During the last few weeks we get more and more alerts from our beta tester regarding the Thunderbird e-mail client. The problems are not OS specific. Some testers described them as "like having a virus in the system that occupies ALL the resources - CPU, memory, disk and network".
Indeed, PerfProtector detected Thunderbird as the source for this problem. Turns out that almost all of its statistics were significantly deviated from the known profiles of Thunderbird behavior.
It looks as someone replaced Thunderbird with another application that has a completely different behavior.
However, this new behavior doesn't look right at all. It dramatically affects the performance of the clients while having many negative side effects. The machines are hogged for hours - the CPU utilization is very high, the memory consumption increases continuously, the hard disk constantly performs many read/write operations and gigs of data are transferred from the network side. And on top of that, it is impossible to use Thunderbird at all - can't read e-mails, can't send e-mails, can't reclaim the inbox...
It sounds like a very aggressive virus/Trojan that took control over the machine and then went completely out of control.
What was common between the infected machines? Thunderbird 3.
Users, which used for years previous versions of Thunderbird, said that once it was upgraded (automatically) to version 3, their machines were stuck for days. Fresh installs of Thunderbird 3 produced the same behavior.
What went wrong from version 2 to 3? How bad it is?
To understand the exact problem we reproduced it in our lab. We set up Linux (Ubuntu) and Windows (XP, Vista and 7) machines and tested them first with Thunderbird 2 (version 2.0.0.24) and then with Thunderbird 3 (versions 3.0 and 3.1.1). On each machine we configured a GMAIL IMAP account with a real mailbox that has about 9,000 e-mails from the last 4 years where the total size of the mailbox was less than 5 GB. This mailbox reflects an activity of about 5 e-mails per day with an average size of about 500KB per e-mail.
Then, the machines were monitored by PerfProtector.
The following graphs compare the behavior of Thunderbird 2 to Thunderbird 3 during the first 48 hours after their fresh installation. In each graph, the blue line represents the behavior of Thunderbird 2 the red line represents the behavior of Thunderbird 3.
CPU Utilization:
While CPU utilization of Thunderbird 2 is usually between 0% to 10% with an average of 0.3%, Thunderbird 3 CPU utilization is between 5% to 80% with an average of 30% - 100 times more than Thunderbird 2. In addition, during long periods of time, Thunderbird 3 used more than 50% of the overall CPU resources. This behavior slows dramatically the whole machine.
Memory Consumption:
Thunderbird 3 memory consumption is twice as Thunderbird 2 consumption.
Read Operations from Hard Disk:
While Thunderbird 2 performs minor read operations from the hard disk, Thunderbird 3 performs thousands of read operations during long periods of time.
Thunderbird 3 reads gigs of data from the hard disk during long periods of time. Thunderbird 2 reads almost none. This behavior affects the response time of the hard disk and the behavior of other applications that read from the hard disk.
Write Operations to Hard Disk:
While Thunderbird 2 performs minor write operations to the hard disk, Thunderbird 3 performs thousands of write operations during long periods of time.
Thunderbird 3 writes gigs of data to the hard disk during long periods of time. Thunderbird 2 writes almost none. Similar to the read behavior, it affects the response time of the hard disk and the behavior of other applications that write to the hard disk. In addition, it occupies gigs of free space from the hard disk.
Network Activity:
While Thunderbird 2 performs minor download operations from the Internet, Thunderbird 3 performs thousands of download operations during long periods of time.
Thunderbird 3 downloads gigs of data from the Internet during long periods of time. Thunderbird 2 downloads almost none. This behavior affects the response time of the Internet connection. It hogs the connection and slows dramatically the Internet activities. In case your Internet is paid according to the bandwidth that you use, your bill will increase significantly.
We can see that Thunderbird 3 has a completely different behavior than Thunderbird 2. While Thunderbird 2 behaves like a normal balanced application, Thunderbird 3 has a sure anomalous behavior - its deviations from normal behavior are crystal clear. Thunderbird 3 reclaimed the CPU, reclaimed the memory, reclaimed the hard disk and reclaimed the network.
What is the source of these problems?
These problems are due to a combination of two features:
- The first feature is the Global Search and Indexer. This feature is new in Thunderbird 3. It enables fast search of e-mails in the mailbox. However, Thunderbird has first to index all the e-mails and this process is time and resource consuming. It took Thunderbird 3 days to index our mailbox. During this time, its CPU utilization was between 5% to 80%, its memory consumption was between 100 to 150MB and it made thousands of read operations from the hard disk. In addition, even after the indexing was completed, we still noticed that Thunderbird 3 continue to index and re-index from time to time thus consuming more resources from the machine. Moreover, according to Thunderbird's official site, "if you enable Global Search/Indexing it normally uses about 3.5 KB per message in the SQLite database". So our 9,000 e-mails database should theoretically consume about 30MB. However, in reality it consumed 150MB and it keeps growing and growing.
The worrisome thing about this new feature is that it is turned on by default. It happens either when you migrate from Thunderbird 2 to 3 or when you install Thunderbird 3 from scratch. This feature has a huge impact on the behavior of the client: for at least couple of days, the machine is hogged from all directions and the user is helpless. A typical user can't understand what is going wrong with his machine and why he can't do even simple tasks. A new feature, which has this major side effect, shouldn't be turned on by default. It's completely wrong. Users around the world spent hours and days on finding the source of this problem. They conclude that it's due to this new feature in Thunderbird 3. The funny thing is that we didn't find any significant difference between searching with or without indexing . It seems that for a typical user, this feature is not needed.
- The second feature is the Message Synchronization of IMAP accounts. Thunderbird synchronizes the IMAP folders and saves the messages locally on the hard disk of the machine. It means that all your IMAP accounts reside on the machine hard disk. This feature was already in Thunderbird 2 but its default was turned off. In Thunderbird 3, its default was changed to be on. In addition, if Thunderbird 2 was migrated to Thunderbird 3, this feature was turned on automatically, even if it wasn't turned on in your Thunderbird 2 settings! This feature has a huge impact on the computer resources since it downloads all your e-mail messages and stores them on the hard disk of the machine. It means that it uses your Internet connection to download gigs of data and uses your hard disk to store them. It took Thunderbird 3 days to synchronize our mailbox and during this time it hogged our Internet connection by downloading our entire mailbox while making thousands of write operations to the hard disk. Since Gmail maps messages to labels, multiple copies of the same message are held when synchronization is enabled, thus increasing the occupied disk space. As a consequence, our 5GB online mailbox increased to 40GB of data that were downloaded from the Internet and were stored on our hard disk without getting any warning alert from Thunderbird!
This feature may be useful to many users but since it affects dramatically the resources of the machine, it shouldn't be turned on by default. Some of our machines, which have "only" 30GB of free space, were stuck during the synchronization operation. Thunderbird used the whole disk space and not even a single sector was left free.
How to solve these problem using PerfProtector?
The solution is very simple. In the main dashboard of PerfProtector, right-click on the Thunderbird alert and choose either "Disable Global Search and Indexer Functionality" to disable the global search or the "Disable Message Synchronization Functionality" to disable message synchronization.
How to solve these problems manually?
To disable the Global Search and Indexer functionality:
- Open the options window. On Windows, click Tools-> Options. On Linux, click Edit-> Preferences.
- Click on Advanced -> General -> Advanced Configuration.
- Uncheck the Enable Global Search and Indexer checkbox.
To disable the Message Synchronization functionality:
- Open the accounts settings window. On Windows, click Tools-> Account Settings. On Linux, click Edit-> Account Settings.
- For each IMAP account that is configured in your Thunderbird, choose Synchronization & Storage and under Message Synchronization uncheck the Keep Messages For This Account On This Computer checkbox.
I was a big fan of thunderbird. I use Mac OSX and have my own postfix/dovecot server on FreeBSD. Thunderbird 3 is just plain garbage, aside from what you have listed here it's also plagued with bugs.
ReplyDelete1) When I first upgraded it was randomly marking messages in my mailbox as unread. I had to turn off whatever "condstore" is in some buried and hard to find advanced section.
2) Every time I close it the process spins consumes 100% of one core until I kill it. This also just randomly happens while it's running.
3) I moved a folder on a different mail client (same account, this is IMAP after all) and thunderbird not only did not recognize the change and keep insisting the folder is in the old location but then it just constantly errors "this folder doesn't exist".
Maybe some of this has been fixed in newer versions (I pretty much gave up immediately after trying 3.0..) but really...really... and this is just off the top of my head! how can you take such a fantastic mail client and ruin it so bad... Lets not even get in to performance issues with trying to index my entire mailbox.. I only have two things to say to that..
$ du -sh Maildir
3.5G Maildir
$ find Maildir | wc -l
227015
You characterize this change somewhat incorrectly... and by using google as your IMAP source, you're also invoking a worse case scenario (that is: least benefit for most cost).
ReplyDeleteWith a modern system running windoze 7, 150meg of memory isn't really an issue and for systems with 2 to 8 cores, occupying one (only for a few days, mind you) is also insignificant. Even the downloading of mail is skewed by your analysis (by the use of google tags and a large persistent mailbox where nobody deleted things). The costs of running thunderbird, will only be significant for a small minority.
The benefits, too, are skewed in your analysis. Google is fairly good at searching the mailbox (searching is their business, after all). Many IMAP systems are less spry at searching. I have found that the new search in thunderbird3 to be amazingly faster at searching my mailbox (which is much bigger than yours). In fact, I let thunderbird index everything and only prevent it from downloading a few overcrowded and rarely used sub-mailboxes.
Maybe you can make a case for the options to be more clear, but I don't have any issues with the default values. Simply put, thunderbird 2 didn't have these capabilities, so thus it didn't have these options. Part of what convinced me that the upgrade to thunderbird was a good one was the speed at which it searched.
First: I, personally, use the KMail for all my mail needs (well, I use the 'Kontact' KDE PIM suite for everything - but KMail is the mail client it embeds) and it has some similar problems because of the indexing, but...
ReplyDeleteI actually install Thunderbird for relatives that use Windoze and several of them that I first introduced to Thunderbird back with version 2 actually insist on staying with it. Many of them refuse to touch version 3 - complaints of "it doesn't work right", "it's slow" and many others are all provided as reasons why.
So regardless of what the anonymous poster has said, this article does seem dead-on when it comes to the problems Thunderbird 3 has.
We have about 500 users running thunderbird 2.
ReplyDeleteIf we were to upgrade to thunderbird3 we would need to buy another 500 cores to run tasks on the machines
because they do use the CPU when the user is idle.
That would cost us about $100,000 to $200,000.
Then we would need to spend another couple hundred thousand on file servers to deal with the additional network traffic for the user's home directories and switches and power to run 500 cores will run about $500 a month. I think we will hold off on upgrading.
I can not see why it needs to use 100 times the CPU
for a new "feature".
This hasn't been my experience at all. While I don't have an inbox as large as yours, I use Thunderbird 3 daily to check 3 IMAP mailboxes and 2 POP3 accounts.
ReplyDeleteMy CPU stats are nowhere near consumed and the app routinely uses under 100mb of RAM.
The caching features increased performance, instead of requiring a new connection to the mail server. I don't miss v2 at all.
Interesting..I guess I haven't either paid attention ...or haven't noticed any slowdown.
ReplyDelete100+ users also haven't reported any issues.
We are using TB v3.x on
OS X (Tiger, Leopard, Snow Leopard)
Windows (XP SP3 & 7)
Linux (Redhat & Gentoo)
Mail Server running Gentoo/Postfix/Courier-Imap
Avg. Mailbox size 4G
Avg Msg Count 13,000 per mailbox.
Hardware:
Dual QuadCore XEON
8G Ram
Raid1(Serial SCSI) for System
Raid10 for Mail store
No problems at all using thunderbird.
What? taking 1 core completely out of a 2 core system is non-issue! You probably don't do anything else other than checking emails and browsing the web ;)
ReplyDeleteI'm actually quite grateful for all the bloat in TB3. I've been using TB since it was known as Netscape Mail/News and for each new release the damned thing kept getting more and more annoying.
ReplyDeleteThe TB3 "global search" fiasco finally pushed me over the edge. I spent a weekend learning Mutt, and haven't looked back.
They have a slogan of sorts, which sums up my experiences quite well: "All mail clients suck. This one just sucks less."
If you're used to TB it does take some getting used to, but if you're prepared to spend 5 or 10 hours learning a new client, you'll not regret it. ..
Just one hint; if you do decide to give it a go, use the manual found on dev.mutt.org, the manual on www.mutt.org is outdated!
I noticed the indexing issue months and months ago(around the time the release of Fedora 11 or 12) and it took me a while to figure it out but it was only an issue at the time because I didn't know it had been enabled and I have a rather large inbox(some of it dates back to 1992 when I first got online if that tells ya how much there is) which didn't help doubly so since I needed every byte of my bandwidth at the time.
ReplyDeleteAll I did was turn off the indexing and the subscribed folders for syncing, then turned the folders on one-by-one to sync when I had the free bandwidth. It actually runs a LOT better than with V2 where I could NEVER search the bodies of my messages...now it's a breeze.
Viruses are self-replicating programs. This only sounds like a virus to the ignorant. Trojan's involve intent. Do you really think the intent is to harm your computer? Modern viruses and trojans try not to slow down your computer because they don't want to be detected. Sounds even less like a virus or a trojan. Really, get a clue about your terminology
ReplyDeleteDespite disabling indexing and sync across the board, I've had the slowdown problem since the first version of TB 3. On Linux (several dists, multiple versions each), Mac (couple OS versions), and Windows (XP/2003/7).
ReplyDeleteTB used to be good. But it has lost focus. When TB came out, the downgrade was so severe that I exported all my years' of stored email to a neutral format the very next day. I've actually gone back to Outlook (gasp!) on Windows (++gasp!) and Evolution on Linux.
ReplyDeleteGreat work guys, it's the most professional analysis about TB I read so far.
ReplyDeleteThumbs up!
And for the one who wrote "Do you really think the intent is to harm your computer? ... Really, get a clue about your terminology",
why are you so hard on them. They used the virus as a metaphor - they didn't say that it intends to harm the computer: "It sounds like a very aggressive virus/Trojan that took control over the machine and then WENT COMPLETELY OUT OF CONTROL". Just a metaphor and a bit of cynicism ;-)
Anyway, if this your comment, are missing the point of this article.
What software did you use to generate these graphs?
ReplyDeleteWhat shocks me the most is that there are still people using desktop mail clients. Wow...
ReplyDeleteI use Thunderbird 3 on a netbook and have not seen any of these issues. I also use it on my desktop, and my wife uses it on her laptop. Those systems run fine as well. I don't think Thunderbird is the issue.
ReplyDeleteI have had excellent success overall with Thunderbird. I really like how I can take my Linux ~/.thunderbird folder in its entirety and copy it to a Windows system and put that folder in the right place, right name and run the program and all my stuff is there - mail, accounts, passwords et al. I always find portability to be important. When the upgrade occurred from 2.x to 3.x, it told me about the indexing and the downloading of IMAP and I specified to not do those things (well, the imap sync I left for a little while and quickly decided I did not want the local storage consumption - this is why I have IMAP email at a hosted web site to begin with) and all continues to be well. I am a very happy Thunderbird user for years now. While I read this article with interest and curiosity, it in no way compels me to even consider moving to something else. Thanks for listening ;-)
ReplyDeleteKAS
I wondered why my Thunderbird install at work was running super slow for about a month now. Now I have some indications what happened.
ReplyDeleteMy work machine had about 1 GB in pop emails on a P4 2.8 Northwood with 1 GB of DDR. It finally pushed me over the edge and running on my x200s only (U9400 cpu at least has two cores, in case one gets pegged).
This so inspires me to be pissed off again.
ReplyDeleteI've been writing so many bug reports against TB3 -- it's so crapified....it never should have been released -- they don't have things working from TB2 *STILL*, that I used every day. I tried upgrading for the 3rd time a few weeks ago -- trying to get used to it --- but I wondered today why logging off my DOMAIN server took over an hour today -- It had 1.8G of local IMAP mail stored from my *server* -- which I keep on my server so I won't waste space on my Windows machine!!! IDIOTS!... I have about 150 folders -- all were on server, except for a few commonly used folders.
I wish they would dump 3.0 and go back to working on 2.0.
If they had done 3.0 as *totally* compatible with 2.0 and let users turn on features as they wished, it might have grown into a usable product. But instead, they have tried to force users to do things their way without our input!
Idiots! Idjots and Idiots!
Why do large companies start acting so *paternal* and ignoring their users and instead start thinking they know what's best for their users better than their users.
I DID see a performance issue. I already did as you have suggested. You just confirmed it.
ReplyDeleteThank you taking the time, where as Mozilla couldnt be bothered.
I didn't noticed any performance degradation. But the message synchronization had cost me lots of money, since I use mobile wireless network to access the Internet!
ReplyDeleteAm running Thunderbird 3.1.1, one IMAP account with 2.8 GB mail, on Fedora 13 and i have not noticed the slow down. Specs: 64 bit, 4 GB ram, Dual Core
ReplyDeletemy comp was slow for one day when i upgraded.
ReplyDeleteand really guys. did you read the manual for tb3? the faq?
it's clearly explained inthere why it does things, and why it might slow down initially.
personally I have not experienced anything beyond that initial indexing.
http://support.mozillamessaging.com/en-US/kb/FAQ+Upgrading+Thunderbird+2+to+3
for those not capable of pushing F1 in TB
Biggest issue I have with TB3 is that IMAP download is on by default.
ReplyDeleteWhile it isn't a big problem on my private pc, it made it almost impossible to log into my account at work, because the quota there is limited.
TB3 had done a great job filling that up...
I would like to see Mozilla's response to this article as it really is as serious as you make it out to be.
ReplyDeleteI have an entire company of people all using Thunderbird 2. There is no way we could possibly handle all these users upgrading to Thunderbird 3, so our choice is to either wait for Mozilla to do something about it, or find another client (of which there are very few reasonable choices).
What do we do, Mozilla?
I have been Using TB3 since its release and noted all kinds of bugs, random crashes especially. They have been making it more stable and it does not crash as much anymore but there are some other issues. I have not authorized my company employees to upgrade from the TB2 that they use because I fear this would cause productivity problems for my company. I hope they fix it someday because it is my favorite mail app.
ReplyDeletehello,
ReplyDeleteI've also noticed that indexing takes a long time at first
I've been using thunderbird 2.0 for years
I've tried TB3.0 but it was missing the local search I was used to so I directly jumped from 2.0 to 3.1
I've completely migrated to TB3.1 and I'm pretty satisfied with the upgrade.
I was already downloading all messages with TB2.0 but TB3.1 does it better.
After the indexing which takes a long time, it runs ok
also TB3.1 can see mail coming directly via server filter to sub folder much better than TB2.0
Working with tab for mail is also good.
It's not perfect but I still think people should upgrade to 3.1 so that thunderbid developpers can concentrate ressources on the next version, TB2.0 being very old code from a maintenance/security point of view.
It would be good to have a light mode for thunderbird which would allow TB to be used for example in a mobile environment maximising ressources used on the server.
We have some hundreds of users of 3 without any issues.
ReplyDeleteAnonymous, but not a troll. It works fine - but then we don't have any local mail.
Man!
ReplyDeleteDo we "really" need this potions?, at least in my office everyone creates folders to organize everything and it's "very" strange for someone to search for something in the mail (as it might be far easier to just ask someone who has the information at hand)... on the other hand... keeping a copy synced in the local drive when the IMAP works fine as it is doesn't have much sense unless you want to be able to work offline (for example if you have a laptop) in which case you should be able to optionally set this option.
Some folks have posted that they have some huge number of installations to babysit, and that upgrading tb to 3.x would cost them millions, and that the world would end, etc.
ReplyDeleteIf you are babysitting these workstations, can't you just set the configuration to disable these features as part of the upgrade?
I've used tb for years and years and when I went to move my profile dir from one machine to another, I realized it was HUGE (this was before the 3.x upgrade). I deleted the content directories in order to move the profile and noticed that tb was re-fetching everything. I did a bit of research, found that it was caching everything locally, and disabled mailbox synchronization. Problem solved.
Yes, I might be more willing to research and troubleshoot a problem such as this than a lay user, but I'd like to stress two points:
1. These "problems" aren't new
2. These "problems" aren't unsolvable
Finally, the Mozilla team (whether it be Firefox, Thunderbird, or their other fine products) is constantly under fire by a huge population of Internet trolls, Microsoft fanboys, and frustrated users. It's not surprising that they take a while to get back to everybody's complaints. Yes, these features impact performance, but their analysis was that the performance penalty (which is temporary, remember) was worth the added benefit.
Just my two cents.
-chris
I have TB3 on the same machine as my IMAP server. I have the spam filtering on (though smapbayes catches most of it before) and I have about 3 rules for highlighting messages. When mail arrives, I get full-core CPU usage for about 3 seconds, and the TB UI is unresponsive (i.e., can't type in message composition windows). I turned off global indexing already.
ReplyDeleteI use Thunderbird 3.0.6 with Xubuntu and the "Enable Global search and Indexer" was unchecked by default here.
ReplyDeleteI've been using TB3 for a while now on Windows7 and its been stellar. The indexing feature is really handy since it lets me global search through ALL my accounts (I have 3 IMAP and 1 gmail) in no time.
ReplyDeleteAs for CPU, when its just running in the background, its almost always at 0 (don't have decimal precision in task manager). When I'm doing email related work, yes the CPU spikes up, but only for a second or two, which I can totally deal with.
Oh, as for the offline feature being turned on by default, I don't think you are even considering people located outside the US (and maybe other developed nations), where Internet is not always as reliable, not always on, and not unlimited. So, the initial download is a problem, yes, but then being able to have all my messages available offline makes it worth the initial wait.
I have been using TB for quite a while on my linux boxes. I have not noticed a slower systems, nor have seen TB using more memory and processor use. I have one gmail account at 130MB with 1197 msgs in INBOX, 4 other IMAP boxes for it, I do have Indexing turned off, I am syncronizing all messages and boxes.
ReplyDeleteI also have a gmail box with 177 msgs in the INBOX and 4 other IMAP boxes. Two other accounts, one with 2.3G and 34 different IMAP folders all syncronized. The other with around 20G, and over 200 boxes.
I have all my mailboxes syncronized, and have the indexing turned off.
Thank you for saving me time and money. If they must include global indexing it should be switched off by default and the user should be warned it will eat up CPU. I saw an immediate drop in my idle CPU usage from about 10-15$ down to 2-4%
ReplyDeleteI operate more than 10 imap mail accounts and my PC was continuously reading/writing to disk which was getting really annoying probably contributed to occasional performance and hanging issues. I still sync most of these accounts locally but it appears disabling global indexing has immediately resolved my problem. Thank you very much. John
ReplyDeleteOne more thing mates. Don' forget the Compact function in order to restore to the old size. :)
ReplyDeleteGood analysis, thanks.
ReplyDeleteVery wrong decision of Thunderbird team to use such default values in new software release. I wonder whether there are other hidden wrong decision in this process of 'upgrading'.
Nice post, I'm running Thunderbird with IMAP on kubuntu here, and it kept running with full CPU load and crashing almost instantly until I turned off Global indexing
ReplyDeleteThis is how I finally solved my performance issues. I have a few google apps imap accounts in Thunderbird 3. Both have excess of 80,000 emails in the Inbox and about 50,000 in the Sent.
ReplyDeleteI tried making new profiles, compacting, turning off Global Search, using gmail to archive a bunch of the emails, getting rid of Lightning calendar. The problem was switching from one folder to another would take minutes and usually just freeze TB altogether.
Solution: change the Folder view from Unified to All Folders. That one simple thing is what finally did it. GRRRRR.
This definitely helped my situation (on osx using 10 imap email accounts). But also very suspect were two add-on, Remove Duplicates, and Send Later. when I turned off global indexing, synchronizing, and these two add-ons I go the cpu usage down from ~30% to 3%. saving me at least an hour of battery life.
ReplyDelete-Paul