ist.berkeley.edu/ciocalmailupdates/november-30-2011
IST Service Status November 30, 2011 Update 3:00pm CalMail supports 70,000 accounts over 100 subdomains for students, faculty, staff, emeriti, and retirees, and also provides forwarding services for 140,000 alumni, handling more than 3 million messages a day. The current environment is five-years old and is reaching its normal end of life. The hardware was scheduled to be replaced this year during a normal refresh cycle; however the replacement is expensive (over $1M) and with the acceptance of the OE Productivity Suite project, as well as the strong interest in external services such as Google and Microsoft, the decision was made to pursue those options rather than investing in a platform we would only be shutting down in the near future. This summer, we began evaluating our sourcing options while maintaining CalMail in preparation for replacement. Our original timeline was to continue to support CalMail through 2012 while we undertook the selection and migration to a new platform. Excess demand Unfortunately, usage patterns did not conform to our expectations and demand on the system is significantly greater than anticipated. CalMail, like most email platforms, is impacted by the number of accounts, the number of devices connected to it, as well as how those devices connect. In the past, we had a predictable pattern that closely correlated to the number of accounts. During the last 18 months, we have seen a tremendous increase in the number of unique devices connecting to CalMail. Each of those devices consume connection resources, increasing both the frequency and the number of simultaneous attempts to sync or download mail. Much of this can be attributed to the ongoing explosion in smartphones and mobile devices (Android, iOS iPad, and iPhone), which are often configured to automatically check for new email. The additional load from increased connections combined with an already aging fully utilized environment, made for the slower performance experienced during October. We expected to manage under those conditions until we had a hardware failure on October 25. The storage subsystem that normally would have been able to handle such an outage through its redundant configuration caused CalMail to crash. While the environment was brought up later that day, that one event combined with the increasing high connection load has contributed to the extended outage we are dealing with now. Essentially with each new outage, more messages back up and more people repeatedly check for email, each action driving more load until the system exceeds its ability to keep up with the volume. To try to manage this load, the CalMail team "throttles", or attempts to slow down, those connections (hence the "failed to connect" messages that many people have received) via a rolling connection brownout to make sure everything stays up and all mail continues to be sent and delivered. That was working, albeit unsatisfactorily, until Thanksgiving weekend. More hardware failure On Friday, November 25 at 9:53 am, the storage array had an additional hardware failure. The system crashed as it had in October, and on-call staff proceeded to work on the environment to bring it back up. During that process, the database that maintains all login information and account details (the "who you are" and "where to deliver your emails") was corrupted and couldn't be cleanly recovered. That meant that mail couldn't be delivered and if you tried to log in, and if you could, the system wouldn't know who you were. The database had to be rebuilt from scratch, which required recovering information from backups, then adding in all needed changes from many different log files. It was an exceptionally challenging activity that took a team of people working around the clock from Friday morning until Sunday night to achieve. The primary impact beyond lost access during that period related to outside mail servers having to continually retry sending email (per regular internet mail protocol) until the CalMail environment was again available and messages were able to be delivered. The team completed its work Sunday night around 11:00 pm and confirmed that email delivery queues were slowly coming back under control. Then, at 12:45 am, another disk failed in the storage array. Disk failure is a relatively common occurrence, however the storage environment needs system capacity to be able to recover all the data off of the failed disk and move it to other disks. This would have been managed under normal load, however once the campus community returned from the Thanksgiving holiday, we again had a huge spike with everyone logging in and syncing for the first time since the beginning of the holiday weekend. When combined with the added mobile device demand, it simply overwhelmed the system. The team again throttled connections to allow the system to stay up and deliver the backlogged email more slowly, while the rebuild of the failed disk proceeded. Tuesday, the system simply ran out of capacity and locked up - pushed so far beyond its operating capacity that while it was still running, it was so slow as to be unusable. Processes implemented To allow mail to keep flowing, we have made several difficult decisions that we felt were necessary under the circumstances to both allow access to email while protecting the integrity of each person's email box. First, we encourage students to forward mail in order to remove some load from student email connections off of the server. More than 20 percent of our students are already forwarding from CalMail to other email accounts. Encouraging students to take this action is intended to achieve some "headroom" on the environment while allowing mail to keep flowing. We also shut off the ability to connect from most mobile devices and regular email clients (like Outlook or Apple Mail) via IMAP and POP protocols, and left open the ability for both webmail clients, SquirrelMail and RoundCube, to connect. The result is the system can be effectively throttled while still allowing everyone to access mail and to have mail flowing in and out. It also puts a tremendous unprecedented load on the servers that support the webmail clients, which can result in slow performance. Those 10 servers are being monitored closely and are being tuned and augmented where we can, but system performance will continue to be an issue. Next steps While we continue to pursue the end objective of migration off of CalMail to a third party email solution (either Google or Microsoft), we have multiple efforts underway, each with the objective of supporting the system through the end of semester, at which time we will change out the back end storage to faster equipment. Those interim plans include bringing in some of the world's leading technical experts on email to evaluate what options we have for creating more headroom in our specific email architecture. We are also bringing up an additional email server and storage environment and will be moving some accounts off to create capacity on the primary cluster. Finally, we have engaged with a hardware professional services firm to both assist with maintaining the existing environment and accelerate the implementation of the new hardware, which is expected to arrive late Wednesday night or early Thursday morning. This equipment is very large and contains hundreds of disk drives that must be "burned in" before we can put production email on the server. The risk of moving to it before that testing is completed isn't viable as we would be risking permanent mailbox loss if there was a failure.
|
http://csua.com/feed/