WSO Server Restoration January 2007

WSO Server Restoration January 2007

Synopsis

Server Status: Working

As of 10:20 PM Wednesday January 3, 2007.

We are currently running from a copy of the backup, with Irene (our iMac) standing in for Ursula (our Xserve). All services should be functioning "normally" from the backup (in their September 9th state), but they may be slower than usual, though Irene hasn't had much trouble with the load yet.

What happened?

WSO's user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the software system designed to prevent data loss due to the physical failure of one of the machine's hard disks had been corrupted. After numerous troubleshooting attempts and support calls, we concluded that it is unrecoverable and restored from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006. We are currently running with all normal services but with data from this backup.

What was affected?

The following are affected:

• user accounts (including passwords, email forwarding, and files)
• organization websites
• email lists
• A very limited amount of email may have been lost on December 31. Since then mail to any address @wso.williams.edu may have bounced (the sender would receive an undeliverable mail message) due to Ursula's downtime. Mail sent between December 31 and January 3 may be delayed.
• The password reset tool

The following services have NOT been affected:

• WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
• Databases

What does this mean for users?

All data on the server now exist as they did on September 9. Any changes made after September have been lost.

• Any accounts created after September 9 no longer exist.
• Mailing list subscriptions now stand as they did on September 9.
• Passwords and email forwarding settings stand as they did on September 9.
• Organization websites will look as they did on September 9. (Databases have not been affected).

What is WSO doing?

We are currently running a different machine (Irene the iMac) as a replacement for our user server (Ursula). It has files from the September 9, 2006 backup. Changes by users should will be retained here (barring any further disasters), but we strongly encourage users to make and keep backups of their personal data or organization website data. We do not have backups set up for the temporary user machine yet.

Now that we have a working replacement for Ursula, we will continue working on issues on Ursula. In the next few days we will be working on rebuilding Ursula from scratch to have a machine that we know well at our disposal. When this process is complete, we will switch to the new Ursula. At this point, a short downtime for copying the most recent user data will be needed.

How can I get my files and settings back?

It is now fine to proceed here.

Files

Restore any personal backups. We apologize for the inconvenience that the outdatedness of our backup may cause for users.

Account info

For accounts created after September 9, 2006, please visit the accounts page at <http://wso.williams.edu/accounts/> to sign up for an account (again). We have not had the opportunity to test the functionality of this post-restore yet.

For email forwarding updates or password changes, please use the appropriate utilities linked from the WSO home page (on the lower left). All of these utilities should now be functional.

Reporting problems

Please report any other problems, updates, or changes you need WSO to make by sending email to <email>restoration@wso.williams.edu</email>. Also feel free to offer your help.

We apologize for any inconvenience caused by data loss etc. We ask for your patience as we work to restore our services as best as possible.

Thank you,
WSO Staff