WSO Server Restoration January 2007

Server Status: Working

As of 10:20 PM Wednesday January 3, 2007.

We are currently running from a copy of the backup, with Irene (our iMac) standing in for Ursula (our Xserve). All services should be functioning "normally" from the backup (in their September 9th state), but they may be slower than usual, though Irene hasn't had much trouble with the load yet.

What happened?

WSO's user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the RAID (the software system designed to prevent data loss due to the physical failure of one of the machine's hard disks) had been corrupted. After numerous troubleshooting attempts and support calls, we concluded that it was unrecoverable and restored from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006. We are currently running with all normal services but with data from this backup. This means that all changes since September 9th have been lost permanently.

What was affected?

The following are affected:

• user accounts (including passwords, email forwarding, and files)
• organization websites
• email lists
• A very limited amount of email may have been lost on December 31. Since then mail to any address @wso.williams.edu may have bounced (the sender would receive an undeliverable mail message) due to Ursula's downtime. Mail sent between December 31 and January 3 may be delayed.

The following services have NOT been affected:

• WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
• Databases

What does this mean for users?

All data on the server now exist as they did on September 9. Any changes made after September have been lost permanently.

• Any accounts created after September 9 no longer exist.
• Mailing list subscriptions now stand as they did on September 9.
• Passwords and email forwarding settings stand as they did on September 9.
• Organization websites will look as they did on September 9. (Databases have not been affected).

How can I get my files and settings back?

It is now fine to proceed here.

Files

Restore any personal backups. We apologize for the inconvenience that the outdatedness of our backup may cause for users.

Accounts

New Accounts

For accounts created after September 9, 2006, please visit the accounts page at <http://wso.williams.edu/accounts/> to sign up for an account (again). We have not had the opportunity to test the functionality of this after the restoration of our backup.

Passwords

If you need to change the password on your WSO account, use the tools described here.

• Change your password: http://wso.williams.edu/password
• Get a temporary password mailed to you: http://wso.williams.edu/reset_password

Email Forwarding

Change your email forwarding settings at https://wso.williams.edu/updateinfo.html

Mailing Lists

If you need a mailing list owner password reset or a mailing list recreated, please email <email>restoration@wso.williams.edu</email>. If you need to resubscribe to (or unsubscribe from) a list, please visit http://wso.williams.edu/mailman/listinfo/listname, where listname is the name of the list. If this page does not exist, please wait until the list owner for that list has recreated the list.

Reporting problems

Please report any other problems, updates, or changes you need WSO to make by sending email to <email>restoration@wso.williams.edu</email>. Also feel free to offer your help.

We apologize for any inconvenience caused by data loss etc. We ask for your patience as we work to restore our services as best as possible.

What is WSO doing?

We are currently running a different machine (Irene the iMac) as a replacement for our user server (Ursula). It has files from the September 9, 2006 backup. Changes by users will be retained here (barring any further disasters), but we strongly encourage users to make and keep backups of their personal data or organization website data. We do not have backups set up for the temporary user machine yet.

Now that we have a working replacement for Ursula, we will continue working on issues on Ursula. In the next few days we will be working on rebuilding Ursula from scratch to have a machine that we know well at our disposal. When this process is complete, we will switch to the new Ursula. At this point, a short downtime for copying the most recent user data will be needed.

Thank you,
WSO Staff