WSO Server Restoration January 2007

Updated 10:20 AM Monday January 29, 2007.

Server Status: Working

We are currently running from a copy of the backup, with Irene (our iMac) standing in for Ursula (our Xserve). All services should be functioning “normally” from the backup (in their September 9th state), but they may be slower than usual.

WSO‘s user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the RAID (the software system designed to prevent data loss due to the physical failure of one of the machine’s hard disks) had been corrupted. After numerous troubleshooting attempts and support calls, we concluded that it was unrecoverable and restored from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006. We are currently running with all normal services but with data from this backup. This means that all changes since September 9th have been lost permanently.

The following are affected:

• user accounts (including passwords, email forwarding, and files)
• organization websites
• email lists
• A very limited amount of email may have been lost on December 31. Since then mail to any address @wso.williams.edu may have bounced (the sender would receive an undeliverable mail message) due to Ursula’s downtime. Mail sent between December 31 and January 3 may be delayed.

The following services have NOT been affected:

• WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
• Databases

All data on the server now exist as they did on September 9. Any changes made after September have been lost permanently.

• Any accounts created after September 9 no longer exist.
• Mailing list subscriptions now stand as they did on September 9.
• Passwords and email forwarding settings stand as they did on September 9.
• Organization websites will look as they did on September 9. (Databases have not been affected).

Restore any personal backups. We apologize for the inconvenience that the outdatedness of our backup may cause for users.

For accounts created after September 9, 2006, please visit the accounts page at <http://wso.williams.edu/accounts/> to sign up for an account (again). We have not had the opportunity to test the functionality of this after the restoration of our backup.

If you need to change the password on your WSO account, use the tools described here.

• Change your password: http://wso.williams.edu/password
• Get a temporary password mailed to you: http://wso.williams.edu/reset_password

Change your email forwarding settings at https://wso.williams.edu/updateinfo.html

If you need a mailing list owner password reset or a mailing list recreated, please email restoration [at] wso.williams.edu. If you need to resubscribe to (or unsubscribe from) a list, please visit http://wso.williams.edu/mailman/listinfo/listname, where listname is the name of the list. If this page does not exist, please wait until the list owner for that list has recreated the list.

Please report any other problems, updates, or changes you need WSO to make by sending email to restoration [at] wso.williams.edu. Also feel free to offer your help.

We apologize for any inconvenience caused by data loss etc. We ask for your patience as we work to restore our services as best as possible.

We are currently running a different machine (Irene the iMac) as a replacement for our user server (Ursula). It has files from the September 9, 2006 backup. Changes by users will be retained here (barring any further disasters), but we strongly encourage users to make and keep backups of their personal data or organization website data. We do not have backups set up for the temporary user machine yet.

After talking with Apple support, WSO has determined that the Xserve seems to have a CPU or logic board problem. Apple is sending replacement parts which we will install soon. Once the Xserve is back in commission, we will rechristen her as Viola, and build a new user server setup, copying user data from the temporary Ursula. A short downtime will be necessary, but users should not see any other interruptions of service as a result.

Thank you, WSO Staff