Difference between revisions of "WSO Server Restoration January 2007"

 
(8 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= WSO Server Restoration January 2007 =
+
[[Category:WSO]][[Category:History]]
  
== Synopsis ==
+
For up-to-date information on the restoration efforts and server situation, please see <http://wso.williams.edu/wso/services/restorejan07>.
  
<div style="margin: 10px 0px; padding: 0px 8px 8px 8px; border: 1px solid #495; background: #9da; float: left;">
+
== What happened? ==
'''Server Status: ''Working'' '''
 
</div>
 
<div style="clear: both;"></div>
 
As of 8:20 PM Wednesday January 3, 2007.
 
  
We are currently running from a copy of the backup, with Irene (our iMac) standing in for Ursula (our Xserve). All services should be functioning "normally" from the backup (in their September 9th state), but they may be slower than usual, though Irene hasn't had much trouble with the load yet.
+
WSO's user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the RAID (the software system designed to prevent data loss due to the physical failure of one of the machine's hard disks) had been corrupted. After numerous troubleshooting attempts and support calls, we concluded that it was unrecoverable and restored from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006. We are currently running with all normal services but with data from this backup. ''This means that all changes since September 9th have been lost permanently.''
  
== What happened? ==
+
After further conversations with Apple support, WSO determined that the Xserve (the user server) had hardware issues with either the CPU or the logic board.
 
 
WSO's user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the software system designed to prevent data loss due to the physical failure of one of the machine's hard disks had been corrupted. After numerous troubleshooting attempts and support calls, we concluded that it is unrecoverable and restored from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006. We are currently running with all normal services but with data from this backup.
 
  
 
== What was affected? ==
 
== What was affected? ==
  
The following are affected:
+
The following were affected:
  
 
* user accounts (including passwords, email forwarding, and files)
 
* user accounts (including passwords, email forwarding, and files)
Line 23: Line 17:
 
* email lists
 
* email lists
 
* A ''very limited'' amount of email may have been lost on December 31. Since then mail to any address @wso.williams.edu may have bounced (the sender would receive an undeliverable mail message) due to Ursula's downtime. Mail sent between December 31 and January 3 may be delayed.
 
* A ''very limited'' amount of email may have been lost on December 31. Since then mail to any address @wso.williams.edu may have bounced (the sender would receive an undeliverable mail message) due to Ursula's downtime. Mail sent between December 31 and January 3 may be delayed.
* The password reset tool
 
  
The following services have NOT been affected:
+
 
 +
The following services were NOT affected:
  
 
* WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
 
* WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
Line 32: Line 26:
 
== What does this mean for users? ==
 
== What does this mean for users? ==
  
All data on the server now exist as they did on September 9. Any changes made after September have been lost.
+
All data on the server after the crash and restoration exist as they did on September 9. Any changes made after September have been lost permanently.
  
 
* Any accounts created after September 9 no longer exist.
 
* Any accounts created after September 9 no longer exist.
Line 39: Line 33:
 
* Organization websites will look as they did on September 9. (Databases have not been affected).
 
* Organization websites will look as they did on September 9. (Databases have not been affected).
  
 +
== More Info ==
  
== What is WSO doing? ==
+
For up-to-date information on the restoration efforts and server situation, please see <http://wso.williams.edu/wso/services/restorejan07>.
 
 
We are currently running a different machine (Irene the iMac) as a replacement for our user server (Ursula). It has files from the September 9, 2006 backup. Changes by users should will be retained here (barring any further disasters), but '''we strongly encourage users to make and keep backups of their personal data or organization website data. We do not have backups set up for the temporary user machine yet.'''
 
 
 
Now that we have a working replacement for Ursula, we will continue working on issues on Ursula. In the next few days we will be working on rebuilding Ursula from scratch to have a machine that we know well at our disposal. When this process is complete, we will switch to the new Ursula. At this point, a short downtime for copying the most recent user data will be needed.
 
 
 
== How can I get my files and settings back? ==
 
 
 
'''It is now fine to proceed here.'''
 
 
 
=== Files ===
 
 
 
Restore any personal backups. We apologize for the inconvenience that the outdatedness of our backup may cause for users.
 
 
 
=== Account info ===
 
 
 
For accounts created after September 9, 2006, please visit the accounts page at <http://wso.williams.edu/accounts/> to sign up for an account (again). '''We have not had the opportunity to test the functionality of this post-restore yet.'''
 
 
 
For email forwarding updates or password changes, please use the appropriate utilities linked from the WSO home page (on the lower left). All of these utilities should now be functional.
 
 
 
=== Reporting problems ===
 
 
 
Please report any other problems, updates, or changes you need WSO to make by sending email to <email>restoration@wso.williams.edu</email>. Also feel free to offer your help.
 
 
 
 
 
We apologize for any inconvenience caused by data loss etc. We ask for your patience as we work to restore our services as best as possible.
 
 
 
Thank you,<br />
 
WSO Staff
 

Latest revision as of 10:34, January 29, 2007


For up-to-date information on the restoration efforts and server situation, please see <http://wso.williams.edu/wso/services/restorejan07>.

What happened?

WSO's user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the RAID (the software system designed to prevent data loss due to the physical failure of one of the machine's hard disks) had been corrupted. After numerous troubleshooting attempts and support calls, we concluded that it was unrecoverable and restored from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006. We are currently running with all normal services but with data from this backup. This means that all changes since September 9th have been lost permanently.

After further conversations with Apple support, WSO determined that the Xserve (the user server) had hardware issues with either the CPU or the logic board.

What was affected?

The following were affected:

  • user accounts (including passwords, email forwarding, and files)
  • organization websites
  • email lists
  • A very limited amount of email may have been lost on December 31. Since then mail to any address @wso.williams.edu may have bounced (the sender would receive an undeliverable mail message) due to Ursula's downtime. Mail sent between December 31 and January 3 may be delayed.


The following services were NOT affected:

  • WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
  • Databases

What does this mean for users?

All data on the server after the crash and restoration exist as they did on September 9. Any changes made after September have been lost permanently.

  • Any accounts created after September 9 no longer exist.
  • Mailing list subscriptions now stand as they did on September 9.
  • Passwords and email forwarding settings stand as they did on September 9.
  • Organization websites will look as they did on September 9. (Databases have not been affected).

More Info

For up-to-date information on the restoration efforts and server situation, please see <http://wso.williams.edu/wso/services/restorejan07>.