Difference between revisions of "WSO Server Restoration January 2007"

m
 
(19 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= WSO Server Restoration January 2007 =
+
[[Category:WSO]][[Category:History]]
 +
 
 +
For up-to-date information on the restoration efforts and server situation, please see <http://wso.williams.edu/wso/services/restorejan07>.
  
 
== What happened? ==
 
== What happened? ==
  
WSO's user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the software system designed to prevent data loss due to the physical failure of one of the machine's hard disks had been corrupted. After numerous troubleshooting attempts and support calls, we've concluded that it is unrecoverable and we are now restoring from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006.
+
WSO's user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the RAID (the software system designed to prevent data loss due to the physical failure of one of the machine's hard disks) had been corrupted. After numerous troubleshooting attempts and support calls, we concluded that it was unrecoverable and restored from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006. We are currently running with all normal services but with data from this backup. ''This means that all changes since September 9th have been lost permanently.''
  
'''All services on Ursula (organizations, user pages, email, lists, mailman interface, password tools, etc.) are currently off while we do some work.''' -- 11:00 AM Wednesday January 3
+
After further conversations with Apple support, WSO determined that the Xserve (the user server) had hardware issues with either the CPU or the logic board.
  
 
== What was affected? ==
 
== What was affected? ==
  
The following are affected:
+
The following were affected:
  
 
* user accounts (including passwords, email forwarding, and files)
 
* user accounts (including passwords, email forwarding, and files)
 
* organization websites
 
* organization websites
 
* email lists
 
* email lists
* A very limited amount of email may have been lost on December 31. While we cannot guarantee that no mail was lost, the scope of the loss is extremely limited at most.
+
* A ''very limited'' amount of email may have been lost on December 31. Since then mail to any address @wso.williams.edu may have bounced (the sender would receive an undeliverable mail message) due to Ursula's downtime. Mail sent between December 31 and January 3 may be delayed.
* The password reset tool
 
  
The following services have NOT been affected:
+
 
 +
The following services were NOT affected:
  
 
* WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
 
* WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
 
* Databases
 
* Databases
* Email (Any email sent after the crash will wait in queues for 2 days before bouncing if it cannot be delivered via our server.)
 
 
  
 
== What does this mean for users? ==
 
== What does this mean for users? ==
  
When the restoration process is complete, the affected services will appear as they did on September 9. Any changes made after September will be lost.
+
All data on the server after the crash and restoration exist as they did on September 9. Any changes made after September have been lost permanently.
  
 
* Any accounts created after September 9 no longer exist.
 
* Any accounts created after September 9 no longer exist.
Line 33: Line 33:
 
* Organization websites will look as they did on September 9. (Databases have not been affected).
 
* Organization websites will look as they did on September 9. (Databases have not been affected).
  
 +
== More Info ==
  
== What is WSO doing? ==
+
For up-to-date information on the restoration efforts and server situation, please see <http://wso.williams.edu/wso/services/restorejan07>.
 
 
We are currently working to restore the backup and apply all subsequent system updates. Our subsequent plans are currently as follows.
 
 
 
# Restore the backup -- when this is completed, services as they existed on September 9 will resume.
 
# Apply system updates
 
# Restore the password reset tool
 
# Deal with individual requests and other small configuration change which are necessary.
 
 
 
 
 
== How can I get my files and settings back? ==
 
 
 
'''We will make an announcement when you should proceed with the following steps.''' (Any attempts made before then may be a waste of your time.)
 
 
 
=== Files ===
 
 
 
Restore any personal backups. Please remember that WSO does not guarantee the integrity of user data. We apologize for the inconvenience this may cause.
 
 
 
=== Account info ===
 
 
 
For accounts created after September 9, please visit the accounts page at <http://wso.williams.edu/accounts/> to sign up for an account (again). For email forwarding updates or password changes, please use the appropriate utilities linked from the WSO home page (on the lower left). The password reset utility and the tool to update your email forwarding are currently not available but will be soon. Please wait until they are ready (we will announce this).
 
 
 
=== Other problems ===
 
 
 
Please describe any other problems, updates, or changes you need WSO to make on the [[Talk:WSO Server Restoration January 2007 | talk page]]. Please do '''not''' email us, '''unless''' your request involves sensitive personal information... in this case email your request to restoration@wso.williams.edu. (Remember that email to any address at WSO will not be received until our restore is complete.)
 
 
 
Also feel free to offer your help on the Talk page or by emailing us at restoration@wso.williams.edu.
 
 
 
 
 
We apologize for any inconvenience caused by data loss etc. We ask for your patience as we work to restore our services as best as possible.
 
 
 
Thank you,
 
WSO Staff
 

Latest revision as of 11:34, January 29, 2007


For up-to-date information on the restoration efforts and server situation, please see <http://wso.williams.edu/wso/services/restorejan07>.

What happened?

WSO's user and mail server experienced a severe crash on December 31, 2006. After installing a system security update, the server did not finish rebooting successfully. The server has been down or nonfunctional since then. Upon arriving on campus, we discovered that the RAID (the software system designed to prevent data loss due to the physical failure of one of the machine's hard disks) had been corrupted. After numerous troubleshooting attempts and support calls, we concluded that it was unrecoverable and restored from a backup. Unfortunately, and contrary to what the WSO staff thought, our most recent full backup dates to September 9, 2006. We are currently running with all normal services but with data from this backup. This means that all changes since September 9th have been lost permanently.

After further conversations with Apple support, WSO determined that the Xserve (the user server) had hardware issues with either the CPU or the logic board.

What was affected?

The following were affected:

  • user accounts (including passwords, email forwarding, and files)
  • organization websites
  • email lists
  • A very limited amount of email may have been lost on December 31. Since then mail to any address @wso.williams.edu may have bounced (the sender would receive an undeliverable mail message) due to Ursula's downtime. Mail sent between December 31 and January 3 may be delayed.


The following services were NOT affected:

  • WSO homepage and web services (e.g. Willipedia, Discussions, Announcements, etc.)
  • Databases

What does this mean for users?

All data on the server after the crash and restoration exist as they did on September 9. Any changes made after September have been lost permanently.

  • Any accounts created after September 9 no longer exist.
  • Mailing list subscriptions now stand as they did on September 9.
  • Passwords and email forwarding settings stand as they did on September 9.
  • Organization websites will look as they did on September 9. (Databases have not been affected).

More Info

For up-to-date information on the restoration efforts and server situation, please see <http://wso.williams.edu/wso/services/restorejan07>.