Every 2 years the HSPH IT Team conducts disaster recovery (DR) exercises during Winter Recess. Conducting a regular DR test reduces the risk of data loss and computing outages to our community and demonstrates the business continuity abilities of HSPH. The excercise allows the IT team to ensure that network and computing services can be restored in case of emergency. In addition, they provide an opportunity to update written DR plans and procedures.
Core HSPH IT services are run from a secure data center located at the main HSPH campus. During a DR test services are shut down and brought back up on duplicate hardware at Harvard’s Disaster Recovery facility in Downtown Boston. After running on the duplicate hardware for three days, the services are transferred back to the primary hardware on campus. The IT Team runs through a series of checklists to ensure that all systems are properly restored and running.
A disaster scenario might involve a fire, a broken water pipe, a security breach, or sliced network cables in the street. HSPH IT partners with HSPH Operations, HUIT, and other teams and vendors to restore service as quickly and securely as possible. Our DR exercise is conducted in a nondestructive manner by gracefully shutting down servers in a coordinated and preferred order before bringing them back up on duplicate equipment. This method takes more time but reduces the risk of data loss to our community.
Here is a recap:
On Friday morning (12/27) at 8:00am:
- 1 hour to shut down production servers
- 1 hour to ensure data synced between data centers
- 1 hour to power on disaster recovery servers
- 1 hour to test and verify operation
- 71 servers moved and tested
- 85TB of data moved
Services tested:
– File and Print services
– Mailman listservs
– Scan-to-email for network copiers
– Local SMTP email services
– FILR
– Powerfaids and NetPartner
– Wasabi
– Master Calendar
– LabPoint for DFCI
– Course Evaluations
– Faculty Activity Report
– CCure and Passage Point
– Administrative forms (account requests, visitor pass requests, computer setup, etc.)
– ICF VDI/Virtual Desktop System
– Staff VDI/Virtual Desktop System
– Kiosk VDI/Virtual Desktop System
On Monday afternoon (12/30) the process was reversed and all systems were tested to verify proper operation.
Thanks to everyone for your patience during the brief interruptions of service.
Please contact our IT Helpdesk at (617) 432-4357 or email helpdesk@hsph.harvard.edu for any questions or concerns you may have regarding this information.
Harvard T.H. Chan School of Public Health
Department of Information Technology
