NVCC IT Disaster Recovery Plan

This disaster recovery plan describes the methods and procedures to be used by Northern Virginia Community College (NVCC) in order to safeguard and restore business operations in the event of a disaster.

Requirements for disaster recovery success

For disaster recovery to be as quick and inexpensive as possible, college management must be aware of, and supportive of, the unique requirements of a disaster recovery operation.

· Disaster recovery procedures must take precedence over existing college procedures when needed to insure timely restoration of key college services.

· The disaster recovery efforts of personnel on disaster recovery teams must take precedence over their normal business operations. Personnel may need to be assigned full time to disaster recovery efforts until the project is complete.

· Needed funds for disaster recovery efforts must be released in a timely manner to support the disaster recovery project.

· Purchasing efforts must be streamlined as much as possible in time of emergency to limit productivity losses in the college due to waiting for a procurement action to occur.

· Members of the disaster recovery teams must have all necessary training and equipment needed to carry out the disaster recovery procedures.

· Users must continue work if an alternate procedure is available to perform the work, even if the procedure is less efficient than the one they are accustomed to.

Lack of management support in any of these key areas has the capability of severely impacting disaster recovery efforts.

5 levels of disaster

IT related disasters can be categorized into five different levels:

1. Partial server component failure

2. Total server component failure

3. Total server failure

4. Multiple server failure

5. Loss of physical plant

Levels 1-3 are addressed by redundancy in server components and live backup of data between servers on a nightly basis. Also, enough spare capacity should be maintained so that key services from a failed server can be restored on other servers on other campuses in the college.

This disaster recovery plan addresses levels 4 and 5. When there is loss of physical plant or multiple servers fail in a disaster (regardless of the type of disaster), generally there will not be enough spare capacity to quickly restore services on existing equipment. This plan contains the information and procedures needed to restore services after a level 4 or 5 disaster.

Likelihood of different types of disaster

Disasters may be caused by any number of reasons. It is helpful to keep in mind when creating disaster plans the most likely causes of disaster. According to industry literature, here is a list of the seven most likely cause of disaster.

1. Flood

2. Equipment outage

3. Power outage

4. Fire/Explosion

5. Earthquakes

6. Hurricanes/Tornados

7. Building facilities problems

Since no campus of NVCC is currently in a low lying or flood prone area or an area with a high likelihood of earthquakes, these two types of disaster are less likely in the case of NVCC. Therefore this disaster recovery plan is written with equipment outage, power outage, fire, hurricane/tornadoes and building facilities problems in mind. Although procedures specific to each disaster type are not listed, the general procedures given should work well with any of these types of disasters.

DATA PROCESSING/BUSINESS ENVIRONMENT

NVCC currently utilizes both networked servers and mainframe applications for its day to day business activities. The mainframe systems are maintained by the VCCS, so this disaster recovery plan will focus on recovering the NVCC network. The NVCC network is based on Microsoft Windows servers being accessed from Windows 98, Windows 2000 and Windows XP on the end users workstation.

Logical Drawing of NVCC Data Network

See NVCC WAN attachment.

Physical Drawings of NVCC IT Networks

See NVCC Physical Network attachments.

Backup Locations

Data Backups:

A full backup to disk will be performed daily on every server and domain controller. The backups shall stored locally on the same server. An tape backup the of local servers’ disk backup shall be performed daily to store the disk backup files on tape. In addition to the disk backup, the mail server will be backed up directly to tape. On Fridays, the full set of the tapes from the Thursday backup will be transferred to an offsite location for storage. All tapes will be labeled with date and time of backup. A minimum of 2 months or 8 weekly backup sets will be stored at the offsite location. Tapes older than 8 weeks may be reused. Tapes will be transported in a proper tape carrying case to protect against damage and loss of data. The tapes will treated as any other confidential data. Tapes will not be left alone in any unsecured area, or any area where damage to the tapes may occur. Backup tapes will be stored in data grade fire safes.

Backup work areas for end users:

Depending on the severity of the loss of service, end users will be relocated to a different work area, building or campus. Relocating the end users will be the primary focus of the Customer Support team. If the loss of service is localized to a particular area of a building, end users will be moved to other areas of the campus. If the loss of service is to an entire building or campus, users will be relocated to other buildings or campuses as necessary. When a disaster occurs at one of the college sites, employees will have the ability to relocate the office to work in the office of another specified employee.

Backup areas to hold classes:

Classes will be moved to unaffected sites in the event of a disaster, per the college disaster recovery plan (which is separate from the IT disaster recovery plan).

Network Equipment List

Here is a list of equipment we currently use on the NVCC network:

Cisco 7206 Routers
Cisco 3660 Routers
Cisco 3640 Routers
Cisco 1010 ATM Switches
Cisco Catalyst 6509 Switches
Cisco Catalyst 4006 Switches
Cisco Catalyst 3524 Switches
Cisco Catalyst 3548 Switches
Cisco Catalyst 3550 Switches
Cisco Catalyst 3508 Switches
Cisco 515E Pix Firewalls
Cisco VG248

Network Servers

Applications (Prioritized List)

See Business Impact Analysis document.

DISASTER RECOVERY TEAMS

Disaster Planning Coordinator

The disaster recovery coordinator is the Director of IT Support Services. The responsibilities of the Disaster Planning Coordinator are:

Serve as the primary contact and coordinate the recovery effort.
Contact all support personnel involved in the recovery effort.
Provide all support personnel with a copy of the plan.
Contact the following individuals as soon as possible: College President, Provosts, IT Director, etc.
Maintain the disaster recovery plan.

Emergency Management Team

The members of the Emergency Management Team are the following:

Network Services Manager
Telecommunications Engineer
Server Engineer
Security Engineer
Infrastructure Engineer
Database Engineer
Campus IT Managers
IT Purchasing Specialist

The responsibilities of the Emergency Management Team are as follows:

Assessment of the damage.
Provide a detailed status of the disaster to the disaster planning coordinator as soon as possible.
Contact all vendors, contractors or external resources necessary to restore services to the damaged areas.
Provide a general status of the disaster to college personnel.
Determine the priorities. There should be a minimal accepted time frame the college will function with degraded operations before the backup plan is implemented.
Ensure all needed support staff have been contacted to provide assistance.
Determine a general time frame for when all services will be restored.

Technical Support Team

The members of the Technical Support Team are the following:

Currently the Emergency Management Team
Other personnel as needed

The responsibilities of the Technical Support Team are:

Determine what computer hardware/software has been damaged.
Review the risk assessment analysis and business impact analysis and determine what the critical/non-critical applications are and to determine whom is responsible for each application.
List procedures to create a new environment for the hardware or for the purchase of new hardware.
List procedures to restore critical software/applications.
List procedures to restore non-critical software/applications.
Contact application owners to determine their role in the recovery process.

Special Projects Team

The members of the Special Projects Team will be:

ITSS Secretary
Campus IT Staff

The responsibilities of the Special Projects Team are:

Provide transportation to/from backup facilities.
Make any necessary telephone calls.
Order supplies, complete necessary paper work, provide assistance as required to all support groups.

Customer Support Team

The members of the Customer Support Team are:

IT Customer Services Manager
IT Security Officer
Campus IT Manager and Staff

The responsibilities of the Customer Support Team are as follows:

Notify computer customers of the disaster and give them a time frame for recovery.
Help customers develop manual procedures to accomplish work if resources are unavailable for a long duration of time.
Have customers list the priority of their day to day work.

Formal Declaration of Emergency

In the event of a disaster, the disaster planning coordinator should be contacted. The coordinator should contact the emergency management team. Designated members of the emergency management team should go to the area of the disaster, assess the damage and provide the coordinator with the results of the assessment as soon as possible. The disaster planning coordinator should decide which other teams to contact depending on the type and severity of the disaster. Disaster recovery operations should not begin until the coordinator has designated the plan of operation.

Working with the VP of Instructional and Information Technology, VP of Financial & Administrative Services, VP of Student and Academic Services and the College President, the Disaster Planning Coordinator shall determine when a disaster has in fact occurred and make a formal declaration of emergency in order to release needed funds and begin disaster recovery procedures

EMERGENCY RESPONSE PROCEDURES

In any emergency the following general procedures should be followed:

During an emergency

1. Ensure safety of individuals by evacuating premises if necessary.

2. If personal safety is not threatened, secure equipment to minimize damage from the disaster.

3. Notify the correct authorities by pulling fire alarm, calling 911, calling campus police, etc. as needed depending on the type of emergency

4. Notify the NVCC IT Help Desk as to the loss of computer services.

5. The NVCC IT Help Desk will notify the Disaster Planning Coordinator that a disaster may have occurred.

6. Working with the College VP’s and College President, the Disaster Planning Coordinator will declare that an emergency exists and will begin disaster recovery procedures.

After an emergency

1. After the cause of the disaster has dissipated, members of the emergency management team will inspect the damage and report findings to the Disaster Planning Coordinator.

2. Backup sites will be set up and begin operations if necessary to support college business until the primary site is restored to service.

3. Network services shall be restored in priority order as much as possible with functional equipment on hand. Key data and systems shall be restored before less critical data and systems.

4. The additional equipment required to bring the network to full operations shall be listed and ordered from vendors listed in the disaster recovery plan.

5. Once equipment has arrived and the network has been restored to full operations, then the backup site shall not be used any more and end users will return to their offices and assume normal business procedures.

6. The Disaster Planning Coordinator, working with the College VP’s and College President, will formally declare that the disaster recovery efforts are complete.

Procedures for specific degrees of disaster

This section will address major failures such as multiple servers, entire buildings or entire campus. Minor failures, such as single server or partial server are not covered in this document because of the use of redundancy and backups.

Loss of Multiple Servers

· In the event that multiple servers are lost and cannot be repaired in one day or less, data will be temporarily restored from tape backup to another server. Users will be notified of new location for data. Changes to login scripts and drive mappings will be done as necessary.

· If the server(s) that were lost were one of the following; mail server, web server, WINS, DNS or DHCP server, they must be built on a new spare server. These servers need to operate on separate servers. If replacement servers are to be built in same location they originally resided, no changes to firewalls, DNS or routing will need to be done. If the site is not safe or unusable, replacement servers will have to be built at another location. Depending on which location is chosen, changes to firewalls, DNS and routing will have to be made. (for information on IP addresses and DNS information, refer to the NVCC Network Configuration manual).

· Restoration of mail servers could take 3 – 5 days to complete, because of the database integrity checks that must be performed after the backup has been restored.

II. Loss of WAN Services

· The NVCC IT Help Desk shall call the VCCS Customer Service Center to inform them of the loss of WAN connection.

· Individual campuses shall set up dialup connections to the Annandale modem server to allow a small bandwidth network access between the college campuses.

· Annandale shall procure and dial up an Internet Service Provider (ISP) to establish a basic Internet connection.

· All users should be notified that the network is running at reduced bandwidth and they should limit interactions that run over the WAN.

III. Widespread Virus Problems

· The virus should be isolated and identified as soon as possible.

· Once identified, a removal method should be determined. Users should then be notified as to what to the characteristics of this virus, how to detect the virus, and how to remove the virus.

· An updated virus image file should be obtained from the college virus protection software vendor in order to prevent spread of the virus in the future.

MAINTAINING THE PLAN

Maintenance of the plan is the responsibility of the Disaster Planning Coordinator. The steps the coordinator should take to maintain this plan are as follows:

· Ensure that the plan is updated at least once per year.

· Ensure that the plan is tested at least once per year.

· Ensure that the plan is signed-off by the V.P. of Instructional and Information Technology, V.P. of Financial & Administrative Services, V.P. of Student & Academic Services, and the College President

· Update the plan at the earliest convenient time after significant network or business changes have occurred that affect the plan.

· Make sure plan is stored in a secured area accessible to the Disaster Planning Coordinator and ensure that at least one copy is kept at the Provost office on each college campus, at the admin council member offices, and at the College Presidents office in secure locations.

Appendix A. Contact Information

Key Services

VCCS Customer Support Center

(540) 591-5861 or (804) 423-6757

Equipment Rooms

Annandale

Server Room – CG 402D

Main Data Closet – CG 110A – 703-323-3517

Alexandria

Server Room – AA433 - 17004 (internal only)

Main Data Closet – AA235 – 703-845-6453

Brault Annex

Server Room & Main Data Closet on 6^th Floor

College Staff

Server Room & Main Data Closet – CH105

ELI

Server Room & Main Data Closet – Storage area

Manassas

Main Data Closet - MH 225 - (703) 368-6689

Server Room - MH 222B – 46501 (Internal VOIP)

703-392-0495 (Verizon)

Medical Center

Main Data Closet (DMARC Only) 110

Server Room – 325B – 703-822-6670

Loudoun

Main Data Closet - LC116

Server Room - LC228

Woodridge

Main Data Closet WO117 – 51033 (internal only)

Server Room WO326

College Computer Center

Server Room – CT 231 –

703-323-2419

703-323-2577 (Verizon)

Campus IT Managers

Name Work Home Cell Phone

Annandale

Bruce Ghofrany (703) 323-3257 (703)501-2605

College Staff

Peter Tharp (703)323-3705 (703)864-5479

Alexandria

Dave Babel (703) 845-6019 (703)501-3648

Medical Center

Jeff Howlett (703)822-6666 (703)509-1642

Loudoun

Kevin Kelley (703) 450-2569 (703)795-0818

Manassas

Lynn Bowers (703) 257-6652 (703)409-1799

Woodridge

Lynn Feist (703) 878-5659 (703)973-0064

Campus IT Staff

Alexandria

Tariq Arsalayee (703)845-6117

Abdul Askaryar (703)845-6118

Max Alleyne (703)845-6048

Vandarine Ky (703)845-6042

Annandale

Bang Nguyen (703)323-3847

Howard Treichler (703)323-3138

Claire Weaver (703)323-3191

Annie Law (703)323-4272

Brault Annex

Dave Campbell (703) 536-1180

College Staff

P-14’s (703)323-3705

ELI

Ella Moore (703)323-3512

Financial Aid

Kirsten Delashmutt (703)323-2183

Loudoun

Andrew Hayden (703)450-2660

Scott Wood (703)450-2584

Mark Worthington (703)450-2507

Manassas

Frank Dreistadt (703)257-6651

Justin Sowa (703)257-6510

Mary Lou Davison (703)257-6653

Medical Center

Lyle Fanelli (703)822-6669

Woodbridge

Charles Tulowetzke (703)878-5675

Frank Echevarria (703)878-5675

NVCC Technical Support Services

Director

Allen Sinner (Acting) (703)323-3338 (703)307-3543

TSS Secretary

Helen P. Anderson (703)323-3278 N/A

IT Purchasing Specialist

Janet Zilczer (703)323-3742 N/A

Chief Network Engineer

Allen Sinner (703)323-3338 (703)307-3543

Network Engineer

Athar Hameed (703)323-3311 (703)209-0909

Customer Service Engineer

Rae Smith-Watson (703)323-3595 (703)254-8416

Security Engineer

Randy Dotson (Acting) (703)764-6033 (703)463-6677

Infrastructure Engineer

Cap Bromley (703)323-3735 (703) 627-8627

Database Engineer

Tony Vu (703)323-3345 (703)501-3385

Telecommunications Engineer

Choon Lee (703)323-3308 (703)609-8731

Hardware/Software Vendors

ATS

Computer Room Maintenance

810 Cromwell Park Drive

Suite A

Glen Burnie, MD 21061

Fax: (410) 487-0071

Phone: (410) 487-0081

Mobil: (410) 591-5645

24 HR #: (301) 210-3528

Black Box Corp.

Datacomm. equipment/connectors/cables

1000 Park Drive

Lawrence, PA 15055-1018

FAX : (724)746-0746

Phone: (724)746-5500

Cable Express, Inc

Datacomm. equipment/connectors/cables

33 Collegeview Road

Westerville, OH

FAX: (800)875-2225

Phone: (800)783-0222

Computerware, Inc.

Madge equipment, printers, Compaq notebooks, and hardware/software in general

8480-I Tyco Road

Vienna, VA 22180

FAX: (703)821-1746

Phone: (703)821-8200

Dell Marketing

Dell Servers

ONE DELL WAY

ROUND ROCK, RX 78682

FAX: (800)365-5329

Phone: (800)274-7799

HCI Technologies

EIC software, Dialogic hardware

11414 Sunset Hills Rd.

Suite 300

Reston, VA 20190-5207

FAX (703)736-3019

Phone: (703)736-3000

Micro Center

3089 Nutley St.

Fairfax, VA 22031-1931

Phone: (703)204-8400

VASI

Attn: Desi Arnaiz

9034 Mathis Ave

Manassas VA 22110

Phone 703 330 8268

Fax 703 330 6698

Virginia Cable Specialties

7823-AA Loisdale Road

Springfield VA 22150

703 541 2727

703 541 2722

Juanita Byerly

Datacom Network Enterprises

Attn: Joe Hall

301 Maple Ave W. Poplar Building C

Vienna VA 22181

Phone 703 255 0879

Fax 703 255 4762

Hardware/Software Technical Support

Vendor Phone FAX Representative

ADC Kentrox (800) 733-5511

APC (UPS) (800)555-2725

Ariel (800) 422-4633

Autodesk (800) 225-1327 (415) 507-4936

BlackBaud (Raiser's Edge) (800) 468-8996 Selma Permenter

Blackbox (724)746-5500 (724)746-0746

CACI (703) 875-2900

Cisco TAC (contract # 1276587) (800) 533-2447

College Board (PowerFaids) (703) 707-8999

Dell (800)-822-8965

FileMagic (800) 949-3453 (203) 483-3350

HCI (EIC) (877)-848-1973 (703) 736-3019

HP (970) 635-1000

McAfee (800)338-8754

MicroTest (800) 526-9675

MS (# 0005017629-M401) (800) 936-5800

NCS (ED Express) (800) 330-5947

T2 Systems (800) 434-1501 (905)389-1444

Computer Rental Vendors

Affordable Computer Rentals

Leesburg, VA

703-858-1327

AvalonTC

45880 Shepard Drive, Unit 12

Sterling, VA

Phone: (800)355-1197, (703)430-8649

FAX: (703)430-8290

MacHeaven

4500 Daly Dr.

Chantilly, VA 20151-3701

703-378-1201

Rent-A-Computer, Inc.

Arlington, VA

703-525-0455

Rent-A-PC, Inc.

(877)736-8272

Rentex

Washington, D.C.

(800)574-1702