Objectives

Why exercise in the first place? The primary objective is to ensure that the plan works when it is needed.  But it is not enough to exercise parts of a plan. Ideally all elements of disaster recovery and business continuity plans should be exercised at least once a year if not quarterly. Each exercise may have different objectives, beside the primary one.

Main exercise objectives include identifying weaknesses and shortcomings, verifying recovery objectives and procedures, validating global efficiency of plans, verifying the adequacy of emergency operations centers (EOCs) and alternate sites, and achieving specific recovery time objectives (RTOs) and recovery point objectives (RPO).

How much should you test?

Tests can be simple or complex. A table-top exercise can establish a plan performance baseline. A specialized test, such as one which focuses on crisis management procedures at an EOC, provides valuable information about specific activities. At a higher level, an integrated exercise can address multiple disaster recovery and business continuity plans or plan components. Finally, an entire plan, with all components, can be exercised. It is far better to err on the side of exercising too much, rather than not enough.

Managing human resources

Tests present human resource issues. Tests are important for validating team member expertise and identifying training opportunities. Conversely, people could refuse to work overnight, weekends or be away from home even a few days. Be sure to discuss and resolve these issues with human resources management.

During disaster recovery and business continuity plan tests, it is good practice to treat team members well, especially when they are away from home or working difficult hours. Be sure to budget for appropriate hotel accommodations and food, while managing costs.

Effective test strategies

The test options will help improve disaster recovery and business continuity plans and train staff. But no matter how often you exercise plans, when reality strikes, your response capability could be much different than in the exercises.

Key strategies for testing include starting simple; raising the bar in terms of difficulty; involving vendors and stakeholders in exercises; making objectives increasingly difficult to achieve; and launching surprise exercises. When launching an exercise program, start with plan reviews and walk-thrus. This will help staff get comfortable with the exercise process. As they improve, increase the level of exercise complexity. Remember that if an exercise fails, it is not a failure; rather, it is a success. It is far better to identify systems and procedures that may fail, and rectify them, before a real incident occurs. Finally, a true test is to launch a surprise incident. This will truly test how well prepared the organization is to address a real incident.

What is a successful test?

The primary reason to exercise is to identify limitations of disaster recovery and business continuity plans. Recognizing that most organizations change frequently, even mature business continuity plans may be inappropriate in a given situation or at a given time. Tests that appear to be successful and uncover no problem should be suspect. Maybe the objectives were too easy or the situation was unrealistic. Exercises present opportunities to fix problems before a disaster happens.

A successful test uncovers and documents problems. Once the problems have been fixed, consider running a follow-up test to ensure the repairs work. Measuring the success of disaster recovery and business continuity tests means having relevant objectives that will help uncover problems. Testing is your chance to push your disaster recovery and business continuity plans increasingly closer to the reality of a disaster.

US Smart Grid Could Cause Business Interruptions - Disaster Planning Consideration

A cybersecurity coordination task force released a report that assesses various security and privacy requirements for the U.S. Smart Grid, as well as strategies needed to address them. It looks at security and disaster planning issues.

The 256-page document was compiled by the task force, composed of individuals from the government, industry, academia, and regulatory bodies, and led by the National Institutes of Standards and Technology (NIST). Now open for comment, NIST will release a final version of the document in March 2010 describing a overall Smart Grid security architecture and security requirements.

The draft report highlights the need for planners to address threats that could potentially allow attackers to penetrate the smart grid, gain access to control software, and alter load conditions to cause widespread disruptions. Cybersecurity strategies for protecting the smart grid need to address not only deliberate attacks but also inadvertent compromises resulting from user errors, equipment failures and buggy software, the report said.

Released as part of the report was a Privacy Impact Analysis that examines some of the privacy implications of establishing a smart grid for power distribution.

A smart grid uses digital technology to transmit, distribute, and deliver power to consumer in a more reliable and efficient manner than traditional electricity systems. A key component of the smart grid is the real-time, two-way communication it establishes between consumers and power distributors for tracking energy use and enabling smarter consumption and pricing. Current plans call for nearly 17 million two-way connected smart meters to be installed in U.S. homes over the next few years.

Disaster Recovery Planning International Standard Set by Janco

Disaster recovery continues to be an area of high risk and high cost

A recent survey by Janco Associates showed that organizations of all sizes considered that the loss of IT systems was the threat most likely to have an impact on costs and revenue and that it is the most commonly experienced disruption.

The regulated nature of the IT environment, combined with the statutory obligations of clients' data protection, means that having a disaster recovery system in place is essential. Until now, enterprises of all sizes have faced enormous costs and inflexible regimes to implement effective IT disaster recovery provisions. Many have therefore been forced to settle for a mere plan of action or ineffective options, which may in reality, do little to reduce their risks. So what are the options for protecting critical IT systems for your firm?

Have a backup

Most organizations take backups, but it is the barest minimum requirement for protecting your firm from a disaster. Backups are for getting you out of a hole when you accidentally delete/lose/corrupt data on your working machines. If you lose those machines completely then the backup will only help once you have replaced and rebuilt your systems. In addition, replacing and rebuilding is not as simple as it sounds and can take a long time before you have working systems again.

CIOs should also know that taking a backup is not the same as having a good working backup. Backup processes have a reputation for letting enterprises down when they need them most. If the recovery plan in based on backups only, CIOs should check regularly that backups are actually working and understand that they have only covered the first step and plan to be without working systems for typically around 3 to 7 days).  Also, remember that if you want to guard against a disaster that physically destroys your machines, then your backups need to be off-site - well out of harm's way.

How to calculate the cost of downtime

One overlooked truth is that downtime costs accelerate in a non-linear fashion every hour. If a system fails for five minutes, the costs are fairly low because manual methods (paper and pencil) of making records or communicating by telephone instead of e-mails can suffice to conduct business. Over an extended period, however, the volume of work overwhelms the manual processes. Yet some businesses -  such as Amazon or e-Bay - cannot run at all on manual processes. Business and financial operations increasingly deteriorate, and the rate of dollar losses grows - sometimes to the point of fatally damaging the business.

In addition, when assessing the financial impact of downtime, you need to consider factors such as potential lost revenue, reductions in worker productivity, and damaged market reputation. In some cases, downtime can even reduce shareholder confidence, which can create unnecessary and unplanned costs. Financial analysts and accountants at your company can help you come up with the factors at your company that are affected by downtime and contribute to its costs.

Swine Flu - DRP - BCP - CIO Issue

What swine flu has done is reminded us all of the necessity to plan for threat scenarios that affect people more than they do data centers and other physical corporate facilities. Alternate work area facilities, mobile recovery units, and other workforce recovery strategies are not effective when people are home sick or there are travel bans in place. In these scenarios, your workforce recovery strategy must rely on remote access solutions or virtual workforce solutions.

Large numbers of employees out sick will affect the business (revenue) and cost your company a lot of money in productivity loss (you still pay employees their salary when they are out).  In a recent Janco Associates survey, they asked over 300 DRP/BCP decision makers if their company had strategies for workforce recovery in their BCPs, 71% said yes. This means that 29% of you out there have a lot of work to do. Of the 71% that have strategies in place, 82% use remote access procedures as part of their strategy.

The US Center for Disease Control (CDC) has confirmed thousands of cases of swine flu in the United States and as other countries including Canada, New Zealand, the United Kingdom, Israel, Spain, and all of Europe has confirmed cases. This means health officials have confirmed that the disease can spread person-to-person and has the potential to cause "community-level" outbreaks.

IT disaster recovery is not necessarily business continuity.  In addition there is a good chance that the plan is out of date and that it has not been exercised in a long time.

A plan walk through is no substitute for a more thorough exercise but it is a good place to start.

• Validate the currency of the plan and the procedures.
• Validate team member, roles, and responsibilities.
• Understand what technology and services you currently have in place.

Disaster Recovery Business Continuity for Remote Offices

Data residing outside the data center at remote and branch offices (ROBOs) accounts for a significant portion of an enterprise's information store, yet it often either is protected with inefficient backup processes or is not protected at all -- leaving companies at risk on many fronts.

In a recent research report, high priority projects for ROBOs included improving information security measures; ensuring compliance with government, industry or corporate governance mandates; and improving Disaster Recovery Business Continuity processes.

Which Files Need to be backed up

Hard drives often contain hundreds of thousands of files. Many of them should be backed up every day, others only occasionally, and still others - including temp files, the hibernation file, and your browser cache--not at all.

• Documents: You should back up your word processing files, spreadsheets, and similar documents every day. Most basic backup program perform incremental backups, in which the program copies only the files that have changed since the most recent previous backup. (Several backup programs also perform versioning;  they keep several iterations of the same file on hand and enable you to choose which version to restore.)
• Recent Documents: If your backup program can handle incremental backups, you don't have to worry about recent documents as separate entities. But if you often work on these files on other people's computers, you may want to carry a copy of them on a flash drive or store a copy of them online.
• Application Data: Applications create and maintain data files such as e-mail messages, browser favorites, calendar entries, and contacts that require daily backing up. Many programs store them in a hidden folder inside your user folder (in XP, C:\Documents and Settings\your name\Application Data; in Vista, C:\Users\your name\AppData). Also, in XP, Microsoft stores Outlook and Outlook Express data in C:\Documents and Settings\your name\Local Settings\Application Data). Fortunately, any well-designed backup program intended for everyday, nonexpert users (as opposed to IT departments) knows where to look for Outlook data.
• Operating System: You can always reinstall Windows and your apps, if you have the original discs or can download the programs. But if Windows becomes unusable or your hard drive crashes, switching to a system backup (also called a disaster recovery backup) that you create a couple of times a year can get your machine up and running smoothly without much effort.
• Media: These large files require a separate backup strategy because of the amount of storage space they require..
• Heirlooms: Files that you want to keep forever need backing up and extra protection.

Budget cuts impact disaster plans

IT staff cuts spurred by the economy are likely to continue throughout the remainder of the year. According to a survey of 300 IT center managers last year, half of all data centers were planning to cut 2009 budgets by an average of 15%. Respondents at 14% of those companies said the cuts would include layoffs of IT staffers.

The PayPal electronic payment system is one of many Internet-based services that have been hit with outages. And based on news reports, the number of such incidents appears to have been increasing in recent months, analysts said. They cited shutdowns of the Google Apps software hosted by Google Inc., outages at data centers run by Rackspace Hosting Inc. and a distributed denial-of-service attack on Twitter.

Observers pointed to several possible reasons for the apparent uptick in online outages, including IT budget and personnel cutbacks, increasing corporate dependence on hosted applications -- and bad luck. Companies are not doing the maintenance we should be doing, and when you do not do maintenance, they increase the probability of catastrophic failure.

UK Pandemic system for disaster fails

The UK Government has rolled out the National Pandemic Flu Service in England today. Scotland, Northern Ireland and Wales have decided to opt out of the service as demands in numbers are significantly less than that of England.

According to the BBC, the UK may have over 100,000 cases of H1N1 infection along with roughly 30 deaths as a result. The US is reported to have 40,000 cases with over 250 deaths. But because the flu pandemic has spread so far and wide, it is difficult to determine whether someone’s death is a direct result from swine flu, or whether the figures and statistics are accurate. There are simply too many cases and not enough resources being spent on data collection; some would say at least governments have their priorities right.

The National Pandemic Flu Service will be primarily a web based service, alongside a call center which will not be operated by health staff or qualified professionals to allow an "ease of burden on the NHS". It will act as a checklist service that algorithmically determines whether your symptoms are severe enough to require Tamiflu, the main anti-viral drug used to combat the illness.

Backup Window Must be Planned For

Rather than add more bandwidth, or invest in expensive, dedicated storage networks, WAN optimization can improve IP network performance sufficient to turn recovery into continuity. To help meet the objectives outlined above, a WAN optimization solution must be able to do three separate tasks for true business continuity: restrict bandwidth to backup applications during the allowed window and allocate it to critical applications in the event of a disaster, overcome latency and bandwidth limitations on the wire, and provide acceleration to roaming or displaced users redirected to alternative data sources.

 

 

 

Regardless of whether the data is being replicated from a massive cabinet, over IP-based storage or off a user’s hard drive for compliance purposes, during the backup window maximum bandwidth should be available to ensure completion. This requires granular bandwidth management that can isolate applications on the network and provide a predictable, policy-based service level. Further, the solution should be able to distinguish between a user initiated file copy and one started by the backup daemon, and apply different bandwidth allocations to each.

 

 

 

Also, the solution must remove latency and protocol inefficiencies that constrain current WAN backups. Caching and compression technology combined with inline protocol optimization of commonly used file transfer protocols form a technology suite that improves the performance characteristics of a WAN, adding bandwidth and reducing the time needed to complete backups and restores. Moreover, it should be able to do this for individual devices and accommodate displaced and roaming users without the need for bulky appliances.

- more info

---

What is the optimal method of back up for an enterprise's disaster recovery plan?

The Backup and Backup Retention policy is an 11 page sample policy that is a complete policy which can be implemented immediately. 

The document is provided in both Word 2003 and Word 2007 format and is easily modified. 

Solution	Benefit	Cost
Local Backup	Shorter backup times Reduced bandwidth	More hardware and staff Security risks
Central Backup	Less hardware and staff	Increased bandwidth costs Increased backup times
Central Backup	Shorter backup times Reduced bandwidth Less hardware and staff	One-time technology investment

 

 

 

 

 

- more info

---

Roles in Developing a Disaster Recovery Plan

The disaster recovery policy must be reviewed at least annually to assure its relevance. Just as in the development of such a policy, a planning team that consists of upper management, and personnel from information security, information technology, human resources, or other operations should be assembled to review the disaster policy. Roles and responsibilities of the planning team should be as follows:

• Perform an initial risk assessment to determine current information systems vulnerabilities.
• Perform an initial business impact analysis to document and understand the interdependencies among business processes and determine how the business would be affected by an information systems outage.
• Take an inventory of information systems assets such as computer hardware, software, applications, and data.
• Identify single points of failure within the information systems infrastructure.
• Identify critical applications, systems, and data.
• Prioritize key business functions.

The Disaster Recovery Plan Template has tools that can be used immediately and defined in detail all of these responsiblities and provides a work plan that can be use as is.

- more info

---

Questions to Ask About Your Disaster Plan

Does your datacenter have the right procedures and equipment in place to recover your business from a disaster? Can your business survive extended downtime without your computing resources? Is your company prepared for a planned D/R event? What about an unplanned event?  Who's in charge?  Which technicians are driving the project?

A real disaster recovery effort is much different from a test.

• People work around the clock in cramped quarters, getting very little sleep.
• There often are too many people involved in the data cener leading to questions of who is in charge.
• There are not sufficient LAN drops for the all the necessary technicians to be on the network simultaneously. 
• The equipment being using needs to be refreshed, so there is an equipment refresh along with a data recovery, which posed additional problems during the recovery.

- more info

---

Credit Card Processor Disaster

Talk about a serious outage. Payment gateway service provider Authorize.net was down several hours. The service is used by tens of thousands of e-commerce vendors to accept credit card and electronic checks payments on their websites.

A fire in Seattle’s Fisher Plaza appears to be the reason what has taken down Authorize.net.

With its website down, Authorize has set up a new Twitter account to provide updates and address the many customer complaints and questions.

On July 2nd at approximately 11:10 pm, an incident in a garage-level electrical room disrupted power to Fisher Plaza East and knocked out the facility's backup generation system. The electrical room is where Fisher Plaza East receives its power from Seattle City Light.  One of the services affected was Authorize.net, the largest credit card and e-check payment processor in the world, with tens of thousands of partners and processing millions of transactions on a daily basis. Authorize.net set up a Twitter account to keep its customers informed and transaction processing has been restored with a backup data center. ARB transactions will be rerun over the weekend thought there are still issues with CIM, VPOS and api.authorize.net.

 

- more info

---

Maximum Tolerable Period of Disruption (MTPOD) is an issue

The concept of Maximum Tolerable Period of Disruption (MTPOD) is an issue with the introduction of British Standard 25999-2.  When applied appropriately, MTPOD will improve management's understanding of your disaster recovery business continuity program and clarifies your enterprise's recovery priorities.

BS 25999-2, Section 4 says that the goal of a business impact analysis is to "determine the impact of any disruption of the activities that support the organization's key products and services." A key aspect of determining the impact of a disruption is identifying what BS 25999 calls the "Maximum Tolerable Period of Disruption," or MTPOD. BS 25999 defines MTPOD as the "duration after which an organization's viability will be irrevocably threatened if product and service delivery cannot be resumed."  MTPOD is the maximum amount of time that the organization's key products or services can be unavailable or undeliverable before its stakeholders realize unacceptable consequences.

The full application of this concept can mean rethinking how a business impact analysis  is approached. While many DRP / BCP professionals start a business impact analysis   by gathering data from individual departments, MTPOD forces them to first look at products and services. Disaster Recovery and Business continuity professionals should understand downtime tolerance, taking into account:

• Customer expectations
• Regulatory requirements
• Reputational issues
• Financial and operational impairment
• Strategic consequences.

Based on management input, disaster recovery / business continuity professionals can propose preliminary Maximum Tolerable Periods of Disruption for key products or services within the scope of the business continuity program.

Once MTPOD is established for key products and services, the traditional business impact analysis  or service. From there, the business impact analysis  can either validate or disagree with preliminary MTPOD conclusions. In addition, the business impact analysis  does identify the department, function and process details that are needed to achieve the MTPOD.

Perhaps most importantly, the disaster recovery / business continuity professional must understand the amount of time required to perform the process or activity in order to deliver the product or service to its key stakeholders (internal or external). This is referred to as cycle time. For example, in a manufacturing company, cycle time would be how long it takes to obtain the necessary stock, manufacture the product, and deliver it to the customer.

With an understanding of MTPOD and cycle time, the business continuity professional can identify what is commonly accepted as the core output of the business impact analysis   - the recovery time objective, or RTO. RTO is the point in time following a disruption when operations must resume (at a minimum level) in order to meet downtime tolerances.

- more info