Quest logo

Quest Technology Management

Helping clients manage their technology for over 30 years.

Master your disaster: Investing in your disaster recovery plan

(reprinted with permission from California Banker Disaster Recovery Special Issue Vol. XXIX, 2007)

by Mike Dillon, Chief Technology Officer, Quest

Think your disaster recovery plan is up to the test? Consider this Northern California investment trust whose executives thought their disaster recovery plan prepared the organization for any emergency.

The Infamous Backhoe Incident

The firm was, as one would expect in California, ready for an earthquake, but what shook it up was something on a much smaller scale: the Infamous Backhoe Incident. The accidental cutting of a power line during construction of a nearby shopping plaza brought the business to its knees.

This crippling eight-hour outage unearthed the company’s vulnerabilities. The Backhoe Incident caused a loss of communications that put tens of millions of revenue dollars at risk — a tough way to discover that one’s disaster recovery plan cannot get critical systems back online in a timely manner.

Fortunately, this story has a happy ending: today, the systems needed to ensure the firm’s essential communications are now duplicated at an offsite Disaster Recovery Facility.

Even a small disruption of operations can be costly, not just in productivity and dollars, but also in customer satisfaction and a bank’s reputation. Whereas some businesses can get away with extended periods of downtime — eight, 24, even 48 hours — before real damage is done, this is not so for banks. When a customer wants his or her money, a bank risks a very great deal if it has to say, ‘Come back in a day or two.’

Clearly, maintaining the continuity of business operations is impossible without a proactive plan to recover from disasters large and small. And, of course, your institution has a disaster recovery plan. After all, bank board members and senior managers are required by law to identify, assess, prioritize, manage, and control risks.

Complex by any measure

As financial institutions become ever more dependent on digital technology and data to do business, disaster recovery is no longer just about getting computer systems working again — it’s about quickly and efficiently restoring and recreating interrelated business processes.

By any measure, this is a complex task, even for a small institution — and it’s made notably more complex as information once resident on paper becomes digitized data that must be housed in digital storage devices and networks and is accessed by desktop PCs and department-level systems. These new technologies open financial institutions to an ever-changing array of threats and vulnerabilities, which is why a disaster recovery plan has become a regulatory requirement.

Some disaster recovery plans, however, are better than others. A disaster recovery plan must be encompassing enough to deal with not just technology breakdowns like power outages and systems crashes but also cataclysmic events (fire, flood, and so on) and human behavior, such as robbery, strikes, or hacking.

And to stay in business, you’ll need more than backup copies of your data safely stored away on a tape drive. To restore the business processes necessary to sustain operations after a compromising event, your institution needs to:

  • Replicate its systems in their entirety — including all the safeguards and procedures necessary for regulatory compliance — so you can continue to operate if your facilities are unusable or if just your data was compromised,
  • Maintain backup and storage procedures, so constantly changing data is safely and consistently preserved according to regulatory requirements,
  • Conduct continuous monitoring of systems for performance and intrusions, and
  • Keep informed, via reports and/or alarms, of current conditions while you make the changes necessary to get your business processes back online.

Needed: disaster recovery expertise

Crafting a truly effective disaster recovery plan — one that enables your financial institution to continue operating in the face of technology breakdowns, cataclysmic events, or people behaving badly — requires not just commitment. It takes skill and experience that comes with expertise.

So how good is your disaster recovery plan? Even if your plan meets all regulatory requirements, if you can’t answer yes to these questions, your bank may be more vulnerable than you think:

  • Does your disaster recovery/business continuity plan address what happens when whole departments suffer an outage?
  • Does it determine how work and information will flow from department to department, from platform to platform, from site to site?
  • Does it consider how to cope when the people in charge of your systems are stranded and can’t initiate the plan or verify that systems are up and running?
  • Has it been updated since that new branch was opened or since completing that recent acquisition?
  • Has your disaster recovery plan been tested in the last six months?

These are just a few of the more general issues that a solid disaster recovery/business continuity plan must address. The unique needs and circumstances of each financial institution will drive many more.

Getting prepared, getting secure:

Today’s disaster recovery options

So where should you start? You need to learn a little something about the key options in a disaster recovery plan. That way you’ll be able to work with a disaster recovery/business continuity expert to develop the best plan possible.

Backup services. Tape backup isn’t dead. Indeed, it’s getting faster all the time, though disk backup is faster still. Meanwhile, data backup software now does compression and data deduplication (saving only a single iteration of a file or block and providing pointers to duplicates), thus reducing the number of disks needed and reducing storage costs.

Because of compliance mandates, a bank’s business units should be involved in determining how stored data is backed up, retained, secured, and retrieved. This is something a qualified disaster recovery expert will see to. Other issues, like power, cooling, and disk reliability, can be avoided by outsourcing data backup functions to a qualified managed services provider.

Remote data backup. Regulations require that banks back up data remotely to protect it against regional disasters. Among the technologies used in remote data backup are mirroring, replication, snapshots, and the tracking and recording of changes to data on the fly in real time, called continuous data protection.

Replication services. Once a disaster recovery expert has figured out your bank’s requirements, you can use replication services — which continually back up data to remote sites so that no single point failure will threaten data safety — to efficiently replicate data at the block level, reducing network traffic.

When it comes to recovery, data replication services can have your systems and processes operational within minutes. Your institution’s historical email and data can quickly be synchronized. Applications such as spreadsheets, databases, CRM, and more will run as if nothing ever happened, and your branches and locations will communicate and conduct business effortlessly.

Co-location. When multiple enterprises locate business-critical network, server, and storage gear at a managed services firm’s data center where this equipment (and its resident applications and data) is linked to a variety of telecommunications services, the cost and complexity of remote data backup can be cut significantly, thanks to the sharing of data center and telecom infrastructure. Also, network latency is reduced, as are traffic backhaul costs.

Electronic vaulting. A flexible service enabling financial institutions to maintain duplicate data and systems at a highly secure, remotely located recovery site, an electronic vault offers automatic backup. Data reduction and encryption techniques enable data to be securely moved offsite using little network bandwidth. And data is automatically restored on demand while a secure offsite backup is maintained.

Disaster recovery services. It’s easy to sign up for disaster recovery services that maintain your backup and storage procedures, monitor your systems for performance and intrusions, and keep you informed through reports and/or alarms — all while you make the changes necessary to get your business back online.

Making sure you’ve configured these services to meet your bank’s actual needs is a bit tougher. It takes a careful assessment by an expert who will conduct a thorough audit of you storage, backup, and restoration components, followed by a gap analysis. It’s important that before you sign onto any service, you engage an expert to provide:

  • Analysis of growth trends supporting a roadmap for your storage, backup, and recovery planning initiatives,
  • Identification of existing and potential vulnerabilities,
  • Documentation of system flows, and
  • Optimization of storage, backup, and recovery practices through design, development, deployment, and rollout.
  • Hoping nothing bad will happen is not a strategy. Every financial enterprise needs to proactively prepare for when things go wrong. That means building a solid plan to ensure fast and efficient data backup, disaster recovery, and replication capabilities.

Related Disaster Recovery Solutions:

Contact Information

Barbara Klide