The importance of understanding how to restore data and systems efficiently cannot be understated, even when backup copies or replicas exist.
Ransomware brings disaster recovery planning into the limelight. While it’s easy to believe that your company may not be affected by natural disasters like hurricanes, earthquakes or tornadoes, assuming that it is immune to ransomware is as misguided as believing in a guaranteed lottery win. The likelihood of a company being targeted by ransomware is significantly high, necessitating a sturdy disaster recovery plan enclosed within an incident response plan.
Years of experience in this industry have taught me that simply having backups or replicated copies does not equate to having a comprehensive disaster recovery (DR) plan. I have personally experienced critical system recoveries and seen the fallout when assumptions are made and planning is incomplete. Proper preparation for recovery is vital, particularly in the current climate where ransomware is prevalent.
A well-devised DR plan starts with the assumption that you’ll potentially need to restore data and systems on a completely new infrastructure. The original system might be unavailable due to various reasons such as being retained for forensic analysis, or irreparable hardware damage. Relying on the restoration of physical servers post an attack or disaster is highly risky. The reality is that it might not be feasible, thus, it is essential to be prepared to commence recovery using entirely new hardware.
Procuring standby equipment or failover hosts in advance for temporary deployment post-event is essential. Using cloud infrastructure is an effective method, as you can set up the configuration ahead of time, but only pay for it when necessary. Avoid scrambling to buy replacement servers amid pandemonium. It will only intensify the crisis. Therefore, plan where replacement systems will come from, well before you actually need them.
People often overlook the need for backup at the DR site post-failover. When a switch to a DR site occurs, either on-premises or in the cloud, backup of the new environment should start immediately. The last thing you want is a secondary ransomware wave affecting the DR location due to a lack of backups. Planning to secure recovery hardware and preparing a backup for your recovery site in advance are crucial steps. Design it so that backups begin automatically upon failing over to your recovery site. Don’t consider the recovery complete until legacy backups are reinstated.
Many business leaders will stress the need for immediate recovery with zero data loss. However, they often hesitate when faced with the high costs to meet that requirement. It is important to have necessary discussions regarding true recovery time and data loss tolerance thresholds before a disaster. The more C-suite stakeholders can align organizational priorities and balance business continuity needs with IT capabilities and expenses, the more successful recoveries will be. Remember, success is relative. If the set objectives are realistic and adequately funded, then you are set for success.
Defined objectives must be accompanied by documented and tested runbooks, especially during a crisis. Having backup copies or replicas without understanding the restoration of data and systems at scale becomes ineffective. You need a detailed procedure for recovering assets based on their importance, including system dependencies. The documentation should also include a comprehensive inventory of the recovery environment, a contact list of all staff and vendors, and an issue escalation process. Critical and repeated testing under simulated scenarios grants responders the practice needed to retrieve vital data when it’s most important.
Despite initial disaster recovery (DR) steps often leaning heavily towards manual methods, it is key to continuously strive for increased automation. The more recovery tasks that can be executed using scripts or programmatic triggers increase the likelihood of a successful recovery. However, one should never let perfection stand in the way of progress. The drive towards total automation should not detract from the importance of deploying even a rudimentary DR strategy. Any measure of planning is better than none. Commence with highly detailed documentation and then make a gradual shift towards automation based on feedback from tests. Remember, before you can run a marathon, you must first learn to crawl and then walk.
Many contemporary organizations find the cloud a more flexible medium to replicate data, and particularly to allocate short-term disaster recovery infrastructure. Cloud services offer a simple solution to spin up and spin down test environments, greatly facilitating the rehearsal of disaster scenarios. Even organizations that rely on on-premises infrastructure should consider cloud-based replication for backups intended for DR purposes. For a large number of organizations, the cloud is already an integral part of their cyber resilience preparations – it’s crucial to also make sure it plays a role in DR readiness.
While having DR documentation is undoubtedly beneficial, its usefulness is greatly undermined if it is only tested when a disaster strikes. Frequent testing not only helps to develop a quick and accurate disaster response, but it also helps to identify any gaps in the process. Proactive testing involving forced failures and simulated disasters is essential. During such testing, occasional setbacks should be anticipated as opportunities to examine new capabilities or routines.
Twenty-nine years ago, within hours of the birth of my daughter, I found myself fielding a phone call asking for assistance with a botched restoration at work. However, thanks to my comprehensive and well-documented recovery process, I was able to quickly address the issue and return my focus to my newborn child. There is no telling when a fully documented recovery process will be required. So, the best time to start creating one is right now.
