Data is an incredibly important asset. Unfortunately, too many agencies continue to treat data as an easily replaced commodity.
But we’re not talking about a database administrator’s (DBA) iTunes library. We’re talking highly sensitive and important government data that can be too easily lost or compromised.
It’s time to stop treating data as a commodity and create a secure and reliable data recovery plan by following a few core strategies.
1. Establish objectives
Establish a Recovery Point Objective (RPO) that determines how much data loss is acceptable. Understanding acceptable risk levels can help establish a baseline understanding of where DBAs should focus their recovery efforts.
Then, work on a Recovery Time Objective (RTO) that shows how long the agency can afford to be without its data. Is a two-day restore period acceptable, or does it have to be 15 minutes?
Finally, remember that “high availability” and “disaster recovery” are different. An agency DBA managing three nodes with data flowing between each may assume that if something happens to one node the other two will still be available. But an error in one node will undoubtedly get replicated across all of them. Have a plan in place to recover from this error and keep it from becoming replicated.
2. Understand differences between backups and snapshots
There’s a surprising amount of confusion about the differences between database backups, server tape backups, and snapshots. For instance, many people have a misperception that a storage area network (SAN) snapshot is a backup, when it’s really only a set of data reference markers. Remember that a true backup – either on- or off-site -- is one in which data is securely stored in the event it needs to be recovered.
Consider the backup rule of three, which dictates that DBAs should save three copies of everything, in two different formats, and with one off-site backup. Does this contain hints of paranoia? Perhaps. But it also perfectly illustrates what constitutes a backup, and how it should be done.
3. Make sure those backups are working
Although many DBAs will undoubtedly insist that their backups are working, the only way to know for sure is to test the backups by doing a restore. This will provide assurance that backups are running -- not failing -- and highly available.
4. Practice data encryption
Data-at-rest on the server should always be encrypted, and there should be backup encryption for the database. There are a couple of options for this. DBAs can either encrypt the database backup file itself, or encrypt the entire database. That way, if someone takes a backup, they won’t be able to access the information without a key.
DBAs must also ensure that if a device is lost or stolen, the data stored on the device remains inaccessible to users without proper keys. Bio-level encryption tools like BitLocker can be useful in this capacity.
5. Monitor and collect data
Real-time data collection (RTC) and real-time monitoring (RTM) should be used together to protect data. Combined with network performance monitoring and other analysis software, RTM and RTC can improve performance, reduce outages, and maintain network and data availability.
With RTC, administrators can capture events as they come in, allowing them to establish a real-time collection of information they can then store to do proper data forensics. This will make it easier to track down the cause of an intrusion, which can be detected through monitoring.
RTM, database analysis, and log and event management can help DBAs understand if something is failing. They’ll be able to identify potential threats through things like unusual queries or suspected anomalies. They can compare the queries to their RTC historical information to gauge whether or not the requests represent potential intrusions.
6. Test, Test, Test
This is assuming a DBA has already tested backups, but let’s make it a little more interesting. Let’s say a DBA is managing an agency environment with 3,000 databases. It’s impossible to restore them every night; there’s simply not enough space or time.
In this case, DBAs should take a random sampling of their databases to test. Shoot for a sample size representing at least 95 percent of the 3,000 databases in deployment, while leaving a small margin of error (much like a political poll). From this information DBAs can gain confidence that they will be able to recover any database they administer, even if that database is in a large pool. If you’re interested in learning more, check out this post, which gets into further detail on database sampling.
Aside from people, data is quickly becoming government agencies’ most precious asset. Don’t treat it like it’s anything but that. Make sure no one is leaving server tapes lying around cubicles, practice the backup rule of three, and, above all, develop a sound data recovery plan.
By Thomas LaRock, Head Geek, SolarWinds
Related Blog Posts
