In the last few months the news reported severe cases of database nightmares: first COVID test errors were caused by information improperly stored in Excel, then police records were wiped by a human error.
As much as it is not common to read in the news about these types of stories, unfortunately, data loss in business happens fairly frequently and the effects can be devastating. The key problem is that too often businesses realise the risks of data loss only when it’s too late, when the data has gone, and there is no longer any way to retrieve it.
Think about your organisation, how much business data is stored to ensure the running of operations?
From invoices to bills, from clients to employees, from suppliers to sales, each business unit constantly acquires an incredible amount of data that is stored in order to be retrieved when needed.
Now imagine that one day your sales team is no longer able to access any data about your prospects, or that your finance team can’t access any information of due payments, or that all of your clients’ sensitive information, including credit card details, are stolen and shared on the Web. Well, that is data loss. And that is a nightmare nobody wants to have.
Fortunately, there are times when an expert might be able to rescue some of the data, but relying on an expert’s skill and luck is not a good strategy, the risks are simply too high.
To help businesses understand the real consequences of data loss, why it’s critical to set up systems in order to prevent it, and how to do it we spoke with our DBAs and asked them for their tips to help us navigate through the challenges of secure data storage and data loss prevention.
What is data loss and why does it happen?
Data loss specifically refers to the deletion or corruption of data that was considered stored and accessible. In practical terms, it means that data can no longer be retrieved because it’s been deleted, misplaced or made unavailable. It might also mean that sensitive data has been leaked and, as such, is no longer secured.
There are many causes of data loss:
- Corruption by hardware failure
- Site disaster (flood, fire, water damage, power surge, etc)
- Human error
- Loss of encryption keys
- Cyber attack and data breach
However, the very core of the issue is the lack of planning.
Data is lost forever only when disaster recovery plans and back-ups are not in place.
This is why prevention is the only secure way to avoid loss of data.
How serious can data loss be?
One of our DBA’s recalls the case of an organisation that came to WellData for help: “They had an unsuitable infrastructure: they were storing critical business data on a single disk that was over 5 years old. The data backups were taken to a spare disk on the server, but unfortunately no backups of the server were taken. When a disk failure happened on the server, it remained undetected for 3 days. By the time an investigation was organised, they discovered that the logfiles had completely filled one of the disks, due to the database stalling (lack of diskspace), that the underlying disk had been corrupted, and that the data was not recoverable by any conventional means. Fortunately, in that case, after an expensive and non-stop week of work using highly specialised tools, WellData was able to recover most of the data.”
However, having only incurred financial and down-time expenses; other companies are not so lucky: only a few months ago, Travelex, the foreign exchange firm, had to close operations as a result of a cyber attack to their database.
Indeed, when data disaster strikes in small organisations, over 70% of them cease operations within the year following the incident.
Data loss is often overlooked by a business and they only realise how important their data is when they lose access to it. Data is the lifeblood of any company and without it any company will effectively cease to function.
How to prevent data loss
Nowadays businesses can decide where to store their information and data: on physical databases or in the cloud. More often than not, they choose both systems, depending on the data.
Our top recommendation on how to prevent data loss in the cloud computing include:
- The majority of the platforms-as-a service (PaaS) offer high availability and disaster recover options. However, you should always ensure that they are configured correctly.
- It is also critical to ensure that the frequency and retention of backups adhere to your Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO).
- Don’t rely only on the provided corruption checks, instead also carry out your own checks. As they say, better safe than sorry, and in this case it could not be more true.
- Ensure that security is as tight as it can be, including Network and Access rights. In particular, ensure that access and permissions are set to a minimum following the Principle of Least Privilege, to avoid data breaches and human errors.
- Prevent cyber attacks with additional security measures.
- Make sure that you backup to multiple regions/multiple locations/multiple clouds.
- Finally, a simple way to secure your data is to copy the local backup files to the cloud. This gives a simple offsite, and therefore unreachable, copy.
When it comes to traditional databases, our recommendations for data loss prevention are:
- Make sure that you have proper security/access/privileges on all database accounts. Importantly, ensure that security meets the latest practices and that servers, hard drives and software are patched to the latest patch sets.
- Like in the cloud, prevent cyber attacks with additional security measures
- Have a full set of backups in place, preferably following the 3-2-1 rule: at least three total copies of your data, two of which are local but on different mediums, and at least one copy off-site and unreachable.
- Do regular checks to make sure that you can restore from the backups.
- Ensure that your backup plan meets your RPO and RTO requirements.
- Ensure corruption checks for databases and database backups are in place.
- Ensure backups are restored regularly to ensure their validity.
- Ensure high availability and data recovery processes are documented, tested and followed.
One additional recommendation in particular deserves extra consideration: very regular transaction backups.
Taking a backup of the transaction log every minute almost doesn’t take any extra storage than one taken every 30 minutes. The only extra overhead is the file header that holds a few bytes of data. However, the benefit can be huge: taking regular transactions backups means less data loss when things go wrong as you can restore to the point in time of your last log backup.
How do I get my data back?
If for any reason your database was not secured and you are struggling to retrieve your data, there might still be a chance that an experienced DBA can come to the rescue.
The truth is that the type of help depends on the nature of the data loss. If data has been deleted and there are no backups, there is, unfortunately, nothing anyone can do.
However, if the data loss is due to corruption, companies like WellData have tools to try to recover the data; for instance, if there are backups, systems could be rebuilt and damaged files can be recovered.
In a recent instance, as an example, a non-client came to WellData after a ransomware attack. Their database and backups had been encrypted. They were able to recover some of the database files from a forensic recovery company, but the database file was not recognised.
WellData used tools to extract data from the recovered file and create a new database. Unfortunately, there was still data loss as the recovered file was a month old, but nevertheless, this was considerably better for the company than no data at all.
Additionally, after solving the immediate crisis, WellData advised on a number of measures to mitigate against this scenario happening again and prevent data loss in the future.
In fact, the crucial point is that businesses need to think about what could cause data loss before it happens and take mitigating actions. This is where hiring experts like WellData can make the real difference: we can help to prevent data loss by setting up and advising on backup, on strategies and corruption checking, security, high availability and disaster recovery configurations.
