How to cripple your business in seconds.
We didn't know
A few years back I was approached by a school for some advice. It seems that they were having some issues with viruses and wanted guidance.
It turns out that they had a randsomware virus hit and it locked a large number of their files. Many of the files were very important to the administration of their school.
I advised them to clean up the virus and restore their files from backup. At the time they had an MSP who was taking care of their systems. It turned out that this particular MSP had not been checking the backups and when they went to restore there were no good ones they could use.
The client had to re-create files from scratch while others were completely lost forever.
The administration thought they were doing everything correctly. They had an MSP on retainer, they had backup software running on schedule and their network was maintained. So why did this happen?
In college I had one of my Computer Science professors tell me in a networking class that the more I learned about this stuff the more I would be amazed that it actually works. I have been maintaining IT systems professionally for over 15 years and can attest that this is very true. Managing technology is like trying to fly a plane where every part was made in a different plant each of which don't know they are making parts for an airplane. It requires a very broad level of knowledge and planning to make these parts work together and create a safe ride.
This is why Multi Service Providers are an invaluable tool for business. They are able to take the complexity of a business network and manage it for you across the wide technology spectrum.
The issue is that even though these MSPs are working hard, they often have blind spots. In this particular situation the MSP was very good at customer service and keeping things running but was weak on process review. They did not have it in their service portfolio to regularly validate backups. This is a common mistake. In my career I have many stories where admins thought they were backing up but ended up explaining to their customers why they cant restore a file. In a 2016 study it was discovered that over half of businesses have bad backups. It isn't enough to simply be backing up, you need to be backing up properly.
Oversights like these often have disastrous consequences which include companies going out of business overnight.
Backups have become such a large problem over the past few years that most governance standards prescribe backup procedures.
However, backups are not the only complex technology problem that businesses have to face. We have networks, kiosks, email, and databases.
As business owners we want to work on developing our product, we want to avoid overhead which is why we outsouce.
Separation of Duties
In my career I have been through many audits and no matter how big the corporation, or how mature the IT department, things are always uncovered. Some of the best insights come from third party audits. Often these audits are required for compliance purposes on a regular basis. The items they uncover vary from innocent oversights to blatant ignorance. Either way the risk gets identified and the business can then decide what to do.
You cannot fix risks you do not identify.
An audit does not fix the problems, it simply shines light on them so you can decide what is important as a business. Perhaps in your company backups are not important but maybe you run a business where customers can use wi-fi and the liability is a problem for your company. You can determine what is an acceptable risk and what is not.
These assessments cannot come from inside your MSP or contractor, they must be external to be objective. You don't want to get stuck with excuses or in a sales meeting. You just want to know the honest truth about what is going on.
What if they knew?
Had the school done an IT audit they would have discovered the threat with the backups not being validated. They also would have found several other items. Then they could have worked with their existing MSP and ensured there were processes in place to cover their major concerns.
The school would have avoided the loss, strengthened their MSP relationship, and been in a much better place.
Technology is complex and oversights happen all the time. The trick is monitoring your services and holding them accountable. A good service provide will welcome an assessment. It is their moment to shine. A poor one will avoid it at all costs.