It was a Tuesday morning after a bank holiday. 90 minute trek into the office, which was quicker than the usual two hour slog out of Cornwall.
Turned the laptop on, and grabbed a coffee whilst the laptop finished installing its selection of updates for Windows 10 tech preview.
Upon firing up our service board application, I was greeted with hundreds of server alerts from the evening before.
Careful inspection of the alerts showed that the aircon had failed in one of our customers backup server rooms.
Luckily, the extractor fan we insisted on having installed (after a secondary aircon unit was considered too expensive) was able to assist with cooling the room. Not enough to keep the room cool, but helped prevent damage to the hardware.
The local aircon maintenance company were called out to perform an urgent repair.
After under an hour onsite, the aircon engineer had resolved the problem and the disaster had been averted.
The problem? Dandelion seeds.
An excess of dandelion seeds had blocked the external inverter units fan, causing it to overheat and trip out the power. This of course shut down the aircon to the server room.
Unbelievable, but true. If you have external aircon inverters, its worth getting them checked out!
What can I do to prevent this:
Of course having redundant aircon units on separate supplies is a great idea (with an annual maintenance contract) , if funds allow, but having email alerts coming through is a great warning system, and its little to no cost to implement. It’s also possible to shutdown servers if they get hot (though of course that can have its own issues with regards to continuity of service).
I recommend taking a look at this guide from the guys over at howtogeek