Business Continuity Operations Group this morning (BCOG), where most of the business arose out of the recent exercise we carried out. We're particularly focusing now on getting departments to carry out Business Impact Assessments, and write their BC plans to fit in with them. Also some procedures we need to clarify - we spent some time discussing how to evacuate a building when there isn't a fire, so you cant use the fire alarms. An interesting one. Run round with a loud hailer shouting "GET OUT" was one suggestion. At the end of BCOG we always go round the table and note down Incidents and Near-Misses. This can cover everything from a power outage in the student residences, to bad weather, to a serious student illness, to systems related incidents.
We were pleased to report that this morning we'd had a "near miss", (not a good description), and not an incident. Tuesday night we had a serious hardware failure on one of our filers. Failover to the other one worked perfectly, but as we were then vunerable and running at risk we decided we had to repair it asap. Parts were brought up by the engineer yesterday afternoon, and taking into account what exams were being held and when, and availability of staff, an emergency CAB yesterday decided to do the work early this morning. We thought it was low risk - but obviously very high impact - if the filers didn't come back properly we would lose many systems. However, we decided the risk was higher after the engineer told us that they'd only done this twice before, and it had failed both times. Cue the taking down of our VLE, increased resistance on the web site, more people around, an earlier start time and ---- a hashtag!!
I woke up early, and with myiPhone under the bedcovers followed the progress of #fixthefiler (sorry to everyone else who'd got up at unearthly hours to come in). We're very lucky that nearly everyone involved tweets, so I got a running commentary, and all the banter that goes with it. There was some debate about the value of a hashtag, and we felt it definitely increases the team spirit - as well as keeping the people who follow us amused. Our official twitter feed tweeted regular updates, and everything went swimmingly well, apart from a mysterious orange light which appeared on one of the filers. But, the VLE and other things we'd taken down as a precaution were brought back, and the team had done a great job. But, there was more to come. In order that they could keep an eye on the mysterious orange light, a webcam was pointed at the server, and a new hashtag was born - #filercam. For most of the day, some of us have had a screen open showing a filer with a small orange light on it. Quite hypnotic, but also very silly. Social media is now an integrated part of our communications, and is so valuable, but its good that it can also be a bit silly at time. We all need a bit of fun in our lives.