This weekend we carried out a major upgrade to our services. I won't go into great detail about what we did, you can read all about it in this excellent blog, but I do want to use this post to thank all concerned. The team started the task at 6pm, after a full days work, and began taking services down. Then they began the upgrade, as well as turning round some half ton cabinets in our data centres to improve the cooling and power consumption. With the hope that the upgrade would be complete by about midnight, and services could be brought back either then or first thing in the morning, things were looking good late in the evening. However, it's fair to say that things then did not go as well as hoped, and the team hit a problem which was difficult to diagnose, and prevented systems being brought back.
So, they carried on working. All night. And into the next day. I was away, but keeping up with events by watching the twitter hash tag, #cicsupgrade, and by phone and I have to say by Saturday morning was not bothered at all about the systems, but about the team's welfare! Pizza, fruit, drinks and donuts were provided, and eventually the problem was solved and systems back by mid afternoon. At that point the core team had worked for over 30 hours without a break. (Although I have heard rumours of one of them catching a nap or two.)
The comms during the event were excellent, with the blog updated regularly, and the twitter feed being very active. At one point we were trending in Sheffield, and being watched by people from all over the place. It was compelling viewing, a bit like watching a soap opera! The support for the team was amazing, with not a single comment about inconvenience. There's a great storify here which picks up some of the action.
So, a big thank you to the team who worked through the night, everyone who supported them, and everyone who helped get systems back on Saturday. I am proud to have such a dedicated, professional team working in CiCS.
- Posted using BlogPress from my iPad