~ Data Center Fire! ~
This is a story of trials & tribulations in running a data centre overseas in the late '90s...
An oil company ran a data centre in the Indonesian capital for all its SE Asian operations. As all data centers then, it had a staff running the mainframes the original repository, and workstations to manage the mainframes. You see, mainframes were the 18-wheelers of the day - no, the truck trains like in Australia with one tractor pulling three trailers - un-sexy but reliable and simple to operate and to teach the local staff. Workstations on the other hand ran backups on newfangled Exabyte tapes that could easily be stored and shipped by expatriate (expat) staff.
So when the Jakarta riots happened in 1998, the military transport that repatriated the expats also took a container full of Exabyte tapes... There wasn't even time to power down the data centre! I was in the Houston HQ of a service co. asked to restore the Exabyte tapes locally, so that operations could continue until things settled down abroad. [ Ed.: little did we know we were building a digital twin over two decades before it became the fashion with the internet and modern software! ] But I was given the unenviable task as senior consultant to tell the oil co. IT execs that their backup tapes were empty! The process had been set up on the relatively new workstations but neither tested nor verified, as would have been by old hands in the mainframe world.
Fortunately the events unfolded rather quickly, and a few months later when everyone returned to Jakarta, the data centre had been spared because it had been hosed with water as a retardant to stop various fires from sweeping through. That's how reliable mainframes were, that they kept humming along! And meanwhile the more fragile workstations were simply rebuilt (operating system recreated and apps and data restored). So we simply fixed the backup routines, and before anyone touched anything else, all mainframes were backed up over a 5 day period. We took advantage of that faux-pas to train local staff in both mainframe and workstation routines... we literally had expats working daytimes and locals doing maintenance night time. Voilà!