For me, my high bar that I have yet to beat, was the time I pivoted the running OS (ubuntu) into RAM over SSH so I could unmount and image the boot drive without rebooting and loading a live USB (Which would have required a ticket with my provider to enable IPMI)

  • Noxy@pawb.social
    link
    fedilink
    English
    arrow-up
    5
    ·
    17 hours ago

    Recovered a legacy COBOL 911 dispatch system after the hard drive containing the root filesystem died, which wasn’t RAID protected at all and had no complete backups except for a few days prior when I started running daily rsync backups out of paranoia because the idiot dipshit sysadmin who set it all up left the company to work directly for one of our customers.

    Thankfully the data volume was RAID protected and didn’t die so the critical data was pretty much all there ready to go again after some rebuilding of shit.

    Still, took 15 hours to recover. Was a RHEL 4 system when RHEL 6 was current, and we had no way of obtaining the install media or licensing, so I dug up an archived CentOS 4 iso and installed from that, and got stuff working mostly just by copying various files from backup.

    Fucking nightmare come true. Drive crashed at noon and we didn’t go home until about 4am.