A guy in our data center couldn’t figure out who owned a particular machine that he needed to work on. So his solution to figure it out was to let them come to him. He went and pulled out the network cable and waited. He was escorted out a little while later. The moral of the story is don’t go disabling production machines on purpose.
Where I worked we had a very important time sensitive project. The server had to do a lot of calculations on a terrain dataset that covered the entire planet.
The server had a huge amount of RAM and each calculation block took about a week. It could not be saved until the end of the calculation and only that server had the RAM to do the work. So if it went down we could lose almost a weeks work.
Project was due in 6 months and calculation time was estimated to be about 5 1/2 months. So we couldn’t afford any interruptions.
We had bought a huge UPS meant for a whole server rack. For this one server. It could keep the server up for three days. That way even if wet lost power over the weekend it would keep going and we would have time to buy a generator.
One Friday afternoon the building losses power and I go check on the server room. Sure enough the big UPS with a sign saying only for project xyz has a bunch of other servers plugged into it.
I quickly unplug all but ours. I tell my boss and we go home at 5. Latter that day the power comes back on.
On Monday there are a ton of departments bitching that they came in an their servers were unplugged. Lots of people wanted me fired. My boss backed me and nothing happened but it was stressful.
At a startup a long time ago, I was working on the weekend and brought my 3 year old with me. We had a customer coming in next week and this one machine was 5 days into a 7 day model build.
We had to go into that office to help someone with something unrelated. The little shit saw the blinking light and headed straight for the button.
On this computer (HP 710), it didn’t shut off until you released the button. He actually was just pressing it but got spooked when I tried to get to it.
The next day our CEO told the guys that built that app that it had to be made so it could recover from crashes and restart from where it left off.
I guess it depends on where you work. This was a large datacenter for a very large health insurance company. They made it a point later that day to remind people that it was a fireable offense to mess with production machines like that on purpose. And evidently the service he disabled was critical enough that it didn’t take long for the hammer to come down. There were plenty of ways to find out who owned the machine, he just chose the easiest and got fired on the spot for it.
Yeah, I’ve done that before – after asking literally everyone in IT, plus our external consultants, and getting the go-ahead from my team lead and the head of IT.
A guy in our data center couldn’t figure out who owned a particular machine that he needed to work on. So his solution to figure it out was to let them come to him. He went and pulled out the network cable and waited. He was escorted out a little while later. The moral of the story is don’t go disabling production machines on purpose.
Where I worked we had a very important time sensitive project. The server had to do a lot of calculations on a terrain dataset that covered the entire planet.
The server had a huge amount of RAM and each calculation block took about a week. It could not be saved until the end of the calculation and only that server had the RAM to do the work. So if it went down we could lose almost a weeks work.
Project was due in 6 months and calculation time was estimated to be about 5 1/2 months. So we couldn’t afford any interruptions.
We had bought a huge UPS meant for a whole server rack. For this one server. It could keep the server up for three days. That way even if wet lost power over the weekend it would keep going and we would have time to buy a generator.
One Friday afternoon the building losses power and I go check on the server room. Sure enough the big UPS with a sign saying only for project xyz has a bunch of other servers plugged into it.
I quickly unplug all but ours. I tell my boss and we go home at 5. Latter that day the power comes back on.
On Monday there are a ton of departments bitching that they came in an their servers were unplugged. Lots of people wanted me fired. My boss backed me and nothing happened but it was stressful.
At a startup a long time ago, I was working on the weekend and brought my 3 year old with me. We had a customer coming in next week and this one machine was 5 days into a 7 day model build.
We had to go into that office to help someone with something unrelated. The little shit saw the blinking light and headed straight for the button.
On this computer (HP 710), it didn’t shut off until you released the button. He actually was just pressing it but got spooked when I tried to get to it.
The next day our CEO told the guys that built that app that it had to be made so it could recover from crashes and restart from where it left off.
I’d be super gluing those plastic toddler plug covers all over that thing.
fuck those other departments.
Honestly we do that when we ask and no one speaks up. Lovingly called the “scream test” as we wait to see who screams.
I guess it depends on where you work. This was a large datacenter for a very large health insurance company. They made it a point later that day to remind people that it was a fireable offense to mess with production machines like that on purpose. And evidently the service he disabled was critical enough that it didn’t take long for the hammer to come down. There were plenty of ways to find out who owned the machine, he just chose the easiest and got fired on the spot for it.
Scream tests are a last resort though.
If you fear reprisal for a scream test then you need to make it look like an accident.
Yeah, I’ve done that before – after asking literally everyone in IT, plus our external consultants, and getting the go-ahead from my team lead and the head of IT.