Server room A/C turned off to save energy

[snopes]

Comment: I was just wondering whether you have heard of this ... Urban
Legend?

http://worsethanfailure.com/Articles...-Can-Deal.aspx

[diddy]

I am A bit skeptical about this. I work in the IT business and we have equipment that monitors various environmental considerations for our servers. Temperature happens to be one of them. If the temperature gets above a certain temp, we get alerted via email, text message, as well as an audible alarm. We had an alarm before the weekend (our system failed) and my boss found out and rushed in.

A state based system has to have a similar system to prevent the AC from turning off and alerting somebody.

[Hans Off]

You would hope so!

But I't might depend on how long ago it happened...

We had this problem back when NEXRAD was still an "experiemental" system. At OU, where they developed NEXRAD, they built a new university building for the computers and displays. And at 5:00 PM, the A/C units shut off, like they do in all university buildings. It took a day or so, but they were able to reprogram they system to keep the A/C unit working in that room.

Now the UL part of the story (i.e., I can't verify it because I was chasing at the time) is that the first incident happened when they had a bunch of VIPs over to demonstrate the use of the NEXRAD data. There were storms in the area, so they had the demo start at about 4 PM. When 5 PM rolled around, the room got very hot (due to the people, the computers, and the A/C shutting off).

[PallasAthena]

I don't know about this story, but I can share a couple of my own. I work as an account manager for an IT company, and one of my customers happens to be a very large chain restaurant. We have had several of their individual restaurants experience issues with their equipment rebooting over and over again.

In one, a restaurant manager thought her office (which doubled as a server room) was too cold so she brought in a space heater.

At another site, a manager decided that there was "too much racket" coming from the server rack so he muffled the noise by leaning flattened cardboard boxes against the rack. The results were predictable.

We've also had cleaning services who unplug our equipment to vaccuum around it, and a manager who, whenever her PC locked up would just unplug the entire power supply for the rack. When I asked why she didn't just hit the power button on her PC to reboot, she told me she had heard that was bad for the computer to do that.

Another account manager has another even larger restaurant chain whose account she manages. This account had a rash of dead routers last year. Several hundred sites had routers that just died completely. Turns out, the restaurant managers had placed some of the routers next to the big ovens. The rest were dead because of people stacking things on top of them. At one site, we dispatched our own tech to help deal with the problem. He pulled a stack of plates off of the top of the router and found scorch marks on the plates. Talk about running hot!

All true stories. All things I have seen and dealt with and had to clean up. So the OP to me doesn't seem farfetched in the least, especially if the company isn't a huge one and doesn't have a 24/7 IT staff. At a big datacenter, it would never happen, but in some company's IT closet, sure. I've seen stuff like it before!

[diddy]

Originally Posted by PallasAthena

All true stories. All things I have seen and dealt with and had to clean up. So the OP to me doesn't seem farfetched in the least, especially if the company isn't a huge one and doesn't have a 24/7 IT staff. At a big datacenter, it would never happen, but in some company's IT closet, sure. I've seen stuff like it before!

According to the OP:

Not too long ago, there was a small furor in the local media about a major disaster at The State's Technology Services Division.

It was a state-run technology division, My guess is that they have a dedicated IT staff.

Although I'm sure things like this have happened on a smaller scale, the story, as it is presented, is just a little too perfect.

Besides, if the facilities guy simply left his keycard unguarded at his desk, he should "take an early retirement" as well...

-Tim

[PallasAthena]

Originally Posted by diddy

According to the OP:

It was a state-run technology division, My guess is that they have a dedicated IT staff.

I wasn't sure it it was a business name (I have another account with a sort of ambiguous name like that) or if it was actually a government thing.

[diddy]

Originally Posted by PallasAthena

I wasn't sure it it was a business name (I have another account with a sort of ambiguous name like that) or if it was actually a government thing.

This:

I came in today (Monday) to finish up a project I was working
on before our big meeting with the State ----- Commission tomorrow,

The fact that they refer it as "The State"

And:

the network administrators had to figure out how to bring online the four Exchange Services, six Domain Controllers, a few Sun servers, and the entire State Tax Commission’s server farm.

Makes me think that its a government job.

Edit: The general consensus from the Page is that it is complete nonsense (they have seen variation of it with different details), lack of any unverifiable details, etc.

Originally Posted by diddy

It was a state-run technology division, My guess is that they have a dedicated IT staff.

From my experiences, I could see such an agency not have 24/7 IT staff on-site (costs too much, and they do have a limited budget) and if they move from a building where they have control of the HVAC system to a building where they don't (say during consolidation due to budget cuts)...I could see this happen.

In my NEXRAD case, they moved the "experimental" NEXRAD equipment from a dedicated building over to the new Weather Service building the university built for them. The university then put the HVAC units on their "standard" program, which meant shutting off the system between 5 PM and 7 AM to save money.

$200K buys a lot of servers, and that number wasn't counting the staff pay. I could see spending $200K or more to build a datacenter, but not replace a few bum servers (they admitted not all of them died).

Most server-grade equipment would be designed to shut-down before it got to the self-destruct point...

-Tim

[PallasAthena]

Originally Posted by Rehcsif

$200K buys a lot of servers, and that number wasn't counting the staff pay. I could see spending $200K or more to build a datacenter, but not replace a few bum servers (they admitted not all of them died).

Most server-grade equipment would be designed to shut-down before it got to the self-destruct point...

-Tim

Well, I don't think you could build a datacenter of much consequense on $200K (you have to think of the power needed, security, connectivity, POPs, etc), but $200K I would think would get you a hell of a lot of servers!

This story could be bogus, but the scenario is very much within the realm of possibility in my experience.

[WildaBeast]

I doubt that specific story is true due to the lack of verifiable details mentioned above, however we did have a problem like this at a startup company I worked for briefly in college. The company leased space in an office building, and the building's owner contorlled the A/C. At 5:00 on Friday, the A/C turned off automatically. Over the weekend the servers sensed the increase in temperature and shut themselves down. The temporary solution was to put a portable A/C unit in the server room (it wasn't a huge server farm, just a few servers in a large closet). We soon discovered that the collected condensation in a pan in the bottom of the unit, and when this pan filled up the A/C unit would shut down, and she servers soon followed. So, we had to have employees voulenteer to come in over the weekend to empty the drip pan. Eventually, we got the building owner to leace the A/C in our space on over the weekend.

Originally Posted by PallasAthena

Well, I don't think you could build a datacenter of much consequense on $200K (you have to think of the power needed, security, connectivity, POPs, etc), but $200K I would think would get you a hell of a lot of servers!.

But that was exactly my point. The heat would not destroy the power taps, security measures, connectivity (most likely), etc. The only thing that stands to really be killed by heat are the servers themselves, and those usually have fail-safe mechanisms for it. Even if they didn't, as you say, $200K buys a lot of servers. Which is why I think it DOESN'T fall in the realm of possibilities, at least as-told.

Besides, isn't this essentially This Old UL in new clothing? Substitute "bumbling non-IT type" for "janitor", "servers" for "patients", and "trying to reduce energy costs" for "trying to clean the floor"...

-Tim

[diddy]

Originally Posted by Doug4.7

From my experiences, I could see such an agency not have 24/7 IT staff on-site (costs too much, and they do have a limited budget) and if they move from a building where they have control of the HVAC system to a building where they don't (say during consolidation due to budget cuts)...I could see this happen.

In my NEXRAD case, they moved the "experimental" NEXRAD equipment from a dedicated building over to the new Weather Service building the university built for them. The university then put the HVAC units on their "standard" program, which meant shutting off the system between 5 PM and 7 AM to save money.

Our systems at work are far less extensive than he ones mentioned in the server and we have a system in place that lets us know in a minute if anything happens to the network. You can;t just shut down the AC either. An audible alarm goes off. You don't need to have a 24/7 staff, but you always have someone on call. Dedicated doesn't always mean 24/7. I should also point out that we are moving to be a green company as well.

Trust me, the story makes no sense. The guy leaves a key card just lying around that can get someone access to the server. No way. What is suillier is that someone admits to it and reprimands the entire IT staff for it.

[PallasAthena]

Originally Posted by Rehcsif

Besides, isn't this essentially This Old UL in new clothing? Substitute "bumbling non-IT type" for "janitor", "servers" for "patients", and "trying to reduce energy costs" for "trying to clean the floor"...

-Tim

Except for the fact that I actually have had janitors unplug my equipment to clean around it. On 2 separate occaisions. The things I have seen people do just blow my mind! Don't get me started on the time I had to troubleshoot a friggin' modem with a "tech" for one of my restaurant sites. I kid you not it took me 15 minutes to get her just to find where the power chord came out of the back of the modem. Then when she finally did, I had her trace it to the wall and describe what she saw:

"Well, there's this small box thingie at the end of the wire with two little flat bits sticking out."

"Uh, you mean the plug that you stick into the power outlet?"

"I guess that's what it is."

"Is it plugged into the wall?"

"No."

"Well, there's your problem!"

I kid you not. The call just got worse from there because then she had to find a phone jack to plug the phone chord into. ("What's a phone jack?" "Like you have in your house that you plug your phone into?" "Uh... I don't know what that is."). But dang she was proud of herself when she plugged that modem into the wall and the lights started blinking. *shudder* Then we had to fix the "broken" (i.e. not plugged in or connected) router.

Yeah, I'm a bad person. I put her on speaker after the first few minutes of the call so the people in the surrounding cubes could hear. They wouldn't have believed me otherwise.

Anyway, I guess my point is, there are a lot of morons out there who have way more access than they should to expensive equipment. It's perfectly plausible that one would not realize that these machines need to be kept cool. It's just the details in this that don't add up.

(plus, I got the chance to relate some crazy stories about the people I deal with on a semi-regular basis. )

[diddy]

Originally Posted by Rehcsif

Besides, isn't this essentially This Old UL in new clothing? Substitute "bumbling non-IT type" for "janitor", "servers" for "patients", and "trying to reduce energy costs" for "trying to clean the floor"...

-Tim

Heck. Thats got more details than that one.

[snopes]

Besides, isn't this essentially This Old UL in new clothing? Substitute "bumbling non-IT type" for "janitor", "servers" for "patients", and "trying to reduce energy costs" for "trying to clean the floor"...

I'd say it's distinctly different. In the older version, the person who unplugs the vital equipment is completely oblivious to what he/she is doing -- it's just a tragic accident. In this version, the person knows full well what he's doing (i.e., turning off the A/C in a server room), but he's too full of self-importance to comprehend the consequences of his act.

- snopes

Originally Posted by snopes

I'd say it's distinctly different. In the older version, the person who unplugs the vital equipment is completely oblivious to what he/she is doing -- it's just a tragic accident. In this version, the person knows full well what he's doing (i.e., turning off the A/C in a server room), but he's too full of self-importance to comprehend the consequences of his act.

- snopes

Well, I won't doubt the expert! But I guess I was looking at it as:

• The Janitor didn't know the consequences of unplugging the cord
• The person in the OP didn't know the consequences of turning off the AC
• Neither meant to kill anything, be it a patient or a server, and thought they were "just doing their job".

-Tim

[diddy]

Anyway, I guess my point is, there are a lot of morons out there who have way more access than they should to expensive equipment. It's perfectly plausible that one would not realize that these machines need to be kept cool. It's just the details in this that don't add up.

The OP dealt with that. The guilty party stole an access card to shut to the HVAC system in the server room. I don;t know about most mid sized companies that have somebody working in IT - one of the basics that you do is actually lock your servers in its own room.