Had a call from the Operations tech on duty last night, to tell me temperature in one room of the remote (but critical) North Campus was unusually high. 85°F. Bad but, I thought, not too bad. I told him to call Facilities and keep an eye on it.
I called back about fifteen minutes later. Yes, Facilities had been notified, and by the way, "that column, bar-graph thing on the remote meters, where it says POWER? It went down to 75."
Oops. That is very much in my bailiwick. I did my level best to not sound as if I was rolling my eyes while asking, "Are there any alarms in red on the screen?" Red would be active rather than cleared and logged. "Have you looked at the security cameras?"
"Oh, we looked at them first thing. Nothing out of the ordinary. And," long pause, "...no red alarms. I stepped through the whole list."
"Okay," I replied. "I'm going up there. Did Facilities give you an ETA?"
"Um, he was going to check things on his computer from home." (There's a whole separate remote control for the cooling systems. For stupid department-rivalry reasons, I don't have access to it.)
I drove up, wondering what was going on, and arrived to find the front room -- only! -- mildly warm at the door, and in the 85° to 90° range near the critical equipment. Each room has its own cooling; the front room has six 10-ton units, out of sight above the ceiling in a long row above a long row of equipment, half of which sits idle since we went digital. At the end where the working equipment sits, two were not moving air at all and one was pushing hot air. There's not as much mixing of air as you'd expect and as a result -- with 10,000 cubic feet of air per minute air going into the equipment and coming out ten degrees hotter than it went in -- was a hot-ended room and a distinct reduction of output power.
The little workshop/office has its own cooling and the double doors to it were closest to the warmth; I propped one door open far enough to clear a couple of fans and set them running while I called up the Facilities guy to explain that this was A Problem Of Importance. He was already en route, his telemetry software having revealed not three but four of the six units weren't cooling the air.
He was twenty or more minutes out, so I checked room temperature and power out -- down a little and up a little, respectively, then went to the back room: much colder. Colder than the workshop by several degrees, at least. If I moved the fans to that door, they'd pull in some really cold air, and help move air from the less-warm end of the front room to the hot end -- and there was a wire-rack shelf I could use to vertically stack them!* I did it, and it helped; output was up to 80% by the time Facilities arrived. Two new belts and some inspired fiddling later, he had two of the four offline air handlers back to making cool air, and the room was rapidly heading towards its normal 70° temperature. The other two units? Turns out they had been down for some time, needed either extensive repairs or replacement, and the second-floor folks were skeptical of the expense. (Ahem. Good to know. Ahem.)
We both hung around for another half hour while things stabilized -- a big HVAC system has a "capture transient" like a phase-locked loop. It's usually very well damped, but recovery from a large excursion can produce interesting overshoots and the last thing we wanted was to freeze up a coil as the cooling units worked hard to recover. Didn't happen. Just a nice smooth ramp-down (and ramp up) to normal temperature and 100% power, and we went away happy, having agreed to gang up on the accountants to get the remaining two units taken care of.
But what a way to spend a late Sunday night!
____________________________________
* When you stick a fan in an opening, the amount of air moved goes up as you close off the non-fan portions of the opening. The is one of the Rules Of Fans. Another rule is that the moved air needs an exit or a return to the back side of the fan; a room or duct or whatever can't just expand like a balloon. I was breaking that rule but it's a huge room and they were small fans. Had it come to that, I would have had to go to the 24-hour five-and-dime to buy more fans, and opened more doors so I could move air between the rooms in a big loop. The building is laid out for it; we did have some inkling that we might want to do such a thing. But this time, I was pretty sure even a small change would make a big-enough difference, and I was right.
I'm assuming the North Campus has network access, as in Al Gore's Intertubes or at least an internal network to which you have access. What would be involved in installing 8-10 thermometers in strategic locations - such as adjacent to the primary equipment and in front of the cooling system outlets - to provide that info?
ReplyDeleteSome years back we encountered budgetary recalcitrance toward a similar objective (fortunately, only temporarily) and rendered a cheap quick fix with some wiring, a few LEDs and a vendor-supplied Molelecular Collision Frequency Display (a thermometer from Walmart) and a web cam. Light #1 on, main power OK; light #2 on, UPS output OK; light #3 on, the night operator had been in the room in the last 30 minutes (motion sensing switch); thermometer reported temperature in real time. It became written that whomever had remote supervisory responsibilities (e.g., "working from home") stay logged in and maintain at all times an open window displaying the Production Control Facilities Monitoring System status. Never mind that the web cam was supported by a $40,000 4-processor server.....
There is an extensive remote-control system and the one temperature sensor already in that room is enough for my purposes -- I may need to change the alarm setpoint for it.
ReplyDeleteIf that changeover to digital equipment is at all like the equipment I've changed over to digital, then the room in standard config as it stands should be well over 3 times redundant in terms of heating load.
ReplyDeleteIn a few cases I've managed to grease the skids on the process for obtaining new toys by explaining what the upgrades would do for our "carbon footprint".
Good luck dealing with financial people using logic. I have tried in the past doing so, and have often run into a brick wall. It isn't enough to say that by spending the money to fix X you can keep Y from breaking down. They want to see payback within a certain timeframe, usually 12 months, or they can't justify the expense to the stakeholders.
ReplyDeleteOf course, they are not the ones who have to shlup down in the middle of the night, usually on a weekend, to fix something that should have been taken care of properly in the first place.
Just like the old Fram oil filter commercial. You can pay me now, or you can pay me later.
Too bad you couldn't open a few windows and doors, but that would probably violate some set of security protocols...
ReplyDeleteThe accounting/bean counter types never change. We lost most of a weekend at a phone center because they wouldn't spring for a couple of thousand dollars for a replacement part that the self-diagnostics were calling out as failing. It failed at a time to cause the longest possible outage.
The bean-counter didn't even get a reprimand. (Cost the company a LOT, both money and in reputation.)
I always thought that anytime something like this happened they should have to work graveyard shift for a week.
Got an e-mail one day just before leaving for the weekend from one of my mission critical communications servers that it was not happy with the temperature where it and 2 other servers lived. Apparently 115F was slightly out of spec.
ReplyDeleteThe building's new automated climate control system had 1. Determined that since no one was working in that section of the building that day it required no cooling. And 2, for some reason, it was a dandy day to turn the heating units into full blast and fans on high, dumping into the server room.
If nothing else, rousting the facilities folks back in just having got home while I shut down the servers and made notifications seemed to focus them on coming up with a solution in re-programming the system to keep the AC on in that room regardless of what else the building was doing.
Since the bean counters in question worked for my boss's boss, and she was impressed with the cost ration of replacing all the servers in that room vs putting in an auxiliary A/C unit that was NOT tied into the building controller(something like 93:1), and losing a critical emergency communications system while new servers were ordered, installed, and configured... Let's say it's nice 62F in there with all servers running in the middle of summer. Sometimes you get lucky.