Had a call from the Operations tech on duty last night, to tell me temperature in one room of the remote (but critical) North Campus was unusually high. 85°F. Bad but, I thought, not too bad. I told him to call Facilities and keep an eye on it.
I called back about fifteen minutes later. Yes, Facilities had been notified, and by the way, "that column, bar-graph thing on the remote meters, where it says POWER? It went down to 75."
Oops. That is very much in my bailiwick. I did my level best to not sound as if I was rolling my eyes while asking, "Are there any alarms in red on the screen?" Red would be active rather than cleared and logged. "Have you looked at the security cameras?"
"Oh, we looked at them first thing. Nothing out of the ordinary. And," long pause, "...no red alarms. I stepped through the whole list."
"Okay," I replied. "I'm going up there. Did Facilities give you an ETA?"
"Um, he was going to check things on his computer from home." (There's a whole separate remote control for the cooling systems. For stupid department-rivalry reasons, I don't have access to it.)
I drove up, wondering what was going on, and arrived to find the front room -- only! -- mildly warm at the door, and in the 85° to 90° range near the critical equipment. Each room has its own cooling; the front room has six 10-ton units, out of sight above the ceiling in a long row above a long row of equipment, half of which sits idle since we went digital. At the end where the working equipment sits, two were not moving air at all and one was pushing hot air. There's not as much mixing of air as you'd expect and as a result -- with 10,000 cubic feet of air per minute air going into the equipment and coming out ten degrees hotter than it went in -- was a hot-ended room and a distinct reduction of output power.
The little workshop/office has its own cooling and the double doors to it were closest to the warmth; I propped one door open far enough to clear a couple of fans and set them running while I called up the Facilities guy to explain that this was A Problem Of Importance. He was already en route, his telemetry software having revealed not three but four of the six units weren't cooling the air.
He was twenty or more minutes out, so I checked room temperature and power out -- down a little and up a little, respectively, then went to the back room: much colder. Colder than the workshop by several degrees, at least. If I moved the fans to that door, they'd pull in some really cold air, and help move air from the less-warm end of the front room to the hot end -- and there was a wire-rack shelf I could use to vertically stack them!* I did it, and it helped; output was up to 80% by the time Facilities arrived. Two new belts and some inspired fiddling later, he had two of the four offline air handlers back to making cool air, and the room was rapidly heading towards its normal 70° temperature. The other two units? Turns out they had been down for some time, needed either extensive repairs or replacement, and the second-floor folks were skeptical of the expense. (Ahem. Good to know. Ahem.)
We both hung around for another half hour while things stabilized -- a big HVAC system has a "capture transient" like a phase-locked loop. It's usually very well damped, but recovery from a large excursion can produce interesting overshoots and the last thing we wanted was to freeze up a coil as the cooling units worked hard to recover. Didn't happen. Just a nice smooth ramp-down (and ramp up) to normal temperature and 100% power, and we went away happy, having agreed to gang up on the accountants to get the remaining two units taken care of.
But what a way to spend a late Sunday night!
* When you stick a fan in an opening, the amount of air moved goes up as you close off the non-fan portions of the opening. The is one of the Rules Of Fans. Another rule is that the moved air needs an exit or a return to the back side of the fan; a room or duct or whatever can't just expand like a balloon. I was breaking that rule but it's a huge room and they were small fans. Had it come to that, I would have had to go to the 24-hour five-and-dime to buy more fans, and opened more doors so I could move air between the rooms in a big loop. The building is laid out for it; we did have some inkling that we might want to do such a thing. But this time, I was pretty sure even a small change would make a big-enough difference, and I was right.