KFOR-TV knocked off by UPS failure early morning

Current events discussions relating to Broadcast Engineering
Post Reply
User avatar
NECRAT
Site Admin
Posts: 2949
Joined: Sat Nov 17, 2007 9:13 pm
Location: Taunton, MA
Contact:

KFOR-TV knocked off by UPS failure early morning

Post by NECRAT » Wed Jul 20, 2016 5:53 pm

http://kfor.com/2016/07/20/kfor-crews-t ... JPplKKZDBa

Some interesting behind the scenes vids further down. Their Leibert UPS had a failure and a control circuit burned up.
http://www.necrat.us

"Arguing with an engineer is like mud wrestling with a pig. After a couple of hours, you realize the pig likes it"

grich
Posts: 308
Joined: Fri Sep 02, 2011 11:19 am
Location: MP89.5, Mason City Subdivision

Re: KFOR-TV knocked off by UPS failure early morning

Post by grich » Thu Jul 21, 2016 3:58 pm

Capture2.jpeg
Frame grab from video of some of the damage.

User avatar
Shane
Posts: 765
Joined: Fri Feb 01, 2008 12:08 am
Location: Omaha
Contact:

Re: KFOR-TV knocked off by UPS failure early morning

Post by Shane » Sat Jul 23, 2016 12:15 am

I could hardly get my iPhone to stand still on that site so I could actually read the story. Forget video. Sheesh. Hate sites like that.
Mike Shane, CBRE
---Omaha---

User avatar
RGORJANCE
Posts: 1387
Joined: Mon Dec 31, 2007 7:48 am
Location: RACINE, WI

Re: KFOR-TV knocked off by UPS failure early morning

Post by RGORJANCE » Sat Jul 23, 2016 7:17 am

Then UPS thingys are dangerous. Had a waist high unit BARF white and yellow sparks and flame at me a few years back. Singed my britches. A second small one a couple months later barfed on my shins when I was doing some wiring in a rack. Nearly got it again.

Are they trying to tell me something? :?

Fossil

dbuckley
Posts: 103
Joined: Mon Jun 16, 2014 4:18 pm
Location: North Canterbury, New Zealand

Re: KFOR-TV knocked off by UPS failure early morning

Post by dbuckley » Tue Jul 26, 2016 3:19 pm

So a single UPS failure takes out their servers. Hmmm... I'd have thought they would be running Tier 3 infrastructure as a minimum, so a UPS on fire is no more that a minor inconvenience, rather than an event that interrupts on-air capability. Also without being Tier 3 minimum one cant run 24x7x365 reliably as you need to bypass and/or take stuff down for maintenance.

If one's power system isn't "concurrently maintainable", one is looking for a hiding sooner or later; either one maintains and runs with reduced (or no) ability to withstand power disturbances during maintenance, or one doesn't maintain because its too disruptive. To run a data center non-stop for years takes more than just good luck.

Explanation of data center tiers from Wikipedia.

User avatar
Shane
Posts: 765
Joined: Fri Feb 01, 2008 12:08 am
Location: Omaha
Contact:

Re: KFOR-TV knocked off by UPS failure early morning

Post by Shane » Tue Jul 26, 2016 10:21 pm

Um, please take this in the humorous spirit in which it is offered but we can tell you don't live here!

Hate to admit it but I do not know what Tier 3 infrastructure is. So, I'm a little less surprised than you are about KFOR-TV's problem, but still surprised.

In our little radio complex, I have two automation servers and two audio engine servers each with dual power supplies being fed thru two different UPSs. So the failure of one shouldn't take us down...unless one catches fire. They ARE at the bottoms of the racks after all.

[On top of that, the automation is very network-dependent. It can't work if the network goes down and guess what? The network switch has only one source of power (which I accidentally unplugged a few months back - place got real quiet as all the workstations "coded." Reckon it'll be a few years before I live that one down!).]

Couple years ago a power outage hit Fargo, ND, and took out one of the "big three" networks across the whole state! The A/C wasn't on generator power and the servers overheated and shut down. Doh!

Oh and I had to stop awhile and think about what you meant by "hiding." In the states, it's what I do after pulling the power plug on the network switch.

Love your contributions, dbuckley; please don't stop. I always learn something.
Mike Shane, CBRE
---Omaha---

User avatar
Bill DeFelice
Posts: 216
Joined: Wed May 18, 2011 6:29 pm
Location: Fairfield County, CT
Contact:

Re: KFOR-TV knocked off by UPS failure early morning

Post by Bill DeFelice » Wed Jul 27, 2016 5:02 am

I was always taught to avoid a single point of failure. In my 9-to-5, a K-12 school district, our data center has multiple UPS systems not only for load balancing but to ensure any one hiccup doesn't take out the entire plant. I've seen things in such a disarray in some data centers it makes me cringe.

At one station I have my hands in there's three separate UPS units for the entire building, where business equipment, audio equipment and transmitter control gear are each on their own independent system. With the way things are wired any UPS can be bypassed via a heavy duty twistlock extension. For the decades I've been involved I've never hear of them losing any significant airtime.
Webmaster
History of Westport Connecticut Radio
The WMMM Tribute Site

CampusBroadcaster.net
Legal, low power, license-free broadcasting for educational institutions

kcbooboo
Posts: 451
Joined: Sat Dec 10, 2011 4:53 pm
Location: East of the Mississippi

Re: KFOR-TV knocked off by UPS failure early morning

Post by kcbooboo » Wed Jul 27, 2016 6:01 am

APC, and probably other companies, make a bypass device. It gets power from the incoming utility as well as the UPS, and you plug your loads into this device. If the UPS has output, that powers the loads. If the UPS shuts off, the utility powers the loads. I don't know what else is inside it besides a relay.

An engineer I know built a similar device using a 120V relay with 30A DPDT contacts. The relay is powered by the UPS output and the NO contacts supply the load with UPS power. If that goes away, the relay drops out and the NC contacts feed utility power to the load. Some status indicator lamps and the relay all fit inside a standard 4 inch square electrical box with a pair of duplex outlets.

As we found out, most 30A DPDT relays have that rating on the contacts that are engaged when the relay is pulled in. The other contacts are only good for 5A. You have to check the ratings carefully to find something that will handle 30A through both the NO and NC contacts.

Bob M.

dbuckley
Posts: 103
Joined: Mon Jun 16, 2014 4:18 pm
Location: North Canterbury, New Zealand

Re: KFOR-TV knocked off by UPS failure early morning

Post by dbuckley » Wed Jul 27, 2016 3:59 pm

Shane wrote:Um, please take this in the humorous spirit in which it is offered but we can tell you don't live here!
Humour always welcome.

My day job is IT, and keeping computer stuff running is a challenge; everywhere you look there is something waiting to trip one up. And the problem is that Da Management often doesn't understand (or, sometimes, dont want to understand) how critical It infrastructure is to the continued operation of a business.

Your dual UPS / dual power to servers is the tier three approach; you can unplug one UPS, take it outside, hose it down to get the muck out of the fans and filters, leave it in the sun to dry, and bring it back in and power it up, whilst all the time the system is still running on the other UPS, so if the street power goes down the system will stay up. That's "concurrently maintainable", and its a damned good thing to have.

The problem, as you've noted, is that some equipment only has one power inlet. Whats a chap to do? The answer is a bit of kit called a static switch, or sometimes a transfer switch, for example, found by a quick google, the APC AP7752, which takes two power sources in, and gives one highly available source out. Of course, then you have to trust that the static switch is more reliable than a UPS... And my experience in that area is that small static transfer switches are not that great. The small static switch from the manufacturer that delivered on its promises (https://web.archive.org/web/20020622164 ... rackmount/) is no longer available.

dbuckley
Posts: 103
Joined: Mon Jun 16, 2014 4:18 pm
Location: North Canterbury, New Zealand

Re: KFOR-TV knocked off by UPS failure early morning

Post by dbuckley » Wed Jul 27, 2016 4:12 pm

kcbooboo wrote:APC, and probably other companies, make a bypass device. It gets power from the incoming utility as well as the UPS, and you plug your loads into this device. If the UPS has output, that powers the loads. If the UPS shuts off, the utility powers the loads. I don't know what else is inside it besides a relay
Automatic bypass "devices" intended for no-break operation, as opposed to a manual bypass switch, don't use relays, they use solid state devices (thyristors or triacs) to do the switching, as a relay is far too slow. Though some bypass switches of the non-automatic variety do the actual switching with semiconductors, and then use a contactor to carry the load continuously, as there is less losses and thus less heat generated. But these can only be used in manual mode as they are too slow for no-break switching.

Every UPS already has a semiconductor bypass switch inside it, though its not there for the reason that most folks think. UPSs cant actually deliver a lot of power beyond their rated output, their overload rating is usually something pathetic like 150% of rated load for a couple of cycles. If a fault occurs downstreeam of the UPS, like a short circuit, that requires a breaker to trip (or a fuse to blow), a UPS simply hasn't got the power reserves to open that breaker. The UPS output will just collapse. So... what the UPS does is as the output collapses, it switches the output back to street power, which has got what it takes to open the breaker. Then a couple of seconds later the static switch put the output back onto the UPS output. This is true of even quite large UPSs, those with ratings of 100s of KVA.

The corollary of this is that faults are bad when a UPS is running without street power as the normal fault clearing process cant work. So faults can be far more disruptive than would be the case when street power is present.

Post Reply