Sync, time code, and insanity

[PID_Stop]

Several Saturdays ago our quiet evening came to a sudden halt, along with our automation system (which shall not be named): each of the twelve air channels stopped dead, and not even rebooting the gear would bring it back. The support engineer's first comment was, "that's not possible" -- never a good sign. After several hours, we traced the problem back to our master time code generator, a rather expensive GPS-referenced master clock that had gone rogue and decided that the current date and time and time were simply "8".

Fortunately, the master clock came back to life after being unplugged and restarted -- but not without blurting out about fifteen minutes of the wrong time, before it acquired enough satellites to lock back up and rejoin reality. (We did find out what Chicago wanted to know years ago: no, nobody really knew what time it was, and yes, our station's general managers really cared.) Even then, the clock woke up thinking we had moved three time zones west, which didn't make the automation any happier and got the attention of our operators, who were afraid their shift was being digitally extended three hours. Finally we got things back on track and settled down, but the carnage on the air was epic.

We've used different automation systems for years and seen all kinds of failures, but this is the first time we've run across equipment that depends on time code, but is utterly incapable of free-running on an internal clock if the code goes away briefly. Can't begin to fathom why anyone specifying the hardware or designing the software thought this would be acceptable... but that's what we have to deal with.

So because time code has been shown to be so incredibly critical -- and by the way, some of the gear wants LTC while the rest expects it as VITC in the reference -- we quickly decided that we need a far more robust sync generator / master clock system, which we have ordered and are now eagerly awaiting. This is not a cheap piece of kit: it's roughly the same price as my car, and it comes with about as many amenities (but not heated seats).

I've already made up the 42 BNC cables (yes!) to go between the generators and the changeover switch, and I'm reading the manual so that the gear will get configured and tested as soon as the UPS driver wheels it through the door. And on page 3, the manual contains this gem:

"Unplug this apparatus during lightning storms or when unused for long periods of time."

Seriously? Has any station ever shut down its sync generators in anticipation of bad weather? What are these people smoking?




Jeff

[NECRAT]

We had a similar epic failure several years back in what is now one of your sister hubs, where the automation system (now 3 incarnations ago), required LTC to run. In the middle of one Sunday afternoon it stopped. All the lists stopped, and the operators were in a panic. Of course I got the trouble call, got in as soon as I could, and sure enough, the system flagged it lost time code. However the backup "automation appliance" did not! So this was a 2 for 1 on this shot, as I decided to switch over to the backup using the change over switch. Only to then discover the device control change over switch was also dead!! (Different issue). I ended up swapping the two time code cables (with RCA connectors), and the time code came back. However the lists were still fubar and I had to reboot all the "playlist machines" to get them running. When all was said and done, we lost one break on our CBS and one on our FOX. (Plus our national channel went down but no one cared). Oh and the issue? The integrator when installing the DAs for the Time Code never actually went through and set the levels for 5vP-P. They were all over the place, and the cable that had been on the main device server was around 2~2.5.

(For what it's worth, I am JUST about to put in our brand new sync generator pair on where I currently work. They run on GNSS vs. GPS but will free run and keep very accurate time with the loss. However in the planning stages, discovered the integrator here (different guy than above) had mis-labeled a bunch of cables, so the decision was made to run 40 new cables to the new DAs. This will take our "house shaking" frame offline too. This all happens at 1am, including being on the ladder on the roof to swap the antennas out. This is a big leap that we need to take to get some quite old, long end of life GPS sync gear offline...)

[PID_Stop]

Your experiences with integrators is sadly similar to my own. In every case where others have built up systems for us, they have left behind serious errors in design and execution; and generally, whatever documentation they leave us is marginal, and in a form that is impossible to update. That's why I greatly prefer to do our own integration (which I did for the current automation system): we know exactly how everything is wired, know where to look if there's a problem, and the documentation is consistent with the rest of our plant drawings and wire list. (And who can pass up the opportunity to wire up about 140 cables with HD-BNC plugs to a two rack-unit routing switcher?)

Our new generators -- an Evertz system with separate GPS receivers for the primary and backup -- are impressive, but then I'm old enough to think a Tektronix R146 is pretty neat. The idea that two separate units, neither locked to the other, can produce essentially synchronous outputs within a few dozen nanoseconds is amazing.


Jeff

[NECRAT]

Your experiences with integrators is sadly similar to my own. In every case where others have built up systems for us, they have left behind serious errors in design and execution; and generally, whatever documentation they leave us is marginal, and in a form that is impossible to update. That's why I greatly prefer to do our own integration (which I did for the current automation system): we know exactly how everything is wired, know where to look if there's a problem, and the documentation is consistent with the rest of our plant drawings and wire list. (And who can pass up the opportunity to wire up about 140 cables with HD-BNC plugs to a two rack-unit routing switcher?)

The 5700 series from them is what we're putting in. I have been very impressed with them so far. What I really like about the ACO, is that it's fully metal to metal and doesn't do any actual buffering. The GNSS units keep such accurate time to each other, that a switch "on air" should be clean and not cause any issue without the need to have any buffering. The only thing I don't like is the HD-BNC connectors. They're a PITA to use. I would've rather seen an octopus breakout. I had Ramtronix out of Long Island make me about 26 2' jumpers HDBNC to BNC Male, to keep the install clean and neat.

[Dale H. Cook]

Fortunately, the master clock came back to life after being unplugged and restarted -- but not without blurting out about fifteen minutes of the wrong time, before it acquired enough satellites to lock back up and rejoin reality.

It should not take that long for a GPS which is already in service to get a fix and that may point to a cause. On first boot a GPS (any fairly recent on that is) has to acquire the almanac and ephemeris and then get the first fix, and a new GPS on first boot has no almanac or ephemeris, requiring (typically) 15 minutes or so to get to first fix. Any GPS that sits unused for too long (say a few months), or that is transported a significant distance unused and unpowered, will have an out-of-date almanac and ephemeris. It sounds to me as though something corrupted the almanac or ephemeris. When you rebooted it did you do a factory reset?

[PID_Stop]

When you rebooted it did you do a factory reset?

Not intentionally... but given the number of settings that changed from where they had been, it's apparent that this was the effect. Temperatures have gotten quite a bit lower lately, and with them the humidity has dropped like a rock. We had several apparently static-related problems over that weekend, and I put a humidifier in the room Monday morning. Haven't had any problems since. It will all be a moot point in a couple weeks.

One thing I appreciate about the Evertz system is that the cable to the GPS antenna is a data line, not coax like the ESE's (or the Leitch that came before the ESE)... so the system's RF performance is independent of the cable length. The other great thing is that for the first time, our time reference will be redundant and just as robust as the video reference signals.

If someone had told me 35 years ago that our master clock would be more immediately air-critical than the sync generator, I would have laughed. My, how things change.

Jeff

[Dale H. Cook]

... so the system's RF performance is independent of the cable length.

That is my case as well, but my mom-and-pop radio clients don't need GPS. I use it only in the Explorer to control my police/fire/EMS scanner. As I drive around the scanner switches, based on location, to whatever radio systems are used locally, be they city/town or county systems, the local state police division, or other services like DOT highway repair crews and snowplows, utility crews, or airports. I actually use my old hand-held Garmin GPS (long ago replaced by a Magellan that displays topo maps for DA proofs), which can output GPS data through a serial cable in a format that my scanner can read. The Garmin lives in a cradle on my dashboard. Last year it would not acquire any satellites when I turned it on, and I had to do a factory reset to force it to (I presume) load new copies of the almanac and ephemeris.

[NECRAT]

So reason #3 (or shall I say cable #3, why we are replacing cable).

Intergrator labeling.
(Sync side) -- (Rack side)
<146 ----- 90004>

< 90004 ----- ????>

[jawsborne]

Wow, pretty crazy and scary that all MC systems stopped without the timecode. Was anything on air while the master clock was messed up?

Did you go with an Evertz as your new master clock?

[PID_Stop]

We are a regional hub, so our automation controls twelve feeds in addition to another twelve feeds that are passthrough with minimal local insertion (e.g., Bounce, Laff). So yes, all twelve main program feeds came to a grinding halt. What we have since discovered is that the automation can free-run... until midnight GMT (7pm local time), when the system date is supposed to change. If time code is absent, the system "sees" a massive jump backward in time, amounting to over 2.5 million frames. And this without a flux capacitor!

And yes, we bought a whole new Evertz system with two 5601MSC sync generator / master clocks and a 5601ACO2 automatic changeover switch. It's very impressive equipment, with each sync generator locked to its own GPS receiver; as a result, the two generators stay within nanoseconds of each other so that changing from one to the other is relatively non-disruptive.

Our two Tektronix SPG422 boxes get put out to pasture after about twenty years. Amusingly, the 45+ year old Tektronix R146 that had been our main sync generator when I started is still at our shop bench going strong. I have a definite fondness for equipment that comes with detailed manuals and schematics, with the expectation that it can be readily repaired.

Jeff

[PID_Stop]

As an aside, thinking about that Tektronix R146 is a powerful reminder of how profoundly broadcast equipment has changed: most of the signals it produces have not been relevant in at least 25 years, but our Ampex VTRs and GE cameras depended on them:

• Subcarrier (continuous 3.57MHz sine wave)
• Composite sync (H and V sync plus blanking, but no color burst)
• Burst flag (essentially a mask for where subcarrier would go in composite sync)
• Vertical drive
• Horizontal drive
• Blanking

It also produced NTSC color black and various test signals like color bars, multiburst, graticules, and ramps. Today only color black is still used by modern equipment (but the R146 wouldn't be a good choice, because it doesn't keep the subcarrier in a consistent phase relationship to horizontal sync).

Yup, I'm feeling my age now!