Tobogganing with Satellites, or How Murphy's Law was Thwarted on a Remote Broadcast

Recently I was fortunate enough to be tapped to engineer a live remote in the snowy reaches of Camden, Maine. The folks at American Public Media’s Weekend America wanted to cover the US National Tobogganing Championship, so they sent intrepid reporter Keith O'Brien to cover the fierce, yet admittedly whimsical, competition.

Keith went up a week before the event to get a lay of the land and interview several competitors. He put together a fine pre-recorded piece and sent it off to the Los Angeles studios of Weekend America. But producer Marc Sanchez and technical director Philip Richards weren’t going to settle for only a pre-recorded piece…they wanted Keith to actually take a ride down the official chute. That meant a live interview!

This presented something of a problem, as about half the toboggan chute extends out over the frozen expanse of Hosmer Pond. There’s no AC power out there, nor any telecommunications landlines. There was a cellphone tower not far away, with good line of sight…but for a nationally-distributed public radio show, we wanted better audio quality than a regular cellphone would deliver. And there wasn’t time to get a POTS-codec up there for testing. So we needed something that was known to work, and could work completely independent of any existing infrastructure in Camden. That meant satellite! Specifically, ISDN over satellite. With the right satphone, you can establish a 64kbps channel, which is more than sufficient for a decent quality ISDN connection.

Now as you might imagine, there’s about a thousand things that could go wrong with this setup. And go wrong they did! It was a comedy of errors for practically the entire process. But with foresight, planning, backups, testing, more backups, more testing and yet more backups, we kept it a comedy rather than a tragedy.

Keith’s initial visit had us pretty confident that we’d have a good view of the sky for the satellite. But first we had to make sure the equipment survived shipping to me in Boston. A few days before the live event, I received a Thrane & Thrane M4 TT-3080A portable satellite telephone in a nice flight case, and a Telos Zephyr Xstream ISDN (with built-in mic mixer) in another. The Thrane satellite phone folds into a very portable and lightweight package. Besides the somewhat weighty batteries in the UPS’s, I could’ve carried the ISDN, satphone, dish and assorted gear in one trip by myself! Okay, after some fiddling, both powered up just fine. At first there was some confusion over the lack of SPID’s for the ISDN. Eventually, with the help of Telos tech support, we realized that we didn’t need SPID’s because our satellite’s telco provider is located in France; SPID’s are usually only needed in North America. A quick change of the ISDN’s “telco” setting to ETS-300 and we were in business.

Our first test was from my apartment in Boston. I had a narrow…very narrow…view of the sky from my back porch. After aiming the satphone’s “dish”, I was able to get a single little bar of signal from the Inmarsat AORW floating over the western Atlantic Ocean region. That was enough to start dialing and, success! I connected and locked to Weekend America’s ISDN in Los Angeles, and Phil heard me loud and remarkably clear. However, I couldn’t hear Phil. Odd…no lock on his end. We re-dialed a few times and which end had the lock would change, but we couldn’t get both ends to lock. We had started with the AAC-LD 64 (low delay) codec, and discovered that changing to other codecs, like L3 or L2, made the problem worse. That usually points to data throughput problems, so I theorized that one bar of signal just isn’t enough.

After much fiddling with the satellite dish, I discovered that you must aim the dish very precisely, but you’ll get four or five bars of signal as a reward. The dish has a built-in gimbaled compass that helps, but it’s mostly done by a steady hand and lots of little changes. It was very difficult as the dish isn’t designed to hold anything but a 0, 45 or 90 degree tilt. Anything else required external bracing to hold it steady. Still, once the proper aim was achieved, we got a solid lock on both ends, and our ISDN’s were very happy.

So we’ve proven that we don’t need any local telco connections, now what about the lack of power? I knew that a standard little “brick” UPS/Battery Backup, such as the APC ES350, ought to work for our needs. A stress test demonstrated over 25 minutes of battery life – more than enough. Even so, a second UPS was available and seemed a good idea. Furthermore, I wanted to go for a “belt and suspenders” approach of having a DC-to-AC inverter charging the UPS, and the UPS powering the ISDN and satellite phone. Unfortunately, the vagaries of backup power proved problematic. When you look at AC power coming out of your average household outlet, in theory it looks like a nice 60Hz sine wave. The power coming out of your average inverter, like my $50 Target 400W model, typically looks like a “square wave”. So much so that the UPS didn’t think it was seeing valid AC power, and insisted on “staying on battery”. While there are ways to “clean up” the power, they’re both expensive and we likely couldn’t have gotten them in time. Still, we had two UPS’s and could run off inverter direct if necessary.

The day of reckoning approached, and Friday night I headed out for the four-hour trek from Boston to Camden. Apparently Murphy hopped on board, because this is where the problems started in earnest. To begin, the only real winter storm this season decided to blow through that evening…making a four hour drive into six. Late that evening, I arrived to the toboggan chute and found a good spot at the edge of the frozen lake. Problem is, now there was a “freezing rain/snow” mix steadily falling, and not all this gear is weatherproof! Fortunately, I’d brought a five-foot-wide patio umbrella with me and clamped it to the trunk lid, which kept the sensitive stuff dry. After some aiming of the dish, I had a perfectly successful “real world” test connecting to LA again.

I retired to my hotel room and re-packed all the gear with an eye towards carrying it by hand if I couldn’t get my preferred parking spot. All batteries were topped off and I nodded off for the evening, hoping the weather would clear.

Fortune smiled on us, as day came with beautiful clear skies, a fresh layer of ice on the chute, and plunging temperatures. Good thing I packed extra sweaters and long underwear! I met with Keith and, after some arguing with the parking guards, was able to get my car situated in the same spot where I did my test, and got everything set up. Except I realized that the UPS that had initially provided a perfectly-sufficient 25 minutes of power, was not providing jack squat! I had left it charging for two days, it should’ve been fine…but later tests showed it just refused to take a charge at all. Perhaps the stress test damaged the battery. Regardless, it's good thing we had a spare UPS. And as it turns out, despite valid concerns about an inverter’s “square wave” causing audible noise on the ISDN, there wasn’t any noise and we just ran on the inverter alone.

Then not an hour later, a dogsled team with ten noisy huskies set up shop not 20 feet from us! Trust me, it's still a comedy! A little chat with the owners and they said it'd be no problem to take the dogs on a run across the frozen lake when we were actually live...which actually worked great from an ambient sound perspective.

After both I, and Keith’s wife, remarked how amazing it was that nobody had died on that toboggan chute…Keith took the plunge with the chutemaster during a break in the trials. He says for most of the nine seconds, he didn’t scream because he couldn’t breathe. But when he found his breath...well, I think I heard him from across the lake! And let’s just say his choice in words wasn't FCC-friendly! Afterwards, Keith met up with competitor Andy Hazen, who was to be interviewed live at about 12:30pm.

Suddenly, disaster loomed! The morning’s test on the satellite phone had drained the satellite phone’s battery alarmingly fast. I put it on the charger and decided to run the live interview that way. But the satphone’s charger put out so much electrical noise that the ISDN would no longer connect. Fortunately there was a second satphone battery that had been sitting in the nice warm car; these batteries specifically say to keep them above 5 degrees Celsius (about 41F). A quick swap and we re-established the ISDN link. But even the warm battery was quickly running out of juice! Suddenly this 15 minute connection looked mighty tenuous.

Fortunately we had earlier determined that Keith had solid cellphone service at the chute, and that Keith had a Verizon cellphone (as do I). The latter is important since Verizon’s CDMA technology, unlike Cingular & T-Mobile’s GSM, or Nextel’s iDEN, doesn’t cause any audible interference to audio gear. We quickly dialed Keith’s cellphone into Weekend America’s studios and patched it in to their mix board, so we had a hot-standby ready to go.

I outfitted Keith and Andy with Sony 7506 headphones and ElectroVoice RE50B mics with hefty windscreens. After much testing, we got the levels for both Keith and Andy set about right, and confirmed they could talk back and forth with Weekend America just fine. We tried different L3 and L2 codecs but ultimately stayed with the AAC-LD at 64kbps and 48kHz sampling. The quality was quite good, I’d compare it to a solid FM radio station. The Low-Delay part was remarkable…it was no longer than a cellphone’s delay.

Feeling a little nervous but nonetheless hopeful, 12:30 rolled around and we listened to Keith’s pre-recorded piece fed back down the line to us. At the end, host Bill Radke introduced us and we were off to the races. The background noise of dogs, snowmobiles and PA announcements set the scene perfectly. Keith shared his toboggan run story, Andy shared his secret toboggan wax formula of “moose drool”, and I nervously watched the satphone’s battery indicator steadily drop and the satphone start to beep softly. I whispered down my own cellphone to Marc that we might have to go to our cellphone backup, but fortunately the satphone battery held in there long enough to finish the interview. You can hear it at http://weekendamerica.publicradio.org/programs/2007/02/03/toboggan_championshi.html (in the February 3, 2007 archive)

The ultimate point here is that pretty much everything that could go wrong, did. Even comically so; my polarized sunglasses – ideal for snow glare – made reading all the LCD displays very difficult. Fortunately we figured out most of what could go wrong in advance and had time to prepare backups. In fact, the one thing we didn't test to real-world conditions was the one thing we were nearly bitten by: the battery life of the satphone. Admittedly there was little reason to test it beyond a minute or two; satphone calls run around $5/minute (ouch!) and we had every reason to expect that two batteries would be more than sufficient, or that we could run off charger if we had to. Everything had at least one, if not two levels of backup…but that was one area we needed three.

So what’s the moral? Test, test, test! Always do tests as if they were the real thing; never assume something will work as you expect it to, and always have at least one solid backup to every part of your system (extra UPS’s and the inverter, extra batteries), and a separate backup that’s completely separate and redundant to everything (Keith’s cellphone).

In conclusion, I’m not sure I’d recommend this method for doing remote recordings. It’s got several potential points of failure and is prohibitively expensive. A cynical observer might remark that Keith could've made one trip with a portable flash or minidisc recoder, edited the piece, and filed it for the following week's show.

But at the same time, that's not really what Weekend America is about. I'm told the show strives to have a "live" feel to it and it definitely contributes to its overall "sound" every week. Plus who wants to hear about the toboggan championships that happened last week?? It sounds a lot better when you're talking about something happening this weekend!

So with that kind of mandate in mind, it's good to know that you've got options like ISDN-over-satellite in your toolbox. Plus I got paid to have a fun trip to Maine and watch guys hurtle down a chute on an old couch. Ain't public radio grand?!? :-)

Photo Credits: Weekend America, Keith O'Brien & Aaron Read