This is how things collapse

96 hours to the stone age: How quickly our connected lives crumble when the power goes out

Gigaom

By Jeffrey Belk, ICT168 Nov. 23, 2011, 5:00pm PT

We may be closer to Thanksgiving than Halloween, but I hope this piece scares you. A lot.

Today, we take for granted that we will have full Internet access and connectivity to the world 24/7/365 on our smartphones, tablets and notebooks. We expect to be able to check a sports score or connect with a loved one in 10 seconds or less.

However, we don’t really consider that our smartphones and wireless device are connected to cell sites and cell towers. Which in turn are connected to the wireless operator’s main switching facility. All that needs lots of power, which after a blackout is provided by backup systems. If and when those systems run out of juice, at about 96 hours, we have a big problem.

Consider this. On Thursday, September 8, 2011, an equipment failure in Arizona caused an electric utility cascade failure, leaving millions of people from the San Diego area in the dark. One moment, power was on for a several thousand square mile area. The next moment it was gone.

In August, Hurricane Irene temporarily took out 6,500 cell phone sites on the east coast. At the end of October, a freak snowstorm left millions without power in parts of the Eastern Seaboard. Although weather problems are challenging, at least there’s usually some prior notice so utilities and cell operators can prepare. And there are often pockets where power is still available. When power goes down everywhere simultaneously, instantaneously, like it did in San Diego, it makes you think.

By 96 hours after the power shuts down, power better be turned on again, our connectivity restored, or we’ll be in the Stone Age.

That worries me.

What happens when the lights go out

When the power goes down, cell service “gets shoddy.” That’s going to happen when everyone grabs their phones at the same time. It’s the wireless equivalent of everybody getting on the same roads at the same time. But when 3G systems get congested, the coverage area of cell sites can actually shrink, resulting in potentially bigger coverage holes in addition to capacity issues.

However, just like I took it for granted when I was a kid that the wireline networks would always work, the vast majority of folks think their wireless devices should keep on working when power goes down. For shorter outages, this has mostly been the case. Like the Wizard of Oz behind the curtain, the wireless operators keep the systems up.

Let’s look at what happens. There are three pieces to the puzzle:

. Our phones/devices

. The cell sites

. The operator’s central switching facility for a given geographic area

(There are actually a bunch more, but I’m going to keep it simple.)

Power for your phone and laptop should be easy. Phones have small batteries, and everybody should have spares, external battery packs, solar chargers, or a cheap crank charger (buy one online). It’s your fault if your phone runs out of juice. Remember, even though your handy laptop might still have power, your wi-fi will be toast, since your router/modem will be down.

The cell sites themselves are backed up. There are hundreds of thousands of cell sites across the country, and operators have put battery backup in many of them, especially sites they view as critical. Beyond battery backups, operators have the ability to attach generators to existing cell sites, or rapidly deploy COW’s (Cells on Wheels) or COLTS (Cells on Light Trucks) to augment coverage and capacity in times of a crisis, and they have done well to this point. When there is a natural disaster, there is often a staggering amount of effort behind the scenes to restore service.

The problem is that those backup batteries run of juice, and when that happens, that’s it for service in a given area or neighborhood.

That third piece that you don’t think about? Start thinking about it.

Earlier this year, I visited a central switching facility of one of the large wireless operators. It is responsible for the operation of hundreds of cell sites in a geographic area. The wireless voice and data traffic from the cell sites in the region are routed back to this location though various methods of transport.

The facility itself is about 40,000 square feet, and is a monument to sophisticated technology. Spotless, shining, largely empty, and highly automated, it felt a bit like an old Star Trek episode where the people were gone and the advanced technology kept running on its own.

One room is filled with rows of switching equipment, the only sound the whirring of cooling fans. Another area has rows and stacks of large batteries providing eight hours of primary backup power to the facility.

Another layer of redundant backup is located outside: a giant diesel generator, wisely placed on seismic mounts, with fuel to keep the facility going for another four days without resupply. Four days. 96 hours. Ever since my visit, that 96 hours figure kept on sticking in my mind. If the switching facility went down, our smartphones were 96 hours away from being stupid. No power, no connectivity. No connectivity, no smarts.

If and when the power goes down, there will be limited staff at the facility, and if there is damage, my guess is that technicians and spare parts for the sophisticated equipment won’t be on-site.And that’s assuming there aren’t other problems (i.e. earthquake) that further disrupt transport, or even worse, a bunch of bad guys who break into the facility and play ‘telecom baseball’ with a bunch of steel pipes and the equipment. Personally, I would have been a lot happier to see deeper physical security on site. Maybe even rabid security dogs roaming the empty halls.

Let’s count down the 96-hour clock

So imagine power goes down one morning. No notice, it is just out. For the first few hours, we bitch that our wireless Internet connection is slow, or that we get network busy tones for some of the calls we want to make. Through our phones, tablets and other connected devices, we get news and updates from our local municipality along the lines of: “hang in there, we are sorting it out.” If we are not stuck at work or in a traffic jam, we make our way home to our dark houses.

24-48 hours: Enter the information abyss. The next morning, many of us will not have cell service. Operators will get portable generators to key sites — but not all sites. After 48 hours, more of us are disconnected.

So how do we find out what’s happening? Our TVs and cable modems don’t work — no power! Battery-powered radio? If you are one of the rare people who owns one, you’ll still have a problem. Radio stations are increasingly high tech, and guess what, most stations were off the air during the September San Diego blackout.

48-72 hours: Your wallet is empty and so is your fridge. How will you handle even simple purchases without power, communications or cash? As we increasingly transact via credit cards, online and even cell phones, cash has become much less prevalent. If the ATMs are down, and you don’t have enough emergency cash on hand, what do you do?

Already, it seems that for a broad range of demographics, especially those under 25, cash is already dead. Or, if there are emergency radio broadcasts and the broadcasts says that emergency help is located at a certain park in a certain city, what good is that information to a GPS reliant person who never learned to read a map and doesn’t own any maps?

72-96 hours: No gas, no water. Now what? Cars have run out of gas. The roads are so clogged, they’re non-functional. Public safety will be dealing with all of these issues — with a degraded communications infrastructure. And are the pumps from your local water facility still running?  Remember, all of the sewage and water plants are increasingly automated. I don’t know about you, but I will be cranky by that point.

Acts of humans will be worse than the proverbial “acts of God.” But wait. We’ve been discussing natural disasters and equipment failures. In another scenario, what if some bad guys launched a cyber attack on the utility grids? Kind of like what “may” have happened in Brazil in 2005 and 2007 (though the government attributed it to “soot on wires”). I’m not a data security guy, but in looking up articles on utility vulnerability, I stumbled upon the Grey Goose Report,  which is scarier than anything I’ve written.

The key to preventing this is keeping the power on or at least getting back up as soon as possible. On a more positive note, I visited the SDG&E (San Diego Gas and Electric) Emergency Operations Center earlier this year. They had this stuff thought out, worked out, and implemented. They maintain redundant disaster centers, redundant public and private communications systems, and an entire company dedicated to stopping the power outage clock as quickly as possible. 

However, stuff happens.

This is a serious threat, and we need to take it seriously

As I’ve thought about our reliance on pervasive connectivity over the last year, I’ve spoken with C-level executives from both the tech side and the utility side. They get it. But they have businesses to run, customers to serve, business targets to achieve to keep their jobs.

It is critical to recognize that the pace of our reliance on pervasive connectivity via our wireless devices is rapidly outstripping our ability to deal with the absence of those services. We need to recognize the extent that our wireless infrastructure is increasingly core to our personal, family, and societal existence. For now, it is a fragile core.

So, the next time the lights go out, look at the clock on your smartphone. Or start your stopwatch application to measure how long the power stays out. And hope the stopwatch doesn’t get to 96 hours.

Jeff Belk is Managing Director of ICT168 Capital, LLC, investing and working with wireless firms globally. He spent almost 14 years at Qualcomm, in roles including SVP, Global Marketing, and SVP, Strategy and Market Development. Belk is guilty by association as he helped make this smartphone and mobile broadband stuff happen in the first place while at Qualcomm and predicted the demise of WiMax six years ago. He really hopes he is wrong here.