| << Back to Press Menu | |
 |
After the Storm by Merrill Douglas Feb 08, 2006 VIEW ORIGINAL ARTICLE As hurricanes ripped through the Gulf Coast in 2005, government CIOs swung into action. Soon after the waters of Lake Pontchartrain flooded portions of New Orleans, CTO Greg Meffert handed Mayor Ray Nagin a voice over Internet protocol (VoIP) phone he "commandeered" from a looted Office Depot. This was the only option left for conducting a call between the mayor and President Bush on Air Force One. Phone service was down throughout the city, and radio towers had toppled to the ground. The phone was plugged into City Hall's only functioning Internet connection, which stood just a few inches above water. Ten weeks later, Meffert still faced "a huge rebuilding effort" on the city's IT infrastructure. Although anything plugged in was washed away in the flood, Meffert said, the city's Web applications and services were running safely on servers in Dallas, and its mainframe applications were high and dry in Orange County, Calif. "That's why I'm still in business," said Meffert in mid-November 2005. "All those basic systems are back online. But I've had to, over the past two months, completely build a whole operations center to run everything." Impossible Endeavor Like so many others who responded to hurricanes Katrina and Rita in 2005, IT officials at every level of government from Florida to Texas dedicated long hours of hard work and exercised tremendous creativity. Whether they were fishing ruined equipment out of the floodwaters, helping house and comfort evacuees, or managing the offers of aid that poured into their region, CIOs and their departments played a key role in the recovery. They often pulled off the seemingly impossible in record time. The immensity of what happened that week in September continues to emerge. By mid-December 2005, the U.S. government had spent more than $20 billion in hurricane relief, with estimates that the final price tag would "pale in comparison," according to an article in the Washington Post. The storm and the ensuing floods along the Gulf Coast highlighted how vulnerable the physical infrastructure for telecommunications networks is to natural disaster, according to testimony in October 2005 during a congressional hearing on homeland security on government operability during catastrophic events. CIOs in both the public and private sectors are ill prepared for such a catastrophic event. A poll of CIOs in November 2004 found that one in five did not have a formal disaster recovery plan in place, and one in 10 said their organization would last less than two weeks if local or regional technical infrastructure failed to recover quickly, according to the CIO Executive Council, which conducted the survey. Only 31 percent of CIOs rated their disaster recovery and business continuity plans as "extremely or very effective." But as federal, state and local government CIOs know, citizens expect government to be available -- at all hours -- to help during such disasters. That means being electronically and physically accessible for those in need. In New Orleans, Meffert turned to business partners and new technology to keep a semblance of government operations above water. For example, when the city had only six weeks to determine whether it was safe for residents to return to 110,000 homes, software developer Accela offered technology to automate permitting and inspections. Intel and Tropos Networks built a Wi-Fi network based on mesh technology. "Panasonic stepped up and gave us a bunch of Toughbooks with GPS," Meffert said. "My internal team wrote some code to automate the GPS linkup with the GIS database to map everything, and voilà! I went from total devastation to probably the single most state-of-the-art inspection and permitting system in the country." By combining the new Wi-Fi network with another that New Orleans installed to support public safety surveillance cameras -- which survived Katrina -- the city now also has wireless communications for police applications. "Electronic incident reporting, suspect tracking -- all these kinds of things directly into the cop cars," Meffert said. "Now my MDTs [mobile data terminals] are flooded, so I need more of those. But I've got all the infrastructure and the apps ready now. "Companies are stepping up and saying this is now the ultimate proving laboratory," Meffert continued, adding that it's a place where they can demonstrate new solutions in extreme conditions without accommodating legacy technology. "That's one of the few advantages of so much wreck. I don't have to figure out how to work around that [old] system. That system's gone." Tracking Health Risks As the storm waters and winds from Katrina receded, government CIOs and IT specialists swung into action. An IT support specialist with the Director's Emergency Operations Center at the Centers for Disease Control (CDC), Darrell McQueen flew from Atlanta to Louisiana to provide IT support to scientists. McQueen was part of a Logistics Support Team (LST) consisting of IT, communications and logistics specialists. The CDC's mission was to analyze the floodwaters in New Orleans and assess the danger that bacteria posed to people returning to the city. After a short stay at a tent city in Baton Rouge and a few nights sleeping in their cars, the LST settled into quarters on the USS Iwo Jima, docked in New Orleans. Members began converting a New Orleans hospital -- its patients and staff evacuated, its IT infrastructure destroyed -- into a workspace for the CDC scientists. Army reservists brought power generators, and the team started establishing data communications so the scientists could exchange information with their headquarters in Atlanta. "We had one satellite unit set up. One computer," McQueen said. "And that's the way we operated for a day or two." Soon the CDC team started working with BellSouth. "In one week's time, we were up and operating on a T1 connection with about 30 users -- computers, printers," McQueen said. Since the phone lines weren't working, staff members relied on cell phones for voice communication. But, McQueen said, they had video conferencing running, network communications, data sharing, print sharing -- everything they normally have at their office. In all, 15 to 20 IT specialists worked at the site, rotating in and out, with three to five in place at any one time. Each day, the team assessed what equipment was needed and had it flown in from Atlanta. Setting up an IT infrastructure from scratch wasn't much of a problem once he had the necessary equipment, said McQueen. "The biggest challenge for me was mental, dealing with the devastation." Images on TV and descriptions in the news didn't prepare him for the reality on the ground. "It was breathtaking. It was life changing. It really made me think about how quickly you can lose everything," he said. "My heart goes out to those people." Reconnecting Infrastructure Mississippi's central IT facilities in Jackson are well away from the devastated coast, but Katrina left much of the city without power for a week. "We happened to be on a downtown grid that did not go down, so all of our central computing facilities stayed up. But all of our government office buildings were down," said state CIO David Litchliter. Desktop computers at state offices throughout Mississippi are linked to servers in Jackson, so agency facilities on the Gulf Coast didn't have much IT equipment to lose. But many of the local communications lines connecting those offices to the state's communications backbone "were down hard," Litchliter said. "In a couple of cases, the central offices of BellSouth were completely washed away and there was nothing left but some wires sticking out of a slab." As of late October 2005, state officials, the Federal Emergency Management Agency (FEMA) and state telecommunications experts from Florida were still working with BellSouth to re-establish voice and data communications for state employees on the coast -- at least, for those who had something resembling an office. "A lot of them are working out of tents," Litchliter said. By a fortunate coincidence, just before Katrina struck, the state's IT department completed a business continuity plan that Litchliter said proved quite helpful. One thing state IT officials didn't anticipate before Katrina was how deeply involved they would become in the post-hurricane humanitarian effort. When state agencies and non-governmental organizations arrived with relief supplies, they needed technology to support their distribution facilities. "Some of those people needed systems and some brought their own," Litchliter said. State IT staffers helped by setting up wireless capability in the distribution centers so they could establish local area networks and have applications to use. They also provided the expertise needed to impose order on the numerous missing persons databases established by different aid agencies. "There is a master database now that was accumulated from the multiple databases and then scrubbed to get rid of duplicates," he said, adding that the master database became the one used by the state's public safety agency. In addition, the IT department helped colleagues in the state's Department of Human Services, the Mississippi Emergency Management Agency, the Department of Finance and Administration, the Department of Employment Security, and other state organizations quickly set up call centers so they could start providing aid to constituents, he said. Channeling Information, Aid in Alabama Although Katrina didn't damage Alabama nearly as badly as Louisiana and Mississippi, the hurricane and its aftermath kept the state's Information Services Division (ISD) busy. A major focus was on channeling information to the Alabama Web sites and ensuring that everything stayed up in spite of the high volume of hits received, said Alabama CIO Jim Burns. A "Hurricane Center" link on the state's Web portal provided a centralized source for information on road closures, emergency management, weather forecasts, relief agencies and more, said Sloane Wright, director of marketing at Alabama Interactive, Alabama's e-government solutions provider. The ISD and Alabama Interactive also built a link to a second Web site, where concerned citizens could donate money and services. Burns' division worked with Thomas Marshall, associate professor of management at Auburn University, on an application that matches volunteers and donated goods with areas of need. The software allows streamlining of donations made to the state in a way that's never been done before, said Terri Hasdorff, executive director of the Alabama Governor's Office of Faith-Based and Community Initiatives. At the direction of Gov. Bob Riley, Alabama's state parks became homes-away-from-home for evacuees from all three states devastated by Katrina. FEMA provided trailers to house them, but the parks lacked telephone service and Internet access, and many of them didn't have cell phone coverage, Burns said. Working with several of the state's technology partners, the ISD started running T1 lines to each of the 12 state parks, so residents could use the Internet to apply for services, contact family, apply for jobs and communicate with insurance companies. BellSouth helped with the circuits, Dell donated laptop computers and Cisco donated equipment to link those machines to the state's network. Another vendor, Blue Coat, donated software for filtering out inappropriate content. Vonage -- with which the state had not yet done business -- provided VoIP phones, interface boxes, and free local and long distance telephone service. State IT facilities saw little damage. "The only thing we had of consequence was the telephone node for the state telephone systems in Mobile, which was taken out. It was actually flooded," Burns said. "It was a fairly straightforward thing to redirect those calls." Houston's Double Whammy By the time Hurricane Rita took an indirect swipe at Houston in late September, that city's IT staff had already been working on hurricane response for several weeks. Tens of thousands of Katrina evacuees from New Orleans poured into the city to take shelter in the Astrodome, the George R. Brown Convention Center and other facilities. "The first thing we focused on in the IT shop was the evacuee registration process," said Houston CIO Richard Lewis. While the Red Cross and county personnel logged information about new arrivals onto paper forms at the Harris County-owned Astrodome, an IT team from the local electric utility set up a computer-based registration process at the city-owned convention center. Lewis worked with Steven Cooper, CIO of the American Red Cross and Steven Jennings, CIO of Harris County, to get registration data from both locations into the Red Cross's Web-based family locator system. "I think we had 27,000 registrations loaded initially in the week after Katrina hit," Lewis said. Using 75 PCs donated by Hewlett-Packard and United Way, Lewis and his deputy worked with Cooper at Reliant Park -- home to the Astrodome and two other facilities used as shelters -- to start a center where evacuees could access the locator database. Then the Red Cross converted a church into a disaster recovery facility, Houston's IT team set up computers and a wireless network, which volunteers used to issue emergency assistance checks to thousands of evacuees. That deployment took a mere 24 hours. "As soon as we finished that, FEMA asked us to help them set up a disaster recovery site," Lewis said, adding that the site was located in an old grocery store building. The city's IT Department, its Building Services Department and FEMA got that up and running in just a few days. The IT Department also worked with a joint housing task force and staff from IBM to develop a Web site where evacuees could register for housing. Property owners, hotel operators and residents who wanted to sponsor evacuees also used the site to register their offerings. "On the heels of all that, we get a [Category] 5 hurricane headed right up Galveston Island, and we thought we were going to take a direct hit," Lewis said. To prepare for Rita, the city established a team of its network personnel to restore connectivity, both voice and data, in case the hurricane took out the city's communications systems. "We pre-positioned those people in six different locations around the city." Fortunately Rita swerved east and the city's IT infrastructure suffered little damage. Lewis also took on non-IT responsibilities, developing recovery plans and managing an operation that brought special-needs evacuees back to Houston after Rita passed. Losing a Cold War Mentality One lesson that emerged for IT executives after the hurricanes was a cheerful one: In time of need, technology vendors are ready to lend a hand. "It took just a few phone calls to our partners to receive computers, networking gear, software," Burns said. Lewis concurred. "The vendor community couldn't have been more helpful," he said. To prepare for future disasters, government IT professionals should make sure they have application software to meet the needs of humanitarian efforts related to the disaster response, Litchliter said. "This would include software for missing persons, warehouse and distribution management, and volunteer/donations management," he said. "We were forced to write or locate software for these functions." As Mississippi replaces the communications towers and public safety operations centers that Katrina destroyed, it will probably site the new ones farther from the coast, so there's less risk of flooding, Litchliter said. "If possible, the state may want to have more ready access to portable towers they can bring in and set up in certain areas. We had a few of them, but probably not as many as were needed." In addition, the hurricane has given new urgency to an initiative to build a statewide radio system that will provide better communications among first responders. "I think we'll move forward with that project rather quickly," he said. In Alabama, Katrina showed that the state could use "a little more GIS," Burns said. With better geographic information, rescuers can more quickly locate people who need help in areas where most signs and landmarks are no longer standing. The post-Katrina work in New Orleans taught the CDC team that it's important to bring along as many resources as possible, such as power generators and food, when you go into a disaster area, said McQueen. For Meffert, the big lesson learned from Katrina is that when governments plan for disaster response, they need to lose their Cold War mentality, which assumes a centralized command center and centralized backup for power and data systems. "In a true disaster, nobody's in the same room," he said. "They're not even in the same building." A backup generator is great as long as there's diesel to run it, but after Katrina, "I couldn't get fuel for 10 days." Governments must develop command and technology structures that operate more like the Internet, which constantly reroutes communications paths to move information effectively, Meffert said. "You could have that apply to communications back and forth between agencies." For instance, voice calls would have a centralized number that would automatically rotate, so that if the first person on the list didn't respond, the call would automatically roll over to the next contact. If New Orleans had had a more decentralized system in place before Katrina, Meffert said, "it would have made a huge difference." |