How Treasury, AT&T botched billion-dollar network upgrade

Feds, carrier scramble to get massive network consolidation effort back on track

Three years ago, the U.S. Treasury Department awarded a network consolidation project worth as much as $1 billion to AT&T, which claimed to win the deal due to its superior "transition plan."

Today, both the federal agency and its carrier admit the Treasury Network (TNet) -- which involves consolidating 15 separate networks into a single, secure IP backbone -- is notably behind schedule and continues to experience performance problems.

Learn from Treasury's networking mistakes|
Fed's IPv6 plan called a "game changer"

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

Three years ago, the U.S. Treasury Department awarded a network consolidation project worth as much as $1 billion to AT&T, which claimed to win the deal due to its superior "transition plan."

Today, both the federal agency and its carrier admit the Treasury Network (TNet) -- which involves consolidating 15 separate networks into a single, secure IP backbone -- is notably behind schedule and continues to experience performance problems.

Learn from Treasury's networking mistakes|
Fed's IPv6 plan called a "game changer"

Indeed, TNet has encountered so many issues that the umbrella program that funds it -- which goes by the acronym ITT TSS for IT Infrastructure Telecommunications -- recently landed on the Office of Management and Budget's list of 26 high-risk federal IT projects. Federal CIO Vivek Kundra released the high-risk list a month ago, threatening to discontinue any of the projects that can't be turned around.

How did TNet get so far off track?

In September 2007, TNet was the first major award under Networx Universal, which is a governmentwide telecommunications program managed by the General Services Administration. AT&T won TNet in a competitive bid believed to include Networx Universal rivals Verizon and Qwest.

TNet is approximately two years behind its original schedule. The transition of the Treasury's predecessor data network -- called TCS for Treasury Communications System -- was supposed to begin in November 2007 and take about a year, according to a January 2010 report by the agency's Office of Inspector General. Instead, the agency just finished migrating the TCS circuits to AT&T's Networx contract in August 2010.

Treasury blames the delays on new network security requirements that were put in place by the Bush Administration in November 2007, shortly after TNet was awarded.

These requirements -- dubbed the Trusted Internet Connection (TIC) initiative -- mandate that all federal agencies deploy a standard set of security tools including antivirus, firewall, intrusion detection and traffic monitoring on their networks to protect against hacking attacks launched by foreign governments and criminals.

"Treasury's TNet transition was complicated by the introduction of additional requirements introduced post-award (e.g., the Trusted Internet Connection initiative)," said Diane Litman, Acting Deputy Assistant Secretary for Information Systems and Chief Information Officer of Treasury, in a written response to questions. "Treasury was also challenged in melding different operational models of the legacy telecommunications systems of our bureaus into a single model compatible with the managed service nature of TNet."

Treasury said it has made progress in getting TNet back on schedule, but that it still hasn't reached its goal of consolidating all of its networks onto a single WAN.

"A concerted effort on the part of Treasury, its bureaus, and the vendor (AT&T) culminated in the migration of its legacy data telecommunications circuits to Networx in August 2010," Litman said, adding that the legacy data circuits were disconnected by the end of September.