In the difficult days after 9/11, White House officials quietly passed the word through Washington's alphabet soup of intelligence agencies: tell us which weapons you need to stop another attack. At the supersecretive NSA, the National Security Agency (also known as No Such Agency), the request came back: give us permission to collect information on people inside the United States. The NSA had been struggling, without much success, to listen in on terrorists who use cheap and easily available encrypted phones, and officials eagerly drew up a wish list, according to a participant in the discussions. This source, who declined to be identified discussing sensitive matters, said NSA officials did not really expect the White House to say yes to domestic spying. After scandals over wiretapping erupt-ed in the 1970s, the code breakers and electronic sleuths at the NSA had been essentially restricted to eavesdrop-ping on conversations between foreigners abroad. American residents and even most foreign visitors to the United States were off-limits to "Big Noddy," as NSA insiders call their giant "Ear in the Sky" surveillance capability.

But after 9/11, president George W. Bush wanted fast action. He believed that most Americans thought their government should do whatever was necessary to catch terrorists before they struck again. Though the details remain highly classified, the "National Security Presidential Directives" issued by Bush called for an all-out war on terrorism, including, it is generally believed, expanded electronic surveillance. Out went the old rules—a 1980 document called "U.S. Signals Intelligence Directive 18," which sharply limited domestic surveillance; in came a new, still dimly understood regimen of domestic spying.

Desperate times call for desperate measures. In times of war, open societies have been willing to accept the need for secret spy services. Americans now spend upwards of $40 billion a year on intelligence. Given a hard choice between security and privacy, most Americans would probably choose to sacrifice some of the latter to get more of the former. The harder question is whether the techno wizards at the NSA, overwhelmed by tidal waves of digital data, searching for tiny poisonous fish in a giant sea, can provide true security from another 9/11.

There can be no doubt that Bush correctly read the public mood in the days and weeks following the 2001 attacks. And had the president sent a bill up to Capitol Hill giving the NSA broad powers to wiretap and eavesdrop inside the United States, in all likelihood, the lawmakers would have shouted it through. But the president did not ask for public support. Instead, like most chief executives charged with running the modern national-security state, he chose the path of secrecy. True, the administration's spymasters confidentially briefed congressional leaders on the new eavesdropping program. But some of the lawmakers now claim they were confused, or misled, or somehow did not fully understand what the spooks were telling them. Perhaps the legislators weren't fully informed. Or perhaps they didn't really want to hear what they were told.

In any case, the story eventually, and inevitably, leaked. Last December, The New York Times revealed that the NSA had eavesdropped on thousands of phone calls between people in the United States and foreign countries without first obtaining warrants. Then, last week, USA Today reported that the NSA had amassed a vast database of billions of calls inside the United States—not the content of the calls themselves, but a record of when and to which phone numbers the calls were made and for how long. (The government did not ask the phone companies for names and addresses, but the simplest Internet search of a phone number can divulge that information.) The revelation was another blow to Bush, whose approval rating in the new NEWSWEEK Poll dipped to 35 percent, his record low in the survey, and it may slow the administration's plan to find a CIA director who can restore morale at the beleaguered intelligence agency. The brewing scandal is likely to entangle the government and the phone companies that helped in a legal morass.

Administration officials have always insisted that any eavesdropping or "data collection" had been narrowly focused on Al Qaeda terror suspects. It is hard to determine if the NSA goes on fishing expeditions. A senior administration of-ficial, who declined to be identified discussing classified matters, acknowledged to NEWSWEEK that the NSA had crunched through vast databases to help identify suspects who may have then been subjected to electronic eavesdropping, either without a warrant or under court order. This official claimed that the NSA program had helped gather evidence that had foiled terrorist operations, though the official would not be more specific. If the program "leads to one disruption of another 9/11, then it would be worth it," said the official. But other administration officials interviewed by NEWSWEEK questioned whether the fruits of the NSA program—which they doubted, though not publicly at the risk of losing their jobs—have been worth the cost to privacy. And many Americans naturally wondered whether Big Brother was watching or listening in ways that are still unknown. There are hints, for instance, that the government has been fishing the Internet as well as the phone lines.

Michael Hayden, Mith McConnell
In San Francisco, a privacy group called the Electronic Frontier Foundation has filed a lawsuit based in part on the testimony of Mark Klein, an AT&T technician for 22 years who claims he witnessed the construction of a "secret room" for the NSA at AT&T's San Francisco headquarters in early 2003. Later that year, Klein says, he discovered that cables from the secret room were tapping into massive volumes of Internet communication. Klein says he discovered similar operations in other cities on the West Coast, and now concludes that the NSA had created the capability of "vacuum-cleaner surveillance" of all data crossing the Internet. AT&T says it has always obeyed the law and worked to safeguard the privacy of its customers. The federal government has mostly remained mum, though at a Dec. 19 White House briefing, Attorney General Alberto Gonzales somewhat cryptically referred to "many operational aspects" of the eavesdropping program "that have still not been disclosed." After the USA Today story, President Bush told reporters, "We are not trolling through the personal lives of millions of innocent Americans."

Whether that is strictly true will likely be on the agenda this week as lawmakers on the Senate intelligence committee grill Air Force Gen. Michael Hayden, Bush's choice to take over the troubled CIA. Hayden ran the NSA before and after 9/11, when the agency was expanding its surveillance programs. "I have substantial questions about his credibility," Senate intelligence committee member Ron Wyden, an Oregon Democrat, told NEWSWEEK. He points to Hayden's public statements that the NSA monitored only international calls. "There was never any mention of establishing a domestic database," says Wyden.

Republicans defending Hayden's nomination can counter with some early polls showing that most Americans support expanded electronic surveillance to catch terrorists, even if it intrudes on their privacy. (Much depends on the wording of a poll question, of course, and later polls showed more skepticism. The NEWSWEEK survey found 53 percent agreed with the statement that NSA data collection "goes too far in invading people's privacy," while only 41 percent agreed that the collection program is "a necessary tool to combat terrorism.") Most legal experts seemed to agree that the government could collect a huge database of phone records without violating the Constitution's ban on "unreasonable searches and seizures." Still, the phone companies that cooperated with the NSA—AT&T, Verizon and BellSouth—will be hauled into court, accused by their customers of violating the arcane and murky restrictions of various federal communications laws. All of them have protested that they were complying with the law, though it has been noted that they were paid for their cooperation, and lawyers suing the phone companies will undoubtedly want to know if they were pressured by threats to withhold valuable federal contracts. One much smaller phone company—Qwest, based in the Rocky Mountain states—refused to turn over its call records, arguing that the NSA never satisfied the company's legal doubts about the agency's request.

Americans are not naive about the need to snoop at home and overseas. In 1929, Secretary of State Henry Stimson shut down a secret code-breaking operation called the Black Chamber by saying, "Gentlemen do not read each other's mail." But America's enemies are apt to play dirty, and during World War II and the cold war, the federal government decided, in effect, to play dirty, too—to steal secrets and eavesdrop, at home as well as abroad.

Washington spun a huge web of intelligence agencies with acronyms familiar (like CIA and FBI) and obscure (like NRO—for National Reconnaissance Office—to operate spy satellites). The attitude toward secret or "black" operations was, at first, rather "stiff upper lip" and British. Policymakers did not want to know too much about what the spooks were up to. Presidents were protected by the doctrine of plausible deniability. They were supposed to be able to say, plausibly, that they really didn't know how that secret was stolen—or that a journalist's phone was tapped or that a foreign government was overthrown. If caught, American spymasters were supposed to fall on their swords and take responsibility.

Of course, blametaking didn't quite work so stoically in practice. During the Watergate scandal, it emerged that the Feds had been carrying on a program of domestic spying, tapping phones and opening the mail of real and imagined enemies of the state. At the 1975 Church Committee hearings, intelligence officials squirmed and pointed fingers. New laws were enacted, including the 1978 Foreign Intelligence Surveillance Act, which requires the Feds to get a warrant from a secret court before eavesdropping on foreign calls in and out of the United States.

The NSA was banned from any domestic espionage. At those 1975 hearings, Sen. Frank Church, the chairman of the committee appointed to investigate intelligence abuses, made a statement that today seems ominous and possibly prescient. The Idaho senator said he was most worried about the NSA. The secret agency's capabilities were so great they "could be turned around on the American people, and no American would have any privacy left, such is the capability to monitor everything, telephone conversations, telegrams, it doesn't matter. There would be no place to hide."

The NSA does have vast capabilities. One senior U.S. intelligence official, speaking anonymously because of the sensitivity of the subject, told NEWSWEEK that the heat generated by the NSA's secret supercomputers has been so great that officials have been talking about carting in snow and ice to mask the machines from the prying sensors of foreign spy satellites.

But increasingly, there has been talk of the agency's "going deaf." The NSA had its best luck monitoring Soviet lines of communication—for example, a microwave transmission from Moscow to a missile base in Siberia. But the new enemy is more shadowy and elusive. In 2002, General Hayden told NEWSWEEK, "We've gone from chasing the telecommunications structure of a slow-moving, technologically inferior, resource-poor nation-state—and we could do that pretty well—to chasing a communications structure in which an Al Qaeda member can go into a storefront in Istanbul and buy for $100 a communications device that is absolutely cutting edge, and for which he has had to make no investment for development."

According to most accounts, the NSA remains behind the telecommunications curve. A December 2002 report by the Senate intelligence committee noted that only a "tiny fraction" of the NSA's 650 million daily intercepts worldwide "are actually ever reviewed by humans, and much of what is collected gets lost in the deluge of data." Hayden told NEWSEEK that year that the NSA had been slow to catch up to new technology, and that he was obsessed with turning the enemy's "beeps and squeaks into something intelligible."

One of Hayden's most ambitious initiatives was called Trailblazer. It was a program aimed at helping the NSA make sense of its many databases—to put them to use. By more efficiently locating and retrieving messages, Trailblazer could help the NSA "data-mine," to find patterns in the huge volume of electronic traffic that might help lead sleuths to a terror suspect. Instead, the program has produced nearly a billion dollars' worth of junk hardware and software. "It's a complete and abject failure," says Robert D. Steele, a CIA veteran who is familiar with the program. Adds Ed Giorgio, who was the chief code breaker for the NSA for 30 years: "Everybody's eyes rolled when you mentioned Trailblazer."

What went wrong? The NSA apparently tried a clunky top-down approach, trying to satisfy too many requirements with one grand solution, rather than taking a more Silicon Valley-like tack of letting small entrepreneurs compete for ideas. John Arquilla of the Naval Postgraduate School at Monterey, Calif., a renowned "network" intelligence expert, says: "The real problem Big Brother is having is he's not making enough use of the Little Brothers"—the corporations that have become expert at manipulating databases for commercial use.

"Data mining" has been a boon to credit-card companies that can match customers and products. It has also helped the Feds track drug dealers who constantly buy and throw away cell phones (the technology can monitor frequent phone-number changes). Identifying and tracking terrorists may be a taller order. For one thing, terrorists have learned not to even use phones. A computer disk or message between, say, Osama bin Laden and Iraqi insurgent leader Abu Mussab al-Zarqawi is hand-delivered. Some terrorists have learned to leave messages hidden in Web sites. Others are given passwords to go on the Web sites and find the messages. Since that process involves no electronic communication—no e-mail or phone call—the NSA is kept in the dark.

Effective data mining might have averted 9/11, notes Philip Bobbitt, who served as a National Security staffer in the Clinton administration. On Sept. 10, 2001, the NSA, monitoring pay phones in Qaeda-controlled Afghanistan, intercepted two messages, "The match begins tomorrow" and "Tomorrow is zero hour." No one knew what to make of these messages, which in any event weren't translated until Sept. 12. But the CIA and FBI had the identities of two of the hijackers, who had been linked to earlier Qaeda plotting, in the agencies' computers. "Had we at the time cross-referenced credit-card accounts, frequent-flier programs and a cell- phone number shared by those two men, data mining might easily have picked up on the 17 other men linked to them and flying on the same day and at the same time on four flights," Bobbitt recently wrote in The New York Times.

There are doubts within the upper levels of the U.S. government that the NSA, four-and-a-half years after 9/11, is any better equipped and run to piece together the next "Tomorrow is zero hour" intercept. NEWSWEEK has learned that some top government lawyers were troubled by the NSA data collection and search program—not on legal grounds so much, but because they doubted its efficacy. A senior administration official who was involved in legally vetting the NSA program but declined to be identified discussing sensitive matters says that a crude cost-benefit analysis left him uneasy. The NSA program ran a risk of intruding on the privacy of Americans. There are always "false positives." National Journal's Shane Harris conjured up the example of a book agent who represents a journalist who once interviewed Osama bin Laden. A faulty pattern analysis could make him a terror suspect. To justify the risk of dragging such innocents into government investigations, there needs to be evidence showing a high probability of return on the investment—the prospect of actually catching a terrorist.

So far, the best catch the Feds have offered up is a truckdriver named Iyman Faris, who conceived a rather farfetched plot to cut down the Brooklyn Bridge with a blowtorch. (Faris was apparently identified by a captured Qaeda leader; it's not clear the NSA played any role.) Of course, intelligence services do not always brag about their successes, and one U.S. official privy to the intelligence tells NEWSWEEK that another attack on an urban area in the United States was averted as well. The official would not discuss the plot for fear of revealing NSA listening methods.

There has been at least some debate inside the administration over how much license to give the NSA. In the spring of 2004, senior Justice Department lawyers objected to warrantless eavesdropping. For several months, until new rules to safeguard privacy were adopted, the program was suspended. It is not clear whether the NSA's data-collection program was also put on hold or altered in some way.

The administration is not eager to air its internal debates. At the Justice Department, an internal watchdog, the Office of Professional Responsibility, began an investigation into whether DOJ lawyers had behaved unethically by interpreting the law too aggressively—by giving a legal green light to coercive interrogations and warrantless eavesdropping. But the OPR lawyers had to drop their investigation last week when the administration refused to give them the necessary security clearances.

Catching Al Qaeda or some shadowy terrorist offshoot before it strikes again will take all the tools of spy tradecraft—old-fashioned human intelligence (HUMINT) as well as signals intelligence (SIGINT) like electronic eavesdropping. It is frustrating to think how close the CIA and FBI came to stopping 9/11. After Al Qaeda bombed the American embassies in Kenya and Tanzania in 1998, local police managed to catch one of the would-be bombers who had decided not to commit suicide in the blast. The conspirator was turned over to American intelligence officials, who persuaded the man to give up the phone number of a Qaeda safe house in Yemen. The NSA began listening in on the phone line of the safe house. American agents were tipped to a Qaeda terror summit in Kuala Lumpur in January 2000. Two of the 9/11 hijackers—Nawaf Alhazmi and Khalid Almihdhar—were at that summit. Somehow, the CIA failed to hand over the identities of these two terrorists to the FBI in time for the slow-moving bureau to track them before they flew into buildings on 9/11.

That was a human error, but it was caused in part by the culture of secrecy that permeates the national-security state. The CIA and FBI are renowned for their turf wars and unwillingness to share secrets. It's hoped that intelligence reform and the shame of failure have prodded the intelligence agencies to share a little more. As the late senator Daniel Patrick Moynihan observed, during the cold war excessive secrecy did more to hurt national security than to help it. In an overly secretive world, assumptions go untested and rigorous thinking is stifled. The CIA, for instance, failed to predict the collapse of the Soviet Union, in part because agency analysts refused to reach out to outside economists and experts.

It is true, as the old World War II saying goes, that "loose lips sink ships." But by refusing to tolerate an open discussion of new rules post-9/11, the Bush team lost a chance to gain public support for the necessary trade-off between security and privacy. Figuring out how to track and find Internet-savvy terrorists is a daunting task. Government officials—even the superspooks of the NSA—need all the help they can get.

With Michael Hirsh, Michael Isikoff, Daniel Klaidman, Richard Wolffe, Holly Bailey and John Barry

© 2006 MSNBC.com