DARPA Touts Deep-Web Search in Hunt for ISIS

ISIS Humvee

An experimental Pentagon tool to search the deep web is now being used in the hunt for Islamic militants, an official said.

The research project, known as Memex and developed by the Pentagon’s research and development arm, was discussed this week at the first annual Future of War conference. The event was organized by the New America Foundation, a nonpartisan research organization in Washington, D.C.

Arati Prabhakar, a physicist and director of the Defense Advanced Research Projects Agency, said the program has already been used to help police in Dallas trace sex trafficking networks to North Korea by searching the so-called deep web, or websites not crawled by popular search engines such as Google or Bing.

“Our law enforcement colleagues, they were sort of taken aback I think initially by how rich that data set was,” she said. “You can imagine how that might give you a way to see how the ISIS global community that’s spreading like this cancer — how they are using that infrastructure similarly.”

Prabhakar confirmed that the software is helping to track the online activities of militants affiliated with the Islamic State in Iraq and Syria, or ISIS. “That work is just beginning,” she said, but didn’t go into specifics. “Because it’s live and we’re in a wartime situation, that’s not going to be an area that we can talk about in a lot of detail,” she said.


Instead, Prabhakar talked about how the project was used in Texas.

“We looked at back page ads … and from that we were able to build a very quick assessment of where the same phone numbers kept showing up on multiple websites,” she said. “If you’re looking across thousands and thousands manually, you wouldn’t have seen it. But we were able to scoop up these high-value phone numbers and hand them to law enforcement.”

She added, “Many of those numbers tied to criminal violations that they already knew about through conventional law enforcement means. More interestingly from a national-security point of view, they found that some of those numbers linked to fund transfers in the region around North Korea, and that started them on the trail of looking for a trafficking network.”

The Memex project also seeks to address other shortfalls associated with today’s search process by saving sessions, allowing imprecise search terms and multiple queries, and archiving results beyond a list of links, according to a release on Darpa’s website. Technical areas of interest include domain-specific indexing and search.

The effort is headed by Christopher White, an engineer with a doctoral degree in electrical engineering from the Johns Hopkins University.


“We’re envisioning a new paradigm for search that would tailor indexed content, search results and interface tools to individual users and specific subject areas, and not the other way around,” he said in a release on the site. “By inventing better methods for interacting with and sharing information, we want to improve search for everybody and individualize access to information. Ease of use for non-programmers is essential.”

The program draws its name and inspiration from a hypothetical device described in a 1945 Atlantic Monthly magazine article, titled “As We May Think” and written by Vannevar Bush, director of the U.S. Office of Scientific Research and Development during World War II, according to the release.

About the Author

Brendan McGarry
Brendan McGarry is the managing editor of Military.com. He can be reached at brendan.mcgarry@military.com. Follow him on Twitter at @Brendan_McGarry.
  • Mitch S

    Using the popular, ordinary, search engine Google, I came up with this link for those who want to see Bush’s article:

  • GI dude

    Why not just nuke the entire middle-east and be done with it?

  • blight_

    Throw in extensive deep packet inspection while you’re at it…since the NSA does control the hubs where the world’s internet traffic goes through. Like in Persons of Interest, they probably discard everything that has nothing to do with terrorists. Kiddie porn? The NSA probably has a huge pile of it gzip’ed away in internet snapshots.

  • تحميل فايرفوكس

    The world is on the edge.

  • NMI

    Ozzy Osbourne said it best “Thank God for the Bomb. Nuke `em! Nuke `em!”

  • blight_

    Blight touts his twitter-search for ISIS.

  • oblatt22

    Terrorism is justa smokescreen. We know from the FISA courts that 99% of this will be used to spy on Americans so that their can be blackmailed at the authorities leisure.

  • steve

    This is good, the internet and social media are an essential tool for today’s terrorists.

    What the Talking Heads got wrong with ISIL recently is. it isn’t enough to fight them. You have to go after them on the internet, shut down their funding, and try to cut down recruitment. Everybody jumped down that woman from the State Department’s throat for suggesting we try to create jobs for them. It really showed the ignorance of commenters. It’s telling that in some areas, joining ISIL is the only/best way to support your family.

    Most people don’t even understand how the NSA scans the internet. The NSA doesn’t give two craps about the average internet user. The software scans before a human even looks at the data. If the software tags you for more observation, chances are, you’re shady as sh@t.

    • blight_

      The NSA is interested in finding bad guys. Maximizing sensitivity (true positive rate) and reducing the false positive rate is where you want to be. This means putting aside a good deal of data that is “normal people”, since storing data that you have no interest in is an expensive prospect.

  • blight_

    Curious how they intend to go after websites that are unlinked. I guess they could go and cross-reference all ICANN domains and check them all for content. But for the websites that don’t have domain names and just IP addresses…brute-forcing every IP address is an option as well. In addition to that, brute-forcing all combinations of URL and extension may be options as well, but a savvy administrator running his own cluster for kiddie pornos may notice someone trying to scrape content from his servers, and assume it is a Ddos attack.

    • steve

      The kiddie porn guys are already sweating. There’s a whole other bag of tricks they use. Image analysis and such. One group came out with how to ID the kids and figure out a location. Their first proof it worked is they tracked down a girl in a video and found out the guy had already been busted for it, but, it proved their technique correct. Another group busted a guy because they figured out the location from the Sun and a road sign that could barely be made out by the window. I keep tabs on that field since I stumbled upon the tricks they were using to hide a decade ago. I was working at an internet provider and my manager suggested I search the newsgroups for info about computer security. These dirtbags were posting on how they secured their data. From what I gather, if you share kiddie porn, eventually, you’re getting caught, which makes my day.

  • John

    I believe nuclear weapons have been advanced to the stage of significantly reducing nuclear fall out.

  • IT 2 IT

    WHY don’t they go for the head of this “SIR—pent’?

    WHY don’t they LAY HOLD the CFR-RIIA-Trilateral globalist mafia?

    WHY don’t they LAY HOLD the ‘SORE—Os’?

  • Cataldo

    From my point of view, in mediterraneo, i can’t find a real reason for fighting ISIS, all their action in ME is useful to US interest ;) all about fighting ISIS is a bad propaganda.

  • oblatt22

    American companies are being removed from approved buyer lists in quite a few places because American equipment cannot be trusted.

    NSAs activities are going to end up costing trillions of dollars in lost sales in one of the few remaining industries where we are competitive.

  • rtsy

    Barely veiled excuse to spy on everyone and everything that moves on the internet. This MIGHT help you catch a few disaffected citizens thinking of joining up with the Islamic State but won’t come close to touching the real ISIL threat.