Darpa Is Developing a Search Engine for the Dark Web

A new search engine being developed by Darpa aims to shine a light on the dark web and uncover patterns and relationships in online data to help law enforcement and others track illegal activity. To achieve this goal, Memex will not only scrape content from the millions of regular web pages that get ignored by commercial search engines but will also chronicle thousands of sites on the so-called Dark Web---such as sites like the former Silk Road drug emporium that are part of the TOR network's Hidden Services. Although sites do exist that index some Hidden Services pages---often around a specific topic---and there is even already a search engine called Grams for uncovering sites selling illicit drugs and other contraband, the majority of Hidden Services remain well under the radar. The content on Hidden Services is public---in the sense that it's not password protected---but is not readily accessible through a commercial search engine.