Information Warfare: September 1, 2004


The Internet has become a major source of useful intelligence information, even though the people leaving messages on the web dont believe they are talking out of class. The U.S. Department of Defense, NSA, NRO and CIA have long worked at creating tools to pull useful information from the masses of data they accumulate. In the past there have been mainframe and workstation based systems, that tended to get overwhelmed by the sheer mass of data to be searched. As a result, these earlier attempts at data mining, or simply data query, were not terribly successful. Although these systems were classified, eventually they word got out on some of them. 

In the last decade, computers have gotten much more powerful, and cheaper. Then the Internet came along, and before long there were a billion people on it. This had a largely unforeseen impact on the intelligence business. In the last few years, the U.S. government has made reference to Internet chatter as proof that al Qaeda was up to something. What few people realize is that there is probably a new generation of search tools the intel people are using, that work much more effectively than the old ones from the 1980s (and earlier.) How do these tools work? That can be seen from an examination of commercial versions of this software that are beginning to appear. For more than a decade, financial firms developed data mining tools for the news, the better to get advance warning of changing trends. These search systems were generally kept secret, as they were very lucrative tools for those who worked the markets in stocks, currency and related items. Some of these systems now mine the Internet as well. 

Now an Israeli firm, Trendum, comes along and offers to sell anyone the ability to mine the Internet. Openly using the same tools (neural networks, rule induction, decision trees, and pattern analysis, text and linguistic analysis) everyone else has been using in secret , and propelled by lots of powerful, but cheap, computer hardware. Trendum mines the Internet for developing trends. Major corporations, news organizations, and the UN, are lining up to use Trendums services. What Trendum will reveal is what the chatter of thousands, or millions, of people (depending on the subject) indicates. Most questions are of commercial significance, like how popular a new car, beer or movie is. But such systems can also reveal if terrorists are really about to make some attacks, or are just chattering. 


Article Archive

Information Warfare: Current 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 2004 2003 2002 2001 2000 1999 



Help Keep Us Soaring

We need your help! Our subscription base has slowly been dwindling. We need your help in reversing that trend. We would like to add 20 new subscribers this month.

Each month we count on your subscriptions or contributions. You can support us in the following ways:

  1. Make sure you spread the word about us. Two ways to do that are to like us on Facebook and follow us on Twitter.
  2. Subscribe to our daily newsletter. We’ll send the news to your email box, and you don’t have to come to the site unless you want to read columns or see photos.
  3. You can contribute to the health of StrategyPage. A contribution is not a donation that you can deduct at tax time, but a form of crowdfunding. We store none of your information when you contribute..
Subscribe   Contribute   Close