[FoRK] Why We Are Losing The War on Terrorism

J.Andrew Rogers andrew at ceruleansystems.com
Mon May 3 22:08:30 PDT 2004

On May 3, 2004, at 9:14 AM, Contempt for Meatheads forwarded:
> We desperately need adult supervision and high quality minds in the 
> intelligence business!  I am growing more convince that the security 
> clearance process, the government hiring/promotion process, and 
> information silos are overwhelming our ability to get even a 
> marginally adequate level of intelligence needed to fight terrorism.  
> Wow, this is depressing.
> My confident belief (100%):  we will continue to lose the war on 
> terrorism until we fix our intelligence system.

I think this analysis is correct, but also a bit too shallow to be 
really insightful.  While there are some significant institutional 
problems and byzantine self-defeating regulations, these things are 
masking a much bigger technical problem that desperately needs to be 
tackled from their perspective.

The volume of data they collect has reached the point where good 
analysis is no longer tractable in a theoretical algorithmic sense with 
the best tools they currently have at their disposal, particularly when 
you have a data space as broad and diffuse as "terrorism" to sift.  
Institutional procedures and problems aggravate this, but the 
underlying issues are deeper.

One of the ways I keep track of what the US DoD is up to is by analysis 
of the open research programs, contracts, and grants that they publish. 
  By threading the many, many programs together over time, you can see 
how fast different technologies are progressing and you can chain 
inferences to make an intelligent estimate as to when specific 
capabilities (which may require the intersection of multiple research 
tracks) could theoretically be available to the DoD.  Furthermore, the 
program managers have a habit of mildly editorializing their program 
descriptions in response to some of the proposals they have received 
and the success of the proposals they have actually funded, which also 
gives some added insight.

One thing that I have noticed for several years is that the advanced 
data mining and automated intelligence analysis research programs have 
been essentially stalled for many years now despite aggressive 
marketing and a large number of agencies willing to liberally fund 
proposals.  And the editorializing of the program managers on this 
research track makes it clear that they are quite frustrated both with 
the lack of progress in this area and with the fact that research 
proposals keep trying to beat the same dead horse over and over.  
Furthermore, while most programs have a shelf-life after which they are 
either closed (both on good progress or no progress), these particular 
programs keep getting extended and re-funded over and over, sometimes 
under a different name but always with roughly the same parameters.

As long as this program track is stuck in neutral, the intelligence 
agencies will have serious problems that will be all but 
insurmountable.  The US intelligence service is a victim of its own 
ability to acquire data.  This isn't a problem that they can simply 
throw money at in the sense that it requires pretty substantial 
algorithm breakthroughs to even be tractable for high-quality analysis. 
  To date, private research organizations have clearly been unable to 
solve this problem in any meaningful way, and there is substantial 
evidence of this fact.  In the mean time, they are left using narrow 
brittle algorithms to sift and analyze the data, with holes you could 
drive a truck (bomb) through.

Someone who fully understood the theoretical limitations and likely 
implementation parameters of the current state-of-the-art could likely 
defeat the automated analysis.  Fortunately for the intelligence 
agencies, few people have those skills and they get by on a pretty 
broken system hampered further by institutional problems.

j. andrew rogers

More information about the FoRK mailing list