The recent tragedy at Fort Hood has people griping about missed signals that could have been used to prevent the attack. However, I will argue that is likely to be impossible to ever have a system that can screen out all terrorists without also flagging a lot of innocent people. The calculation is a simple exercise in probability theory that is often given in first year statistic classes.
Suppose we have a system in place that gives a yes Y or no response of whether or not a person is a terrorist T. Let P(T) be the probablity that a given person is a terrorist, P(T|Y) be the probability that a person is a terrorist given that the test said yes. Thus P(~T|Y)=1-P(T|Y) is the probability that one is not a terrorist even though the test said so. Using Bayes theorem we have that
P(~T|Y)=P(Y|~T) P(~T)/P(Y) (*)
where P(Y)=P(Y|T)P(T) + P(Y|~T)P(~T) is the probability of getting a yes result. Now, the probability of being a terrorist is very low. Out of the 300 million or so people in the US a small number are probably potential terrorists. The US military has over a million people on active service. Hence, the probability of not being a terrorist is very high.
From (*), we see that in order to have a low probability of flagging an innocent person we need to have P(Y|~T)P(~T)<< P(Y|T)P(T), or P(Y|~T)<< P(Y|T) P(T)/P(~T). Since P(T) is very small, P(T)/P(~T)~ P(T), so if the true positive probability P(Y|T) was near one (i.e. a test that catches all terrorists), we need the false positive probability P(Y|~T) to be much smaller than the probability of having a terrorist, which means we need a test that gives false positives at a rate of less than 1 in a million. The problem is that the true positive and false positive probabilities will be correlated. The more sensitive the test the more likely it is to get a false positive. So if you set your threshold to be very low so P(Y|T) is very high (i.e. make sure you never miss a terrorist), you’ll most certainly have P(Y|~T) to also be high. I doubt you’ll ever have a test where P(Y|T) is near one while P(Y|~T) is less than one in a million. So basically, if we want to catch all the terrorists, we’ll also have to flag a lot of innocent people.