Recent News - Articles and Discussion

[Kasey Wertheim]

http://www.nytimes.com/2009/05/12/scien ... 1&emc=eta1

Plugging Holes in the Science of Forensics
HENRY FOUNTAIN
Published: May 11, 2009

It was time, the panel of experts said, to put more science in forensic science.

A report in February by a committee of the National Academy of Sciences found “serious problems” with much of the work performed by crime laboratories in the United States. Recent incidents of faulty evidence analysis — including the case of an Oregon lawyer who was arrested by the F.B.I. after the 2004 Madrid terrorist bombings based on fingerprint identification that turned out to be wrong — were just high-profile examples of wider deficiencies, the committee said. Crime labs were overworked, there were few certification programs for investigators and technicians, and the entire field suffered from a lack of oversight.

But perhaps the most damning conclusion was that many forensic disciplines — including analysis of fingerprints, bite marks and the striations and indentations left by a pry bar or a gun’s firing mechanism — were not grounded in the kind of rigorous, peer-reviewed research that is the hallmark of classic science. DNA analysis was an exception, the report noted, in that it had been studied extensively. But many other investigative tests, the report said, “have never been exposed to stringent scientific scrutiny.”

While some forensic experts took issue with that conclusion, many welcomed it. And some scientists are working on just the kind of research necessary to improve the field. They are refining software and studying human decision-making to improve an important aspect of much forensic science — the ability to recognize and compare patterns.

The report was “basically saying what many of us have been saying for a long time,” said Lawrence Kobilinsky, chairman of the department of sciences at John Jay College of Criminal Justice in New York. “There are a lot of areas in forensics that need improvement.”

Barry Fisher, a past president of the American Academy of Forensic Sciences and a former director of the crime laboratory at the Los Angeles County Sheriff’s Department, said he and others had been pushing for this kind of independent assessment for years. “There needs to be a demonstration that this stuff is reliable,” he said.

It’s not that there hasn’t been any research in forensic science. But over the years much of it has been done in crime labs themselves. “It hasn’t gotten to the level where they can state findings in a rigorous scientific way,” said Constantine Gatsonis, director of the Center for Statistical Sciences at Brown University and co-chairman of the National Academy of Sciences committee. And rather than being teased out in academic papers and debated at scientific conferences, “a lot of this forensic stuff is being argued in the courtroom,” Mr. Fisher said. “That’s not the place to validate any kind of scientific information.”

Much forensic research has been geared to improving technologies and techniques. These studies can result in the kinds of gee-whiz advances that may show up in the next episode of the “C.S.I.” series — a technique to obtain fingerprints from a grocery bag or other unlikely source, for example, or equipment that enables analyses of the tiniest bits of evidence.

This kind of work is useful, Dr. Kobilinsky said, “but it doesn’t solve the basic problem.”

DNA analysis came out of the biological sciences, and much money and time has been spent developing the field, resulting in a large body of peer-reviewed research. So when a DNA expert testifies in court that there is a certain probability that a sample comes from a suspect, that claim is grounded in science.

As evidence to be analyzed, DNA has certain advantages. “DNA has a particular structure, and can be digitized,” Dr. Gatsonis said. So scientists can agree, for example, on how many loci on a DNA strand to use in their analyses, and computers can do the necessary computations of probability.

“Fingerprints are a lot more complicated,” Dr. Gatsonis said. “There are a lot of different ways you can select features and make comparisons.” A smudged print may have only a few ridge endings or other points for comparison, while a clear print may have many more. And other factors can affect prints, including the material they were found on and the pressure of the fingers in making them.

Sargur N. Srihari, an expert in pattern recognition at the University at Buffalo, part of the New York state university system, is trying to quantify the uncertainty. His group did much of the research that led to postal systems that can recognize handwritten addresses on envelopes, and he works with databases of fingerprints to derive probabilities of random correspondence between two prints.

Most features on a print are usually represented by X and Y coordinates and by an angle that represents the orientation of the particular ridge where the feature is located. A single print can have 40 or more comparable features.

Dr. Srihari uses relatively small databases, including an extreme one that contains fingerprints from dozens of identical twins (so the probability of matches is high), and employs the results to further refine mathematical tools for comparison that would work with larger populations.

“These numbers are not easy to come by at this point,” he said. The goal is not individualization — matching two prints with absolute certainty — but coming up with firm probabilities that would be very useful in legal proceedings.

Other researchers are compiling databases of their own. Nicholas D. K. Petraco, an assistant professor at John Jay College, is studying microscopic tool marks of the kind made by a screwdriver when a burglar jimmies a window. It has been hypothesized that no two screwdrivers leave exactly the same pattern of marks, although that has never been proved. So Dr. Petraco is systematically making marks in jeweler’s wax and other materials, creating images of them under a stereo microscope and quantifying the details, assembling a database that can eventually be mined to determine probabilities that a mark matches a certain tool.

Dr. Petraco, a chemist with a strong background in computer science, looks to industry for ideas about pattern recognition — the tools that a company like Netflix uses, for example, to classify people by the kinds of movies they like. “A lot of computational machinery goes into making those kinds of decisions,” he said.

He figures that if something works for industry, it will work for forensic science. “You don’t want to invent anything new,” he said, because that raises legal issues of admissibility of evidence.

The work takes time, but the good news is that the data stays around forever. So as software improves, the probabilities should get more accurate. “Algorithms and data comparison evolve over time,” Dr. Petraco said.

But it may not be possible to develop useful databases in some disciplines — bite mark analysis, for example. “Using a screwdriver, that’s very straightforward and simple,” said Ira Titunik, a forensic odontologist and adjunct professor at John Jay College. But bites involve numerous teeth, and there are other factors, including condition of the skin, that may make it difficult to quantify them for purposes of determining probabilities.

A few researchers are looking at how errors creep into forensic analysis. The National Institute of Standards and Technology recently established a working group on fingerprints, with statisticians, psychologists and others, “to try to understand the circumstances that lead to human error,” said Mark Stolorow, director of the Office of Law Enforcement Standards at the institute.

In Britain, Itiel Dror, a psychologist who studies decision-making processes, is already looking at human factors. “I like to say the mind is not a camera, objectively and passively recording information,” said Dr. Dror, who has a consulting firm and is affiliated with University College London. “The brain is an active and dynamic device.”

He has conducted studies that show that when working on an identification, fingerprint examiners can be influenced by what else they know about a case. In one experiment, he found that the same examiner can come to different conclusions about the same fingerprint, if the context is changed over time.

The same kinds of contextual biases arise with other decision-makers, said Dr. Dror, who works with the military and with financial and medical professionals. He thinks one reason forensic examiners often do not acknowledge that they make errors is that in these other fields, the mistakes are obvious. “In forensics, they don’t really see it,” he said. “People go to jail.”

Forensics experts say the need for research like Dr. Dror’s and Dr. Srihari’s does not mean that disciplines like fingerprint analysis will turn out to be invalid. “I have no doubt that fingerprint evidence and firearms evidence, once looked into by the appropriate research entities, are going to be shown to be very reliable and good,” said Mr. Fisher, the former American Academy of Forensic Sciences president.

Dr. Kobilinsky said people should not jump to the conclusion that forensic science is bad science. “There’s a lot of experience and knowledge that goes into somebody’s expertise,” he said.

“It’s not junk science. But that doesn’t mean it shouldn’t be improved.”

[Gerald Clough]

As evidence to be analyzed, DNA has certain advantages. “DNA has a particular structure, and can be digitized,” Dr. Gatsonis said. So scientists can agree, for example, on how many loci on a DNA strand to use in their analyses, and computers can do the necessary computations of probability.

“Fingerprints are a lot more complicated,” Dr. Gatsonis said. “There are a lot of different ways you can select features and make comparisons.” A smudged print may have only a few ridge endings or other points for comparison, while a clear print may have many more. And other factors can affect prints, including the material they were found on and the pressure of the fingers in making them.

I'm not sure I agree with the assertion that fingerprints are more complicated. The nature of DNA is that the way it is characterized grew out of the essential research required to understand it. It certainly wasn't uncomplicated. It's just that it had to be clearly represented to be of any use for any purpose.

Fingerprints are not more complicated. They have been, however, usable (in the traditionally little-challenged way) without a strict coding. Pore unit ridge structures likely can be modeled in a fairly straightforward way, and the features used in identification and their relationships likely can be described in an uncomplicated way, things that will almost certainly have to be done to craft the needed studies.

Dr. Gatsonis does, as people often do, mix the issues of ridge structures' ability to discriminate with the very different issue of humans interpreting features in impressions of varying quality and subject to various distortions. It is incorrect to consider the two together to judge the complexity of the whole. That accurate interpretation requires considerable experience and some degree of inspiration and native ability doesn't make it "complicated." It just renders it beyond present machine capabilities. It is indeed a difficult task to model, but it's not on account of being complicated that it's an expert task.

If we just today had noticed fingerprints and thought of using them to identify individuals, we would likely rather quickly develop a way to clearly code details and relationships among them and would take that as a matter of course, just as we do DNA coding.

[Outsider]

DNA analysis came out of the biological sciences, and much money and time has been spent developing the field, resulting in a large body of peer-reviewed research. So when a DNA expert testifies in court that there is a certain probability that a sample comes from a suspect, that claim is grounded in science.

This is only partly true. While we know the random match probability of a DNA match very precisely we have very little idea of the likelihood of a false positive due to lab error or contamination. This is similar to discussing "individualisation" in fingerprints while ignoring the practical likelihood of human error.

Profs Thompson, Taroni and Aitken argue (regarding DNA evidence):

"that ignoring or underestimating the potential for a false positive can lead to serious errors of interpretation, particularly when the suspect is identified through a “DNA dragnet” or database search, and that ignorance of the true rate of error creates an important element of uncertainty about the value of DNA evidence."

http://www.bioforensics.com/conference/ ... %20Pos.pdf

And, of course, all forensic identifications are open to errors of logic (the likelihood of the evidence given that the suspect is innocent vs. the likelihood that the suspect is innocent given the evidence)

[Gerald Clough]

DNA analysis came out of the biological sciences, and much money and time has been spent developing the field, resulting in a large body of peer-reviewed research. So when a DNA expert testifies in court that there is a certain probability that a sample comes from a suspect, that claim is grounded in science.

This is only partly true. While we know the random match probability of a DNA match very precisely we have very little idea of the likelihood of a false positive due to lab error or contamination. This is similar to discussing "individualization" in fingerprints while ignoring the practical likelihood of human error.

This is exactly why we need to maintain a strict awareness of the differing natures of the "science" versus the practice. We test the science with ideal data, completely unambiguous and presumed accurate. There's no other way to test the science. The testing of forensic sciences for validity is a matter of determining the valid probabilities, virtually all results being probablistic on account of the unavoidably finite test data sets.

I question whether "error rate" has any useful meaning in many fields. I don't know enough about mechanical tools used in some fields to comment on their rate of error or tolerances. I presume that any mechanism may have some error rate that may be peculiar to the individual machine or class of machines or may apply to all mechanisms. But I think the actual nature of the error must be known if it is to be applied practically. For example, are the errors random or occurring under specific conditions, which might affect how retesting was interpreted.

But when we deal with human observation and interpretation, any overall error rate means nothing, other than the practical judgment on whether or not to reexamine evidence. Individual analysts have their own unique rates of error in observation and interpretation, and these cannot really ever be known with confidence. Where error is possible - and it is always possible - the remedy is reexamination and assessment of the potential for contamination and similar effects and how they may affect results.

[Outsider]

I question whether "error rate" has any useful meaning in many fields. I don't know enough about mechanical tools used in some fields to comment on their rate of error or tolerances. I presume that any mechanism may have some error rate that may be peculiar to the individual machine or class of machines or may apply to all mechanisms. But I think the actual nature of the error must be known if it is to be applied practically. For example, are the errors random or occurring under specific conditions, which might affect how retesting was interpreted.

I think this is a very good point. In industry any problem such out-of-specification goods can be due to common causes (where something that is inherent in the process occasionally produces an extreme result that is unacceptable) or due to special causes (where something happens that is not always present in the process).

Operator error is usually the latter. Special causes are unstable over time. Even if we could test the skill of an operator it would not necessarily be the same next week or even after a cup of coffee. The idea of defining a defect or error rate that has any use in a predictive sense is a non-starter.

For the forensic processes the question is, how to you allow for a small and unpredictable likelihood of error? The question is not so strange, I am sure most evidence has an unpredictable likelihood of error (the RMP element of a DNA match is the exception to this, but not the false positive probability). Professionals using forensic evidence need to know that a small general likelihood of error does not necessarily mean a small likelihood of error in a specific case. This is particularly true if the suspect was only selected as a result of a forensic report and there is nothing else to link him or her to the crime.

[Gerald Clough]

For the forensic processes the question is, how to you allow for a small and unpredictable likelihood of error? The question is not so strange, I am sure most evidence has an unpredictable likelihood of error (the RMP element of a DNA match is the exception to this, but not the false positive probability). Professionals using forensic evidence need to know that a small general likelihood of error does not necessarily mean a small likelihood of error in a specific case. This is particularly true if the suspect was only selected as a result of a forensic report and there is nothing else to link him or her to the crime.

This is where research can only address part of the question. The rest depends on proper advocacy. One cannot depend on overall rates of human perceptual and interpretive error to guide a finder of fact in a case. To do so would be to assign erroneous chances of error in both directions. Proper advocacy has the best potential for managing observational error through the simple process of independent review. But plain error is not always the case, and one role of the advocate's expert is to advise on when the advocate should conduct a more rigorous inquiry in the entire process and how to do it. It is conceivable that an expert properly playing that role might have induced Cowans's counsel to demand production of all potentially useful materials that could relate to the case that might have allowed the elimination records problem and others to be revealed. And, had the Mayfield case gone so far as a trial, I am confident that a truly independent examination would have revealed the interpretive errors, since the OIG had no difficulty demonstrating the nature of the errors.

The latent print examination community simply cannot devise anything within the community that will address everything that must be considered when fingerprint evidence enters a case. When I look at the roster of attendees at the NAS Report seminar I attended and form what I casually hear about similar programs, I realize how very early we are in trying to bring legal professionals into the issue, especially those who deal with criminal trials. The judges present were manly the justices of the highest state appeals court (they funded it). Unfortunately, working down from the top is an expensive way to proceed. I suppose that's just the nature of the system. State habeas attorneys and appeals judges are the agents of change by redefining what should happen at trial. I know from past and ongoing lessons that the trial courts and trial counsel will generally be moved to change by adverse rulings. But, ultimately, even if we were go set our scientific and procedural houses in order tomorrow, it would still be a long time before the legal system does its part.

[Outsider]

This is where research can only address part of the question. The rest depends on proper advocacy.

This needs to be brought right up front to the police officers who first see the forensic reports and law officials who decide when to prosecute. They need to understand how to handle the possibility of error. If it is not rational to be confident that someone has done something wrong they should not be churned through the justice system.

[Gerald Clough]

This is where research can only address part of the question. The rest depends on proper advocacy.

This needs to be brought right up front to the police officers who first see the forensic reports and law officials who decide when to prosecute. They need to understand how to handle the possibility of error. If it is not rational to be confident that someone has done something wrong they should not be churned through the justice system.

Criminal investigation does not proceed in such a nice order that you can determine the entirety of the evidence or even the potential evidence at any stage. It is often necessary and proper to presume reliable and accurate forensic results at an early stage and to move rapidly to a formal charge as the investigation continues. There are issues of public safety, volatility of evidence, and flight that absolutely must be considered. When the early evidence is physical and the investigation can best be done by securing a suspect and other evidence, the mere possibility of error is NOT reason to risk people and evidence by avoiding actions that require charging the suspect. If further evidence then strongly suggests error, right-minded investigators will concentrate the investigation in the directions needed to settle that issue. This is both for the benefit of the suspect and of the investigative goal to reveal the actual actor. Waiting until other evidence happens to confirm the implications of the forensic results is often not an option. It is, in fact, very commonly the case that any possibility of erroneous forensics or forensics that fall short of trial proof can only be determined by charging a suspect and getting him into custody. (I speak here from experience in a state in which there is no ability to take someone into custody "for suspicion." A full criminal charge is required.)

I have personally done many, many cases in which the evidence at the time of the arrest was subject to error, often far more subject to error than fingerprint evidence. Those cases could not be completed without proceeding against the suspect indicated by the evidence. In most of them, the case would not have been completed successfully. You may ask if I have ever arrested someone on the basis of evidence that turned out to be erroneous. Yes. The suspect interview directed the investigation to evidence that the suspect could not have been the person associated with the crime. Even a barely competent defense effort would have discovered it. But had they been the actual actor, as the evidence first indicated, there was no practical way to have discovered what turned out to be exculpatory evidence without wrecking the investigation. I could also recount a case with evidence more than sufficient for conviction in which diligent investigation beyond that point cleared the suspect and placed a very clever individual in jail for contriving the evidence. Police investigation did that, and I am confident that no defense investigation would have revealed the fraud later.

It is not proper for either police or prosecutors to refrain from prosecution in all cases in which they are not absolutely certain of guilt, any more than it would be proper for an attorney to refuse to defend anyone they were certain was innocent. Prosecutors, too, have a client to serve. For a prosecution to be irrational, the weight of the evidence would have had to have been to the side of the defendant being more likely to have not been the actor.

This is not to say that in a case in which there is little more than damning associative evidence, the evidence shouldn't be subject to especially rigorous examination and consideration of any factors that might make it erroneous. This is equally the responsibility of each entity, including the defendant. The controlling entities in the ultimate contest at trial are the prosecutor, defense counsel and judge. And they too often know far too little about the evidence in play. Trial judges in their role in the U.S. system get something of a pass, in that they are to be educated by the attorneys on these issues and should not be making their own inquiries into matters before their courts. In the end, it comes down to who is determined to simply be good at their job. And attorneys for both the State and defense (and judges to some extent) commonly have their standards of reasonable performance established by the not very efficient process of negative feedback in the form reversed convictions, including the unpleasant reversal for having provided ineffective assistance to the defendant.

[Outsider]

The problem with forensic evidence such as fingerprints or DNA is that they are SO reliable that it is difficult and rather unintuitive to know how to translate the small general likelihood of error into the specific likelihood of error in the context of a case.

When the early evidence is physical and the investigation can best be done by securing a suspect and other evidence, the mere possibility of error is NOT reason to risk people and evidence by avoiding actions that require charging the suspect. If further evidence then strongly suggests error, right-minded investigators will concentrate the investigation in the directions needed to settle that issue.

I take the point about needing to act quickly but I don’t think that there should need to be evidence suggesting error before it is considered to be a possibility in some situations. For example, if the subject was unknown to the inquiry prior to the forensic finding, and after investigation no other connection to the crime can be established. Or if it is unrealistic for the subject to have committed the crime under investigation, yet there is no sign of any other wrongdoing in the location for the subject to be lying about. I think in these circumstances error is not a remote possibility, even without specific evidence suggesting an error has occurred.

I am currently in private e-mail correspondence with a criminal justice professional who has to make judgements about prosecution (not in very serious cases). He genuinely finds the logic of the Prosecutor’s fallacy and Defence Attorney’s fallacy difficult to grasp at times, particularly in relation to forensic evidence which has been very reliable in other cases (fortunately prosecuting drunks fighting after the pubs close seldom needs logical reasoning). In theory, judgements about the importance of evidence should be left up to the jury but if we cannot be sure that criminal justice professionals fully understand how to handle expert evidence with a small general likelihood of error, why should we expect juries to do any better?

[Pat A. Wertheim]

The problem with forensic evidence such as fingerprints or DNA is that they are SO reliable that it is difficult and rather unintuitive to know how to translate the small general likelihood of error into the specific likelihood of error in the context of a case.

Hi Steve

Calculating a likelihood ratio for DNA is a rather straightforward proposition. Unfortunately, for fingerprints it is not so simple. Whereas DNA has only four basic components to begin with, and it is the combinations of these components that allow the calculations, fingerprints have a virtually infinite variety of ridge features. Even using the simplest models and considering only ending ridges and splitting ridges, the weights of those two "points" varies in different parts of the print and in different combinations with each other. Still, progress is being made. I was very fortunate to be able to attend a course taught by Cedric Neumann and Glenn Langenburg a couple of weeks ago in basic statistics theory and its application to fingerprints. Cedric has developed a computer model that allows an examiner to plot the ending ridges and splitting ridges in an identification and the program will calculate a likelihood ratio. The model is just that -- a model. It does not make the identification, nor is there any claim that it is perfect. But it is an effort to provide statistical support for an identification. I look forward to the day when a program like Cedric's has been refined and validated so that we can apply it in actual casework. That would not make fingerprints any more scientific because, in my opinion, what we do already is good science. But it would add one more tool to help us understand and present what we do as good science.

[Outsider]

Hi Pat,

The software does sound like a very interesting development. As with all expert evidence what the software tells investigators and the court will have to be interpreted in the context of the facts of the case. I thought of another analogy the other day (I can’t remember if I have previously posted it on this forum):

If you throw a dice 6 times, the odds that you will get 6 sixes is 1 in 6 to the power of 6 (1 in 46,656 according to my calculator). If the police get a tip-off that someone is using a loaded dice, and they go and observe the suspect throwing the dice 6 times and all come up sixes, what are the chances that this happened by chance? (lets assume that it is impossible to inspect the dice, you can only go by the outcome). The likelihood of the evidence (6 sixes) given that the suspect is innocent is 1 in 46,656. Since the police had a tip-off and observe exactly 6 throws, this is also the likelihood that the suspect is innocent given the evidence (I think).

Now, instead of the police getting a tip-off, you read in a newspaper that there was a fight in a casino somewhere in Russia a few months ago. The fight was because someone threw 6 sixes and was assumed to be cheating. What are the chances that the gambler was innocent? There are a awful lot of dice being thrown in the world and we can be pretty certain that 6 sixes WILL be thrown purely by chance at sometime somewhere. Who is to say that this incident in Russia is not that rare event? A statistical expert can say very precisely that the likelihood of the evidence if the suspect is innocent is 1 in 46,656 (same as the tip-off case), but the likelihood that the suspect is innocent given the evidence is not at all clear, it could be more likely than not that he is innocent. It will depend on a lot of things such as if it is easy to get hold of a loaded dice in Russia.

[Gerald Clough]

The problem with forensic evidence such as fingerprints or DNA is that they are SO reliable that it is difficult and rather unintuitive to know how to translate the small general likelihood of error into the specific likelihood of error in the context of a case.

"Reliability" is too vague to really be useful. We have to view each aspect of associative evidence before we can begin to sort out appropriate uses and weights.

Frequency of Matches in The Population
While obviously varying with the population, if that population is well-understood with respect to the characteristic being examined, this is simply what it is. Error occurs when the logical implication is misrepresented, the simplest being that of representing the results as a probability of guilt. When associative evidence is properly viewed, there can be no issue of "erroneous identification," because any scientifically valid analysis will inevitably produce a probablistic conclusion. There are indeed significant issues in how the population is assumed, including potential undiscovered effects, but the numbers are mostly so large that in practice the weight of the evidence can be rightly applied. This is, of course, the missing link at the moment in fingerprint identification.

Inherent Process Error
This does not include perceptual/cognitive errors. It may be a question of tolerances in measures or sensors. In fingerprint identification, this is pretty much limited to the possibility of undetectable artifact appearing in the impression or image being examined.

Perceptual/Cognitive Error
This may be a conclusive error in a poorly understood process such as eyewitness recognition of faces and objects. Or it may be interpretive error on the part of a human observer who has expertise in interpreting particular observed phenomena, a person who has the requisite knowledge, experience, and ability to interpret correctly but who may interpret in error. In fingerprint identification, this includes interpretative error on account of simple failure to observe correctly or on account of bias that affects interpretation.

Error In Implication of Realm
Not a very good label, I admit. By this, I mean the cloaking of clinical estimation with the appearance of scientific conclusion. Psychological exposition is often elevated to the realm of scientific conclusion. When clinical evidence is in play, it is almost inevitably contested on the basis of arguing the relative degrees of witnesses' experience and is presented as competing views that cannot be resolved. The view that prevails may do so on "common sense" agreement with other evidence, the apparent rigor of the practitioner's efforts, or simple rhetorical ability.


These issues are not easily disentangled in fingerprint identification. In cases in which considerable interpretation of less poorly represented detail is required, some degree of clinical judgment must be applied. No matter how mechanically an examiner may trace ridge structures, there may be appropriate difference of opinion on how the image of one or another structural detail should be interpreted. It becomes a matter of clinical argument for one or another interpretation and arguments that one or the other is a biased interpretation.

Strongly validated scientific forensics like DNA analysis lend themselves well to lessons on The Prosecutor's Fallacy and The Defendant's Fallacy. It is well to note that while DNA evidence is often powerful because it is often deposited through intimate contribution of body material or through bloodletting during the crime, as its sensitivity improves and more analysis is done, more detectable casual and innocent deposition may require more care. But I think we have to make a distinction between evidence that can be represented with validated probabilities and evidence that cannot. Results of both can be misrepresented, but one is misrepresented by erroneously misinterpreting a valid numerical result. The other is misrepresented by the more fundamental misrepresentation of a scientifically unvalidated result of analysis or an unspoken and unchallenged implication of valid conclusion.

The Sally Clark case often comes into discussions of The Prosecutor's Fallacy. Indeed, it is applicable, but it is a poor example because it is mixed with the fact that the expert conclusion was founded on a frankly erroneous theory. That theory, the fundamental assumption that led to the conclusion, was that the two deaths were independent. That assumption must have been made before Meadow calculated the probability of happenstance. It was not a case of a validated result of analysis being used to misrepresent a likelihood of guilt. It was a case in which the conclusion of the analysis was not only unvalidated but was frankly invalid. I am not entirely familiar with the state of statistical research on multiple SIDS deaths in families at the time (except that, while not fully developed, it was known that there was a greater incidence in some families), but I think the process of concluding the probability as a product of independent events would have been found impossible to validate.

Errors in perception or interpretation of observations should not be much a matter for argument over representations of probabilities. Surely, statistics must be properly applied if they are presented, but questions of observational and interpretive error must either be resolved by additional examination or must admit to two differing interpretations. The statistical presentation of the damning conclusion remains the same and remains correct for anyone who accepts that interpretation. That presentation will either be accepted or rejected by the finder of fact, based on the credibility of the differing experts. To presume that some overall "error rate," even if one could be found, should apply to any individual examination is to commit quite the same error committed when Meadows ignored the possible differences among families.

[mdavis]

I've been out of these threads for a while, but did think a response to Pat's post is in order regarding mathematical models to approximate probabilities of friction ridge individualization, and Gerald's comments on reliability. Everyone is trying to equate the comparison disciplines with DNA science. To do so with any degree of accuracy will, in my opinion (and what's that worth?...) continue to be a gross oversimplification of the not-so-obvious. Consider the mechanics of DNA analysis (with appropriate oversimplification).

Assuming a DNA sample is obtained legally and in an uncontaminated state, it is "blindly" chemically processed for extraction of the DNA molecule(s), and then introduced into an unbiased, highly mechanized "test tube" process which chemically "snips" a specific length at an absolute site from the molecule for analysis. It is then introduced into a machine that reads the various "markers" obtained from alleles, and charts the results. There is a minimal possibility of "Inherent Process Error." Interpretation of the data is a relatively simple statistical comparison of markers obtained from the sample with data obtained from hundreds of thousands of collected samples stored in a master database. There is no interpretation of what is a marker and what isn't a marker. If it's there, you use the individual marker's probability of existence in comparison to the database and obtain a total probability of a specific DNA sample's uniqueness by multiplying the probabilities (Frequency of Matches in the Population) of each individual marker together. There is essentially no "Perceptual/Cognitive Error" factor in looking at a computer print-out -- there is either a marker or there isn't, and you do the math.

To apply a similar unbiased probability analysis to any comparison discipline, one must overcome virtually every factor that relies on human interpretation. Even studies by statistical gurus are limited to Level 2 details. Assuming a legally obtained and "uncontaminated" fingerprint, we stumble at the very first step -- that of unbiased recognition of Level 2 details. Many of the rare but highly over-publicized examples of error in latent print identification are the result of the inability of examiners to recognize or agree on specific Level 2 details -- the "Perceptual/Cognitive Error." Until we can create unbiased computer recognition of Level 1, Level 2 and Level 3 details, there is no way we will approach statistical approximation of uniqueness.

Let's assume, for the sake of example, that someone is able to write such a program to bring us "up" to the level of DNA. The program would have to rely on pre-processing (Inherent Process Error). Was the latent print photographed "as is", was it processed with powder, cyanoacrylate, fluorescent dye, amido black? What color is the substrate and the matrix that determine the contrast? Are we allowed to enhance the image by altering camera or scanner settings or lighting? Can we enhance the image with a digital image editor prior to introduction into this unbiased computer analyzer? If we convert an image to grayscale, what is the 8-bit (256 level) of grayscale differential allowable to count a ridge detail matrix against its substrate? All of the above can be said to fall within or be affected by "Inherent Process Error" -- affecting the unbiased ability of the computer to accurately recognize the pattern, the ridge endings, and more detailed individual characteristics before the computer can eliminate the "Perceptual/Cognitive Error." Can the image be a reversal, or a forgery? What happens if the latent has a scar that has occurred since the control was obtained, a crease that appears on one but not the other due to pressure differential which results in a series of "ending ridges?" We can't get beyond the Inherent Process Error. Garbage in, garbage out.

Unless we can account for these common anomalies, any simplified computer algorithm that is limited to nothing but Level 2 details (assuming we can overcome to everyone's satisfaction the Inherent Process Error and Perceptual/Cognitive Error) will be only a token shot at throwing a bone to the court system. What we are seeing with the current "state of the art" computer algorithms, those being continually tweaked by competing AFIS vendors, is that often the highest scoring candidate from the list (depending on your depth setting) is not the actual identification. So we're a very long way from solving the "Perceptual/Cognitive Error" problem as well.

What I think I'm seeing is that we think we are being forced to provide the courts with some statistical data to match the DNA science. So we are cutting corners. We don't have adequate control of the Inherent Process Error to provide grist for the unbiased computer algorithm (the Perceptual/Cognitive Error eliminator), so we are working feverishly to create that algorithm and in the meantime, using what pathetic programs that we have. We are using relatively small, test databases to build up minutiae templates of Level 2 details in an effort to compare templates to determine the statistical frequency of (4, 5, 6, 7, 8, whatever) Level 2 details to determine a statistical probability that we can apply to comparisons of friction ridge captures in a database of relatively controlled origin (i.e. inked or scanned ten print cards as opposed to latent prints obtained as evidence).

And we must beware of Gerald's "Error in Implication of Realm." If you want me to begin using statistical probabilities in my reports, can I report that a latent print and a control share three Level 2 details in common with a (let's say) 15% statistical probability of individualization? And what if I include the probability that this pattern (an arch of 5% occurrence in the general population) renders my probability even better? If we ignore just Level 1, what does that do to the statistical probability of the Level 2 estimates? Does a 1% probability of Level 2 in a loop have the same probability of occurrence as a 1% probability of Level 2 in an arch? Should we set a threshold below which we simply must not report any results to insulate the court from having to determine uniqueness?

We are going from the remote possibility of "Perceptual/Cognitive" errors to potentially misunderstood but highly damaging "Errors of Implication of Realm" being transferred from experienced, trained examiners to naive juries for evaluation. This does not "validate" comparison disciplines, rather it plays into the hands of desperate attorneys on both sides looking for ways to minimize or maximize the impact of experienced expert witnesses on their side of the case. If a latent print comparison has a 1:1000 probability of individualization, what does that mean? Sounds pretty good, until you consider the millions of latent prints on file. So who determines the number for uniqueness? We are back where we started unless we give it to the courts "as is" and let them sort it out.

We have gotten into more trouble with this statistical modeling than we will ever get out of. We are headed down a narrow path and the vines are getting closer. What was wrong with staying on the pavement, using all three levels of detail and years of training and experience, and calling our results "expert opinion?" A newborn can quickly recognize with essentially 100% accuracy its own mother. The fact that a comparison discipline does not lend itself to "scientific validation" does not render it "invalid."

[Pat A. Wertheim]

Dear mdavis,

I don't disagree with a thing you say, but let's start with a simple question: Should there be research into statistical models to support fingerprint identification?

If you answer "No," the conversation does not need to go any further. Along with attempts to develop new, sophisticated computer models, you have rejected Galton, Locard, and dozens of others who have proposed various statistical models throughout the history of our science.

If you answer "yes," let's consider the beginnings of fingerprint identification as an acceptable science. The first statistical model was proposed by Galton, and it was his model published in 1892 that gave credibility to the idea of fingerprint identification as science and launched our careers, yours and mine and thousands of others through the past century. Models subsequent to Galton's all attempted to refine his, from Locard to the current day. Now that we have computers, might they not be capable of calculations beyond the pencil and paper available to Galton?

As I said, the computer model Cedric Neumann has been working on does not make the identification, and it is not perfect. It is not an AFIS algorithm, although it may have some similar features (I know very little of AFIS algorithms and even less of Cedric's). But it is a starting point. As I said, when it has been refined, tested, and validated, then it may be used to add statistical support to an identification that has already been made.

But I reject the idea that we should block any attempts to design a new model simply because fingerprints are complex and some decisions made in the comparison process are subjective. If fingerprint identification is science, then continued research to develop and refine the tools we use is mandatory. We need to encourage the development of these tools, not disparage them because of perceived problems in their development.

[mdavis]

Thanks for the reply, Pat. Sure, research is always welcome. However, after 100 years of attempts at statistical models, we have little to show for it, even today with massive computer power and modeling. That is not to "dis" Galton and Locard to whom we owe so much. Statisticians have been working on statistical modeling for decades as you know. We can presume to come up with something "statistical" if you allow for exclusion of much of the minute detail we all use in making comparisons, but creating a computer model that takes into account what we all "see" in a latent print in making decisions is not likely to happen in my lifetime. We cannot digitize a Monet painting, put it through a computer, and tell if it is a fake if the computer has nothing with which to compare that specific painting.

What we are doing is using a database of relatively "clean", controlled ten print cards, generating minutiae templates and then telling the computer to calculate the probabilities of uniqueness for each number of positional markers. So one marker (I studiously avoid the term "point") has a 0% chance of being unique, 2 markers, maybe 1% and so on (it would no doubt be a logarithmic scale). The probabilities between, say 8 markers and 9 markers would be huge, 10 even more so. So what do we do with that information and how accurately does an un-manned computer find those markers? Anyone working with AFIS will tell you that the input should always be "cleaned up" to eliminate false markers. Printrax used to require tracing (may still do so), but that relies on input operator decision making, so we can't use that. In addition, much of the Level 2 detail in tight areas (core, delta) is ignored by the computer. So we have questionable data gathering and incomplete "analysis" providing statistical numbers that mean .... what? We are back to point counting, and the international forensic community never could agree on that magic number. By sending out reports with statistical probabilities, we are asking someone else to determine the threshold. If we come up with a probability of 1x10 to the 16th power as "individualized" then what about 1x10-15? Is that inconclusive?

Consider the ident made on the thumbprint on the rifle scope that had no Level 2 details. Granted, I've been taken to task by the point counters who argue that incipient ridges are, in fact, Level 2 details -- an argument for another time. But zero Level 2 details is a 100% failure to individualize unless you broaden the definition (and what is the definition, by the way) of Level 2. Or we can take the hypothetical situation of a plain arch with no Level 2 details. So I take that example to court, introduce my report saying that there is a zero percent statistical probability of this being an ident, but "I think it is." What have we gained? Actually, we've lost because the statistical models will override the education, training and experience of an examiner. We will have lost the ability to use ET&E on any marginal latent print such as Mayfield, McKee, which is perhaps good? I don't know. What I do believe is that we have too many arrogant old timers and inadequately trained newbies who become overconfident, push the envelope just a tad too far, and fail to admit that "inconclusive" is a very valid and safe conclusion. In fact, it follows constitutional guarantees and court logic of "innocent until proven guilty." And it is these rare (and I do believe them to be quite rare) conclusions that have overridden the incredible reputation for accuracy that we have enjoyed for the past 100 years. The latent print community in which I started 20 years ago was very effectively self-policing, and we seldom if ever saw bad idents. Today we and the public and court system are paying a terrible price for those mistakes. And we are being asked to "fix something that ain't broke" by well-meaning people who are not experienced examiners.

We have always agreed that the basis of friction ridge uniqueness is based on the science of biology. What we have not been able to do is demonstrate the complexity of friction ridge skin in a statistical model that is capable of describing the myriad detail of an imprint. We would not hope to describe that Monet painting in a paragraph or a page, which is what we would be doing, in effect, if we dumb down our statistical modeling of fingerprints to Level 2. Does a statistical model of only Level 2 detail validate the "science?" I'm not saying we shouldn't let the statisticians and computer hackers work on it, but I think we are in danger of either accepting a premature result, or hoping for a miracle. Heck, if the courts think so, full speed ahead! I don't think it flies.