‘Difference and the non-numeric system’

[printlady]

DNA has its own issues...

http://missionlocal.org/2009/10/dna-pro ... y-be-next/

The 236,970 profile submissions in the first nine months of 2009 already exceeds the roughly 200,000 from last year.

And amongst these samples, legal experts argued, there are errors.

With so many samples it is inevitable that there will be mistakes, said Risher, citing a cross-contamination at a state-run Sacramento lab where DNA discovered on a cigarette matched the profile of a sexual assault victim from another case. Cross-contamination occurred when the sample from the cigarette was processed close to the victim’s vaginal sample.

“Everybody thinks DNA is this gold standard, but it’s still just as susceptible to human error,” said Ortega. “Inevitably there are humans interacting with this sensitive material. That allows room for error.”

Another possible source of error is actually the improvement in crime lab technology.

“When you start talking about crimes in a public place, combined with increasing sensitivity of DNA equipment, it raises the possibility that you can find DNA at a scene for someone that had nothing to do with the crime,” said Risher.

Errors do occur, said Michael Rushford, president of the Criminal Justice Legal Foundation. But the positives far outweigh the negatives.

“The pressure needs to be on doing this right and doing this well,” he said.


The statistics that are provided by DNA analysis will tell you the likelihood of that particular profile appearing in the population; but it won't tell you if the sample was contaminated or if the technician made an error. We are reliant on the QA/QC measures a facility employs and the training/experience of those conducting the tests. Any endeavor involving human beings is susceptible to error.

[Gerald Clough]

Speaking as a lay person, it is becoming increasingly apparent to me that fingerprint identification is not nearly as straightforward and "infallible" as I previously understood. If you had asked me a year ago (even in the knowledge of the accepted misidentifications in the McKie and Asbury cases but before hearing evidence at this Inquiry) which form of identification I "trusted" more, fingerprints or DNA, I would have said fingerprints beyond question. Yet DNA identification has a proper statistical basis for its conclusions (which fingerprint "science" doesn't have) and DNA identification is devoid of any element of subjectivity on the part of the examiner (which fingerprint identification clearly has). So much relies, in fingerprints, on the competency and integrity of the individual expert. It would be difficult to imagine a DNA practitioner setting out, on his or her own, to mislead without involving investigators, scenes of crime officers etc. How is it that fingerprinting acquired the myth of infallibility, which has never attached to DNA identification? Could it be to do with history, that DNA identification was born in a much more sceptical age?

There's a lot going on in your post. First, you are correct in now recognizing the infallibility is not to be found, and your previous assumption agreed with what most lay people believe. The implications of that and the reasons for it can be found everywhere you look in the ongoing arguments and discussions.

Before going on, I want you to clarify for yourself exactly what you mean by "subjective." I think you mean what many people mean but, like them, you haven't thought it through, because we are so used to using "objective" and "subjective" in very imprecise. The two terms do not describe two states differentiated by whether or not judgment was required, rather than some mechanical measurement. The difference between objective and subjective is that, contrary to an objective process, a subjective process allows something other than the object of the process to influence the observer by altering his perception. In fingerprint terms, an objective analysis looks only at the impressions. It would be subjective only if the analysis introduced other data that recreated the analysis as an analysis of a more complex construct of his own mind, rather than analysis of the object, the impressions. But the fact that the analyst makes judgments and assigns values does not make it subjective. Fingerprint examination, done properly, is never subjective.

I can show you publicly revealed deliberate falsification of DNA analysis by the analyst. And it was not done by altering the sample. It was by misrepresenting the statistical data to suggest a false probability of the named individual being the source. There is no fundamental difference between that false representation and a deliberate misstatement that an individual is all but certain to have been the source of a fingerprint. Both misrepresent the meaning of the data. It makes no difference that the process of creating DNA data is mechanical and fingerprint data is derived by expert interpretation. It was simply the nature of DNA analysis that it can be interpreted as probabilities. We cannot misrepresent the absolute probabilistics of fingerprint analysis, because we simply don't know the probabilities. In other words, we could lie, but we couldn't know big a lie we were telling. Imagine now. Had we somehow not been able to examine or logically derive the DNA numbers in the population, we could still, by accumulating experience in the same way we accumulated experience with fingerprints, have rendered opinions from observations of DNA. Those opinions would be that the analyst had formed an opinion concluding that an individual was the source. (Remember - you couldn't retort that X number of others also match, because you wouldn't have the information to substantiate your objection.) In that hypothetical circumstance, we would be having exactly these same discussions about DNA.

But we're not talking about deliberate error. (I know. Simple minds see only conspiracy or simple error and nothing in between, because it suits them, but things are almost never that simple.) Let's talk about the simple possibility that a person associated as the source of some evidence is not the source. You now feel pretty good about DNA. But statistically, there is a known probability that someone else may be the source. (And if you ask the fingerprint examiner if there is not an extremely small chance that someone else could have made the impression, he must answer that there is that very small possibility. If he does not admit that, he is a logical fraud. Any portion of skin exhibits a finite number of features, and the number of possible variations is therefor also finite, so inescapably, there is that possibility.) By the gaming truth that dice have no memory, the fact that the probability is extremely low does not mean the guy next door to the suspect isn't the true source of the sample. Probability does not say you will fail to find another match until you examine that extremely large number of people. In a very real way, it only implies how surprised you will be if you find the other match next door. We know this intuitively. It's why we're willing to gamble, and it's also why we're pleasantly surprised when we win.

When the fingerprint analyst declares his belief in the identity of the source of an impression, he is essentially expressing how extremely surprised he would be to find that someone else was the true source. And he would be very surprised, indeed. The DNA analyst is expressing a similar anticipation of extreme surprise to find another match in the population from which a reasonable suspect could be drawn. It's not a greater potential surprise, just because the DNA analyst can put a number on it. And it's not an entirely precise number, since the DNA analyst cannot know the actual distribution of characteristics in that population limited to only those who could have contributed the sample. It's not possible for them to know. Arguably, the experienced fingerprint examiner may have a more reliable feel for the nature of the suspect community.

The bottom line. DNA is, by its frank probabilistic conclusions, fallible. So any imagined infallibility is a myth. But because people do not understand probability, they take it as practically infallible. Fingerprint identification has never been taken as less fallible than that. It was simply accepted as extremely unlikely to be mistaken, just as DNA is taken. Since attempts to present the practical application of probability in court has never played well, they both get a pass. I suspect if you were watching an intense discussion about DNA analysis and its application, you might well be thinking that DNA isn't quite what you thought it was, just as you're now feeling about fingerprints. I do not think it has anything to do with being in a more sceptical age. I think it's more that challenges to many forensic conclusions have evolved to include academics and that the law has evolved to allow more involvement of science. But you should note that courts in general do not share your feeling that fingerprint conclusions as suspect, because their perspective is rather different from that of the academic scientist, and the courts understand the value of forensic expertise, including expertise partially or wholly unsupported by valid science.

It is indeed often said that much of fingerprint examination's power is the result of uncritical history. I do not agree. I believe the value was recognized early and is still recognized as just as powerful and that that recognition is as valid today as it was 100 years ago. I enjoy being in good company there, since almost all courts agree with me.

[sharon cook]

[David Fairhurst]

Upon re-reading my previous post, I thought that the last step coud be expanded somewhat, and also that I could simplify the whole process to bullet points. So here goes.

1.Sufficient detail in agreement to individualise and then I find a difference.... Consult analysis.

a, The difference was red-flagged..... Report individualisation.
b, The difference was not red-flagged.... reanalyse.

2. Reanalyse the mark.... is the cause of the difference evident upon objective and honest reanalysis?

a, Yes, the distortion can be red-flagged..... Report individualisation.
b, No, there is no evidence of distortion..... recompare.

3. Recompare the prints.... are the features previously thought to be, truely in agreement?

a, No... report exclusion.
b, Yes.... re-evaluate.

4. Re-evaluate the detail in agreement... do I sill beleive the observed detail in agreement is sufficient to individualise?

a, No.... report exclusion.
b, Yes.... RETRAIN and.... report exclusion.

I hope this emphasises the importance of differences in the examination process and allays fears of the "lucky bag" dipping that Iain brought up.

LPEs... Feedback would be welcomed on this process's validity.

[ER]

LPEs... Feedback would be welcomed on this process's validity.

I would think that at some point (depending on the 2 prints being compared, of course) the result of inconclusive would have to factor in here somewhere.

If the prints are so close that it has come down to one point being out of place because of possible distortion, reporting an inconclusive in some cases may be the safer route than saying that you're absolutely certain that this person did NOT make this print.

[Ernie Hamm]

An erroneous identification is a miscarriage of justice.

An erroneous elimination is a miscarriage to justice.

Avoid making either.

[Atonement]

Gerald

Thank you for your very interesting post which has given me much to think about. I will be re-reading several times and studying it.

I should perhaps explain that the reference to misidentifications of Y7 and Q12 in my last post was based on what I understand to be the official Scottish Government position, although not of course accepted by everyone. Hopefully we will get an answer which everyone WILL accept when the fingerprint inquiry reports.

I still do have considerable trust in the science (is that the right word?) of fingerprint identification, but not perhaps the blind and unquestioning faith which I previously had. Surely a good thing. And of course I am aware that DNA evidence is not infallible. It needs careful watching at all times.

[sharon cook]

[David Fairhurst]

ER,

Thanks for bringing that up. I'll try to give my perspective on the issue.

Inconclusive results can only result from:
a) existence of features in agreement that are not sufficient to individualise
AND
b) absence of differences that cannot be accounted for.

So inconclusive results cannot come out of this process because my starting point is one opposite to both these conditions.

I am of the opinion that the so-called "one difference rule" (and I don't particularly like that term) dictates that in the presence of one unaccounted difference you have to exclude.

Yes, there will be times when you have some detail in agreement, but not enough, and then some differences that you are unsure about; they're just not strong enough to exclude on - by all means report inconclusive on those, just as I would.

The overall point is that you cannot have sufficient detail to individualise and unaccounted differences in the same print, it just isn't possible. If you think you have.... you need to retrain.

[ER]

David,

I'm not sure if I completely agree with your definition of inconclusive. Of course, prints meeting both of those criteria would fall into the inconclusive result, but I feel that it is a bit too narrow of a definition for inconclusive.

The FBI's recent study included three reasons to give a conclusion of inconclusive:
1. There is no overlapping area (basically, the known prints are incomplete)
OR
2. Potentially corresponding areas are present but contain insufficient corresponding or contradicory data.
OR
3. Corresponding areas are present with no contradictory features but are insufficient for identification.

Some other situations that may result in an inconclusive:
- pattern looks good, but there are no clear points at all
- there are a few points that match and a few that don't, but the ones that don't are red-flagged
- you know it's friction ridge skin, but you have conclusive clues about where it came from, and you just can't find it

That last one may show the need for more training, but there are some prints that you just can't find. Inconclusive seems to be the best result to report.

[kevin]

It would be difficult to imagine a DNA practitioner setting out, on his or her own, to mislead without involving investigators, scenes of crime officers etc. How is it that fingerprinting acquired the myth of infallibility, which has never attached to DNA identification?

I think it is simple - most people just do not have a working understanding of DNA and laboratory practices to question the techniques and practices (heck most of us don't either). It would actually be alot easier to mislead in a DNA case in my opinion-be it for the prosecution or defense. Go swab one of the suspect's personal items, or swab some unknown item and get a mixed sample for the jury to ho and hum over in deliberations.

The general public just does not have the base of knowledge to question DNA practitioners on many of their methods like they can in fingerprints-it is a visual exam they can see with their own eyes right there in the courtroom. Freelancing (or even flat out lying) of the type we see out of SCRO or any other organization with a bad practioner can occur in DNA just as it does in fingerprints, trace, firearm and toolmark identification, footwear examinations, and crime scene investigation. With DNA you simply have a statistical model for populations to quote along with your findings-I just don't see the difference besides that. It is still subject to interpretation as to the how and why. Transfer of DNA samples (cross-contamination) can and does happen and I personally would not concede much of anything to DNA as a superior form of identification. It is simply a piece of the puzzle and should be used in conjunction with other disciplines-statistical model or not.

I personally think that if you came up with a statistical model for fingerprints you would find that (taking into account intervening ridge counts, position and type of minutiae along the lines of what we see with AFIS scoring) the minimum number of points needed to actually indentify would be so low it would probably make the statistical model moot....you are still going to need an examiner to verify what the computer model is telling you at the end of the day.

[David Fairhurst]

David,

1. There is no overlapping area (basically, the known prints are incomplete)
OR
2. Potentially corresponding areas are present but contain insufficient corresponding or contradicory data.
OR
3. Corresponding areas are present with no contradictory features but are insufficient for identification.

1 and 2 both deal with incomplete exemplars and I wholeheartedly agree with inconclusive results, qualified with a statement that with full exemplars a conclusive result may be possible.

3 is my scenario above.

- pattern looks good, but there are no clear points at all

I would say that this is also an insufficiency of detail to individualise. Only this time it's the quality that's deficient not the quantity.

- there are a few points that match and a few that don't, but the ones that don't are red-flagged

I totally agree with this one, you do not have sufficient detail in agreement to individualise and the differences you have are accounted for by red flags.

you know it's friction ridge skin, but you have conclusive clues about where it came from, and you just can't find it

I'm not quite sure what you mean by this one. If you're sure which part of the hand (or foot) made the mark and you have that bit to compare it with on the exemplars, why can you not be conclusive with an ident or exclusion?
If you're not sure about the source area you have to assess whether your exemplars are complete and report accordingly.

[L.J.Steele]

[quote="Gerald Clough]I do not think it has anything to do with being in a more sceptical age. I think it's more that challenges to many forensic conclusions have evolved to include academics and that the law has evolved to allow more involvement of science. But you should note that courts in general do not share your feeling that fingerprint conclusions as suspect, because their perspective is rather different from that of the academic scientist, and the courts understand the value of forensic expertise, including expertise partially or wholly unsupported by valid science.
It is indeed often said that much of fingerprint examination's power is the result of uncritical history. I do not agree. I believe the value was recognized early and is still recognized as just as powerful and that that recognition is as valid today as it was 100 years ago. I enjoy being in good company there, since almost all courts agree with me.[/quote]

I am deeply concerned with confusing acceptance by the courts with validation of any field. The National Academy of Science report concluded that the courts have been utterly ineffective at dealing with most forensic evidence, and they may be right. The vast majority of judges and attorneys are not trained in science or statisitics -- there's a huge learning curve to present, or evaluate, a well-done challenge to forensic evidence. It is a vastly different audience than talking to either other examiners or other academic scientists.

Yes, the courts see utility in fingerprints, but accepted bullet lead comparisions, and the old arson science, and lots of other things that they probably should not have. That's not to say that there is utility in fingerprints, or many other areas, but to recognize that the courts should not be the arbiters of what is, and isn't, good science.

[ER]

I'm not quite sure what you mean by this one.

Sorry about that. It should have read:
-you know it's friction ridge skin, but you DON'T have conclusive clues about where it came from, and you just can't find it

Funny how much details matter sometimes.

In any case, it looks like we're mostly on the same page.

[Gerald Clough]

I don't think it can ever be the courts' role to decide what is and isn't good science. The appropriate role can only be to decide what has value as evidence. I think what is true is that courts often imagine that they are making judgments on scientific validity but do not always get as complete a presentation as they could. There are a number of areas of expert evidence that have some characteristic of science, some characteristics being use of statistical analysis, science-like institutions, doctoral level qualifications, etc. But they are not sciences and do not, within their communities, pretend to be sciences. (Well, they do take on the mantle of science by implication, but their professional bodies don't pretend.) Courts find them valuable, both as providers of information and because they may satisfy required legal definitions. And juries give those opinions great weight. An expert opinion does not lack value merely on account of lack of the conclusions being validated to a scientific standard.

The NAS is closely focused on applications of science. They choose to ignore fields that they rightly do not accept as sciences. They are not really interested in judicial process. There is no outcry in the scientific community (or the legal community, for that matter) against the heavy use of the expert opinions of psychologists. There is no outcry against the testimony of physicians, which is generally almost wholly from clinical experience and in no way scientific. Now, the scientists have a legitimate complaint when something is presented as valid science when it has no sufficient scientific validity. I will indeed fault the courts for entertaining some attacks on expert evidence on the basis of lack of full scientific validity, but even then, science has something to say, and maybe it's well that the courts hear from science. But I also recognize that the courts work from what the advocates bring to the courtroom. The pretended position that evidence should be excluded because it doesn't pass a test of having been fully validated to a scientific standard forces the issue into an argument over the science, when the test can only ever be whether it is useful to a trier of fact. In this, the courts have generally held the line in favor of what's useful. How much courts should meddle in the structure of arguments placed before them is a difficult question. But I can only answer it by viewing the courts are the disciplined avatar of society, a distillation of the better nature of the people, not as an institution in thrall to scientists.

Specific to fingerprints, scientists object to the supposed implication that an opinion that an analysis convinces the expert of some fact. They object, because, as scientists, they cannot draw a factual conclusion of that kind from the object of the analysis. They would disallow the opinion. And that is wrong. It is as wrong to let scientists arbitrate what a court may hear as it is for a court to pass on scientific validity. Scientists should definitely educate on the nature of science and the distinction between valid (and always tentative) scientific knowledge and clinical knowledge. They should also point up where genuine scientific research may well make possible more reliable expert conclusions. That is their essential role, to contribute knowledge to human affairs. In that role, they have something to say to society in its judicial affairs, but they cannot BE the court. I think courts also recognize that an exhaustive recitation of just why something isn't scientifically valid becomes an academic exercise and inappropriately damns offered evidence. They who guard the legal standard of what has value as evidence must also guard that evidence and preserve its value. Sometimes, that means not allowing a scientific authority to make it less valuable than it should be.

And science has not been squeaky clean in the legal arena. Scientists have lent their expertise to attempts to deprive courts of valuable evidence by pretending that it can have no value unless they have passed on it as valid. That is in no way a rational and balanced point of view. They have agreed to a narrow view to suit the advocates employing them. Perhaps they truly believe. That doesn't matter. They are subject to the same tendencies of other people when they take what I might call a hard science-centric view. Their view that the courts have been utterly ineffective in applying forensic evidence is entirely science-centric. The courts have been, if not always perfect in working with forensic evidence, appropriately balanced in how they apply it. We cannot look to past applications of forensics that have since been shown to be unreliable and claim it as evidence that courts can't handle forensics, any more than we can look to past scientific error and claim scientists can't handle science. Arguably, past scientific errors have been more fundamental and have more profoundly perverted truth than anything in forensics. If forensic evidence is beyond the capacity of judges, science is beyond the capacity of scientists.