Earlier today (19th February 2018) the University of Warwick put out a press release about what one of the authors, Paul Thornalley, described as a ‘small biomarker discovery study’ in autism. The paper has the, perhaps, not very sexy title “Advanced glycation endproducts, dityrosine, and arginine transporter dysfunction in autism—a source of biomarkers for clinical diagnosis”. It was a study using complex proteomics techniques of blood, and urine samples taken from 38 Italian children, aged between 5 and 12, all severely affected by autism, and only 9 recorded as having a normal IQ, and 31 age and sex matched community controls. The press release made three points, that the test believed to be the first of its kind (which is not true); that a link found between autism and damage to proteins in blood plasma (which is a stretch, but might be right); and that this could lead to earlier diagnosis of the condition (which is not even considered in the paper). It finished with the lines ‘The outcome was a diagnostic test better than any method currently available. The next steps are to repeat the study with further groups of children to confirm the good diagnostic performance and to assess if the test can identify ASD at very early stages and assess if treatments are working.’.
The media went crazy. As I write, at 10pm that day, the paper has been on all the main radio news programs in Ireland, and is prominently covered by the Irish Times, The Guardian, The Telegraph, the BBC website, the Nursing Times, and many many more. Most of these sites, with a few honourable exceptions, including the Irish Times, and the Huffington Post, repeat the press release, and add a few lines about how wonderful it all is.
So, what went wrong? First, of all, I do not criticize the researchers. I spoke with one of them, Peter Thornalley, on the Matt Cooper show earlier this evening. He and his colleagues did a fairly typical biomarker discovery study, looking at a series of markers, in a very small group of children. I am about as competent to judge their chemistry as my cat, but they have a good reputation, and all the right words, apparently in the right order. They looked for particular proteins and protein fragments in the blood and urine of children with ASD, and control children. There are many tens of thousands of proteins and fragments, any of which could have been examined. They used reasonable methods of data analysis too, which is less common. They make one serious mistake, which is that they do not seem to realize that the study is too small, and the result, most likely, is a false-positive. However, Peter Thornalley said that he saw this as a preliminary study, which might form the basis for further applications for funding.
Warwick are mostly to blame, for a very overwritten press release, but reporters must take some responsibility too. After all, the goal of a press release is to get the client talked about, and this one has certainly hit the spot. So, what was wrong, and what could a journalist have spotted, without a detailed technical knowledge of this area?
Matt Cooper, the Irish radio journalist who interviewed me, and Paul Thornalley, this evening, put his finger straight on two key points. First of all, this is a small study, and small studies have a well known bias – in a study like this you will look at many markers, and report the ones which show a positive result. This is normal. The problem is that in a small study, and especially in a small high-throughput study like this one, it is overwhelmingly likely that the interesting result, that the researchers dutifully write up, is a false-positive result. This means, that although this experiment gave a positive result, future experiments will not confirm these findings. This is a well known and much discussed problem in science.
The second point that Matt spotted was that while the press release talked about early diagnosis of ASD, most kids with ASD are diagnosed before the age of 5, that is before the age of the youngest child in the study. This means that the study only recruited kids with a well-established diagnosis of ASD, and only measured them well after the time you might expect a diagnosis to be made. In other words, it has nothing to do with early diagnosis of ASD.
Another issue is that to be useful a test like this has to be better than existing tests. Showing this is really hard to do. As Max Davie points out in the Huffington Post, such a test would have to either predict future autism at an early age, or improve upon existing clinical methods of assessment. Doing this would require big studies, lasting several years, and costing a million euro or more. It’s really hard to see a place for such a test in ASD.
What is the value of this work? Very simply, it points to a possible difference between the biochemistry of people affected by ASD, and those not affected. Such work is a necessary part of exploring the biology of ASD. At the moment, very little is known about any such differences. This paper is of interest, but to other researchers, and not to the general public. No-one has been served by the massive publicity given to this work. Parents will be confused, clinicians will be quizzed about this work, and no child with ASD will benefit. Editors and reporters can, and should, do better.