The British Medical Journal Embarrasses Itself by Reproducing That Notorious 1 in 58 Figure

The deluge of bad reporting flowing from St. Maximilian Kolbe and Blessed Titus Brandsma Day continues. The mojo of this day is so powerful that it has even influenced the British Medical Journal for the worse: GMC hearing against Wakefield and colleagues opens.

After listing some of the charges against Dr Andrew Wakefield, Profs Walker-Smith and Murch but then reporting that the full list of charges won't be announced until today, when the hearing opens, the BMJ concludes this item with a blaze of shame.

Researchers from Cambridge University's autism research centre will conclude in an as yet unpublished study that autistic spectrum disorders are almost twice as common among British schoolchildren as current estimates indicate. The lead researcher, Simon Baron-Cohen, said that this study, which examined some 12 000 primary school children in Cambridgeshire, will conclude that one in 58 children has such a disorder.

No, no, no. Prof. Baron-Cohen has written a letter to The Observer (amongst others) to correct that canard about the 1 in 58 figure.

[Your] article linked MMR and autism.

The research does not...

The best estimate of the prevalence of autism is the 1 per cent figure published in the Lancet in 2006.

My view is that any apparent rise is likely to be driven by better recognition, greater awareness, growth in services, a widening of the definition of autism and a shift towards viewing it as a spectrum rather than a categorical condition.

Once more and with feeling, the researchers used several screening tools on that population. The 1 in 58 figure is the result of the Childhood Asperger Syndrome Test (CAST). When used with a general population such as a mainstream school, the CAST generates 50% false positives but it has a reasonable specificity (the negatives are true negatives) so it might seem reasonable to evaluate its potential usefulness as a comparatively low-cost first pass screening tool for epidemiology research. All of this was quite obvious to anyone familiar with screening who just read the newspaper articles: however, it was up to the journalists to explain this to the general reading public.

If the journalists and BMJ had had any further questions, they might have actually talked with one of the professors at the Cambridge University Autism Research Centre (ARC). They might even have consulted the ARC website and looked at the current research projects where they would have read about: Screening for Asperger Syndrome in primary and secondary schools. Despite the frequent accusations that there is no research into autistic spectrum disorders (ASD), this project is exploring ways of lowering the age at which diagnosis is practical. The researchers are aware that early intervention will help those children who are affected by ASD and they hope that effective screening would "reassure the worried well" (pg 2 of following report).

Currently, AS is picked up far too late. It should be possible to identify AS in children in primary school (age 5-11). We have developed a screening instrument for this purpose, called the CAST (Childhood Asperger Screening Test) and are testing it at a population level.

The team links to some relevant reports, including The CAST (Childhood Asperger Syndrome Test): test accuracy (pdf).

If the journalists or BMJ had done any of that then they would have learned that the researchers do not support that 1 in 58 figure and report it as an artifact of using the CAST. They conclude that although CAST may be useful in epidemiology research, there is not enough evidence to support its use as a general screening tool.

The Childhood Asperger Syndrome Test (CAST) is a parental questionnaire to screen for autism spectrum conditions. In this validation study, the CAST was distributed to 1925 children aged 5-11 in mainstream Cambridgeshire schools. A sample of participants received a full diagnostic assessment, conducted blind to screen status. The sensitivity of the CAST, as a designated cut-point of 15, was 100 percent, the specificity was 97 percent and the positive predictive value was 50 percent, using the group's consensus diagnosis as the gold standard. The accuracy indices varied with the case definition used...The CAST is useful as a screening test for autism spectrum conditions in epidemiological research. There is not currently enough evidence to recommend the use of the CAST as a screening test within a public health screening programme in the general population. [My emphases.]

More relevant quotations from the report are as follows but feel free to page-down through this if you feel that the researchers have already set out their stall.

Currently there is insufficient evidence to recommend screen for autism spectrum conditions as a public health service (National Screening Committee Child Health Subgroup, 2001). One of the gaps in the evidence is the lack of a screening test that has been fully validated and shown to be effective in the general population...

An effective screening test for autism spectrum conditions would also be invaluable for epidemiological research. Due to the resource implications it would not be possible to undertake a detailed assessment of all children in a large population-based study. A screening test can be used in the first phase of an epidemiological survey to sift out the children who require further detailed assessment in a second phase of the study, and hence make large studies feasible.
[pg 2 of 24 or 46.]

Many promising screening tests are being developed, but there is currently no screening test for autism spectrum conditions which has been fully validated in the general population, which has been shown to be effective, and for which information about validation is available in the public domain. The aim in further developing the Childhood Asperger Syndrome Test (CAST) was to validate a test for use in the general population rather than clinical populations, and to develop a test that is sensitive to autism spectrum conditions, including pervasive developmental disorder not other specified (PDD-NOS), not just to typical autism.
[pg 3 of 24 or 47. It is worth noting that the authors acknowledge that the name of the test is a little mis-leading, it has been retained for continuity purposes. They emphasise that CAST is for ASD.]

The consensus diagnosis captured children with wider spectrum conditions. When using the consensus diagnosis, a cut-point of 15 appeared to be appropriate where sensitivity (100 percent; 95 percent CI 74-100 percent) and specificity (97 percent; 95 percent CI 93-99 percent) were high. At higher cut-points, the sensitivity dropped. The positive predictive value was low at a cut-point of 15, at 50 percent (95 percent CI 28-72 percent). Using the assesment diagnosis, a higher cut-point may be more appropriate, such as 18 where sensitivity was 100 percent (95 percent CI 63-100 percent) and specificity 99 percent (95 percent CI 96-100 percent).
[pg 13-15 of 24 or 57-59 (intervening pp of tables).]

The drawback of the CAST is the low positive predictive value, which is a function of low prevalence of the condition in the general population (O'Toole, 2000). There are major resource implications of assessing a large number of children who are false positives. There could be much anxiety associated with false positive screen results, as has been demonstrated with other screening tests (Marshall, 1996). A positive screen results brings uncertainty about health status (Marteau, 1994), in the case regarding the presence of a developmental disorder, until further assessment is under-taken. It should be noted, however, that a child who is a false positive for an autism spectrum condition may have another developmental problem which may be clinically important...
[pg 16 of 24 or 60.]

There is no absolute gold standard test for a developmental disorder...
[pg 18 of 24 or 62.]

If there had been complete response to the screening test, proportionally more respondents with low scores would have been expected and lower prevalence estimates, compariable to those in other studies (e.g. Scott et al., 2002b), could be expected.
[Researchers discuss the type of family that was most likely to have filled out and returned the questionnaire. They missed some known children with ASD because the parents of those with confirmed diagnoses were less likely to participate in the study: to that extent, the prevalence rates might have been higher. However, the rate of participation in the study means that in general, there should probably have been a much greater proportion of low scores, so the prevalence rates were more in line with those reported elsewhere.]

The CAST is demonstrating good sensitivity and specificity but low positive predictive value...As the aim is to develop a screen for the general population, however, a more pragmatic method of increasing the positive predictive value [is needed]. It might be possible to introduce an additional phase prior to using the CAST, such as asking if the parent has concerns over the child's development. The CAST could then be used in a higher-risk population, and the positive predictive value may be considerably increased (O'Toole, 2000).

The CAST can be recommended as a screening test for autism spectrum conditions in epidemiological studies, as the low positive predictive value and subsequent false positives are unlikely to cause anxiety because a range of children from low to high scorers would be invited for further assessment...It is not appropriate, however, to recommend the use of the CAST as a general population screening test in a public health or educational setting...
[pg 19 of 24 or 63.]
[My emphases.]

[Edited: July 18] Simongates commented:

I think I read somewhere that the sample size was actually 116 i.e. there were 2 autistic spectrum children. If this is right the 95% confidence interval around the estimate of 1.7% would be 0.5% to 6.0%, which includes pretty much every value that has ever been claimed. So claiming an autism epidemic (let alone one caused by MMR) from these sorts of figures would be misleading even if the numbers were correct.

As a general point, in the absence of open access journals we rely upon journalists of various media to read and understand studies that are of general interest and to report upon them accurately. However, as Goldacre expresses it, although

newspapers like to fantasise that they are mediators between specialist tricky knowledge and the wider public...I wouldn’t be so flattering. In fact, if you have access to the original journals, you can see just how rubbish things can get.

In this case, the actual report is freely available but the journalists concerned, and those who were subsequently involved in supporting their actions, do not seem to have read it. It's almost as if the source of this figure is of no relevance to the heat and light that can be generated by the mis-reporting.

It was bad enough that this canard of the 1 in 58 prevalence spread like wildfire in the newspapers; it is a new nadir that the BMJ reported it so uncritically.

More when the feeling of sadness goes away.

Update: 15:00 Dr Fiona Scott was one of the implied sources for the MMR-autism connection in the original story that broke the 1 in 58 prevalence figure. A Dr Scott (identity unconfirmed but claimed) has posted this response to The Observer Readers' Editor response. Comment July 16, 2007 11:31 AM.

I feel, given that I was one of the two 'leaders in the field' (flattering, but rather an exaggeration) reported as linking MMR to the rise in autism, that I should quite clearly and firmly point out that I was never contacted by and had no communication whatsoever with the reporter who wrote the infamous Observer article. It is somewhat amazing that my 'private beliefs' can be presented without actually asking me what they are. What appeared in the article was a flagrant misrepresentation of my opinions - unsurprising given that they were published without my being spoken to.

It is outrageous that the article states that I link rising prevalence figures to use of the MMR. I have never held this opinion. I do not think the MMR jab 'might be partly to blame'. As for it being a factor in 'a small number of children', had the journalist checked with me it would have been clear that my view is in line with Vivienne Parry of the JCVI. The 'small number' was misrepresented by being linked inappropriately and inaccurately with 'rise in prevalence', leading readers to arguably infer that it is in fact NOT a small number!

I wholeheartedly agree with Prof Baron-Cohen, and many of the posts and responses received to date, that the article was irresponsible and misleading. Furthermore I reiterate that it was inappropriate in including views and comments attributed to me and presented as if I had input into the article when I had not (and still have not)ever been contacted by the journalist in question. I am taking the matter under advisement.

How many more times will Dr Scott and Prof Baron-Cohen have to issue clarifications?

Update: Denis Campbell's article, New health fears over big surge in autism has been removed from the Observer archive so I have linked to a new URL.