 |
| A
critical appraisal of the
Wakefield et al paper The term
‘critical
appraisal’ means a
systematic analysis of a
study that has been
published in a scientific
journal (or a study
submitted to a scientific
journal whose authors
seek to publish it),
using a checklist of key
questions. When a
scientist submits a paper
to the Lancet, it is sent
out to two or more
reviewers with expertise
in the field, who are
required to send back a
formal report based on a
critical appraisal
checklist.
|
|
1. Is the
topic area important and relevant to
the Lancet’s readership?
Yes. In 1998, MMR
vaccine had recently been introduced
in the UK. Autism was rising in
incidence. Questions were being asked
about a possible link. A paper
describing a study that explored this
link would certainly have been
appropriate to the Lancet and highly
relevant to a general medical
readership, so long as it was
scientifically robust.
2. Was the study
original?
Yes. At the time, no
previous study had explored in this
way the link between MMR vaccine,
bowel problems and autism in
children.
3. Was the
research hypothesis clearly stated?
No. The paper does not
state a research hypothesis at all.
Implicitly, the research hypothesis
might be stated, “The
administration of MMR vaccine to
infants increases their risk of
developing (a) a particular pattern
of inflammatory damage in the
gastro-intestinal tract and (b)
autism or an autism-like
syndrome.”
4. What was the
study design?
The study design was a
descriptive report on 12 children who
had been referred to a paediatric
gastroenterology clinic with both
bowel symptoms (diarrhoea, abdominal
pain, bloating, and food intolerance)
and pervasive developmental
disorder characterised by loss of
skills that had been previously
acquired. Various blood tests,
gastrointestinal biopsies, and a
sample of cerebrospinal fluid were
taken from the children. The samples
were examined to explore the extent
of inflammatory reaction in the bowel
and to exclude other diseases (such
as thyroid disease, inherited
metabolic syndromes and so on). Of
dozens of tests done on each child, a
number were abnormal, though no test
was consistently abnormal in all the
children. Eleven of the 12 children
had microscopic evidence of
inflammatory reaction in their bowel.
The parents were asked to remember
back and identify if and when MMR
vaccine was given. In 8 of the 12
children, the onset of developmental
delay was said to have occurred
within 2 weeks of having the MMR
vaccine, and in 3 it was said to have
occurred within 48 hours.
5. Was this design
an appropriate way to test the
research hypothesis?
No. If the hypothesis
was that there is a causal link
between MMR and autism-bowel
syndrome, this study design was
incapable of proving that link one
way or the other. There are six main
reasons for this:
The sample was
highly selected – that is,
the authors deliberately picked
out the tiny number of children
who had been referred to a major
specialist centre because they
had both bowel symptoms and
an autism-like syndrome. So the
fact that these rare conditions
occurred together proves nothing
at all. The fact that children
with diarrhoea or other chronic
gastro-intestinal symptoms have
microscopic evidence of inflamed
bowels is also, in itself,
unsurprising.
The sample was
extremely small. I would expect a
scientific study claiming a
causal association between two
events (in this case, giving MMR
vaccine and developing
autism-bowel syndrome) to have a
formal statistical calculation of
the number of individuals that ought
to be looked at. This is
known as a power calculation. The
reason why the Lancet does not
normally publish studies on just
12 individuals (the usual number
of research participants is
several hundred, and not
uncommonly, several thousand) is
that the smaller the study, the
more likely it is that an apparent
causal link will turn out to
be due to chance association.
The study had no
control group. When studying the
possible harmful impact of a
vaccine or environmental agent,
it is standard scientific
practice to include a control
group of individuals who have not
been exposed to the putative
harmful agent. Indeed,
epidemiologists can occasionally
perform robust studies to explore
the causes of rare but
devastating events when only tiny
numbers of individuals are
affected (this was done, for
example, to explore the possible
link between overhead power lines
and leukaemia) by carefully
matching these individuals with
others who are alike in many ways
(age, sex, social class, smoking
status and so on) but who have
not been exposed to the putative
causal agent. This sort of design
is known as a case control
study. No matching was not
done by Wakefield and colleagues,
which means we have no way of
knowing that the unusual
combination of bowel disease and
autism-like syndrome might be
equally frequent in children who
had not received the MMR
vaccine.
The alleged link
with MMR vaccine was made on the
basis of retrospective parental
recall – in other words,
parents (who had just signed a
consent form to take part in a
study of whether there is a link
between MMR and autism) were
asked to consider how closely in
time the vaccine was with the
onset of autism-like behaviour
pattern in their child. Whilst
there is no suggestion that
parents deliberately fabricated
the closeness of the link, the
authors of the paper took no
steps to guard against what is
known as ‘recall bias’
(that is, remembering a closer
association between two events
than actually occurred). The
notion that a previously healthy
child was ‘normal’ one
day and showed clear signs of
autism the next day is at odds
with the clinical course of
pervasive developmental disorder.
Such syndromes tend to have a
period of weeks or months during
which the child’s behaviour
is causing some concern but is
not clearly abnormal. Hence the
firm statement of a “48
hour” or “2 week”
interval between the
administration of the vaccine and
the diagnosis of autism is
scientifically implausible and
requires further explanation.
The follow-up period
during which the children were
studied was short – days or
weeks rather than months or
years.
The investigators
were not ‘blinded’
– that is, the people who
examined the children and
analysed the specimens all knew
that the children had received
MMR vaccine and that a question
had been raised about the its
link with autism-bowel syndrome.
They would also have been aware
that ‘positive’
findings would be highly likely
to lead to a prestigious
publication whereas
‘negative’ findings
would not. In a scientifically
robust study, the people who do
the tests should be unaware of
the status of the samples.
6. Were the
study’s conclusions supported by
the data?
No. Whilst Wakefield and
colleagues stated at one stage in
their paper that their findings did
not prove a causal link between MMR
vaccine and autism-bowel syndrome,
the overall tone of the paper
strongly suggests that they believed
that they had demonstrated such a
link. This conclusion is unjustified
for the reasons given above.
7. If the answer
to (5) is no, would a more robust
design have been practically possible
to test the study’s main
hypothesis?
Most certainly, yes. It
is worth noting the key principles,
set out by Sir Austin Bradford Hill
in 1965, of any scientific study that
seeks to prove causation rather than
merely association:
Is there evidence
from true experiments in humans?
Is the association
strong?
Is the association
consistent from study to study?
Is the temporal
relationship appropriate (i.e.
did the postulated cause precede
the postulated effect)?
Is there a
dose-response gradient (i.e. does
more of the postulated effect
follow more of the postulated
cause)?
Does the association
make epidemiological sense?
Does the association
make biological sense?
Is the association
specific?
Is the association
analogous to a previously proven
causal association?
A study from Finland,
designed to be large enough pick up
even very rare events, followed 1.8
million children prospectively from
the day they received the MMR vaccine
for a full 14 years. The total number
of reported vaccine-associated events
was 437, and they included allergic
reactions and convulsions, but none
of the children had autism linked
with the vaccine.
Several subsequent
studies, all much larger and better
designed than Wakefield’s, have
confirmed an absence of excess cases
of autism or bowel disease in
children who have received MMR
vaccine. In 1999, for example, Dr
Brent Taylor looked at 500 cases of
autism in the Royal Free Hospital and
found no excess in immunised
children. A very large and well
designed study in the USA – with
a proper case control design and with
all vaccination dates confirmed by
medical records – looked at the
incidence of inflammatory bowel
disease in people who had received
MMR vaccine, other measles-containing
vaccine, and no vaccine. The risk of
inflammatory bowel disease was the
same for vaccinated or unvaccinated
people. The average time between
vaccination and the development of
bowel disease was 12 years. Only 1%
of cases developed inflammatory bowel
disease within a year of vaccination
– and 1% of controls developed
inflammatory bowel disease during the
same time period. This study was
cited in the medical journal
‘Bandolier’ and its results
are available free online on
http://www.jr2.ox.ac.uk/bandolier/index.html
In conclusion, the
Wakefield study was scientifically
flawed on numerous counts. I am
surprised that neither the editor nor
the reviewers spotted these flaws
when the paper was submitted. Had
they done so, the public would have
been saved the confusion and anxiety
caused by false credibility conveyed
by publication of the study in this
prestigious journal.
-
- Professor
Trisha Greenhalgh OBE MD FRCP
FRCGP
-
- Professor
of Primary Health Care and
- Director,
Unit for Evidence-Based
Practice and Policy
- University
College London
[1] This combination has been given
the informal name ‘autism-bowel
syndrome’ though no such
syndrome is officially accepted in
the mainstream medical literature
[2]
Bradford Hill, A, 1965. The
environment and disease: association
or causation? Proceedings of
the Royal Society of Medicine. Volume
58, 295-300, cited with permission in
Greenhalgh T, How to Read a Paper:
the basics of evidence-based
medicine; London: BMJ Publications, 2nd
edition 2002
[3]
A Patja et al. Serious adverse events
after measles-mumps-rubella
vaccination during a fourteen-year
prospective follow up. Pediatric
Infectious Diseases Journal 2000 19:
1127-1134.
[4]
B Taylor et al. Autism and measles,
mumps, and rubella vaccine: no
epidemiological evidence for a causal
association. Lancet 1999 353:
2026-2029.
[5]
RL Davis et al. Measles-mumps-rubella
and other measles-containing vaccines
do not increase the risk of
inflammatory bowel disease. Arch
Pediatr Adolesc Med 2001 155: 354-359