Just how much new research can we trust?

The Sunday Times, May 21 2006

Brian Deer says scientific studies are not as reliable as they pretend to be

Last month brought welcome news for nicotine junkies everywhere. According to press reports from an American conference, “new research” suggested that “smokers who’ve tried but failed to kick their habit may want to pop a daily aspirin, ibuprofen or naproxen (Aleve) to help cut their risk of mouth cancer”.

The reports, however, neglected to mention the other side of the coin: that these painkillers cause more medical emergencies than any other pharmaceutical product. “Research”, in fact, suggests that stomach bleeding triggered by these so-called “non-steroidal anti-inflammatories”, or NSAIDs, leads to about 3,500 hospitalisations and 400 deaths among Britons over 60 each year.

Family doctors are exasperated by patients demanding new drugs or treatments based on what they read. “Patients come in all the time under the influence of these stories,” says Dr Mike Fitzpatrick, an east London GP. “Like cholesterol-lowering drugs. One minute new research says they are better than we thought. Then more new research links them with serious side effects.”

READ HOW DEER’S WAKEFIELD
INVESTIGATION WAS CHECKED

Maybe somebody should start a league table to rank the credibility of the new research’s sources. The mouth cancer research, for instance, is the baby of Dr Jon Sudbo, a consultant oncologist at Norway’s Rikshospitalet and, until recently, an adjunct professor at the reputable Oslo University.

Reporting last October in The Lancet, once the flagship of British medical publishing, Sudbo and a team of 13 other doctors laid out an impressive nine pages of text, tables and statistics pointing to the painkillers’ benefits.

That seems to nail it. But on the other hand, the stomach research’s source is impeccable. The risks of painkillers and bleeding are most notably the work of Professor Michael Langman, who is not only the former dean of medicine at Birmingham University, but also sat for many years on the government’s late committee on safety of medicines. He is an expert’s expert.

No assistance yet, then, with Fitzpatrick’s dilemma. As a wise aunt of mine regularly advised me in my younger and more vulnerable years: “Believe nothing you hear and hardly anything you see.”

If you have the time, however, further inquiry helps. For instance, Sudbo has recently joined a growing list of doctors and scientists who have been condemned by their colleagues as cheats. Amazingly, two months before the press reports announcing his dramatic “new research” for smokers, The Lancet had published a worrying 52-word statement buried on a left-hand page.

“We have received confirmation,” Dr Richard Horton, the journal’s beleaguered editor, admitted, “that the paper published by Jon Sudbo and colleagues in The Lancet contains fabricated data . . . and we now retract this article in full.”

This inclines me to think that Langman’s research is the weightier of the two. If I smoked, I wouldn’t also “pop aspirin”. True or false, Sudbo’s suggestion should therefore perhaps be considered a sort of urban myth — like advice to pregnant women to smoke fags for a smaller baby and that inhaling blue asbestos clears the lungs.

What about this Langman chap? This is a tricky one. Langman is no faker. Yet his enthusiasms have sometimes run amok. A Sunday Times investigation revealed last summer that, while sitting on the government’s drug safety watchdog, in April 1999 he flew to the United States for the Merck drug company and lobbied regulators in support of Vioxx, Merck’s painkiller which was later withdrawn after an epidemic of heart deaths. Ooops.

“There is a crisis of credibility in medicine and science,” says Dr Joseph Sonnabend who, as a former virologist for the Medical Research Council and a retired Aids physician, has watched the basis for public confidence decline. “Less and less information seems purely disinterested. You have to check everything out.”

If you cannot always trust the doctors, however, what about the journals in which they write? About half the top 20,000 medical titles trumpet their status as being “peer reviewed”, which means that research is supposed to be vetted before publication: not merely with a “spellchecker” but by relevant specialists.

“A rationale for peer review is that it is rare for an individual author or research team to spot every mistake or flaw in a complicated piece of work,” is how an anonymous Wikipedia editor (obviously a doctor) explains it. “In a new and perhaps eclectic intellectual product, an opportunity for improvement may stand out only to someone with special expertise or experience.”

It was failings on this front, however, which helped to unleash the MMR vaccine debacle. Although place of honour lies with Dr Andrew Wakefield, the former gut surgeon whose discredited allegations were published in The Lancet, it was Horton, and at least one peer reviewer, who failed to spot the telltale signs.

“Many cases of poor science, or even scientific fraud, are also failures of peer review,” says Dr Evan Harris, Liberal Democrat MP for Oxford West and Abingdon and a member of the House of Commons science committee. “The process is critical in distinguishing mere assertion from something that can be tested and judged by others.”

Failings in this system have particularly stung The Lancet, which in addition to publishing claims by Sudbo and Wakefield (both retracted), has been regularly roasted for printing twaddle. Most famously, in 1999 the journal set off yet another public scare over genetically modified food, which was based on tests (later challenged) with potatoes and rats.

The Lancet’s troubles, however, are just examples, albeit the most spectacular. Only last week The Wall Street Journal revealed that an editor of The New England Journal of Medicine (the world’s No 1 medical title) had admitted that “lax editing” had contributed to a research paper in 2000 which gave a misleading impression of Vioxx’s safety, and that the journal had sold reprints grossing nearly $700,000, most of which had been bought by Merck.

A few years ago the British Medical Journal published a paper from researchers at University College London claiming — again with meticulous tables — that 53.8% (a majority) of heterosexually active women, aged 44 or under, suffered from a sexual dysfunction. When I phoned to ask whether this could really be true, the paper’s chief author hung up.

Whether fraud or foolishness, the cause is always the same: the need to publish something — anything — in a journal. Researchers gain promotion (especially to the magic “professor” circle) on the quantity and purported quality of their papers. And the journals, in turn, are ranked by arcane “impact factors” which, broadly, measure how often they are quoted.

Each year in June these impact factors are recalculated, to be gawked at like Strictly Come Dancing. The last release I could find clocked The New England Journal of Medicine at a lofty 38.570, Britain’s top science journal Nature at a credible 32.182, and The Lancet trailing at 21.713.

For the journals and their publishers, the bottom line is money. The Anglo-Dutch Reed Elsevier group, which owns The Lancet and a library full of other medical journals, scientific book imprints and technical databases, enjoys a healthy profit margin. Any aim to inform sometimes seems incidental to the mission. Accuracy is good but not essential.

After decades in which newspaper journalists were regarded as the medically ignorant, paradoxically it is now the medical journals that are under the spotlight over whether or not their stories are true. At least that’s what new research suggests.