David Tuller has published several long pieces on chronic fatigue syndrome but his piece “TRIAL BY ERROR: The Troubling Case of the PACE Chronic Fatigue Syndrome Study” on the UK’s federally funded PACE CBT/GET trial may be the most impactful. The trial with it’s many problems proved to be catnip for this investigative journalist – there was just so much there…
Tuller’s findings are being published in a three part series on Vincent Raccaniello’s Virology Blog. Parts I and II have been released.
Why is the PACE trial worthy of such attention? Because of all the attention it got. The $8,000,000, 600 person plus trial published in 2011 is still surely the biggest most expensive study ever done in ME/CFS. Proponents hoped the trial – and its sister trial the FINE trial – would prove that CBT/GET was the answer to chronic fatigue syndrome. The trials were expected to provide the foundation for the UK government to further broaden it’s commitment to these practices.
Seven years and two and half million dollars later the 300 patient FINE trial, however, flamed out, producing insignificant effects after one year.
With the first leg of the UK governments attempt to make CBT/GET the treatment of choice failing miserably, the PACE researchers must have been under considerable pressure to make sure their trial succeeded.
Getting their study published in The Lancet in 2011 – one of the most prestigious medical journals in the world – was an excellent first step to rehabilitating the government’s CBT/GET goals. In 2014 The Lancet ’s impact factor was ranked second only to The New England Journal of Medicine. Given the respect the Journal engenders simply getting the results published there gave the study a strong seal of approval.
At first blush the study appeared to be moderately successful. The authors concluded that “CBT and GET can safely be added to SMC (standard medical care) to moderately improve outcomes for chronic fatigue syndrome”
That was hardly a ringing endorsement but commentary published by Lancet (“30 percent of the study participants achieved recovery” ) and the author’s public endorsements (“twice as many people on graded exercise therapy and cognitive behaviour therapy got back to normal.” ) ramped up the media response to the study. The media jumped on statements like these to assert that exercise and behavioral therapies worked well and could even be the ticket out for ME/CFS patients.
Tom Kindlon, Graham and other ME/CFS patients were not sure. Their analyses – which eventually extended to virtually every aspect of the study – indicated that the authors had had to put the trial through remarkable contortions to get even the modest results they did.
Many of their findings have been published before but Tuller is wrapping them up and adding to them in a way that only an investigative journalist of his ilk can. This blog provides an overview of some of Tuller’s findings plus Graham’s videos explaining some of the issues.
Stumbling at the Gate: The PACE Trial Issues
Changing Horses in Midstream – Determining how to assess results is a critical part of study development. Given the scope of this trial the researchers surely thought long and hard about the best way to do that. It was important that they get it right because changing result assessments after a trial has begun is frowned upon in scientific circles.
Tuller reported, though, that the researchers began the trial intending to employ one set of tools to measure physical functioning and fatigue, but changed horses in midstream and ended up employing different tools. In fact many parts of the study protocol were changed over time.
The Slippery Slope – Redefining What Recovery Is – The researchers started out with four criteria for recovery – and by the end of the trial had relaxed them all – again suggesting that they were trying to find a way to meet their targets.
Guaranteeing Recovery – In a bizarre twist – possibly never seen before in clinical trials – the authors made the criteria for “recovery” so low that some patients could enter the trial already being “recovered”. In fact, Tuller points out that some patients could have gotten worse during the trial and still be labeled as “recovered”.
Downplaying Negative Results – Despite the researchers claims that CBT/GET were moderately effective none of the objective tests ( walking test, step test, employment data) provided evidence, that that was so. Instead the authors relied on subjective tests which we will see were prejudiced by the researchers actions.’
Rallying the Troops! – Objectivity is the touchstone of medical research. The authors of the PACE trial, however, gilded their own lily by a) sending the participants — in the midst of the trial – a glowing newsletter indicating how well everyone was doing, and b) by informing the participants that the “best evidence available” guided the decision to produce the trial; i.e. they essentially told the participants the methods were effective.
Not surprisingly some statisticians have had a field day with the PACE trials bizarre twists:
Bruce Levin, professor of biostatistics at Columbia University
“I have never seen a trial design where eligibility requirements for a disease alone would qualify some patients for having had a successful treatment…I find it nearly inconceivable that a trial’s data monitoring committee would have approved such a protocol problem if they were aware of it.”
Arthur Reingold – head of Epidemiology at the University of California, Berkeley
“Given the subjective nature of the primary outcomes, broadcasting testimonials from those who had received interventions under study would seem to violate a basic tenet of research design, and potentially introduce substantial reporting and information bias,” said Reingold. “I am hard-pressed to recall a precedent for such an approach in other therapeutic trials. Under the circumstances, an independent review of the trial conducted by experts not involved in the design or conduct of the study would seem to be very much in order.”
Tuller has done such a thorough job it’s clear that Lancet should reassess the paper.
Whatever the authors of the study were selling the funder of the pace trials didn’t appear to be buying. For the first time, the MRC in 2011, devoted all of its $2.6 million to physiological ME/CFS research studies.
Meanwhile the CBT/GET juggernaut roles on. Despite the Cochrane report indicating that the majority of patients (60%) in CBT studies showed no significant clinical improvement at all, no less than eight CBT/GET studies have appeared this year. A recent examination of non-behavioral RCT clinical trials in ME/CFS over the past five years found a grand total of eleven.
Much more is available in Tuller’s articles including a fascinating story of how Tom Kindlon and others uncovered the PACE trials flaws, how the study authors use of language caused the media to overstate the studies findings, the unrevealed ties the authors had to the insurance industry, the unwillingness of the study authors to comment on Tuller’s piece.
Part III focuses on the participants responses to the trial.
It should be noted that in a remarkable display of generosity David Tuller has now produced several very lengthy feature articles on chronic fatigue syndrome – none of which he’s been paid for.