Was the most expensive ME/CFS study ever built on a house of cards?
David Tuller has published several long pieces on chronic fatigue syndrome but his piece “TRIAL BY ERROR: The Troubling Case of the PACE Chronic Fatigue Syndrome Study” on the UK’s federally funded PACE CBT/GET trial may be the most impactful. The trial with it’s many problems proved to be catnip for this investigative journalist – there was just so much there…
Tuller’s findings are being published in a three part series on Vincent Raccaniello’s Virology Blog. Parts I and II have been released.
Why is the PACE trial worthy of such attention? Because of all the attention it got. The $8,000,000, 600 person plus trial published in 2011 is still surely the biggest most expensive study ever done in ME/CFS. Proponents hoped the trial – and its sister trial the FINE trial – would prove that CBT/GET was the answer to chronic fatigue syndrome. The trials were expected to provide the foundation for the UK government to further broaden it’s commitment to these practices.
Seven years and two and half million dollars later the 300 patient FINE trial, however, flamed out, producing insignificant effects after one year.
With the first leg of the UK governments attempt to make CBT/GET the treatment of choice failing miserably, the PACE researchers must have been under considerable pressure to make sure their trial succeeded.
Getting their study published in The Lancet in 2011 – one of the most prestigious medical journals in the world – was an excellent first step to rehabilitating the government’s CBT/GET goals. In 2014 The Lancet ’s impact factor was ranked second only to The New England Journal of Medicine. Given the respect the Journal engenders simply getting the results published there gave the study a strong seal of approval.
At first blush the study appeared to be moderately successful. The authors concluded that “CBT and GET can safely be added to SMC (standard medical care) to moderately improve outcomes for chronic fatigue syndrome”
That was hardly a ringing endorsement but commentary published by Lancet (“30 percent of the study participants achieved recovery” ) and the author’s public endorsements (“twice as many people on graded exercise therapy and cognitive behaviour therapy got back to normal.” ) ramped up the media response to the study. The media jumped on statements like these to assert that exercise and behavioral therapies worked well and could even be the ticket out for ME/CFS patients.
Tom Kindlon, Graham and other ME/CFS patients were not sure. Their analyses – which eventually extended to virtually every aspect of the study – indicated that the authors had had to put the trial through remarkable contortions to get even the modest results they did.
Many of their findings have been published before but Tuller is wrapping them up and adding to them in a way that only an investigative journalist of his ilk can. This blog provides an overview of some of Tuller’s findings plus Graham’s videos explaining some of the issues.
Stumbling at the Gate: The PACE Trial Issues
Changing Horses in Midstream – Determining how to assess results is a critical part of study development. Given the scope of this trial the researchers surely thought long and hard about the best way to do that. It was important that they get it right because changing result assessments after a trial has begun is frowned upon in scientific circles.
Tuller reported, though, that the researchers began the trial intending to employ one set of tools to measure physical functioning and fatigue, but changed horses in midstream and ended up employing different tools. In fact many parts of the study protocol were changed over time.
The Slippery Slope – Redefining What Recovery Is – The researchers started out with four criteria for recovery – and by the end of the trial had relaxed them all – again suggesting that they were trying to find a way to meet their targets.
The changing definitions of recovery allowed investigators to claim people had recovered who had not
Guaranteeing Recovery – In a bizarre twist – possibly never seen before in clinical trials – the authors made the criteria for “recovery” so low that some patients could enter the trial already being “recovered”. In fact, Tuller points out that some patients could have gotten worse during the trial and still be labeled as “recovered”.
Downplaying Negative Results – Despite the researchers claims that CBT/GET were moderately effective none of the objective tests ( walking test, step test, employment data) provided evidence, that that was so. Instead the authors relied on subjective tests which we will see were prejudiced by the researchers actions.’
Rallying the Troops! – Objectivity is the touchstone of medical research. The authors of the PACE trial, however, gilded their own lily by a) sending the participants — in the midst of the trial – a glowing newsletter indicating how well everyone was doing, and b) by informing the participants that the “best evidence available” guided the decision to produce the trial; i.e. they essentially told the participants the methods were effective.
Not surprisingly some statisticians have had a field day with the PACE trials bizarre twists:
Bruce Levin, professor of biostatistics at Columbia University
“I have never seen a trial design where eligibility requirements for a disease alone would qualify some patients for having had a successful treatment…I find it nearly inconceivable that a trial’s data monitoring committee would have approved such a protocol problem if they were aware of it.”
Arthur Reingold – head of Epidemiology at the University of California, Berkeley
“Given the subjective nature of the primary outcomes, broadcasting testimonials from those who had received interventions under study would seem to violate a basic tenet of research design, and potentially introduce substantial reporting and information bias,” said Reingold. “I am hard-pressed to recall a precedent for such an approach in other therapeutic trials. Under the circumstances, an independent review of the trial conducted by experts not involved in the design or conduct of the study would seem to be very much in order.”
Is the PACE report worth the paper it’s written on
Tuller has done such a thorough job it’s clear that Lancet should reassess the paper.
Aftermath
Whatever the authors of the study were selling the funder of the pace trials didn’t appear to be buying. For the first time, the MRC in 2011, devoted all of its $2.6 million to physiological ME/CFS research studies.
Meanwhile the CBT/GET juggernaut roles on. Despite the Cochrane report indicating that the majority of patients (60%) in CBT studies showed no significant clinical improvement at all, no less than eight CBT/GET studies have appeared this year. A recent examination of non-behavioral RCT clinical trials in ME/CFS over the past five years found a grand total of eleven.
Much more is available in Tuller’s articles including a fascinating story of how Tom Kindlon and others uncovered the PACE trials flaws, how the study authors use of language caused the media to overstate the studies findings, the unrevealed ties the authors had to the insurance industry, the unwillingness of the study authors to comment on Tuller’s piece.
Part III focuses on the participants responses to the trial.
It should be noted that in a remarkable display of generosity David Tuller has now produced several very lengthy feature articles on chronic fatigue syndrome – none of which he’s been paid for.
Really glad to see Mr. Tuller pick up the good work by Tom Kindlon. Before I still didn’t know who to believe, since Tom is a “small fish” and the Lancet and the study authors are such “big fish”. Tuller’s articles and your quotes from Levin and Reingold convince me fully that the PACE study is not legit.
You make a key point, Johannes – that people don’t assume that patients have the necessary expertise to assess a trial – not even other patients.
But experts such as Davis, Edwards, Reingold and Levin have such authority that people immediately take them seriously and they’re to be applauded for their frankness and their willingness to break ranks.
But of course, without Tom Kindlon and the other patients who helped him, we’d never have got to this stage and we wouldn’t now be hearing these illustrious voices.
Congratulations to Tom and his colleagues!
I was impressed as well by the firepower Tuller brought to the conversation….That’s what investigative reporters do I guess…
The little guys were right this time! It’s kind of remarkable.
A lot of the little guys probably have a background in science. ME/CFS is an equal opportunities disease and being a scientist doesn’t protect anybody from getting sick. In a UK ME/CFS population of 250,000, there are going to be scientists.
We probably have a lot of expertise in our community. It certainly appears that we have more than the PACE trial did.
There is a bit of worrying commentary here. The concept of “Big Fish” and “Small Fish” is half the here. When it comes to solid critical analysis there is no fish-just science where we are all equal if we follow the correct procedure which Tom has always done.
I come from a scientific background – and know that research scientists are just normal people. There is no reason why a patient cannot analyse data if they are prepared to do the work and understand the basic tenets of science. A dismissal of patient feedback or research due to a sort of snobbery or misplaced sense of authority is dangerous. As we have seen. It is a concept we should be sure of for our own safety and work hard to ensure and support it.
Tom Kindlon is a hero and it is even more impressive he has conducted such solid research whilst being affected by the illness.
I meant to say “big fish small fish is half the problem.”
Thanks for this article, Cort. This situation is absolutely shameful. All patients owe a great debt of gratitude to David Tuller for writing his piece, which must have taken him a very long time.
We should also be grateful to those scientists who have been willing to stand up and say publicly what a piece of utter dreck the PACE trial was.
I hope other scientists will now sit up and take note and that the trial authors will be forced to publish the results according to the original statistical protocol.
Yes, thanks for mentioning David. I am now going to acknowledge him in the piece. Thanks for the reminder.
I have the MS like symptoms without lesions let’s call it Fibromyalgia have been able to exercise and then injure myself over the years because finally a physical therapist decided my body could only take low impact activities like water aerobics at 15 minutes building up, but this wasn’t at cure. My dear, dear CFS buddy who got ill her first year teaching Early Ed. I tell you, little children are dangerous! 40 new viruses a year, and they are Petri Dishes waiting to happen! But Linda got Epstein Barr and has had very few periods of non-flares since 1994. Her immune system cannot get a grip. She like me is a two time cancer survivor. And I think if you told her exercise would cure the woman who is too tired to even check out this site. She would shoot you, and rightly so
I’m glad these experts are now agreeing with us on the terrible piece of ( non) science. My only worry is that most are retired and have nothing to loose in speaking up for us. We need to hear from clinicians. Problem is they are too scared to speak out for fear of loosing their funding. I hope I’m wrong in this assertion.
I think the most telling thing about this so called PACE ‘trial’ is that they have repeatedly refused to release the raw data. This shows that they KNOW their data, and every other part of this ‘study’ is riddled with a total lack of scientific rigour.
Ironically I was to attend a CFS clinic that was included in the trial, apparently I just missed a course that had just started ie I could have started immediately if only I was referred a week earlier, So by rights I was top of the list for the next one, during this period I was getting worse and was unable to work. By the time the next Course arrived apparently it was full up!!!!! and this happened again, eventually one year later after the deadline for those to be included in the PACE trial had passed I managed to get on the course. All except one other person had not been on any waiting list they had been immediately placed on the course, they had known others on previous courses that also had no problems being admitted to a course. The other person that had same problems as myself was also deteriorating and had ceased work. I obviously have no concrete evidence of what I am implying but I know what I make of it.
Secondly even when I was on the course do I think that the clients all had CFS/ME No!!! you be lucky if it was 50%
At last! Congratulations to all concerned. I came across another paper published this year from the same PACE authors, describing the difficulties they had to overcome carrying out the trial, and unbelievably providing helpful advice to others on how to conduct such a trial in the future!!!! it’s available online at:
http://pb.rcpsych.org/content/39/1/24
Thanks – this is how they start off their paper:
!
and then this:
Our deliberate policy, to help allay concerns about the trial, was to be as transparent as possible regarding what we did,
Gotta love em…
I have one word for the peer review of the pace trial: Nepotism!
Maybe someone should talk to all the people who have been physically hurt and have had their changes of recovery destroyed by graded exercise and CBT and who are then blamed for relapsing because they are thought to be doing it to themselves. Graded exercise and CBT can and has been used to abuse patients.
My immune system is in overdrive, sending out too many chemical fighters and their answer has been to stimulate it even more with exercise. This has never made any sense to me.
as has been said many scientists get ME too. Our brains my not work as well as they did but red when lying down and some of us can still spot flaws when we see them.
This has always been a case of the Emperor’s New Clothes https://en.wikipedia.org/wiki/The_Emperor%27s_New_Clothes (wiki link for anyone who hasn’t heard the story)
The significance of all this goes way beyond one trial and its merits or flaws.
Not just in the UK but in many other countries, too, programmes centred around CBT and GET continue to be introduced as “evidence based”, and all too often the UK’s PACE trial is cited as key evidence for the “efficacy” of these therapies.
Rarely can one, shambolic piece of work been so influential. It is truly a victory for spin and dogma over substance.
Thanks for helping bring attention to this, Cort.
Personally, I am very thankful for CBT. The clinical psychologist I saw wrote a strongly-worded letter that enabled me to get ill-health retirement. That was what I needed. The only part of her initial report that I question was the premise underlying her statement that ‘From our own research trial we know that 73% of patients respond to this form of treatment and 27% do not. I am afraid Diane is in the non-responding group.’ My own thought about this is that so much of what I was taught was common-sense, and as an intelligent woman who had suffered from this disease since childhood, I had learnt good coping strategies. In fact, the first person I saw for my CBT sessions (a nurse specialist) said at the end of our first session ‘But you already do all the things I normally teach people.’
(Also of interest is that a letter written a year later cited figures of 70% and 30%. Memory error or different tests?)
Anyway, my reaction to CBT is that it can be a useful aid to help one evaluate daily practices and think through procedures – but only for those with the energy to do this (I was given 4 months off work to do it) and the will to change. It does not cure the illness; it helps one maximise the available limited energy.
Thanks for mentioning that Diane. I am not anti-CBT. I agree that it can be helpful. Indeed I fail to see how it could not be helpful – if done right – given how difficult this disease is. It’s the overstating and the overwhelming emphasis on it in the UK and other areas that is troubling.
I agree with you completely Cort. I found it frustrating that the only help I was given here in the UK was CBT. In fact I was initially told that I would not get ill-health retirement because I had not completed the CBT course. That was what sent me back.
Anything that has been of help to me has come through an Australian doctor I used to see when I went there every 2 years on holiday, or through research and blogs such as this.
Thanks for writing this Cort! I typically wait for your analysis when news comes out because My cognitive difficulties make it really hard to read long articles and studies.
Your posts are much appreciated!
Thanks Snookum96 🙂
Glad to hear they help.
Thanks to all who have contributed to this work of recognising exactly what the PACE Trial has achieved. Damage, anger, frustration, doctor struck off after that group of mental health workers falsely accused that doctor of malpractice( took to court and won the case) – reinstated by NHS but has private practice now!! Dr Myhill!
Decades of trying to get this group of psych’s exposed for their disgraceful behaviour and children snatched from families due to not being diagnosed quickly enough.
See Film ‘Voices from the Shadows’ and that will tell you the damage done by CBT and GET.
How do we get the UK Government to listen??? They don’t listen to our ME Experts and ignore Expert Biomedical Research papers!! NICE refuse to review their guidelines and NHS Official says NICE don’t have the capacity to review everything. They deal with their priorities.
Please, for all our sakes, help us make the changes.
Thank you.
Forever hopefulx