PACE Trial Gets Most Devastating Critique Yet

Cort

Founder of Health Rising and Phoenix Rising
Staff member
The PACE trial may, in the end, be about more than an injustice done to the chronic fatigue syndrome (ME/CFS) community. It may become, if the latest devastating criticism of it takes hold, exhibit number one in the medical community of how not to do a clinical trial.

[fright]
Wrecking-Ball.jpg
[/fright]It also presents a challenge to the orthodoxy of the medical profession. To think that a bunch of sick patients and a journalist working without pay could bring down an $8 million dollar study and embarrass one of the most respected medical journals in the world. It shouldn't happen but with the latest critique emanating from a statistician one wonders how much time the PACE trial and Lancet have left.

Key Figures

The PACE trial has taken many shots but the first shots taken - by ME/CFS patients - may have been the most important. Tom Kindlon, Graham McPhee and others in the chronic fatigue syndrome (ME/CFS) community created the foundation for the trials trouble. It wasn't that they objected to the trial's conclusions; it was how methodically they did it. The rigorous manner with which they presented their case give their critique the legs it needed to get out of the community.

That happened in a big way when David Tuller blew the whole thing open with his multi-part series "Trial by Error" in Virology Today last October. Tuller couldn't get a major media outlet to bite so he published his huge piece in a small virology blog. You wouldn't expect it to get much traction there, but Tuller's exhaustively researched series began a firestorm that has not quit. Tuller is the second essential piece in the PACE saga. Without Tuller - PACE stands. After Tuller - who knows what will happen.

David Tuller may not get a Pulitzer Prize for investigating PACE trial on a blog; but his service to—and we do not exaggerate—millions of sufferers around the world make it hard for us to think of another work of journalism so deserving of commendation.

[fleft]
Tuller,-David-2012.jpg
[/fleft]Jon Cohen followed Tuller's first blow with a short review in Science. In it he quoted, one of the PACE authors, Michael Sharpe, saying he doesn’t think there’s “a growing army of people upset about this". (Boy, was he ever wrong.)

In November six researchers demanded that Lancet review the 2011 study, James Coyne got in the act with a call for the release of the PACE data, and patients filed an Freedom of Information Act to get access to the same.

Julie Rehmeyer, a journalist with ME'/CFS who recently won an award for excellence in statistical reporting got the PACE trial into the mainstream media with a November piece in Slate. The Wall Street Journal picked up the ball in March after 43 scientists asked Lancet to reanalyze the data and an 11,000 person MEAction petition was submitted with an Amy Dockser Marcus article.

Statistics Organization Speaks Out

All that was prelude, however, to the latest and perhaps most devastating blow to the PACE trial yet - an open critique from Rebecca Goldin, the director of Stats.org, and Professor of Mathematical Sciences at George Mason University. Goldin's entry into the debate indicates that the PACE trial controversy is now bigger than ME/CFS; that it's being held up and examined in the medical community as a case study of a major research effort gone wrong.

The question of how all this happened and how the criticism is being handled have sent shockwaves through medicine. The results from PACE (including these) have been published in prestigious journals and influenced public health recommendations around the world; and yet, unraveling this design and the characterization of the outcomes of the trial has left many people, including me, unsure this study has any scientific merit. How did the study go unchallenged for five years?

[fright]
Keeping-eye-on.jpg
[/fright]If this keeps up the 600 person plus, $8 million PACE trial may end up making the textbooks in a way its authors could not have imagined: a case study of how not to produce a clinical trial.

Goldin, who has no connection at all to ME/CFS, but is very committed to statistical rigor in research, was scathing in her critique. Stating that "problems with the study (existed) on almost all levels" Goldin focused on just one area - study design. Goldin doesn't just review the problems in the PACE trial - she dissects them, going through them step by step.

Selection Bias

Problems with "selection bias" lead the authors of the trial to select out patients who would be most likely to benefit. Goldin reported that the patients in the trial were "more likely to suffer from depression, less likely to meet the clinical criteria for ME, more likely to be able to function a little bit, and less likely to have found an effective way to manage their illness."

The authors got the population they wanted. The strangest thing about the trial in the end, though, may not be all the shenanigans the authors went through to up their recovery rates of the participants in the trial but how poorly the trial did even after doing so.

Shifting Recovery Criteria

[fleft]
Pace-Recovery.jpg
[/fleft]"Dramatic" alterations in recovery criteria as the trial was underway made it easier for the authors to label the patients as "recovered". Goldin cited the now notorious alteration which made it possible for patients to meet the criteria for ME/CFS and be classified as recovered from it at the same time.

Goldin seemed like nothing if not shocked by the manipulations of the physical functioning criteria. The physical functioning score required for someone in the trial to be considered recovered from ME/CFS was 60 - just five points lower than the normative scores for 75 year olds. Those normative scores, by the way, are not from healthy 75 year olds - they're the average score for all 75 year olds - healthy and sick.

With 39 being the average age of the PACE trial participants, the PACE trial authors were essentially saying that if we can get a 39 year old to function at the level of a 75 year old, they would be considered "recovered". (The normative value for a 39 year old is 93). The PACE authors justified their criteria but in doing so exposed a basic mathematical error they made when they confused median values for mean values.

Goldin goes through two more of the changes to the recovery criteria that occurred over time including one which was so shocking that she asked the authors to confirm she had it right. (She did).

A PACE "Recovery Story"

She then presented an example of a patient the PACE trials might have helped to "recover" from ME/CFS.

For the sake of illustration, we can imagine someone who came into the study with extreme and debilitating fatigue, scoring a 6 on the bimodal fatigue scale, and 65 on the physical function scale. The person meets the clinical definition of CFS according to the Oxford criteria, but is generally an upbeat person.

While in the trial, she has not improved in her overall fatigue or her physical function, but she is quite happy to be getting expert medical care, and she is also sleeping a little better thanks to sleep medications. She rates herself as “much improved”, but not “very much improved” on the CGI, but she records no differences in her assessments of physical function or fatigue. She still cannot walk a mile, remember her words, or hold a job. Yet she is a case of someone who has “recovered” thanks to the CGI.

Doomed Study

In this long piece Goldin goes on to do more analyses that poke holes in the study and the authors reasoning. In the end the best she can say about the trial is that it provides an example of how to throw $8 million down the tubes.

It seems that the best we can glean from PACE is that study design is essential to good science, and the flaws in this design were enough to doom its results from the start.

Lost Trust

The burning question now is what Lancet, the journal the original study was published in, will do. One of the most prestigious medical journals in the world, studies published in Lancet carry enormous weight. I recently read a book on a large effort designed to assess the world's health problems. Tens of millions of dollars were spent on it over several years. The authors first goal was to get their publications into Lancet. Every analysis and every word was poured over with an eye to meeting Lancet's exacting standards - because, if it appears in Lancet, it's trusted.

[fright]
Bias.jpg
[/fright]The PACE trial results (which have been published in several journals) were trusted. In an editorial "On PACE: An Editorial" published alongside Goldin's critique Trevor Butterworth noted that the Independent's headline was “Got ME? Just get out and exercise say scientists.” The Medical News Today reported that Fear of exercise' is biggest barrier to chronic fatigue syndrome recovery". Others media pieces Goldin cited were:
  • “Psychotherapy Eases Chronic Fatigue Syndrome, Study Finds”—New York Times
  • “Pushing limits can help chronic fatigue patients”—Reuters
  • “Brain and body training treats ME, UK study says”—BBC
  • “Therapy, Exercise Help Chronic Fatigue Syndrome”—WebMD
  • “Helping chronic fatigue patients over fears eases symptoms”—Fox News
  • “Chronic fatigue syndrome patients’ fear of exercise can hinder treatment – study”— The Guardian
  • “Study supports use of 2 controversial treatments for chronic fatigue”—CNN
  • “Chronic Fatigue Treatments Lead To Recovery In Trial”—Medical News
Butterworth reported that because of the PACE trial "the UK’s National Health Service, the Centers for Disease Control, the Mayo Clinic, and Kaiser all ended up recommending cognitive behavioral therapy and exercise for ME/CFS" and that "PACE has become the paradigm for understanding a condition affecting millions of people."

That's of course very troubling to the ME/CFS patients who have been stymied again and again by their inability to be active, let alone "exercise". That a study of this size and cost and potential impact could be so shoddily run, and then show up in one of the best medical journals in the world, basically blew science reporter Julie Rehmeyer's mind.

But when you dig down into the details, you find that the data doesn’t support the researchers’ claims. The most amazing problem (among many) is that “recovery” was defined so loosely that patients could get sicker over the course of the study and still be said to have recovered! But this study is considered top notch, gold standard work. The media has fallen down here too—no article in the mainstream media has ever seriously analyzed this study, even though patients are being injured by it regularly.

The whole thing has been really shocking for me. It’s had a huge impact on my perspective on science and the world as a whole, and there are a lot of controversial issues where my emotional stance has changed in a really big way.

That's a tough thing to lay on a journal that's built up an impeccable reputation over many years. With this critique from an established statistician, the stakes for Lancet and other journals publishing the PACE studies just got higher. Will they lance the growing PACE boil, and let outside experts re-assess the studies or will they continue on their way, an embarrassment to themselves, and if Goldin is right, the medical profession itself.
 
Last edited:

Cort

Founder of Health Rising and Phoenix Rising
Staff member
"major research effort gone wrong" or "major research effort being intentionally deceptive"?
Major research effort doing everything they can -including things they shouldn't have done - to improve their results...and getting away with it for about five years.

The problems are so obvious, in retrospect, that they also seem to be that gang that couldn't shoot straight.

How did the Brits pass review on this? How did a Journal like Lancet (and other Journals) give it a pass?
 
Last edited:

Nico

Member
Cort, thanks again for another great piece of reporting! I want to answer Dr. Golding's question. It's only from my very small perspective of course.
"How did the study go unchallenged for five years?"

I've been following Tom Kindlon on a social media platform for a couple of years now, and have seen his repeated posts and challenges regarding this trial. I think I may have tried reading the information a couple of times, but probably gave up. One reason is, most likely, the illness itself and how it affects me cognitively. Next to that, is that I don't think my brain is wired to analyze data and scientific explanations.

Also, not everyone has the same level of tenacity within - even the human population let alone - the ME/CFS population. I know that Tom's efforts were ongoing. I think too, there is possibly a degree (and this, by no means is a criticism) of learned helplessness in our community. I know for me, my ability to vie and advocate for different things really fluctuates, and sometimes I just think: "I give up, and I'll wait for _________ (researcher) to get the science right." It's not really a chosen learned helplessness, it's by default due to lack of stamina.

This might be the first time that our patient community (and, I could be wrong) has attracted attention of those not somehow directly tied to the community itself. I think we all know of ME/CFS's general "swept under the rug" status (so I feel, anyway)…that it's just not daunting enough or scary enough for a wider populous to be that interested in our affairs. We now have gained a wider audience, and an audience with a vested interest in true science and gold standard practices. I've been ill for 10 years, and to me this kind of ripple effect seems to be a first.

Just my 2 cents' worth.
 

gowwab

Member
"major research effort gone wrong" or "major research effort being intentionally deceptive"?


The latter is the only thing that makes a bit of sense and it fits the pattern we've seen in our country of wanting to make us go away. Then the question is who wants to burry us and why. There is already know conflict of interest with the researchers, the NHS and insurance companies that they did not disclose to the patients in the trial which they were required to do by law.

Follow the money usually gets you the answer.

Cort you should send this to all the media outlets listed to ask if they are going to follow up with their reporting in the light of new revelations. It's very helpful to see the timeline of recent events concerning Pace put into one place. Thanks.
 

Bottsie

Member
I'm unsure as to how all of this works, but I have a concern... These erroneous articles that have been generated and published are 'out there'. Is there any way to remove/delete/retract them so that as medical professionals gain interest and begin to try to educate themselves, they won't be misinformed by this mess? I am deeply grateful for all of those who have been so diligent in trying to right this grievous wrong to the ME/CFS/SEID patient community.
 

tatt

Well-Known Member
The problems are so obvious, in retrospect, that they also seem to be that gang that couldn't shoot straight.

Cort I think in the past - and not that long ago - you have written things suggesting that PACE wasn't as bad as we Brits have made i out to be and that Wessely wasn't as bad as we perceived him to be. The problem is that people with ME have been perceived as mentally ill and therefore it's very difficult to get anyone to listen. Most people with ME dont have the energy to go on and on presenting the same case. All credit to those who have managed to repeat the criticisms over and over and to do so with politeness and great patience in the face of extreme provocation.

However the latest criticisms are still from what will be seen in the UK as a minor university. It needs something like the UK's Royal Statistical Society to have much weight with the Lancet.
 

KweenPita

Active Member
Thank My Father in Heaven, finally a Statistician is looking at these "researchers" findings. I have been beating this dead horse to death. They 1# target a carefully screened population to prove there hypothesis, without enough numbers to be considered mathematically a cross section of "our society". Even if they wanted similar ages, backgrounds, socio-economic, they could get at 1500 of us, no prob Bob. You host to have the have the numbers, untainted numbers, "not massaged" as the data collectors used to say. Well, I guess they done been stroking this study a bit too much, and things got messy. But cold hard math facts don't lie.

And these piddly 28 person break throughs, "When people with Fibromyalgia scratch the bottoms, it has proven it's less itchy" That's not good science. That's not statistically proven it is worth a tinkers damn to easing our, but it helps ease more fund raising power from people who have money and want to feel they are putting it to a good solid cause and keeps them comfortably employed.

I know what fundraising is all about. It's about getting people with money to get emotionally invested wallets behind your "cause". No better than some of the hack Tele Evangelist, same principle, we be healing sick folks, just write your check and you can help us help them poor suffering creatures.
 
Last edited by a moderator:

Sarah L

New Member
Well, we attracted a lot of attention with XMRV, both as a possible cause and then for scandal, retraction, when XMRV turned out to be a contamination in several laboratories and not a virus at all. That is how virologists came to know us so well. The second open letter to the Lancet included more than one person whose specialty was proper study design, prominent in that field, and thus statisticians. I am glad to hear from one more.

I have always been glad for Tom Kinldon's work. The PACE study was a mistake from the start, with a poor design, and following a similar study that had no useful results. I never designed medical studies, but I did design studies for my own field, and the flaws in PACE were there from the start, a real waste of money. That the Lancet let itself be roped in, well, that led me to think that there is an in-group for UK medicine, where it is who you know and not what brilliant work you do, that gets your work published. Cynical I suppose, but what is a person to think? Especially when the authors act like spoiled children when criticisms arrive, valid criticisms.
 

Kim Garrett

Member
Major research effort doing everything they can -including things they shouldn't have done - to improve their results...and getting away with it for about five years.

The problems are so obvious, in retrospect, that they also seem to be that gang that couldn't shoot straight.

How did the Brits pass review on this? How did a Journal like Lancet (and other Journals) give it a pass?
This is disgusting. A complete diservce to the chronically ill. And how they manipulated the findings , it should be criminal . I think there was a second study at the time , if memory service especially me right . That found viral involvement of CFS. That seemed to basically get swept under the rug . For the Great New Hope of a baffled medical community called PACE .
 

sasha

Member
I'm unsure as to how all of this works, but I have a concern... These erroneous articles that have been generated and published are 'out there'. Is there any way to remove/delete/retract them so that as medical professionals gain interest and begin to try to educate themselves, they won't be misinformed by this mess? I am deeply grateful for all of those who have been so diligent in trying to right this grievous wrong to the ME/CFS/SEID patient community.

cleaning up google's offensive "medical summary" that is the first thing to pop up in a search on CFS would be nice, too (a product of google's partnership w/ mayo that started last year). GET/CBT, antidepressants and referral to a psychologist/psychiatrist.
 

Kim Garrett

Member
My question is why would Mayo , the CDC and other health entities not have reviewed this reseach before making it their new treatment protocol? And if they had reviewed why did they choose to make it their new treatment protocol?
 

Cort

Founder of Health Rising and Phoenix Rising
Staff member
My question is why would Mayo , the CDC and other health entities not have reviewed this reseach before making it their new treatment protocol? And if they had reviewed why did they choose to make it their new treatment protocol?
I don't think Mayo really digs into the minutiae of studies - I'll bet they did just what everyone except for patients and some ME/CFS researchers did - they relied on the study findings - and trusted Lancet.

The CDC is another story. I don't know that their take on the PACE trials is...I'll bet most major medical sites were espousing CBT/GET long before the PACE trial though. I think he's a bit off on that.
 

Cort

Founder of Health Rising and Phoenix Rising
Staff member
Cort I think in the past - and not that long ago - you have written things suggesting that PACE wasn't as bad as we Brits have made i out to be and that Wessely wasn't as bad as we perceived him to be. The problem is that people with ME have been perceived as mentally ill and therefore it's very difficult to get anyone to listen. Most people with ME dont have the energy to go on and on presenting the same case. All credit to those who have managed to repeat the criticisms over and over and to do so with politeness and great patience in the face of extreme provocation.

However the latest criticisms are still from what will be seen in the UK as a minor university. It needs something like the UK's Royal Statistical Society to have much weight with the Lancet.

I think I am probably less hard on Wessely et al than many but I have never in any way supported the results coming out of the PACE trial. Quite the opposite in fact. It was a mess from the beginning. Please check out these blogs.
Hopefully the Royal Statistical Society will get on the PACE case.
 
Last edited:

Cort

Founder of Health Rising and Phoenix Rising
Staff member
Thanks. It's actually a huge subject and there's more to it than in this paper.....The PACE had a lot of problems...
Thanks for tying all these ends together, Cort. I didn't have the cognitive function to make sense of this before. Well done.
 

Cort

Founder of Health Rising and Phoenix Rising
Staff member

Get Our Free ME/CFS and FM Blog!



Forum Tips

Support Our Work

DO IT MONTHLY

HEALTH RISING IS NOT A 501 (c) 3 NON-PROFIT

Shopping on Amazon.com For HR

Latest Resources

Top