The Pace Trial
PACE Trial authors were moderate in their assertions about the PACE Trials effectiveness…but did even they go too far?
In March 2011, the outcome of the rather infamous PACE trial was published in Lancet. The trial sought to prove that behavioral therapies were both effective and cost-effective in treating chronic fatigue syndrome (ME/CFS). At over 8 million dollars in cost and years in development it was easily the most expensive trial ever for chronic fatigue syndrome. This was the study to rule them all; a successful trial would set UK treatment priorities on a CBT/GET course for the forseeable future.
The results were underwhelming to say the least but while the study authors were careful to note the ‘moderate’ benefits achieved, some of the media were less cautious, stating that an effective treatment had been found. A closer look at the study suggested that even the more moderate statements by the study authors may have overstated the results. The overly positive media reception and the authors statements prompted a group of ME/CFS patients with math and statistical backgrounds (plus some fantastic html skills (see below)) to take a closer look.
They labored over the PACE study for over a year to produce their report; a standard setter in rigor, scope and creativity from the patient community.
The PACE Trial lives on. The 2011 report was just the first analysis; in 2012 a study asserting the PACE trial proved CBT/GET was both effective and cost-effective was published, and a 2013 study asserting a 22% recovery rate for CBT/GET participants in the trial lead the authors to state
This study confirms that recovery from CFS is possible, and that CBT and GET are the therapies most likely to lead to recovery
A Tour de Force
The patients analysis of the study suggested that positive results were, intentionally or not, baked into the study design. If that’s so, it’s a bad sign for PACE, as its results were underwhelming to begin with. Even worse some of the same problems appear to be present in some of the standard tools used to assess patients well-being. If this is true then the assessments of many studies may be up for grabs.
One of the most troubling aspects of the PACE report was its lack of objective measures of functioning. The original trial proposal included actometers to measure physical activity levels but after they were dropped the authors relied on a 6 minute walk test. Be sure to check the often hilarious animation as the healthy controls and CFS patients race each other to see who walks farthest…..
As you read the blog be sure to check out the entertaining and educational animations – they’re a blast. The first one takes a look at the exercise component of the trial.
Setting the Pace….For Futility (?)
The authors came up with 10 conclusions…among them
- some scales designed for CFS and widely used in CBT/GET and other CFS studies have fundamental faults
- some of the statistical measures the PACE trial used erred on the side of showing improvement
- patients assigned to Specialist Medical Care actually showed a greater degree of improvement than patients given CBT/GET
- the statement that all ME/CFS patients can receive at least moderate benefits from these therapies is not borne out by the study results
- CBT/GET does appear to help a subset of patients but we don’t know which type that is
- The positive effects CBT/GET had on CFS in the PACE trial were about the same as these therapies do in disorders like heart disease and multiple sclerosis.
Their reasons for these conclusions are explained in a wonderful, down to earth manner. If at some points you get stuck (statistics are involved in some areas) simply go to the animations, which often have soundtracks and sit back and enjoy because the animations are simply astounding; using the latest html 5 technology, the authors are able to get across complex factors in often disarming fashion..
This analysis suggested the CBT/GET field itself has a ways to go, something a Cochrane Report alluded to in its overview of the field.
A Talk With Graham
One of the leaders of the PACE analysis effort, I asked Graham how it all got started and about what they found..
Can you talk about how and why this effort got started and how many people it involved?
Originally Bob and I were working on a project together analysing the PACE results and we published a blog that received a number of positive comments. While this was going on, a group of people in Phoenix Rising started what turned out to be an enormous, high-quality thread, looking in detail at the PACE study. Bob persuaded me to join Phoenix Rising, and I started to read through the PACE thread.
Many people there were discussing what to do with all the issues that they had found. Several people wrote papers or submitted letters for publication (e.g. Tom’s “harms” paper): there was a lot of discussion on how to put all these ideas together. I raised the subject of perhaps trying to take a different, more accessible approach alongside the conventional analyses. A number of people liked this idea, so we discussed it further and so formed a team to produce this response.
Not all the contributors have been from PR, the numbers have fluctuated according to energies, and health and other projects have also dictated how much each person could contribute, but overall I would say that around a dozen or more people have been involved.
There seemed to be lots of problems with the PACE study: you found that their use of averages instead of medians skewed the results upwards; the control group was not adequate; there was no attempt to assess functionality; the study design was poor. Have you looked at other CBT/GET studies and, if so, do you find similar problems?
I haven’t spent the same amount of time on other studies (this project has taken nearly a year), but I have looked at hundreds of different studies. Most have major problems with the sample size being small. Almost all of them seem to be similar in their reliance on subjective assessments, and completely fail to impress me.
“60- the New 75” or How to Change the Design of a Study Twice to Achieve the Results You Want (refrain from eating prior to viewing)
There are studies which compared the use of actometers/pedometers with subjective assessments and concluded that actometers were more reliable indicators of activity levels, but expensive. One trial, using actometers and questionnaires to assess the value of GET, found that the questionnaires suggested activity levels had improved, but the hard evidence from the actometers showed that activity levels had not changed. PACE was funded to include actometers, but decided to drop them. Apparently wearing a wristwatch type of fitting around the ankle was considered to be too great a demand on the patients.
It is pretty clear that the questionnaires can lead to falsely optimistic conclusions.
Your statistical analysis of a group of patients who tracked their wellness over a year suggested that some of the CFS patients are too sick for the scale to handle ie; since they’re already at the bottom of it, if they worsen it wont show up in the statistics. Again, given that these scales are used in other studies, is this kind of a field-wide problem?
So many people with ME/CFS score 3 (the maximum) on most of the questions. This clumping of patients’ scores at the most severe end of the scale must be a major problem, and clearly prevents many patients from recording when they feel even worse. That is probably why there is such a conflict between studies claiming that GET does no harm and surveys of patients which claim it does. The Chalder fatigue scale in particular is hardly used outside studies of ME/CFS, so it is difficult to make many comparisons with other illnesses, but the few that we can make puts the ME/CFS scores at the worst end.
Can you briefly explain how it is that someone could improve enough for the authors of the PACE trial to say they’ve recovered but they’re still ill enough to meet the criteria for CFS. (Did I get this right?).
Yes, you got it right. One can simultaneously qualify as severely fatigued and able to enter the study, and as “back to normal”. The basic reason is very simple – they changed the way that they measured the answers to the fatigue questionnaire. They also have a rather peculiar standard for “normal”. But it is very hard to explain properly in words. That is why we have so many graphics in the project (it’s much easier if you follow the graphics in the extra layers to conclusions 3 and 6 in particular).
How Turn A Small Gain into An ‘Effective Treatment’
The authors changed both the way that they scored the answers on the fatigue scale, and changed the criterion for “normal function”. Originally a patient was enrolled on the trial if they indicated that six or more of the eleven items on the questionnaire gave them difficulties. The target was to halve the number of problems that they had noted on the questionnaire, which seems a valid aim (though not necessarily what you or I would call a return to “normal”). After the trial had started, the authors changed the assessment to a scoring system, where each item was scored 0 to 3, and they decided upon a target of 18 (which itself is quite controversial – six out of eleven items could still be scoring 3 points each – maximum difficulties). According to our survey, a more accurate match to halving the original number of problem areas would have been a target of 12 on the new marking scheme. (Remember that the aim is to reduce the fatigue score).
According to your analysis, patients actually improved more at the Specialist Medical Clinics than they did with the CBT/GET practitioners. To put it another way, if the authors’ original focus had been to prove that Specialist Medical Clinics were more effective at treating ME/CFS, they would have had ample evidence to do that. In fact, the improvement with CBT/GET was so small that it could have been explained by the natural swings of the illness that most people experience. I don’t remember reading anything about this. How could this have been missed?
I don’t think it has been missed at all, but most of the submissions in reply to the PACE trial have been very lengthy and complex. It is easy to show what is happening with the right graphics (although it takes a while to get that right!), but much harder to explain clearly in words.
It would be wrong to believe that the Specialist Medical Clinics were effective at improving patients’ health. The improvements were tiny: it’s just that the improvements that GET or CBT added on were even smaller. But the fact that 40% of the people sent to the Newcastle ME centre were found not to have ME/CFS hammers home the need for people to be assessed by specialists, particularly the severely ill who are lucky to even have their GP visit them.
Also, I don’t believe that there is any evidence out there about the extent of the natural variation in fatigue levels for people with ME. We could be a first!
You believe that CBT/GET should not be labeled as a treatment or therapy but more as a support system that provides help with sleep, pain control and other areas. Why do you believe this?
It is more than that. A couple of years ago I fell over and broke my hip, and I hadn’t even had a drink! While it was healing, I followed all the proper advice about exercises etc. and when it finally healed, I had no need for physiotherapy. But of course there are some people who would be unsettled by breaking their hip, and would be reluctant to engage in the full range of stretching and moving while the hip was healing. They may also suffer from a lack of confidence afterwards in repeating whatever it was that they were doing when they broke their hip in the first place. But a broken hip is purely a physical problem. Physiotherapy may be needed to help people come to terms with it and regain their confidence.
Cha, Cha, Changes Galore…Plus A Missing Bottom: Conclusions and Questions
I see no reason whatsoever to think that ME/CFS is any different from a broken hip, and what people need is time to adjust to the illness, support in learning to make the most of things, and education and experience in recognizing the warning signs before doing too much. One study found that people who were members of an ME/CFS support group showed less improvement when they attended an ME clinic than those who did not. The study decided that ME/CFS support groups have a negative influence. But of course, the sensible conclusion would have been that ME/CFS support groups start patients down that path of adjusting and learning.
In fact, what I would like to see is a trial that applies CBT and GET to a random group of healthy adults. I have the feeling that they would show greater average increases in stamina and fitness than shown by the patients in the ME trial. What would that tell us?
________________________________________________
Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial.White PD, Goldsmith KA, Johnson AL, Potts L, Walwyn R, DeCesare JC, Baber HL, Burgess M, Clark LV, Cox DL, Bavinton J, Angus BJ, Murphy G, Murphy M, O’Dowd H, Wilks D, McCrone P, Chalder T, Sharpe M; PACE trial management group. Lancet. 2011 Mar 5;377(9768):823-36. doi: 10.1016/S0140-6736(11)60096-2. Epub 2011 Feb 18.
PLoS One. 2012;7(8):e40808. doi: 10.1371/journal.pone.0040808. Epub 2012 Aug 1.Adaptive pacing, cognitive behaviour therapy, graded exercise, and specialist medical care for chronic fatigue syndrome: a cost-effectiveness analysis. McCrone P, Sharpe M, Chalder T, Knapp M, Johnson AL, Goldsmith KA, White PD.
Psychol Med. 2013 Jan 31:1-9. [Epub ahead of print] Recovery from chronic fatigue syndrome after treatments given in the PACE trial. White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M; PACE Trial Management Group.
Very effective use of animations and You tube to simplify a medical paper and highlight the importance of method and measurement in research design.
Isn’t that something…these guys; I think it was Graham’s son – who has ME/CFS – and perhaps others who did those…If you go to the paper you’ll see further html advances – it’s really something.
Terrific discussion and videos. Thanks, Graham (love your voice and nice dog!), Bob, and others who worked on this. Thanks, Cort, for making this available.
Thanks, Merry and others. Your support is greatly appreciated.
We wouldn’t have finished the project if it hadn’t been for Janelle’s tireless work in producing content, improving legibility, and raising the scientific standard, so she deserves an enormous public cheer. And also for my son who had the patience of a saint improving the layouts, inserting graphics, and in creating or improving my efforts at animations. The amount of work that the others in Phoenix Rising put in the thread, whose ideas we used, who advised us on content, and who are working hard still to refute more recent claims, is amazing; many of them preferred to remain in the background, but they know how grateful I am to them.
We all have ME: that’s why it took us a year to complete it.
It was an honour to be part of such a great team.
Also Alex Young deserves a particular mention as he wrote some of the content as well. Of course there were a lot of people involved in the project, and it was and is an honor working collaboratively with such an amazing team.
Janelle
Thanks Cort,
It is good to see patients “undressing” the lies of PACE and you putting it together.
There is little doubt that the Wessely school psychiatrists at King´s College is producing “made to order” research aimed at psychosomatizing large groups of patients with serious and chronic organic diseases, labelling them MUPS. Please check out this on Wikipedia:
http://en.wikipedia.org/wiki/Medically_unexplained_physical_symptoms
You talk about Specialist Medical Clinics at one point – is that right? Can you please doublecheck? The term could be Specialist Medical Care (SMC).
Thanks for this blog.
Lots of complicated issues mentioned here, but there’s one bit I noticed which could need clarifying now that the new PACE recovery paper is out (it could also be worth mentioning that this interview took place prior to that paper’s release, assuming that it did).
re – “Can you briefly explain how it is that someone could improve enough for the authors of the PACE trial to say they’ve recovered but they’re still ill enough to meet the criteria for CFS. (Did I get this right?).
Yes, you got it right. One can simultaneously qualify as severely fatigued and able to enter the study, and as “back to normal”.
It is the criteria for those that PACE’s researchers classed as “back to normal” which overlaps with their criteria for CFS. Lots of other people (researchers and journalists) took “back to normal” to mean ‘recovered’, but PACE’s researchers did not use the term. They have now released a new paper with a hugely loosened criteria for recovery compared to the one that they laid out in their protocol, but also one which specifically requires thaty patients no-longer fulfil their criteria for recovery.
I’d suggest an edit to:
“Can you briefly explain how it is that someone could improve enough for the authors of the PACE trial to say they’[re back to normal], but they’re still ill enough to meet the criteria for CFS. (Did I get this right?).”
Maybe explaining that others used this to claim that patients were recovered?
The new PACE recovery paper uses a criteria which makes it much more difficult to talk about these things concisely.
Also, this point is possibly confusing:
“patients assigned to Specialist Medical Care actually showed a greater degree of improvement than patients given CBT/GET”
Patients assigned to SMC showed greater improvement than the additional improvement which occurred when CBT/GET was added to SMC. Patients who received SMC+CBT and SMC+GET reported greater improvements than those who received SMC alone. (All patients received SMC, some SMC+GET, some SMC+CBT, some SMC+APT). It could be that some sacrifices need to be made to prevent things from becoming to complicated, but as this was one of the main bullet points, I thought that I’d mention that it could be misinterpreted. Considering the fact that SMC was such a brief intervention, and the reported improvements from the addition of CBT and GET were such minor changes in questionnaire scores, it does look like the positive affects could just be a rather worthless placebo affect, particularly as the addition of CBT/GET did not lead to improvements in things like employment rates, or patients on disability benefits.
I’ve bored myself with all that! If I’d written it, this blog post would be unreadable.
Hi Esther,
The ‘normal range’ is used as one of the individual ‘recovery’ criteria in the new ‘recovery’ paper, so I don’t think it’s inaccurate to say that the authors consider the ‘normal range’ to equate to ‘recovery’. Actually, I think the new ‘recovery’ paper confirms what’s written in Cort’s blog.
“We chose domains for defining recovery on the basis of the previous literature and the measures available from the trial. The thresholds defining our criteria for recovery on each domain were based either on population normal ranges, case definitions or trial entry criteria.”
See Table 1. in the ‘recovery’ paper, and the ‘normal range’ is described as a ‘domain’ and ‘criteria’ for recovery.
In any case, an SF-36 PF score of 60 is used as a minimum threshold for the ‘trial recovery’, as far as I understand it.
That’s my interpretation anyway.
Bob
Hi Bob. Yeah, but they’ve now required that patients are no longer able to fulfil the criteria for CFS for their criteria for recovery. Explaining how that affects things is complicated, and could lead to confusion if patients were to claim elsewhere that PACE’s criteria for recovery allowed patients to get iller during the trial, yet still be classed as recovered by the end of it. They class a score of 60 as both normal and abnormal in the same paper, but there are other parts of the criteria for recovery to be accounted for to.
I love all these videos by Graham! I’ve seen them before but it’s always nice to see them again. They are perfect at breaking down complex issues into something most anyone can understand.
Thanks for posting them here.
Marjorie
I do too…There are video’s and there are video’s. They are well done in all phases of the game; visually and educationally…
Thanks so much to everyone involved in the analysis of the skulduggery that is the PACE trial. The graphics are great for simplifying the complexity. And, yes, their definition of recovery should win the Booker fiction prize.