Pragmatic Clinical Trials

Join Dr. Heather Allore, from Yale School of Medicine and Dr. David Reuben, from University of California, Los Angeles, as they delineate between explanatory and pragmatic trials and discuss issues to consider when conducting pragmatic trials in health care systems, including commitment to supporting clinical personnel, study infrastructure, and fidelity. 

To view a transcript click here then select the transcript tab.

Transcript

Heather Allore, PhD, MS, MA: Hello. I am Heather Allore, Professor of Internal Medicine in the section of Geriatrics at the Yale School of Medicine, as well as Professor of Biostatistics at the Yale School of Public Health. I'm here with Dr. David Ruben, the Archstone Foundation Chair and Professor at the David Geffen School of Medicine at UCLA.

We will be discussing key points from the AGS/ AGING LEARNING Collaborative: clinical and translational learning curriculum that focuses on the science of multiple chronic conditions. As co-chair of the research design module, Dr. Rubin contributed a presentation on Pragmatic Clinical Trials. 

Thank you, Dr. Rubin, for joining me, and I wonder if you'd tell us a little about yourself. 

David Reuben, MD: Oh, sure. I'm Dave Rubin. I am a, geriatrician and, mostly an implementation science researcher. I've had a lot of [00:01:00] experience with both traditional clinical trials and, more recently pragmatic trials. They are, a lot of fun and keep me up at night.

Heather Allore, PhD, MS, MA: Well, that's great. 

Maybe you can start off the conversation by telling us about some of your, your feelings on the differences between traditional and pragmatic trials, especially in the context of multiple chronic conditions. 

David Reuben, MD: Sure. It turns out everybody thinks that pragmatic trials are, are recent addition to research design.

But, in fact, some of the articles that I used in preparing this go back to 1967. So they, they've been around a long time and, in part because, they've gotten a lot more attention of late. Now, the best way to think about, traditional or the use of term explanatory clinical trials and pragmatic trials relates to, you'd like to have both.

You'd [00:02:00] like to have a, a traditional trial to really determine efficacy and a pragmatic trial to determine effectiveness. So they would be sequential in a sense. So the traditional trials confirm a physiological or a clinical hypothesis. This is kind of under the best conditions possible. Whereas a pragmatic trial informs a clinical or policy decision by providing evidence for the adoption of the intervention into real world clinical practice.

So in a sense, pragmatic trials, ideally [if] budgets were not, constrained, would always follow an explanatory trial because you wouldn't do a trial, something that didn't work under ideal conditions. 

Heather Allore, PhD, MS, MA: So we know that there have been many successful explanatory trials that never really get incorporated into clinical practice or healthcare systems.

Do you see this as a failure of moving to [00:03:00] this implementation or pragmatic clinical trial stage? 

David Reuben, MD: Well, the pragmatic clinical trials are easier to go from trial phase into actually incorporating in practice, because they're, they're conducted in real world settings with real world clinicians and real world staff.

So the, the leap from a pragmatic trial into clinical practice should be less than the leap from an explanatory trial into clinical. 

Heather Allore, PhD, MS, MA: Well, that's kind of a nice segue into part of the planning that might go into a pragmatic clinical trial. In your presentation, you provided some information on the pragmatic explanatory continuum indicator summary, typically known as PRECIS-2.

And it's been a helpful tool for planning pragmatic clinical trials, but the diagram and criteria may not be familiar [00:04:00] to our listeners. Could you provide maybe some examples of how you used it when designing one of your own pragmatic clinical trials? 

David Reuben, MD: Sure. The PRECIS-2 construct is really, nine components around a wheel.

These include: eligibility, who's selected for the trial; recruitment, how they're recruited into the trial; setting ,where the trial's being done; organization, the expertise and resources needed to develop and deliver the intervention; flexibility in how the intervention is delivered; flexibility in adherence to make sure that participants adhere to the intervention; follow up, how closely primary outcomes- how relevant they are to participants; and then the primary analysis, whether, all data are included. 

And this wheel goes from, one to five, five being the most pragmatic. [00:05:00] And, for each of these, you consider for the individual trial, how far out you can be on that, number five. And to say that any trial is entirely pragmatic or entirely explanatory is probably overstating the case. They all have some more or less pragmatic features to them. 

And, and you have to make some decisions. I have my own kind of, rubric in terms of what I think is inviolate and I think, where, where I think, you know, in real world settings is truly a, and, and using real world staff is more or less an inviolate principle of the PRECIS-2 to try to adhere to as much as possible.

There are other things, such as adherence and fidelity that I think, I tend to dip more towards the traditional trial because of the, the simple fact is if people aren't receiving the intervention, you're not really testing the intervention.[00:06:00] 

Heather Allore, PhD, MS, MA: Well, I think that's very helpful. And I wonder, you know, since our theme is Multiple Chronic Conditions, maybe we can touch on, eligibility and recruitment.

Because in your presentation on STRIDE, it was really quite evident how potential participants should have, one or more of the risk factors. And so this fits very nicely into, the multiple chronic condition theme. Can you talk about how people might be selecting those if they were looking at different sorts of, you know, we know so many explanatory trials use chronic conditions as exclusion criteria. Can you give us your thoughts on, on how one might do this? 

David Reuben, MD: Sure. In contrast to traditional trials, you want the recruitment population in pragmatic trials to be messy. You, you [00:07:00] don't want to exclude people, for comorbidity as much as possible. 

Now, obviously if you're doing a study that requires people to fill out questionnaires or something like that and they have dementia, it, it may not work. Or if they speak, a language that is, very seldom used, and, you don't have translators for that. You know, that's another example of exclusion. But in, in general, pragmatic trials try to include as many people as possible, particularly those who would be eligible for the intervention in real world settings and after the, the evidence is in. For example, in, the other trial that I use as a, example in the slideshow is the D-Care Study. And the only people - virtually the, the major criteria for being excluded is if that you wouldn't have received the intervention anyhow because you're living in a nursing home or you're on hospice - that [00:08:00] you wouldn't be eligible for the intervention anyhow, even in practice settings.

Heather Allore, PhD, MS, MA: And so would you find, that it is maybe useful to see, say, for adherence if there were certain combinations of chronic conditions that may have made it harder for individuals to adhere to the intervention or be able to be measured on the primary outcome? 

David Reuben, MD: Well, the question that's a great question, and the question is, how much of that is the planning going into the study and how much of it is the post hoc analysis?

In other words, you, you certainly wanna exclude people who couldn't you, you know, in, who are going into the study who couldn't complete the outcome measures. But on the other hand, you might, once they're included and once they receive the intervention and the study outcomes are measured, you might look at, number one: how well the different subgroups performed in terms of the efficacy of the intervention. And [00:09:00] number two: how well they performed in terms of leading measures. 

So, that's a good question is, is how much of this do you do upfront as exclusionary criteria and how much in the backend as post hoc analysis? 

Heather Allore, PhD, MS, MA: You know, I think one of the points you touched on in the, the PRECIS is that, flexibility in delivery. And since you've, run some very large trials (STRIDE and D-CARE among them), where you were using cluster designs, there must have been, some variation in the way the pragmatic trial was fitting into care delivery, across all those different sort of, sites and settings.

Can you say a few words about that? You know, often with explanatory trials, they want such tight monitoring. I think this may be a, a real difference in the way people [00:10:00] conceive of a pragmatic clinical trial.

David Reuben, MD: And it's attention. It's always attention, it's how much wiggle room do you, do you give in in these trials?

We, we try to give as much wiggle room as possible, realizing that,all politics and all cultures are local. But on the other hand, insisting that the core principles of the intervention are maintained. That's actually a, a good challenge for investigators who have an intervention that they want to test: are what are the core principles of that intervention? If you don't have them, then it- the intervention isn't the intervention. 

Heather Allore, PhD, MS, MA: I think that's very helpful for people in the planning stage. And that kind of leads to then the elements of follow up in the primary outcome. You know, some trials, the pragmatic trials are trying to capture primary outcomes through [00:11:00] electronic health records or standardly collected data in nursing homes or other settings. And so that follow up interval is based on when those visits would occur. 

What has been your experience and do you have any advice on follow up and, and if you should have standardly recorded outcomes to, you know, the regular record system or add on for, that's a trial specific outcome. 

David Reuben, MD: Oh yes. Well, this is one of the vulnerabilities of pragmatic trials.

If you're, you're looking at your outcomes, you want to make sure that they're collected correctly. And indeed, one of the, one of the liabilities of using non-research staff, including clinical, staff to collect and enter data, say, data that are in the electronic health record, is that they've had no [00:12:00] training and, they may be, may be making things up. You, you never know. So this is one of the areas where I tend to just personally come down on the side of having that much more standardized. And in, in, in both of, of these studies, we relied on, data that were collected specifically for the study, rather than just trying to abstract data from, from the medical records. 

It depends upon what your outcomes are. For some, some you don't even need the electronic health record. You can just get it from claims data. But it really depends if you're, if you're measuring things like, functional status, those things are not captured well in electronic health records. So, or anything that is a patient reported outcome measure, PROM. Those are things you're gonna have to really, collect yourself and not rely on existing data. 

It's kind of a trap that people think, geez, you know, every, we're recording so much these days that all we have to do [00:13:00] is dig into the electronic health record. But what's collected and how it's collected and may not be sufficient.

Heather Allore, PhD, MS, MA: I think that's very true. Do you want to talk, maybe just give a little insights, you know, over the length of a trial, when, when working with older adults, and especially those with multiple chronic conditions, they may have met eligibility to, participate in the trial. They're, recruited and either they're individually consented or they follow whatever the consent procedure is for that trial. But over the course of the trial, they may be developing, additional geriatric syndromes or other chronic conditions. 

Can you say anything about your experience? Essentially, the population you recruit looks actually different in their profiles by the time you're getting that primary outcome. [00:14:00] 

David Reuben, MD: Yeah, well, we're seeing a lot of that right now in the, D-Care Study. They look very different because they're dead. we're losing about 12 to 15% per year, in the population and in- indeed, you know, these, particularly with,multiple chronic conditions, they have high, mortality and high morbidity, and they, they drop out because of other illnesses, et cetera.

And, one of the things that, I learned from the statisticians is that this is a, a good reason for collecting data and multiple time points. So that you have data, that can be carried forward, so that you have data on, on pretty much everyone, unless they, they drop out or die very early. But, these kind of longitudinal assessments of, of the outcomes can be very, very helpful.

But yeah, this happens. You're, you're dealing with people who have multiple chronic conditions. 

Heather Allore, PhD, MS, MA: I think that, you know, there, you're touching on that primary analysis and, to what extent all the data is [00:15:00] included. Are there scenarios where you might find that you wouldn't want to include all the data?

Can you give us any examples where, where that might be the case? 

David Reuben, MD: I would think only if you're masochistic. You know, it's, you want to use every bit of data that you can, that as long as it's valid data, you know, it, it may be that some data aren't, are no longer valid because you have a different reporter and that might not be something that was a self-report that a proxy may be giving you the answers to, and the data may not be valid because you're getting a different responder answering the questions. But if the data are valid, I would recommend using as much as you can with it. Otherwise, you tend to have high dropout rates and a lot of missing data. 

Heather Allore, PhD, MS, MA: One of the things that I've been, working with, some people on recently is trials that were being run pre-COVID, during COVID, [00:16:00] and in this period where COVID is less severe but still existent. Even looking at, you have the same sort of treatment effect in almost these three periods. So sometimes there's other analytic ways. If there was, some large shift that may be, contributing to the change in the effectiveness of the intervention, especially in trialswhere elements may have been- had to be dropped, such as home visits or a number of things that were excluded, say, during COVID or, or other situations.

So, so I think working with your, biostatistical team can generally address all these unusual occurrences. 

David Reuben, MD: Yeah, and COVID is a very interesting example there because it can affect two things. One is how the intervention is delivered. And the second is how responses are collected. And for [00:17:00] each of those, you can have people in a trial that the entire time they were in the trial was pre-COVID so you're operating in those conditions. You have people who recruited after COVID started. And they got all of their care and all of their outcome measures assessed after COVID. And then to to boot you've got p- some people who are enrolled pre-COVID and continue to receive the intervention post-COVID and the whatever changes there were in data collection. A lot of the data collection in COVID times has moved from in-person to virtual and you can't just necessarily assume that they're identical. Some of the questions, particularly in multimorbidity, you know, for example, for cognitive testing, if you're doing cognitive testing, it is different over the phone than it is in person. 

Heather Allore, PhD, MS, MA: As well as functional testing?

David Reuben, MD: Correct, yes. 

Heather Allore, PhD, MS, MA: So, although there's no perfect [00:18:00] pragmatic clinical trial, as as you've discussed here and in your presentation summary, could you share with us some of the advantages you found using a pragmatic approach in STRIDE and D-CARE?

David Reuben, MD: Some of the advantages, once again in real world settings, these are are people who, if the trial ended and you could magically have the results the next day, and they were glowingly positive, the the funding source for these people just moves from the research grant to the, health system. So it really sets up things for implementation in real world practice if the study was conducted in, in real world setting.

The second is fewer exclusionary criteria, so, you don't have to kind of be searching for the, the right person for the study. Pretty much all comers . 

And, I've also found that at least in the studies that we have done, the institutions, they, they've had to contribute some [00:19:00] money towards implementing the pragmatic trial. Whether that's a share of costs for the clinical personnel who are involved or space or support for them. And I think that buy-in is really important. I think it, it's hard to get, but once a health system, or a practice is, has bought into saying, you know, we're part of this, they're willing to commit resources, and I think that's very valuable. It, it's also much more integrated into the fabric of what they're supposed to do. Some of the sites where we've done these pragmatic trials are not particularly academic and they've been committed as well. 

Heather Allore, PhD, MS, MA: Yeah. I think that really fits into the organization aspect of the PRECIS-2, diagram. And I think we can forget at times that the healthcare system and administrators of them are stakeholders, especially when looking at pragmatic [00:20:00] clinical trials.

Could you say a couple words about how you've interacted with, organizational stakeholders to get some buy-in? 

David Reuben, MD: You have to have a mole, somebody on the inside. For the investigator coming in from outside, you're just another stranger who's asking them for something. And, I, I've been incredibly fortunate in that the people who, who I've known, who I've recruited to be site principal investigators have been phenomenal.

They have been incredibly integrated and respected into their, not only their academic institutions, but also the clinical operations. And they are the ones who really carry the water. They have been so great and they're the people who, the, people in the C-suite or the administrators who are gonna have to commit resources, see every day.

So in many respects, that is a local phenomenon and that getting, [00:21:00] local people who are really well- respected in the institution is kind of one of the most important things that, a principal investigator can do. 

Heather Allore, PhD, MS, MA: Well, I think this has been really helpful. In closing, is there any advice you could share with listeners who'd like to design a pragmatic clinical trial for persons with multiple chronic conditions?

David Reuben, MD: O ther than starting on antidepressants, early? 

Heather Allore, PhD, MS, MA: Not anti-anxiety medications? 

David Reuben, MD: Tho -those too, you maybe get a little of both. No, I gotta tell you, they are a lot of fun. They're a lot of fun, but they are big. And the commitment that you need to make as a, as a principal investigator is large because there will be, even perhaps more so than in the case of traditional trials, unforeseen things that'll happen.

It's interesting in, in, several of the studies that the ownership of the clinical operation changed. And the, [00:22:00] priorities might be different. So there are gonna be the, the daily headaches, the daily worries. But on the other hand, when you see healthcare systems changing in response to an intervention that you have helped design, it's, it's impressive.

It's impressive and, and it's fun. And, the, the big surprise is at the end of the rainbow, when you finish the study. 

The thing you wanna be able to do is to say, at the end of the day, we've given this intervention a fair trial. 

Heather Allore, PhD, MS, MA: I think that's wonderful. And it's really, I think what all researchers want is to really have their, their interventions used to benefit people.

And this is, I think, one of the final steps in that, move towards implementing throughout healthcare systems that are really the way practices get embedded. 

David Reuben, MD: Agree 100%. You summed it up very nicely. 

Heather Allore, PhD, MS, MA: Well, thank you very much, Dr. Rubin, and I think [00:23:00] that'll wrap up our section on pragmatic clinical trials.

Thank you. 

David Reuben, MD: Thank you.