Evidence Generation for Complex Older Adults

Join Karen Bandeen-Roche, PhD, Johns Hopkins Bloomberg School of Public Health, and Ravi Varadhan, PhD, John Hopkins University, as they discuss the challenges and limitations of randomized clinical trials, and observational data for providing evidence relevant to older adults with multimorbidity. They review a cross-design synthetic approach combining trials and real-world data to generate evidence for older adults with multimorbidity.

To view a transcript click here then select the transcript tab.

Transcript

Copy Transcript Copied

Karen Bandeen-Roche, PhD: Well, welcome to another podcast in the AGS /AGING LEARNING Collaborative series. Today, it's my great pleasure to be here with Ravi Varadhan who's going to speak with us about Evidence Generation for Complex Older Adults. Ravi is a professor in the Quantitative Sciences Division at the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins University, but also a very long time researcher on aging and someone with whom it's been a great privilege for me to be a colleague.

And so Ravi, welcome today. It's great to have you. 

Ravi Varadhan, PhD: Thank you very much, Karen, for the kind introduction. 

Karen Bandeen-Roche, PhD: Well, so, you know, you've prepared this really lovely module for the series. And one of the things you do at the very beginning is to motivate what you call a fundamental limitation of the clinical trial paradigm for treating patients.

And indeed, you provided this great [01:00] quote from David Mann, which reads "the paradox of the clinical trial is that it is the best way to assess whether an intervention works, but it is arguably the worst way to assess who benefits from it." And so I'm wondering if you can just elaborate on that for us a bit.

Ravi Varadhan, PhD: Yes. So this quote is taken from a Lancet article in 1999. And I don't know if you're familiar or not, but Lancet had a series of articles on treating individual patients. So, that was led by Peter Rothwell, who's a well known internal medicine specialist from the UK. And so, the series of articles, which was later put into a monograph on treating the individual patients, sort of highlighted this issue of the paradox in clinical trials, which is the gold standard for providing evidence on whether something works. [02:00] Because way back, one of the statistical pioneers Ronald Fisher came up with this idea that randomization is the best way to evaluate whether something works or not. And then starting with Jonas Salk's, polio vaccine, there was a large randomized trial to assess the efficacy of polio vaccine. So, ever since, randomized trials have become the way to evaluate whether something works. 

But then practitioners who are treating individual patients started to recognize the fundamental limitations of randomized trials. Because for one, the randomized trials ended up recruiting a very selective population that was not necessarily representing the population that they were treating in the clinic.

And so there was always this question of, does the trial evidence generalize to the population that's being cared [03:00] for? So that was sort of the initial question, the generalizability of randomized trials. But then later on, more subtle questions also came up with randomized trials, such as even if this trial sample is representative of the larger clinical population, is the treatment effect relatively homogenous? All right, are there subgroups of patients whose treatment effect differs substantially from the average treatment effect? 

So these two fundamental issues of generalizability and heterogeneity sort of really made the practitioners question the relevance of randomized clinical trial evidence for their individual clinical practices.

So this was nicely articulated in Peter Rothwell's series. And again, David Mann's quote sort of captures that very succinctly. 

Karen Bandeen-Roche, PhD: You've also argued that the limitations you've [04:00] just talked about are particularly serious for complex older adults, and I think the generalizability limitation will be familiar to many on this podcast.

I think it's very widely known that representation of older adults in trials is is often severely wanting. But the heterogeneity limitation, I would love to hear you talk a little bit more about that, why that may be particularly challenging for complex older adults. 

Ravi Varadhan, PhD: Yes. So one could make the argument that, that generalizability is not necessarily a big issue, as long as the treatment effect is homogenous, right?

You know, if the treatment effect doesn't differ substantially among different subgroups of patients, then the fact that you did not include a particular subgroup is not a big issue. Right? So you have to combine the generalizability along with heterogeneity of treatment effects.

So Mary [05:00] Tinetti wrote a nice article that discussed why the treatment effects that are found in healthy older adults may not generalize to complex older adults. She gave a variety of reasons, which I had highlighted in one of my slides. So, she basically argues that one reason could be that the treatment effect itself might not be the same because of physiological changes due to multiple chronic conditions that complex older adults have. And the other could be that presence of one disease may impact the effect of treatment on the primary disease that you're prescribing the treatment for.

And also, the complex older adults are more likely to experience serious adverse effects of medications and therefore they may not continue with their treatment. In fact, there is more likely to be noncompliance with the treatment and [06:00] therefore the treatment effect shown in clinical trials may not pan out for the complex older adults.

Karen Bandeen-Roche, PhD: I think that's really clearly articulated. And so, you know, given these severe challenges, just at a high level, you know, how can we hope to generate appropriate evidence for complex older adults? 

Ravi Varadhan, PhD: Yes, I guess that's the most important question. Because a lot of people know about the limitations. So we want to talk about the solutions, right?

So I highlighted some approaches in my slide number 10. I talk about the primary approach, which is a direct approach. So there are two primary approaches. One direct approach would be to make sure that the trial is representative of the population that's being treated. While it's easier said, but it's very difficult to do to get representative samples.

And even if you get a representative sample, [07:00] during the conduct of the study or during the consenting process, people may not consent to participate. And even if they participate, there may be challenges in following up and collecting the relevant data on complex older adults. So the representative clinical trial idea, while it's appealing, it's difficult to do.

But what if we did a focus study on, on the population of interest, right? The targeted study that is designed specially to take into consideration the challenges of working with complex older adults, that's probably a more realistic or a better approach where we target the study to get evidence from complex older adults.

Of course, that may also be challenging because we really have to define what we mean by complex older adults in order to have a sharp eligibility criteria for participation in such trials. So those are the [08:00] two direct approaches. 

Karen Bandeen-Roche, PhD: All right, well, why don't we take on, you know, the second of the things you just talked about first and give people a little bit more of an idea of how you would actually leverage the strategy that you foresaw at a high level, which is to use propensity scores to best leverage observational data.

Basically, how does that work? You know, both the mechanics of how it works and then how it seems to work in practice. 

Ravi Varadhan, PhD: Yes. The observational data, or what is recently becoming popularly known as the "real world data" provides a very attractive source for generating evidence on complex older adults because this is actually getting data from in a representative condition where patients are being treated.

And so while it is very appealing to get the evidence from this source, it [09:00] also presents a lot of challenges. The main challenge is that the treatments are not assigned in a randomized fashion. So the patients who are getting the treatment are more likely to differ from those patients who are not getting the treatment for a variety of reasons, and those could be associated with their prognostic factors.

In other words, people who are taking the treatments may be either better or worse compared to those who are not taking the treatment. So if we ignore that fact and estimate the treatment effect in a naive fashion by comparing the two groups, you may not get a valid estimate of the treatment effect. And therefore, we need to account for the confounding variables that present in the real world database. 

It is possible to eliminate confounding of factors that we know are confounders and those that are actually [10:00] available in our database. And the most popular way to do that is through what is known as propensity score methods.

So in the propensity score method, what you do is you fit a statistical model, it's a regression approach, where you regress all the patient's characteristics that are available. Potentially all the prognostic variables should be in that model, as well as other variables that may not be prognostic for the outcome, but they may affect the choice of treatment selection. 

So all those variables are put into this regression model. And these days we have really powerful tools, such as machine learning tools, right? And so you could put a lot of variables into the model, and these machine learning tools can generate really powerful approaches to predict treatment assignment based on these variables.

And then we take the predicted value for treatment assignments. And [11:00] we can follow a variety of approaches. One simple approach would be to divide people into strata of propensity score. Like, you can divide them into tertiles or partiles or quintiles. Then within each strata of propensity scores, the patients are assumed to be comparable.

Those who are treated and those who are not treated within that strata, they are comparable, so we can estimate the treatment effects within those strata, and then we can combine them, right? That's one approach.

Another approach is to match people at the individual level. So I will match someone who is receiving the treatment to someone who is not receiving the treatment. But they both have very close or almost the same propensity scores, and then I can match people like this, and I create matched pairs. Or I can match one person who is treated to more than one person who is not treated, right? I create these matched people, and [12:00] then I estimate the treatment effect within the match groups. Those are the two common approaches to estimating an unconfounded treatment effect using propensity score approach.

However, a major caveat is that it's only unconfounded with regards to the characteristics that we know and we have measured in our data set. So if there are important prognostic variables or things that are not captured in the database, and the propensity score cannot address that issue it's called unmeasured confounding. So it's always a big issue in observational data. 

People don't really trust the treatment effects. And there are many examples where the observational treatment effect differs from randomized treatment effect because of suspicion that unmeasured confounders are potentially quite important and they were not captured in the observational databases.[13:00] 

Karen Bandeen-Roche, PhD: Well, two brief follow ups, then. You know, one has to do with the propensity score. It sounds like such a powerful idea to compare, say, two individuals who both were likely to have received the treatment, but one of whom actually did, and one of whom actually didn't. Similarly, two people not likely to have received the treatment, but one who did and one who didn't.

And my impression is that strategy has worked. pretty well in practice recognizing the limitations. Is that also, you know, your opinion on the matter? 

Ravi Varadhan, PhD: Correct. 

Propensity score is by far the most popular way to address measured confounding. And it not only does that, but in the example that I show in my slides, it comes from bone marrow transplant, you would really see that, you know, analyzing registry data, which is a very important source of data for real world assessment of [14:00] evidence, the treated group and the untreated group generally tend to be very skewed in terms of the sample sizes.

Okay, so, so that generally gives an indication that there is potential for differences between the groups and, and, and that's the propensity score approach would be the first thing that will come to your mind. And so when you attempt a propensity score approach, not only it provides a way to address confounding, but it also reveals factors on how these two groups differ. 

So what types of people are actually getting the treatment, one treatment versus another treatment? That's a very important piece of knowledge that you can gain from propensity score model itself, apart from its ability to control for confounding, right? 

Karen Bandeen-Roche, PhD: Absolutely. And I thought your example was lovely, you know. Both for that, but also for demonstrating how the unadjusted or naive estimates could really [15:00] differ from ones that, you know, appropriately take into account of propensity score, you know, potential differences in those who receive the treatment and those who don't.

So the other thing that you mentioned a minute ago had to do with unobserved confounding, which propensity scores cannot address. And so you did at least briefly raise one strategy in your presentation that might be able to advance in this area, which is to combine observational or real world evidence with randomized evidence seeking to leverage the strengths of both.

And so can you elaborate on that just a little bit? 

Ravi Varadhan, PhD: Yes, interestingly, this strategy was highlighted way back in the early 90s. There was a report that General Accounting Office put out in 1992, and I want to say that it was might have been led by statisticians such as [16:00] Don Rubin and others because it was very thoughtfully written report where they actually talk about the potential value of this approach, where suppose you, you go to something like meta analysis or systematic reviews.

Okay, that's a well known paradigm in evidence generation. But there, You systematically review or meta analyze studies of the same design. So they're all randomized studies, okay? So there, it has a lot of value, but then, this report highlighted the fact that rather than combining studies of the same type, why can't we combine studies of different types?

So they can actually complement each other. And a study that has a particular strength also has a particular weakness, whereas another study... which overcomes the weakness of the first design, but it has a limitation that is addressed by the first design. So in other words, they are very complementary in nature.

[17:00] And so you put them together, and if you have methods to combine them, then hopefully you can mitigate the weaknesses of both designs. and accentuate the strengths of both designs. So that's the idea behind cross design synthesis, which was nicely illustrated in that, in that report. And they actually give a very simple example, which I also highlighted in my talk.

It makes a very simplistic assumption, but it illustrates the main idea. You, you take that in the, in the real world database, or registry data, or electronic medical records, you have the whole representation. There is a fair number of complex older adults, and then there is a fair number of normal or healthier older adults. So now you can compare the treatment effects between the two groups in the real world data. 

 now when you go to the randomized trial, you only have the healthier older [18:00] adult population. You know, their treatment effect, but you don't know the treatment effect for the complex older adults because they're not there in the randomized clinical trials.

So in that setting, you have a two by two cell. I know three cells in that two by two. So there is one cell that is missing and I can extrapolate that treatment effect using some very simplistic assumption. Okay, so it just gives you a flavor of how the cross design synthesis would work. And of course, in actual implementation, you would take into account in a more sophisticated fashion the distribution of confounders, how they differ between the two groups, the healthier group and the complex older adult group, and you put it all into a global model and you figure it out. 

Karen Bandeen-Roche, PhD: I think one really nice thing about what you've just said also is that it envisions a little bit how one might begin to address heterogeneity generally.

You know, you [19:00] could imagine not just healthy and unhealthy, but you know, a number of groups whose difference in treatment, treatment, quote, unquote effects could be estimated in the observational data and then more broadly extrapolated either in the, the randomized data or, you know, creative designs in which you work really hard to be able to study these things in a stop set in a randomized setting really opens up all kinds of interesting directions for research, I think. 

Ravi Varadhan, PhD: Yes, yes, this is certainly one of the areas that is receiving a lot of attention now. Groups from Harvard and other places, they're publishing methodological results in Epidemiology and other journals. But this is going to have a lot of interesting developments in the near future. 

Karen Bandeen-Roche, PhD: Well, I only have one more question for you, but before I do, is there anything [20:00] you had hoped we would talk about for potential students of your module?

Ravi Varadhan, PhD: So one of the things would be, in addition to methodological ways to address this problem, as well as encouraging participation of complex older adults in clinical trials, I was also thinking that maybe some of the modern technological advances could help mitigate the situation to some extent, such as, you know, the obvious thing that comes to mind is, you know, wearable computing and wearable things where we can collect data from folks, not necessarily in a clinical setting, right?

They don't have to come to a clinical setting to provide the extensive data that is required for assessing treatment effects in randomized trials. So if we could minimize the burden on patients, you know, in terms of how they can provide the data, I think it can open up a lot of [21:00] doors for really getting representative sampling of people.

Because right now the challenge is that even if you enroll them into the trials, they drop out because it's very challenging for them to come and provide data on regular basis, you know, to assess treatment effects. So I'm hoping that the technological advances can help us collect data in minimally invasive less burdensome ways.

Karen Bandeen-Roche, PhD: That's a really wonderful insight, and it is the complement of what my final question was, which does have to do with representativeness of older adults in clinical trials. Absolutely, you've, you've just addressed what we can do on the participant end of things, but older adults really do prove to be, you know, quite faithful participants within their capability, and so there is still a real gap on the researcher side on, you know, [22:00] including older adults, particularly representatively. And do you have any thoughts on how we can address that side of the problem or advocate broadly in the field, you know, about the importance of addressing it? 

Ravi Varadhan, PhD: Certainly we can do the advocating part in terms of highlighting the importance of that. In terms of how to promote or actually go ahead and try to improve recruitment of older adults, maybe having registries of patients who are complex older adults such as, you know, frail older adults or older adults with delirium or incontinence. So maybe having registries of these patients, perhaps maybe allowing us to recruit from those registries into randomized trials might be one way to make sure that these trials have representative sampling.

And perhaps the funders can also make [23:00] it mandatory that these more complex groups need to be represented in an adequate fashion in the trials. So data safety monitoring boards or things of that nature, they can hold the study accountable for recruiting these people as well as following them through thoroughly.

So there's obviously there's a lot that that can be done. 

Karen Bandeen-Roche, PhD: Absolutely, and I'm very glad you said that given that the NIH really has made substantial efforts in the last few years to mandate broader inclusion. The FDA has beefed up its guidelines. And then the other nice thread in there is the important role of organizations in making this sort of research downhill, so to speak. You know not leaving it all to the individual researcher, but providing the infrastructure and the resources that are really needed to make it something that feasibly can be done. 

Ravi Varadhan, PhD: That's [24:00] great, actually. That reminds me about PCORI, for instance, Patient Centered Outcomes Research Institute. So if you were to apply to a grant through PCORI, they would ensure that you do all these things, actually.

You need to have patient advocates, you know, advocates for complex older adults. You have to make sure that the study design considers the advocacy groups for such patients. And so it becomes an integral part of the study proposal itself before it gets funded. So I think it's very important that the funders are very actively promoting these things.

Karen Bandeen-Roche, PhD: Ravi, it's been such a pleasure to speak with you today, and I thank you for your time. And I also thank you for the wonderful module you've prepared as a resource for the field. 

Ravi Varadhan, PhD: Thank you very much, Karen.