Turning Research into Practice

Join Jeff Williamson, MD, MHS, Wake Forest University School of Medicine, and Kathryn Callahan, MD, MS, Wake Forest University School of Medicine, as they discuss health system coalition building and the different kinds of skillsets of team members in the Learning Healthcare System (LHS). They also discuss implementing initiatives in an LHS.

To view a transcript click here then select the transcript tab.

Transcript

Jeff Williamson, MD, MHS: Welcome again everyone to this podcast, sponsored by the American Geriatric Society, on Integrating Research into a Learning Healthcare System and making this a true learning health care system enterprise around our nation, and even around the world. And I am so pleased today to welcome Dr. Kathryn Callahan.

I've known Kate for many years and she's a leader in this field, she's. Had a National Institute on Aging Beeson Award focused in this area and other projects, Duke Endowment Award. And so Kate has a lot of experience, practical experience, I would say, in helping us think through these problems. 

And today we really want to talk to Kate about some of the lessons she's learned about the practical aspects of this journey. And it's a journey and we're all somewhere in the midst of it. 

So, Kate, tell us about yourself a little bit more. Tell us a little bit about your work. And then I have a few questions for you. 

Kathryn Callahan, MD, MS: [01:00] Absolutely. Well, thank you for having me. I'm always excited to talk to my good friend, Dr. Williamson. And I'm excited to be here today.

So, I have the wonderful pleasure of being at Wake Forest University School of Medicine. And it's been an exciting time as the relationship has been established with Atrium Health and now Advocate Health. We continue to look to how we can fully understand how solutions for older adults can be scalable. So thinking not only about how we can deliver high quality care to older adults, but my real passion is thinking about how can we do that at scale and think about how all older adults can benefit from geriatrics influenced care, even if they are not able to access some of the elite geriatricians among us.

So my focus in [02:00] particular has been on frailty and on the function metrics, how we can integrate those into the health system in a way that is easy to use for frontline clinicians and can inform shared clinical decision making for older adults. And specifically I have partnered with Dr Nick Piaski, who's a statistician to bring an adapted version of Ken Rockwood's frailty index into practice. The e(lectronic) frailty index has been live in the Wake Forest electronic health record since late 2019, and we have had the opportunity to see it used by the health system, both practically as well as in some research spaces. So excited to talk about any or all of that today. 

Jeff Williamson, MD, MHS: Hey, thank you I should say Kate is an associate professor here, and as you can tell we have worked together for a while. 

And I was just thinking Kate one question is is [03:00] at one point you were our fellowship director for geriatric medicine and I think there may be people listening to this podcast who are in that situation who love education are thinking, "Oh, I just don't know. I don't know if I could ever, you know, make, make that bridge to discovery and bringing new knowledge and discovery into the healthcare system." But I suspect that your time as a fellowship director really informed the things that you're doing now. And, I know I didn't prepare you for this question or many of the questions I'm going to ask you, but could you just off the top of your head talk a little bit about that transition and talk a little bit about how being a fellowship director really influenced your skill and really being a leader in creating an academic learning health care system. 

Kathryn Callahan, MD, MS: Absolutely. The first thing I would say is that I think that the scholarship of teaching of education is sometimes undervalued. [04:00] And I say that because I think it's undervalued even by educators ourselves. You know, I had a wonderful conversation at one point with Joe Ouslander when I was sort of figuring out my path. 

And as passionate as I was becoming about research, I didn't want to leave behind the education. And what I found is that there's a really, you know, big overlap between the world of implementation science and education, in that both are predominantly concerned with behavior change. You know, for education we think about it, you know, ultimately as, you know, you go back to the concept of the Kirkpatrick pyramid or the understanding at what level does an educational intervention impact. Often we'll see that an educational intervention will think about focusing on a learner's experience or on their knowledge but .The holy grail of what happens with education is that we want to see [05:00] this brought home to our older adult patients. We want to, we want to change and make the course of care better for these individuals.

That really is the focus ultimately of research too, right? It's just a different methodology of whether this is knowledge about a process of improving care. And that requires education, but as, as Joe told me, it also requires tools. And we need to put tools in the hands of our learners that can help them categorize patients to understand their function and their cognition to approach the process of their care in a way that is usable, that is feasible, that is respectful of their time and expertise.

So yes, I would say that part of what I learned as a fellowship director is how very strapped learners are for [06:00] time, that learners want to do the right thing for the patients in front of them. And another thing I learned is that the majority of people caring for the older adults whom we have dedicated our careers to are not geriatricians.

So thinking about the need for geriatrics education for me drove a lot of my passion for developing, adapting, and integrating, especially implementing tools that can help frontline clinicians meet those needs and that their education, their learning is supported by tools that make sense and can get the biggest bang for their energy.

Jeff Williamson, MD, MHS: So what I'm hearing you say is that educating fellows was a great joy. And yet even as you went through that process, you [07:00] began to see that you weren't going to be able to educate enough fellows to fill the void, and that we needed a better way to identify knowledge gaps and to get them into the healthcare system more broadly.

I think that's what I'm hearing you say. So... 

Kathryn Callahan, MD, MS: It is. I was wondering if I could share one very quick anecdote, Jeff. 

I was at a medical student research day, and I had worked with a student, this is some years ago, and I was approached by a student who had worked with me, who was planning on going into a non geriatrics, not really a non, you, IM based field and. You know, great to catch up and hear how he was doing. And he said, um, it told me what he was doing and sort of apologized. And I said, never apologize. We all, we all take care of older adults. I said, you know, what you've learned will stay with you. 

And he smiled and said, "I really hate to say this, Dr. Callahan, but I'm never going to use those tools. You know, I'm, I'm not going to do a lot of the assessments because they just take so much time." And he said, "I really [08:00] wanted to when I was doing my summer internships or acting internships. It just, there never seemed to be enough time." 

And he was bothered by that. He was troubled by that. And I was too, you know, it was, it was hard to wrap my head around that. And I started thinking, you know, is there a way, Jeff, you and I have talked for years about, you know, how do we help learners understand the heterogeneity of the aging population? 

And then, you know, one of the top questions we invariably get whenever, whenever I would be recruiting fellows is, so what does it mean to take care of a geriatric patient? What is a geriatric patient? And how do we know who is frail and who is not? How do we know who's vulnerable and who is not? In whom do we use these, these tests and these tools?

So yes, I think, you know, even as an educator, I was struck by a need for a [09:00] practical approach that incorporated a high standard of evidence, but that could have a broad enough reach. And then as I got more into the study of implementation science, realizing that that was, essentially a formula: you know, what is, what is the impact of a given tool or intervention multiplied by, by its reach. And, you know, recognizing that the tools, you know, outside medicine that have really transformed the world are often those that are able to be used most broadly.

Jeff Williamson, MD, MHS: So, let's fast forward a little bit. You've learned those lessons you just talked about, and you're beginning to say, here's the question I want to answer in order to increase knowledge in the health care system by caring for older adults. But sometimes I think what happens is, is that we get so wrapped up in the question, we forget to build that coalition at the same time.

So can you tell me at what point you began to build your health system coalition and kind of [10:00] how that happened? Because again, many researchers have great questions, even discover great things that end up in wonderful journals, but they never go anywhere because I think they forgot that coalition building.

Talk to us a little bit about that because that's such a key part of academic learning health system success. 

Kathryn Callahan, MD, MS: Oh, it's critical. And so much of it is approaching these questions with a sense of humility. I have been humbled many times. We'll continue to be humbled by how fast our healthcare system is, how much there is to know, and, you know, how very, very hard we're all working for the benefit of the people we care for. 

So I'll say that I think some of the coalition building happened while I was still predominantly an educator. I think that was one thing that was really important. You know, we at Wake Forest had had the joy and privilege of Reynolds Foundation funding, and that really started my relationship with [11:00] a lot of other specialists. Whether these are folks in hospital medicine or orthopedic surgery, urology, anesthesia, cardiology, you get the idea, but, you know, our focus had been on integrating geriatrics principles into specialties. And through that I formed a lot of really wonderful relationships.

And the other thing that I learned through that process is how, you know, we could look at the same patient and have a very different perspective on this individual, and that mine was neither better nor worse than the other. For, you know, physicians, it was simply, simply different. And starting to learn what sort of, just in time teaching, what sort of point of care testing, you know, the importance of speed and utility in understanding these concepts.

The other thing that I learned was that I knew one very tiny piece of the pie of how the process of caring [12:00] for patients happens and how very different that looks from my office space in the Sticht Center [on Aging and Rehabilitation] or on the acute care family unit versus trauma surgery, or the cath lab, or the endoscopy suite. And that that these new processes and these pain points, they often seemed to differ on the surface.

But I also started noticing that there were a lot of themes and that I got a lot more in terms of understanding what was going on for our patients and how I might be able to help, whether as a teacher or as a researcher, by starting out with just saying "tell me about what it's like to care for older adults and what worries you, what stresses you out about caring for older adults." 

Opening with that question rather than, "Hey, I got a great tool for you," um, got me way further. [13:00] One of the critical ideas- so one critical idea is knowing how you, you're an expert of your own domain, but not necessarily the rest of the systems that there's a lot to learn there. 

The second is as researchers, we often want to find the truth, right? We want to find what is the answer to this underlying question. I think when you start looking at health systems change, a lot of it is a little, it's more about solving a problem. 

So going to potential partners and saying, "Hey, what is it about caring for this population that keeps you up at night? What do you worry about? Where's a gap where we might be able to help?" And a lot of times, those would filter down into what we would recognize as the 4Ms or the 5Ms. People ultimately are worried about multimorbidity, you know, the multiple chronic conditions, that's the focus of this series. They're worried about [14:00] frailty, function, cognition. They're worried about doing harm, whether it's through medications or through procedures.

But I got a lot further in building relationship if the first thing I did was listen rather than going in and saying, "Hey, let me tell you about X, Y, Z." 

Jeff Williamson, MD, MHS: Right. 

Kathryn Callahan, MD, MS: So I would say the first few times that I would meet with people, I would listen. And then I would try to do something for them, you know, to help them understand or to share an article or see a patient or just, just build a relationship and start to understand the challenge from their perspective.

Jeff Williamson, MD, MHS: And it sounds like you started doing that before you had it all figured out on the answering the question side. You started listening, listening, and you're still listening to people I know. 

I have a couple other questions. Before that, I would like a just a short description. Many people understand the electronic frailty and so they've heard of it. I want a short description of that, very [15:00] short, because then I want to ask you a few questions related to, not the eFI, related to the team that helped you build it and how that team differs from say, let's say traditional research teams. 

Kathryn Callahan, MD, MS: The eFI is a passive digital marker for frailty. And what that means is it draws from multiple aging related deficits or, you know, as it's called in the frailty index that are gathered in the routine course of clinical care.

So just as in the traditional frailty index will be populated often by elements that are derived from the comprehensive geriatric assessment, these are breadcrumbs that are in the electronic health record. We have a universe of aging related deficits, you know, 50 plus, and the score is generated by seeing how many of these [16:00] diagnoses, lab studies, functional measures, etc. are present. That generates a proportion, and that proportion between 0 and 1 is the frailty score with higher scores meaning higher frailty. 

Jeff Williamson, MD, MHS: So the first question is, is in a learning health system endeavor, based on your experience now, what kind of team members, what's the, there are some differences in my perception, I'm more of a traditional researcher, but your team looks different than my team. And so, uh, you know, they have different skill sets. Sometimes they have the same skill set, but many are different. 

What sort of skill sets have you learned really help speed the process of developing a learning health system project from start to finish? 

Kathryn Callahan, MD, MS: The first question is often, you know, where in the health system are we considering the application of this tool? Whether it's an intervention, whether it's a risk assessment tool like the eFI. A [17:00] critical early partnership for for us was with population health. And this occurred during the COVID 19 pandemic when folks were really trying to figure out and understand how could we stratify risk when a lot of our usual tools were upended by the disruption in delivery that was brought on by the pandemic.

And I had the pleasure of working with folks like Keya Eaton and Jennifer Houlihan in our system through our education pathway and learned a lot through that time and then circled back when this opportunity came forward. And, I mean, Keya is a nurse and is, and is an administrator and has a fabulous perspective on the concept of value based care and also on the strategic deployment of staff.

And so a really wonderful understanding of what it means to deliver services. [18:00] And, you know, what are all of the moving parts in the health system? You know, truly a staggering amount of information. So someone like Keya, or we've also had a great partnership with Jennifer Houlihan. 

Jeff Williamson, MD, MHS: Why don't you say what Jennifer's role is in the organization?

Kathryn Callahan, MD, MS: Jennifer is a vice president, particularly focused on our value based contracts. And so, her understanding is really about how the health system payment models are changing, the structure of an accountable care organization, extradition accountability organization, and recognizing all of the different levers that are at play in the delivery of high quality care.

These are folks who have their ear to understanding reimbursement structures, how policy is changing, and the conversations that are happening around, around health policy. While I work with, of course, Nick is a biostatistician, we have a master's level statistician who works with us. Her focus, though, is almost exclusively [19:00] on the derivation of data and the cleaning of data that comes from the electronic health record.

And, in addition, we work with someone who has a background in IT. A few people who have backgrounds in IT, where their focus is, you know, not building models, you know, I'm thinking about what has been done for some, you know, traditional research studies where, say, you're building a database or something like that, critical work.

We're kind of trying to interface with the database that is already in existence and figuring out how to collaborate on both the clinical front, so that For instance, when, when we were putting together our Duke Endowment Foundation project, we needed to figure out how we could track specific outcomes.

Jen Gabbard masterfully in her work with advanced care planning. And what we worked for is really understanding is sitting down with Adam Moses and understanding, well, how does this happen? You know, how does a [20:00] note get put into Epic? What are the pathways? What are flow sheets? What are what are places where data reside?

Jeff Williamson, MD, MHS: You mentioned Adam Moses. Tell the audience what Adam Moses, what's his title? What does he do? 

Kathryn Callahan, MD, MS: Oh, sure. So Adam is, Adam works in information technology and really has a focus on how to extract data from the electronic health record, how to interpret data, those data, how to, in terms of their, their location and their optimal placement. He has some expertise in user centered design. So, where is it comfortable and optimal to put this information?

Jeff Williamson, MD, MHS: So, I know we've only got a couple minutes left, but I want to so congeal some of what you just said. 

What I hear you say is the, the traditional researcher is thinking about payment systems from the NIH. We're thinking about how we get our NIH dollars to move through. We're thinking about how do we construct data sets for research purposes. 

But your work has said, how do I [21:00] leverage people who understand how payments come through the health care system? How does data come through the health care system? And actually someone like Keya Eaton, who's a frontline supervisor of care navigators, what are their needs on the health care system?

So you put together that coalition to really begin to say, how can something like a frailty index help the healthcare system, help frontline workers, etc. 

In the last couple of minutes we have, I've got two things I want to ask you. First is, I know that you've not only experienced successes, but you've experienced some failures.

And we have this concept in innovation of failing fast. Fast is in the eye of the beholder, I realize. But nevertheless. I know i'm thinking of one example when you fail when you didn't fail, but we just realized that this was not going to be a success; therefore it was a failure. At what point did you know that a particular learning health care system initiative was [sic] not going to be a failure and you ought to just put that on the shelf for a while, maybe come back to it. You know, the one I'm talking about.

Kathryn Callahan, MD, MS: I do. [22:00] I do. And I think in that case, what we realized was that one of the key elements for a health system initiative or intervention is demand. People need to want it. Now, that could be patients, that could be providers, but we were seeking to intervene with frail older adults at a somewhat tumultuous time for them. You know, we were, we were thinking about the perioperative space. 

Jeff Williamson, MD, MHS: I think what you're telling me is you found that the perioperative space required a whole lot more coalition building than we- and an understanding of what the community caring for those individuals was willing to sacrifice in order to improve care and we didn't get there, right?

Kathryn Callahan, MD, MS: Oh, we didn't get there. 

Jeff Williamson, MD, MHS: And what is it that made you, because we can't keep doing things that just, you know, beating our head against the wall. What, was there anything that came to your mind that said, "Oh, this is not a particular place in the learning healthcare [23:00] system. That I need to spend my efforts. I want to move on." And you did move on. What happened? And you can be, look, we're all among friends. It's all Las Vegas here. Go ahead. 

Kathryn Callahan, MD, MS: So, I mean, a few things happened. One was that there are a couple of external things and a couple of internal things. 

An external thing is that our numbers did not look good. I mean, you know, if we had been trying to recruit for an NIH trial, you know, I would have been losing sleep every night. 

Jeff Williamson, MD, MHS: So the numbers of people whose lives you were touching patients really were slow in the end. 

Kathryn Callahan, MD, MS: Yeah. As one of our friends says, the juice was not worth a squeeze. You know, we were not getting the numbers that we needed.

The second is that we were not in a position where the people who could control- we did not have the people who controlled timing and patient flow on the team. 

I think another thing we learned that was really valuable was how stressful this particular period of time was for older adults and how they did not have the capacity to add one more item or one more thing to their world [24:00] during this time.

The other thing was a it was an internal feeling. And I think that just as we often say, when we're seeing a patient or when we're, you know, when we're working the space, you know, you'll get a feeling of concern or anxiety or what have you. And I was starting to notice that the meetings that we had about this were really difficult.

You know, the, the bloom had gone off the rose. You know, what had started as a really exciting relationship and partnership had become really difficult. It's kind of like trying to make a relationship work that isn't working anymore, and we needed to part ways. 

Not that we wouldn't work together in the future, but what we were able to say was, now is not the time.

You know, until there is such a time where XYZ can be different, we're not going to be able to make this work. 

Jeff Williamson, MD, MHS: Great. I've got one last question for you. 

One of the things that I've learned through you and with you is how sometimes in a learning healthcare system environment, unlike traditional research, our learning and our discovery shows up in places we [25:00] would never expect.

And it even wins competitions sometimes where we, we didn't even know that it was being entered. 

So can you just close by just telling our audience about just a few examples in the healthcare system, but then in the recent student competition that we had sponsored by the dean about where this particular learning health tool of the electronic frailty index kind of just started showing up everywhere.

Can you do that? And then we'll we'll close after that. 

Kathryn Callahan, MD, MS: Absolutely. Three examples come to mind. One is, pharmacy resident used the eFI along with another, another score too as part of her residency project to test what was the optimal way of identifying folks who would benefit from a post discharge medication reconciliation program. That was wild. You know, you get a notification of publication and I was really excited to see that. 

Another example was a [26:00] colleague of ours in anesthesia who recently published in JAMA Open with the eFI looking at outcomes in the perioperative space, a really, elegant examination and, and a great example of how, you know, we had started in that space and we will likely be able to come back to it and, you know, that we, we iterate over time.

Jeff Williamson, MD, MHS: Fail is not a cap- there's not a capital letter starting that word fail. That's just a, that's an example that even though we thought we failed or you thought you quote failed. Yeah, here comes a publication out of this out of nowhere. 

Kathryn Callahan, MD, MS: Here's the kinds of publication. And the other thing is that, you know, that may generate interest, opportunity and sponsorship at a higher level that may make our ideas possible.

And then the other one, which was just great fun was I was going through the medical student posters and at medical student research day, which is one of my favorite days of the year and met a young woman who is presenting work on frailty in the HIV [27:00] population. And what was great is that she was talking about the tool and how this was, you know, integrated and asked if I'd heard of it. And which was a fabulous, a fabulous moment. And I said, "Yeah, just keep going. Just keep going. This is this is great." 

But that was a wonderful feeling to see that it was not only You know, ubiquitous. I think there were eight or nine posters that had to do with it, and I only had one student this summer. So it was, they were not all my students. 

Jeff Williamson, MD, MHS: And you did finally reveal your secret identity to the person at the end, right? 

Kathryn Callahan, MD, MS: I eventually did. I eventually did, but I told her, you know, how well she had done.

Jeff Williamson, MD, MHS: Yeah, they wanted your autograph and all that sort of stuff. And then a winning, one of the winning entries also, is that the one that won?

Kathryn Callahan, MD, MS: That was the one that won. 

Jeff Williamson, MD, MHS: Tremendous. So, yeah, I think that that's, you started off this talk about one of the key ingredients to successful, I think, academic learning health systems success is just being humble and not, you know, knowing that many times [28:00] your discovery is going to be used, and just, just being able to get joy out of the fact that people are using things in ways you never would have imagined.

Kathryn Callahan, MD, MS: Well, there's a, there's an amazing person named Arti Hurria, who's well known in research circles, but would talk about how much you can get done if you don't care who gets the credit. 

Jeff Williamson, MD, MHS: Yeah. 

Kathryn Callahan, MD, MS: And I think that recognizing that there's a lot of really hardworking people who deserve that credit, and that if we can help our ideas get to the person, you know, at the end of the day, it's all about the patients.

Jeff Williamson, MD, MHS: John Wooden, the great UCLA basketball coach, said the same thing. I'm trying to figure out where that actually originated, that whole statement. But, anyway, it's been wonderful talking to you. I knew, always, I always knew it would be. 

We could probably talk for another two hours, even though you and I have talked about these things many times. There's always new alleys to go down. And so I just always enjoy it. And I really want to thank you. 

I just thank our audience for listening. And I want to thank the American Geriatric Society for sponsoring [29:00] this series of podcasts. And hopefully this has been helpful to many of you out there. If you want to get in contact with us, I think you'll, you'll know how to do that, but we're out ththeren the, on the web and different places.

But thanks again, Dr. Callahan. I really appreciate it. 

Kathryn Callahan, MD, MS: As always, Dr. Williamson, thank you.