Strategies for Keeping Clinicians Trained Up on Key Apps

Aired on: Thursday, Jun 02, 2022

314e at HLS Symposium 2022 | 314e Corporation

In this webinar, speakers discussed the ways to beat clinician burnouts that usually occur due to various software upgrades. The session was well attended and ended with an engaging Q&A session. The key questions answered during the webinar:

  • How to work with software vendors and understand the changes involved with any upgrade?
  • How to communicate and get feedback from operational and business leaders after understanding the changes?
  • How to schedule, measure effectiveness and collect feedback on training?
  • What kind of training and support to offer employees- from live to recordings to in-person?
Donna Roach

Speaker 1: Donna Roach
Chief Information Officer at University of Utah Health

With decades of experience in the healthcare industry, Roach has primarily focused on healthcare IT for 30 years now. Some of her experiences include working as the Vice President Information Services for BJC Healthcare and as the Market Information Officer at Ascension Information Services, to name a few.

Lee Milligan

Speaker 2: Lee Milligan
SVP/CIO at Asante

Lee’s experiences include working as the Senior Vice President and Chief Information Officer (CIO) at Asante, as well as the Medical Director of Informatics (MDI) in the same company. Lee has made some leading innovations in healthcare services along with engagement and transformation of individuals through coaching, training, mentoring, and accountability.

Ryan Seratt

Speaker 3: Ryan Seratt
Director Training & Development at 314e Corporation

Currently holding the position of the Director of Training and Development at 314e Corporation, Ryan’s 20 years of work experience include working as the Learning and Development Consultant at Real World Learning LLC, Director of Training and Development at SCL Health, Lead Sales Development Instructional Designer at MoneyGram International, to name a few.

Anthony Guerra

Speaker 4: Anthony Guerra
Founder & Editor-in-Chief at healthsystemCIO

Having completed his Masters in Journalism at Boston University, Anthony’s 20+ years of work experience prior to his current commitment at HealthsystemCIO include working as an Executive Editor at Wall Street & Technology, as a Managing Editor at Home Textiles Today, and as the Editor-in-Chief at Healthcare Innovation.

Here is the transcript of the webinar

Anthony Guerra 0:06

Good afternoon and welcome to Strategies for Keeping Clinicians Trained Up on Key Apps, a healthsystemCIO Media Inc. Production, sponsored by 314e. Just a little housekeeping before we get started. My name is Anthony Guerra. I'm the editor in chief of healthsystemsCIO, and I'll be your moderator today. We're looking forward to your participation, you can send in your questions or comments at any time in the q&a box, and we'll take them later in the program. Just so you see how we're going to spend our time today. First, we're gonna go about 35-40 minutes with our main panel discussion featuring Donna Roach CIO at University of Utah Health, Dr. Lee Milligan, SVP and CIO at Asante, and Ryan Seratt, Director of Training and Development at 314e; and then we will have our q&a. So let's jump right in. Donna, let's start with you. Can you give us an overview of your organization and your role?


Donna Roach 1:01

Sure, happy to; so I had been the CIO, I was saying for a little bit for 18 months, I'm coming up on 20 months now. So, I have almost been here for two years. So I'm pushing in my two-year mark, so the University of Utah Health is located in Salt Lake City in Utah. We are a five-hospital system. Soon to add a sixth one into our mix. We have, about 2 million patient visits that take place every year; we're very much growing in our footprint. If you think about Salt Lake City, we are right up against the Wasatch Mountain Range. An interesting fact about University of Utah Health, we are embedded into the University of Utah. So we're, we're kind of all combined together. We are a five-state quaternary referral in the Mountain West. So we are the only academic medical center and a five-state region, and so we bring we have a tremendous amount of partnerships out in those other states that are, have very rural systems or sometimes access to care. So that's a big component of what we do. I would say last year, we had about a 30% growth. So interestingly enough, our margins did quite well; when I see all of these other organizations that took kids, our margins have done quite well. So you know, I think more credit to our executive team and to our staff who really helped manage the expense side during the pandemic. So…


Anthony Guerra 2:47

Very good. Thank you, Donna. Lee?


Lee Milligan 2:49

Hey all! Lee Milligan, CIO for Asante health system in Oregon. We're a three-hospital system in Southern Oregon, serving about nine counties in Northern California and Southern Oregon. But 100,000 ER visits per year for 1.1 billion in annual revenue. We have an ACO that's been set up for about five years now, we're still kind of plodding along with that, best we can. And it's our super high-quality health system. We've won Truven, the top 15 Health System, nine years in a row. The only other system that can claim that is Mayo; all three hospitals are CMS five star, we take quality very seriously here, and we've attempted to leverage the heck out of technology to be a part of that quality journey.


Anthony Guerra 3:34

Thank you, Lee. Ryan?


Ryan Seratt 3:37

Hi! I am Ryan Seratt with 314e, and I'm the Director of Training. 314e provides consulting services around technology and technology implementation. So we all work very, very deeply into data management and also training, which is where my expertise falls.


Anthony Guerra 4:00

All right, very good. Ryan. Thank you. All right. Donna, we're gonna start with you. Please discuss the challenges around providing users with software training, indoor support in the following scenarios, new employees, existing employees on new software, and existing employees on changes to existing, existing software. And I think, what are the theories is that that third one is the trickiest to manage. But anyway, take us through that.


Donna Roach 4:24

Yeah. So you know, I have a training department, you know, just in, what I would call our IT shop of around 38 FTE. So they are a makeup of E-learning developers, instructional designers, classroom, virtual training, and phone support. We do about 7500 hours of classroom training, a year. So kind of I wanted to give you that picture of what we do. I would say for our new employees, it's actually we're pretty structured, and we work really well with HR Bring the new employees. And I think the hardest thing is, and a lot of you might identify with this, is more than contract people, getting them access into the systems so that it can kick off some of their training, that, you know, access into the LMS is sometimes a little trickier, but for the most part, we get really good feedback. We, we ask for feedback after our training for the new employees and do a fairly thorough job of that. existing employees, I think the hardest thing is the outreach that we need to do, and making sure that we do it at a multitier level, they can sign up through LMS, but it's also we do a lot of At-the-elbow support. So we can do just in time training for them. The, the last one, and it's kind of interesting, you bring that up that that may be the hardest, and I think so we're we are primarily an Epic shop. And I think we've done a lot of different ways in which to do outreach of training to people. So it's not just one size fits all, right, and we make sure that we have things like sending out different training material that is either just in time videos, or they can sign up for an in-classroom training event, which I think is really good. I have a whole physician informatics team that will come to At-the-Elbow, because sometimes it's very difficult to get physicians to sign up and to do some of the training programs, but we really tried to approach it in a multitier way. So that we can kind of capture, you know, people during before the upgrade goes in, or before there significant changes in the system, you know, you know, we're not perfect, people fall through the cracks, and so it's kind of that At-the-Elbow. When we do go live, that's really critical. One of the pieces I didn't bring up is we have phone support hours, and for as many hours as we do in the classroom, I have just as many hours of phone support that take place. So about 7500 hours a year. You know, we're about 14,000 employees on the health, hospital, and clinic side. So kind of put that in perspective.


Anthony Guerra 7:26

Very good. Donna, thank you for that. Lee?


Lee Milligan 7:28

Yeah, I agree with everything Donna said about kind of framing it up. For us, we spend about $3.1 million a year on training; training really is that the last mile of really getting everything you need to get out of your investment; I've jokingly, tongue in cheek, told my team that they're really the technology, full value extraction team, right? Because they allow us to be able to get that full value out of that incredible investment that we're that we're making, and that applies to everything right, it applies to I think about down on my car, there's buttons on there, I don't know what they do, right. And so there's always something like that. And maybe there are some buttons, that doesn't really matter, there's probably some buttons, that would be nice for me to know, right. And it's the same thing with technology, there's always aspects that we can we can leverage that we're not currently leveraging. In terms of the breakdown of these three categories, new employees, as Donna mentioned, is the easiest. It's very structured, have a very specific approach to that, it gets a little dicey with folks who are added on last minute, we've had folks, you know, start in the NICU, and we got three hours warning. So that makes a little difficult for the team; our team has risen to the occasion and, you know, jumped on it to fix it. existing employees on new software that gets challenging because pulling people out of their clinical scenario is always a dogfight and operations folks, God bless them, but they never understand the value of the training as it relates to how efficient their staff will ultimately be. So like Donna, we take a variety of approaches that are one of the ways we do it is, is we try to incorporate it into their current meetings. So, for example, for the physicians, the hospitalist, for example, we have a pretty, pretty deep bench when it comes to our physician builders and clinical informaticists or physicians and what they do is they actually carve out dedicated time at each staff meeting each month, and they actually meet in a room with computers. So they can go over a few of those, those things. And then in terms of the third bullet point their existing employees on changes to existing software. You know, we take we're an Epic shop, they release quarterly upgrades, we don't take them quarterly. We take them twice a year, and we try to strike that balance between upsetting people's applecart and also at the same time allowing them to have access to new stuff, and so we feel like that, that's a reasonable, reasonable balance Epic didn't like it, but it really works for us. Prior to those twice-a-year upgrades, we have this all-out effort to get folks to understand the key elements of it, everything from our official advisory councils to we have kiosks set up in the hospital, to rounding at the clinics, and kind of showcasing these new elements. So it's really a multi-modal approach, and I would say it's imperfect but certainly better than it was before.


Donna Roach 10:31

Yeah. And Lee, I would add, you know, I think we probably do something very similar. And then, they the communication prior to an upgrade or change is pretty good. I mean, because of you want your physician and your clinician population to really know about the upgrade. So sometimes the training or the, you know, covering off on how to do it is it's kind of anti-climax, right? It's like they already know about it. I think and I'm wondering, you know, just one of the things that we struggle with a little bit is how, you know, so we've done it, and now we can kind of see At-the-Elbow, how they're doing. It's how well how well have they ingested the material? I think we need to get better at that. Because we know it's like I said before, we know it's not perfect, it, there's got to be ways that we can improve upon some of the materials so that when somebody hits the new upgrade, that's they're hitting it, and it's, you know, there, it's actually the learning curve is very small for them. And so they can kind of use the new functionality very quickly. Do you do anything like that…?


Lee Milligan 11:46

That is such a great point, and the vendor's Epic, or whoever, you know, they want to, they want to showcase their features, and I get that; if I was dealt with developing software, I'd want to showcase the heck out of this cool feature. The doctors and nurses don't care. They want to understand within their workflow what is going to change? And how will they navigate that change? And so one of the things we've attempted to do is every time lessons and training plans come out, we try to adapt them specifically to the workflow of the individual who will experience it. And that for us has really made a big difference.


Anthony Guerra 12:23

Ryan, there's a tonne there. I'd like to get your reaction to anything you've heard.


Ryan Seratt 12:28

Yeah, I'm taking great notes. I think, you know, one thing that both Donna and Lee had kind of spotlighted is, you know, what I call personalized training, that one size doesn't really fit all, and that using different modalities to reach people is definitely one of the keys to success to these ongoing changes. I think, you know, we live in a world where there's a lot of information flying at us, and helping the physicians sort through that, like what Lee was talking about, is making sure that this is what's important. This is how you're going to apply this, and doing some of that work for them is definitely something that I've seen in very successful organizations.


Anthony Guerra 13:12

Lee, I'd like to talk a little bit more about the dynamic you described, with Epic putting out upgrades four times a year and you only taking them twice a year. Obviously, that was a major decision. What are you losing? What are you gaining? And I guess it was too much disruption, to do this four times a year to put everyone through this? We just can't do it. We're going to do it twice. But take me through that.

Lee Milligan 13:40

Yeah, I mean, in theory, I like what Epic has done here. You know, I recall when we were doing it every, you know, twice a year, or once every two years, once every 18 months. And those were big changes. So the idea of the kind of breaking that apart and having smaller, more incremental changes makes sense. The challenge is you have to do the same amount of work. Right, you have to do all the same testing. You got to test everything from end to end. And so, if you do it that way, your team is all they're doing is preparing for the next upgrade. And they're not working on, you know, capital projects or optimization requests even break-fix sometimes can get put in the back burner, and so we had to strike that balance between how do we get the most amount of new stuff in front of our folks using the technology, while at the same time not breaking the back of our IT team attempting to get that out there and so far, it seems to have struck the right balance for us we'll reassess as time goes on. But for now, it seems to work pretty well.


Anthony Guerra 14:40

Donna, you seem to describe one of the challenging aspects is may not be actually providing the training, but it's determining to what degree that it's been absorbed and to what degree has the information that's been communicated about these upgrades been absorbed. Can you talk a little bit more about working through that and trying to determine that?


Donna Roach 15:00

Yeah, I think that's where we really rely on our informatics team. So I have a position informatics team and a nursing informatics team that do a lot of work being At-the-Elbow. We also have a learning dashboard that kind of help through that process. But you can, you know, it's gotta be multitiered. Like, I can't just depend on the folks that are kind of in the trenches, I think, you can look, we utilize ServiceNow, for example, and doing that analysis and ServiceNow to see well, how well are we doing? You know, what are the tickets that are coming in that are kind of questionable, I do a lot of rounding with my CMIO, and I am always astonished when I go out there, and I hear how they're using the system, you know, whether it was shortly after an upgrade or before, you know, coming into an upgrade. You just can see that, you know, they're trying to make it work as best as possible. But this is not what we wanted or intended for them. So it's kind of going back and having different ways to approach the improvements. I think I think we do a good job of working with people, sometimes one on one, sometimes in groups, because we do we have a big group called Our Value Officers where we make sure that are we delivering on things that we've said that we would deliver on, but I think it's spending time making sure that okay, did the training, did the upgrade, do what we said it was going to do? Is it achieving what we want it to do? We actually take three upgrades a year, so we do probably one more than Lee, and some of that is, you know, in my mind, some of that's trying to get to more the baseline of the of the system, because the more you're, you haven't done some of those specializations optimizations in the system, it's easier to take some of those upgrades. So it's kind of getting people through some of the changes, I think, I tell people, one of the best things that our you know, our smartphones, our Apple phones and the Android phones have done is they just push out upgrades. And so it's kind of learning in time, right? Like, oh, here's, here's what changed on your phone. So, the little things that, you know, maybe 10 years ago, people would get all upset about they change the colors, they move this around, you know, so some of what we've been indoctrinated with on our iPhones and Android phones, it's like, it's it's, you know, we've already set the stage for these changes are going to always happen. Now let's look at the bigger changes or the improvements in functionality. And so it's kind of like taking it to that next level.


Anthony Guerra 17:57

Ryan, so Donna takes three upgrades a year, Lee takes two, and Epic wants four. So do you have any advice there for people on the line regarding how they should navigate that? Or should they do just what the vendor recommends? Or what are they looking at internally to decide how often to do these things?


Ryan Seratt 18:18

Yeah, I don't I don't think that there's one right answer. I think it really working in a lot of different organizations, that I've seen different models work. When I personally have taken double upgrades, I don't recommend those to anyone when they were year-long. So two years' worth of upgrades was very, very hard to do. But the idea is, and I think that we're seeing this more and more in the information age, is that smaller chunks of information are easier for people to understand and apply to their work. So, by having more, more but smaller upgrades, the ideas, hopefully, instead of teaching you 20 things, I'm teaching you 10; and hopefully, we can support those and do really well at the 10. And then there is a higher frequency, but different organizations, I think culture, there's a large cultural component to that doing for I know that talking to some people that they feel like they do work on the upgrade constantly is that it's like congratulations, we finish, tomorrow we start the next one. And so, there is a balance there that needs to be struck.


Anthony Guerra 19:33

And any advice regarding the issue of post-training analysis to determine if the information was absorbed?


Ryan Seratt 19:43

I think the and going back to what Donna was saying, the clinical informatics teams that are actually working elbow to elbow is really the way that you look, and you can measure that. You know are people using the suggested work? Flows are the new tools and the way that they're designed? Just because there's a new feature doesn't mean that people are applying it the way that the analyst built it. We see that quite often that if you that, it's like, okay, I have these tools, I'm doing the best I can with them. But it might not be optimized by doing the elbow to elbow support by getting out and doing the coaching and going, oh, did you if you click on this, this will bring up these screens, which will then automate this workflow and get some things taught some times things are missed. And really, you have to individually find that out from people. There's not an easy answer to that for sure.


Anthony Guerra 20:41

Lee, we did a webinar on this topic few months ago with 314e. One of the CIOs mentioned, it talked about a situation where one of their vendors there was a change to the software. Apparently, they either missed it, or the vendor did not think it was a big enough deal to let the customer know. And it turned out it was a fairly big deal to the users, which made the CIO fairly unpopular for a short period of time for, you know, where is this thing all of a sudden that I used to do this? Is that an issue that you've seen come up? And how do you navigate that?


Lee Milligan 21:15

It is, on occasion. You know, I'll tell you that there is a disconnect, I think, between folks who build the software and those who use the software. And I think because of that's where, you know, informaticists swim, right, they're able to kind of connect the pieces on this. And I agree with what Donna said and what Ryan said around kind of going to them first understand how effectively it's happened. But, you know, I've seen scenarios where things that seemingly were small ended up being big deals. And I think it's primarily because I'll take doctors, for example, doctors, you know, you're caring for a patient, you're, you're there for just you and the patient trying to figure this whole thing out. And all of a sudden, the thing that you're familiar with, that you you know, you latched on to you felt some trust for, is gone. Right, you're kind of floating there, and the patient is looking at you and can see that you're like trying to figure things out, you don't quite have it all sorted out, you're trying to try to portray some level of confidence in your in your competence as a provider. At the same time, you can't even figure out this user user interface because it's changed. It's a very discombobulating situation for a provider to find themselves in. So it's really our duty to get ahead of that. And it makes sure that we've made every effort to inform them about how their workflow specifically will change. So they don't find themselves in that circumstance.


Anthony Guerra 22:41

Yeah, and the physician that I've spoken with before said the worst thing you can do to a doctor is disrupt their train of thought interruptions, disruption. This, this is exactly that, right? I mean, this disrupts the train of thought you're trying to focus on a million things to deal with the patient, assimilate all the information that's coming in from the chart from the patient to make a diagnosis. And boom, that's just gets blown up. Right?


Lee Milligan 23:08

Yeah, and if you think about the work that the doctor does, or the nurse does, really, as it relates to the, the interface, it's really they're trying to find something, or they're trying to do something. Right. And so, historically, finding things has been awful. You know, if you look at even the best EHRs out there, it's just, it's still not great. It's gotten better over time; Epic has the search feature, which is actually quite helpful. They've done a good job of optimizing that, and whatnot. But if you look at, you know, to make search really effective, you have to have all the data governance behind the scenes that allows the proper metadata and tagging. They'll pull things up in a way that is appropriate for that scenario. And so, so there's that piece, and then there's the doing piece. I'm going to order something. I'm going to get something on the record. Same thing there, right, how do we make that be an easier, totally intuitive scenario? I would love it if we get to a point at some point where training is unnecessary. Right, because the, the user interface is just so dang intuitive. You think about you know, Apple, you know, when they send the updates that Donna was referring to before, right? They don't send you a manual. They don't say go train for 12 hours to learn how to do this new thing. They just make it really intuitive. And if it isn't 100% intuitive. They have ways of kind of, you know, having some pop ups and whatnot that kind of guide you where you need to go. We're not there yet. We need to get there.


Donna Roach 24:40

Yeah, you know, what I would add to I think he brings up a good point. You know, one of the things that I've noticed, as you know, we've designed systems, even in Epic, we've designed so that the information is close to the clinician at that unit, at the bedside, right. But if there's anybody that crosses over different units, we make it very difficult. So, you know, take a hospitalist who may cover three, you know, like, if they're internal medicine, they may, they may be on two units. And we make it very difficult for them to get into that one unit and do what they need to do. And then they have to get into the other unit and, you know, haven't helped them if all of a sudden, they have to manage both and say, Okay, well, where should a patient go? They may have to look at seven units across the board and go and then have to navigate each. So I think, you know, we've done I think the EHRs have done a good job in meeting the needs closest at the bedside. But it's this kind of rolling up of the data, like you said, the search data and making it a little bit easier. So we have the dashboards that they can look at and go, oh, okay, this is where maybe I can put that patient because there's two openings on that unit versus there's, you know, that the, you know, my other unit is full. And so it's those kinds of things that you have to understand the difficulties people have run into with the workflows that we've given them and the systems. So it's it is that constant improvement.


Lee Milligan 26:22

Yeah, I want to roll with on this point there. You know, I think, you know, kind of beginning with the end in mind, though, Stephen Covey's stuff, right, so what do we want this to actually do? Specific for finding stuff, that first bucket? You know, ideally, the doctor shouldn't spend their time trying to find something at all? Right? It should be they should be thinking about what to do with that thing that they found. What does that mean clinically? And how should they act on it? Right, so we shouldn't be at the point. Now, we should have at the point now where we can say, you know, hey, Epic, or hey, Cerner, or hey, Allscripts, bring up the last five creatinines of Mrs. Smith; it's a marker of kidney function, and graph it out. Then you can look at it you can see the trajectory, you see a graph, you're like, oh, yeah, she's doing fine. Or oh, no, something's going wrong here. Now, I'm going to do X, right, but the doc is pulling that stuff up and trying to find it still in it. We need to get to that next layer.


Donna Roach 27:19

Yeah. And I want to add on something because she brought up something Lee, which is a good point. It's like, okay, now I can graph that, right? If I could do that. And then I can show it to the patient, or I can show it to the family and go, Okay, see, this is what we're trying to control for, or to bring up an easily bring up an image to say, look at this is what's causing your pain and your back and why you're, you're having this kind of, you know, lower back pain. And it's, you, you, we have to give the tools to support the clinicians as best as possible. But then it's this flowing to how do we, you know, do the teaching to the patient and their family, so that they are fully part of kind of that caregiver team, right. Yeah. And that's where it really becomes, you know, that's the ideal. That's where that's kind of where we want to get to,


Anthony Guerra 28:14

You know, Lee, I'm picturing you actually getting finding the data on the computer in a difficult way and actually drawing a graph on a piece of paper, so you could show the patient. I have


Lee Milligan 28:25

I have done that many times. Okay, okay. Mr. Smith, you're supposed to be here; you're actually here. We got to do something about that.


Anthony Guerra 28:36

Right. So, Ryan, this is, I mean, it's very interesting, because what have we done so far? In this conversation? We were talking about training. And from training, we go, yeah, you know, we have to train on a lot of things. And it's complicated. And it's, it's, there's lots of changes that happen. It's complicated stuff. It's not intuitive. That's why we have to have lots of training, and we have to figure out if the training is working. And it morphs into a conversation of God, wish we the stuff was better. So we didn't have to do all this training. Right. It's just a natural outgrowth. But you're here to talk about how the training that is the reality of today, based on the tools of today, is best done. So any thoughts that you have based on some of the complaints we've heard about the tools to talk more about the training?


Ryan Seratt 29:24

Yeah, I think the, you know, the challenges are challenges that are everywhere. And a big part of that and kind of in my world is matching up the, the processes that exist, and the tools and everybody has different processes. Even those departments have different processes. So as I move from one department to another, or one location to another, I get to actually have different factors which affect the way I do the work, which therefore means the tool works differently because it's just not one easy flow. And you know, 10 years ago, I worked in finance. And in that it was really easy to teach people Salesforce because there was only six roles. And in my first year of working in the hospital, I think I was working on training for 217. I think and I quit counting at that point rolls, and the processes are more complicated. So going back to the what is successful training is, successful training prepares people to do the, give them the knowledge and the skills to perform tasks; when you really break it down, they need to have the confidence to be able to use the systems in their flow of work as they're doing that. And the best way to do that is that we put them through the classroom, but we know that they don't remember everything from the classroom, that we need to support them in other ways. With informatics, we were also talking about help desk with small updates, going to the meetings and trying to do that continuous training and to develop them and coach them over time and, and to help them advance to make their life easier. A lot of times, when we talk about physician burnout and actually working with us physicians, they're not using the system optimally. So how do we help them use it better so that they're so that they their lives are better, they're not spending as much time at home after dinner, documenting what happened during the day?


Lee Milligan 31:31

Can I touch on what Ryan said there? Just one anecdotal example. And we'll get into this in more detail in a few minutes. But recently, we had a physician who was experiencing difficulties with Epic. And finally, she reached out and said, Look, you know, white flag waving the white flag, I need some help. So one of our trainers spent one hour with her. And between her in basket, her note creation, and her order generation, she saved 28 minutes per day for normal workflow. We measured it measured before and after we use signal data and other data to measure it. Given the number of days she works a year, that's 104 hours of time saved by spending one hour with a good trainer who understands your world and can identify the low-hanging fruit and go at it. So, I just think it's just such a great example of operationally how we can do training; good training can impact the organization. Well, let's


Anthony Guerra 32:39

Well, let's talk a little bit more about that Lee, she, it's a she in this case, she raised the white flag. Let me pivot over to Donna and let her talk about this. We want to not have to wait for someone to raise the white flag the data that Lee talked about finding and when he did his measurements, we want to see if it was, I would imagine, get some sort of notification that physician so and so is spending way more time than the average, we need to reach out and try and do some kind of intervention. Does that make sense?


Donna Roach 33:16

Oh, yeah. And Lee brought this up, you know, using those signal reports. And what's kind of built into Epic is really key. And that's where nursing informatics is, especially on the ambulatory side goes out there constantly looking at that. And also then on the inpatient side, the physician informatics works very closely and making sure monitoring that ahead of time, I would rather catch somebody, you know, you can kind of see that somebody's having trouble, then having them complain or just kind of quietly struggle. You know, that the quiet, strugglers are the ones that are going to be the, they're the ones that are getting burned out, because they don't know, I think we've done a good job, at least in reaching out and making sure that we're monitoring those reports, which is, you know, it's a test. It's somewhat of a testament to Epic that they've got that built into the system and, and we can use that effectively.


Anthony Guerra 34:14

Lee, I will let you comment on that. Your thoughts. I mean, you when you live, when someone comes to you, they're in a position of wanting help. So that's a good thing. Right? So rather than you going to them, they may be ready to get help, but it's good to reach out. But you also, you know, that also means they've gotten to extreme frustration, which is not great, but good to catch them before. What are your thoughts?


Lee Milligan 34:37

I like this to Alcoholics Anonymous, you know, somebody has to actually want to do it, right make the change in order to actually be open to the changes when we've attempted to do this proactively, which is my pension as well. Just like Donna said, I love that idea being proactive. The level of engagement can be not what it needs to be So when you have somebody who finally says, Look, I can't take this, I need some help. And they come in, they're open minded, and they're actually listening, and they're willing to try new things, because the old way is it wasn't working for them, then all of a sudden, you have this magic opportunity. This, this very anecdotal, this particular case really highlighted that. I feel like this is something that we could probably get ahead of better, using different tools. Epic has the signal data also has for nursing has neat data and EAT data. And then on the inpatient side, they have a tool, but it's pretty clunky. Right now. I don't know if Donna, you're familiar with that or, or others. But it's just not is not as robust. Yeah, as the signal data. So they need to get there because inpatient obviously, is just as important as ambulatory. Yeah,


Donna Roach 35:51

Yeah, it's just, it's interesting, you bring that up because I hear the squawkers more on the inpatient side, right? Because they're like, you're here, you either hear it through nursing, or you hear through the hospital, as you know, the individuals on the floor. So it's, especially as we go through upgrades, you can kind of feel the heightened awareness. And I think that's, that's why it's nice having a group of informaticists that are, are out there kind of, you know, doing that their job.



Anthony Guerra 36:21

Lee, is there a group of I mean, does it happen sometimes where you hear or word gets to you have a physician, instead of saying, I want to help just going around saying, I hate this, this stuff doesn't work. And then you have an opportunity to make an approach there. It's not that they formally brought a complaint, but you're just hearing, this supposition is very frustrated. Has that happened?


Lee Milligan 36:47

My Yeah, my situation a little bit different. Because I've been a physician here for many years. I know most of these folks. And for better or worse, they have my text number. So I don't just hear about it through the grapevine I receive messages sometimes. But one of the things that we instituted last year that has been really helpful, has been our rounding and are rounding reports. So we started rounding both on the inpatient side and on the ambulatory side. And we were doing great rounding, we're solving problems or identifying issues getting ahead of issues. But I was still hearing about some of the senior executive leaders, we're hearing about anecdotally about things and bringing it to my attention. And they didn't even know the rounding was happening. So I pulled the team together, we had a conversation about it. And we decided to create a template for our rounding reports. And that and that templated rounding report gets distributed on a weekly basis to all of the executives involved. So they can clearly see what clinic was visited, what provider was interacted with, what complaint they may or may not have had, and what the plan of action is associated with that up to an including ServiceNow ticket numbers. It makes it really crisp. And I've only one time since then had executive asked me about it. And I pulled up the email that he received and asked him to go to the second paragraph, subsection three, to see what was done about it. So he understood that it was happening. So it's a very effective tool to have those conversations.


Donna Roach 38:20

Nothing real similar, not not in this organization, but a prior one, where we had a mass or a revolt around an implementation that we had done of a system. And I'm going to try to keep names out of it to protect the innocent. And, you know, I came in, and I and I kind of was the new person and in the, in the situation. I said okay, here's what we're going to do. We know it's a very specific group of physicians, we're going to go to them, and we're going to survey them daily. If they're on duty, if they're, you know, on coverage, we're going to ask them, you know, three or four basic questions. We're going to see how is the performance of the system doing this day? And you know, what is your likelihood of continuing to use the system, and we collect this daily information on a daily basis, then we rolled it up because the people were just going directly to the CEO saying you got to you got to take this system out. And I'm like, Well before we do that, that is a huge capital investment and time investment. Let's make certain we can make some changes before we would do this, and I sit in and also you've got some people who maybe we didn't kind of train them and maybe we didn't bring them along in this process well enough. So let's you know, yeah, it's after the fact but let's try to do something and so it definitely shut down some of the discussion or the I would say the pathway they made to the CEO because I got to the CEO before they did with, you know, the daily report of, here's the five physicians that were on, you know, on-call this day, here's their feedback. And I said, be honest, if you don't like it, if it's if it's bad, tell me, but you got to tie your name to it, you can't be anonymous. Because I, you know, I'm, you need to tell me I'm being I'm being upfront with you, you need to be upfront with me. And sometimes they would see the, you know, I had my staff doing this, and I'd go out and do it, they'd see us coming in, they're like, oh, no, no, it's fine. And I said, Well, can you fill out the form, and they're like, Okay, I'll fill in, some of them are like his, but I said, then what I'm going to do is, I'm going to say, I came to you, and you didn't want to fill out the form, because that was our agreement. And so it was a little bit of, it was kind of a tool that I used to, yes, get the feedback, but also kind of circumvent the, the, you know, pathway to the CEO. So like, I had the information ahead of time, in terms of how best to manage a not-so-good implementation that had happened, and then kept the system in place, which was like, Okay, well, something worked out.


Lee Milligan 41:19

that's, that's brilliant. I'm gonna I'm taking notes over here, by the way. I want to roll that just for a second, if I could, Anthony. Back when I was CMO, back in 2015, we had to clinics, call up the CEO of our ambulatory Medical Group, and tell that CEO they were going to quit, because Epic was broken and needed to be optimized. There was issues and problems that all needed to be fixed. And so I pulled together a team of physician informaticists, small team, four of us. And we went out and spent time with these doctors and sat with them and their workflows to understand what was going on. And what we found was that 70%, we quantify this, we made a huge spreadsheet, 70% of the things that they either want enough to build or told us were broken, actually, work just fine in the system; but they didn't know how to do it, it was a training issue. The other 30% was broken up between, you know, some optimizations that might be nice, and also a few break fixes as well, but the vast majority are training-related. And so to Donna's point, I really feel like you know, going out there spending time with them, seeing them in their in their workflows is key, because you'll number one, you'll gain credibility out of the gate, by actually doing the work quickly, when you have an executive out there willing to spend time in the trenches with folks who are doing this work. And number two, you'll gain insights that will be very powerful in terms of making improvements.


Anthony Guerra 42.57

Ryan?


Ryan Seratt 42:58

I absolutely agree. You know that, you know, the, the way, people's understanding is oftentimes one of the things that kind of hold them back, they're not able to see kind of how something is developed. It's not always evident. You know, when you're seeing patients, people are bleeding. You know, in the hospitals, people are sick, there's a lot of things going on. And it and connections are not always evident. You know, obviously, coming from a training side, hopefully it never gets to one of those type of type of actions where things have gone that bad, like to hit those off earlier. And I'm just a very large proponent of making sure that we're sharing best practices. How do we actually do that continuous training, and finding ways to reach people before it gets gets critical in those situations. Love, I love how you handled it. But whenever that happens, really something bad has happened on the training side, or the build side before it ever gets to the operational.


Lee Milligan 44:05

Can I ask Ryan a question? Please? What are your thoughts on gamification of training? In other words, how do we make this stuff fun? Yeah, or at least I'm interesting to do it for folks. And I think about that because I think, you know, I got four kids, they all play way too many video games, but I watched them as they're navigating it. And they very quickly kind of pick up on the dynamics of the game, based on how the game is laid out. It's again, going back to that intuitiveness of it, and it makes it kind of fun as they're learning. What are your thoughts on gamification of training?


Ryan Seratt 44:44

So, gamification is something that I flirted with for a long time. I'm not the biggest proponent of gamification. I actually fall into the make training relevant and as short as possible, where gamification can take that away, I've seen gamification done really, really well and work. You know, the simplest example is like a Jeopardy game for, to test out. So we recently developed that for a client, where new providers come on board, they go through the Jeopardy game. And if they score high enough, they don't need to take the training, they already know the Epic system. So that's an example of gamification where it's worked. But I've also seen it not work and go; well, what does collecting diamonds have to do with my job? But I think that there's aspects of it is when you actually need to test someone, gamification is good. But when you're actually teaching them workflows, it tends to fall short. It's, and it often seems a little hokey. On the flip side, badges, people love badges. So getting physician champions to create microlearning best practices to record themselves in a one-minute video of, hey, here's how I do it, that we can send out to everyone. And then the more people that watch the video, the they get a badge. Those people love badges, they, you know, we were taught from a very young age, that the gold star is the thing, right? So you ate all your fruit, snack time, here's a gold star that still works today.


Lee Milligan 46:29

Well, Epic. Epic has the power user program, which is, I think, really effective because, you know, Epic pivoted to the physician builder program, a decade ago, and it was really helpful for doctors to understand how things were built, but didn't necessarily make them better at actually using the software. The power user program is really designed to have doctors take their normal use of the software in interacting with patients to the next level in terms of efficiencies that are built-in. So I think there are some badges associated with that.


Ryan Seratt 47:02

Yeah. And I think that's a great program. The really the most successful implementations and rollout of new software is when there's a kind of a three-legged stool between the build team between the training team, and between the physicians or the users that are going to be using them. When you have good champions on all three areas, then the rollouts are usually smoother.



Anthony Guerra 47:29

Ryan, did you have a question for one of both? Or both of your co-panelists?


Ryan Seratt 47:35

Absolutely, I think Donna, you mentioned kiosks, what are these? You just mentioned in passing. But that was one of the ways that you were reaching people. Can you tell me a little bit more about the way that you're using kiosks, to help train?


Donna Roach 47:51

I don't believe I mentioned kiosks. But yeah, but we do have a form of a kiosk in terms of training people that they can do just in time training. But it's more just in time type videos and stuff. And that I, I will put a plug in for so the University of Utah has one of the largest gamification programs in development in terms of engineers and stuff that go through it. And so happy, we could probably talk a whole thing about that. But I'd like Lee, I think I liked your your point of how can you apply that to training versus kind of more outcome based, right. Ours is very outcome based around healthcare. So yeah. So


Anthony Guerra 48:37

Lee, you want to talk about the kiosks at all?


Lee Milligan 48:40

Yeah, sure. So we do, as Donna was saying is, just-in-time training. And so we set up, you know, four or five laptop computers. And we try to put it in opportune locations. So just outside of the Surgery Bay, up outside of our ICU and CCU area, and in some of our basic floors, and because we recognize folks work all kinds of hours, right? Healthcare 24/7. So as people who you know, the only time they're ever going to get exposure to this is at three in the morning on their break. So it's a great opportunity for folks to swing by, and even if they pick up, you know, three things from this new upgrade it gets in their brain, it can be helpful, and it also once again, kind of showcases that we're willing to be there at these off-hours, willing to attempt to train to the extent we can.



Anthony Guerra 49:33

Very good. Donna, did you have a question for one or both of your co-panelists?


Donna Roach 49:37

I do, I was thinking about this topic, and I really do like the topic, but I want to maybe take it to a different level and ask Lee what he's what he might do in his organization. So I, you know, so I'm, I'm a big, I'm always kind of reading and one of the books I've read recently is called Beyond digital. It's about the, you know, it's a fairly recent publication 2021, I think it came out. What I loved about what he talks about, and it's like in chapter seven of the social contract to have with your employees and, and really start because we're introducing so many digital tools, and we don't really do a good job of training at all levels around our digital tools, and I think our biggest gap is at our executive level. So my, my, kind of the next level up of the operational manager and then go higher. And I'm just wondering what it do you do any kind of training, or any executive training around tools and expectation of tools, even your how best to use your dashboards? And, you know, whether it's Power BI, or, or even into Epic? Do you do you have a program in place?


Lee Milligan 51:02

I don't put, you know, as you're, as you're describing this, with light bulbs going off in my head, it's, you know, if you think about what we need our executives to do, they need to be you know, deep thinkers, they have to be good communicators, they have to be good leaders. In order to do all of those jobs, they have to be able to access tools that help them understand their world. And all those tools nowadays are digital. So I agree with you, if we can put together some sort of either a boot camp out of the gate for folks, or some sort of like continuous training, that, you know, once a year, they sit down for four hours, and we pound through some stuff. That would be awesome. I love what you're doing with that.


Donna Roach 51:43

Yeah, I mean, even just, you know, like, cybersecurity, we make everybody go through the cybersecurity and then if they get caught in the fishing, they have to go through the fishing thing. But even just little stuff, like on a regular basis, like how do I how do I run that dashboard? And what's that dashboard telling me and, and just little things that I think there's a gap of people that just don't know. I mean, I, the people like closest to the units and to the patients, they're amazing what they know, it's there, there's a little bit of a gap happening at the, you know, kind of one level above. So…


Lee Milligan 52:21

I mean, even using, like, effectively using Outlook, right, or creating a team's channel? Yeah, right. Or just basic Excel? There's a lot of opportunity there for sure.


Anthony Guerra 52:35

Ryan, we didn't get too much of it. But I know that one of the things you've talked about is in the workflow support. That's kind of a unique, unique thing in our last webinar. The executives on the line said, that's something sounded great to them, but they didn't have it. Just talk to the audience, tell us what's available, what can be done? And just best advice around that.


Ryan Seratt 53:02

Yeah, I think, you know, that concept of the just-in-time support. It's been mentioned on the call by both Don and lead today. And how do we, how do we take learning from, after the event? So everyone goes through their their new class of new software, new hire, but how do we support them when they're actually trying to apply that there's a lot of different ways to do it. The kiosk definitely love that idea. You know, learn, you know, self-guided learning labs, there's also tools a 314e makes one about how do we connect people to all the information that exists in the flow of work, in a way that they can consume it. I think that, you know, a lot of times people go, well, that information is available. Well, when I looked at it, I got a flyer from three years ago, where we did healthy at a cookout, that was not what I was looking for. And so but actually giving the right information to people when they need it, in their moment of need. The help desk that Donna was talking about is another great example of that. There's many ways to connect people to the information, and digital tools can actually help out, especially ones now that do that add metadata to the item that are more self sustaining that people and that allows the team to actually deliver all the content exists to the people when they need it, so they can continue their work.


Anthony Guerra 54:35

Alright, we've got time for what I'll call a lightning round of final thoughts and advice. I'd like to frame it up this way for your comments. I didn't


Lee Milligan 54:43

Wait, wait, I didn't ask a question.


Anthony Guerra 54:44

Go ahead, Lee.


Lee Milligan 54:46

Donna, I went to the University of Utah for undergrad. So my question for you is, have you eaten up the pie pizzeria?


Donna Roach 54:54

No, no. Okay.


Lee Milligan 54:57

Best pizza west of the Mississippi. Sorry, Anthony.


Anthony Guerra 55:01

Oh, no, no, very good. Very good. All right. Again, framing up your final comments, I would like you to pretend you're speaking to someone in your in your position at a comparable-sized organization who feels like they are struggling both with making sure everyone who needs to be trained in any situation is getting training, and in addition, they're not feeling great about that. They're also not feeling great about the effectiveness of the training that they've got going, because they're hearing grumbling. So frame up your best final piece of advice for that individual. Donna, I'll start with you.


Donna Roach 55:41

So probably three-parter, you should be partnering with some medical, whether it's your CMIO, informaticists, that's your you're going to have the best combination of addressing something by doing that. Get out of your office, start doing some listening tours, and start having your hit list of what are the important things. And then I think applying Agile to say do some quick wins. don't Don't try to eat the whole elephant. You know, get some quick wins. And you will, I think, quickly see some progress made and improvements.


Anthony Guerra 56:27

Lee. Last word?


Lee Milligan 56:29

I started with a why right? I would in the why is quantifiable, right? So provider satisfaction, for sure. You can you can measure that whether you use class or you do something internal. And then looking at signal and other data, you can see efficiency and you can translate that directly into provider turnover, which has $1 cost associated with it plus is ethically the right thing to do. As well as efficiency, you can identify how much financial benefit there is to be more simply more efficient in the system. Start with that and then build a business case for why you're going to put in place a comprehensive plan that improves their experience using the technology.


Anthony Guerra 57:11

Ryan, final word?


Ryan Seratt 57:13

On the training standpoint. And there's a wonderful model by Kathy Moore that I think should be more famous than what it is. And some of that's really helped me is that she envisions a bullseye for training is not where you want it to be. Start with the business outcome as the bullseye, then figure out what people need to do in the next ring. And then figure out what they need to know. Outside of that, that'll condense your training and make sure that it's very targeted.


Anthony Guerra 57:45

Ryan, do you have the spelling of top your head of that name? When we want to look at…


Ryan Seratt 57:50

I think it's Moore…


Anthony Guerra 57:57

Okay. All right. Very good. Anyone on to look that up. All right. Well, that's about all we had time for today. Regarding continuing education, you can use the final slide in this deck, you'll get an email when the on-demand recording of this event is ready for viewing. If you want to sponsor an event with us, you can reach out to Nancy, Nancy Wilcox from our team and you can go to our website to register for upcoming events. With that, I want to thank our tremendous panel. Donna Roche, Dr. Lee Milligan and Ryan Seratt. I want to thank 314e for sponsoring and you for attending. With that, everybody have a wonderful day. Thank you.