Computer-based Training Will Not Steer You Wrong

Georgianna Kelly:
Set at about, at about 52,000. So yes, we are a large health system largest in Texas, if you know anything about Texas, that is saying something.

Georgianna Kelly:
All right, we'll get the next one. Ryan. So a little bit about who we are in terms of technology learning services.

Georgianna Kelly 0:22
My team was created; Technology Learning Services was created out of these three teams. Our North Texas ambulatory arm had been live, has been live on Epic for what is it now eight years, and they had actually developed their own computer-based training modules. This worked out very well, our HTP and clinics or North Texas ambulatory clinics are spread over a very large geography. So getting somebody to drive from Paris, Texas down to Dallas, that's about an hour and a half. And as you can well imagine that that was, that was just a waste of time number one. And number two, some of our team members share cars, and they have childcare responsibilities, that just wasn't, that just wasn't meeting the needs of the business. So boom, you know, developed computer-based training modules. So that little pocket of our business, that little pocket of our training business, if you will, had computer-based training modules. Now we moved to central Texas, we have 51 team members, all of that Epic training was delivered in the classroom instructor-led, but the geography challenge remain, if you're going to drive from South Boston in the temple where our classrooms are, again, you could be looking at a, at a 90-minute drive. So although the business did it, it really wasn't a satisfier for the business. And then in IS, we had 15 team members, mainly our principal trainers.

Georgianna Kelly 2:02
Now Nortex is our hospitals were on Allscripts until of June of last year, we brought all of those hospitals live on Epic. Well, Allscripts training consisted of PowerPoint presentations simply provided by Allscripts. So the training wasn't nearly as in depth or, or complex as our Epic training is. So Nortex, if you hired into North Texas, you get to work in one of our hospitals. It was simply here some PowerPoints, but there was a heck of a lot of OJT

Georgianna Kelly 2:36
going on. And we realized that that methodology simply wouldn't work with Epic. But there were no training rooms in North Texas. None. And if you know, I'm sure many of you been working in healthcare for many years, the most valuable thing in healthcare is real estate. So you can imagine as I go to these hospital leaders and say, Hey, would you like to pony up some space so I can develop a training room, you can imagine what the answers were, let's just say it was all in all in the negative. So my position was created to lead this combined team

Georgianna Kelly 3:16
taking from all of these parts. So you'll notice we started with 89. Members, we are now down to 63. Those 63 team members support 52,000 Epic-end users. So that's a ratio of one to 195, one to 125, one to 825. And as I like to remind my IS partners, their support ratio is one to 88. So they have one ISP member for every 88 end users. I have one team member for me 825.

Georgianna Kelly 3:51
And we really wouldn't be able to maintain that ratio without a, our robust computer-based training library.

Georgianna Kelly 4:01
So, Ryan, you want to go to the Oh, before we leave, Ryan, I'm sorry. Can you back up one more thing? Very important piece, you'll see our commitment statement in the, in the middle of this puzzle diagram. So each of us at Baylor Scott White Health is encouraged to have a personal commitment statement meaning how every day are we going to get up and, and in? What is our commitment to the organization? What is our commitment to, our to our members, and to our patients? So when we think about developing this team from three very different areas, and in some cases, different cultures, Texas is one state, that many cultures, how do we all come together as one, one of the ways was to develop our own commitment statement, and this is ours. We commit to empowering the adoption of EHR-related technology so that patients and members are treated by the most proficient users in the nation. Not provisioned. The Epic sense it gets a little gets a little tricky there.

Georgianna Kelly 5:00
We have our own proficiency continuum that we have developed, we're not going to cover it in this session. happy to chat with you offline about that. But our proficiency continuum then walks people through what happens after new hire training? How do we, how do we get to proficient? And how do we make, maintain your proficiency? So just wanted to point that out? Because that is, that is our professional Northstar. That's very important to us.

Georgianna Kelly 5:28
Alright, so to answer the question, you know, what is it that your team, what is it that your team does I get this question a lot, especially from, you know, from executives in the organization, who are new name, title, write new, new team names, have to be able to answer that question, you know, my elevator pitch? Well, our activities can really be divided into three broad categories. But we build and optimize the technical framework required to provide learning services and to deliver continuous learning and in user support. So last year, in 2020, we trained 37,337 team members, this includes new hire training, it includes cross-training during COVID, it includes some advanced training, so 37,337 End Users went through some sort of training that TLS provided. And again, with those kind of numbers, we wouldn't have been able to come close to those kinds of numbers without a significant uplift in our budget so that we could, we could add headcount and

Georgianna Kelly 6:40
go rent some training rooms.

Georgianna Kelly 6:44
All right. All right, next slide.

Georgianna Kelly 6:48
Alright, so what value do we, do we bring to the organization?

Georgianna Kelly 6:55
Really, again, proficiency is our North Star, professional North Star organization not going to come close to maximizing its investment in Epic without having proficient end users

Georgianna Kelly 7:10
You know, it's, it's much like having a Ferrari and keeping it in the garage, every organization after they drop hundreds of millions of dollars on Epic, their boards are asking, their C suites asking, great now what? Well, the now what is whether it's Baylor Scott & White, or really any Epic shop, it really all starts and ends with proficiency really without it, you don't have proficient users, you're not going to meet any strategic goal tied to, you know, tied to your technology are tied to predictive analysis.

Georgianna Kelly 7:47
We found that the more proficient our users, the higher our engagement scores, the better our retention,

Georgianna Kelly 7:56
the better able to standardize our workflows or processees. So for us, again, our success, that's our professional Northstar, that, that that's really where we are, where we are focused on. And one thing I do want to call your attention to, is with CBTs. We spend 30% less time in Epic new hire training versus instructor-led classes. And with the number of nurses that we onboard every year, we're easily giving seven figures well into seven figures back to the business, because we have quantified that, it's, it's easily 30% less time

Georgianna Kelly 8:38
with CBTs than instructor-led training, because we're not we're not necessarily pacing our classes to

Georgianna Kelly 8:47
you know, that, that, that end user who may be struggling, it's however, whatever your pace is, is whatever your pace is. So

Georgianna Kelly 8:56
obviously, our CNO, CNO is, is very happy with, with that figure.

Georgianna Kelly 9:05
All right, right.

Georgianna Kelly 9:07
Next slide.

Ryan Seratt 9:10
All right. So Georgie, on a couple of years ago, when you first started to convert to CBT, is you're in a little bit different place. And instructor-led training was the primary method across the organization. And many leaders on the call today might have some of the same challenges you did. Can you tell us about what really prompted the, you looking into alternative solutions to the instructor-led training classes? Sure.

Georgianna Kelly 9:38
We'll get to the next slide. Ryan, if this is our Rhode Island slide.

Georgianna Kelly 9:47
This is our Indiana slide. There we go.

Georgianna Kelly 9:51
So all of the blue that you see in the state represents our service area and if you would

Georgianna Kelly 10:00
You know, with the exception of, you know what's in West Texas, if you really just focus on what's central Texas and East, that geography is either 30 Rhode Islands or one state of Indiana. So if you can imagine, it's just not feasible to, you know, send a trainer to, you know, get from Dallas, Dallas to Austin, I mean, that person's going to have to pack a lunch and dinner because our roads are always under construction number one,

Georgianna Kelly 10:31
number two distance. So geography was a significant business driver in our decision to, to transition fully to new hire computer-based training modules.

Georgianna Kelly 10:47
And then, really, because of our again, because of our size, and because of the number of facilities and the number of clinics, we have the ability to deliver a consistent learning experience for new hires and new affiliates.

Georgianna Kelly 11:04
We didn't at the time, have an established super user program. So we really couldn't leverage super users to help define that consistent message. We have big geography. And we wanted to ensure as we, you know, as we all come on board with Epic, that, that there was a consistent message. So if, if you are in a clinic in, you know, South Boston Horseshoe Bay, you're hearing the exact same message that you are, if you're in a clinic in Paris, Texas, same same, exact same. And so the hope, then, is that you, you at least initially, understand, here's our common workflow. Here's how here's how our instance of Epic works before you

Georgianna Kelly 11:54
dive into your, into your job, if you will. And then again, I mentioned it before. So first one is geography, second ability to deliver a consistent message. And then third, real estate, no budget to build out training rooms, nobody was going to give me training rooms, any facilities, no budget to add to headcount.

Georgianna Kelly 12:16
And, again, even when we were; had training rooms in Temple, Texas had training rooms in Dallas, even that geographic spread was just a bit too much for our, for our end users. And our North Texas ambulatory arm made it very clear, we have CBTs, we're keeping CBTs we are not, we are not straying from, that you are not taking our CBTs away from us. So really, all of, all of those factors, those four factors were our initial impetus to

Georgianna Kelly 12:50
going down the computer-based training route, full time, full board.

Georgianna Kelly 12:57
Yeah. And then

Georgianna Kelly 12:59
efficient use of resources, some of you may have, you know, had budget reductions in the midst of COVID. I know I did, I took a 20% hit to my budget. But even before then, again, we were, we were driving towards, we were driving towards efficiencies.

Georgianna Kelly 13:18
So how do we, how do we take scarce resources, resources that,

Georgianna Kelly 13:25
you know, again, I wasn't going to get a budget increase. So how do I, how do I, in essence, wring as much efficiencies as I can out of my existing resources,

Georgianna Kelly 13:35
and that certainly, was again, was not going to be delivering in-person training. On top of that, we are on a quarterly release cycle. So the pace of change is rapid. And initially, I thought, oh, you know, Epic says we're not going to have that many changes. It's, it's, you know, they're going to be there not going to be that many. That has not been my, that has not been our, that has not been our case. And I would venture to guess it's not yours as well. So in the ambulatory space, for example, we routinely have 400 changes every quarter 400 changes, 400 changes. So how on earth, you know, you keep up with that in terms of not only getting your new team members up to speed but your existing team members up to speed on that many changes occurring every, every three months. So

Georgianna Kelly 14:32
again, we find it much more efficient, to be able to leverage our CBTs whether that's for new hire training, and in some instances for

Georgianna Kelly 14:45
quarterly update training, special updates, whatever the case may be,

Georgianna Kelly 14:52
to, you know, to help him, that with that communication, then back to our, back to our end users.

Georgianna Kelly 15:03
So our solution really couldn't be

Georgianna Kelly 15:08
about, you know, what's best for Technology Learning Services in the midst of, you know, a budget cuts, really, again, our professional Northstar is everything we do, every action we take needs to lead to either getting the end-user proficient or, or increasing that proficiency. But again, balancing, balancing the tools that we have at our disposal to, you know, to make all this make all this happen. So really, we need something that's scalable, and flexible, scalable, so that, you know, this summer we plan on, you know, bringing on, you know, close to 1000, nurses, okay, I need something scalable, so I can meet that demand and really meet it without, you know, with, without any consulting dollars, made it without encountering any bumps in the road. Again, computer-based training offers that see me because really, in a matter of minutes, I can have all of that curricula, as long as I have a clean list of end-users, I can have all of that curricula

Georgianna Kelly 16:11
assigned. Another thing I would point out is that during COVID, we were able to develop 31 cross-training modules. And we really did it, we did this in a matter of hours. And we did it before the business even asked for it. So as we started shutting down, surgeries, we realize pretty quickly that they're going to want to redeploy those nurses to other areas of the hospital, tough to take an OR nurse into the ED

Georgianna Kelly 16:41
a little bit easier, so to speak, to take that, take that OR nurse and, and you know, send him or her to a med surge floor. So that's really where we, that's really where we started. And then, on our own, we were able to identify 20 user roles that we had high confidence that the business would say, okay, we need to redeploy these, these resources, and then another 11 use cases were identified in the course of our

Georgianna Kelly 17:12
you know COVID. And then COVID, recovery, recovery in terms of a business model. So, again, I can't tell you how many Epic clients I've heard from during, during that period, you're saying Holy smokes, what are you? What are you doing? How can you know, how can we be as nimble? And how can we be as responsive and really, at that point, if you don't already have something developed? I really didn't have a good answer for them. But I know COVID has definitely sparked a lot of, a lot of organizations to start thinking about, you know, moving to a full CBT model, or at least a blended model.

Ryan Seratt 17:55
Speaking of the different models that are out there, you know, not all I hear a lot of e-learning a lot of virtual learning. And really, that can mean a lot of different approaches. I think that when you know when we're talking with your computer-based training model, that really it's a highly interactive scenario-based online learning experience. So it's not passive. It's not watching a PowerPoint online, people are actually in scenarios, performing duties that they would on a day-to-day basis. I think that's important to point out. And then you would also mention a little bit about blended learning when we were talking earlier.

Ryan Seratt 18:36
And it's not a full blended approach, that different departments use it a little bit differently. And they blend based on their, on their own needs, and what makes sense in their area. Can you tell us a little bit more about that?

Georgianna Kelly 18:50
Yeah, so all of our providers are in a; on a full 100%

Georgianna Kelly 18:56
computer-based training for their new hire or new, new affiliate.

Georgianna Kelly 19:03
And we only employ 39% of our, of our provider. So you know, 61% are affiliates, meaning they have a choice. They have a choice to practice in one of our facilities or, or go somewhere else.

Georgianna Kelly 19:19
And some of our providers are coming here from other states, actually.

Georgianna Kelly 19:26
Apparently, we're hiring a lot of providers from New Jersey, because I'm, you know, I'll get a couple of phone calls from providers who are in the credentialing process. You know, they're there in New Jersey, and they're able to complete their CBTs, they want to do it at 2am-3am, whatever the case may be, and the feedback I've heard is, oh, this is awesome. I can get it done. And then you know, I'm in essence ready to go hit the ground running. So that's for our providers, full CBT. For our, we partnered most closely with nursing education, so nursing education,

Georgianna Kelly 20:00
We'll take, take the curriculum, the computer-based training curricula, and they have decided how at what pace our new hire nurses will consume that content. So they'll do a little bit of CBT, they'll stop, they'll place those, they'll place what that nurses learned in the clinical and clinical context, maybe do a skills assessment, and then they'll, they'll, they'll start another CBT, placing clinical context, do a skills assessment, that's what works best for nursing radiology is, is the same way. So it really, you know, the, the benefit that we have found with CBTs is that it is grounded in simulations, it can be broken up very, very easily to meet the needs of all of your stakeholders in the business. And now we're empowering nursing education, to say, you know, take our content, consume it, as you will consume it at your own pace if you need to, if you need to, you know, rewind and do something, again, you're free to do that. In an instructor-led environment, we would have gathered all those nurses for a week FHIR those, right, you're gonna learn everything we need you to understand about Epic, then you're going to take a test, and then you're going to go to your facility, or then you're going to go to your clinic, and you will have probably forgotten 90% of what you learned. So now we're gonna have to, you know, retrain you.

Georgianna Kelly 21:35
So that's, that's how we found that the blended learning works, works best. But really, it's not me telling the business, here's how you're going to consume my training, it's me telling the business, this is available to you, you the business then decide what's, what's best for you. And the business has been very, very receptive to that approach.

Lisa 22:03
Lisa, from South Coast, can I just ask a question? So you,

Lisa 22:08
so for at South Coast, we don't allow access to Epic until people are trained. So are you saying that you allow open access without any

Lisa 22:17
I see, shaking your head

Georgianna Kelly 22:18
No, they still have to complete Epic training, they still have to complete all of the curricula. It's just they don't have to complete all of that curricula,on my timeline. They complete that they complete their curriculum then on, you know, nursing education's timeline

Georgianna Kelly 22:37
for radiology's timeline.

Georgianna Kelly 22:40
On the provider side, in order to be credentialed, you have to have completed your Epic training. So that takes care of providers but in terms of nursing,

Georgianna Kelly 22:50
you still have to complete your Epic training, you stopped a passing assessment, it's just, I'm now not telling the business, hey, give me all your nurses for a week, we're gonna run them through Epic, they're going to pass an assessment. It's, it's nursing education now in charge of, and, and really, you know, driving that bus to say, we're going to have these nurses for, you know, 10 days. Why do they need to pass their Epic training on day five, when they're not really even going to be hitting the floor until, you know, day? 11?

Lisa 23:24
Okay.

Lisa 23:26
Okay, I got it. Yeah, we use okay. It's just, you know, we have a hard time with that getting buy in from, from operations for that, because most of the time, they want them to have full access right away, which I guess is a limiting factor, which it seems that you've overcome. Yeah. Okay. Thank you.

Ryan Seratt 23:47
So going back to, you know, the, the projects that we're launching, there was a couple of different roles. Can you tell us a little bit about the way that we define the roles that would be involved in the conversion to CBT, or computer-based training? Yeah, and I will say this, this wasn't, this wasn't our initial thought.

Georgianna Kelly 24:08
You know, as far as Ryan can tell you, we went through a couple of iterations until we, you know, really figured out how to make this process.

Georgianna Kelly 24:20
But first are your subject matter experts, these are, you know, credential trainers end users.

Georgianna Kelly 24:27
I would say for us, you know, some, some folks on the informatics team, for sure. And then the CBT authors, you want folks who are, you know, definitely familiar with that Epic, definitely do not need to be certified in Epic. But the higher degree of computer literacy, I would say the better off that you're going to be in that particular bucket. And then instructional designers to really put that instructional design wrapper around it in terms of intros, you know, the pay

Georgianna Kelly 25:00
The sequencing,

Georgianna Kelly 25:03
really all of, all of those factors we have

Georgianna Kelly 25:07
three, no we have four people on our team,

Georgianna Kelly 25:11
One of whom has a PhD in instructional design, the other three have their master's degree in instructional design. So we were very fortunate in that, in that regard, to have, to have that knowledge on our team.

Ryan Seratt 25:29
And one of the keys, I think, the and as you were saying that, you know, looking back on it, you know, we learned a lot through the process. And sometimes it wasn't moving as fast as we thought it should. And, you know, there were definitely changes that we made. But ultimately, I think this is one of the things that made that large amount of work 144 courses, in that six month period we had worked is that those roles were really it was kind of an assembly line.

Ryan Seratt 26:01
You know, Nick Edwards, like from your team says that it's the assembly line that made it possible.

Ryan Seratt 26:07
And this is really what it looked like, is, with the tools that you had implemented, there was something called the Rapid Recorder. So the subject matter experts were able to make the initial recordings for the simulations. And having them do that. And they're the closest to the work. So having them capture, I think, was really a large factor. And then that was passed on to the, the edits, where we started going through the simulation edits, which definitely is the longest process in, in this, in the course design, putting the bubbles in the right place, making sure that someone can complete those functions themselves, we actually did three modes, where you know, they actually watched a demonstration than they could do one without or with on-screen help point them to the right areas, and then point, and then those games were gone. And I had to complete it without the on-screen help. So we had those three different levels we put in initially.

Ryan Seratt 27:09
Then we assembled the courses with the instructional designers, like you were saying they're the people that took all the pieces and put it together, made it make sense, kind of like a producer of a movie is really how does all this work together. And so that the learner has a good experience. And then there was the review process. Also the review process. And Nick will say that, that took longer than expected, where, where you actually have to go through the entire course, making sure that everything is perfect. So that the end-users have that great experience.

Ryan Seratt 27:43
Everything was done. And as it finished, it started passing through this process. So it took a little bit of time, just to get ramped up. If you're going through a process like that, be prepared for that. Before things start coming out the end fully assemble, it does take some time to get that, the flow going.

Georgianna Kelly 28:02
Yeah, you're correct. And I would have a countdown on my whiteboard. And you know, it stayed at 144 as total number of courses we had to convert and we had a we had a fixed date because our fixed date was the point at which we had to have a computer-based training, new hire solution ready to go because we were bringing on our largest facility in North Texas live on Epic. And so

Georgianna Kelly 28:32
the week ago live, they hired, they had new hires. So we had to be ready to go, went live on a Saturday, that Monday, we had to be ready to go with new hire training. And so it was

Georgianna Kelly 28:46
I would not recommend trying to do this in six months. It, it wasn't enough time. For sure. But

Georgianna Kelly 28:56
you know,

Georgianna Kelly 28:59
hats off to 314e for they were, they were, you know, willing and more than able to, to make that journey with us and found some excellent, excellent people that literally we would, you know, throw work to Ryan and he would package it and give it back to us. And

Georgianna Kelly 29:19
so, yeah, Ryan, let's not ever do that again.

Georgianna Kelly 29:24
In that amount of time. That was, that was definitely stressful.

Ryan Seratt 29:29
Yeah, I could, seven months would be great.

Ryan Seratt 29:34
I'll take nine if, if possible.

Ryan Seratt 29:37
So here's a little bit on just the timeframe that we were looking at and just to talk a little bit more about the work that Georgiana is talking about. So the team did 3303 workflow recordings in 100 day time period. So we also edited each one of those and assemble them

Ryan Seratt 30:00
To that multilayer sequence training that was hands-on, and we put it all into 144 courses, and the 44, that was actually towards the end of the project, is putting it all together. And we did that in 40 days, there was a little bit of nail-biting there. But we were able to accomplish it and make that happen.

Georgianna Kelly 30:27
Yeah, so we use it, uPerform®. And so as part of uPerform®,

Georgianna Kelly 30:34
you get Rapid Recorder,

Georgianna Kelly 30:38
and so in the midst of bringing our largest facility live on Epic, implementing uPerform® and uPerform® is not a

Georgianna Kelly 30:49
it's, it's not necessarily intuitive. It's gonna take if you're thinking about uPerform®, it's going to take your team months to get comfortable with it. I still have team members today who will tell who will tell you, they're not necessarily all that comfortable with it.

Georgianna Kelly 31:05
So we had new technology uPerform®, we had massive conversion, plus the team was getting ready to take our largest facility live

Georgianna Kelly 31:16
in Epic. So it was definitely there were definitely some added stressors in there. This in itself would be stressful, but we had added stressors.

Ryan Seratt 31:28
So we have the question on in the chat room,

Ryan Seratt 31:32
to the sneeze recording the workflows, that screen capture recordings or audio recordings.

Georgianna Kelly 31:39
So yeah, it, it was screen capture initially. And then we identified some folks who

Georgianna Kelly 31:48
we refer to collectively as the voice. And so we had folks who had, you know, just good voices for, you know, for some of these recordings, some we just had to use the, you know, the artificial voice, the robot voice, simply because there were just there are too many courses in order to do the human voice for all of them. I will say we just signed with Amazon Polly

Georgianna Kelly 32:14
in their neuro voices sound, almost human-like

Georgianna Kelly 32:21
it's incredibly low, incredibly low price point. So if any of you are looking for a cost-effective solution, and you know, you don't want to have to do the human voice, then I would suggest you look into Amazon Polly.

Ryan Seratt 32:41
And Google also makes a similar product that is, and those voices have drastically improved over recent years, they're, they're near human quality now.

Ryan Seratt 32:54
But for the initial recording, so the tool that you were using uPerform®, allow the subject matter experts to make those recording. If you're using a different tool, like Articulate or Captivate, then the way that we go about capturing that is we do a live WebEx session, and where the client actually walks us through their training. And then, are the 314e instructional designers then record that session, and then we go back and we replicate that instructional designers capture with their, with their tools, Articulate Captivate like Torah, inside the tool later on.

Ryan Seratt 33:41
Go, so.

Ryan Seratt 33:44
So moving to computer-based training really impacted the work that your team was able to do Georgianna and took some of the, I would say the day to day tasks freed you from those so you can work on a little bit deeper training work. Can you talk about that? I think that was a big learning for us.

Georgianna Kelly 34:05
Yeah. So we were, we are able to take many of our team members.

Georgianna Kelly 34:14
And I will say this well, I will caution anybody who's thinking of making this transition, that if you hire trainers, you most likely have hired people who want to be around people. They want to interact with people. So we did lose team members who said I don't want to sit with my earbuds in and just do technology. Fair enough. So some of those folks were able to you know, find them positions within the organization, others, other chose to leave. But for those who have a penchant for the you know, for the technology, embrace it, you know, dive deep into it. They they absolutely love it and

Georgianna Kelly 35:00
So that's been, that's been great for our team when I had to lay some folks off.

Georgianna Kelly 35:07
You know, common refrain was, you know, that I've heard from people is, yeah, don't worry, I'm gonna, I'm going to get a job really soon. Because I know, I now know how to create computer-based training modules. Amazon in Austin, Microsoft in Austin, scooped up our people, I don't even know if they had more than a five-minute interview in our people were scooped, up, up here in Dallas, you know, scooped up, our person, I think Neiman Marcus scooped up are another one of our folks. So

Georgianna Kelly 35:44
that was, you know, those things, stay with me, you know, feel responsible for my people when I have to lay him off his things, you know, those things stick with me. So that, to me is a positive that that will stay with me except like who, you know, you guys found you guys found the job. But

Georgianna Kelly 36:02
the opportunity now to not have people in the classroom. And now we can, you know, take a deep breath and say, Okay, what advanced learning opportunities are out there. So we partner closely with nursing and

Georgianna Kelly 36:16
blood administration, we have an advanced learning pathway on that. And we've really taken our provider personalization, which Epic now cause user setting sessions. And we've really enhanced that into three distinct sessions that are incredibly detailed, incredibly meaty. And so again, I just wouldn't have had the bandwidth to really focus on more of those advanced learning opportunities. If, if my folks were, you know, we're in the classroom every day, so and hands down, our physicians love our CBTs. I mean, they might not like having to go through them. But if training is a requirement, they absolutely love that they don't have to come into a classroom and that they can FHIR up your laptop at 2 am and complete them that works best in their schedule.

Ryan Seratt 37:14
And lessons learned.

Ryan Seratt 37:18
Always one of the pearls of wisdom.

Georgianna Kelly 37:22
Yeah, besides, you know what you have here on the screen, I will say that

Georgianna Kelly 37:27
I hear from some

Georgianna Kelly 37:31
some CIOs who, who are interested in making this transition. And, and understandably, one of their first thoughts is awesome, I'm going to be able to cut headcount not necessarily, not necessarily, especially as your as you are getting, as you're getting this platform stood up. And then if you have uPerform® again, it's, it's a meaty, complicated product. And it does take folks to keep it up to date. And I will tell you a tale of caution perhaps. So we've, we've converted everything. Ryan and team did an awesome job, got everything ready to go in September. And then I called Casey in January, I think it was January, Casey. And I said, Guess what, we get to do it all over again because a storyboard. So we converted. And then when storyboard came along, we converted again. And now I hear that Epic's going to change in basket, they're going to be some substantial changes to the end basket won't impact us as much as storyboard. So that's a bit of a tale of caution that, you know, always have in your budget when you can, consulting dollars to, you know, to help with these kind of

Georgianna Kelly 38:51
evolutions, because that is a heavy lift when you have to, you know, when you have to

Georgianna Kelly 38:58
totally redo your CBTs. But hopefully, they're not going to be that many more things coming out of, coming out of Epic.

Georgianna Kelly 39:08
Yeah, those are besides what's listed here. Those are, those are my two biggest, you know, lessons learned and, you know, attract a team that is really, really going to whatever technology you have, that they're really going to embrace it and you may need to be looking for new skill set, and it's not necessarily

Georgianna Kelly 39:30
you know, the person who has worked as a nurse for 20 years and then makes the transition to training. You may be looking for more of the technology folks and then partner with informatics to really bring in that you know, the, the clinical pieces that you need.

Casey Post 39:51
Well, great session, as Ryan, you have on the screen here if anyone has questions, by all means, please, please Chime in.

Casey Post 40:00
I do have a question for you Georgiana, what, what would you say was one of the biggest challenges, either for you and your team to make this transition to CBTs? Or for the organization to make this shift to from in-person training to computer-based training?

Georgianna Kelly 40:19
For, for me personally, it was, it was the timeline, you know, six months, definitely just, it wasn't enough time.

Georgianna Kelly 40:28
So I would absolutely not recommend that. For the organization,

Georgianna Kelly 40:35
We do still have some hiccups,

Georgianna Kelly 40:38
specific to assigning curricula for our affiliates. So in our AllScripts world, again, it was, it was, it was really easy. Medical staff services would just literally send that, send that, attach PowerPoint to the providers and say, Here's your training, answer these questions. And, you know, away you go. So now Medical Staff Services has to get into our LMS and, and assign training. So that's a bit of a hiccup that we're, we're still trying to work out, I didn't necessarily foresee that as a challenge going into it. But

Georgianna Kelly 41:18
that, that is indeed a challenge,

Georgianna Kelly 41:22
then

Georgianna Kelly 41:24
just socializing the change in every communication channel or every, you know, pocket of the organization definitely takes time. So having a robust communication plan that really outlines not only the what, but the why, and the benefits and what what is, you know, what would your business partners expect to gain out of a computer-based training solution versus, versus in person? And, you know, some of my business partners said, Yeah, I don't want that I want in-person training, I don't want I don't want anything to do with CBTs. And their reluctance was because, hey, look, I don't have space for these folks to complete CBT. So our solution to that is, we do offer in Temple and in Dallas, we do offer, if you want to send your folks we do have a space, we will proctor them. So you know, you can, you can send them. We do offer virtual proctoring, which is basically a team's you know, either chat, or they can call in. And if they have, if they have a question about content, or even if they have a question about how do I, you know, how do I log into my LMS? What am I doing? Those kinds of questions, so, just be ready to support, you know, support the business, be nimble, be flexible, that kind of thing.

Ramon Meiena 42:54
Great thing, I have a question. Ramon, Northern California, I'm, I'm spearheading a pilot program for launching CBT training.

Ramon Meiena 43:04
And they're not on board on it yet. I'm still creating one that's

Ramon Meiena 43:11
more to more toy department that they would like to see it with. I've already created some for laboratory.

Ramon Meiena 43:18
But they want to see when creative radiology My question for you is, what style CBT Have you felt has the biggest impact?

Georgianna Kelly 43:28
Definitely simulation. Okay.

Ramon Meiena 43:32
I am with that because I went down the same route. Currently, I build mine out of Captivate, and they're fully interactive, and every which way possible that it can be.

Ramon Meiena 43:44
I'm now trying to tinker with fully interactive and game-based trainings. Nice. Nice. Yeah. So with that being said,

Ramon Meiena 43:56
Captivate isn't really known for game-based training development. So do you have software that you would recommend to tap into that?

Georgianna Kelly 44:07
No, I wish I did. We're on the lookout for the same thing. Because

Georgianna Kelly 44:14
you know, that, that's been one of the challenges I've posed to my team is okay, what's next? So we, you know, we have this and we, I think we, I think the, the industry agrees that you know, gamification can be a very effective tool in training, but I don't know if the, if the industry really has a good platform, to then you know, create something that you know, that can be easily gamified. So, no, but if you find something I would be very interested

Georgianna Kelly 44:54
to know what that answer is.

Ramon Meiena 45:00
Okay, last question; with, with developing these CBTs… And never mind, I lost my train of thought.

Ramon Meiena 45:11
There that went.

Ryan Seratt 45:15
Right, almost looking for that question. Then we do have another one in chat is did you see physician engagement scores improve after conversion as proficiency with Epic equal to IoT instructor-led training?

Georgianna Kelly 45:32
So, yes, on the Yes. On the on the

Georgianna Kelly 45:38
physician and a PP side? Yes, engagement scores

Georgianna Kelly 45:45
have increased, not wildly so. But yes, they have increased. And then Ryan, what was the second part? Sorry,

Ryan Seratt 45:52
speaking, proficiency with Epic equal to instructor-led training.

Georgianna Kelly 45:58
So we define proficiency

Georgianna Kelly 46:01
differently. So if you're if you're talking about Epic version of proficiency which you get out of signal data and what you can get out of signal data for,

Georgianna Kelly 46:11
you know, for ambulatory physicians, we, we do pull that data. We don't know that there's a core, we don't know that there's is causation there. But yes, signal data for providers, a PP's and physicians going through CBTs? is

Georgianna Kelly 46:30
higher, better ever way you want to look at that. Then our providers who went through instructor-led training for our North Texas Epic implementation? Yes. Their data I saw was 16% improvement in the signal scores. And it's a pretty low sample size, that it's not statistically significant. But that it is suggestive of performance. Yeah, because we're just looking at our providers. So for Nortex's Epic implementation, we did provide instructor-led training. And so we have that cohort. And now we have providers who are joining the organization, whether they're, well, on the ambulatory side, they're employed. And so now we're we can do, do that comparison. But to Ryan's point, yeah, we, our sample sizes is a bit low.

Ramon Meiena 47:28
I remembered my question.

Ramon Meiena 47:31
In regards to assessment.

Ramon Meiena 47:35
Since you're doing CBT, are your assessments also scenario-based when they're going to have to complete it? Or are they at the end of the CBT? Do they have a questionnaire?

Georgianna Kelly 47:49
So on the, on both, and so I'm going to give you two answers my pre-COVID Answer and my COVID Answer. My pre-COVID answer is, we developed something that we internally call robot grader said, you know, what Epics auto grader is? No,

Georgianna Kelly 48:07
Well, there's not a whole lot of auto in the auto grader. Spoiler alert. So it takes a lot of, takes a lot of time, a lot of time to build out an auto grader, I'm happy to chat with you and anybody else offline who's interested. So we've taken we've taken the auto grader

Georgianna Kelly 48:29
concept, and we've created Robo grader. So if you want to test out, you would you would have Robo graders assigned to you. So for example, we know that our providers in the in the inpatient space struggle with our discharge process, because our discharge process isn't really aligned with our competitor in North Texas, even though we're both using Epic. And so when they practice it, both places, they're, you know, they don't, we know that's a stumbling block. So we have a robo grader, that's basically we, we've taken one of our training environments and converted it into a robo grader, you're hitting that training environment, and we're giving you just enough information, we're not leading you through, or we're giving you just enough information that you would need to then in this case, discharge, discharge the patient. And then within a matter of minutes, it's graded, it's returned to you, it's returned to your manager. And step two, and that version two, and that process is going to be now we're going to attach tip sheets. So if you're, if you're a nurse, and let's say you passed your robot grader, but just barely, and you're, you know, we've identified some knowledge gaps. Now we're gonna, we're gonna attach some links to some tip sheets, you know, to help fill those knowledge gaps before you, before you hit the floor. That's my pre COVID Answer. Post or during COVID We went back to just our multiple choice exams because our nurses are completing some

Georgianna Kelly 50:00
they're training at home. And we were just getting too many too many questions about Citrix. What am I doing? You know, we just couldn't support, we just couldn't support that number of questions. So we were going to return to our Rober greater version in July of this year. And when we expect those nurses to complete all of their training in, in a nursing education room somewhere, so

Ramon Meiena 50:30
Are your trainings housed in

Ramon Meiena 50:33
within Epic as well, like so after a person has the assessment, and they get access to Epic? Do they have an area in Epic to go back and retake those if they wanted to?

Georgianna Kelly 50:47
No, they could get into their, Cornerstone is our LMS. And so they could always, they can always retake them in LMS. But, but now we have integrated,

Georgianna Kelly 50:59
we've integrated uPerform® into Epic.

Georgianna Kelly 51:03
So it's it's very easy for end-users to access all of our training content within Epic.

Casey Post 51:13
Yeah, well, we are. We're at the hour and so

Casey Post 51:18
I need to meet with you offline and just walk you through what we've done. Okay. I'll be I'll be happy to make that connection for you, Ramon, and, you know, for everyone that that joined us. Thank you for attending. Georgiana and Ryan, thank you both for your time. I really enjoyed the conversation here. And it's at your agreement, Georgiana, I can connect folks to you if they have follow-up questions. Absolutely. Wonderful. Well, thank you all, and have a wonderful afternoon. Thanks, everyone.