Everything You Need to Know About On-Demand Evaluations

The need for more assessment data, collected more frequently, to inform and track student performance and progress over time continues to increase. Combined with the COVID-19-influenced concern over sharing personal devices, we knew that we had to prioritize a solution. That is why DaVinci Education has created a new, flexible assessment tool called On-Demand Evaluations. 

Below, we explore some of the common barriers to gaining accurate assessment data, and share how this new functionality can help overcome those challenges to get you the data you need, when you need it.

Q: We need more data gathered over time in order to make informed, summative decisions on entrustment, graduation, etc. How can we easily gather more assessment data, more often?

A: With Leo, you can collect both summative and formative assessment data at the same time. In addition to summative evaluations scheduled at the end of a course, On-Demand Evaluations are a great tool for gathering assessment data at any time, in any environment. Since students can request to be evaluated, they can do so during or after any clinical situation - allowing them to seek feedback on a particular type of interaction or procedure. Students also can see all requests they have made, identify gaps where they may need to request additional evaluations, etc. 

Faculty can easily complete evaluations of students from any device. Each evaluation request sends a unique email link, bringing faculty directly to the evaluation to be completed. If there is more than one request available, preceptors can complete and submit multiple evaluations from the same link, just by clicking on the Next Evaluatee button. Additionally, there is no need to wait for a request from a student. Faculty who have access to the system can complete an On-Demand Evaluation at any time with a click of a button in the Eval Portals. This information can then be used for formative feedback, mentoring or coaching, or as summative data for student performance in or across clerkships for entrustment decisions. 

Q: COVID-19 has changed our practices - in order to help mitigate spread and maintain social distancing, we are no longer sharing devices or papers for evaluations. How can Leo help us maintain social distancing and still get the assessment data we need?

A: With Leo’s On-Demand Evals, there is no sharing of devices, no paper to carry around (or try not to lose) - no need to get close at all. Students can request to be evaluated at any time, in any location, straight from their personal device. Evaluators will immediately receive an email with a unique link, allowing evaluations to be completed in the moment, directly after a clinical encounter or from the comfort of home. Scheduled evaluations work in the same manner, with a unique link via email, and can also be completed any time, anywhere. Stay safe and socially distant while still getting the assessment data you need. 

Q: We have preceptors at locations across the state, and trying to give everyone access to a new system that they have to learn and remember a password for is a headache. How does Leo address this?

A: Leo helps you lessen the cognitive load for your clinical faculty and preceptors - no password or training required. If they can access and open an email, they can complete an evaluation. Leo sends an email with a unique link for each evaluation with no need to sign in or learn a new system. Complete and submit multiple evaluations with just a few clicks or taps - all at once or come back later to finish the rest. We have also created this overview video and also have Leo User Guide articles for you to share with your faculty and students. Less work for your evaluators, no headaches for you.

Q: Some of our community preceptors and clinical faculty don’t check their official email, preferring to use a personal account. How will they receive evaluations and reminders?

A: Leo has an option to store more than one email address, including a preferred or personal one outside of the official school or clinic associated email. This information can be updated in a user’s profile at any time, allowing evaluation emails, announcements, reminders, etc. to be sent to the preferred email on file. 

Q: Many of our clinical sites have firewalls or use outdated browsers, will our preceptors still be able to access the evaluations?

A: Yes! Evaluation links are sent via email to a preferred email address, so preceptors can complete them on personal devices, using wi-fi or cell data. Additionally, the Evaluation Portal is mobile friendly, making it easier to complete and submit and track evaluations from personal devices. With Leo, preceptors and faculty are not reliant on a desktop browser and do not need a password. As long as they can access their email, then evaluations can be completed.

Q: We often find that our preceptors can’t evaluate students in the moment, but then too much time passes until they can. Sometimes they forget or don’t even know the name of the student they were observing. How can we be sure of accurate data in these cases?

A: With Leo’s On-Demand Evals, clinical preceptors can complete evaluations at any time. Evals can be completed immediately following an observation, or even in the moment (if their active participation and support is not needed). Evals requested by students also include important information like name, location and date of the encounter, that can help jog memories.

Q: It is a lot of work to try to track down clinical preceptors who haven’t completed their evals and to send them reminders. Can Leo help with this?

A: Absolutely! For scheduled evaluations in Leo (i.e. course evals, end of rotation summary evals, etc.), you can schedule reminders to go out automatically based on your scheduling and frequency preferences. The system will send these reminders only to those who have not yet completed the evaluations, saving those who did complete them from being bombarded with unneeded messagest. There are also multiple reports where you can track completion for a given rotation or site. Since you can send a reminder at any time, even outside the scheduled frequency, these reports can be used to check if that additional reminder is needed. For on-demand evaluations where students are requesting that a preceptor complete an evaluation on them, students can also track completion and send reminders - giving the power and responsibility back to the students. Additionally, administrators can see a list of all evaluation requests for a particular course section.

Q: Our students are smart, that is why they are here. But sometimes they are too smart for their own good and try to optimize their assessment data by choosing only days, preceptors or situations or where they felt that they performed well. If they are in charge of choosing when to request an assessment, what can be done to ensure that the assessment data is reflective of their performance and not just on their best days?

A: On-Demand Evals can be easily integrated into existing assessment workflows, as an additional data collection instrument for summative assessment decisions or as a formative assessment tool to provide immediate feedback to the student. Institutions can also create guidelines for the use of On-Demand Evals, such as setting a minimum number of assessments that must be completed in a given course or rotation. When requesting an evaluation, Leo prompts students to choose from a list of clinical faculty associated with that course - a built-in reminder of different evaluators. Additionally, administrators have the ability to track both requested and completed evaluations, as well as utilize Leo’s Evaluation Workflow to review, summarize and submit completed assessments. This process can be utilized as an additional check to ensure diversity of evaluators, etc.

‍Q: It is important to us that our students can see the evaluation results in order to make adjustments and keep improving. We also want advisors to have access so that results can be utilized in coaching. How can students, advisors and administrators track evaluation results?

A: Summarized evaluation data is available in the Academic Portrait, which will show evaluations on a specific student, as well as evaluations they have completed. This can help in tracking as well as utilizing summarized eval data to learn from. Advisors and other coaches can be provided access to a student’s Portrait as a Mentor. They must be given explicit access, which in turn protects confidentiality while still supporting the information sharing necessary to support students. 

Please contact us to learn more about On-Demand Evaluations. We would be happy to discuss with you how best to implement this valuable and flexible assessment tool at your institution as a complement to your existing evaluation protocol practices or as part of a new strategy.