Basics

The new Checklist Prefill feature provides a checklist prefill solution for faculty using AI-based analysis of transcribed interactions (such as patient-provider simulations).
It aims to assist teachers in completing checklists for individual students during video reviews.

Using the transcript text as a basis, the AI attempts to answer questions and provide recommendations - which teachers can accept or ignore.

Example:

If a student greets the SP during an encounter, this will be evident from the conversation transcript.
Let’s say the student greeted like:”Good morning Mr Wolf”.
If the FON checklist includes a question asking whether the student greeted the SP, the AI could automatically provide a "yes" response or suggestion.

This tool significantly

• reduces the workload of faculty

• reduces the time and effort needed for human reviewers

• automatically maps specific sentences from interactions to relevant checklist items

• maintains accuracy.

By structuring responses in this way, the application produces clear and consistent outputs for evaluation purposes.

AI Prefill is available to admins and faculty members (all users who previously had access to data entry functions).

Pre-Conditions for Prefill Generation

The Checklist Prefill feature only activates when the following preconditions are met:

o A video has an associated transcript.

o The video has an associated FON checklist.

o The Checklist Prefill Availability setting is enabled in your LearningSpace instance (System Manager).

Enabling Checklist Prefill

The Checklist Prefill Availability setting can be enabled in the System manager (Local settings) by Admin or System Manager privileged users.

Once the admin user has enabled the Transcript and then the AI assistant (Checklist Prefill), the features becomes available to users (based on their permissions).

Once Checklist Prefill is globally enabled, the system automatically generates a Prefill for the FON checklist related to the relevant videos IF:

• a video has an associated transcript

• and an associated FON checklist.
  
  During this process, the user is presented with feedback about the status of the generating process - and can use the Refresh button as well.
  
  

How does AI Prefill work?

1. Once a video recording is completed (when the user clicks the stop button), the transcript begins generating.

2. Immediately after that, the AI starts evaluating the answers.

3. Once this is done, both the Transcript and the AI assistant become available.

4. The AI assistant is visually represented by the color purple throughout the LearningSpace.

If a recording has an AI assistant, a purple box appears below the top row on the Video Review/Data Entry tab. This box contains a brief description and a toggle that the user can use to reveal the AI's answers.
If the AI has not yet run or the transcript has not been generated, the toggle will be disabled, and a hover message will indicate that the system still needs some time.

Faculty users can toggle the Prefill for a given checklist at the top of the checklist during Data Entry.

Enabling the AI Assistant: By toggling the switch at the top of the screen, the AI becomes visible.

Users can toggle the AI on and off at any time. If you manually accept the AI's answers, these answers will remain in the input field even if the AI is turned off. The answers can still be edited, but all purple elements (indicating AI suggestions) disappear from the screen.

Jump to Source

The AI makes its recommendations based on the transcript, specifically on what was said in the video. These can be considered evidence that the user can verify. The Jump to Source function provides timestamps that the user can click on to view this evidence. This could be a single timestamp or multiple timestamps.

Confidence Level

The AI can score the confidence of its answers. This allows faculty members to know whether they need to double-check a particular question.
There are 3 categories, indicated by stars: ⭐️ = least confident, ⭐️⭐️⭐️ = most confident.

If you wish to accept the answers suggested by the AI (Chat GPT), click the Use AI Answer button.

You can withdraw the selected AI answer with the Undo button.

Supported Checklist Items

Users can apply the prefill service for the following types of Checklist Items:

• Radio Button

• Yes/ No

• Correct Technique - IT - ND

• Drop-down

• Checkbox

• Text

• Matrix

  
  

  For checkboxes and radio buttons: items are highlighted in purple with a shadow behind them.

  
  

  

  Radio button items highlighted

  
  

  
  

  

  Checkbox items highlighted

  
  

  
  

  
  

  
  For dropdown and free text: A purple box containing the recommended answer appears. If the user hovers over a purple item, a tooltip appears with additional information.
  For dropdown questions, the purple box displays the correct answer from the dropdown, which the user can accept by clicking the button.

  
  

  
  
  

  
  

  Any text entered by the user in the free text or dropdown fields will be replaced by the AI-generated answer upon clicking the "Use AI Answer" button. This behavior is highlighted by a warning message displayed to the user. “Clicking the button will overwrite your current response.”

  
  

  

  Items not numbered in the checklist (Item counter turned off in the Case Content)

  
  

  
  

If AI cannot generate answers, the following notice is displayed:

If the AI is unable to find a suitable response to any of the questions asked, this message is displayed to the user. “Unfortunately, the AI was unable to find an answer to any of the questions in the checklist. We are unable to assist you at this time.”

If LS cannot reach the AI server for any reason, an error message will appear: “AI server is currently not available. Please try again later.”

User Rights and Responsibilities

• Acceptance of Terms

By enabling the Checklist Prefill feature, you agree to these Terms of Use set forth by Elevate Healthcare.
A disclaimer below the submit button states that the AI may make mistakes. This is a legal requirement.

• Access to Checklist Prefill Feature

If you disable the Checklist Prefill Availability setting, LearningSpace will cease generating prefills for FON checklists.

• Data Privacy and Security

You retain ownership of all content provided to LearningSpace, including transcripts and FON checklists. You have the right to access, correct, and delete your data in accordance with Elevate Healthcare’s data management policies.

Limitation of Liability

Elevate Healthcare and its providers are not liable for unauthorized access to user data or indirect, incidental, or consequential damages associated with the Checklist Prefill feature. This limitation does not intend to restrict any legal rights you may have.

Amendments to Terms

Elevate Healthcare reserves the right to modify or update these Terms of Use to reflect changes in our services or compliance requirements. Users will be notified of any material changes, and continued use of the Checklist Prefill feature after such modifications constitutes acceptance of the revised terms.