# AI‑Ready SOAP Note Evaluation Checklist Guide ## Creating AI‑Ready SOAP Note Evaluation Checklists User Guide & Best Practices for LearningSpace – AI Assistant The LearningSpace – [AI Assistant](/doc/oeqs-ai-assistant-guide-O0ye9YpFeG) supports faculty by automatically pre-filling SOAP‑note–based checklists using **large‑language‑model (LLM)** analysis. To ensure *accurate, reliable, and verifiable* pre‑evaluation, the checklist's structure must be intentionally designed for natural‑language inputs rather than human-scoring shorthand. This guide provides practical instructions for faculty, SP educators, and OSCE designers on how to build checklists that the (OEQS) AI Assistant can interpret with high fidelity. ## 1.  Core Design Philosophy **1.1  SOAP Notes Are Natural Language → Your Checklist Should Be Too** Learners write SOAP notes in **free text**. Therefore, AI performs best when checklist questions and answer choices are also written in *clear, natural language,* **avoiding abbreviations or coded scoring logic**. :::tip **Key principle:** *Write checklist questions the way a human would describe correct or incorrect performance in plain English.* ::: **1.2  Humans Think in Abstractions — AI Needs Explicit Information** Traditional checklists might compress scoring logic into minimal text because faculty members, based on years of experience, apply **heuristic rules** automatically. AI, however, needs **explicit, self-contained statements**. **1.3  What Information the AI Receives Today** At the current stage of implementation, the AI Assistant evaluates learner SOAP notes using only the information provided in the checklist itself and the learner’s written response. It does not receive **case materials, instructions, or scenario details** unless they are explicitly included in the checklist items. :::info Specifically, the AI receives: 1. The learner’s **SOAP note text** (*Subjective, Objective, Assessment, Plan*). 2. The **evaluation question text** for each checklist item. 3. The **full natural-language evaluation criteria** embedded in the *radio buttons, checklist items, or matrix elements*. ::: Because no case description or external rubric is passed to the AI, every aspect of evaluation—*correct actions, expected findings, common errors, red-flag omissions, and unacceptable practices*—must be represented as self-contained evaluation criteria within the checklist. ### Why this matters If a checklist item assumes the AI knows details about the case (e.g., “Did the learner document the correct differential diagnosis for this case?”), the AI has no way to infer what the correct answer should be. :::info To ensure accurate scoring: * Each **evaluation point** should be expressed as a **standalone statement**. * **Expected findings** should be written **in full** (e.g., “The learner documented the expected diagnosis of acute appendicitis”). * **Error states** must be **explicit** (e.g., “The learner documented an incorrect diagnosis not supported by the case presentation”). ::: This ensures that the AI evaluates the learner’s response by directly comparing it with the **clearly defined, text-based criteria**—without relying on hidden or inaccessible case context. ## 2.  Best Practices for AI‑Ready Checklist Construction **2.1  Write Questions in Full Sentences** :::success Good: “Did the learner provide a concise assessment that identifies the most likely diagnosis and key differential diagnoses?” ::: :::warning Bad: “Assessment accurate?” ::: **2.2  Use Natural‑Language Answer Options** AI performs best when each answer choice is: –  complete (a full, standalone statement) –  unambiguous –  self-contained **2.3  Include Both Correct and Common Error States** Checklist items should explicitly capture: –  correct actions –  partial actions –  incorrect actions –  omissions **2.4  Avoid Shorthand, Abbreviations, or Internal Coding** :::warning Avoid: “Hx/ROS appropriate?” “Plan → R1/R2/R3 categories” ::: **2.5  Avoid Nested Scoring Logic in Headers** :::warning Bad practice: A header with scoring logic (Poor/Fair/Good/Excellent) but radio buttons containing only labels. ::: :::success Correct approach: Convert scoring logic into full statements under each radio button. ::: **2.6 Specify Quantifiable Clinical Information** :::warning Bad practice: ”DURATION: dyspnea is persistent.” ::: :::success Correct approach: ”Duration: Did the learner document that the duration of the patient’s dyspnea was 3 weeks, including that it was persistent?” ::: **2.7  Never Use External Tables, Images, or Links for Scoring Rules** AI cannot access **images or external PDFs**. :::tip Convert every rubric element into text. ::: ### Summary Characteristics of non-AI compatible checklists:  ·      Uses unresolved abbreviations (HPI, meds, NKDA, etc.) ·       Vague, subjective language (“some,” “several,” “clear picture”) ·       No explicit list of what the AI should search for ·       No measurable thresholds (“most,” “adequate,” etc.) ·       References information that the question never defines · Depending on the case knowledge, the AI does **not** have ·       Contains ambiguous or contradictory scoring options ### AI-compatible Checklist Development Guideline :::success ·       **Try to resolve abbreviations** such as HPI, meds, NKDA, ROS, etc., without expansion. ·       **Reference the complete information set relevant to the case** - The question must explicitly list what the AI should look for (e.g., the exact medications, or the exact symptom attributes). ·       **Use measurable, countable criteria** NOT DONE / PARTLY DONE / WELL DONE must be based on numeric thresholds (e.g., 0–2 attributes, 3–6 attributes, ≥7 attributes). ·       **Avoid any case-knowledge dependency** that is not stated in the question The evaluator must not *guess* what is “appropriate for the case.” Everything must be self-contained. **·       Avoid subjective language** No: “some,” “several,” “clear picture,” “adequately,” “important,” etc. **·       Avoid conditional relevance** No “if relevant,” “as needed,” “if appropriate” inside the criteria. (All attributes are simply counted, not interpreted.) #### ·       Avoid extraneous or interpretive statements The AI should only evaluate the *presence/absence* of listed data, not clinical correctness. ::: ## 3.  Recommended Checklist Structure **3.1  Question Format** Use a **full natural-language question** describing what the learner should demonstrate in S/O/A/P. **3.2  Answer Option Format** Use structured sets: ·     Correct ·     Partial ·     Incorrect ·     Omitted ## 4.  Illustrative Example To illustrate how the checklist design principles should be applied in practice, the following example uses a short, self-contained clinical case. It demonstrates both ineffective and well-constructed checklist items. :::warning The ***“bad” or “not AI compatible”*** versions show **common pitfalls**—*unresolved abbreviations, vague scoring terms, missing reference information, and reliance on unstated case details*—making them unsuitable for AI-assisted evaluation. ::: :::success The ***“good” or “AI-ready”*** versions are built for each item by explicitly listing all relevant case elements, resolving abbreviations, and defining measurable thresholds for *NOT DONE, PARTLY DONE, and WELL DONE.* ::: Together, these examples show how a clear, unambiguous, and fully self-contained checklist enables consistent, objective scoring, whether done by human reviewers or AI. ### **4.1  Sample case description** **__Patient Overview__** A 55-year-old man presents to the clinic with increasing shortness of breath. He has a history of hypertension and type 2 diabetes, both of which are generally well controlled. He lives independently, walks several kilometers daily, and has no history of heart or lung disease. He does not smoke and drinks alcohol occasionally. #### __History of Present Illness__ The patient reports that his shortness of breath began three days ago and has progressively worsened. At first, he only noticed it when walking uphill, but over the past day, he has felt breathless even at rest. He describes the sensation as “not being able to get enough air.” He denies chest pain, fever, or cough, but mentions feeling unusually tired. He also noticed mild swelling around both ankles and says he had difficulty sleeping last night because lying flat worsened his breathing. Sitting upright provides some relief. #### __Symptom Details__ ·       **Onset:** 3 days ago ·       **Character:** Constant shortness of breath, worse with exertion ·       **Severity:** Moderate; limits daily activities and causes discomfort at rest ·       **Timing:** Worse at night and when lying flat ·       **Aggravating factors:** Physical activity, supine position ·       **Alleviating factors:** Sitting upright ·       **Associated symptoms:** Fatigue and mild ankle swelling; no chest pain, cough, or fever #### __Medical History and Baseline Function__ The patient is usually active and independent. He routinely walks 2–3 km per day without difficulty and has had no recent hospitalizations. He reports no changes in appetite, weight, or exercise tolerance until this current episode. #### __Medications__ ·       **Lisinopril 10 mg PO once daily** ·       **Metformin 500 mg PO twice daily** ·       **Albuterol inhaler 90 mcg, 2 puffs every 4 hours as needed** (used rarely) #### __Allergies__ ·       **Penicillin**, which causes a rash ·       No food or environmental allergies reported ### 4.2 Sample checklists demonstrating AI-ready/not-AI compatible practices  | **Non-AI compatible formulation** |  **AI-ready formulation** | |-------------------------------|-----------------------| | **==HPI – baseline info==** **[ ] NOT DONE:** Learner didn’t really talk about baseline status or didn’t give enough useful info **[ ] PARTLY DONE:** Learner mentioned *some* baseline information, but it’s not detailed enough **[ ] WELL DONE:** Learner included all relevant baseline HPI information and explained the patient’s usual functioning clearly | **==Document the patient’s baseline health and functioning as follows:==** 1.         Chronic medical conditions (hypertension, type 2 diabetes)2.        Baseline physical activity (walks 2–3 km daily)3.       Independence in activities of daily living4.        Smoking status (non-smoker)5.       Alcohol use (occasional)6.       Recent hospitalizations (none)7.        Baseline functional stability (no prior cardiac/lung disease)**[ ] NOT DONE:** Fewer than **3** of the listed baseline elements are documented.**[ ] PARTLY DONE: 3–5** of the listed baseline elements are documented. **[ ] WELL DONE: 6 or more** baseline elements are documented. | | **==HPI – symptom details==** ·       onset·       duration·       severity·       other relevant features [ ] NOT DONE: Missing several key parts of the HPI, doesn’t give a clear picture [ ] PARTLY DONE: Has some of the HPI symptom attributes but lacks important ones [ ] WELL DONE: Describes most HPI attributes well enough to understand the problem | **==Document the attributes of the patient’s presenting symptom (shortness of breath) as follows:==**1.         Onset (3 days ago)2.        Character (“cannot get enough air”)3.       Location (chest, non-localized dyspnea)4.        Radiation (none)5.       Severity (moderate; at rest and with exertion)6.       Duration (constant last 24 hours)7.        Timing/pattern (worse at night / when lying flat)8.       Alleviating factors (sitting upright)9.        Aggravating factors (exertion, lying flat)10.     Associated symptoms (fatigue, ankle swelling)**[ ] NOT DONE: 0–3** symptomsattributes documented.**[ ] PARTLY DONE: 4–7** symptom attributes documented.**[ ] WELL DONE: 8 or more** symptom attributes documented. | | **==Meds documented==** [ ] NOT DONE: Medications not listed or missing required info [ ] PARTLY DONE: Med list is complete, but not all meds include dose/route/freq [ ] WELL DONE: Meds listed with complete information for each medication as per case | **==Document all current medications as described, including dose, route, and frequency:==** 1.         Lisinopril 10 mg PO once daily2.        Metformin 500 mg PO twice daily3.       Albuterol inhaler 90 mcg — 2 puffs every 4 hours as needed**[ ] NOT DONE:** Learner lists **0–1 medications**, or medication details include dose/route/frequency for **fewer than 30%** of listed items. **[ ] PARTLY DONE:** Learner lists all medications, BUT includes complete dose/route/frequency for **30–79%** of them. **[ ] WELL DONE:** Learner lists all medications AND includes dose, route, and frequency for**≥80%** of them. | |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **==Allergies noted (incl. rxn)==** [ ] NOT DONE: No allergies listed OR NKDA stated incorrectly [ ] PARTLY DONE: Allergies listed, but reactions missing [ ] WELL DONE: All allergies documented with reactions and NKDA if appropriate for the case | **==Documented the patient’s allergies according to the following:==** 1.         Penicillin allergy2.        Reaction: rash3.       Absence of food/environmental allergies (no others reported)**[ ] NOT DONE: 0–1** of the listed allergy elements documented, OR contradictory use of NKDA. **[ ] PARTLY DONE: 2** of the required allergy elements documented. **[ ] WELL DONE: All three** allergy elements documented precisely as stated. | ## Examples ### **Examples of Good Practices** ![](https://outline-production-attachments.s3-accelerate.amazonaws.com/uploads/03fa1014-cf0e-4191-ba80-50d24eef65d4/54bc2f3b-6df2-4348-880d-ab01ceb692cb/K%C3%A9p1.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4EOUDTOVUICLPZ4P%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T120000Z&X-Amz-Expires=86400&X-Amz-Signature=3f2ac3e600a1e0bf1ab301d635458a2f4a37a4f2373598ac8753e606004cae11&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject " =471x566") ![](https://outline-production-attachments.s3-accelerate.amazonaws.com/uploads/03fa1014-cf0e-4191-ba80-50d24eef65d4/a810ff5e-f044-4e4f-8b47-200ef95018db/K%C3%A9p2.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4EOUDTOVUICLPZ4P%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T120000Z&X-Amz-Expires=86400&X-Amz-Signature=08ad1877293f8d027fada17010027b84cb7398656ab859f118cdf1d192eae875&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject " =471x520") ### **Examples for Mixed Practices** Use of abbreviations should be avoided, but checklist items that contain text allow pre-filling with good accuracy. ![](https://outline-production-attachments.s3-accelerate.amazonaws.com/uploads/03fa1014-cf0e-4191-ba80-50d24eef65d4/ab39e69d-9e3d-4480-90b8-5cf593e374f2/K%C3%A9p7.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4EOUDTOVUICLPZ4P%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T120000Z&X-Amz-Expires=86400&X-Amz-Signature=509edf68a3e45bc038819318c07a8fd68cfd6461a15feed3e0f52e2e8470e990&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject " =471x517") ![](https://outline-production-attachments.s3-accelerate.amazonaws.com/uploads/03fa1014-cf0e-4191-ba80-50d24eef65d4/f0058251-7064-4ff2-bed9-08a8d48d7846/K%C3%A9p6.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4EOUDTOVUICLPZ4P%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T120000Z&X-Amz-Expires=86400&X-Amz-Signature=00621a4dd8a85d84c904c7377822429c10b174b17b144afae7ba5cc7f9bd0dcf&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject " =471x635") ![](https://outline-production-attachments.s3-accelerate.amazonaws.com/uploads/03fa1014-cf0e-4191-ba80-50d24eef65d4/96064515-8f88-4042-ac9d-6d5c6445ce40/K%C3%A9p4.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4EOUDTOVUICLPZ4P%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T120000Z&X-Amz-Expires=86400&X-Amz-Signature=99646d9244f4026397395d10d8a825ec0f8430fb05e1dfe2377f3c4eb3a2033a&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject " =471x483") ![](https://outline-production-attachments.s3-accelerate.amazonaws.com/uploads/03fa1014-cf0e-4191-ba80-50d24eef65d4/cae2e49b-d496-4bd2-a3ad-fdad838b1d96/K%C3%A9p5.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4EOUDTOVUICLPZ4P%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T120000Z&X-Amz-Expires=86400&X-Amz-Signature=93220309e2cf3444b27a89cbd0a16706c75be561110ceeb36919279053e64be8&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject " =471x568") ![](https://outline-production-attachments.s3-accelerate.amazonaws.com/uploads/03fa1014-cf0e-4191-ba80-50d24eef65d4/886fc4eb-bf15-4599-bc0f-6e7019432512/K%C3%A9p3.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4EOUDTOVUICLPZ4P%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T120000Z&X-Amz-Expires=86400&X-Amz-Signature=6055f8373f65b1922d969c068884a04bd049750431b925ead5c92cf4c5770acf&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject " =471x552") :::success [AI tools in LearningSpace](/doc/ef7be453-2b82-473c-9ec1-aae32b3311ec)  [AI tools in LearningSpace - How to](/doc/747f6b97-0c16-4c7e-9c7f-eb32426543c7) ::: --- **Documents** - [User Guide](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/user-guide-WPv4VxhtWB) - [FAQ](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/faq-YoxOQOjvnY) - [Training Videos](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/training-videos-D2ycsEwqPt) - [Module Introduction Videos](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/module-introduction-videos-I6OQ6H7SUT) - [Release Notes](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/release-notes-BcFFMsc0RV) - [Distance Learning with LearningSpace](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/distance-learning-with-learningspace-eAfwW23Hxg) - [Preventing Copy-Paste in LearningSpace](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/preventing-copy-paste-in-learningspace-sPD0IIUuk0) - [Important Technical Notification Regarding Adobe Flash Player](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/important-technical-notification-regarding-adobe-flash-player-eQtMLwLmTJ) - [iPad support - Technical information](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/ipad-support-technical-information-dLmeFvKofC) - [TeamViewer Link](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/teamviewer-link-QyKZBMqUT9) - [YouTube videos about LearningSpace](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/youtube-videos-about-learningspace-TMb3zdc0LC) - [Customer Aftercare One-Pager](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/customer-aftercare-one-pager-GBWMe1BcfS) - [Case Studies](https://kb.learningspace.elevatehealth.net/s/ent-hs/doc/case-studies-VcAbFqXSz7)