Inter-Rater Agreement

One device for evaluating SPs is for an Observer to complete the same questionnaire the SP completed and then compare the two.

Observers can review the video recording of encounters, thereby ensuring what the SP and the learner did and how they did it during the encounter.


Difference Value

To evaluate the Inter-Rater Agreement, the system calculates the difference value. For an Item, this value is the absolute difference between the answer selected by the SP and that selected by the Observer. The section's difference value is the sum of the difference values for the non-open-ended items.

The Difference value is in points, not percentages.


Oi = value of the answer selected by the observer for an item
SPi = value of the answer selected by the SP for an item
n = number of the items in the SP part of the case


NOTES:

  • If both the SP and the Observer select the same value for an item, the difference for this item is zero.

  • The difference value for an item is zero even if they selected different answers if those answers have the same point value.
    The reason is that the difference does not affect the learner's score: it is the same whether the SP does exactly as the Observer.

IMPORTANT: If selecting an answer for an item is important and the SP's choice affects the student's scores, the answers should have different point values.

  • The difference value calculation uses only responses to items with predefined answers. Responses to open-ended items, such as text boxes, are NOT used, even if they are scored.


Section Agreement Value

The Difference Value is used for calculating the Section Agreement Value. This percentage value refers to one section of the SP part, and is the difference of the total achievable non-open-ended point value in the section and the difference value, divided by the total achievable point value.

The Section Agreement Value is a percentage value that refers to one section of the SP part: it is the difference of the total achievable non-open-ended point value in the section and the difference value, divided by the total achievable point value.

t = total achievable non-open-ended point value in the section
d = difference value for the section

From the section agreement value of each section of the SP part, the system calculates the Inter-Rater Agreement Value: their weighted average using the section weights as set in the Case Manager for learner scoring. This ensures that disagreement about points in more important (hence higher weighted) sections will affect the Total Agreement Value more.

Inter-Rater Agreement Value

Inter-Rater Agreement Value: the sum of the products of each Section Agreement Value within one particular SP part and the corresponding section weight divided by the sum of the section weights.




ss = section agreement value for each section of the SP part of a certain Case
ws = section weight belonging to each section within the SP part, defined in the Case Manager
n = number of sections in the SP part

Inter-Rater Agreement Score

The Inter-Rater Agreement Value refers to one observation the SP has about their performance for a certain learner encounter. However, SPs may be evaluated according to several observations in a Case.
The total Inter-Rater Agreement Score is the average of the Inter-Rater Agreement Values across all observations of the SP for a given Case.




io = inter-rater agreement value of an SP for one particular encounter (i.e., observation)
n = number of observations the SP has for a certain Case

For further information, see the Reports - How to pages and the Reports Module Training Videos.