A Metric to Quantify if a Software Application Meets the User’s Expectation for Completing a Representative Task

A usability test was conducted to evaluate medical software (ECGSim). A total of 112 tasks were recorded (14 users each attempted 8 tasks). We wanted to evaluate two metrics that could be used to determine if the software meets the user’s expectation in carrying out a representative task. Each user answered two questions before completing a task, (Q1) how difficult do you expect this task to be out of 1-10? (where 10=most difficult) and (Q2) how long do you expect to complete this task? After attempting the task, the user was then asked to rate how difficult the task was. Thus two metrics were calculated for each user task, (Metric 1) Δ between the user’s expected task-completion time and actual task-completion time (derived from time-stamps via a screencast) and (Metric 2) Δ between pre-task difficulty rating and post-task difficulty rating. A paired t-test (p