Conference-Based Usability Evaluation Methodology for Medical Web Applications

Background: In the US, 100,000 deaths and 1,000,000 injuries occur each year due to medical errors. Whilst all medical errors are not caused by poorly designed software, many are. According to the 2007 ISO/IEC 62366 standard, a usability evaluation process must take place before the introduction of medical software into healthcare. Thus, software including Medicine 2.0 applications must also be evaluated for their usability. The standard method for assessing usability is via the ‘Usability Test’, which normally takes place in a ‘laboratory setting’. However, a laboratory-based approach can be expensive and time consuming.Objective: As such, we investigated the feasibility of carrying out a ‘Usability Test’ at a conference setting.Methods: In our previous work two Web-based systems were developed to visualise Electrocardiogram data. We carried out a usability evaluation of these applications at the 2010 International Conference on Computing in Cardiology. We recruited 17 delegates to partake in the evaluation. Each delegate attempted representative tasks using the software whilst ‘thinking aloud’. Each session was recorded using screencast software and a microphone.Results: From analysis of the video recordings, 70 previously unknown usability issues were discovered in the two applications. This alone indicates the utility of a conference-based approach. As part of the study, we also identified a number of advantages and disadvantages for carrying out a conference-based usability evaluation. The main advantage is the high concentration of experts at one geographical location. These experts also represent the international stage and it would be very expensive to arrange such worldwide experts to partake in a laboratory-based evaluation. Furthermore, in a conference setting, a usability evaluation can be carried out over a few days as opposed to weeks or months. This is contrary to a laboratory-based approach where weeks are dedicated to mailing invitation letters. A conference-based approach also allowed for effective advertisement of the study through the dissemination of information sheets within the delegate packs. It also allowed for face-to-face discussion regarding queries from potential subjects. Although delegates can volunteer to participate, a conference setting does also facilitate the solicitation of subjects. One of the most pertinent advantages is the fact that a conference-based approach does not require subjects to take additional time out and money to travel to a laboratory.Given the busy schedule of most conferences, a significant disadvantage is that a limited amount of time can be dedicated to each participant. Within this study, the average duration of a usability session was 34.69 minutes (SD=10.28). Given conference session breaks usually last somewhere between 30 minutes and one hour, this data shows the feasibility of integrating usability studies within a conference setting. We also found that only four user tasks could be performed within a 30-minute time period.Conclusions: Not only have we shown that the ‘Usability Test’ can be effectively carried out at a conference setting to detect ‘use errors’, we have also shown that this approach is less time consuming and more economic when compared to the conventional laboratory approach.