The usability of personal digital assistants (PDAs) for assessment of practical performance

Context: The administration of an objective structured clinical examination (OSCE) using paper checklists presents problems such as illegible handwriting, missing student names and/or numbers and lost checklists. Calculating and entering results is not only time-consuming, but is subject to human errors, and feedback to students is rarely available. To rectify these problems, personal digital assistants (PDAs) and HaPerT software were acquired to replace paper checklists and provide automated results and feedback. This study sought to determine the usability of the PDA assessment system. Methods: The usability of the PDA system was evaluated according to effectiveness, efficiency and user satisfaction. Effectiveness was judged by comparing the results of an OSCE conducted in 2003 (paper-based method) and repeated in 2004 and 2005 (PDA method). Efficiency was determined by calculating the amount of time required to organise the logistics for 2 consecutive PDA OSCEs and deliver results and grades, compared with the time required for the paper-based OSCE. User satisfaction was established by using questionnaires to obtain feedback on the assessors' experiences during their first assessments. Results: An independent groups t-test used to compare the means of scores achieved by students in the PDA and paper-based OSCEs, respectively, showed that the difference in effectiveness was not significant. In terms of efficiency, 77% less time was used for the PDA OSCE in 2004 and 93% less in 2005. Assessor feedback on PDA assessment was overwhelmingly positive. Assessment by PDA was found to be just as effective as and more efficient than paper-based assessment in practical examinations, and was highly rated by assessors.