Abstract

This paper presents a systematic literature review characterizing the methodological properties of usability studies conducted on educational and learning technologies in the past 20 years. PRISMA guidelines were followed to identify, select, and review relevant research and report results. Our rigorous review focused on (1) categories of educational and learning technologies that have been the focus of usability evaluation, (2) specific usability evaluation methods used, (3) outcome measures, and (4) research limitations. Findings revealed a diverse range of usability evaluation methods employed for different types of educational/learning technologies and the contexts in which those methods were used, with the majority of usability studies being performed on e-learning technologies within higher education contexts. Specific methods, instrumentation, and types of usability research found to be dominant in reviewed studies were further analyzed and classified, with findings suggesting inquiry methods using questionnaires were most prevalent. Prevalent outcome measures were also synthesized, with findings suggesting that the majority of usability research focuses on issues of technological usability, with very few studies considering pedagogical and socio-cultural aspects of usability. A number of limitations were found, including conceptual and procedural flaws, fundamental misunderstanding of usability evaluation methods, and inappropriate application of usability methods, suggesting potentially problematic and unreliable results. These findings are discussed in-depth, and implications for future research are provided.