FLAMINKO: Usability evaluation of work-domain specific applications. Experiences and lessons learnt.

Work-domain specific software solutions represent important challenges for usability evaluation methods; particularly related to the lack of context knowledge resources available to the usability experts involved in the evaluation. In order to meet this challenge we have investigated two methods where work-domain experts (end users with thorough experience of the context of use) are engaged as analysts in usability evaluations.• Group-based expert evaluation is an analytical evaluation method designed to allow non-usability experts to participate as evaluators.• Cooperative usability testing is a combined empirical and analytical method, designed to allow the participating user to reflect systematically on their experience with the test and the tested solution. The two evaluation methods were investigated on basis of their application in two cases: (1) The redesign of a PDA application for mobile sales personnel and (2) the development of an in-vehicle solution for emergency response units. The methods’ performance was investigated with regard to thoroughness, validity and impact – three well defined measures in the literature. The results from the two cases were consistent. Group-based expert walkthrough may be recommendable as a low-cost method for usability evaluation of work-domain specific solutions. However, the method missed important usability issues identified through empirical usability testing. Cooperative usability testing seemed in these cases to be a preferable alternative to regular usability testing, as the inclusion of user reflection served to increase the usability issues covered as well as positively affect the impact on the development. The study also gave results of relevance also outside the scope of investigating the methods in question. In particular they serve to raise concern regarding the adequacy of the validity measurement as applied within the field of Human–computer interaction.

Oppdragsgiver: Norwegian research project, the VERDIKT program