Sawyer, Flanders and Wixon defined a metric called impact ratio to measure the effectiveness of usability evaluation methods. This numerical value presents the proportion of the problems that the development team commits to fix from all the problems found in the evaluations. They inspected ten software products that they had evaluated, and achieved an average impact ratio of 78%.
They found out that the following factors improved the impact ratio:
– Developers’ respect: The usability group conducting most of the evaluations was an internal group in the company, so it had a long-term relationship with the development groups and had earned the developers’ respect.
– Written reports: The usability group gave written reports of the findings to the development groups. The reports included descriptions of the problems and alternatives for fixing them.
– Written response: The group required a written response from the clients to report the commitment to fix the problems.
– Multiple methods: The evaluations involved the use of multiple methods for discovering usability problems.
– Easy response process: The group set up meetings where they went over the report with the clients.
– Specific recommendations: The group provided detailed and technically specific recommendations to fix specific problems.
– Severity level: The group rated each problem by its severity.
– Client participation: The group involved the development team in the evaluations by having a member of the team sit on an inspection or user test.
– Early involvement: The group tried to work with the development teams as early in the development cycle as possible.
To be able to give effective recommendations, Sawyer et al. recommend that the evaluators become familiar with the product and its design goals before the evaluations, so that the recommendations support the product goals. In addition, the evaluators should study the product within the context of its product family, so that the recommendations do not represent inconsistencies with other components in the product family.