Methods: Evaluation Techniques


The IE lab provides a variety of methods to evaluate web applications. The selection of methods in the evaluation process follows the User-Centered Design approach and is based on our clients’ needs, timelines, budgets, and scopes.

Card Sort
Representative users are asked to categorize system content, providing the design team with user input on how to best structure the system.

What is it? Test subjects use note cards to categorize content areas, providing your design team with input from actual users on how to best structure the site.

When is it used? Creating a new site, implementing new site features, redesigning an existing website

Why is it useful? It provides a good foundation for the system structure. The results will show how your users think the site should be designed and how content should be organized.

Deliverables Card sort report, video clips, category diagram, presentation

Heuristic Walkthrough
A hypothetical scenario is designed to identify potential system problems and to consider how a typical user might solve them.

What is it? Using industry-accepted usability standards, experts do a “walkthrough” of your site to evaluate the design and functionality Your website is tested using the most common tasks users perform on your website.

When is it used? If you have a complicated website with lots of information and navigation options. Example: Users of your websites do a number of complicated tasks on your website, such as booking airline tickets, reserving hotel rooms, and renting a car.

Why is it useful? It provides much deeper analysis of the usability and functionality of your website. A prioritized task list and focused questions help to reveal complicated problems not readily discovered on the surface.

Deliverables Video slides and the screen shots of the problematic areas of your website in written reports and presentation

Comparative Evaluation
Systems more or less equivalent to the system under review are identified. The systems are compared to see what can be learned from the approaches others take to solve common problems, incorporate needed features, and meet user needs.

What is it? Comparative evaluation looks at the strengths and weaknesses of a site by comparing it to either competitors or a site deemed to be an aspirational benchmark.

When is it used? It is typically used in the beginning of a redesign.

Why is it useful? This method can identify pertinent needs and industry standards.

Deliverables – Side by side screen shots and information architecture of comparative sites.

Expert (Heuristic) Evaluation
The system is evaluated by three or four experts using a standard list of usability heuristics.

What is it? A group of trained usability experts evaluate the website using industry-accepted usability principles.

When is it used? During the early stages of development and/or when there is not enough time to complete more robust evaluation.

Why is it useful? Because it saves time. Our experts can quickly focus on only the most important aspects of your website and any major problem areas. Also, because we use a team of evaluators, you will get several different perspectives, which helps ensure that a broad range of issues are covered.

Deliverables Screenshots with comments of experts, video clips of certain parts of the expert review process, heuristic evaluation scores, final report and/or presentation

Focus Groups
Six to eight representative users are interviewed as a group to provide input on system content, design, or usability.

What is it? A group interview of 6 to 8 individuals who have been identified as a typical users of your website. Together they provide input on your content, design, and usability with observers asking questions and taking notes.

When is it used? When there is a shortage of time, particularly at an initial stage of development.

Why is it useful?- Because it allows a large amount of high-quality feedback to be collected quickly.

Deliverables Video clips and screen shots with written reports and presentations

Information Horizons
A representative user is asked to recall a specific information related activity and to sketch a diagram of his or her subsequent actions towards satisfying the information need.

This is a method developed using the ‘Information Horizons’ theory by Diane H. Sonnenwald (1999) to study user’s information seeking behavior.

The concept of ‘Information Horizons’ takes into account the variety of information resources that are available to a user socially or individually. The theory assumes many solutions and the problem expands from determining the most efficient path to the best solution, to determining how to make solutions visible to an individual(s) (Sonnenwald, 1999).

Profiles of representative users, complete with user name, photo, and information about user goals, motivations, personalities, and attitudes, are created as a means to provide designers with robust information on the individuals for whom the system is being designed.

What is it? Our team creates a fictional profile of your typical user. The profile consists of a variety of fictional personal details including a name, photo, income, education level, personality, and attitudes. This is created as a means to provide you with an better idea about the users for whom the website is being designed.

When is it used? Usually during the early stages of design so that your users’ needs can be integrated into the website more easily.

Why is it useful? The persona profile can help guide design decision making throughout the design process. It can help you better understand your target audience and their needs, and it helps avoid designing a product for non-existent users.

Deliverables Persona artifact

Task Analysis
Identifies users’ overall goals and information needs and models tasks based on these goals as a means to identify points where users fail to achieve goals, spend excessive time, or are made uncomfortable.

What is it? This is the bread and butter of usability in terms of identifying the actual functionality of a site.

When is it used? It is typically used towards the end of a redesign or to identify problem areas on a current site.

Why is it useful? This method is especially useful for identifying first time user’s experience. Often individuals that are familiar with a site forget about the learning curve to locate or to perform actions, and this method can determine the frustration new users who in turn may abandon the site.

Deliverables – Videos, mouse clicks, time on task, qualitative data of user expectations and frustrations.

Sociotechnical Walkthrough (STWT)
Representative (future) users are asked to model together sociotechnical serive or work processes in a series of workshops that are professional facilitated.

For what purposes? Analysis and integration of technology into existing and new work processes (change management), designing together with future users and stakeholders to identify issues such as ‘holes’ in the process. The models will be modeled step-by-step and users indicate strength and weaknesses in the process as a basis to improve the relationship between task, users and technology.

Which problems are addressed? Gaining mutual understanding of service or work processes, identifying problems and adaptions needs, improving reliability and sustainability of service processes, improving commitment and reducing frictions in service or work processes.

When it is used? This method can be used in different project phases: Users’ and decision-makers’ need analysis; for early stage system design and prototyping, during technical system development and for distribution, engagement and training, if the technical system has been built already.

Deliverables: visualized graphical semi-structured process models, focus group video recordings

Think-aloud Interview
Users are asked to give a running commentary on their thoughts as they perform tasks.

What is it? Users are recorded while they use the website and talk through their all thought processes. Usually 5-6 users are tested in separate think-aloud sessions.

When is it used? At any stage of development to get feedback from real users.

Why is it useful? It helps us understand the real behavior and thought processes of actual users. This process helps to reveal any areas of the site which might conflict with the cognitive and behavioral patterns of the users.

Deliverables Video slides and screen shots of the problematic areas of the website with written report and presentation

Web Analytics
Monitors and analyzes your web site traffic.

What is it? Online tools that monitor the traffic on your website. Analytics code is embedded on each page, and it captures information about who is visiting your site, what pages they are visiting, how long they stay on the site, etc.

When is it used? After the site is live to gather real-time data about your users.

Why is it useful? Web Analytics help to can help you tailor your site to the specific users of your site. Using this information will help you make better decisions about changes or design upgrades that will improve your targeted users’ experience.

Deliverables Video slides and screen shots of the problematic areas of the website with written report and presentation