Methods: Evaluation Techniques

Posted:

The IE lab provides a variety of methods to evaluate web applications. The selection of methods in the evaluation process follows the User-Centered Design approach and is based on our clients’ needs, timelines, budgets, and scopes.


Card Sort
Representative users are asked to categorize system content, providing the design team with user input on how to best structure the system.

What is it?
Test subjects use note cards to categorize content areas, providing your design team with input from actual users on how to best structure the site.

When is it used?
Creating a new site, implementing new site features, redesigning an existing website

Why is it useful?
It provides a good foundation for the system structure. The results will show how your users think the site should be designed and how content should be organized.

Deliverables
Card sort report, video clips, category diagram, presentation


Cognitive/Scenario-based Walkthrough
A hypothetical scenario is designed to identify potential system problems and to consider how a typical user might solve them.

What is it?
Using industry-accepted usability standards, experts do a “walkthrough” of your site to evaluate the design and functionality Your website is tested using the most common tasks users perform on your website.

When is it used?
If you have a complicated website with lots of information and navigation options. Example: Users of your websites do a number of complicated tasks on your website, such as booking airline tickets, reserving hotel rooms, and renting a car.

Why is it useful?
It provides much deeper analysis of the usability and functionality of your website. A prioritized task list and focused questions help to reveal complicated problems not readily discovered on the surface.

Deliverables
Video slides and the screen shots of the problematic areas of your website in written reports and presentation


Comparative Evaluation
Systems more or less equivalent to the system under review are identified. The systems are compared to see what can be learned from the approaches others take to solve common problems, incorporate needed features, and meet user needs.

What is it?
Comparative evaluation looks at the strengths and weaknesses of a site by comparing it to either competitors or a site deemed to be an aspirational benchmark.

When is it used?
It is typically used in the beginning of a redesign.

Why is it useful?
This method can identify pertinent needs and industry standards.

Deliverables
Side by side screen shots and information architecture of comparative sites.


Expert (Heuristic) Evaluation
The system is evaluated by three or four experts using a standard list of usability heuristics.

What is it?
A group of trained usability experts evaluate the website using industry-accepted usability principles.

When is it used?
During the early stages of development and/or when there is not enough time to complete more robust evaluation.

Why is it useful?
Because it saves time. Our experts can quickly focus on only the most important aspects of your website and any major problem areas. Also, because we use a team of evaluators, you will get several different perspectives, which helps ensure that a broad range of issues are covered.

Deliverables
Screenshots with comments of experts, video clips of certain parts of the expert review process, heuristic evaluation scores, final report and/or presentation


Focus Groups
Six to eight representative users are interviewed as a group to provide input on system content, design, or usability.

What is it?
A group interview of 6 to 8 individuals who have been identified as a typical users of your website. Together they provide input on your content, design, and usability with observers asking questions and taking notes.

When is it used?
When there is a shortage of time, particularly at an initial stage of development.

Why is it useful?
Because it allows a large amount of high-quality feedback to be collected quickly.

Deliverables
Video clips and screen shots with written reports and presentations


Individual interviews (Follow-up interviews)
Users are asked to participant in an interview that conducted by the practitioners that are relating to usability experience of the system or product.

What is it?
This is a UX research method during which a researcher asks one user questions about a topic of interest about the system with the goal of learning about that topic.

When is it used?
Interviews take place at the start of the development cycle while the objectives and goals of the system is still under development and review.

Why is it useful?
The interviews do not involve watching a user work, they are different from interviewing user in a usability test or conducting contextual interviews. It resembles focus groups since they involve talking with users.

Deliverables
Individual interview can be face-to-face or online and before or/and after the activity. Practitioners may also use interviews to supplement online surveys. Performing an interview before a survey helps you to refine questions for the survey. Interviewing participants after a survey allows you to probe for details and reasons behind answers that users give on a survey.


Information Horizons
A representative user is asked to recall a specific information related activity and to sketch a diagram of his or her subsequent actions towards satisfying the information need.

What is it?
This is a method developed using the ‘Information Horizons’ theory by Diane H. Sonnenwald (1999) to study user’s information seeking behavior.

When is it used?
Information horizons can be used in any context, situation, or stage exists an information seeking tasks or activities that an individual involved. In terms of the phase in the usability testing, we use information horizons when user is participating in the usability test.

Why is it useful?
‘Information Horizons’ takes into account the variety of information resources that are available to a user socially or individually. It helps to identify the sources people use and their source preferences. The theory assumes many solutions and the problem expands from determining the most efficient path to the best solution, to determining how to make solutions visible to an individual(s) (Sonnenwald, 1999).

Deliverables
Predominately used interviews with informants’ map-drawings. Individual talk and explain the drawing as he/she creates it.


Persona
Profiles of representative users, complete with user name, photo, and information about user goals, motivations, personalities, and attitudes, are created as a means to provide designers with robust information on the individuals for whom the system is being designed.

What is it?
Our team creates a fictional profile of your typical user. The profile consists of a variety of fictional personal details including a name, photo, income, education level, personality, and attitudes. This is created as a means to provide you with an better idea about the users for whom the website is being designed.

When is it used?
Usually during the early stages of design so that your users’ needs can be integrated into the website more easily.

Why is it useful?
The persona profile can help guide design decision making throughout the design process. It can help you better understand your target audience and their needs, and it helps avoid designing a product for non-existent users.

Deliverables
Persona artifact


Pilot Testing (Pilot Study)
The pilot testing is conducted by the usability practitioners to ensure that the full study goes as smoothly as possible

What is it?
It is a type of testing that verifies all the components of the usability testing under a real-time operation condition.

When is it used?
The usability practitioners run through a session or two in at least 1 day in advance of the main, schedule study.

Why is it useful?
Pilot testing helps fine-tune usability studies, leading to more reliable results. It provides an opportunity to validate the wording of the tasks, understand the time necessary for the session, and, if all goes well, may even supply an additional data point for your study.

Deliverables
The pilot testing can be held remotely or on-site depends on the need of the study. The participant for the pilot testing can be the internal team members.


Prototyping
The usability practitioners develop the draft version of the products. In order to make better design, practitioners show the draft to users, obtain the feedbacks from them, then modify the system based on the feedback.

What is it?
A prototype is a draft version of a product that allow usability practitioners to explore the ideas and show the intention behind a feature or the overall design concept to users before investing time and money into development.

When is it used?
Practitioners build the prototypes in the early stage of the development of the product.

Why is it useful?
It is economical to change a product early in the development process than to make change after completing the site.

Deliverables
A prototype can be anything from paper drawings (low-fidelity) to something that allows click-through of a few pieces of content to a fully functioning site (high-fidelity).


Sociotechnical Walkthrough (STWT)
Representative (future) users are asked to model together sociotechnical serive or work processes in a series of workshops that are professional facilitated.

For what purposes?
Analysis and integration of technology into existing and new work processes (change management), designing together with future users and stakeholders to identify issues such as ‘holes’ in the process. The models will be modeled step-by-step and users indicate strength and weaknesses in the process as a basis to improve the relationship between task, users and technology.

Which problems are addressed?
Gaining mutual understanding of service or work processes, identifying problems and adaptions needs, improving reliability and sustainability of service processes, improving commitment and reducing frictions in service or work processes.

When it is used?
This method can be used in different project phases: Users’ and decision-makers’ need analysis; for early stage system design and prototyping, during technical system development and for distribution, engagement and training, if the technical system has been built already.

Deliverables
visualized graphical semi-structured process models, focus group video recordings


System Usability Scale (SUS)
Users are asked to provide feedback through completing the 10-items questionnaires regarding the experience of the system.

What is it?
SUS is a ten-item questionnaire for the user, with five response options for respondents, ranging from strongly agree and strongly disagree, about the feedbacks to the system.

When is it used?
Users will complete the SUS right after interacting with the system. The system could be hardware, software, mobile devices, websites, applications, and so on.

Why is it useful?
SUS is a short quiz and the template has already been created, which does not require a lot of resources or efforts for the usability practitioners to develop other usability components. More importantly, the SUS shows reliability and validity about the user’s tasks performances.

Deliverables
SUS templates can be retrieved from the network easily. This questionnaire can be presented in any form, such as print-out, online survey, or directly input in the system that user interacted with.


Task Analysis (Task-Based Usability Study)
Identifies users’ overall goals and information needs and models tasks based on these goals as a means to identify points where users fail to achieve goals, spend excessive time, or are made uncomfortable.

What is it?
This is the bread and butter of usability in terms of identifying the actual functionality of a site.

When is it used?
It is typically used towards the end of a redesign or to identify problem areas on a current site.

Why is it useful?
This method is especially useful for identifying first time user’s experience. Often individuals that are familiar with a site forget about the learning curve to locate or to perform actions, and this method can determine the frustration new users who in turn may abandon the site.

Deliverables
Videos, mouse clicks, time on task, qualitative data of user expectations and frustrations.


Think Aloud
Users are asked to give a running commentary on their thoughts as they perform tasks.

What is it?
Users are recorded while they use the website and talk through their all thought processes. Usually 5-6 users are tested in separate think-aloud sessions.

When is it used?
At any stage of development to get feedback from real users.

Why is it useful?
It helps us understand the real behavior and thought processes of actual users. This process helps to reveal any areas of the site which might conflict with the cognitive and behavioral patterns of the users.

Deliverables
Video slides and screen shots of the problematic areas of the website with written report and presentation


Web Analytics
Monitors and analyzes your web site traffic.

What is it?
Online tools that monitor the traffic on your website. Analytics code is embedded on each page, and it captures information about who is visiting your site, what pages they are visiting, how long they stay on the site, etc.

When is it used?
After the site is live to gather real-time data about your users.

Why is it useful?
Web Analytics help to can help you tailor your site to the specific users of your site. Using this information will help you make better decisions about changes or design upgrades that will improve your targeted users’ experience.

Deliverables
Video slides and screen shots of the problematic areas of the website with written report and presentation