Specification:

PILLAR:

Technology and Design

Design phase:

Evaluation

SOLO TAXONOMY LEVEL:

Extended Abstract

TIME:

Long

In this teaching activity, students will learn how to plan for a continuous evaluation of a product, system or service during the evaluation phase and after it has been used for a while by different stakeholders in real life contexts. The learning from this teaching activity can for example be used for writing the “future work” section of a project report. It can also be used to plan for a future iteration of an existing product, system or service.

The teaching activity is based on the assumption that a product, system or service is never fully finished, in the sense that it needs iterative or incremental design and development after it has been appropriated into the use contexts that it was directed towards. At the end of a design process, when students are evaluating the reception and impacts of their designs in use, they can perform this activity as part of the evaluation phase.

To maintain the students’ critical perspectives on the impacts of their product, system or service, and the values that it generates in real-world/real-life contexts, students should reflect upon the results of the stakeholder evaluations of their prototypes and how they might differ from the original visions behind a design.

Additionally, students will predict a few potential impacts of their product, system or service on the contexts in which it is deployed. They will set up criteria for how they might evaluate the impacts of their product, system or service, and how different stakeholders have appropriated it to fit their user contexts.

The students will also discuss which parts of their product, system or service that might need to be changed or modified later on, based on the different appropriations that they might discover in the evaluation phase.

The teaching activity ends with a proposal for how the students might follow up on the development of their product, system or service after it has been used for a while. In other words: how they might assess future use patterns and their consequences.

After the teaching activity students will be able to:

• evaluate how different stakeholders use a product, system or service in their use contexts,
• discuss how a product, system or service might be appropriated by different stakeholders in their concrete everyday life situations, based on the above evaluation of a product, system or service,
• present suggestions for how the use of a product, system or service might be assessed when it has been used by different stakeholders for a while.

• Introduce the students to the teaching activity during the last phase of a design process where they evaluate their prototypes.
• Prepare the provided worksheets 1–3 to be handed out to the students.
• Give instructions for the teaching activity including supporting materials such as worksheets that the students can work on independently or in groups.

The teaching activity can be performed individually or in groups and where the teacher is available to the students during supervision.

Step 1:

• The students are asked to list three important qualities of their prototype that somehow manifest the project values. They should list their expectations of how different stakeholders might appropriate their design. See worksheet 1, column A.

Step 2:

• The students identify a list of stakeholders that they can get in touch with in order to evaluate their prototype in the actual use situations together with the stakeholders.

Step 3:

• The students bring the prototypes to the use contexts and bodystorm (Oulasvirta, Kurvinen & Kankainen 2003), or do contextual inquiries (Beyer & Holtzblatt, 1999; Holtzblatt, Wendell & Wood, 2005) with their prototypes in the different use contexts in order to evaluate them together with a selection of stakeholders. Special attention should be paid on the expected use and perceived values that they listed in Step 1.

  In a discussion with the stakeholders in the use situations, the students should keep in mind that the product, system or service might also change the stakeholders’ assumptions of how they might use the product, system or service that they evaluate. Thus, a debrief with the stakeholders after the user evaluation can be appropriate.

Step 4:

• The students list the results of the evaluation of their product, system or service prototypes. By filling in column B and C in worksheet 1, they find out what the differences are between expected use and observed use.

  When evaluating the results, they should discuss how their design might be redesigned, based on the insights from the evaluation by filling in column D in worksheet 1. Based on the list of insights, students create some new evaluation criteria that they might use for future investigations of the product, system or service, column E in worksheet 1.

Step 5:

• In worksheet 2, students should describe how stakeholders appropriated the prototype in a real-world scenario (column A), and then discuss what kind of impact their design might have (column B), based on this scenario.

Step 6:

• Worksheet 3 is related to future work and development, in which students should pose research questions to assess further use of their product, system or service after appropriation (column A). Based on each of the research questions, they should suggest a research method (column B), that is, how they want to investigate the research questions.

• The result of the teaching activity is presented as “future work” as part of a final project presentation in plenum. Furthermore, the outcome of this teaching activity could be described in a future work section of a project report.

To assess whether the intended learning outcomes were attained by the teaching activity the following assessment activities can be carried out (in class or after class).

Assess students' learning by asking them to create a timeline of the evaluation of values within the design process (ipsative assessment) focusing on how values evolve through evaluation of the design containing 1) values of the design ahead of the evaluation phase, 2) values change happening during the evaluation phase 3) values of the final design, 4) values change that (might) emerge after the design has been used for a while.

Assess students' learning by asking them to engage in peer feedback (authentic assessment) on each other’s design after design. Ask students to comment and pose questions in relation to the proposed changes and modifications and how well they are connected to findings coming out of the evaluation phase. Also ask students to give critical but constructive feedback on the suggestions for how the use of a product, system or service might be assessed when it has been used by different stakeholders for a while.

In the assessment activity ask students to focus on:

• explaining how they have planned for continuous evaluation of their product, system or service during and after it has been put into use,
• describing how they might evaluate the impact of their product, system or service, and how different stakeholders have appropriated it to fit their user contexts,
• reflecting on how they might follow up on the development of their product, system or service after it has been used for a while and assess future use patterns and their consequences.

Beyer, Hugh; and Holtzblatt, Karen (1999). Contextual Design. Interactions, 6(1), 32–42. https://doi.org/10.1145/291224.291229

Holtzblatt, Karen; Wendell, Jessamyn Burns; and Wood, Shelly (2005). Rapid Contextual Design: A How-to Guide to Key Techniques for User-Centered Design. San Francisco: Morgan Kaufmann.

Oulasvirta, Antti, Kurvinen, Esko; and Kankainen, Tomi (2003). Understanding contexts by being there: case studies in bodystorming. Personal and Ubiquitous Computing, 7, 125–134. http://10.1007/s00779-003-0238-7