**8. Analysis and interpretation**

Data from the experiment was gathered and interpreted to clarify the communicating effects of instructions, mainly from two aspects of task performance: efficiency and accuracy. Results from different user groups were compared and examined to find out which type of instructions were more effective and inclusive in terms of communication.

## **8.1 Task analysis**

In the test, all components of the product were given to the participant and product instructions were provided on either a printed sheet or a laptop computer. By using Hierarchical task analysis (HTA), the main goal and sub-goals of the tasks were expressed as below (Figure 9).

As shown in figure9, tasks 1 and 2 were skill-based tasks. Users were asked to follow clear and simple instructions to carry out actions and they did not have to make judgements on the aim and plan of their actions. Task 3 was more complicated, it contained 12 steps and these 12 little tasks were grouped into different sections: 3-A, 3-B, 3-C and 3-D, depending on their goals. Many parts of task 3 required users to recall or understand some rules then make their own decisions on what action they should carry out, in order to achieve the required results.

The choices of the participants had to be carefully planned in order to ensure that their human performance characteristics were checked prior to the beginning of the tests. This was done using a method traditionally used by work-study officers to establish what a hundred percent effort or rating looks like. A full pack of cards is dealt into four hands in a period of 52 seconds. It was ensured that the participants could all carry out this task in periods between 50 and 56 seconds. Thus their human performance characteristics could all

They were asked to follow either the printed instructions for the product or the multimedia ones to complete the same set of given tasks. The tasks included 1) using instructions to find information, 2) checking components of the product and 3) assembling. Their actions were monitored; timing was recorded; errors were observed

Data from the experiment was gathered and interpreted to clarify the communicating effects of instructions, mainly from two aspects of task performance: efficiency and accuracy. Results from different user groups were compared and examined to find out which type of

In the test, all components of the product were given to the participant and product instructions were provided on either a printed sheet or a laptop computer. By using Hierarchical task analysis (HTA), the main goal and sub-goals of the tasks were expressed as

As shown in figure9, tasks 1 and 2 were skill-based tasks. Users were asked to follow clear and simple instructions to carry out actions and they did not have to make judgements on the aim and plan of their actions. Task 3 was more complicated, it contained 12 steps and these 12 little tasks were grouped into different sections: 3-A, 3-B, 3-C and 3-D, depending on their goals. Many parts of task 3 required users to recall or understand some rules then make their own decisions on what action they should carry out, in order to achieve the

instructions were more effective and inclusive in terms of communication.

Fig. 8. Screen examples for the multimedia version of product instructions.

be considered similar.

**8.1 Task analysis** 

below (Figure 9).

required results.

and feedback was collected at the end.

**8. Analysis and interpretation** 

Fig. 9. The main goal and sub-goals of the refined tasks.

### **8.2 Performance on efficiency**

To examine efficiency of different types of instructions, the following indices were used:


Overall, Data suggested that all participants, who used multimedia instructions, used on average 1379 seconds (22minutes 59 seconds), with a standard deviation of 220 seconds, which seemed to be slightly shorter to those of printed instruction users, who spent 1390 seconds (23 minutes 10 seconds), with a standard deviation of 213 seconds (Table 1). The difference was small therefore the authors were keen to find out if there was statistically significant difference between the total time consumption of different instruction.

Null hypothesis: T1 = T2

There is no significant difference between the average total time of users using either printed or multimedia instructions.

Alternative hypothesis: T1 ≠ T2

There is a significant difference between the average total time of users using either printed or multimedia instructions.

The authors assumed that there was no statistically significant difference between the groups. A t-test was then carried out. to find out the confidence level of this hypothesis. The 2-tailed t- test returned a P-value of 0.89. Since the P-value (0.89) was greater than the significance level (0.05), the null hypothesis was accepted. This indicated that there was no significant difference between the efficiency of two versions of product instructions.

Product Instructions in the Digital Age 49

Similarly, Figure 10 potentially suggested that the Digital Natives performed better, when using either the printed or multimedia instructions, than the digital immigrants. On the other hand, the Digital Natives seemed to perform better using print instructions than multimedia and the opposite happened with the Digital Immigrants. Again, t- tests were

After another 3 sets of 2-tailed t-tests, the results showed that the amount of time Digital Natives and Digital Immigrants used to complete all tasks was not much different. Also, different types of instructions did not have a huge impact in terms of the overall efficiency. This suggested that multimedia product instructions were as efficient as printed ones for

carried out to find out the confidence level of the significant differences.

Fig. 11. Average time to achieve all task sections in each user group.

of their efficiency in general, especially for Digital Immigrants.

**8.3 Performance on accuracy** 

By looking at the time, which has been spent on each task section in detail (Figure 11), it was discovered that multimedia product instructions generally have a positive impact on Digital Immigrants in terms of their efficiency. Yet, this does not apply when a spatial judgement is involved in knowledge- based tasks. For Digital Natives, different types of instructions have different effects on time consumption; depending on the nature of each task. Printed instructions were more efficient to be scanned when skill-based tasks were simple enough to be completed intuitively. When instructions are necessary to follow for skill- based tasks, the time difference between the use of multimedia instructions and printed instructions was very little. Although it also took longer for the Digital Natives to use multimedia instructions to fulfil a complicated job when a spatial judgement was needed; multimedia instructions were significantly more efficient to use for a transferable process (rule-based tasks). Therefore the authors believe that multimedia instructions are not always efficient as expected, for example for particular types of tasks; but they are still more effective in terms

The accuracy of users' performance was evaluated through error analysis in this investigation and it was achieved by observations. Every time a user failed to achieve the planned goal, a

both Digital Natives and Digital Immigrants.


Table 1. Average time for all tasks

Fig. 10. Time to achieve all tasks in each user group.

Multimedia

All DN

Printed product instructions

Multimedia product instructions Average duration (s):

Average duration (s):

Fig. 10. Time to achieve all tasks in each user group.

Table 1. Average time for all tasks

1375

1390

Digital Natives (DN) Digital Immigrants (DI) Average Duration (s) Average Duration (s)

Printed (DN) 1284 Printed(DI) 1497

(DI)

1425

Average(s) <sup>1461</sup>

(DN) 1332 Multimedia

Average(s) 1308 All DI

Similarly, Figure 10 potentially suggested that the Digital Natives performed better, when using either the printed or multimedia instructions, than the digital immigrants. On the other hand, the Digital Natives seemed to perform better using print instructions than multimedia and the opposite happened with the Digital Immigrants. Again, t- tests were carried out to find out the confidence level of the significant differences.

After another 3 sets of 2-tailed t-tests, the results showed that the amount of time Digital Natives and Digital Immigrants used to complete all tasks was not much different. Also, different types of instructions did not have a huge impact in terms of the overall efficiency. This suggested that multimedia product instructions were as efficient as printed ones for both Digital Natives and Digital Immigrants.

Fig. 11. Average time to achieve all task sections in each user group.

By looking at the time, which has been spent on each task section in detail (Figure 11), it was discovered that multimedia product instructions generally have a positive impact on Digital Immigrants in terms of their efficiency. Yet, this does not apply when a spatial judgement is involved in knowledge- based tasks. For Digital Natives, different types of instructions have different effects on time consumption; depending on the nature of each task. Printed instructions were more efficient to be scanned when skill-based tasks were simple enough to be completed intuitively. When instructions are necessary to follow for skill- based tasks, the time difference between the use of multimedia instructions and printed instructions was very little. Although it also took longer for the Digital Natives to use multimedia instructions to fulfil a complicated job when a spatial judgement was needed; multimedia instructions were significantly more efficient to use for a transferable process (rule-based tasks). Therefore the authors believe that multimedia instructions are not always efficient as expected, for example for particular types of tasks; but they are still more effective in terms of their efficiency in general, especially for Digital Immigrants.

#### **8.3 Performance on accuracy**

The accuracy of users' performance was evaluated through error analysis in this investigation and it was achieved by observations. Every time a user failed to achieve the planned goal, a

Product Instructions in the Digital Age 51

During the tests, by referring to given instructions, users observed and corrected their own errors in the performing progress. When errors were not detected or corrected by the participants, they were recorded and evaluated (Table 2). In this experiment, Digital Natives and Digital Immigrants have both made errors. The numbers of slips and lapses were similar in two user groups. They were caused by each individual's failures of action or failures of memory. Unfortunately, they cannot be perfectly solved by improving product instructions. When looking at the number of mistakes made by the participants, data revealed that multimedia instructions are better for avoiding mistakes, for both Digital

> **Multimedia (DN)**

When each task section was examined (Table 3), it was more evident that multimedia instructions can help users to avoid mistakes, especially in some complicated tasks for example task section 3-B. This section contained the most time consuming jobs in the whole experiment. It was a combination of many small knowledge-based tasks and some tasks required users to make critical decisions. Both Digital Natives and Digital Immigrants have made a few mistakes in this job section when traditional instructions were provided. However, when multimedia instructions with equivalent contents were presented, all users either avoided mistakes or realised errors on their own. Thus it is not difficult to conclude that multimedia instructions are better than traditional instructions (with only images and

Although multimedia instructions are not always more efficient than traditional ones, the authors believe that they are quicker to use for many types of tasks. They allow more flexibility in action and more importantly can help to reduce human errors significantly in the working process. They are effortless to access, can be transferred between different digital platforms and easy to update or replace. It is also proved that they can be used easily

by people who have grown up with or without ubiquitous of digital media.

1 Finding information 0 0 0 0 2 Unpacking- check parts 0 0 0 0 Section 3-A: Make the frame base 1 0 0 0 Section 3-B: Finish the frame 0 3 0 2 Section 3-C: Fix the top sheet 0 1 0 0 Section 3-D: Attach the accessory 0 0 0 0

**Print (DN)**  **Multimedia** 

**Print (DI)** 

**(DI)** 

Slips 2 1 1 2 Lapses 0 1 0 0 Mistakes 4 2 1 0

Natives and Digital Immigrants compare to the traditional instructions.

Table 2. Error analysis

**Tasks** 

**9. Conclusion** 

Table 3. Mistakes in each task section.

text) in terms of ensuring accuracy of actions.

Print DN Print DI Multimedia DN Multimedia DI

human error was found and recorded. The failures mainly appeared in two different ways (Hollnagel, 1993): the user's plan was adequate but the actions were deficient; or actions were carried out as planned but the intention was wrong according to the given tasks.

To define and classify human error in the tasks, three questions were referenced (Reason, 1990), and they all closely related to intentions:


The questions then were used to categorize discovered errors into three main kinds: slips, lapses and mistakes (Figure 12).

Fig. 12. Generic Error Modelling System (GEMS) (Reason, 1990)

#### **8.3.1 Slips and lapses**

Both slips and lapses were errors that happened in the execution process so that users failed to achieve the desired goals. The intentions were proper, but results were not right. A slip mainly involved actions that were not happening as planned, while lapses were the errors that were caused by failures of memory.

#### **8.3.2 Mistakes**

Mistakes referred to failures in the planning and/or judgemental processes; they are errors in users' intentions (Norman, 1983). They might involve the selection of objectives, or the decisions of means, even actions for achieving the objectives. They are normally caused by either a failure of expertise or lack of expertise(Reason, 1990). They could be effectively avoided or reduced by communicating sufficient information, in this case, using product instructions. Therefore, the number of mistakes left by users after each test was the main criteria to evaluate the accuracy performance of the product instructions.


Table 2. Error analysis

human error was found and recorded. The failures mainly appeared in two different ways (Hollnagel, 1993): the user's plan was adequate but the actions were deficient; or actions were

To define and classify human error in the tasks, three questions were referenced (Reason,

The questions then were used to categorize discovered errors into three main kinds: slips,

Both slips and lapses were errors that happened in the execution process so that users failed to achieve the desired goals. The intentions were proper, but results were not right. A slip mainly involved actions that were not happening as planned, while lapses were the errors

Mistakes referred to failures in the planning and/or judgemental processes; they are errors in users' intentions (Norman, 1983). They might involve the selection of objectives, or the decisions of means, even actions for achieving the objectives. They are normally caused by either a failure of expertise or lack of expertise(Reason, 1990). They could be effectively avoided or reduced by communicating sufficient information, in this case, using product instructions. Therefore, the number of mistakes left by users after each test was the main

criteria to evaluate the accuracy performance of the product instructions.

carried out as planned but the intention was wrong according to the given tasks.

1990), and they all closely related to intentions: 1. Were actions guided by prior goals? 2. Have actions been proceed as planned?

Fig. 12. Generic Error Modelling System (GEMS) (Reason, 1990)

3. Was the desired end achieved?

lapses and mistakes (Figure 12).

**8.3.1 Slips and lapses** 

**8.3.2 Mistakes** 

that were caused by failures of memory.

During the tests, by referring to given instructions, users observed and corrected their own errors in the performing progress. When errors were not detected or corrected by the participants, they were recorded and evaluated (Table 2). In this experiment, Digital Natives and Digital Immigrants have both made errors. The numbers of slips and lapses were similar in two user groups. They were caused by each individual's failures of action or failures of memory. Unfortunately, they cannot be perfectly solved by improving product instructions. When looking at the number of mistakes made by the participants, data revealed that multimedia instructions are better for avoiding mistakes, for both Digital Natives and Digital Immigrants compare to the traditional instructions.


Table 3. Mistakes in each task section.

When each task section was examined (Table 3), it was more evident that multimedia instructions can help users to avoid mistakes, especially in some complicated tasks for example task section 3-B. This section contained the most time consuming jobs in the whole experiment. It was a combination of many small knowledge-based tasks and some tasks required users to make critical decisions. Both Digital Natives and Digital Immigrants have made a few mistakes in this job section when traditional instructions were provided. However, when multimedia instructions with equivalent contents were presented, all users either avoided mistakes or realised errors on their own. Thus it is not difficult to conclude that multimedia instructions are better than traditional instructions (with only images and text) in terms of ensuring accuracy of actions.
