**5. Discussion of results**

In this section we discuss and answer the research question of this study.

#### **RQ: How to incorporate the focus groups technique in a real OSS project?**

The usability techniques have been created for another type of software developments, i.e. they have not been conceived with the specific characteristics of the OSS development process in mind. For this reason, it is necessary to adapt these techniques. These adaptations are based on the adverse conditions these techniques present. Some adverse conditions can be overcome using certain web artefacts (for instance, wikis, forums, blogs, etc.), which are known by the OSS community. As a result, many of these adaptations will be familiar to the members of this community, which favours to a certain extent the application of these usability techniques.

The adaptations of the focus groups technique are mainly two. Firstly, users participate online though a web artefact: a forum. Secondly, the usability expert is replaced by a developer, an experienced user or an HCI student under supervision of a mentor. In our particular case, the expert was replaced by an HCI student under the supervision of a mentor.

After applying the focus groups technique to the ERMaster project, we were able to confirm that it is very hard to get a representative set of users. We believe that the main reason for this is that users are unmotivated. We had to be persistent and use different communication mechanisms (for example, personal wikis and electronic mails) to get the consent of the principal developers (only one out of five principal developers responded). The biggest problem with applying the focus groups technique was user availability: most users are volunteers and had very little spare time. In fact, the participants did not have the time to enter their comments in the online forum and ended up emailing their opinions to one of the researchers. Since the focus groups participants had a medium level of experience with respect to both the ERMaster tool and the field of computing, they did not pinpoint any major problems which novice users may have had.

The adoption of the adapted focus groups technique was acceptable as this technique requires a small number of participants to get reliable results. With regard to our proposal of substituting a developer, expert user or HCI student under the supervision of a mentor for the usability expert, the expert was replaced in this case by a HCI student supervised by a mentor. Note that this student was in his final year of the Master of Information and Communication Technologies Research and Innovation at the Autonomous University of Madrid and was taking two HCI courses. Additionally, the student was supervised by two usability experts. On this ground, there is no risk of the proposed adaptation for the selected technique having a negative impact on the quality of the software.

We can conclude that the results of the adoption of the focus groups usability technique were not what we expected. Firstly, we banked on the participation of a large number of users based on the statistics provided on the application web site. Secondly, it was hard to contact and recruit users to participate in the research. Note that OSS community members are all volunteers, and they participate in their spare time. Despite all these problems, however, the adaptation of the focus groups technique was reliable for adoption in the ERMaster project, as it does not take many users to get a reliable result.

The main limitation of our research is the number of case studies (only one). This is preliminary research. Therefore, more cases studies are required to validate the proposed adaptations. Note that there are other usability techniques (for example, user profiles, heuristic evaluation) that might benefit from the proposed adaptations (e.g., HCI students supervised by a mentor standing in for experts) to enhance technique adoption in the OSS development process. Briefly, the results of our research are not very generalizable because we conducted only one case study. Therefore, the focus groups technique needs to be applied to other OSS projects. However, the preliminary results provide a basis on which we can build to improve the performance of other case studies.
