**3. Research results**

Results of the correlation coefficient analyses are shown in **Table 1**. These demonstrate that there are positive, significant correlation between the COTS game and multiple-choice assessments in three of the four studies; Study 1 *r* = 0.23, *p* < 0.01; Study 2 *r* = 0.27, *p* < 0.01; and Study 3 *r* = 0.36, *p* < 0.01; but not Study 4 *r* = 0.02, *p* < 0.05. This shows that in three of the four studies presented here, the score generated by playing the COTS game was significantly related to the score earned by the participant on the traditional multiple-choice assessment. These results are valuable for the area of COTS games because they show that when selected carefully, the scores produced in a COTS game might provide meaningful information for individuals wanting to use COTS games for learning or assessment purposes.



*\* p < 0.05, \*\*p < 0.01. M = mean, SD = standard deviation, COTS = commercial-off-the-shelf. Sex at birth coded as 0 = female, 1 = male. The COTS game scores and multiple-choice assessment scores are percentages. Estimates of reliability are bolded along the diagonal.*

#### **Table 1.**

*Descriptive statistics and correlations for Studies 1–4.*

### **4. Discussion**

The purpose of this chapter was to provide information about the application of COTS games as game-based interventions. In the research series presented as part of this chapter, we sought to build upon a growing body of research literature on COTS games by showing the convergence between COTS game scores with traditional multiple-choice assessment scores on the same content. Overall, the research results answered our research question and showed that COTS game scores significantly and positively correlated with the traditional knowledge assessment scores in Studies 1–3 but not Study 4. Although a non-significant relationship was found in Study 4, we note that these results may be attributed to a limited sample size. Data collection was halted during Study 4 due to the onset of the COVD-19 pandemic and local laws prohibited in-person gatherings including the collection of in-person data which was part of this research protocol. Overall, these results suggest that when a COTS game is considered and chosen intentionally, that the COTS game scores may be reasonably similar to a traditional knowledge assessment score. We caveat these results by stating that these findings cannot generalize outside of these particular games and provide primarily a proof-of-concept to the consideration and use of COTS games. We encourage researchers and practitioners to collect and evaluate their own evidence for the COTS games they select and the intended application of those games.

#### **4.1 Intentional game selection**

The meaningful relationships between the COTS game scores and traditional assessments highlights the potential benefits of using game-based interventions. COTS games may provide viable solutions to learning and assessment contexts but must be considered carefully with the particular purpose of the game in mind. While some

### *Commercial-off-the-Shelf (COTS) Games: Exploring the Applications of Games for Instruction… DOI: http://dx.doi.org/10.5772/intechopen.103965*

COTS games may demonstrate meaningful uses without being intentionally designed to do so [7, 9], it is important to note that not all COTS games will demonstrate similar results. Here we highlight the importance of selecting the right COTS game for the intended purpose. To avoid potential pitfalls of using a COTS game, individuals can prioritize the selection process by incorporating time for planning and review into the early stages of their project. Having a thorough understanding of the differences in game design and content is critical to selecting an appropriate COTS game for the intended purpose. As mentioned earlier in this chapter, learning and assessment are examples of two desired outcomes. To further illustrate how a COTS game could be reasonably connected to the end goal, we provide more detail about the content of the COTS games used in the series of research studies presented in this chapter.

#### *4.1.1 Study 1*

In the game *Quintet*, the primary content focuses on players learning their distinct responsibilities (i.e. the Scientist role can survey the surroundings while the Engineer role can repair and boost various functions) while learning to communicate and coordinate their actions with their teammates. Because the game missions require teammates to coordinate the timing of their actions, gameplay inevitably promotes active communication and collaboration between teammates. Teams will automatically perform better in meeting the mission objectives when they are actively communicating and coordinating their different actions.

#### *4.1.2 Study 2*

In the game *Arm Surgery 2*, the player is presented with factual information about specific surgical equipment and procedures. During gameplay, players are guided through several procedures where they have to select the correct tool and perform the correct action before the game can progress. Progressing through the game naturally promotes active attention towards these tools and procedures as players must monitor and maintain the patient's health while performing their tasks to earn a higher score.

#### *4.1.3 Study 3*

In the game *PC Building Simulator*, players are responsible for their own PC repair shop. In order to improve their score, they must accurately diagnose the computer's problem, order the correct parts, and repair or replace those parts before returning the PC to the customer. When players aren't sure about particular parts, they can go to a guide to click on parts for the name and information on that part. For the gameplay to progress, players must continue to accurately identify and repair the defective components.

#### *4.1.4 Study 4*

In the game *Car Mechanic Simulator 2*, the player owns a car mechanic shop. Similar to the game in Study 3, the player must accurately diagnose the problem, order the needed parts, and replace those parts in the car. The individual can also go to schematics which will give the name and primary functioning of the different parts that will help them when trying to understand and diagnose a problem. Thus, gameplay naturally leads to improved understanding of the different parts of a car and their functioning.

Each of these topics are relevant to workplace learning and may present a relevant context for training or assessment. For example, in Study 1, participants must learn their role in the game but also communicate with members of their team, coordinate their actions, and effectively work together to meet the game objectives. In this instance, the participants cannot be successful together unless they work as a team. Although the environment of being in space does not present a typical work environment, players were actively engaging in workplace applicable teamwork skills. In contrast to this, the physical environment of the games in Studies 2–4 were relevant to the fields of their subject matter (healthcare, computer science, and automotive industry, respectively). As an increasing number of industries continue to consider and apply game-based interventions, the variety of different COTS game contexts becomes more relevant [41–43].

#### **4.2 Low and high stakes contexts**

While COTS games offer a reasonable alternative to the resource intensive commitment of developing a game, they do not provide the same precision and control compared to traditional methods, which can be an important component in some training and assessment contexts [9]. Because of this, it is important to only use COTS in particular contexts when this precision and control is not critical. Beyond assessing how the content of a COTS game is associated with the intended purpose, it's imperative to consider whether the data will be used in a low stakes or a high stakes scenario.

High stakes scenarios occur when results could have a large, consequential influence on a person's access to or receiving of resources. This means it might impact their pay, promotion, or selection into a program, course, or position. A low stakes scenario occurs when the results are used for general information given to the individual or for developmental purposes when shared with a larger institution. For example, using a game to assess applicants for a job would be high stakes. Whereas, using a game to assess high school students' language skills to provide feedback on how to better prepare for college would be a low stakes scenario.

An additional consideration to distinguish high and low stakes scenarios is when a cost is associated with the service and when a claim is made with regard to the actual result of the intervention. For example, someone paying for a training course intended to improve their skills with a software is high stakes and should have evidence that this program improves the skills in individuals that it claims to improve. Similarly, if the program is free but it claims to improve the skill, it should still provide this evidence based on its claim. In contrast, if the program is a free game intended for entertainment purposes and does not make claims about improving cognitive functioning or other skill development then it is low stakes.

High stakes scenarios should be approached with extra care and use instruments or interventions with strong psychometric evidence on the reliability and validity of intended use. To this end, we recommend COTS games to be used primarily in low stakes scenarios. Given the degree of importance and variation that can occur in high stakes scenarios, control over a game or gamified experience is paramount so that any bias or unintended negative consequences can be reconciled. COTS games are typically used without the option to customize aspects of the game, hindering the ability to identify and rectify any pitfalls. Using established and rigorous science-based processes for the development of game-based interventions is especially important for high stakes training or assessment purposes. GBL solutions are best developed following empirical instructional system design (ISD) methodology while GBAs should

follow standard assessment development and validation processes to demonstrate the psychometric properties of the assessment [9].

#### **4.3 Recommendations for software developers**

Given the results of this paper, it appears that COTS games have potential uses beyond entertainment purposes when selected with intention and applied thoughtfully. Below are recommendations for software developers regarding actions which might benefit a COTS game for being used for alternative purposes.

Firstly, software developers can make a variety of game metrics available to collect from gameplay as providing frequent and detailed feedback on performance is a key component of both training and assessment [44]. When games are used for training or assessment purposes, having detailed metrics (e.g., how long a player spent on a level, how many incorrect selections were made etc.) can be a helpful device. COTS games which provide a metrics sheet or summary of data would greatly aid in the game's accessibility for alternate purposes. For example, many Massively Multiplayer Online Role-Playing Games (MMORPGs) and Multiplayer Online Battle Arena (MOBAs) make detailed game metrics publicly available for players to view statistics of their own and other players' performance to better understand and improve their future gameplay [45].

Secondly, when possible, it would help to have software developers create easily customizable options in their games that could increase the usability of the game for alternative purposes. For example, some games could add additional modes (e.g., single player, cooperative, or competitive modes) which allow for alignment with a greater variety of intervention purposes. Games could also provide more content related tutorials so players have the opportunity to learn about the context of the game in a tutorial rather than just learning through game mechanics. For example, a medical based game could provide specific information about tools, procedures, and different considerations when caring for a patient. Additionally, learning and assessment components can be an added benefit when thoughtfully integrated into other contexts such as educational settings [46].

Thirdly, when a game is naturally aligned to an area that might have potential for alternative uses outside of entertainment (e.g., game has content relevant to child or adult learning or the content might potentially assess a relevant topic area) it would help to have software developers seek input or feedback from potential users. This may include contacting educators to ask about particular features they would want in the game if they were to use it for learning or posting summaries of the game on online forums and asking for general ideas on how to improve the game if it were used for assessment purposes. Understanding, and marketing, the particular qualities and game characteristics that would be most beneficial for alternative uses would help in promoting the selection of the COTS game.

#### **4.4 Understanding COTS games and modern technologies**

Considering the rapid emergence of new technologies, it is relevant to consider how some of the most recent innovations in the gaming world might impact the results discussed here. New technologies such as XR and VR offer high fidelity experiences in contexts that can be created and customized for individuals to experience. The potentially high impact of these new experiences involves the degree of realism they create for the player. Several researchers have begun to explore these

technologies and their potential uses for training and assessment purposes, highlighting the hyper realistic environment as a key feature that future researchers and practitioners can capitalize on [3, 27].

This realism can be a valuable training or assessment component of the game or experience for individuals. For example, if trying to teach medical students how to perform a particular procedure, it may be beneficial to have them perform this skill under a variety of conditions such as in a quiet, focused space but also in a loud, and busy space that might better replicate the environment of an Emergency Room. Using XR or VR, the procedure could be adapted to different scenarios, and embedded within a seemingly realistic ER environment that allows common sights and sounds from this experience to be included within the training session.

Similarly for assessment, it may be critical for an electrical engineer to be able to perform specific repairs on power lines and equipment. In XR or VR, these repairs could be assessed in an environment that emulates a heights scenario, where the participant truly feels they are at the top of a powerline. The sights, sounds, and sensations they experience can bring a sense of presence and embodiment that is difficult, if not impossible, to emulate in non-XR or VR environments. In short, it is clear that there are new technologies emerging that could further enhance the usability and application of future COTS games.

### **5. Conclusion**

Research has demonstrated that some game-based interventions may provide benefits in learning and assessment contexts. As the research on game-based interventions continues to mature, more empirical evidence is being produced on how to design and implement games in a more impactful way [47, 48]. Game-based interventions have much potential [9], but game design can be resource-intensive, making COTS games an attractive alternative when the requisite time, resources, or expertise are not available for game development. In the focal research series, we sought to explore how COTS game scores converge with traditional knowledge assessment scores and found preliminary proof-of-concept evidence that in this context, three of the four COTS games demonstrated significant convergence with the respective traditional, multiple-choice assessment. Although these findings do not generalize outside the context of these particular games, they do provide a promising future to the further investigation of COTS games as viable solutions in low stakes instructional and evaluative contexts. Overall, we recommend COTS games within particular contexts where a well-suited game is chosen and encourage individuals to exercise intention in how the COTS games are used.

### **Conflict of interest**

The authors declare no conflict of interest.

*Commercial-off-the-Shelf (COTS) Games: Exploring the Applications of Games for Instruction… DOI: http://dx.doi.org/10.5772/intechopen.103965*
