Subjective Measures of the Influence of Music Customization on the Video Game Play Experience: A Pilot Study
by Alexander Wharton, Karen CollinsAbstract
Many games and consoles today allow for a player to substitute a personal music playlist into the video game. We examined the influence that a player’s choice of music has on the player’s experience in one particular game, Fallout 3: Operation Anchorage. Players specifically chose music for the purpose of relieving anxiety, improving tactics and to experience immersion. Results showed that players were unable to predict what music would improve their immersion, but were able to choose appropriate music to influence game playing tactics and anxiety levels.Keywords: music, customization, players, choice, interaction, empirical study, multimodality
Introduction
Digital technology allows for wide-scale personalization and customization of our media content, from interactive music formats like MXP4 that allow us to choose what instruments we want to hear in a song, to personal video recorders (PVRs) that let us select which programs we want to watch and when. On the Internet, the notion of Web 2.0 and user-driven content has grown in a single decade to the point where it is now taken for granted. Participation is one of the hallmarks of new media, and video games are no exception. As Marshall McLuhan and Barrington Nevitt had predicted as early as 1972 (p. 4), the consumer/producer dichotomy blurs with new technologies, as consumers - through appropriation, customization and personalization - change the end product to suit their own needs and desires. Today we often refer to ’co-creative media’ to describe these producer-consumer relations, in which neither the game developers nor the game players are the sole creators and mediators of a game, but that players, through customization and personalization, bring their own content, meanings and ideas into the game (see, e.g. Morris 2003).The customization and personalization of games through the advent of computer game modifications or “mods” has been one aspect of this drive towards user-customizable game components. Modding describes a number of activities that can overlap, depending on the abilities and desires of the person (or persons) modding the game. This can range from simple changes in graphics or characters, to more advanced level mapping, to entirely new versions of a game based on the original game’s engine. Modification communities or “modders” have become a significant marketing factor for computer game developers, especially in first person shooter, role-playing, and real time strategy games (such as Call of Duty 4: Modern Warfare [Infinity Ward 2007], and Dragon Age [Bioware 2009]). But modding, like other customization and personalization activities, exists in a grey area between fans and the corporate world, which has led to some criticism of the game industry as taking advantage of the free labour of fans to extend shelf life, create brand loyalty, and reduce R&D and training costs (Sotamaa 2007). It has been suggested, for instance, that up to 90 percent of The Sims (Maxis, 2000) content was produced by the players (in Sihvonen 2009, p. 52). And, if your customers are willing to create nearly all of your content, where is the motivation to release a finished product, or to treat a game as a gesamtkunstwerk, a ‘total work of art,’ where the end result is more than the sum of its parts?
At stake for the developers is the issue of control over their intellectual property. For instance, for Microsoft, the ability of a player to input their own music has meant that players can no longer complain about game music’s repetitive nature, but could instead listen to music of their choice. For game composers, however, this particular trend means a step backwards in terms of the idea of the music as a part of the gesamtkunstwerk of the game. While customization might be a desired trait for the consumer, the designers of the game (along with the composer/s) must relinquish control over the musical soundtrack to the game. Such a feature would be unheard of in film, notes sound director Rob Bridgett, who describes, “One cannot imagine, for example, removing Howard Shore’s score from such a fully integrated work as The Lord of the Rings movies and replacing it with user defined content” (2010, 23). He adds, however, that unlike a film, games already offer individualized play- every game is different, and every player is different to a certain degree- and music could be seen as another layer to this individualization, perhaps as part of the larger modding idea of games.
While there are certainly many positive sides to customization, the changes in functionality and aesthetics can alter the intended meaning(s) of media, as the auteur becomes just another cog in the wheel of the production, with their vision infiltrated by the thousands of players adding and changing content. Many questions remain about the impact that this customization might have on the player’s experience of a game. Without a cohesive whole sold as a final product like films or books, what meanings are altered by this co-creation process? When it comes to sound, in particular, game soundtracks are carefully designed around a game’s emotional content, genre, style, action and narrative in order to fulfil a variety of important functions (see Collins 2007). But if players can - as we will show below - customize musical content in their games, what happens to that careful integration of sound, image and gameplay in terms of the experience of the player? This paper seeks to explore the questions through a small pilot study in which players were able to customize the musical content of a game. We first discuss the technical aspects of this co-creative practice, followed by a description of our methods for the pilot study, and finally we present the results with a discussion of the implications suggested by our findings.
Sound and Music Customization in Games
The terms ‘personalization’ and ‘customization’ are often used interchangeably, but many writers agree that there is a distinction to be made between the two terms. The trouble is, however, very few authors agree on what this distinction is (see, e.g. Wachob n.d. or Allen n.d.). For the sake of clarity, we have chosen to define them as follows: The ability to customize a game is provided at the design level: some games are designed, for instance, to allow you to create your own custom avatar. In other words, customization takes place according to a pre-determined series of features or options that are built into the game. Personalization, on the other hand, is not built-in or planned by the designers, but is created or ‘hacked’’ by the end user. In simple terms, player alterations can be intended (customized) or unintended (personalized) by the designers. There is, however, a fine line between the two, as once products become personalized by enough people, the personalization is often co-opted by the company and turned into a customization feature for future versions.Video game music has traditionally been viewed as highly repetitive and often disconnected from gameplay. In the early days of video games, music was often an afterthought, dumped into games in late-stage development without consideration for the needs of the game in terms of genre, emotional content, context, and so on (see Collins 2008). The repetitive nature of game music meant that many players would turn the sound off altogether. This was detrimental to play in some places, as sound effects can often provide the player with important action cues. Over time, the ability to turn off sound evolved into the ability to selectively turn off music separately from other sounds. In part as a response to the repetitive looping nature of game music, game composition has developed to be interactive with the player’s movements and actions, and adaptive to game-play events. In real-time, compositions may adjust to the game-play, as transitional sequences bring the music from section to section to reflect in-game events. This dynamic composition and its integration into games has been further developed over time, as composers have created techniques and tools (i.e. middleware engines) to ensure that game music is responsive to both the player (‘interactive’) and the system-generated game parameters (‘adaptive’ - see Collins 2007, 2008).
Despite the attempts of composers to create dynamic scores and soundtracks for games today, many players still play the game for many hours (Resistance 2 [Insomniac 2008] for instance boasts at least 420 hours of play - see McWhertor 2008), and the repetition of the music is inevitable. One solution to this problem came with the customization of musical content.
It could be said, of course, that all sound in games is customized in the sense that the player has a unique sonic experience every time they play a game. The timing of sound events tied to game actions, in addition to the randomized playback of key sounds, (not to mention the environmental sounds external to the game,) means that the player never experiences the same game exactly the same way twice. While the player may not be intentionally altering the experience (the usual reason for customization), they are nevertheless experiencing something new each time they play. Likewise, there are a variety of ways to customize musical content, including spontaneous online jam sessions (e.g. in The Lord of the Rings Online: Shadows of Angmar [Turbine 2007]), the ability to compose one’s own music (as in Guitar Hero World Tour [Neversoft 2008]), procedurally generated musical content based on player selections (as in Spore [Maxis 2008]), and procedurally generated visual content based on player’s input music (as in Vib Ribbon [SCEI, 1999]). These other means of customizing audio content are interesting, but outside the scope of this paper (for an overview, see Collins 2008).
Perhaps the most significant means of customization of sound content in recent games is the player’s ability to input one’s own choice of music into the game. This has in the past few years been accomplished through various means: the first means by which players can customize their music is in games that have planned for customized musical content. Examples include Habbo (Sulake 2000), which began as the social networking site Habbo Hotel aimed at teenagers but now incorporates Flash-based games into its interface. In the Hotel, players can make their own ‘guest rooms’ which can be customized with their own shapes, furniture, wall patterns, virtual pets, and so on. In 2007, Sulake added the Traxmachine, “putting the power in your hands to fill the hotel with sound!” 1 Players could load up to four ‘Traxpacks’ of 9 samples each to create a soundscape or musical background to their room. Grand Theft Auto (Rockstar 1997), The Sims series, and various Gran Turismo games (Polyphony Digital 1997) also made concessions in the design to accommodate player music (e.g. to be played through a car radio in Grand Theft Auto).
A second means of inputting a player’s own music is occurring in multiplayer online games and virtual worlds, as players use their microphones to input sound and music into a game through voice-over internet protocol (VoIP). In fact, this has become so popular that VoIP software developers Teamspeak have now designed the idea into their third version of their software. It’s also possible in some online games and virtual worlds to stream music directly into the game, thus sharing music with others through the online space.
A third means by which players can achieve this is through modding. Some game companies, as previously mentioned, are quite open to the practice of modding and release source code for the purpose of encouraging fans to edit and share new levels or content based on the game. In these cases, the entire game can be re-designed, and new sounds and music incorporated into it based on the altered needs of the modification. On the other hand, some games are not designed specifically for modification, although it may be possible to edit the library of game assets (e.g. sound files), and thus the player can over-write the original sound files by copying in new files with the same names without changing the actual game in other ways. For this latter practice, there are often game WIKIs set up by fans or by the game designers to help users identify these files.2 Likewise, some players even go so far as to develop modding utilities that aid the user in automatically over-writing music and sound effects. For example, the AudioMod for Minecraft (Mojang 2009) allows the user to add new music and sound effects to the game, alongside the originals.
Finally, when Microsoft introduced their Xbox360 console in 2005 they made it mandatory that all games produced for the Xbox360 must allow for player-customizable soundtracks. That is, the player must be allowed to plug in a CD, their iPod or another MP3 player and the game music would be shut off and the player’s music would play instead. It is theoretically possible to do this with Sony PlayStation and other consoles (PSP, Wii), but few game developers have used the function. In other words, there are a variety of ways in which a player can incorporate their own music into a game today. However, currently no studies have explored this phenomenon or the impact that this has on the player.
Methods
The game selected for the study was Fallout 3: Operation Anchorage (Bethesda Games Studio 2009), an expansion pack to the popular action strategy/role playing game Fallout 3, a game that won IGN’s Game of the Year Award in 2008. The original game is set in 2277, two hundred years after a nuclear apocalypse, but Operation Anchorage sets its game-play in the past, before the bombs. The Chinese have invaded Alaska and the player must progress along catwalks, caverns and cliff edges in fighting their way to disable an artillery base.Fifteen volunteers (8 males, 7 females) aged 19 to 30 - all current or former students of the University of Waterloo - were invited to participate. These participants had varying degrees of experience with the game or with similar games, from no experience at all with the Xbox360 to considerable experience with this particular game. Inevitably those participants with greater experience playing this particular game or other similar games using the Xbox360 controller reported greater overall levels of immersion and less anxiety. In contrast, participants with minimal experience encountered some obstacles to significant immersion, with one participant never being able to overcome the complexity of the controls in the allotted time. Most participants overcame controller-based barriers quickly, thus we felt that the impact of the controls was minimal overall.
In addition to game playing expertise, participants reported a variety of levels of experience with personal music players. One participant, ‘Katherine’ 3, stated that she listened to her iPod “all the time” and found it unsettling when there was no music at all. ‘Julian’ brought his own iPod with selected songs to incorporate into the study, but added that he normally listens to the iPod when travelling, and uses the game’s specially-composed music or no music when playing games at home. It is quite possible, therefore, that the study’s results may vary in terms of how the listeners normally use music in games. For those that never incorporate their own music into a game, but prefer to use the composer’s choice of music designed specifically for that scene, the swapping out of the game’s score for (often) popular music in a game may be particularly jarring. For those that are accustomed to listening to their own music while playing a game, however, it is likely that the impact on the experience of the game would be less. Due to the number of participants in the study, we chose not to separate the results based on their experiences, but to take the sample as a whole as an indicator for future points of study. Future research should seek to determine if there is a difference in experience over time as players acclimatize themselves to listening to interchangeable music tracks in games, and if there are differences between regular music listeners and those who do not regularly listen to portable music players, as well as hardcore versus casual gamers.
As mentioned earlier, it is possible to turn off the composed game music and insert a player’s own music into any game on the Xbox 360. Participants were given the opportunity at designated checkpoints in the experiment to choose from either a pre-selected group of up to 25 songs representing a range of genres, or to incorporate their own musical selection into the game (using an iPod, MP3 or CD player). All players chose to listen to the music through either headphones or speakers, the majority of which elected the use of speakers. At each checkpoint, players could continue with their previous selection or choose new musical material.
The participants selected an avatar and played through a pre-selected level for approximately 45 minutes each. Participants used the think-aloud protocol during game-play, and were asked at each checkpoint to evaluate and reflect on the effects of their choice of music on the gameplay in terms of degree of perceived immersion as well as actions or tactics in the game. The interview at each checkpoint was open-ended, using unstructured questions relating to gameplay and immersion. Here, we defined immersion intentionally very broadly as engagement with, engrossment in, and feeling of presence in the game, as used by Brown and Cairns(2004). We left it up to our participants to describe whether or not they felt immersed (as they themselves defined it). Although the brief interruption at checkpoints may have impacted results to some degree, we deemed that putting off the interview until the final segment would have a larger negative effect regarding participants’ recall ability of the first few game sections. Likewise, we recognize that the lab setting may also have impacted the results, although this is, of course, a perennial problem with empirical participant research. At this stage, our goal was to gather enough qualitative data of a general nature in order to better develop our own quantitative metrics for later studies, and we felt that these limitations were manageable for the purposes of our study.
Results and Discussion
The study revealed that there are several contributing factors when it comes to the influence that musical customization has on the game-play experience. These include an influence on the immersion levels and the emotional involvement reported by the player, multimodal interactions between music and story/action/image, and psychophysiological changes resulting in different levels of anxiety and different playing tactics. A more detailed examination of these will reveal the ways in which a player’s personalization of musical soundtracks can influence their game-play. Please note that these categories do overlap to some extent, thus they are not mutually exclusive.The influence on immersion and emotional involvement
Previous studies have shown music and/or sound in general to have an important effect on the degrees of immersion experienced by the player. Mark Grimshaw has extensively studied sound and immersion in the first person shooter genre by using sound on/sound off subjective and objective tests, arguing that sound effects contribute to immersion (see, e.g. Nacke, Grimshaw and Lindley 2010). Kristine Jørgensen (2008) has likewise shown that sound influences the functional aspects of game-play, as well as the emotional connection to the game world. However, these study findings are not directly transferrable since both Grimshaw and Jørgensen focus on sound effects rather than music, and this condition was not altered in the study here since sound effects were always left on. Lipscomb and Zehnder (2005) tested the absence and presence of the specially-composed music for the game The Lord of the Rings: The Two Towers (Electronic Arts, 2002), and found a variety of impacts that the soundtrack had on the game. These prior studies indicate the importance of the existence of sound or music to the feelings of immersion and emotional connectedness to a video game. It has also been shown that sound and music in other multimedia (when used congruently with the image) increases the emotional involvement of the audience (see, e.g. Bullerjahn 1994, for instance). While these studies are useful at corroborating the overall impact that having sound and music has on media, what has not previously been explored when it comes to video games, however, is the impact that altered music - and the control over that music - has on the player’s experience.Some of our participants chose to try to play the game without music (while keeping the sound effects), with mixed results. ’Sandra’ reported that the game felt more mechanical, that he was “just going through the motions” and was emotionally disengaged from the action when the music was turned off. Others, however, found that without the music, they could better concentrate on the game, feeling that the music highlighted an inability to master the controller, perhaps due to the fact that sound effects (tied to player actions such as button presses) could be more readily heard when the music was switched off. Some players felt that they could become more immersed in the game without music, as the game-play was more life-like and realistic. Sandra also stated that without music she found herself more invested in the game and began “talking to the characters” with nothing to distract her from the situation. She commented “it’s like playing paintball where you want to be in control of everything so there’s nothing ambient going on … I feel like I’m this character more, rather than fooling around bored at home.” Some players found that music was critical to masking external noise and distractions, thus helping to increase immersion. Such use of music was important in the early days of film to cover the sounds of the projector (see Collins 2007). There were construction noises occurring nearby during many of the testing sessions, and participants felt that when the music was turned off, they were more attuned to these external environmental sounds, whereas the presence of music enabled them to successfully ignore these distractions.
In other words, the presence or absence of music in our study had mixed results. Given that many studies of music in film have shown the importance of the music in the emotional engagement of the audience, our own mixed results suggest that the involvement of the player - that is, their control over the game environment, or their control over the musical content - alters the relationship that some players have to game music, depending on their levels of expertise and experience.
Another interesting factor that impacted upon the experience was the degree of familiarity that the players had with the selected music. Facing the initial choice of whether to incorporate their own iPods or musical playlists into the study, some participants desired and appreciated the opportunity to select songs that they knew and liked. A participant’s prior association with the selected music (particularly if tied to other intellectual property) had varying degrees of influence on the level of immersion reported as well as the game-play. ‘Matthew’ tried the indie-rock song, ‘M4 Part 2’ by the Faunts, and found that it reduced his immersion in the game. He reflected that the song had been previously featured in Mass Effect (BioWare 2007), a game he had also played. The selection by ‘Andrea’ of John Mayer’s ‘Heart of Life’ led to what she described as a comical and distracting effect on her play, and she noted she was singing along with the lyrics as the song progressed.
Often, players selected familiar songs (or, at least, familiar genres) in order to achieve a desired emotional effect and/or in an attempt to be more immersive in the game. ’David’, for instance reported feeling apathetic towards the game-play at one state, and chose a heavy rock song (‘Wait’ by Earshot) in order to attempt to re-connect with the game. This trend was common, with other participants using energetic songs to refocus their attention back into the game environment and the combat scenarios. For two participants, the final portion of the level provided a significant barrier in choosing an appropriate song. When informed of the narrative, ‘Paul’ chose to have what he thought would be climactic music, ‘Miserilou’ by Dick Dale & His Del-Tones. However, his first attempt at completing the level was unsuccessful, and after keeping with his choice of music, he said that the repetition of the shorter song was distracting and too intense. His expectations of the music’s effect were not met by the practical result and worked as a dénouement rather than an exciting climax.
What is particularly interesting about the players’ choice of music is that, despite many participants being familiar with the game and bringing in their own songs with which they were familiar, participants were not able to consistently predict and select their own musical songs to increase immersion or enjoyment of the game. In other words, players lacked experience in understanding the complex interplay between music and a game, and were unsuccessful in choosing music that they thought would help them to achieve a desired effect. We shall return to this point later.
Song lyrics also affected the gameplay experience. For one song, ‘Mad World’ by Gary Jules (a cover of a Tears for Fears song), which includes lyrics such as “The dreams in which I'm dyin'/Are the best I've ever had,” participants remarked that it had a strong influence on their game-play. One participant, ‘Caitlin’, decided to change to another song before reaching the in-game checkpoint, saying “I feel like I’m going to die.” Two other participants found themselves distracted by the lyrics of their choice of a Lonely Island comedy hip-hop song, ‘I’m on a Boat.’ After playing through the song to the next checkpoint, Holly said she loved the song, although it may not have immersed her in the game-play as she was so immersed in the song. Paul described his experience as being distracting: he wanted to hear the lyrics rather than the game. After his character perished in a mis-timed explosion, ‘Aaron’ opted for the rave anthem ‘Sweet Harmony’ by Liquid. Although he said that the harmony and peace espoused by the lyrics made the ending sequence epic, he found the anti-communist war theme (“don’t mess with the U.S.”) jarring in this instance. There was a clash between the meanings he associated with song’s lyrics and the war theme of the game. Despite the disparate layers of meaning between the lyrics and the game, the clash did not detract much from the experience, but instead the juxtaposition brought the game’s message to the forefront while the soundtrack became a backdrop and separate from the game. It appears then, that players are unable to ignore familiar lyrics in a song when they are playing a game. Moreover, they (consciously or not) attempt to make connections between the lyrics and the game, possibly due to the years of experience that most Westerners have in seeing popular music (through music videos) as much as just listening to music: we are accustomed to watching a visual that matches the lyrics of a song, and thus making the connections between visuals and lyrics.
In a similar fashion, the choice of musical genre influenced the game-play experience. In particular, the importance of congruence or perceived ‘fit’ between the military style of the game and the musical genre were viewed as important by the participants. For instance, big band, country and classical guitar music were generally viewed among the least immersive with Paul noted of the big band song, “it did not fit (the game), and it was intrusive.” For this participant, the song was so intrusive that he said he “just tuned everything out.” He focused only on visuals, and even ignored ambient sound cues or diegetic sounds from the game (e.g. gunfire) because, as he said, “I just couldn’t listen to it.” ‘Paige’ found that the incongruent classical guitar music increased her anxiety level. She described the song as not fitting “with the aura of the game.” Additionally, attention was drawn out of the game itself, and the participant found herself more “scattered and pressing buttons hoping something would happen.” A similar loss of concentration was found with ‘Sam’, a player who chose a classic rock song and who noted that he became lost in a section and continued to go in circles while the song was playing.
It seems clear, then, that not only certain lyrics, but also certain genres of music are more suited to certain types of gameplay. It would be an interesting future study to determine what these relationships are, and how and why they have historically developed through music’s cultural connotations. While scholars have explored the impact that certain musical traits have on the perception of media (see e.g. Tagg 2002), the impact of genre - and all of the cultural baggage that comes with popular musical genres - has been largely ignored. One reason for the impact that genre has of course, is the influence of musical traits on other modalities.
The influence on multimodal interactions
There have been decades of research into the cross-modal interactions that occur in multimodal media.4 Multi-sensory integration refers to the ways in which we synthesize information from two or more of our senses, such that the combination of those sensory inputs is more than what we would have obtained from each of the sensory modalities separately (Kohlrausch and Van De Par, 1999, p. 35). Film theorist Michel Chion refers to this as synchresis in a discussion of the “added value” relationship between image and sound effects in film (1994, p.1). In other words, when it comes to sound and vision, “the viewer/listener builds verbal and visual representations and connects between them” (Ben-Shaul 2003). This occurs, for instance, in cross-modal matching, which describes the process of using information from one modality to judge the equivalent stimulus from another modality (Storms, 1998). For instance, it has been suggested that there is a cross-modal matching between brightness, pitch and loudness, with increased pitch associated with increased brightness (Storms 1998).Much video game music - at least, for major ‘Triple A’ titles - is composed especially for the video game, and implemented into the game in such a way that it interacts with the player’s actions. For instance, approaching an enemy might trigger a dark, menacing cue, or the music may increase in complexity based on the number of other players nearby. In other words, the soundtrack is designed to adjust in run-time based on a number of in-game parameters (see Collins 2008). Experienced game players would typically recognize this important role that music can have in alerting them to dangers or providing other information about the game. Some players in our study, therefore, had expectations of the music composed for this particular game that left them disappointed. Caitlin wanted to try the game with the scored music because she wanted “to know when enemies are attacking, and to hear the sounds for when the fighting starts.” After playing through the level for a short period of time with the game music on, she reported that she was very dissatisfied. Her expectations were that there would be sound cues in the music that would indicate what was happening in the action. In essence, she wanted the music to mirror the action and to react to her actions. Her disappointment upon hearing a looping background atmospheric song resulted in a lack of immersion in the game. She noted that she usually plays games with the specially-composed game music, and that the dynamic nature of the music in many games (interacting with and adapting to game-play) has meant that she has learned to become attuned to cues in the music. This is an important point that we will return to later on.
Many players in our study were concerned with musical congruence or ‘fit’ with the game. David chose the soundtrack from another video game, Lineage II (NCSoft 2003), and found it “lent itself to coincidences” between the musical score and stealth combat encounters in the game. He relayed that he found himself more connected with the game, more surprised by invisible combatants, and more engaged in the gameplay than with other music. Although the soundtrack was from a different video game, the participant felt that it could have been composed specifically for Operation Anchorage. Sam found a film score (Vangelis’ 1492:Conquest of Paradise) was similarly “more like it was designed for the game” and that produced an “epic” feeling in the context of the game.
Other music - Holst’s ‘Mars, Bringer of War’ - was very popular with players as it was felt to fit with the militant nature of the game. At a checkpoint, ‘Matthew’ said that the song was the most immersive he had experienced up to that point, claiming, “at points I forgot I was even participating in a study.” Comparing it to the game’s original music, Caitlin described, “[‘Mars’] was better. It may sound strange, but I could hear the bullets more when that was playing.” Echoing the previous participant’s experience, Aaron found himself playing more aggressively, as he perceived that ‘Mars’ had “my heart up a bit more.” He also mentioned “the song sounds like it was made for the game.” However, although he lauded the experience as being very immersive, he did not want to keep playing the game with the same sustained level of intensity. Instead, he chose another song of his own which had a “slower heartbeat” which made him more calm and confident.
As described above, lyrics had an influence on the player’s perception of meaning in the game, and similar connections were made between the music and various levels of meaning. ‘Brian’ claimed that he was able to connect and make associations between the music soundtrack and the game, with some “serendipitous moments where things may have clicked in my brain.” Regardless of what was happening between the auditory and visual environments, the participant felt that he subconsciously made connections to fit everything together for his experience.
It is interesting that music influenced the perceptions of image, although this is not surprising given the body of research that illustrates similar multimodal interactions. What is more interesting - and far less researched - is that the inverse also occurred: the image also influenced the perception of the music. For instance, Sam identified “communist sounds” in the vocals of the soundtrack to 1492: Conquest of Paradise (actually sung in Latin). It is possible that the communist association here came from the fact that the combat in the game was between U.S. military members and Communist China soldiers.
A critical point to be made here regarding the influence of music on visuals (and vice versa) is that the players actively chose to attempt to meld the musical world with the virtual world, using each to represent and create meanings for the other. Choosing and listening to music was not viewed as a peripheral experience to game-play; rather, even music external to the game and selected by the player was viewed as an integral part of the gaming experience. It was not seen as something to merely cover up the background noise or provide additional entertainment, but served to add to the meanings and emotions created during the playing of the game. Players sought out this multimodal experience through their choice of music, which highlights the importance of what was referred to the introduction as the gesamtkunstwerk of the game: the experience of a game is not merely visuals, or gameplay, or sound, but a combination of these that work together to create the experience for the player.
The influence on player psychophysiology
One of the most important findings of our study was the role that musical choice played in the emotional, physiological, and physical responses of the player. Many participants described their motivation for song choices as being driven to reduce their perceived levels of state anxiety. Music has been shown to have a significant impact on state anxiety levels (where ‘state anxiety’ refers to the temporary state of anxiety, as opposed to a more long-term personality trait). Smith and Morris (1976) for instance tested students during a stressful situation (exam writing) and found that stimulating music significantly increased anxiety. Hébert et al. (2005), in studying music in games as related to state anxiety found that those who played with the specially composed game music had significantly higher cortisol levels (and thus were more stressed) during gameplay. But this difference in anxiety levels can be altered depending on the type of music played, and one of the significant factors in anxiety is whether or not the participant has control over the music (Thayer 1983). In fact, for those that can select their own anxiety-reducing music, anxiety levels can be significantly lessened (Pitzen and Rauscher, 1988). One study of musical choice found that listeners had a higher heart rate when they could not choose the music. Furthermore, they found that having choice significantly lowered galvanic skin response, indicating lower arousal (excitement/stress). They describe, “The study also found that subject-preferred music was potentially able to reduce the tension subcategory of arousal in older adults. Therefore, being able to listen to music that one chooses could result in lower levels of physiological arousal” (Miller et al. 2008).Participants especially motivated by an intended reduction in anxiety were those with a low level of familiarity with the game, console, or songs. Caitlin, after experiencing anxiety, chose to start with ‘Mars, Bringer of War’, but quickly described the music as being too intense and wanted to change songs even before the first checkpoint was reached. Unlike other participants who relished the intensity and found that it sounded sympathetic to the creative design of the game, this participant’s game-play was impaired by the combination of interacting with a new virtual environment in a combat scenario and the perceived intensity of the music.
The music composed for the game was also found by some to increase anxiety. ‘Katherine’ found herself getting involved with the game, yet discovered she was more anxious “like there was something around the corner that the music was building up to.” However, unlike some participants who found the anxiety unsettling and a potential impairment to game-play, the heightened anxiety was felt by this participant to immerse her more in the game. Paige found that it created a “tense feeling,” and said she was “nervous and uptight when people shot at me.” The song not only involved the participant in the overall game environment, but also provided just enough anxiety to enable the participant to progress through the game and react to combat scenarios intuitively. Although many players enjoyed a certain degree of anxiety, over sustained periods they would switch out anxiety-inducing music for music that they anticipated would calm them down. This suggests that a dynamic range of emotions is desired by most players, and that music can play an important role in facilitating that emotional range.
Another important noticeable effect of musical choice was on the player’s tactics (or perceived tactics, i.e. the players felt that they played differently, but did not necessarily do so). Some players noted a change of tactics depending on whether or not music was present. Paige reported that she was not as cautious without music (i.e. braver and more willing to take risks), but that her actions were more controlled. Paul reported that his tactics changed so that he keyed in on sounds of gunfire, footsteps, and dialogue in order to shape his in-game actions.
The tempo of the music selected in particular was an important factor in the pacing of action. Sandra reflected that music that was more upbeat propelled her “through very high-charged areas.” Once she had made that realization, she intentionally selected similar songs to “power through (areas) and kill people,” although at times she became “caught up in the rhythm of the music” and was not prepared for dangerous encounters. Matthew chose the upbeat trip-hop song ‘The Sleepless’ by Red Snapper and he said he was less cautious, more aggressive, and “more of a bad-ass” when the song was playing. David noted that the classical song he chose, Edvard Grieg’s ‘In the Hall of the Mountain King’ influenced him to use “more sneaky play” whereas other songs made him “just want to knife people” or “go through like Rambo.” Additionally, a slow-tempo LoreenaMcKennitt song led to some players reporting to be less active and non-aggressive, as if the game were in slow motion.
These finding are consistent with research on the perception of pacing of film shots along with different songs, where faster tempo music is consistently influential on creating a perception of faster paced images (Hung 2001). It is well known that fast-paced music is used by stores and restaurants to speed up the visits of customers (Kellaris and Kent 1992). In fact, it has also been shown that music’s tempo can influence the speed with which roulette players place bets in a casino (Spenwyn et al 2009), as well as increase intensity of exercise (Karageorghis et al. 2006).
Some players consciously opted for music that they knew would affect their tactics, in order to help them get through a tricky part of the game. Much like fighters choosing an entry song before they head into the boxing ring, players used music to psych themselves up for the gameplay. Aaron found a Wu-Tang song “kinda good and more spicy,” with “the music keeping up with the firefight.” He felt that he was more connected to a “shoot and kill/get’er done” frame of mind with this song. Aaron also noticed consciously changing actions to suit the music, such as attempting to “be more fluid and go along with the (reggae) music.” Yet another player, Sam, said that playing with a symphonic metal song meant that he was more aggressive and “running with the music as it got exciting,” and described that he “was using VATS (the in-game auto-targeting system) more when this song was running.” The opposite to the wild run-and-gun tactics were also true, in that Paige noted that the film soundtrack that she had chosen made her more cautious, with “less wild-firing” than with previous songs she had chosen. She likewise found the uplifting nature of the music helped to motivate her to “push on and play better.”
It is clear, then, that music can not only change the meanings of the game for the player and alter the level of perceived immersion, but it can actually impact the playing ability and tactics of the player. This is particularly interesting as players can consciously select songs that they know will raise or lower their heart rate to alter not only their experience of the game, but also alter their perceived skill in the game. An interesting future study would test whether the music chosen actually does improve the gameplay of the participants, or whether this is just a perception based on the increased involvement with the game (i.e. immersion in the game) created by the multimodal interactions between music and imagery.
Conclusions
We have presented a pilot study to examine the role of the customization of musical soundtracks in video games. As this functionality becomes more commonplace, further understandings of how music functions in media will be important in order to determine the effects that this has on the experience of the games. In particular, more quantitative methods or a larger group of participants will be necessary in order to determine differences amongst genders, amongst participants with different listening habits, amongst experienced and inexperienced gamers, and amongst a variety of game genres. It is likely, for instance, that the role of music in a match-three puzzle game is less important than in the game studied here.There were many important theoretical findings that came out of the study that will warrant further research and exploration with a larger group of participants. An important (if unsurprising) finding of the study is that altering the music in a game changes meanings, actions, effects, and emotional response to the game. By varying the songs or altering the order of the songs, players not only experienced different immersive and emotional states, but also considerably changed how they play the game. New meanings were created through juxtaposition and counterpoint of music and game. Songs that were chosen at times took on an ironic level juxtaposed with the violence of the game, meaning that the music could be used to heighten awareness and alter meanings. In other words, the overall semiotic meaning(s) of the game can change considerably from what the game’s designers intended.
The fundamental nature of music as a part of games implies that incorporating or accounting for significant player customizability of music is a factor to consider in the design process. Game designers who wish to deliver a more cinematic experience to the player may wish to retain more control over or disable music customization features. Game composers may wish to inextricably tie their music to the visual and haptic gameplay elements in order to create a singular gaming experience that cannot be replicated with customized music. Alternatively, there could be a shift away from a cinematic or narratological focus of video game music that strictly maps composed music with in-game settings, events or scenes. Instead there could be a refocusing on gameplay elements that allow for greater player control over the music heard in-game, as was seen with the Grand Theft Auto example discussed earlier. Games could also be created with a recommended playlist that could be interchanged and manipulated by the player: in essence, the player would be able to customize the game-space within a limited window. It may also be possible to improve the customization experience by using audio “intelligence”, such as beat-mapping or using music file meta-data to match fast-paced music to high-action scenes in the game. Little Big Planet 2 (Media Molecule 2010) for instance includes a music sequencer that lets players arrange a variety of instrument samples into songs, and then share these with other players online. More interesting, however, is the game’s ability to then synchronize the player’s created music to in-game events (Orland 2009). In this sense, Little Big Planet 2 recognizes the importance of the integration of the music with the game content, and provides a (albeit limited) means of customizing music while still maintaining some control over the game.
The music chosen in the game had a variety of impacts on the players’ emotional state and their immersion, but it is worth distinguishing here that some players found themselves immersed in the game, or the music based on their musical selection sometimes at different times. In these instances, players disconnected the musical experience from the game experience and enjoyed the game as different kinds of experiences: sometimes the music influenced the game, and sometimes the game influenced the music. Moreover, while studies have shown clearly that music can influence affective state, anxiety and emotion, this was, to our knowledge, the first study to show how music can influence the player’s actual actions. This is an important finding that requires significant further research, as the pacing of songs in particular had an important effect on how players played the game.
One interesting consideration here is that the Xbox360 console designer (Microsoft), in building in the ability to customize music, has implied that game music is peripheral to the game. The players in the study, however, consciously or subconsciously attempted to make connections between the music that they chose and the game’s narrative, events, imagery and playing tactics. Players found coincidences between elements of the music and actions on-screen, and chose music that they felt would increase their enjoyment of the game. Those players that were experienced with interactive game music scoring were particularly disappointed that the music that they selected (as well as the actual game music) was not more closely tied to the game action. Indeed, when music was incongruent, players reported less ability to play, less ability to concentrate, and less immersion in the game. In other words, music is not a peripheral part of games, but is, rather, an integral part of the overall experience.
Although perhaps an obvious point, it is worth highlighting the fact that players want to customize music in games - as is evidenced by the growing music modding community and the fact that Microsoft built this feature into its console. But more than merely wanting choice over the music, players want to choose music that will help create meanings, produce a dynamic range of emotion, reduce anxiety, increase immersion and aid in their gameplay experience. However, what was interesting in our findings was that while players could predict the effect of selected music on perceived anxiety, and to some extent predict what music would impact their tactical play, players could not predict which music would aid in their perceived immersion in the game. It is possible that, over time, players will gain experience in choosing songs and will become more adept at making these predictions. Just as a composer learns to compose music for specific affect, players may become music supervisors of a sort in selecting songs.5 We may wish, therefore, to consider the notion of musical customization itself as a form of co-creativity, because with a selection as broad as the entire musical repertoire available in digital format, the customization of music is a skill, in that the music is being consciously chosen for a desired effect. We may also wish to consider customization itself as a form of interactivity in games. With the sharing of customized musical content through VoIP and social network gaming, the selection of musical content is, in a sense, a form of interaction between both the game and the player, and between players. The sense of fun and pleasure in the serendipitous moments created by chance suggest that customization can even become a form of play.
Endnotes
1 http://www.habbo.com/groups/Trax-Machine
2 Such as for Soldat (Independent, 2002), http://wiki.soldat.nl/Modding_sfx
3 All names have been changed to protect anonymity.
4 For a useful summary of audio-visual interaction, see Kohlrausch and Van de Par, 1999.
5 Music supervisors in film and television (and now sometimes in games) are responsible for selecting and licensing music in the media. While players here do not have to license the music that they choose, the role of selection is clearly an important one.
Reference
Allen, C. (n.d.).Personalization vs. Customization. Strategic Marketing Planning for Growing Companies. Retreived January 21, 2011, from http://www.allen.com/cgi-bin/gt/tpl.h,content=26
Ben-Shaul, N. (2003). Split Attention Problems in Interactive Moving Audiovisual Texts. MelbourneDAC the 5th International Digital Arts and Culture Conference. Retrieved December 8, 2010, from http://hypertext.rmit.edu.au/dac/papers
Bethesda Softworks (2008). Fallout 3: Operation Anchorage. [Xbox360] USA: ZeniMax.
BioWare (2007). Mass Effect. [Xbox360] USA: Electronic Arts.
BioWare (2009). Dragon Age: Origins. [Xbox360] USA: Electronic Arts.
Bridgett, R. (2010). From the Shadows of Film Sound: Cinematic Production & Creative Process in Video Game Audio. Collected Publications 2000-2010. Self-published, available from http://www.sounddesign.org.uk
Brown, E. & Cairns, P. (2004). A grounded investigation of game immersion. CHI 2004, 24-29 April. Vienna, Austria.
Bullerjahn, C. (1994). An Empirical Investigation of Effects of Film Music Using Qualitative Content Analysis. Psychomusicology, 13, 99-118.
Chion, M. (1994). AudioVision. New York: Columbia University Press.
Collins, K. (2007). An Introduction to the Participatory and Non-Linear Aspects of Video Games Audio. In S. Hawkins & J. Richardson (Eds.). Essays on Sound and Vision. Helsinki: Helsinki University Press, 263-298.
Collins, K. (2008). Game Sound: An Introduction to the History, Theory and Practice of Video Game Music and Sound Design. Cambridge, Mass.: The MIT Press.
Hébert, S., Béland, R. Dionne-Fournelle, O., Crête, M. &Lupien, S.J. (2005). Physiological stress response to video-game playing: the contribution of built-in music. Life Sciences, 76, 2371-2380.
Hung, K. (2001). Framing Meaning Perceptions with Music: The Case of Teaser Ads. Journal of Advertising, 30/ 3, 39-49.
Infinity Ward (2007). Call of Duty 4: Modern Warfare. [Xbox360] USA: Activision.
Insomniac (2008). Resistance 2 [Xbox360] USA: Sony.
Jørgensen, K. (2008). Left in the dark: playing computer games with the sound turned off. In K. Collins (ed.) From Pac-Man to Pop Music: Interactive Audio in Games and New Media. Aldershot: Ashgate, 163-176.
Karageorghis, C.I., Jones, L., Low, D.C. (2006). Relationship Between Exercise Heart Rate and Music Tempo Preference. Research Quarterly for Exercise and Sport, 77/2, pp. 240-250.
Kellaris, J.J. & Kent, R.J. (1992). The Influence of Music on Consumers’ Temporal Perceptions: Does Time Fly When You're Having Fun? Journal of Consumer Psychology, 1/4. 365-376.
Kohlrausch, A., Van de Par, S. (1993). Auditory-Visual Interaction: From Fundamental Research in Cognitive Psychology to (possible) Applications. IS&T/SPIE Conference on Human Vision and Electronic Imaging IV, San Jose, CA. 34-44.
Lipscomb , S.D. & Zehnder, S.M. (2005). Immersion in the virtual environment: The effect of a musical score on the video gaming experience. Journal of Physiological Anthropology and Applied Human Science, 23, 89-95.
Marcinkowski, M. Soldat (2002). [ PC] Independent.
Maxis (2000). The Sims. [PC] USA: Electronic Arts.
Maxis (2008) Spore. [PC] USA: Electronic Arts.
McLuhan, M and Nevitt, B. (1972). Take Today: The Executive as Drop Out. New York: Harcourt.
McWhertor, M. 2008. Resistance 2 Takes 420 Hours to Complete. Kotaku, Retrieved January 16, 2011, from http://kotaku.com/5072781/resistance-2-takes-420-hours-to-complete
Media Molecule (2010). Little Big Planet 2 [Sony Playstation 3] USA: Sony.
Miller, K., Brazeau, C., Fuiten, M. & Woefel, J. (2008). Effects of Music and Choice Listening on Arousal Changes. OshkoshScholar, V. III, pp. 73-81.
Mojang(2009). Minecraft [ PC].USA: Mojang.
Morris, S. (2003). WADs, Bots and Mods: Multiplayer FPS Games as Co-creative Media. Level Up Conference Proceedings. Utrecht: University of Utrecht, November.
Nacke, L.E., Grimshaw, M.N. & Lindley, C.A. (2010). More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game. Interacting with Computers, 22/4, 336-343.
NC Soft (2003). Lineage II [PC] USA: NCSoft.
Neversoft (2008). Guitar Hero World Tour [Xbox360] USA: Activision.
Orland, K. (2010). Media Molecule Reveals Little Big Planet 2’s More-than-Just-Music-Sequencer. Joystiq, Retreived January 18, 2011 from http://www.joystiq.com/2010/08/19/media-molecule-reveals-littlebigplanet-2s-more-than-just-music
Pitzen, L. J., & Rauscher, F. H. (1998). Choosing music, not style of music, reduces stress and improves task performance. Poster presented at the American Psychological Society, Washington, D.C.
Polyphony Digital (1997). Gran Turismo [Playstation] USA: Sony
Rockstar (1997). Grand Theft Auto [Playstation] USA: Rockstar Games.
SCEI (1999). Vib Ribbon. [Playstation] USA, Sony.
Sihvonen, T. (2009). Players Unleashed! Modding The Sims and the Culture of Gaming. PhD. Diss., Turun Ylikopisto, Turku University.
Smith, C. A., & Morris, L. W. (1976). Effects of stimulative and sedative music on cognitive and emotional components of anxiety. Psychological Reports, 38, 1187-1193.
Sotamaa, O. (2007). On modder labour, commodification of play, and mod competitions. First Monday, 12, 9 /3.
Spenwyn, J. & Barrett, D. J. K. & Griffiths, M.D. (2010). The Role of Light and Music in Gambling Behaviour: An Empirical Pilot Study. International Journal of Mental Health and Addiction, 8,107-118.
Stormfront (2002) The Lord of the Rings: The Two Towers [Playstation]. USA: Electronic Arts.
Storms, R.L. (1998). Audio-Visual Cross-Modal Perceptual Phenomena. Ph.D. Diss., Naval Postgraduate School, Monterey, California.
Sulake (2000). Habbo Hotel [Online]. Finland: Sulake.
Tagg, P. (2000). Kojak: Fifty Seconds of Television Music. New York: Mass Media Music Scholars’ Press.
Thayer, J. F. (1983). Effects of Music on Psychophysiological Responses to a Stressful Film, Psychomusicology, 3/1, 44-52.
Turbine (2007). The Lord of the Rings Online: Shadows of Angmar.
Wachob, C. (n.d.). What are personalization and customization? Retrieved January 22, 2011, from https://web.archive.org/web/20110618104958/http://www.workz.com/content/view_content.html?section_id=482&content_id=5200
Discography
Brown, B. (2003). Temple of the Moon. On Lineage II: The Chaotic Chronicle, (NC Soft).
Earshot (2004). Wait. On Wait (single). (USA: Warner Bros. Records Inc.).
Faunts (2007). M4, Pt. II. On M4. (New York: Friendly Fire Recordings).
Grieg, E. (1876/1964). In the Hall of the Mountain King. Peer Gynt Suites Nos 1&2 Piano Concerto, performed by the Berlin Philharmonic. (Berlin: Deutsche Grammophon).
Herb Alpert and the Tijuana Brass (1965/2005).Tijuana Taxi. On Going Places. (USA: A&M).
Holst, G (2002). Mars: Bringer of War. On The Planets, performed by the London Philharmonic Orchestra. (London: EMI Records Ltd.)
Jules, G. (2002). Mad World. On Trading Snakeoil for Wolftickets. (London: Sanctuary).
Liquid (2008). Sweet Harmony. On The First Chapters. (London: XL).
Mayer, J. (2006). Heart of Life. On Continuum. (Chicago: Aware Records).
McKennitt, L. (1994). The Mystic’s Dream. On The Mask and the Mirror (Ontario, Canada: Quinlan Road, 1994).
Red Snapper (1998). The Sleepless (feat. MC Det). On Making Bones. (UK: Warp Records).
The Lonely Island (2009). I’m On A Boat. On Incredibad. (USA: Universal Republic Records).
Vangelis (1992). Conquest of Paradise. On 1492: Conquest of Paradise: Music from the Original Soundtrack. (USA: Warner Music Ltd.).