Sound in a Participatory Culture
by Kristine JørgensenWhen I started researching game sound in 2004, there was no previous comprehensive research on the topic, and existing knowledge had to be adapted from other fields. Now, ten years later, there is a growing body of literature on game sound, and Associate Professor and Canada Research Chair in Interactive Audio at the University of Waterloo Karen Collins’ new book Playing with Sound (MIT Press 2013) is a sophisticated example of theory development on the subject which goes beyond sound design and the interpretation of what sound means in the gameplay context. The novelty of Collins’ recent book is that it analyzes how game sound and music are parts of a greater participatory culture that goes beyond the play situation itself, and includes aspects such as sound modification, in-game creation of music, and the application of game music to new contexts.
In exploring the many ways in which players interact with game sound, the main argument of the book is that interacting with sound is radically different from listening to sound because of our expectation that sound is always embodied; that is, connected to a source. In addition to an introductory and a concluding chapter, the argument is structured around five chapters. In chapter 1, Collins outlines a theory of interactive sound based in neuropsychology and sound cognition research. While our brains have a tendency to merge sounds with physical or visual stimuli, Collins shows how interactive sounds merge, not with image like in films and television, but to actions. She argues that there is an apparent need for listeners to materialize sound, or connect it to a source, either visually or in terms of embodiment. Something creates the sound, and in games interactivity suggests that what creates the sound may often be the player. Thus, Collins reflects my own viewpoint that in games, gameplay is the primary means of communication, and that the visual and auditory channels work as equals in supporting this. While sound may expand the players’ understanding of the game space, Collins also argues that sound works as an intermediary that decreases the spatial proximity between the player and the virtual space. While games’ visual aspects cannot be interacted with directly save through game controllers, the game sound is not delimited to the virtual space but penetrates the physical space, and makes game sound inseparable from other sound produced in actual space. Game sound is for this reason just as real as actual sound, and the player may in this sense respond to it in the same way as s/he would respond to sound created by actual events in physical space. This is expanded through the player’s identification with the avatar through extending the body schema. Collins is arguing heavily for “embodied listening”; the theory that listening is so closely aligned with our bodily functions that it must be understood as an extension of ourselves.
In chapter 2, Collins discusses how game sound works to create identification and immersion, and argues that while visual information in games always remains contained in virtual space, game sound transcends the border of the virtual space and moves into physical space. It is argued that sound in games extends "our sense of self beyond our physical body and into the intermediary space between ourselves and the virtual world” (Collins 2013, 42), and that this contributes to creating a sense of identification between the player and the game character s/he controls, and immersion into the game. Chapter 3 goes further into the role of sound in connection with immersion and identification and shows how interacting with game sound allows players to experience sound and music in new ways. Three cases are used to support her argument; interacting with sound in music games, the role of avatar’s voice for a sense of immersion, and how Alternate Reality Games may invite new ways of interacting with music. In chapter 4, Collins looks at the role of game sound and music in the context of participatory culture and player-generated content. With the idea that games are performative as her point of departure, she analyzes game sound as a tool for performance and spectatorship. She uses examples such as machinima, shared music in online game situations, virtual live performances, and the use of game technology to create music. As performative game activities centered on sound, these are all associated with the need to materialize sound and make it embodied through interaction. With a continued focus on participatory culture, chapter 5 discusses game modification and game sound personalization and customization as forms of interactivity, and what effect these have on the player’s experience.
According to Collins, “we lack the terminology and methodologies to study and discuss the players’ relationship to sound” (Collins 2013, 2). While there is indeed a shortage of terminology and methodology for understanding player experiences with sound - in particular with regards to the social context outside of the game itself - this claim ignores player-centered research on game sound such as that carried out by Grimshaw (2008, 2012), Nacke & Grimshaw (2010), Nacke et al (2010), Roux-Girard (2010), Tinwell et al (2010), and myself (Jørgensen 2009a). My own perspective is that in order to understand this relationship, we must understand how the sound works with respect to gameplay, and that this is most fruitfully understood by exploring player experiences with sound in a gameplay context. However, in order to create tools that can help in the pursuit of understanding players’ relationship to sound in a wider context, Collins applies scientific knowledge on sound perception and cognition to a game context, and existing research on games and participatory culture combined with an analysis of players’ own descriptions of their experiences with game sound on web forums. Her theoretical work is sound, and the application of cognitive psychology and neurology does indeed promise some avenues for future studies that will considerably expand existing research on player responses to sound. The book is particularly valuable, however, when the author goes into an analysis of players’ own descriptions of their experiences with game sound on web forums. The player perspectives used in discussing the player’s interpretation of avatar voices are highly interesting and provide a qualitative account of how actual players experience sound, and reflect the findings of my own research on player experiences with sound (Jørgensen 2009a); in particular the use of avatar voices as a point of identification and information provider (Jørgensen 2009b; 2013, 127, 130-131).
While Playing with Sound generally provides a good read, sometimes the focus on sound tends to disappear in longer discussions about participatory culture and how players interact with games. These discussions are needed in order to provide a background for her argument, but this reviewer was often left asking where the author is going with lengthy discussions of how the interactivity of games creates a new arena for player-generated content, personalization and customization. However, in each of these situations, the relevance for how players interact with and understand sound is unwrapped conclusively, demonstrating that participatory and interactive situations do indeed make us appreciate sound and music in different ways than before. One example is the Alternate Reality Games used by Nine Inch Nails in the promotion of their album Year Zero, which demanded that players interacted with musical pieces in a new and increasingly social way.
Another issue is that the target audience for the book is not always clear. While containing much terminology that only those with a certain familiarity with digital games would be comfortable with, the book sometimes appears to be aimed towards an audience without this knowledge. That said, the examples nevertheless remind the readers that modding and customization are not only about visual aspects or game mechanics, but also about sound. The book thus shows that game sound is also an important and not-to-be-ignored form of content in games that can be manipulated and interacted with in the same way as other features.
The book ends on a somewhat technological deterministic note, on which it is concluded that game sound has impact upon listening, that we may develop new ways of understanding and interpreting sound: “Video games may introduce new ways of listening to, creating and consuming music that go well beyond the game” (Collins 2013, 144). The fact that this happens in a social context driven by the social and creative needs of the players is not ignored, but it is described as an inherent property of modern digital and participatory media more than as connected to the sound itself. The idea that players like to play with sound is repeated throughout the book, although Collins might just as well have extended the thought and stated that “players like to play everything that can be played with”. Participatory culture, including game modification and interaction with sound, is a symptom of something more than what the computer medium can do - it is a symptom of the fact that human beings are basically social and creative beings.
After reading the book, the central idea that I am left with is that sound extends the virtual space into real space and helps immersion by extending the interactive space of the virtual world into the physical space. Another idea which has even stronger impact when it comes to understanding sound in games in terms of other media, is that game music opens up for a new way of listening to music in a similar way to how the music video did: listening to music as one plays a game is a new venue for music listening now, one that may familiarize the player with music s/he did not already know, and to re- contextualize music one already knows.
Researchers and students who want literature on the role of sound in the gameplay context and game audio design may wish to consult Collins’ previous book Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design (MIT Press 2008). For those interested in the broader perspective that takes participatory culture into consideration, and how game sound and music are given meaning outside of the game, or how the social world affects the meaning of game sound and music and frames this within the perspective of cognitive psychology and neurology, Playing with Sound is a good choice.
References:
Collins, Karen (2008): Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. Cambridge, Mass.: MIT Press
Collins, Karen (2013): Playing with Sound. A Theory of Interaction with Sound and Music in Video Games. Cambridge, Mass.: MIT Press.
Grimshaw, Mark (2008). The Acoustic Ecology of the First-Person Shooter: The Player Experience of Sound in the First-Person Shooter Computer Game. VDM Verlag.
Grimshaw, Mark (2012): “Sound and Player Immersion in Digital Games”, in Pinch, Trevor and Karin Bijsterveld (eds.): The Oxford Handbook of Sound Studies. Oxford University Press.
Nacke, Lennart, Mark Grimshaw and Craig A. Lindley (2010): “More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game”, in Interacting with Computers 22, 5, pp. 336.343.
Jørgensen, Kristine (2009a): A Comprehensive Study of Sound in Computer Games: How Audio Affects Player Action. Mellen Press.
Jørgensen, Kristine (2009b): ””I’m overburdened!” An Empirical Study of the Player, the Avatar, and the Gameworld", in Proceedings from DiGRA 2009: Breaking New Ground: Innovation in Games, Play, Practice and Theory. Available: http://www.digra.org/dl/db/09287.20429.pdf
Jørgensen, Kristine (2013): Gameworld Interfaces. Cambridge, Mass.: MIT Press.
Nacke, Lennart and Mark Grimshaw: “Player-game interaction through affective sound”, in Grimshaw, Mark (ed.): Game sound technology and player interaction: concepts and developments. IGN Global.
Roux-Girard, Guillaume (2010): “Listening to Fear: A Study of Sound in Horror Computer Games”, in Grimshaw, Mark (ed.): Game sound technology and player interaction: concepts and developments. IGI Global.
Tinwell, Angela, Mark Grimshaw and Andrew Williams (2010): “Uncanny Speech”, in Grimshaw, Mark (ed.): Game sound technology and player interaction: concepts and developments. IGI Global.