Game Studies

Through the Ludic Glass: Making Sense of Video Games as Algorithmic Spectacles

Abstract

Video game analyses have historically focused on the human act of play or on the events resulting from the player’s act. Until recently, spectating has remained an analytical domain of film theory and visual arts. In game studies, this perspective has changed, with the arrival of the phenomenon of gameplay spectating and game streaming on a mass scale, and its leakage into academic as well as popular consciousness. How does the spectacle change the analytical perspective towards video games as objects of scholarly analysis and video gaming as reflective practice? In this paper, I will approach the video game as an algorithmic spectacle and propose an analytical perspective to study this phenomenon, reaching out to theories of moving (digital) image proposed by the philosopher Vilém Flusser and the filmmaker Harun Farocki.

Keywords: videogame aesthetic, image theory, media aesthetic, spectacle, watching, operationality, Vilém Flusser, Harun Farocki

This contradiction between looking and observing, between “superficial reading” and “close reading”, raises the familiar issue of the distance between the observer and the observed.

(Flusser, 1985/2011, p.33)

Introduction: Through the Ludic Glass [1]

At first glimpse, spectating seems an indirect and distant practice compared to the allegedly direct and close experience of gaming. Perhaps this is why it has not drawn a lot academic attention in the early years of game studies. Watching used to be the domain of non-interactive visual media, such as film and television. Within the context of video games, spectating was perceived as a marker of one’s passivity rather than active doing. Only that which was physically configurable by players had been seen as an emblematic characteristic of new media and in consequence video games. This ability to physically intervene into the allegedly unconfigurable and linear process of displaying images laid ground for dominant analytical perspectives, understanding video gaming as a medial practice facilitating human action, choice and physical effort.

More recently however, livestreaming and professional e-sport have brought spectating, and together with it the digital image, back into the spotlight. Every month, hundreds of millions of people worldwide spend billions of collective hours watching games over the Twitch platform. Such numbers speak to imagination. However, what puzzles me much more than the ubiquity of this mass entertainment format is the algorithmic and operational essence of the digital spectacle. I want to explore what it means to watch, look at or otherwise observe computed images within the context of video games. Does watching a video of gameplay require other visual literacy than film watching, art contemplation or nature spectating? In other words, how do we media aesthetically experience a digital ludic spectacle? Further, more crucially within the context of this special issue and the Game Analysis Perspectives conference, how do we analyze what we look at? In my exploration of the spectacle then, I am not that much concerned with observing other human players at play, but with the question of the configurability and operationality of the displayed video game image.

I want to portray video games as algorithmic spectacles characterized by images that are “…functions in the mathematical realm” (Parikka, 2012). In an algorithmic spectacle, light, the basic “substance” of optics is replaced by the calculus (Parikka, 2012). This foundational transformation of the image provokes questions much deeper than those of purely technical nature. Although high-fidelity graphics and ever more impressive photorealism inscribe themselves in popular consciousness of gamers, it is the media historical and cultural dimension of the technical image that is truly spectacular, for there is more to it than meets the eye. Games are programmed systems. Their images displayed on-screen are effects of algorithmic operations and at the same time, an important part of the game’s operationality. Often, in order to understand the ludic spectacle, the audience needs to know how to interpret the ever-changing visual “muddle” (think of fast-paced multiplayer online battle arenas, also known as MOBAs). An algorithmic spectacle requires from its audience the ability not only to observe images as representations of something but above all to decode imaginary systems in action. Video game images emerge at the intersection of what Aubrey Anable refers to as the “representational/computational fold” (Anable, 2018, p. 50). Spectated play then spans between the configurable image and its displayed configuration.

In this paper I will lean on two entry points to explore the algorithmic spectacle of computed play. The first one starts with the foundational theory of technical image by the philosopher Vilém Flusser while the second one takes on board the operational image by the filmmaker Harun Farocki. Both concepts rethink the image vis-à-vis its traditional representational character. Technical and operational images (also known under the term operative) no longer represent or signify objects found in reality. They construct reality. They do not depict but visualize, model and simulate. In order to prepare some ground for the question of technicity and operationality of the video game image, I will start by providing some rationale behind the choice of the image as an analytical category. I will then move on to addressing the problem of the representation and computation binary, without which the problematic place of the image in the study of video games cannot be entirely grasped.

Why Algorithmic Images Now?

Before delving deeper into the analysis of video game images as algorithmic spectacles, I would like to briefly address the reason of bringing the image back into the conversation. Although, in game studies, many of us have pointed to the importance of that which underlies the visual layer, it is the image that comes as the first association to our mind when we talk and write about particular video games. Even if rules, mechanics or code tend to be treated as core analytical units, they exist in a dialectical relation to the “surface,” often depicted in terms of fiction or representation. The image then, although downplayed, remains a powerful category and I want to show its complexity without relegating it to the visual veneer. In a way, this text is an attempt to go back to the proverbial drawing board. My aim then is not to offer a concrete framework to analyze video games with, but to take a closer look at an ontological status of the digital image to show how problematic it is in the first place to juxtapose rules, mechanics or code against representational elements.

At this point, I also want to stress that my aim is not to interpret games solely through the lens of visual theory. I am not a visual theorist. I belong to the earlier generation of game studies scholars who left their mother disciplines (in my case English Studies) and worked their way through an uncharted terrain, taking from many fields and contributing to building up a more standalone repertoire of game-related research. Nevertheless, I think it is crucial today to look again into foundational categories, image being one of those. Theories of Flusser and Farocki provide for very insightful if not indispensable starting points. Without rethinking the image, game studies will keep falling into the Cartesian trap of differentiation between the visual layer -- that which is represented -- and the rules or mechanics -- that which is computed (I will delve deeper into this dualism in the next section). This recurring analytical fallacy has motivated me to look into the image and to show how both of the above categories fold into it.

Some readers may get the impression that the terms algorithmic spectacle or an operational image are not needed at all. After all, we have long had the term simulation. I believe this argumentation, however convincing in the first gist, does not hold entirely true. If we quickly consider the foundations of the term simulation, we soon end up with that of imitation. Etymologically, imitation (from Latin imitātus) points to a copy and is conceptually connected to the image (from Latin imāgō). In a pre-modern sense, as historian of technology Jessica Riskin notices, simulation is based on imitation and illusion, which implicates some sort of trickery or fakery (Riskin, 2003). It is only with the advent of electronic computing that simulation leaves the confines of representation. As Sherry Turkle argues, the move from the modernist culture of computation to a postmodernist culture of simulation rests on two very different aesthetics of the computer: the first one is founded on linearity and logics whereas the second one embraces complexity and decentring. Intelligence is no longer programmed into computers but instead it is supposed to emerge from a set of interactions. In postmodern terms then, simulation refers to adaptive emergent behaviours of the system (Turkle, 1996).

Espen Aarseth, in one of his earlier contributions, makes a case for simulation in the study of games, starting with an example of a video game fictional character -- an EverQuest dragon -- which apart from being a sign is above all a dynamic model. Models and simulations may be experienced, not only read about or watched, writes Aarseth (2007). This is what differentiates them from fiction. This argument is quite convincing but it seems to fit onto interactable digital objects with assigned functions and attributes much more than, for instance, objects or images that do not seem at first to have any gameplay relevant functionality. A plethora of digital images end up time and again in the corner of decorative visual items responsible for creating game scenery, although they too are digital mathematical objects rather than optical representations.

Despite being analytically equipped with simulation, game studies tend to nevertheless keep falling back on traditional aesthetic categories, juxtaposing representation against computation. My hope is that a closer look at the digital image can lead to a perspective shift.

Computed Representations / Represented Computations

Each medium relies on a tension of some sorts: radio on the tension between the sound signal and silence, film on the tension between the visible and the invisible, literature on the tension between the said and the unspoken (or antagonist and protagonist resulting in a dramatic tension). A video game as the most multifaceted medium of all or a “total cave” to expand on Alain Badiou’s vision of cinema (Badiou, 2015), carries multiple tensions within; those between the non-acted (non-interactive) and the acted (interactive), the observed and the performed, or that performed by the human player and that by the machine or algorithm. But above all, it is an aesthetic form emerging out of the tension between representation and computation; between that which is displayed on the screen and that which is computed behind it. The infamous “Kill Screen” glitch in Pac-Man makes up for a particularly fitting theoretical material. It demonstrates how representation and computation fold into one another. They are not two separate and exclusive dimensions. Together they constitute a digital image or what Frieder Nake calls a “twofold image” (Ger. das doppelte Bild):

A work generated at least in part, if not entirely, by algorithms (programs), exists in an interesting mode of duplicity. It is, at the same time, visible (or other-wise sensually perceptible) and computable. (Nake 2016)

In other words, while the human eye is scratching the surface (Ger. Oberfläche), the digital machine keeps computing the subface (Ger. Unterfläche).

Figure 1. A “Kill Screen” bug in Pac-Man (Namco 1980).

The twofold image of Pac-Man brings back foundational questions regarding the “nature” of the video game. A lot of debates (also those that allegedly have never taken place) (Frasca 2003) and early concerns of game studies revolved around the issues of representation and computation. Theoretical perspectives looking at video games as digital rule-based processes have often pointed towards narrative and visuality as merely supporting roles. The rules, mechanics and computational structures have become the “subface” of gaming. It is the gameplay that is the core of gameness, not the graphics or the story, writes Frans Mäyrä (2018). Espen Aarseth, on the other hand, theorizes the video game as a double-layered object, which consists of mechanics (code, rules, physics) on the one hand and semiotics (text, image, sound) on the other (Aarseth, 2011). As Aubrey Anable notices, “computation/representation has become the structuring binary for game studies” (Anable, 2018, p. 50). After closer examination, however, this representation-computation juxtaposition in game studies seems quite porous from a theoretical standpoint. After all, the surface dimensions of video games are subject to the same computational processes and infrastructures as the core. To classify video game images as representations and juxtapose them against the mechanics, is to turn a blind eye to the digital logic of the image. Hence my attempt to rethink the above juxtapositions, which have become foundational to game studies. After more than twenty years of research dedicated specifically to video games, we no longer need to fear “colonization” on the part of other disciplines (Eskelinen 2001). By now, game studies stand firmly on the interdisciplinary ground and can (or even should) reinspect the old elephants still marking their presence in the room. I see the question of representation/computation as one of the most settled proverbial elephants of our field.

As we will find out in the next section, the image abandoned the realm of representation many decades ago, together with the synthetization and discretization of information. Digital images are no longer representing or signifying and hence cannot be placed vis-à-vis game mechanics or rulesets. Video games are played and displayed. Their operations are often displayed visually and their images are operational in nature. The misconception that they still belong to representational media rests on the erroneous conflation of the image with its sensorial reception on the part of the human player/viewer. The image is digital and discrete. The reception of it must remain analogue and continuous (Nake, 2006). The human observer is able to visually judge only the continuous aspect, so the discrete essence of the image needs to be displayed as if it were no different to the traditional analog image. We can find a similar perspective in the work of Lev Manovich who analyzes a digital image on the one hand as a representation that belongs to human culture and on the other as a computer file that belongs to “computer’s own cosmogony” (Manovich, 2001, pp. 45-46).

The representation/computation fold consolidates the old Cartesian body/mind split disguised under the digital veneer. Perhaps this is why Friedrich Kittler in his famous essay “There is no Software” rejects the distinction between hardware and software altogether. In the end, everything boils down to electric tension:

All code operations, despite such metaphoric faculties as call or return, come down to absolutely local string manipulations, that is, I am afraid, to signifiers of voltage differences. (Kittler, 1997, pp. 147-155)

What may seem like a paradoxical proposition or a rhetorical exercise is rather a materially-grounded philosophy of media. It shows how material processes determine the medial situation or how firmly software is grounded in “hard” matter. And this is what I would like to point towards in this piece, laying bare the video game’s visual and operational layers and bringing them back together, for they cannot and should not be seen as separate units. The computational “cut” performed by many of us, game scholars, places a default borderline where it does not really exist. I do not see it as necessarily productive to think of the system, the rules and the procedures as the core and the instantiation of those rules, their representation and reception as the periphery. The circular trope from the center to the outside marginalizes some aspects of the ludic apparatus while privileging others. I would much rather think in terms of interferences (Dippel and Fizek, 2018), tensions and multiplicities (Jayemanne 2017) to acknowledge the multi-layered dimension of the video game. That is why in my exploration of video games as spectacles, I want to show how intertwined the displayed sequence of images is with the discrete operationality of the system. In other words, how the representational is computed and the computational represented.

Technical Images

Images influence our perceptions, values and experiences. Their transformation carries with it not only a purely technical change, but one that determines our very being-in-the-world. For Vilém Flusser the departure from representation towards computation marks a cultural revolution. In his monograph Into the Universe of Technical Images, first published in 1985, Flusser develops a media philosophy of the technical image. He begins his tour de force by juxtaposing the technical image with the prehistoric traditional image, deriving from early two-dimensional cave paintings. Flusser uses the term traditional to express the contextuality and placement of images within a longer tradition, which makes them decipherable in the first place (Flusser, 1985/2011, p.12). Technical images, on the other hand, are characteristic of contemporary synthetic, electronic and digital media, such as photography, television, cinema and computer imagery. While in the case of traditional images the meaning is inscribed into the surface, in technical images it is constructed from particles and pixels. Traditional images are observed and imagined. Technical images are calculated and computed (Flusser, 1985/2011, p.7). Their essence is a computed universe of particles, which are then assembled into visible images. Flusser differentiates between different types of the technical image: there are the chemical and the electronic images. The chemical ones can be subdivided into the silent and still as well as sounding and moving. The first category describes photography while the second points to film. Computer images belong to the electronic category. So do video game images, originally not mentioned by Flusser.

The computational nature of technical images questions their very ontological position as images in the first place. Inaccessible to human senses, they can no longer depict. What we are able to see are the visualizations of computational processes. As human observers, we must decode technical images as continuous representations; otherwise, they would remain entirely inaccessible to our aesthetic judgement. In other words, “as object of computability, the image must be digital; as object of perceptibility, the image must be analogue” (Nake, 2016).

Flusser does not develop his argument to demonstrate the shortcomings of the human observer equipped with their physical properties of sight not able to see behind the veneer -- quite on the contrary. It is the human observer who has the capacity to turn technical images back to images in the first place. Technical images can only be called images in the strict sense of the term, when looked at superficially (that is on the surface), writes Flusser: “Technical images are images at all only if they are seen superficially. To be images they require that the viewer keep his distance” (1985/2011, p. 34). At the foundation of the technical image, then, lies the concept of distance between the observer and the observed. And although traditional images also require the act of distancing from the concrete experience (they are after all two-dimensional representations and interpretations of it), the kind of distancing involved in spectating computed and simulated imagery renders a completely different experience.

The question of distance culminates in Flusser’s reflections on the critical reception of the technical image. He sees it as necessary to create new criteria according to which a technical image analysis should take place. And these are fundamentally different to the ones known from the traditional realm. Since technical images are no longer representations of the outside world but approximations and models of reality, their “critical reception … demands a level of consciousness that corresponds to the one in which they are produced” (Flusser 1985/2011, p, 22). The meaning of a technical image then is literally encoded. In order to decode a technical image, as Flusser argues, we do not need to read what it shows but rather read how it has been programmed:

We must criticize technical images on the basis of their program. We must start not from the tip of the vector of meaning but from the bow from which the arrow was shot. Criticism of technical images requires an analysis of their trajectory and an analysis of the inten­tion behind it. And this intention lies in the link, the suture of the apparatus that produced them with the envisioners who produced them. (Flusser 1985/2011, p. 48)

Flusser made his observation three decades before the emergence of critical code studies. Whether this influence is direct or incidental is of lesser importance here. It only shows the prognostic quality of his work. It is also a sign of its time. To grasp digital media is to learn the language of operationality in order to be able to critically approach its digital logic. As Marc C. Marino, the author of Critical Code Studies, emphasizes, we must read code for more than what it does -- we must consider what it means (Marino, 2020). To understand digitality, it does not suffice to merely observe the effects displayed on the screen; we need to go beyond the operations of code as visible in its effects. Marino’s statement echoes back into Flusser’s pledge to study the apparatus behind the veneer of the technical image. I will come back to this point later.

Operational Images

Within the context of image theory, operationality is often reflected with and alongside the critical video installation trilogy (2001-2003) of Harun Farocki, especially its last part, the Eye/Machine III. Farocki’s preoccupation with the operational quality of the image goes beyond its aesthetic dimension. He develops the concept of operational image within a highly political and ethical military context. His operational images are concerned with regimes of control and perform specialized tasks, such as guiding remote-controlled missiles (as depicted in Eye/Machine I). This is how Aud Sissel Hoel, a media and visual culture theorist, introduces Farocki’s work:

The catalyzing event for the Eye/Machine trilogy was the outrage and sensation of the 1990-1991 Gulf War, where point-of-view footage from laser-guided bombs (popularly known as smart bombs) was widely broadcasted to TV audiences. The military deployment of eye machines prepared the way for a new type of warfare -- a “war at a distance” facilitated by a new kind of images that Farocki terms operative images (operative Bilder). (Sissel Hoel, 2018)

Farocki’s concept of the operational image complements and expands upon the technical image. Like Flusser, Farocki too sees a major shift in the role played by the digital image. It no longer represents reality but constructs it. Operative images “do not represent an object, but rather are part of an operation” (Farocki, 2004, p. 17). In other words, digital images are no longer approached as representations or signs but rather as instruments capable of acting. Commenting on Farocki’s work, Trevor Paglen writes, “Instead of simply representing things in the world, the machines and their images were starting to “do” things in the world” (Paglen, 2014). This active role of the image does not correspond to deception or delusion, but rather revolves around the idea that images have some kind of agency of their own (Sissel Hoel, 2020, pp. 287-301).

What differentiates Farocki’s operational images from other types of images is their implied audience. They are not intended for human sight. Operational images “… exceed the human scale” (Sissel Hoel, 2020) and are made “neither to entertain nor to inform” (Farocki, 2004, pp. 12-24). They are interfaces mediating between algorithmically generated processes and the human observer. The visual aspect of the operational image remains superfluous for the machine. All it needs to do is process numerical data.

The above interpretation of operationality has become a key reference point in ongoing media theoretical debates regarding the digital image. However, as we learn from Sissel Hoel’s in-depth analysis of operative images, the meaning behind and the use of the term “operation” has remained quite diverse. For instance, in “The Algorithmic Art Manifesto” Frieder Nake uses “operative” as a synonym of computability and algorithmicity:

we cannot see the digital. nor can we hear or smell or taste or touch it. the digital does not exist for human senses. we just cannot perceive it. the computable is the operative and dynamic aspect. computability thus is the primary aspect; digitality is only secondary. the computable is also called the algorithmic. (Nake, 2016, pp. 67-70)

For William Uricchio, operationality of the digital image is also directly connected to its algorithmic construction (Uricchio 2011, p. 26). The new ways of representing are tied to algorithmic intermediations between the subject and the viewed object. In his understanding, algorithms form a layer that separates the calculating subject from the calculated object. Lev Manovich on the other hand, sees operations as “technologically-based cultural practices,” which are not exclusively tied to computer software (Manovich 2001, p. 118).

Within the context of video games, perhaps the most recognizable theory of operationality is the work of Ian Bogost, specifically his concept and method of performing video game criticism known as “unit operations.”

… any medium -- poetic, literary, cinematic, computational -- can be read as a configurative system, an arrangement of discrete, interlocking units of expressive meaning. I call these general instances of procedural expression unit operations. (Bogost, 2006)

Bogost, similarly to Manovich, does not attribute operational logic to digital computational media exclusively. Under “operation,” he understands a process that performs transformations of input information. Operations include “decisions, transitions, and state changes.” They may be mathematical but they may also refer to such mundane processes as brewing tea or steering a car.

My own interpretation of operationality does not expand on Bogost’s “unit operations.” It is closely tied to the technical medium and builds upon Harun Farocki’s perspective, connecting operations with computational and algorithmic processes. Unlike Farocki’s military images, however, video games (even the self-playing ones or the ones played by others) are intended for human sight.

Watching through the Ludic-Glass

Art tends to make the materiality of the medium particularly striking. It has the capacity to bring its most fundamental aspects to light. That is why I would like to open this section with Emissaries (2015-2017), game art simulations about cognitive evolution. The trilogy starts with an ancient community living with a threat of a volcanic eruption, which could lead to its extinction (Emissary in the Squat of Gods). The second part (Emissary Forks At Perfection) depicts an AI-driven world thousands of years after the explosion has taken place. And finally, in the Emissary Sunsets the Self, the AI reaches its own peak attempting to shrug off the god-like agency by mutating in the hope of devising its own generative death. The visual and sonic layer of Emissaries is as surreal as the worlds they generate. Ian Cheng designed his simulated images to be watched rather than physically manipulated. Emissaries were exhibited in galleries and live streamed over the online platform Twitch. They comprise of “computer-generated simulations like those used in predictive technologies for complex scenarios such as climate change or elections,” we can read on MoMA’s website (2017). As algorithmic images, Emissaries are a perfect springboard to reflect the crossover between operationality and spectated video gameplay.

These three ways of looking at Emissaries very accurately depict different cultural techniques [2] employed in the process of looking at games. A video game may be looked at as a collection of images, their visual characteristics being the source of delight. Think of the practice of in-game photography, which involves performing screenshots of the in-game scenery and exhibiting those as digital images to be contemplated (Möring and Mutiis, 2019, pp. 69-94). Many games feature the so called “photo-modes” in order to encourage the players to marvel at and capture in-game scenes. Oftentimes, a video game is watched as a narrative, a sequence of images and events acted out, watched on the screen, remembered and retold as a consistent cause and effect chain. The aspect I find the most mesmerizing from a media theoretical standpoint, particularly with regards to games and simulations, is watching the changing behavior of the performing system. Audrey Anable analyses system observation within the context of another video game art -- Cory Arcangel’s Various Self-Playing Bowling Games (2011, Whitney Museum of American Art) a collection of six hacked bowling games, which would play on their own. She writes:

As viewers, we are completely outside the cybernetic loop, a position that violates the most basic (if exaggerated) principle of video games: that we can control them. Denied the possibility of playing the games, we contemplate the games as systems. (Anable, 2019, p. 107)

Unfortunately, Anable does not develop the concept of system observation beyond a broad statement that the audience instead of watching a looped montage, looks at the gaming system’s enactment of each game’s gutter ball algorithm.

The above two examples (Emissaries and Bowling Games) demonstrate how video game images are closely tied to the notions of functionality, usability and computing. In that sense, video game images often end up being technical instruments in the players’ hands rather than representations. For spectators (or “observers of systems” after Aubrey Anable), they do not only represent the in-game world but become visual instantiations of the game’s operationality. As I have already argued with Flusser and Farocki, the operationality of the digital image usually remains distant and undecipherable to the human eye -- it is buried deep beneath the digital veneer. We can only see approximations and representations of operationality projected onto the screen. In other words, we watch the representational surface of multiple software operations. This algorithmic spectacle displays characteristics of a post-perceptual image; one that “… has attained a certain autonomy from synthetic operations that necessarily involve human forms of perception and sensation” (Hansen, 2016). The digital quality of the image (pixels susceptible to purely machinic protocols) not only marks its break with the legacy of cinema and photography, but, more importantly, makes it impossible to be perceived and what follows, aesthetically judged by the human eye.

Layers of Spectated Meaning

The question still remains: how to perform an aesthetic analysis of spectated play beyond that which is accessible to our senses? At this point I would like to come back to the proposition made by Vilém Flusser to understand the image by examining its apparatus. To perform a Flusserian interpretation would require a systematic analysis of the way, in which the image has been constructed. One of the possibilities, as advocated by the emerging discipline of critical code studies, would be to interpret its underlying code. Conor Mckeown’s close reading of the code within the context of video games is a rare example of the method in question (Mckeown, 2018). Mckeown analyzes an image of a black square he programmed to be displayed on the screen and maneuvered around it by the player. He begins by asking what may seem like a trivial question: “How is the image displayed on the screen?” He then moves on to a rather technical specification of the square’s dimensions (64x64 pixels), the size of its file (239 bytes) and the way, in which the image is called out by the computer program: “image.src = “blacksquare.png”;”

Figure 3. An image of a black square, coded and analyzed by Conor Mckeown in his doctoral thesis Videogame ecologies: interaction, aesthetics, affect (2018). Click image to enlarge.

Figure 4. Selected code lines describing the properties and functions of Mckeown’s black square. Click image to enlarge.

Before the image can be displayed, the computer needs to “know” that there is an object “var bs,” which will be filled with graphical material. As Mckeown states, “only once the image, the digital object and the function are in place can we place the object on the screen” (Mckeown, 2018). For the graphical information to be displayed to the external user, a “renderer” must be specified. Out of Mckeown’s code-analysis we find out that it is not the “blacksquare.png” that is the image. Instead, he defines an image as an ongoing contextualized process. The operations behind the display of as simple an image as a black square demonstrate the complexity of the video game spectacle and the technical conditions of its meaning production.

In an attempt to capture the algorithmic spectacle, we could also follow Friedrich Kittler’s media archeological method of descending from higher to lower levels of observation (Kittler 1997, p. 150); from that, which is visible on the screen, through programs and their infrastructural logic down to the operating system, and low-level programming languages including the very machine code responsible for the operationality of the hardware itself. It is of course technically impossible, if at all necessary, to perform an image analysis cascading to the very electrical impulse. Kittler’s perspective, however, may inspire another medium-centered method of image analysis involving an examination of the drawing and modelling tools (such as Maya, Substance Painter and ZBrush, amongst many others) accompanied by a consideration of the infrastructure of the program, which enables the emergence of the image in the first place. Each program comes with certain design possibilities but also constraints. In that sense, it becomes an active agent in the process of image “envisioning” as theorized by Flusser. As we have learned from the black square example, images in games may be displayed as parts of objects equipped with certain functionalities (e.g., the possibility to move the black square around the screen). In more complex gameworlds, this relation between the image and the object could also be explored by analyzing how a particular image created in a selected visual art software becomes part of a functional game space once imported into the game engine. Once in the game engine, the image not only transforms into a new instantiation of itself, but can also gain a layer of operationality it did not have before. We could say that the image is turned into an instrument-image, which can be acted upon and which behaves in accordance with the simulated physics of the gameworld. To interpret and analyze spectated play, then, is to look at the conditions of its production (software), to look at the underlying code of that image or game scene and, finally, to look into how the image is placed within a gameplay context (how it corresponds with the physics of the engine, which parts of it attain the instrumental qualities of objects, etc.).

In many cases to spectate is also to look into the behavior of the system pictured on the screen in search of relevant pieces of information, which would then allow the player to interpret the gaming situation. Such interpretation of the visual “muddle” becomes a skill demanding a highly developed ludic literacy; especially in the E-sport matches of the MOBA genre (multiplayer online battle arenas), such as League of Legends (Riot Games, 2009). To a spectator unfamiliar with the visual conventions of the MOBA genre, the title and the rules of the game, the operational aspect of the image is difficult, if not impossible, to decipher. They may be able to look at the image as a depiction of the fictional world, recognizing its fictitious figures, foliage, elements of user interface. The position of the playing parties, the color coding during a battle, and the dynamically changing layout will not yield any useful information, which could then be used in order to interpret the outcome of the game. Within the context of esports, a deeper interpretation of the spectacle is oftentimes enhanced by data analytics relying on computer vision live tracking (Maymin, 2020). In such a case, the continuous spectacle playing out in front of the human player is translated back into a discrete set of operations interpretable by the computer. These can include live tracking of the player’s location multiple times every second, their performed attacks, avoided and caused damage and health levels amongst many other variables. In computer science, such methods are used in order to compute and interpret win probability models. I will not delve further into computer-enhanced forms and methods of spectating. However, I see those developing practices as important aspects of the algorithmic spectacle and an area opening new interpretational perspectives to media aesthetics.

Towards Aesthetics of Spectated Play

The spectacle within the context of video games turns out a highly complex topic, partially due to the diversity of games themselves and partially due to the variety of cultural techniques revolving around the practice of watching digital images. There must be a difference between watching a self-playing simulation like Emissaries, watching others at video gameplay and watching a streamed esports tournament. The algorithmic spectacle, then, is always situational. How we look and what we see depends largely not only on our own capacities or the materiality of the image, but on the context of the spectacle and on the apparatus set-up (e.g., watching gameplay in a museum setting, watching on the mobile phone, watching on a computer screen). A digital spectacle puts the player at a distance towards the object of play: spectating is a form of engagement with video games, which requires no direct physical action from the players. It also points towards an aspect of distancing activated by the technical and operational constitution of the image (as argued here with Flusser and Farocki). Computers mediate between the players/spectators and the games, putting the first at an ever-greater distance from their objects of observation.

But above all, it is the digitality of the spectacle, which poses a real challenge to the aesthetics question. After all, video games are both, spectacular (highly visual) and computational (highly operational) media. Can we speak of an image at all if it remains inaccessible to human senses? And how to capture meaning within the context of an image whose operationality reveals as much as it obscures? Such questions culminate in the concept of digital aesthetics, which remains trapped between continuity and discreteness. My aim in this paper was to capture this deadlock within the context of spectated algorithmic play.

Author's Note

This paper is closely based on a chapter “Spectated Play,” to be published in the monograph Playing at a Distance. Borderlands of Video Game Aesthetic (MIT Press 2022).

Endnotes

[1] A playful interpretation of the novel by Lewis Carroll’s Through the Looking-Glass, a sequel to Alice’s Adventures in Wonderland. In the novel, Alice enters a fictional world through the surface of the mirror. As a matter of allegory, video games displayed on the screen can be perceived as such mirrors that allow their player-analysts to look behind the visual veneer into the world of code operations.

[2] Cultural techniques (Ger. Kulturtechniken) is a concept in German media theory referring to the interactions between humans and media. Some of the examples of cultural techniques include: writing, reading, painting, counting or music making. A succinct description of Kulturtechniken may be found in an online wiki “Monoskop” devoted to arts, media and humanities: https://monoskop.org/Cultural_techniques.

References

Aarseth, E. (2011). Define Real, Moron! Some Remarks on Game Ontologies. In S. Günzel, M. Liebe, and D. Mersch (Eds.) DIGAREC Keynote - Lectures 2009/10 (pp. 50-69). University Press. https://publishup.uni-potsdam.de/opus4-ubp/frontdoor/deliver/index/docId/5044/file/digarec06_S050_069.pdf

Aarseth, E. (2007). Doors and Perception. Fiction vs. Simulation in Games. Intermédialités / Intermediality, (9), 35-44. https://doi.org/10.7202/1005528ar

Anable, A. (2018). Playing with Feelings: Video Games and Affect. University of Minnesota Press.

Arcangel, C. (2011). Various Self-Playing Bowling Games. Modded game controllers and various video games. Coding by narrat1ve.com. Whitney Museum of American Art, New York.

Bogost, I. (2006). Unit Operations: An Approach to Videogame Criticism. MIT Press.

Cheng, I. (2015-2017). Emissaries Trilogy: Emissary in the Squat of Gods; Emissary Forks at Perfection; Emissary Sunsets the Self [Live simulation]. MoMA PS1, Queens, New York. https://www.twitch.tv/moma

Cheng, I. (2017, September). Ian Cheng: A Portal to Infinity [Interview]. Louisiana Channel; Louisiana Museum of Modern Art.

Eskelinen, M. (2001). The Gaming Situation. Game Studies: The International Journal of Computer Game Research, 1(1). http://www.gamestudies.org/0101/eskelinen/

Farocki, H. (2004). Phantom Images. Public, (29), 12-22. https://public.journals.yorku.ca/index.php/public/article/view/30354/27882

Fazi, M. B., Fuller, M. (2016). Computational Aesthetics. In C. Paul (Ed.) A Companion to Digital Art (pp. 281-296). John Wiley & Sons.

Fizek, S., and Dippel, A. (2019). Laborious Playgrounds: Citizen Science Games as New Mode of Work/Play in the Digital Age. In R. Glas, S. Lammes, M. de Lange, J. Raessens, and I. de Vries (Eds.) The Playful Citizen: Civic Engagement in a Mediatized Culture (pp. 255-72). Amsterdam University Press.

Flusser, V. (2011). Into the Universe of Technical Images. (N. A. Roth, Trans). Minnesota University Press. (Original work published 1985)

Frasca, G. (2003). Ludologists Love Stories, Too: Notes From a Debate That Never Took Place. In M. Copier and J. Raessens (Eds.), Proceedings of the 2003 DiGRA International Conference: Level Up (pp. 92-99).

Hansen, M. (2016). Algorithmic Sensibility: Reflections on the Post-Perceptual Image. In S. Denson and J. Leyda (Eds.) Post-Cinema: Theorizing 21st-Century Film (pp. 785-816). REFRAME Books.

Hoel, A. S. A. [Aud Sissel]. (2018). Operative Images: Inroads to a New Paradigm of Media Theory. In L. Feiersinger, K. Friedrich, and M. Queisner (Eds.) Image - Action - Space: Situating the Screen in Virtual Practice (pp. 11-27). De Gruyter.

Jayemanne, D. (2017). Performativity in Art, Literature and Video Games. Palgrave McMillan.

Kittler, F. A. (1997). There Is No Software. In John Johnston (Ed.) Friedrich A. Kittler Essays: Literature, Media, Information Systems (pp. 147-55). Overseas Publishers Association.

Manovich, L. (2001). The Language of New Media. MIT Press.

Marino, M. C. (2020). Critical Code Studies. MIT Press.

Maymin, P. Z. (2021). Smart Kills and Worthless Deaths: eSports Analytics for League of Legends. Journal of Quantitative Analysis in Sports, 17( 1), 11-27. https://doi.org/10.1515/jqas-2019-0096

Mäyrä, F. (2008). An Introduction to Game Studies: Games in Culture. Sage Publications.

McKeown, C. (2018). Playing with Materiality: An Agential-Realist Reading of Sethbling’s Super Mario World Code-Injection. Journal of Information, Communication and Society, 21(9), 1234-45.

Möring, S., and M. de Mutiis. (2019). Camera Ludica: Reflections on Photography in Video Games. In M. Fuchs and J. Thoss (Eds.) Intermedia Games -- Games Inter Media: Video Games and Intermediality (pp. 69-94). Bloomsbury Academic.

Nake, F. (2006). Das doppelte Bild [The Twofold Image]. In C. Blümle, H. Bredekamp, M. Bruhn, and K. Müller-Helle (Eds.) Digitale Form, Bildwelten des Wissens: Kunsthistorisches Jahrbuch für Bildkritik (pp. 40-50). De Gruyter.

 Nake, F. (2018). The Algorithmic Art Manifesto. In A. Sick (Ed.) Nevertheless: 17 Manifestos.Textem Verlag. http://17.manifestos.de/

Paglen, T. (2014). Operational Images. e-fluxJournal, (59). https://www.e-flux.com/journal/59/61130/operational-images

Parikka, J. (2012). What is Media Archeology. Polity Press.

Riot Games. (2009). League of Legends [Microsoft Windows]. Digital game published by Riot Games.

Riskin, J. (2003). The Defecating Duck, Or, the Ambiguous Origins of Artificial Life. Critical Inquiry, 29(4), 599-633. https://doi.org/10.1086/377722

Schwingeler, S. (2011, February 24-25). Simulation of Arbitrary Perspectives in Video Games [Paper presentation]. Ludotopia II Conference and Workshop, Greater Manchester, UK.

Uricchio, W. (2011). The Algorithmic Turn: Photosynth, Augmented Reality and the Changing Implications of the Image. Visual Studies, 26(1), 25-35.

Turkle, S. (1996, January 1) Who Am We? Wired, 4(01). https://www.wired.com/1996/01/turkle-2/