Game Studies

From Dead-end to Cutting Edge: Using FMV Design Patterns to Jumpstart a Video Revival

Abstract

This paper argues that the identification and analysis of design patterns in interactive cinema games can be useful in exposing design solutions that can scaffold the creation of new works, including games using emerging forms of captured media like volumetric video. It presents results emerging from a historical analysis of over ninety games using full motion video (FMV) or captured media. The games are analyzed with HACS, a tool that provides an easy way to visualize the most common or original gameplay patterns in any corpus. Interesting configurations taken from this extensive corpus of FMV games are presented in five major categories: spatial navigation, information gathering, social interaction, resource management and activation. The last section explores methods for translating this knowledge back into the creative game design process.

Keywords: Video game history, Video game analysis, Design patterns, Full motion video, Volumetric video

Introduction

At the end of 2018, Netflix released a five hour long “choose your own adventure” TV show under its popular Black Mirror banner. Bandersnatch was flagged in the media as a call-back to the interactive movie games of the late 1990s. As the work of Bernard Perron highlights (2012), many subgenres of interactive movies emerged during the first decades of videogame history; the fascination for “full motion video” (FMV) led to a corpus of over 200 movie games. FMV was quickly superseded by the fully realized motion of 3D animated games, and, perhaps more damningly, contributed to the demise of powerful studios such as Cinemaware and Sierra (Donovan, 2010; Brookey, 2010; Russell, 2012).

Recently, Sam Barlow’s award-winning videogame Her Story (Barlow, 2015) created waves in both the independent and mainstream game communities by reviving a database storytelling model, using video footage in the context of a true crime narrative. The surprise success of Her Story initiated a tenuous surge of interest in videogames that incorporate, and indeed foreground, video footage (Cobbett, 2015; Hughes, 2016). As Hughes suggests, this revival echoes a growing desire to broaden the thematic, aesthetic, and even ludic frame of current videogames. However, the integration of captured [1], indexical media (in particular video) [2] into videogames is not without its challenges. The most recent release in the Guitar Hero franchise (Guitar Hero Live, Freestyle Games, 2015), featuring reactive full motion video filmed from a first-person perspective, was poorly received, derided based on many of the same criticisms lobbed at early FMV games.

Along with this minor resurgence in FMV gameplay among independent and genre-specific mainstream games, videogames using emerging technologies such as hologram-like volumetric video [3] face challenges similar to those associated with FMV. Both traditional video recording and volumetric video share common affordances (Norman, 1999), or more specifically, material constraints to the ways in which they can be incorporated into videogames and gameplay. Some of these constraints are obvious: video is durational, and in many ways tied to the specific material instance of its recording. This gives it a relative immutability compared to the created and/or generated assets that make up the bulk of contemporary videogames. Video is also data intensive -- in some ways, much more so than comparatively economical 3D animation, which can create challenges related to, for example, file size and data transfer. These constraints can result in the difficult integration of recorded linear images into dynamic, interactive game environments. As such, volumetric video, despite wide adoption in areas such as music videos, and non-fiction VR (virtual reality) and AR (augmented reality) contexts, has yet to break through into conventional game design [4]. Given the fragile interest and increased experimentation with games using traditional (flat) or volumetric captured video content, it would be useful for designers to have better access to design solutions embedded in common approaches to similar captured media.

How do we access knowledge from such a relatively underexplored and undervalued corpus? Highlighting a continuity -- in terms of challenges and potential design solutions -- between live-action digitized movie clips (FMV) and volumetric video, this paper offers an approach based on the analysis of patterns emerging in a large sample of games built with captured media. The first section provides an overview of FMV games in academic and design literature, focusing on its potential and challenges. The second section introduces the design pattern extraction methodology (HACS) and research protocols designed for a systemic analysis of our historical corpus (listed in Appendix F). The third section highlights the most interesting results emerging from this analysis. A final discussion explores different ways to assist designers in transferring these findings into the creative process.

1. FMV overview

The commercial convergence of Hollywood and Silicon Valley (which has been termed “Siliwood” in some accounts) attracted a lot of academic attention: at least three monographs on the topic have been published so far (Blanchet, 2009; Brookey, 2010; Russell, 2012). Blanchet focusses on the history of this synergy, from the influence of popular movies on early videogames to the rise of blockbuster adaptations in the 1980s. Brookey highlights the reward function of spectacular movie clips in offerings such as Electronic Arts’ 2006 Godfather adaptation (2010, p. 60) and points out that these cinematic intermissions “remind players of the other products to be bought” in media franchises such as Lord of the Rings (2010, p. 47). Jamie Russell echoes interactive movie game pioneer Greg Roach when he asserts that “the photorealism of FMV set a benchmark for cinematic, realistic looking game visuals that 3D graphics have been working towards matching ever since” (2012, p. 112). For anyone who experienced the craze firsthand in the early 1990s, this technological glorification might appear suspicious: the “glitchy” nature of digitized movie clips often took precedence over their alleged photorealistic qualities. Paradoxically, movie games lost consumer interest at the end of the 1990s, when gaming consoles and computers were finally able to duplicate cinema images more seamlessly.

One of the most useful contributions to better understand the potential of video media in games was published shortly after the FMV crash. Lev Manovich’s The Language of New Media moves from initial strong distinctions -- the titular divide between media old and new -- into a more nuanced examination of how many aspects defining the computer medium have been anticipated to some extent by previous media and, most clearly, cinema (2001). Manovich’s account highlights the difficulty of properly remediating cinematic illusion during the heyday of interactive movie games: “CD-ROM designers have been able to go from a slide-show format to the superimposition of small moving elements over static backgrounds and finally to full-frame moving images” (2001, p. 313). The author points out the “glitchy” nature of most remediated film and video from this era; digitization at the time inevitably implied data compression and artifacting [5].

Citing the rise of digital special effects in the 1990s, Manovich highlights how contemporary cinematic images came to rely on an intricate maillage in which animation techniques hold a privileged position. FMV games offer a distinct constitution of this sort of media collage: to address production and technological constraints, most of the titles in the 1990s assembled still photographic backgrounds, computer animation and live-action characters (2001, 301). Still, as Manovich points out in his analysis of Myst (Cyan, 1993), The 7th Guest (Trilobyte, 1993), and Johnny Mnemonic (Propaganda Code, 1995), these games appear to envy pre-digital cinema in their aim for “the exact duplication of cinematic realism” (2001, 314). While their fabrication can be seen on one hand as the epitome of compositing, the FMV component in these titles nonetheless represents a setback in terms of the new media traits highlighted by Manovich; most notably in modularity (the ability to fraction assets in order to manipulate them independently) and automation (the ability to program reactions and virtual behaviors for all these fragments, a close equivalent to Janet Murray’s procedurality). In other words: “Interactive movies do not foster emergent gameplay, behaviors and strategies. A common complaint is that they offer very limited freedom of action” (Lessard, 2009). Where the dynamic worlds of animated games could realize the full potential of the medium, games incorporating less dynamic, less interactive cinematic media were positioned (both explicitly and implicitly) as suspect and retrograde.

The meaning of the expression “full motion” in video games at the time can better be understood through Thomas Lamarre’s concept of “cinematism” (2009). According to Lamarre, the cinematic potential became associated with the medium’s ability to move us in space through camera movement or editing techniques. Cinema language has developed in a way that favors complicated crane shots and editing between multiple vantage points. Technological constraints made it difficult to implement this intricate visual exploration of space in videogames at the turn of the 1990s. In this context, FMV represented an easy way to remediate such aspects.

As we have seen in this overview, the integration of captured media in videogames can be analyzed as a limitation and even a regression in terms of interactivity. Well before the emergence of FMV, 2D and 3D assets could afford users real-time manipulation, dynamic reformulation of game states, and overall a greater variability as a result of their intrinsic modularity and automation potential. And while FMV was sold on a promise of visual realism -- the forward march of cinematic remediation becoming an attraction in and of itself -- It was severely impeded by compression requirements and poor integration within dynamic game environments. Yet, given that contemporary 3D engines manage to simulate nearly photorealist scenes with ease, the surprising longevity of interactive movie games demonstrates that there’s an interest for captured media outside of cinematic remediation. Furthermore, as Jonathan Lessard demonstrates in his paper on the “premature burial of interactive movies” (2009), design elements integrated within this corpus are still active in contemporary game designs. Perhaps this knowledge should not be discarded so swiftly?

2. Methods for extracting and analyzing

This section introduces the protocols developed for corpus selection and analysis. First, we consulted an extensive database of FMV games developed by Bernard Perron and his team in the course of a research project on interactive cinema (Perron et al., 2008): dozens of titles were selected in order to reflect a variety of genres, production studios, and periods in videogame history. We also wanted to integrate movie games released after 2008 to see how gameplay patterns evolved beyond this initial documentation effort. Finally, a limited number of titles using captured media in a broader sense (beyond video) were integrated in the corpus as a point of comparison (such as photograph digitization in David Wolf Secret Agent [Dynamix, 1989] and rotoscoping [6] in Another World [Chahi, 1991]). The complete list can be consulted at the end of this paper in Appendix F. Furthermore, scholars who want to explore our corpus and our data can access the HACS infrastructure on ludov.ca.

If, as we suspect, there are relevant design solutions embedded in FMV games that can scaffold the design of new works using captured media, how do we identify them, generalize them, and re-integrate them into the design process? Much literature exists on the notion of game ontology and design patterns. Building on their contribution Patterns in Game Design since 2004, Staffan Björk, Jussi Holopainen and their team have now identified over 600 game design elements. The Game Ontology Project wiki encompasses 71 interfaces elements, 58 entries about rules, 37 concepts pertaining to “entity manipulation”, and 12 types of goals (Zagal and Bruckman, 2008). Jesper Juul’s in-depth inspection of matching tile games makes a convincing case for the development of an evolving ontology, at the risk of seeing the number of relevant patterns explode in order to properly account for the history of any game genre (2016). In order to clearly identify specific gameplay patterns, we chose to encode our FMV games corpus through HACS (the Historical-Analytical Comparative System; Therrien, 2017). This section introduces the basic conceptual elements of the system and highlights some of its functionalities.

The breadth and complexity of the expansive knowledge repositories presented above make these tools difficult to use in a consistent manner with a large corpus; coders would need to keep in mind hundreds of concepts, and the coding process in itself would be time-consuming. HACS is based on a restricted set (28 core notions) in five categories, which is meant as a compromise between complexity and operability for encoding. However, interlocking the different notions during the coding process makes it possible to restore the complexity of design patterns. Many concepts have been defined as sub-elements in encompassing continuums that go from the most embodied types of interaction to the most symbolic. At its core, HACS seeks to identify which dimensions of human agency (such as navigation, social interaction, resource management) are modelled in the game (see “interactive figures” in Appendix A) and then provide specific information about how these dimensions are integrated at the level of the interface and performance. For example, a game incorporating spatial navigation might do so in a great variety of ways in terms of interface; HACS is able to encode combinations in order to specify subtypes. The system focuses solely on action modelling, letting go of most classic categories pertaining to space, time and visual perspective commonly seen in game ontologies.

Compared to the vast lexicon of interfaces that have been used in videogame history, a more synthetic approach makes coding more simple and efficient. In HACS, videogame interfaces are conceptualized between three layers: manipulation interface (5 concepts, see Appendix B), mapping of actual player manipulations with virtual actions (5 concepts, see Appendix C), and gameworld feedback (3 main concepts: diegetic, signaletic, ludic; see Appendix D). Through this approach, HACS can be used to effectively transcode the dozens of expressions used to describe interfaces in a functional way. Having only five general categories for manipulation interface might seem overly limited; it implies that a point ‘n click icon interface seen in a graphical adventure game and a drop-down action menu using verbs are both “screen-augmented interfaces”, which provides less historical accuracy than other tools. On the other hand, this simplification makes it possible to encode games more quickly, and thus opens up the possibility of encoding a larger corpus. In turn, the combination of a synthetic approach and a broad corpus makes it possible to track comparative prevalence of specific patterns, common design trends and notable outliers over a historic timeframe. HACS doesn’t seek to replace the more extensive design knowledge repositories; it co-exists in order to open up visualization possibilities.

HACS’s mapping layer encompasses the most abstract concepts in the system and would benefit from some more conceptual context before we present the results of our analysis. HACS delineates five ways of mapping player manipulation on the interface with virtual actions (see Appendix C), integrated in a continuum from the most isomorphic (lifelike gestures) to the most symbolic (typing “move north” and losing all agency while watching the character perform, or reading a textual description of the new locale). The dream of a symbiotic mapping between user manipulations and virtual actions (for example, reaching out and grabbing a virtual object) represents the most natural form of mapping and is often presented as more immersive. Videogames integrating corporeal interfaces rely more clearly on what Rune Klevjer has discussed as miniature gestures, or metonymic mapping (2006). The other extreme -- punctual mapping -- has been inspired by James Newman’s notion of “ergodic punctuations” (2002); it supposes that the mapping between player manipulations and virtual actions is active at a clear point in time, beyond which the system takes over to complete virtual actions. Through HACS, the nature of the interface can be specified for each category of mechanics, effectively segmenting games between major components of action modelling. For instance, while a game might integrate metonymic mapping for neutralization of enemies, spatial navigation can rarely be modelled through more natural action mappings.

Armed with this synthetic analytical tool, research assistants were able to encode games during a one-hour inspection through audiovisual archives such as Let’s Plays. In most cases, members of the team had a prior firsthand experience with the games, and this was essential in a few cases where action modelling couldn’t be inferred properly through audiovisual traces. The encoding procedure involves noting down all the relevant combination of concepts relevant to a particular game. Budget limitations made it impossible to encode all 217 titles unearthed by Perron and his team. Still, with about 40% of known titles, we can safely assert that this sample is a reasonable representation of FMV games for the purpose of this study.

One clear bias appears in our corpus selection: over 50 games can be related loosely to the adventure genre, a dominant model at the time of the initial FMV surge. As we noted earlier, FMV cannot easily be integrated in dynamic game structures. Adventure games were more suited to the technology’s affordances in many ways, and this will become evident as we present specific patterns in the results section below. Action games were also produced and are reflected in our corpus, including shooting galleries such as Firefox (Atari, 1983), Mad Dog McCree (American Laser Games, 1990), Corpse Killer (Digital Pictures, 1994) and Hardline (Cryo, 1997). Beyond specific interactive figures, HACS defines five encompassing modes of performance based on concepts commonly used in the community (Appendix E). The aforementioned shooting galleries correspond to neutralization mechanics, which are present in 40% of our corpus, and almost always performed in “execution” mode. This concept refers to simple “hit the button at the right moment” or very fast eye-hand coordination challenges, implemented through static elements for the most part. Typical action games co-existing with FMV games at the turn of the 1990s were built with modular assets such as tiles and sprites, which could accommodate more variability; this extended possibility space has been defined as “tactical coordination”. Understanding these two modes makes it easy to understand why comparatively fewer action games were produced using FMV. Digital Pictures’ Supreme Warrior, built using FMV segments to depict street fights, leads to very static matches compared to the Street Fighter (Capcom) series, for instance. Action FMV games are heavily restricted and sometimes described as “Dragon’s Lair style”, referring to the classic “hit the button at the right moment” arcade game. Instead of tactical imagination, memorizing skills are put to the test.

Most mechanics in adventure games rely on static game states and in-game consequences, but players are faced with a wider range of theoretical possibilities which lead to more complicated action planning. In HACS, this corresponds to the “resolution” mode of performance. Following a similar encoding of 140 adventure games, the genre has been defined as a resolution challenge where players need to articulate mechanics from five major interactive figures in order to solve linear puzzles (Therrien, Lefebvre and Ray, 2019). The five most prevalent interactive figures remain the same for our FMV corpus: navigation (82% of the games), apprehension (63%), social interaction (71%), resource management (62%) and environmental activation (70%). Data analysis led to the identification of the most common interface configurations for each figure. The following presentation of major patterns results from this tool-assisted analysis [7]. Since the system seeks to bring greater visibility to a broader range of games in history, we will refer to many titles beyond the most typical example for each pattern. Further references beyond our corpus will highlight the persistence of some design solutions throughout videogame history. To make things easier to read, key HACS concepts defined in the appendices have been italicized.

3. HACS results

Engaging with Spaces: Navigation

Navigation of space has been highlighted countless times as one of the major components of videogame play. While essential to many genres, this assertion should be tempered; some genres and games don’t integrate navigational mechanics at all, favoring other figures presented in HACS. In the context of FMV games, navigation mechanics embody a paradoxical design stance. This section introduces two major and contradictory navigational patterns emerging in our corpus; following subsections will discuss the other four relevant interactive figures.

The first pattern associated with FMV navigation is defined through the dissolution of action mapping, or punctual mapping. This model implies that an on-screen character will perform the actions by itself following players’ commands. This is most evident in the infamous 7-CDs horror extravaganza Phantasmagoria (Sierra On-Line, 1995). When players instruct Adrienne to explore her new mansion at the beginning of the game, they kick off the action and then lose control over short animations depicting character movement. This configuration highlights one of the paradoxes of FMV games: in order to enjoy the cinematic, we are ready to be “live” and active with this playable entity for just a few “ergodic punctuations”. This punctual form of mapping was perfectly suited to FMV; Phantasmagoria replaced drawn animations commonly seen in the adventure genre with cut-out live-action sequences playing over still backgrounds (figure 1). However, such opulence came at a heavy cost; in order to produce 7 CDs worth of cinematic content, Sierra invested over 4 million dollars (one of the largest budgets of any game at the time).

Figure 1. Screenshot credits to MobyGames.com. Click to expand.

While media works such as Tracing the Decay of Fiction (Rosemary Comella, Kristy H.A. Kang, Marsha Kinder and Pat O’Neil, 2002) have mapped spatial navigation synchronically to FMV segments, no movie games in our corpus adopted the same solution. Paradoxically, the most common strategy supposes a complete negation of FMV potential, and relies on a synthetic diegetic feedback (visual, to be more specific): world elements are shifted to acknowledge player navigation, but the feedback is elliptical or accelerated. This design acts as a stand-in for longer action sequences. One advantage of synthetic feedback comes in its ability to maintain a limited form of synchronic action mapping. In Myst, players “jump through” space in “real time”; mapping is felt as though it is synchronic even though this synchronicity relies on a visual ellipsis. Synchronic mapping can usually be associated with direct and constant control of a character animated on screen, or of the point of view itself (diegetic visual feedback). The elliptical variant is ideal in a design context where assets are too data intensive or taxing to process in real time, and this was certainly the case for FMV in 1993. Even though players have no control over the implicit virtual actions performed during the ellipsis, they re-orient navigation after each jump. This synthetic navigation design can be found in games such as The Dark Eye (Inscape, 1995), The Journeyman Project: Turbo! (Presto, 1993), and The X-Files Game (Hyperbole, 1998); we can still see this configuration in more recent FMV games such as The Shapeshifting Detective (D’Avekki Studios, 2018).

Even though navigation mechanics are present in the majority of our corpus (82%), most of the games don’t fully depict the virtual actions through video; even Phantasmagoria cuts up these segments. Videogames depicting movement through space extensively often do so in an automated way, with little to no navigation mechanics to manage. This is typical of shooting galleries which use FMV as a background for neutralization mechanics (Sewer Shark, Digital Pictures, 1992; Surgical Strike, The Code Monkeys, 1995). Considering production costs, it is easy to understand why the synthetic model was a more logical design choice at the time. However, the constant resurgence of this navigational pattern in videogame history is noteworthy. Early Myst-like games prolonged an elliptical, “square by square” or “destination by destination” configuration in which the user is teleported directly ahead upon request. The attractive nature of this design configuration became evident early, most notably in Akalabeth’s dungeon (Garriott, 1979) and the many computer RPG series following the success of the Ultima series. Most interestingly, this pattern is integrated in contemporary VR experiments where it is referred to as a “teleportation mechanic”, for instance in Fallout 4 VR (Bethesda, 2017) or Driftwood, a demo environment created for SteamVR (HTC Creative Labs, 2017). The perseverance of synthetic navigation highlights how established ludic conventions can influence design choices to the point of negating FMV affordances. Beyond videogames or VR experiments, this design could even be related with the idea of moving a miniature across a board.

Engaging with/as Static Characters: Apprehension and Social Interaction

One rare instance of synchronic action mapping over a fully animated visual diegetic feedback could be found in our corpus: in The Fear, an FMV horror game released exclusively in Japan on PlayStation 2 (Digital Frontier, 2001), players walk around through punctual mappings in first-person. At some key points, they can zoom in on elements of interest and need to keep pressing on the button to push the point of view forward progressively. This type of mechanic highlights how the static nature of FMV is not completely incompatible with fully enacted synchronic action mappings.

In HACS terminology, the figure of apprehension refers to mechanics involving the sensorial or cognitive exploration of virtual space, from the “mouse look” control in first-person shooters, to “look at” commands in traditional adventure games or the more unusual “taste” mechanic found in Mean Streets (the first episode in the famous Tex Murphy series by Access, 1989). In our corpus, this figure is more typically integrated though a screen-augmented interface where the mechanics are expressed textually. Our analysis revealed a mediatic collage between multiple forms of diegetic or extradiegectic feedback provided to users following input (for reference, see Appendix D).

In the spirit of cinematic realism evoked by Manovich, visual media should show, not tell. However when it comes to apprehension mechanics, verbal feedback is strikingly omnipresent in FMV games. 44% of our corpus integrates diegetic verbal retroaction for this figure (which is present in 63% of all games). Most typically, playable characters sum up their observations in order to communicate relevant information (or red herrings) to players. Since they often address players beyond the 4^th wall, these characters appear to resist their status as a mere prop to enact player actions (even though many players in their alterbiographical accounts would still identity with such protagonists). While FMV was rarely used to depict these verbal “replies”, in CD-ROM games it is not uncommon to replace written text with recorded voices. Action mapping is still punctual in this design: users lose control momentarily in order to read / listen to the feedback. Anyone familiar with the adventure genre will recognize this type of apprehension mechanics, for they have been integrated since the very inception of the genre in textual adventure games. The persistence of this design in game culture at the age of data intensive assets such as FMV is surprising. Legend Software, a corporation that contributed greatly to the textual and graphical adventure genres, still used this form of apprehension in Mission Critical, released in 1995 on multiple CDs. Upon navigating to a new room or looking at parts of the environment, players receive verbal information printed below the main window (figure 2).

Figure 2. Screenshot credits to MobyGames.com. Click to expand.

Many games in the corpus were inspired by police drama and detective stories. These genres often feature computers or other digital tools commonly used in investigations. When such props are integrated in a detective videogame, apprehension mechanics can be implemented in a more naturalistic manner, since there is a close proximity between the users’ manipulations at their computer and the virtual actions occurring on the screen. In Spycraft: The Great Game (Activision, 1996; Figure 3) or Her Story for instance, FMV clips become mere documents to be consulted on a virtual desktop along with all the other textual sources. The loss of synchronic action mapping and the static nature of the feedback become fully expected in this context. Through these fictional scenarios, the typical screen-augmented interface (windows, icons, menus, pointer) becomes techno-mimetic; a sort of metaleptic telescoping between the real and virtual computer occurs, providing symbiotic action mapping through the inherent limitations of the computer interface. Another variation of this design pattern can be seen in the infamous Night Trap (Digital Pictures, 1992): through a surveillance apparatus, users switch between rooms / FMV streams in the hope of saving teenagers in distress at the hands of modern day vampires. The presence of a mediating device can help contextualize the affordances of these images and attune user performance; live action clips are read more explicitly as being recorded or broadcast, which tempers our expectations for real time interaction, co-presence and social cues.

Figure 3. Screenshot credits to MobyGames.com. Click to expand.

Another common figure seen in FMV games involves “talking” with non-player characters. This figure of social interaction is typically overemphasized in adventure games, and absent or automated through cut-scenes in action games. This is the most common figure where FMV has been integrated in our corpus, from its minimal and largely automated implementation in Sherlock Holmes: Consulting Detective (ICOM Simulations, 1991) to the extensive interrogations in the cyberpunk detective thriller Ripper (Take-Two interactive, 1996). For all the intricacies instilled in 2D and 3D characters, there is a degree of visual realism, indexical specificity, and expressivity that only live-action capture can provide. This affords the reading of character posture, expression, and to some degree, presence, which can play a key role in certain narrative genres. The subtleties of unintentional captured information like micro facial expressions are an essential quality for games like Her Story. Moreover, in these scenarios FMV may not represent a perceived trade-off in terms of interactivity: conversations in games have mostly been integrated through dialogue trees offering completely static game states, with minimal modularity or procedurality.

There have been a few attempts in videogame history to use a corporeal interface / symbiotic mappings for verbal interactions (for instance through a microphone with speech recognition software), yet the most obvious way to achieve a more natural model can be related with a classic “chatbot” design: users talk to someone via a computer in order to receive feedback in textual form. Much like the apprehension situation in detective games described above, here the computer becomes a techno-mimetic interface. This was very rarely implemented in FMV games, or in videogames for that matter. The Infectious Madness of Doctor Dekker (D’Avekki Studios, 2017) simulates psychoanalytic dialogues and allows the player to choose between a set of questions or to type their own. Like Her Story, some secret questions are never proposed in the sets and have to be discovered and typed. Nearly 100% of social interaction in our corpus occurs through pre-made question / replies on a screen-augmented interface, each triggering an answering scene from characters that removes control from the player’s hands (punctual mapping).

In early FMV games such as Martian Memorandum (part of the Tex Murphy series), one can observe the same kind of discrepancy noted earlier about apprehension mechanics: some conversations are carried out through audiovisual segments, while others are actualised only through textual feedback. Even with the advent of CD-ROMs, this discrepancy remains visible, for instance in the CD version of Cryo’s KGB; outside of FMV insights given by the protagonist’s father (played by Donald Sutherland), most of the verbal exchanges occur textually. Before the advent of captured media in games, dialogue trees had been acted out through synthetic voices on the Amiga and Atari ST computers (which featured such affordances in their operating systems), most notably in Mortville Manor and Maupiti Island (Lankhor, 1987 and 1988 respectively). This integration, which appears incredibly robotic today, highlights how audiovisual capture was sought after for social interaction mechanics.

Engaging with Objects: Resource Management and Environmental Activation

In conventional adventure game designs, once information has been gathered through apprehension and social interaction mechanics, players need to solve puzzles in order to move ahead. Many puzzles rely on finding the right “key” to activate the right “lock”, which tend to implicate the last two figures presented in this overview: resource management and activation.

FMV games became popular at a time when adventure games integrated mouse controls to a large extent. For many figures, this meant using a screen-augmented WIMP interface through which users could select virtual actions in a menu, and witness or read about the consequences of the action. Diegetic integration of such screen-augmented interfaces occurred relatively early following the introduction of mouse control in computer games; the techno-mimetic situation seen in Spycraft is similar to Steve Cartwright’s design strategy for the Hacker series (Activision, 1985 and 1986), for instance. Beyond this diegetic integration of the user’s computer controls, the mouse can also be used as a natural interface to some extent; after all it was created as a way to “capture” and translate hand gestures in the context of a simulated office environment. This corporeal interface restricts user movements on a 2D plane; it cannot accommodate a fully symbiotic action mapping. The concept of metonymic mapping was introduced in HACS in order to account for the numerous gaming situations were miniature gestures are integrated in the design. This can be traced back to 1985 at least; in Déjà Vu (ICOM Simulations), users drag items from the main view and “drop” them in their inventory window, only to invert this mechanic in other locales in the hopes of finding the right “key” to open all kinds of “locks”. More than 20% of all titles in our corpus integrate this configuration, including The Journeyman Project, Ripper, The Beast Within: A Gabriel Knight Mystery (Sierra On-Line, 1996), and The X-Files Games.

This type of resource management was never depicted using FMV directly. In Toonstruck (Burst, 1996; Figure 4), the screen-augmented interface “registers” that an item has been picked up by modifying the pointer with a figurative icon depicting the object. Players are then invited to drag and drop this item on various parts of the screen to test its affordances, as if they were holding it in their hands. In some cases, picking up items can trigger an FMV segment depicting the action, most notably in Sierra On-Line interactive movie games. By definition, metonymic action mapping dissolves here in favor of a punctual design. In effect, this is a combination of both types of mapping in a more complex pattern. Not every manipulation attempt will trigger an FMV scene, the latter being reserved for actions that are more meaningful or useful for ludic purposes. FMV can thus act as a form of reward or a progression marker.

Figure 4. Screenshot credits to MobyGames.com. Click to expand.

Cumulative is the last type of action mapping defined in HACS. It has been implemented for many different figures but becomes most prevalent for activating elements in the environment in order to progress. This type of mapping involves accumulating many gestures in order to trigger the intended game state modification. Adventure games in particular integrated commands such as “push” or “open” to modify aspects of the various puzzle rooms, and many games such as Phantasmagoria or Lost in Time (Coktel Vision, 1993) used FMV to depict some of these actions / emphasize progression. But most puzzles in the genre rely on a more intricate manipulation sequence: selecting a verb, an item, and a target in the environment. In this three step pattern, it is common for the figures of resource management and activation to become functionally intertwined. For instance in Rex Nebular and the Cosmic Gender Bender, our titular hero explores a foreign planet at the beginning of his adventure. At some point he needs to lure a creature with giant legs into a hidden trap. To do so, players have to select the verb “use” on the screen-augmented interface, then a delicious twinkifruit (collected earlier), and finally “drag” the item on the trap to activate it. Rex then proceeds to set up the trap, and the poor creature jumps right through it.

Rex Nebular, as Toonstruck, is not built with FMV in a strict sense: Gametek used a mix of rotoscoping and video digitization to create animated sequences. However, these sequences function exactly as FMV in this ludic economy: the lavishly depicted sequence acts as a form of reward-spectacle (Surman, 2007), acknowledging player progress within the adventure. Many FMV games such as The Beast Within and Lost in Time relied on the exact same design pattern, which we can define through HACS as a metonymic+cumulative+punctual combination. In such a scenario, the manipulation interface foregrounds both screen-augmented and corporeal elements. This intricate mapping configuration highlights the most important actions performed in the virtual world from a ludic perspective. It also embodies another paradox in FMV design: while parts of the design involve more natural action mappings, the diegetic audiovisual feedback serves as a form of passive spectacle-reward within the same sequence.

Similar action modelling was integrated for an original apprehension mechanic in The Dark Eye (Inscape, 1995). This simple adventure invites players to actualize famous Edgar Alan Poe stories from the perspective of its characters. At various points, it becomes possible to switch to the point of view of another character by waving a virtual hand repeatedly through the mouse; this metonymic, cumulative mapping triggers a short transition cutscene. While such experiments might have been rare outside of the resource management and activation combo illustrated in this section, it is interesting to note that similar designs have spread in contemporary games; most notably in Telltale or Quantic Dream adventures where miniature gestures accumulate to trigger all sorts of virtual actions. More interesting even, this type of design is integrated within action-heavy genres, for instance in modern fighting games (Surman’s main object of inquiry in his paper on reward-spectacle, 2007): players need to master complicated cumulative mappings in order to perform spectacular special moves, which play out independently following player input (punctual mapping).

4. From Pattern to Practice

Many of the patterns and ontology literature listed in the bibliography insist on the transferability of knowledge into the design process: Parkkila et al.’s “Video Game Ontology” project, for instance, clearly puts forward this objective (2017). Transferring expansive knowledge repositories represent a daunting task from a designer perspective. Katherine Neil notes that game designers have resisted attempts to formalize their practice, whether through abstract frameworks or software tools. She argues game design has an epistemic culture more similar to crafting or apprenticeship than architecture, engineering or other design cultures from which formal design tools hail (2015). This tendency is reinforced through both professional practice, and in the ways game creation is taught as an iterative process with a heavy emphasis on troubleshooting through prototyping and playtesting, articulated in the popular mantra: the best way to learn game design is to make games.

While Juul (2016) observes conceptualization akin to design patterns is used in common parlance about games (for example in the description of dynamics like “capture the flag”), these expressions lack the structure of formal design patterns, and are seldom well articulated, standardized, or integrated into a formal process. Design “solutions” and models are more readily expressed using existing game precedents (such as “Dragon’s Lair-style gameplay”) as a short-hand to describe particular mechanics or dynamics (Kuittinen and Holopainen, 2009) with the limitation that new approaches then become bound to specific previous games and exemplars. This presents an obvious challenge to innovation in game design, given that the ways in which we articulate design ideas are ground in unbounded clusters of design decisions from previous games. It creates a particular challenge in genres using FMV, where the corpus is largely unfamiliar, dated, and poorly regarded by designers. The analysis performed through HACS on the FMV corpus sought to address many of these challenges, leveraging a more restricted knowledge repository in a way that can still highlight specific gameplay configurations. In this final discussion section, we reflect on some of our findings and begin to envision ways of transferring knowledge in an optimal way for game designers working with captured media.

The first section of this paper allowed us to highlight how captured media such as live action FMV presents constraints in terms of modularity, automation, dynamic game state generation and global ergodicity compared to other common videogame building blocks. Central to our thesis is the argument that volumetric video and flat full motion video share a series of affordances, and as such, present at least some common design challenges. Throughout our analysis, we have identified patterns that emerge in FMV games, as well as some divergences and special cases that may prove interesting sites for further creative investigation. By incorporating both older and newer titles in our inspection, we have been able to examine a large corpus of games and situate them contextually to some extent. Examining FMV designs historically makes it easier to understand how some patterns were more prevalent because of genre convention and popularity (in this case adventure games), not necessarily reflecting designs that were intrinsically more suited to captured media affordances. Untangling these two elements -- convention and technological affordances -- represents one of the main challenges in our quest to establish proper design strategies for creators working with captured media, specifically volumetric video.

Results emerging from the corpus analysis presented in the third section made it clear that many FMV designs could be situated at the most symbolic end of the continuums through which HACS concepts have been defined. At the same time, in other parts of these complicated mediatic collages, users could manipulate different assets in a more natural manner through the mouse. Some recent games built with captured media have integrated synchronic forms of mappings with FMV, or rather on top of FMV clips, through common game design tricks lifted from modern “Quick Time Event” action segments. A few scenes from The Bunker (Splendy Interactive) enacting the figure of self-preservation use execution challenges reminiscent of Dragon’s Lair-style gameplay, in which feedback appears to react in real time to player manipulation: the screen-augmented interface integrates a signaletic form of visual feedback indicating how much effort users need to “pump” into their controllers. In HACS terminology, this corresponds to a combination of synchronic and cumulative mapping. While this signaletic feedback adapts in real time to highlight player effort, the diegesis depicted through live action video in the background is only affected in a static and binary fashion; failure to execute the straining button presses will trigger a fail state movie clip. In this sense, the design still reverts back to punctual mapping. It is possible to imagine designs that would directly map metonymic or symbiotic gestures onto linear video instead of this signaletic solution, thanks to corporeal interfaces such as the HTC Vive wands. Studios like Quantic Dream have been keen to integrate metonymic+cumulative+punctual solutions using motion-captured 3D assets that play out in a more linear manner; there is no reason why this “on rails” mimetic design could not be adapted to FMV or volumetric video as a means of re-animating interaction with captured images.

Volumetric video is already questioning design solutions implemented in FMV games in a way that could reform navigational and apprehension mechanics. While this new form of captured media shares some of the constraints associated with FMV (most obviously the limitations of presenting the illusion of depth on a flat plane), depth information encoded during the capture process breaks down the linearity and fixed viewing angle of a traditional video format. In other words, where synthetic feedback or punctual action mappings -- particularly for navigation mechanics -- can be seen as a design solution to the challenges of 2D video, synchronic mapping can be integrated for these mechanics more seamlessly in the context of volumetric video. Such a design would maximize the feeling of embodied presence, and the affective implications of being co-present with captured volumetric characters (in VR space, for instance). So far, integration of this type of captured media have afforded users some control over the proximity of the captured characters, or even symbiotic control of the point of view through VR headsets.

While the dream of more “natural” forms of interaction represent a common obsession for videogame creators (see Jørgensen, 2013), designs that build on inherent limitations can still foster user engagement. The quest for immediacy (Bolter and Grusin, 1998) will continue to foster innovation in platform and game design, particularly in genres like documentary, sports and music performance games that trade on reality claims, yet it is clear that designers evolve in an ecosystem where seemingly nostalgic references to the past appear as natural, and are perhaps even more affectively engaging than newer interaction models. In a recent book, Jesper Juul has associated “independent style” to the rise of pixelated aesthetics and classic game molds (2019). Other markers of hypermediacy such as capture artifacts, playback glitches and a general fascination for older representational technologies proliferate in all kinds of cultural forms; one has to wonder if this tendency will go so far as to rehabilitate the compression artifact-ridden look of early FMV games [8]. Based on our analysis, we can envision design directions which would foreground FMV games’ legacy: doubling down on transparently flaunting the constraints of captured media, as an “exercice de style”.

Conclusion

We seldom acknowledge videogames as primarily a form of interactive animation, even as the vast majority of our design knowledge comes from patterns in animated games. For designers working with captured media, and specifically video, standard design approaches fall short in providing design solutions, models, and even inspirations that can help scaffold strong, innovative, and creative design. However, we are sitting on a large corpus of games that, for one brief moment in the early decades of commercial game development, faced the challenges of captured media integration head-on, and devised design solutions surrounding the affordances of this type of asset. At the same time, we are experiencing a surge of interest and experimentation in FMV game design, in part driven by nostalgia, in part driven by the desire to push the boundaries of what games are and what they could be.

Through a systematic pattern analysis of FMV games, we have sought to make-visible common design patterns and highlighted their potential role in addressing the unique affordances of captured media. How can we translate the analytical results back into design tools that can be more fluidly integrated into the design process, at either stages of ideation or the point of design frictions? One potential route, following the lead of Isbister and Mueller (2015) or Björk (2019), would be in creating a design guide or report on captured media in games that can more readily be referenced to inform both current FMV design practice, and future design work using emerging captured media forms such as volumetric video. Alternately, we have envisioned an illustrated library format -- a sort of dictionary listing common implementation solutions for each of the interactive figures we have studied -- similar to Emily Short’s Small-Scale Structures in CYOA or Clara Fernández-Vara’s playable Taxonomy of Narrative Choices. These are “middleware” approaches that attempts to bridge formal models and design practices. Such a tool could be released into the game design community in the form of a trade (Gamasutra) or trade- adjacent (Medium) article, to clearly situate it within a body of game design knowledge exchange.

One of the key features of HACS lies in its ability to point towards other relevant games for specific configurations, thus exploring history through a gameplay-centric lens in a way that opens up to lesser known titles. This work also reinforces the value of preserving historic games of all genres. What the design patterns data provides is a way of generalizing and toggling between specific and patterned instances -- finding, for example, other configurations similar to known design exemplars that might suggest different implementations of these patterns, which might be more suited to particular design goals. It provides a way to network these examples to essentially augment case-based design knowledge. This is particularly useful when dealing with knowledge that has passed out of everyday practice and expertise, as is the case with a less familiar corpus like FMV games. We could envision expanding HACS’s online interface in a way that facilitates troubleshooting design frictions -- for example, finding more exemplars fitting a certain pattern, or reverse engineered from a particular example (“I’d like the gameplay to be similar to Phantasmagoria, but want to see different implementations”).

As a final point, it important to note that this scholarly analysis has been conducted in parallel with two complementary research approaches: the development of an original volumetric video videogame (VVV) prototype, and the hosting of an experimental volumetric videogame jam [9]. This loose “triangulation” of research activities is meant to provide complimentary insights and channels for evaluating and disseminating research outcomes. The game jam became an essential component to communicate early observations about design pattern analysis and gather feedback from community game designers on how to make such a project more useful. Through the volumetric video prototype, we intend to demonstrate the value of channeling historical knowledge into the creative process, and transferring this knowledge back directly to a design community that readily looks to new works, rather than academic papers, for inspiration.

Endnotes

[1] The term captured here is set in contrast to created media forms: captured being media forms like photography and video that you “take,” as opposed to illustration or animation, media forms you “make.”

[2] Indexicality, a semiotic concept proposed by Charles Sanders Peirce, and later linked to the photographic ontology of André Bazin, highlights a relationship of contiguity or causality between a sign and its referent. It is commonly (although not unproblematically) used to describe the relationship between subjects in front of a camera, and the resultant still or video images.

[3] Volumetric video is sometimes described as a 6DoF (six degrees of freedom) technology. As the video simultaneously records surface image and depth data, the recorded images can be re-presented in 3D spatial contexts (such as virtual reality environments) without having to constrain the viewing perspective along a restricted access. In other words, you can look around the volumetric video subject and they are not flat.

[4] That being said, companies such as Microsoft and Intel have been building their capacity in this area (Joyce, 2017; Velhal, 2019), and a few smaller titles have experimented with volumetric video (such as Fire Mammoth’s The Goodbye Room, and Balenciaga's Afterworld: The Age of Tomorrow (2020)).

[5] Interestingly, the same observation could be made for the current state of volumetric video, albeit for different technological reasons.

[6] Rotoscoping, the tracing of video images, can be considered a hybrid capture technique, as it presents a kind of translation of the phenomenological effects of video, onto an animated image.

[7] The HACS infrastructures features an algorithm that can help us identify the most common interface configurations, calculate pattern prevalence, help us identify tipping points, and list all the games relevant for the selected configuration. On ludov.ca, results associated with every concept (or patterns involving many concepts) can be visualized through synthetic and timeline visualizations, and a “game viewer” features allows anyone to have a quick glance at relevant games. Readers who want to know more about the concepts and previous contributions which led to the creation of HACS can go back to the original DiGRA paper (Therrien, 2017).

[8] This can already be seen in some corners of pop culture. For instance, the music video “Champagne Coast” by Blood Orange pays tribute to the movie game aesthetic, and goes so far as to integrate a rendering glitch in one of the FMV loops. https://www.youtube.com/watch?v=nO6y1-erVEw&ab_channel=DominoRecordingCo.

[9] For more information on these parallel initiatives: https://www.instagram.com/volumetricvideovideogames/?hl=en and https://www2.ocadu.ca/research/gameplay/project/vvv-volumetric-video-in-videogames

Acknowledgements

We would like to acknowledge the support of Canada’s Social Science and Humanities Research Council (SSHRC). We would also like to thank the reviewers at the journal who helped improve this article.

References

Access Software. (1989). Mean Streets [Microsoft Windows] Digital game directed by Bruce Carver, published by Access Software.

Activision. (1985). Hacker [Amiga] Digital game directed by Steve Cartwright, published by Activision.

Activision. (1986). Hacker II: The Doomsday Papers [Amiga] Digital game directed by Steve Cartwright, published by Activision.

Activision. (1996). Spycraft: The Great Game [Microsoft Windows]. Digital game directed by Henry Yu, published by Activision.

Barlow, Sam. (2015). Her Story [Microsoft Windows]. Digital game directed by Sam Barlow, published by Sam Barlow.

Björk, S. (2019). Gameplay Design Patterns Collection. Retrieved December 28, 2020, http://virt10.itu.chalmers.se/index.php/Main_Page.

Björk, S. and Holopainen, J. (2004). Patterns in Game Design. Rockland: Charles River Media.

Blanchet, A. (2009). Des Pixels à Hollywood. Paris: Pix’n Love.

Bolter, J. D. and Grusin, R. (1998). Remediation: Understanding New Media. Cambridge: The MIT Press.

Brookey, R. A. (2010). Hollywood Gamers. Digital Convergence in the Film and Video Game Industries. Bloomington: Indiana State University Press.

Burst. (1996). Toonstruck [Microsoft Windows]. Digital game directed by Jennifer McWilliams, published by Virgin Interative.

Cinematronics. (1983). Dragon’s Lair [Arcade] Digital game directed by Rick Dyer, published by Cinematronics.

Cobbett, R. (2015). “FMV Isn’t Dead: A Second Chance for Video in Video Games.” PC Gamer (July 28). https://www.pcgamer.com/fmv-isnt-dead-a-second-chance-for-video-in-video-games/.

Coktel Vision. (1993). Lost in Time [Microsoft Windows]. Digital game directed by Muriel Tramis, published by Coktel Vision.

Cyan. (1993). Myst [Microsoft Windows]. Digital game directed by Rand K. Miller and Robyn Miller, published by Brøderbund Software.

D’Avekki Studios. (2017). The Infectious Madness of Doctor Dekker [Microsoft Windows]. Digital game directed by Tim Cowles, published by D’Avekki Studios.

D’Avekki Studos. (2018). The Shapeshifting Detective [Microsoft Windows]. Digital game directed by Tim Cowles, published by Wales Interactive.

Delphine Software. (1991). Another World [Amiga]. Digital game directed by Eric Chahi, published by U.S. Gold.

Digital Frontier. (2001). The Fear [Playstation 2]. Digital game published by Enix.

Digital Pictures. (1992). Night Trap [Sega CD]. Digital game directed by James Riley, published by Sega.

Donovan, T. (2010). Replay. The History of Video Games. East Sussex: Yellow Ants

Dormans, J. (2013). “Making Design Patterns Work.” Proceedings of the Second Workshop on Design Patterns in Games. Chania: Greece.

Electronic Arts. (2006). The Godfather: The Game [Microsoft Windows]. Digital game directed by Matthew Copeland, published by Electronic Arts.

Fernández-Vara, C. (2019). “Taxonomy of Narrative Choices”. Vagrant Cursor (May 1). https://vagrantcursor.wordpress.com/2019/05/01/taxonomy-of-narrative-choices/.

Freestyle Games. (2015). Guitar Hero Live [Xbox One]. Digital game directed by James Norris, published by Activision.

Garriot, Richard. (1979). Akalabeth: World of Doom [Apple II]. Digital game directed by Richard Garriot, published by California Pacific Computer.

Hornecker, E. (2010). “Creative Idea Exploration Within the Structure of a Gaming Framework: The Card Brainstorming Game”. Proceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction, pp. 101-108. New York: ACM.

Hughes, W. (2016). “The Return of FMV Reflects Gaming’s Growing Interest in Humanity.” AV Club (February 4). https://games.avclub.com/the-return-of-fmv-reflects-gaming-s- growing-interest-in-1798243959.

Hyperbole Studios. (1998). The X-Files Game [Microsoft Windows]. Digital game directed by Greg Roach, published by Fox Interactive.

ICOM Simulations. (1985). Déjà Vu. [Amiga] Digital game directed by Todd Squires, published by Mindscape.

ICOM Simulations. (1991). Sherlock Holmes : Consulting Detective [DOS]. Digital game directed by Kenneth Tarolla, published by ICOM Simulations.

Inscape. (1995). The Dark Eye [Macintosh]. Digital game directed by David Nichols, published by GT Interactive.

Isbister, K., Flanagan, M. & Hash, C. (2010). “Designing Games for Learning: Insights from Conversations with Designers.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2041-2044. New York: ACM.

Isbister, K. & Mueller, F. (2015). “Guidelines for the Design of Movement-Based Games and Their Relevance to HCI.” Hum.-Comput. Interact vol. 30, no. 3-4, pp. 366-399.

Jørgensen, K. (2013). Gameworld Interfaces. Cambridge: The MIt Press.

Joyce, Kevin. (2017). “Hammerhead & Microsoft Open Volumetric Capture Studio for VR Videogames & Experiences.” VRFocus (blog), October 25, 2017. https://www.vrfocus.com/2017/10/hammerhead-microsoft-open-volumetric-capture-studio-for-vr-videogames-experiences/.

Juul, J. (2019). Handmade Pixels. Independent Video Games and the Quest for Authenticity. Cambridge: The MIT Press.

Juul, J. (2016). “Sailing the Endless River of Games: The Case for Historical Design Patterns”. Presented at the 1st International Joint Conference of DiGRA and FDG 2016. Dundee. https://www.jesperjuul.net/text/endlessriverofgames/.

Karhulahti, V.-M. (2013). “Puzzle Is Not a Game! Basic Structures of Challenge”. Defragging Game Studies, Proceedings of DiGRA 2013.

Klevjer, R. (2002). “In Defense of Cutscenes”. Proceedings of Computer Games and Digital Cultures Conference. Tampere: Tampere University Press. http://www.digra.org/wp-content/uploads/digital-library/05164.50328.pdf.

Kreimeier, B. (2002). “The Case for Game Design Patterns.” Gamasutra (March 13). http://www.gamasutra.com/view/feature/132649/the_case_for_game_design_patterns.php.

Kuittinen J. & Holopainen, J. (2009). “Some Notes on the Nature of Game Design”. Breaking New Ground: Innovation in Games, Play, Practice and Theory, Proceedings of DiGRA 2009. https://pdfs.semanticscholar.org/71b7/cf6e008fb422d6cceb5a82ec7cbd42fc35b6.pdf.

Legend Entertainment. (1995). Mission Critical [Microsoft Windows]. Digital game directed by Michael Verdu, published by Legend Entertainment.

Lemay, P. (2007). “Developing a Pattern Language for Flow Experiences in Video Games.” Situated Play, Proceedings of DiGRA 2007 Conference. http://www.digra.org/wp-content/uploads/digital-library/07311.53582.pdf.

Lessard, J. (2009). “Fahrenheit and the premature burial of interactive movies”. Eludamos, vol. 3, no. 2. https://www.eludamos.org/index.php/eludamos/article/view/vol3no2-5

Manovich, L. (2001). The Language of New Media. Cambridge: MIT Press.

Miller, R. (2013). “Classic Game Postmortem: Myst” Presented at the GDC 2013, San Franciso, CA. https://www.gdcvault.com/play/1018048/Classic-Game-Postmortem.

Murray, J. (1997). Hamlet on the Holodeck: The Future of Narrative in Cyberspace. Cambridge: MIT Press.

Neil, K. (2015). “Game Design Tools: Can They Improve Game Design Practice?” Conservatoire national des arts et métiers. Paris : France. http://cedric.cnam.fr/fichiers/art_3970.pdf.

Newman, J. (2002). “The Myth of the Ergodic Videogame, Some thoughts on player- character relationship in videogames”. Game Studies vol. 02, no. 01. http://www.gamestudies.org/0102/newman/.

Norman, Donald. (1999). “Affordance, conventions, and design”. Interactions, vol. 6, 38-42.

Parkkila, J., Radulovic, F., Garijo, D., Ikonen, J., Porras, J. & Gómez-Pérez, A. (2017). “An Ontology For Videogame Interoperability”. Multimedia Tools and Applications, vol.76, no. 4, p. 4981-5000.

Perron, B. (2012). “Interactive Movies”. In M. J. P. Wolf (Ed.), Encyclopedia of Video Games: The Culture, Technology, and Art of Gaming, p. 322-324. Westport: Greenwood Press.

Perron, B., Arsenault, D., Picard, M. & Therrien, Carl. (2008). “Methodological Questions in Interactive Film Studies”. New Review of Film and Television Studies, vol. 6, no. 3 (December), p. 233-252.

Presto Studios. (1993). The Journeyman Project: Turbo! [Microsoft Windows]. Digital game directed by Richard A. Cook, published by Sanctuary Woods Multimedia.

Propaganda Code. (1995). Johnny Mnemonic [Macintosh]. Digital game directed by Douglas Gayeton, published by Sony Imagesoft.

Russell, J. (2012). Generation Xbox: How Videogames Invaded Hollywood. East Sussex: Yellow Ant.

Ryan, M.-L. (2006). Avatars of Story. Minneapolis: University of Minnesota Press.

Short, E.. (2016). “Small-Scale Structures in CYOA”. Emily Short’s Interactive Storytelling (November 5). https://emshort.blog/2016/11/05/small-scale-structures-in-cyoa/.

Sierra. (1995). Phantasmagoria [Microsoft Windows]. Digital game directed by Roberta Williams, published by Sierra.

Sierra On-line. (1996). The Beast Within: A Gabriel Knight Mystery [Microsoft Windows]. Digital game directed by Jane Jensen, published by Sierra On-line.

Slade, David. (2018). Black Mirror: Bandersnatch. [Web Browser]. Interactive movie directed by David Slade, published by Netflix.

Smith, G., Anderson, R., Kopleck, B., Lindblad, Z., Scott, L., Wardell, A., Whitehead, J. & Mateas, M. (2011). “Situating Quests: Design Patterns for Quest and Level Design in Role Playing Games.” Presented at the 4th International Conference on Interactive Digital Storytelling (ICIDS).

Splendy Interactive and Wales Interactive. (2016). The Bunker [Microsoft Windows]. Digital game directed by Allan Plenderleith, published by Green Man Gaming.

Surman, D. (2007). “Pleasure, spectacle and reward in Capcom’s street fighter series”. In B. Atkins & T. Krzywinska (Eds.), Videogame, player, text. Manchester, England:Manchester University Press.

Take-Two Interactive. (1996). Ripper [Microsoft Windows]. Digital game published by Game Tek.

Therrien, C. (2017). “From Video Games to Virtual Reality (and Back). Introducing HACS (Historical-Analytical Comparative System) for the Documentation of Experiential Configurations in Gaming History”, DiGRA conference 2017. http://www.digra.org/digital-library/publications/from-video-games-to-virtual-reality-and-back-introducing-hacs-historical-analytical-comparative-system-for-the-documentation-of-experiential-configurations-in-gaming-history/.

Therrien, C. Lefebvre, I.and Ray, J.-C. (2019). “Toward a Visualization of Video Game Cultural History: Grasping the French Touch”. Games and Culture, online first (October 24), https://journals.sagepub.com/doi/abs/10.1177/1555412019873469.

Trilobyte. (1993). The 7th Guest [DOS]. Digital game directed by Graeme J. Devine, published by Virgin Interactive.

Velhal, R. (2019). “Transforming Hollywood from Pixels to Voxels, Volumetric Video Drives...” Intel Game Development Press Release, September 13. https://www.intel.com/content/www/us/en/develop/articles/transforming-hollywood-from-pixels-to-voxels-volumetric-video-drives-a-big-opportunity-for.html.

Zagal, J. & Bruckman, A. (2008). “The Game Ontology Project: Supporting Learning While Contributing Authentically to Game Studies”. In Kanselaar, G., Jonker, V., Kirschner, P. A., & Prins, F. J. (Eds.), International Perspectives in the Learning Sciences: Cre8ing a learning world. Proceedings of the Eighth International Conference for the Learning Sciences -- ICLS 2008, Volumes 2 (pp. 499-506). Utrecht, The Netherlands: International Society of the Learning Sciences.

Appendix A : Interactive Figures

Categories of action evoked by game mechanics and challenges integrated into the gameworld.

Appendix B : Manipulation Interfaces

Hardware and/or virtual devices through which the player performs actual manipulations (i.e. Player actions) to produce depicted virtual actions (i.e. Virtual actions), organised within a spectrum going from the maximum similarity between Player action and Virtual action (Motor Isomorphy) to minimal similarity (Symbolic Manipulation).

Figure 5. Click to expand.

Corporeal: Virtual actions are implemented through capturing body movement. Ex: performing a neutralization mechanic with a Wiimote, Kinect or PlayStation Move.

Techno-mimetic: Virtual actions are implemented through a hardware device that imitates a specific technology or tool. Ex: a steering wheel shaped controller.

Generic: Virtual actions are implemented through a non-evocative manipulation of a videogame control device. Ex: navigating using a directional pad.

Screen-Augmented: Virtual actions are implemented through visually encoded elements. Ex: using a WIMP interface (Windows, Icons, Menus, Pointing device) to select Virtual actions.

Verbal transcription: Virtual actions are implemented by writing in a text parser with a hardware keyboard or a virtual keyboard. Ex: writing short sentences in a text-based adventure game.

Appendix C : Action mapping

Type of mapping between Player actions and Virtual actions, organized from the most isomorphic in terms of motor activation to the most symbolic.

Figure 6. Click to expand.

Symbiotic: Player action = Virtual action. Ex: turning a hardware steering wheel to turn a virtual car.

Metonymic: Player action = miniature version of Virtual action. Ex: manipulating a virtual object with a mouse.

Synchronic: Player action is linked with Virtual action in terms of duration, but without any mimetic aspect. Ex: pressing a button to jump.

Cumulative: A sequence of Player actions triggers a Virtual action. Ex: a combination of player actions launches a special move in a fighting game.

Punctual: Player action triggers Virtual action, which is subsequently performed autonomously by the system. Ex: choosing a dialogue line and see it recited afterward.

Appendix D : Gameworld Feedback

Communication channels used to represent Virtual actions and consequences to the player, organised within a spectrum going from diegetic strategies (game state changes are presented through the virtual world) to signaletic strategies (game states are transcoded into abstract elements superimposed on the world) to ludic strategies (game states are transcoded to communicate their meaning in the game system)

Figure 7. Click to expand.

Appendix E: Modes of Performance

Figure 8. Click to expand.

Actualization: Short-term action planning with simple requirements in terms of skillset or challenges.

Execution: Quick reactions to perceptual stimulations in a gameworld with largely predetermined game states (Short-term planning, sensorimotor skillsets, static challenge).

Tactical coordination: Constant evaluation and implementation of concurring possible courses of action in a gameworld which integrates variable elements (Medium-term planning, sensorimotor and imaginative skillsets, dynamic challenge).

Resolution: Elaboration of plans in order to circumvent complicated obstacles in a gameworld with largely predetermined game states (Medium-term planning, sensorimotor and imaginative skillsets, dynamic challenge).

Strategy: Elaboration of plans in order to circumvent complicated obstacles in a gameworld which integrates variable elements (Long-term planning, imaginative skillsets, dynamic challenge).

Appendix F: Full corpus

3DO Studio. (1995). Killing Time [3DO]. Digital game directed by Larry Reed and Al Tofanelli, published by The 3DO Company.

3DO Studio. (1995). Zhadnost: The People's Party [3DO]. Digital game directed by Jim Eisenstein, published by 3DO Studio.

3Vision Interactive Picture. (1994). National Lampoon's Blind Date [Macintosh]. Digital game directed by Michael Berns, published by Trimark Interactive.

3Vision Games. (1996). FoxHunt [Playstation]. Digital game directed by Michael Berns, published by Capcom.

7th Level. (1994). Monty Python's Complete Waste of Time [Microsoft Windows]. Digital Game directed by Robert Tercek, published by 7th Level.

Access. (1989). Mean Streets [Microsoft Windows] Digital. game directed by Bruce Carver, published by Access.

Access. (1991). Martian Memorandum [Microsoft Windows]. Digital game directed by Dougles Vandergrift, published by Access.

Access. (1994). Under a Killing Moon [Microsoft Windows]. Digital game directed by Aaron Conners and Chris Jones, published by Access.

Activision. (1996). Spycraft: The Great Game [Microsoft Windows]. Digital game directed by Henry Yu, published by Activision.

Advance Reality. (1997). A Fork in the Tale [Microsoft Windows]. Digital game directed by Rob Lay, published by Any River Entertainment.

American Laser Games. (1990). Mad Dog McCree [Arcade]. Digital game directed by David Roberts, published by American Laser Games.

Atari. (1983). FireFox [Arcade]. Digital Game published by Atari.

Barlow, Sam. (2015). Her Story [Microsoft Windows]. Digital game directed by Sam Barlow, published by Sam Barlow.

Beam Software. (1995). The Dame Was Loaded [Microsoft Windows]. Digital game directed by Cameron Brown, published by Philips Interactive.

Black Dragon Publishing. (1997). Riana Rouge [Microsoft Windows]. Digital game directed by Gillian Bonner, published by Black Dragon Publishing.

Burst. (1996). Toonstruck [Microsoft Windows]. Digital game directed by Jennifer McWilliams, published by Virgin Interactive.

Cinemaware. (1992). It Came From the Desert [PC Engine CD]. Digital game directed by Randy Platt, published by Microsoft.

Code Monkeys. (1995). Surgical Strike [Sega CD]. Digital game directed by James Riley, published by Sega.

Coktel Vision. (1993). Lost in Time [Microsoft Windows]. Digital game directed by Muriel Tramis, published by Coktel Vision.

Coktel Vision. (1996). Urban Runner [Microsoft Windows]. Digital game directed by Arnaud Delrue and Roland Oskian, published by Sierra.

Compro Games. (1994). Cyclemania [Microsoft Windows]. Digital game directed by Ronnie Yaron, published by Accolade.

CPIO Multimedia. (1995). L'Affaire Morlov [CD-i]. Digital game directed by Didier Poiraud and Thierry Poiraud, published by Titus.

CRC Research Institute. (1993). I Will: The Story of London [LaserActive]. Digital game published by Pioneer Electronics.

Cryo. (1993). KGB [Microsoft Windows]. Digital game directed by Yves Lamoureux, published by Virgin Interactive.

Cryo. (1995). Hardline [Microsoft Windows]. Digital game directed by Dominic Matthieu and Greg Glachant, published by Virgin Interactive.

Cryo. (1997). Versailles 1685 [Microsoft Windows]. Digital game directed by Pierre Sauley and Jean-Michel Oullion, published by Canal+ Multimedia and Réunion des Musées Nationaux.

CTRL Movie. (2016). Late Shift [Microsoft Windows]. Digital game directed by Tobias Weber, published by CTRL Movie.

D'Avekki Studios. (2017). The Infectious Madness of Doctor Dekker [Microsoft Windows]. Digital game directed by Tim Cowles, published by D'Avekki Studios.

Digital Frontier. (2001). The Fear [Playstation 2]. Digital game published by Enix.

D'Avekki Studios. (2018). The Shapeshifting Detective [Microsoft Windows]. Digital game directed by Tim Cowles, published by Wales Interactive.

Delphine Software. (1991). Another World [Amiga]. Digital game directed by Eric Chahi, published by U.S. Gold.

Delta 4 Interactive. (1991). Psycho Killer [CDTV]. Digital game directed by Sean Koppans, published by On-Line Entertainment.

DigiFX Interactive. (1996). Harvester [Microsoft Windows]. Digital game directed by Gilbert P. Austin, published by Virgin Interactive.

Digital Pictures. (1992). Night Trap [Sega CD]. Digital game directed by James Riley, published by Sega.

Digital Pictures. (1992). Sewer shark [Sega CD]. Digital game directed by John Dykstra, published by Sony Imagesoft.

Digital Pictures. (1993). Double Switch [Sega CD]. Digital game directed by Mary Lambert, published by Sega.

Digital Pictures. (1993). Ground Zero Texas [Sega CD]. Digital game directed by Dwight H. Little, published by Sony Imagesoft.

Digital Pictures. (1994). Corpse Killer [Sega CD]. Digital game directed by John Lafia, published by Digital Pictures.

Digital Pictures. (1994). Kids on site [Sega CD]. Digital game directed by Heidi Hoffman, published by Digital Pictures.

Digital Pictures. (1994). Prize fighter [Sega CD]. Digital game directed by Ron Stein, published by Sega.

Digital Pictures. (1994). Supreme Warrior [Sega CD]. Digital game directed by Guy Norris, published by Digital Pictures.

Digital Pictures. (1996). QuarterbackAttack [Sega Saturn]. Digital game published by Digital Pictures.

Drew Pictures. (1992). Iron Helix [Microsoft Windows]. Digital game directed by Drew Huffman, published by Spectrum Holobyte.

Hypnotix. (1997). Soldier Boyz [Microsoft Windows]. Digital game directed by Louis Mourneau, published by Dream Catcher.

Dynamix. (1989). David Wolf Secret Agent [Microsoft Windows]. Digital game directed by John Burton, published by Dynamix.

Dynamix. (1990). Rise of the Dragon [Microsoft Windows]. Digital game directed by Jeffrey Runnell, published by Sierra.

Eko Software. (2018). #WarGames [Microsoft Windows] Digital game directed by Sam Barlow, published by Eko Software.

Electric Dreams. (1995). Angel Devoid: Face of the Enemy [Microsoft Windows]. Digital game directed by Anthony Gomez, published by Mindscape.

Moloto Productions. (2002). Zelenhgorm: The Land of the Blue Moon Episode 1: The Great Ship [Microsoft Windows]. Digital game directed by Nils-Petter Ekwall, published by Federation X Media Group.

Freestyle Games. (2014). Guitar Hero Live [Xbox One]. Digital game directed by James Norris, published by Activision.

Epstein, Hannah. (2011). The Immoral Ms.Conduct [Microsoft Windows]. Digital game directed by Hannah Epstein, published by Hannah Epstein.

Her Interactive. (1995). McKenzie & Co [Microsoft Windows]. Digital game directed by Jeffrey Baker, published by American Laser Games.

Hyperbole. (1993). Quantum Gate: The Saga Begins [Microsoft Windows]. Digital game directed by Greg Roach, published by Media Vision.

Hyperbole. (1998). The X-Files Game [Microsoft Windows]. Digital game directed by Greg Roach, published by Fox Interactive.

ICOM Simulations. (1993). Dracula Unleashed [Sega CD]. Digital game directed by Mike Plant, published by Viacom New Media.

ICOM Simulations. (1991). Sherlock Holmes: Consulting Detective [CDTV]. Digital game directed by Kenneth Tarolla, published by ICOM Simulations.

Inscape. (1995). The Dark Eye [Macintosh]. Digital game directed by David Nichols, published by GT Interactive.

Amazing Media. (1996). Mummy: Tomb of the Pharaoh [Microsoft Windows]. Digital game directed by Keith Metzger, published by Interplay.

Interworks. (1995). SFPD Homicide / Case File: The Body in the Bay [Microsoft Windows]. Digital game directed by Julie Marsh, published by Grolier Interactive.

iXL Interactive Excellence. (1995). Snow Job [3DO]. Digital game directed by Michael Seitzman, published by Studio 3DO.

Lankhor. (1987). Mortville Manor [Amiga]. Digital game directed by Bernard Grelaud, published by Lankhor.

Lankhor. (1988). Maupiti Island [Amiga]. Digital game directed by Dominique Sablons, published by Lankhor.

Legacy. (2000). Emergency Room: Code Blue [Microsoft Windows]. Digital game directed by Andre Guttfreund, published by Legacy.

Legend. (1995). Mission Critical [Microsoft Windows]. Digital game directed by Michael Verdu, published by Legend.

Mass Media. (1995). Thunder in Paradise interactive [CD-i]. Digital game directed by Robert Weaver, published by Philips Interactive Media.

Mechadeus. (1995). The Daedalus Encounter [Microsoft Windows]. Digital game directed by Mark Giambruno, published by Virgin Interactive.

MPS Labs. (1992). Rex Nebular and the Cosmic Gender Bender [Microsoft Windows]. Digital game directed by Matt Gruson, published by MicroProse.

Mylstar Electronics. (1984). Us Vs. Them [Arcade]. Digital game published by Mylstar Electronics.

Nova Productions. (1994). Cops [Arcade]. Digital game published by Atari.

Philips P.O.V. Entertainment Group. (1994). Voyeur [CD-i]. Digital Game directed by Robert Weaver, published by Philips Interactive Media.

Presto. (1994).The Journeyman Project: Turbo! [Microsoft Windows]. Digital game directed by Richard A. Cook, published by Sanctuary Woods.

Private Moon Studio. (2006). Yoomurjak's Ring [Sega CD]. Digital game directed by Andras Komlos, published by Private Moon Studios.

Quarium. (1997). Blue Heat: The Case of the Cover Girl Murders [Microsoft Windows]. Digital game directed by Michael Schroeder, published by Orion.

Raw Thrills. (2004). Target: Terror [Arcade]. Digital game directed by Eugen Jarvis, published by Raw Thrills.

Revolution Software. (1996). Circle of Blood [Microsoft Windows]. Digital game directed by Charles Cecil, published by Virgin Interactive.

Sanctuary Woods. (1995). Ripley's Believe it or Not! The Riddle of Master Lu [Microsoft Windows]. Digital game directed by Lee Sheldon and François Robillard, published by U.S. Gold.

Sega. (1995). Fahrenheit [Sega CD]. Digital game directed by Raymond Elias, published by Sega.

Sega. (1994). Midnight Raiders [Sega CD]. Digital game directed by Sam Nicholson, published by Sega.

Sierra. (1995). Phantasmagoria [Microsoft Windows]. Digital game directed by Roberta Williams, published by Sierra.

Sierra. (1996). Phantasmagoria 2 [Microsoft Windows]. Digital game directed by Andy Hoyos, published by Sierra.

Sierra. (1995). The Beast Within: A Gabriel Knight Mystery [Microsoft Windows]. Digital game directed by Jane Jensen, published by Sierra.

Simon & Schuster. (1996). Star Trek: Borg [Microsoft Windows]. Digital game directed by James L. Conway, published by Simon & Schuster.

Some Interactive. (1996). The Crystal Skull [Microsoft Windows]. Digital game directed by Jeff Cretcher, published by Maxis.

South Peak Interactive. (1997). Temüjin [Microsoft Windows]. Digital game directed by Lee Sheldon, published by South Peak Interactive.

SpectreVision. (2018). Transference [Microsoft Windows]. Digital game directed by Benoit Richer, published by Ubisoft.

Splendy Games. (2016). The Bunker [Microsoft Windows]. Digital game directed by Allan Plenderleith, published by Green Man Gaming.

Star Maid Games. (2015). Cibele [Microsoft Windows]. Digital game directed by Nina Freeman, published by Star Maid Games.

Stargate Films. (1994). Bug Blasters: The Exterminators [Sega CD]. Digital game published by Stargate Films.

Sunstar. (1994). Arc of Doom [Macintosh]. Digital game directed by R. W. Moulton, published by Keystone Productions.

Take-Two Interactive. (1996). Ripper [Microsoft Windows]. Digital game published by GameTek.

The Neverhood Inc. (1996). The Neverhood [Microsoft Windows]. Digital game directed by Doug TenNapel and Mark Lorenzen, published by DreamWorks Interactive.

Trip Media. (1994). Burn: Cycle [Microsoft Windows]. Digital game directed by David Collier, published by Philips Interactive Media.

TSI. (1996). Noir: A Shadowy Thriller [Microsoft Windows]. Digital game directed by Jeff Blyth, published by Cyberdreams.

Tsunami Media. (1994). Flash Traffic: City of Angels [Microsoft Windows]. Digital game directed by Tony Markes, published by Time Warner Interactive.

Tsunami Media. (1995). Silent Steel [Microsoft Windows]. Digital game published by Tsunami Media.