Meta-Game Studies
by Espen AarsethQuantitatively, as far as research fields go, game studies is a success. We did it. New conferences, teaching programs, even departments and institutes spring up and thrive. In Sweden, the University of Skövde hosts no less than five bachelor and two master game programs (design, writing, graphics, sound, programming, media and narrative aestetics, and serious games) with over five hundred students. Last year, China held its first game studies conference in Ningbo, and we also saw the first Central and Eastern Europe Game Studies conference in Brno, both with follow up conferences this year in Beijing and Krakow, respectively. The number of publication venues is no longer limited to a handful of journals (not that it ever was); in a brilliant and highly useful paper at this year’s Foundations of Digital Games conference, Melcer et al. (2015) analysed 8207 game research articles from the last 15 years and identified no less than 48 “core venues” of publication. In Denmark alone, in the years 1984-2014, there have been 1235 game research publications, produced by around 200 people, with an annual average of around 150 publ. in the last five years (Aarseth and Hammar 2015). In addition, two related trends can now be observed: game research is increasingly accepted in traditional research communities such as HCI and media studies, and specialized game studies venues have appeared: Music, Queerness, Philosophy, History, and Literary Theory, to name but a few. From the heyday of DiGRA in 2003, when the Utrecht conference had over 500 participants, there seems to have taken place an academic diaspora, into smaller and more glocalized foci and events. And, at least for the non-disciplinarians among us, this is a good thing. We now have more boats to rock.
So the field, as seen from 2015, appears to be alive, well, and growing. However, the critical reader will already have noticed an ambiguity in this text: Are we discussing game research, or game studies? Or perhaps it does not matter? But surely, they are not the same? It is an open question whether educational/learning approaches and applications should be considered its own field (or part of education, or psychology) rather than as a subfield of game studies, and most technical and psychological researchers would probably not feel comfortable, or even consider, using “game studies” as the label for their work, though there are exceptions. However, this is not important. Game studies is, among other things, a nexus where fruitful connections can be made between all of these approaches, and many more. It certainly never was, and never will be, a discipline or -ology, that is, a coherent domain defined by a single set of methods, empirical objects, and research questions and motivations. Instead, it should be viewed as a fuzzy set, equally hard to define as its mother object, games.
Yes, says the critical - and now slightly bored - reader, so what else is new? And what is the problem?
Our field is very young, not just in history, but also in average age of its professional participants. It is still dominated by PhD students and untenured faculty, not to mention master students. From the start around 1980, pioneering PhD students such as Thomas Malone (1980) and Mary Ann Buckles (1985) wrote the first substantial, high-quality studies, and later, when the field came together organizationally, PhD students organized the conferences. This was a necessity, but at a cost. Young, enthusiastic students have the best energy to drive a field forward. What they don’t have yet is the academic skill-set. A field without solid quality standards is not a field in good health. If game studies, no longer the cute baby it once may have been, is to succeed qualitatively and not just as a popular but ill-respected inter-discipline, the teenager needs to grow up and learn how to be self-critical. If we don’t instill proper standards for academic quality control, if we don’t respect the most vital traffic rules of academia, we will never gain the respect of other fields, and we will lose good people we cannot afford to lose. There is a limit to how many times a scholar will submit papers to a venue where the peer reviews make no sense, and that number is closer to one than two.
Proper peer review is not rocket science. And yet, it must be learned, and its ethics understood. Here are some easy rules that, if followed, will make an enormous difference both to the individual and to the field as a whole. This is an elementary lesson, but in my recent experience, significant parts of the field still need to learn it.
1. In an inter-discipline, paradoxically, it is vital that disciplinary competences are observed. Don’t put all the papers in one or two big categories (e.g. “theory”), but organize tracks that make disciplinary sense, such as design, humanities, social sciences, etc. Also, use track chairs that know the subfield and the competences and specialties of the chosen reviewers. These can also assign papers in ways that avoid conflicts of interest.
2. Use more than two reviewers per paper. Three is good, four is better. With four, one can fail and you still don’t need to find a new one to resolve a tie. With only two, one reviewer can effectively torpedo a paper (unless the other gives a top score, and that seldom happens), and that goes against the very core of peer review principles: one person should never have decisive power. Yet, this year I witnessed the use of only one review as basis for a rejection. This is the sort of thing that gives the venue and indirectly, the field, a bad name.
3. Some venues have experimented with the use of “bidding”, which means the reviewers can take a look at paper titles and suggest themselves as interested or disinterested. This is an unethical practice, as it lends itself to several forms of abuse: reviewers may think they can review in an area where they lack competence, reviewers may want to help their friends, or they want to crush topics and approaches conflicting with their own. A system that allows this is unethical. Combined with the lack of proper tracks and track chairs and only using two reviewers, it is diabolical.
4. Always meta-review. The track chair should not send out reviews that make no sense or clearly do not contain relevant arguments. Have you ever received an empty review? So have I. Worse, I once saw a review that complained that the paper, while otherwise good, was too critical! A track chair that does not catch such nonsense is worse than useless. There will always be cases where the submitters will disagree with the reviewers, but this is often unavoidable, and not a problem in itself, as long as it is possible to follow the reviewer’s reasoning, and learn from it.
5. The venue needs to have a policy for how to deal with conflicts of interest. Can the program chair or editor submit a paper? Should they? If so, how should this be handled? When is a reviewer too close to an author? How is this policed?
6. Above all, when it comes to conference series, don’t reinvent the wheel every year. Document the process, set up routines and guidelines and keep a collective memory in the form of track chairs and a pool of vetted reviewers. It is probably a good idea to keep local organizers and program committee separate, connected trough a very simple interface: the locals specify time slots and the number of tracks, and the program committee fills them. End of story.
There are many more finesses to the art of peer review, and I am not implying that the present journal is perfect or even near perfection in this respect. (Personally, I spend both too much and too little time on peer review.) But following these simple guidelines will insure that the field is taken seriously, and that even rejections can contribute to improving the quality of the field. Game Studies has a 85-90% rejection rate, and yet we have received extremely few complaints about review quality in the fifteen years we have operated. Instead, we are often thanked for useful feedback by those rejected. And as editor I am frequently in awe and extremely proud of the excellent and diligent work of our reviewers, who will write long and thoughtful feedback with no rewards or recognition for their time.
If you do experience what you perceive as unfair peer review treatment, from this journal or from any other publication venue, I encourage you to contact those responsible for the peer review and put the issue forward. No one deserves to suffer an unfair peer review, least of all the field of game studies, and pointing out the problem is a good start in fixing it. The field may have come a long way and has already achieved much, but we still have a lot of leveling up to do.
References
Aarseth, E. and Hammar, E. (2015). “Game Research Concerns in Denmark”, paper presented at the workshop Cultures Of Video Game Concerns, Bochum University, January 2015.
Buckles, M. A. (1985). Interactive Fiction: The Computer Storygame Adventure. PhD dissertation, UCSD.
Malone, T. W. (1980). ”What Makes Things Fun to Learn? A Study of In- trinsically Motivating Computer Games” Xerox Palo Alto Research Center Technical Report No. CIS-7 (SSL-80-11).
Melcer, E., Nguyen, T. H. D., Chen, Z., Canossa, A., El-Nasr, M. S., & Isbister, K. (2015). “Games Research Today: Analyzing the Academic Landscape 2000-2014." Foundations of Digital Games 2015.