Usability and accessibility in guitar-based rhythm games

Intro

When delving into discussions about user interfaces, the spotlight often falls on the non-diegetic elements provided by in-game visual interfaces, leaving meta, spatial, and diegetic elements somewhat in the shadows. Yet, a more significant aspect is frequently overlooked: the physical interface, specifically the input devices. This oversight is particularly noteworthy considering that the physical interface stands as the initial and primary component through which players interact with a game — or, put more precisely, assert their control over the entire gaming experience. This underscores the importance of game designers giving due attention to physical interfaces, subjecting them to the same scrutiny as other interface types when evaluating usability and accessibility.

This essay endeavors to analyze the usability and accessibility of physical interfaces, using guitar-based rhythm games as an example where these interfaces, or their absence, serve as core pillar of the gaming experience. Through an examination of four games, the first part of the essay strives to compare the sense of control afforded by different types of physical interfaces. In the second part, the focus narrows to one of these games, using its distinctive lack of an interface equivalent to its counterparts to identify and propose solutions for potential accessibility issues.

PDF version

Verses: about the games

Verse 1: Guitar Hero 3: Legends of Rock

Guitar Hero 3 stands as the third iteration in the renowned Guitar Hero series, a rhythm game that revolutionized the genre by introducing a guitar-shaped peripheral complete with color-coded buttons mimicking guitar frets, a switch for strumming, and even a whammy bar to tweak note pitches.

The visual user interface of the game showcases a top-down black rectangle, emulating the guitar neck, where color-coded notes “fall” towards the player. The objective is to press the correct button combination and strum at the right moment to strike the chords. It could be a single note, a combination, or even a prolonged note requiring the player to maintain button presses while manipulating the whammy bar for a distortion effect. Successfully striking the chords earns a score, ranging from “Bad” to “Perfect,” culminating in a final tally. If impressive enough, this score unlocks access to new songs or modes.

The score may be boosted through accumulating star-shaped notes and activating a mode where notes yield higher scores, or by avoiding failures and obtaining a high chain on the rock meter. On the flip side, repeated note failures result in a low score on the rock meter, potentially leading to the dreaded game over screen.

However, Guitar Hero’s triumph wasn’t solely due to its groundbreaking controller. The game’s success can be attributed to its accessible entry point, a well-paced difficulty curve in the career mode, and an extensive catalog of licensed songs that could be expanded through DLC.

Verse 2: Frets on Fire

In the wake of Guitar Hero’s well-deserved acclaim and popularity, it was only a matter of time before someone created an open-source alternative. Enter Frets on Fire, winner of the Assembly 2006 game development competition, which not only offers many of Guitar Hero’s fantastic features for free, but also grants players the ability to tweak numerous options (including mapping), play their own tracks, and install mods that add even more functionality to the game.

In its current version, Frets on Fire mirrors the gameplay of Guitar Hero, except for the lack of a career mode, a whammy bar, or a custom peripheral. Despite the game’s intended keyboard-as-guitar playstyle, the beauty lies in its customization, allowing players to use a diverse array of controllers, from console gamepads to the Guitar Hero and Rock Band custom guitars.

Verse 3: Rocksmith

The main critique leveled against Guitar Hero, particularly from real guitar enthusiasts, was its perceived lack of realism, giving players a misleading sense of virtuosity (Stuart, 2009) akin to becoming the next Jimi Hendrix. Recognizing the unmet needs of this yet-to-be-pleased audience, Ubisoft responded with the ultimate guitar simulator – Rocksmith.

What sets Rocksmith apart is its departure from custom, non-realistic controllers. Instead, players can plug in any electric guitar via USB. The use of a genuine guitar in a Guitar Hero-like setting enhances immersion but also introduces a steeper learning curve. With more frets to navigate and an additional dimension – the strings – players face a more intricate challenge.

Despite sharing a similar gameplay concept with other guitar-based rhythm games, Rocksmith brings a distinctive experience. Notably, the visual UI presents a wider black rectangle mimicking the guitar neck, scrolling left or right based on the player’s hand position. It also adds an horizontal bar at the bottom of the screen, providing a color-coded representation of the guitar strings. Striking a chord requires players to synchronize both frets and strings, aligning with the onscreen position and color.

Rocksmith also offers additional modes that add to its charm. It provides training sessions for real guitar techniques, jam sessions allowing players to set up a band with instruments responding to their guitar input, and Guitarcade, a mode that blends arcade games with advanced guitar technique learning.

Verse 4: Unplugged

While Ubisoft focused on real guitar enthusiasts with Rocksmith, Anotherway tapped into a different yet widespread practice among rock music fans – air guitar. And what could be better than air guitaring? Doing it in virtual reality, of course. In the Oculus Quest version, Unplugged keeps it simple: all you need is a VR headset and your hands.

In terms of gameplay, Unplugged incorporates many mechanics from Guitar Hero, such as star energy, and plays similarly to Rocksmith, involving both frets and strings. However, it simplifies the equation with a lower count of both, resulting in a more compact representation of the guitar neck. Players move one hand up and down an intangible guitar neck, adjusting fingers to match onscreen chords. The other hand strums an array of immaterial strings.

Despite featuring only a career mode, Unplugged distinguishes itself with a notably steep difficulty. Some players even find certain challenges so demanding that reaching the final set of songs remains elusive for many, adding an extra layer of challenge to the experience.

Chorus: the need for a sense of control

To recap the previous section, the four games under scrutiny employ four distinct types of controllers: a guitar-like peripheral, a keyboard, a real guitar, and nothing but the player’s hands. Despite their diverse nature, these physical interfaces share a common thread in facilitating interaction with a fairly similar gaming concept. Given the subjective nature of their quality and appropriateness, the pertinent question isn’t “which one is better?” but rather “how usable and accessible are they?”. Ultimately, guided by the assertion that “the goal of an interface is to make players feel in control of their experience” (Schell, 2014), the key query emerges: “do they provide the player with a sense of control?”

To assess this, the four games will be analyzed using both Glinert’s dimensions for measuring interface usability (2009) and Schell’s lenses regarding interfaces (2014). However, before delving into the examination of physical input devices, it is crucial to briefly explore potential issues that may arise from the rest of the interfaces. Addressing these early concerns is essential, as they could potentially be interlinked with any further issues uncovered later in the analysis. o prevent this part from becoming overly extensive, the focus will be solely on the active game scene – when the player is playing.

As all four games share a similar in-game appearance, examining the aesthetic dimension first feels apt. In this regard, there’s little to remark upon. Despite minor variations in visual effects or the use of supporting 3D models—bearing in mind their span across three different generations—all four games depict the guitar neck, frets, and strings in some fashion. The game scene is not metaphorical; rather, it offers an almost 1:1 representation of a real guitar. What adds an element of interest is that a substantial portion of the game scene exists beyond the spatial realm. Whether Guitar Hero 3 incorporates avatars and an occasional audience or not is inconsequential, as the game screen is not designed for them; they remain unaware of its existence. This unique aspect means that, unlike many other game genres, rhythm games, or at least the four under scrutiny, predominantly feature non-diegetic elements not only in their user interface but also within the game scene itself.

While crafting a visual replica of the instrument being played constitutes an apparent push towards immersion, which undoubtedly stands as the main pillar of these games, it concurrently aims to enhance the presence cognitive status. Nevertheless, this is likely just one component in a more intricate mechanism. Equally crucial is what this facsimile communicates to the player, as it actively contributes to fostering the flow cognitive status. This is where the remaining dimensions from Glinert and most of Schell’s lenses come into play. Do players require constant information from it? Are players receiving sufficient feedback during interactions? Does it maintain intuitiveness and a robust feel, especially in high-stress moments?

In the examination of all four games through the lens of channels and dimensions, the UI emerges as a robust element. Crucial information, such as the sequence of notes and their positions on the guitar’s neck —the details players need at all times—, dominates the screen. It occupies most of the screen space from top to bottom and at least 80% of the horizontal axis. Supplementary information, like the score or the rock meter, is deliberately placed at the periphery of the screen, in a less prominent plane. Any other information is hidden under menus on other layers, like the pause menu.

Shifting focus to the lenses of feedback and juiciness, all four games excel in providing players with continuous feedback. Visual effects and sounds accompany players when they strike a chord, maintain pressure on a long note correctly, or when they fail and break their chain. The games even add reinforcers to the secondary areas of the screen, like the “bad” and “good” indicators.

Considering the limited set of play modes and easily distinguishable actions in all four games, it is reasonable to assert that they also excel in the simplicity and learnability dimensions, closely aligned with Schell’s lens of transparency. However, as mentioned earlier, an important distinction lies in their complexity, dependent on how many parts of the guitar they emulate. A higher degree of complexity may compromise simplicity, learnability, and transparency. Rocksmith exemplifies this, with part of the active game scene hidden and only revealed through scrolling left or right. Additionally, it introduces extra colored layers to the guitar neck replica to represent the strings that need pressing at each moment. Were it not for accommodating novice players with lower requirements, the entry barrier would be insurmountable.

Regardless of the latter, all four games appear to perform admirably in terms of the usability of their visual interfaces. However, the pivotal question remains: how do their physical interfaces, not only the primary point of distinction among them but also the unique selling point compared to other rhythm games in the market, measure up?

As previously mentioned, the cornerstone for all four games undeniably lies in immersion. According to Schell’s lens of control, “meaningful control is essential for immersive interactivity.” Now, it is time to address the initial question posed at the start of this section: “Do these physical interfaces (or their lack) provide players with a sense of control?”

Rocksmith, with its allowance for the use of real guitars, emerges as the game offering the pinnacle of expressive power. Mastering Rocksmith songs not only bestows a complete sense of control but also imparts the satisfaction of having learned to play a real instrument.

Guitar Hero’s custom input device enables players to interact with the game’s limited set of actions while fostering the sensation of holding a real guitar. The need to press the right set of buttons, strum the switch simultaneously, and move freely around the room contributes to a feeling of control over the game. However, the awareness that the controller is merely a replica, falling short of a real guitar, might diminish the perceived power when compared to actual guitar players.

Frets on Fire developers are not kidding when they say that the correct way of playing is by holding the keyboard as shown in Figure X, utilizing intentional keyboard mapping with keys like F1, F2, F3, F4 for frets and the Enter key for strumming. While not as immersive as holding a real guitar, this serves as a metaphor, helping players feel in control of their play. Nevertheless, the sense of control may diminish for players using a laptop or a similar device without a detachable or light keyboard.

On the other end of the spectrum is Unplugged. The absence of a physical input device caters to air guitar aficionados and strives for immersion through VR goggles. Despite the freedom of movement afforded by VR, the lack of a tangible input device can undermine the player’s sense of power. The constant reminder that they’re not playing a real instrument limits the perceived control they have over the game.

However, the assessment of control extends beyond the subjective feeling of power over the game; it encompasses various usability aspects that ultimately contribute to the sense of control.

For instance, adhering to Schell’s lens of transparency, “the ideal interface becomes invisible to the player, letting the player’s imagination be completely immersed in the game world.” While the physical devices of the first three games may achieve transparency over time through use, Unplugged compels players to continuously look at their hands. This is not solely due to the absence of a tangible reference element but also influenced by the characteristics of VR goggles, such as the camera’s location. Even among physical interfaces, the sense of control may fluctuate based on their learnability and simplicity. Gaining control of the 5-button Guitar Hero peripheral is likely faster and easier than mastering the use of specific keys on an entire keyboard or navigating the 6-strings, 21-frets schema of electric guitars.

A similar argument holds for feedback. Schell asserts that “experiences without feedback are frustrating and confusing,” leading to a weaker sense of control over the game. None of the physical devices presented in these games offer feedback beyond their mechanical characteristics—the sensation of pressing a button, a key, or the whammy bar, the vibration of strings when strummed, etc. Unplugged presents a unique case, lacking any physical device and, therefore, providing no tangible reference. However, since there is still constant feedback from other interfaces, this may not pose a severe usability issue. As explored in the next section, it can, however, represent a significant accessibility challenge.

Bridge: music is for everyone

Rhythm games are typically characterized by their fast-paced nature, demanding exceptional reflexes and substantial eye-body coordination. Introducing additional devices, such as the Dance Dance Revolution pad or the Guitar Hero guitar, further amplifies the requirement for dexterity. These demands pose inherent limitations for individuals with various disabilities, including visual, cognitive and physical. However, what if a game not only necessitates an entirely different tool with its unique challenges but also lacks physical support for input?

This scenario is exemplified in Unplugged, a game that not only combines the challenges inherent in rhythm and VR games but also lacks any physical input device whatsoever. However, it’s important to note that VR games themselves are not without their issues. They grapple with accessibility challenges affecting individuals with a range of disabilities, from visual impairments to mobility impairments. While there have been strides toward a more equitable situation, such as Samsung’s Relúmĭno or Microsoft’s Canetroller (Cutrell, 2020) designed for people with low vision, there’s still a considerable journey ahead.

Given the myriad limitations posed by both VR and rhythm games for individuals with disabilities, what additional challenges arise from the absence of a physical input device? It’s not that this absence exacerbates existing issues; rather, it fails to contribute to a comprehensive solution.

Starting with individuals with visual impairments, particularly those who are blind, there are limited options that can enhance their inclusion in a game like Unplugged. While sound cues providing warnings for specific button presses may prove beneficial in other game genres, the rapid pace and musical nature of Unplugged might render this option less usable. A more effective alternative could be a physical input device with haptic feedback. An example is Blind Hero (Yuan & Folmer, 2008), a derivative of Frets on Fire, which introduces a pair of gloves translating visual stimuli into haptic feedback. Although compromises in certain gameplay elements may be necessary, it represents a noteworthy effort toward inclusion.

Other individuals with visual impairments who may encounter issues with Unplugged include colorblind individuals and those with low vision. Colorblind players might find it necessary to change the theme used by the visual interface, but their challenges are not directly linked to the absence of a physical device. Similarly, those with low vision do not face challenges directly tied to the lack of a physical device. However, in their cases, they could potentially benefit from physical devices that help them address other issues arising from the VR nature of the game, such as the previously mentioned Canetroller.

Individuals with cognitive and mobility disabilities may encounter challenges in following the visual and auditory cues and pressing buttons in a timely manner, particularly if they lack a physical device that serves as a reference for their hands. Specifically, individuals with cognitive disabilities, including conditions such as autism or attention-related disorders, may find it even more challenging to constantly look at their hands, exacerbating existing difficulties stemming from the use of a VR headset.

Certain games, such as Frets on Fire or Rocksmith, have demonstrated a commitment to inclusivity by offering extensive customization options. This approach has empowered individuals with mobility disorders to create and use custom devices tailored to their specific needs, like the LOLO (Stereo Stickman, 2019), showcasing how adaptability in game design can enhance accessibility for a diverse player base.

But what may be a drawback for one population group could serve as a benefit for another. Individuals with physical disabilities can significantly benefit from the elimination of the limitations associated with a physical device. In Unplugged, as it tracks hands based on the calibrated position at the start of each song, players are relieved from the need to adopt a potentially uncomfortable or even painful position.

On a different note, various issues can also impact individuals with no known disabilities, but the majority of these concerns are associated with the use of VR (seizures, vertigo, etc.) and do not have any correlation with the absence of a physical interface, nor would they necessarily benefit from one.

Outro: conclusions

Crafting a custom physical controller is undoubtedly beneficial when it enhances game immersion or empowers players to feel more in control. However, these custom controllers are not exempt from the constraints faced by other input devices. Moreover, due to their rarity, they lack the advantage of “familiarity” that common controllers enjoy, hindering developers from adhering to established industry standards. Therefore, it becomes crucial to carefully consider any usability and accessibility issues that could cast a shadow on an otherwise groundbreaking idea.

As illustrated above, this is no straightforward task. Existing guitar-based rhythm games grapple with numerous challenges concerning the usability, particularly the accessibility, of their visual and physical interfaces. These issues might have proven fatal if not for their factor of novelty. While guitar-based games may not enjoy the same popularity as in years past, any developer aiming to create a new one must navigate the creation of a custom physical interface while learning from the mistakes of the past.

Encore: Beatstar

Even as the conclusions are drawn and the curtains close, there’s always time for one more song. With the gaming landscape shifting towards portable devices (Wijman, 2023), it was inevitable for guitar-based rhythm games to venture into the portable and mobile gaming sphere.

Guitar Hero, ever the pioneer, embarked on this journey, debuting Guitar Hero III Mobile in 2007 for BREW, J2ME, Windows Phone, and Blackberry devices, which, at that time, lacked touch screens. Instead, the game was played using the physical keyboard. Not content with this, just a year later, Vicarious Visions released Guitar Hero On Tour for the Nintendo DS. This installment not only made use of the device’s touch-enabled double screen but also introduced a custom peripheral connecting to the GameBoy port that featured the iconic Guitar Hero color-coded buttons simulating guitar frets, emphasizing the immersion highlighted throughout this text.

From then on, history was made, with numerous clones longing for a slice of the Guitar Hero cake. Some succeeded admirably.

Enter Beatstar, an Android rhythm game launched in 2021 that draws inspiration from Guitar Hero and its successors but shifts the focus to the beat rather than the instruments. Accumulating over 10 million downloads on Google’s Play Store alone, with a user score of 4.5 out of 5, Beatstar offers a user interface reminiscent of the examples discussed above. A vertical surface where notes “fall” towards the player, who simply needs to tap them at the right moment. The game includes single notes, long notes, chain multipliers, and even a version of star power.

Yet, Beatstar deviates from the familiar feel of its counterparts. The game was designed under the lens of primality, opting for intuitiveness and primality over the immersion emphasized by its predecessors. It simplifies many Guitar Hero elements, making it playable with just two thumbs. Onscreen, only three “frets” are represented, and there are never more than two notes to be pressed simultaneously.

While space is limited here, Beatstar could easily warrant its own essay, given the myriad issues it grapples with in both usability and accessibility. Much like Unplugged combined challenges from both VR and rhythm games, Beatstar confronts issues arising from the intersection of mobile games and rhythm games.

Bonus track: References

Side A: bibliography

Cutrell, E. (2020, April 7). Bringing virtual reality to people who are blind with an immersive sensory-based system. Microsoft Research Blog. https://www.microsoft.com/en-us/research/blog/bringing-virtual-reality-to-people-who-are-blind-with-an-immersive-sensory-based-system/

Desconsolados. (2021, October 21). [REVIEW] Unplugged | El Guitar Hero de la VR ya está aquí [Video]. YouTube. https://www.youtube.com/watch?v=RnPdEfZTvbY

Folmer, E. (2007, May 17). Designing Usable and Accessible Games with Interaction Design Patterns. Game Developer. https://www.gamedeveloper.com/design/designing-usable-and-accessible-games-with-interaction-design-patterns

Gam3SetGo. (2021, January 20). Rocksmith 2014 Edition – Remastered (PC) | Review en español [Video]. YouTube. https://www.youtube.com/watch?v=NcLutc87r-8

Glinert, E. (2009, August 19). Upping Your Game’s Usability. Game Developer. https://www.gamedeveloper.com/design/upping-your-game-s-usability

Heilemann, F.; Zimmermann, G.; Münster, P. (2021). Accessibility Guidelines for VR Games – A Comparison and Synthesis of a Comprehensive Set. Frontiers in Virtual Reality, vol. 2. https://www.frontiersin.org/articles/10.3389/frvir.2021.697504/full

Schell, J. (2014). The Art of Game Design: A Book of Lenses, Second Edition. A K Peters. https://doi.org/10.1201/b17723

Stereo Stickman. (2019, October 29). The Lolo: An adaptive, table top acoustic/electric guitar for the disabled. Stereo Stickman. https://stereostickman.com/the-lolo-an-adaptive-table-top-acoustic-electric-guitar-for-the-disabled/

Stuart, K. (2019, September 16). Critics of Rock Band and Guitar Hero are out of tune with the real world. Game Theory. https://www.theguardian.com/technology/2009/sep/16/rock-band-game-theory-column

Ugolik, K. (2020, January 29). Virtual Reality Has an Accessibility Problem. Scientific American Blogs. https://blogs.scientificamerican.com/voices/virtual-reality-has-an-accessibility-problem/

UOC. (n.d.). Design Toolkit. http://design-toolkit.recursos.uoc.edu/es/contenidos/contenidos/contenidos.html

Wijman, T. (2023, August 8). New free report: Explore the global games market in 2023. Newzoo. https://newzoo.com/resources/blog/explore-the-global-games-market-in-2023

Yuan, B.; Folmer, E. (2008). Blind Hero: Enabling Guitar Hero for the Visually Impaired. Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2008. https://dl.acm.org/doi/10.1145/1414471.1414503

Side B: ludography

Anotherway. (2021). Unplugged [Oculus Quest]. Vertigo Games.

Beatstar [Android]. (2021). Space Ape Games.

Frets on Fire [Windows]. (2006). Unreal Voodoo.

Neversoft. (2007). Guitar Hero 3: Legends of Rock [PlayStation 3]. Activision.

Rocksmith [Windows]. (2011). Ubisoft.