Video Demo
WWISE: Adaptive Music System
The basis of this project was to create an adaptive music score for a science fiction turn-based strategy game. Although the music system will not be delivered to a developer, this project was created under this pretence. This meant the project was organised for testing upon completion. The score is generated in the digital audio workstation Cubase, while the music system was put together by middleware software WWISE. The project will cover the various techniques used to implement adaptive music, including Parallel, Transitional, Ornamental, RTPC, Switches and more. The research was carried out with a small sample size, looking into other science fiction turn-based strategy games, with a mindset to see how music is currently implemented in this style of game. After research completion, it was shown that adaptive music never implemented for core gameplay. Upon completion of the music system and its testing, it was demonstrated that adaptive music could work within a science fiction turn-based strategy game, providing ludic information to the player, like enemy direction and combat performance.
Introduction
This report will discuss the process of creating, demonstrating and delivering an interactive score by utilising middleware software, WWISE. It will briefly touch upon the music composition process, but will primarily focus on the implementation of music, into WWISE.
Project Background
The video game XCOM Enemy Unknown (Firaxis Games, 2012), implements transitional linear music, with the music transitioning into new tracks depending on gameplay, for example; the player is listening to exploration music when exploring the terrain, which transitions into combat music once combat has been initiated. Rather than using a linear approach, the music could have been implemented to inform players of their surroundings, provide feedback to player/enemy movements, and inform the player of their performance during combat situations. For this reason, XCOM Enemy Unknown was selected to preview how an adaptive music score could be implemented and showcased to developers in turn based strategy games.
Content
This report will begin by detailing the process of arranging the music composition and preparing this music to be adaptive, using the DAW (Digital Audio Workstation) Cubase. The report will discuss naming conventions to aid in the organisation of tracks and stems, while also looking into optimal sample rates for working with WWISE. The report will explain the method of which the music will be implemented into WWISE. Making use of music playlists, segments, sequences, stingers, events and RTPC (Real-time Parameter Control). While also justifying the implementation decisions, by referencing work from Michael Sweet’s Writing Interactive Music For Video Games, Karen Collins Game Sound, Clint Bajakian on his GDC conference presentations, and other research relevant to adaptive music.
This report will conclude these findings into possible strategies and approaches for music arrangement, implementation in turn-based games, using music to provide feedback to the player on specific gameplay elements.
Adaptive Music & Turn Based Strategy Games
In video games, synchronisation can be an issue, the pace of the game is usually dictated by the player, unlike film, where the speed is set, and music linear (Sweet, 2015). As Sweet (2015) notes in video games, linear music is unable to synchronise and follow changes to gameplay, and for this reason, composers turn to adaptive music, which provides many opportunities to resolve player pacing, flow, synchronisation and musical repetition flaws.
From the analysis of the games illustrated below it appears that science fiction turn-based strategy games commonly use either linear, transitional, or a combination of both, as the primary form of music implementation;
XCOM: Enemy Unknown (Firaxis Games, 2012) Implements a transitional linear approach, also known as Horizontal Resequencing (Phillips, 2014), this method can vary between the extreme and non-extreme. The extreme form consists of multiple branching phrases that quickly adapt to gameplay, examples of this can be heard in the video game Monkey Island 2 (LucasArts, 2017). The non-extreme method works by implementing simple crossfades from any type of track to another kind of track. The crossfading method has been implemented into XCOM Enemy Unknown.
Phantom Doctrine (CreativeForge Games, 2018) Appears to implement a parallel approach, also known as vertical remixing (Phillips, 2014). Percussion elements can be heard ducking for cutscenes; however, it does not appear to serve any feedback functionality to the player. However, there is an exception, in the management system where players can organise their team for missions, another parallel approach was taken, and in this instance, when a player activates the fast forward time, the system introduces a hi-hat percussion layer, signalling that time is progressing forward.
Attack Of The Earthlings (Team Junkfish (2018) Implements either a linear looping track or possibly a transitional approach (horizontal resequencing), with music transitioning within a playlist, creating new arrangements to prevent repetition. Winifred Phillips details how linear music loops can work as a playlist of sequenced tracks, especially if the track composed is at least five minutes or longer (Phillips, 2014), this will explain why it is difficult to determine which method is being used in Attack Of The Earthlings. However, this music does not provide any type of musical feedback to gameplay, with only one style of music for both explore and combat states.
Xenonauts (Goldhawk Interactive, 2014) Uses a similar approach as Attack Of The Earthlings, in it has only one style of music, for both gameplay states explore and combat. It provides no musical feedback that can help or guide players during gameplay.
The above information shows it is uncommon to hear adaptive music in turn based science fiction games. It could be argued the popularity, awards received, and success of XCOM; Enemy Unknown (Game Critics Awards, 2012), has brought with it a surge of science fiction turn-based strategy games. As the examples above arrived after the 2012 release of Enemy Unknown, and all feature similar game mechanics that are not commonly found in games before the 2012 version. The music implementation methods are alike, with Phantom Doctrine being one that stands out due to the implementation of a parallel system, which can provide musical feedback when fast forwarding through time. However, this approach could have been taken a step further, and carried over to the core gameplay, providing ludic feedback for mechanics like enemy proximity and team health.
This genre of turn-based gameplay shares many similarities with other game genres, for instance, the gameplay has an exploration and combat mode, a typical gameplay mechanic in RPGs and Stealth games like Skyrim (Bethesda Game Studios, 2011) and Tom Clancy’s Double Agent (Ubisoft Montreal, 2015). Different genres that incorporate this style of gameplay also integrate adaptive music. Stealth games often use this style, where music can be used to inform the player of an enemies' location. Take for example Splinter Cells Double Agent (Ubisoft Shanghai (2006), the game implements a parallel music approach, introducing a new layer to the music as the players' proximity to the enemy increases. This method could also be applied to a turn-based game like XCOM and extended to include an RTPC linked to team health, introducing new or different layers depending on this variable, which will reflect the players' performance during combat with an increased intensity if the action is going poorly.
In 2013, Clint Bajakian presented Producing Music for AAA Video Games at GDC (Bajakian, 2013). In this, Bajakian discussed adaptive scoring techniques by creating a musical mood playlist. There are methods described in the GDC presentation; one essentially shuffles the tracks within the playlist, which are then triggered by in-game events, activating the stinger, which in this case is a musical mood. Philips (2014) advises stingers can vary in length, being anything from three to twenty seconds long, the objective of a stinger is to inform players of a change in the current state of gameplay. In the game Uncharted 3; Drake's Deception (Naughty Dog, 2011), this technique is used while exploring the terrain, the stinger is used to draw the player’s attention to specific landmarks, collectables etc.
Bajakians (2013) second method is detailed on the GDC slides, which uses a stinger like approach to exploration music. It works by having the default exploration music break away from its usual tone or drone, by transitioning to a random mood track, which leads back to the exploration music. This process will be repeated in sequence, random, or a combination of both. This method acts like a stinger, but the technique is known as transitional or horizontal resequencing. This method could also work in a turn-based strategy game, which could help set the mood of gameplay exploration, while also helping to prevent the music from becoming repetitive.
This project will experiment with these techniques, and forms of implementation to test if this style of adaptive music can be implemented to work in turn-based strategy games previously mentioned, like XCOM, Phantom Doctrine, Attack Of The Earthlings and Xenonauts. Other goals of the project, is to supply adaptive music that will help players gain mastery of the game, by providing useful feedback that can inform the player of how well they are performing, locate enemies, and health status. The project should also be easy to set up for developers to run and test implementation.
Gameplay Mechanics
Before beginning this project, the fundamental gameplay mechanics of XCOM Enemy Unknown were depicted to decide how an adaptive music system could be implemented. The core gameplay breaks down into two categories; exploration, and combat. These will become the two music states for gameplay. Next was deciding how to arrange the music in a way that will provide feedback to the player in these states.
Exploration
Non-repetitive music: Making use of drones, pulses, textures and atmospheric sounds. There will be various variations in the composition stage to prevent repetitiveness, and later in WWISE, this randomness can be implemented and improved upon to further effect. See diagrams below for details:
Musical moods: Theme variations will play at regular intervals to convey mystery. In the above diagram, group 4 contains musical moods. One of these moods will be randomly selected at each interval.
Searching for enemies: An RTPC (Real Time Parameter Control) will introduce a percussion stem. As the player gets closer, the volume will increase and vice versa. A similar function in turn-based games could act as audible feedback, informing players in the direction of enemies. In this project, a parallel approach will be taken, inspired by the method used in video game Tom Clancy's Splinter; Cell Double Agent (Ubisoft Shanghai (2006). This project will incorporate an RTPC to control the volume from the enemies' proximity, increasing in volume as the player moves closer, and decreasing as the player moves away. This will provide musical feedback that will inform the player of the enemies position. See the diagram below for a visual representation of the system.
Stingers that represent the player and alien turns: Each time the player confirms a move, it will trigger a short percussion based stinger. The same is true when it’s the aliens’ turn, which will use strange percussion and textural sounds.
Combat
Non-repetitive music: In combat, the music system will use a combination of transitional and parallel music. The music will be spliced at every two bars to give a natural tail, which will aid the music in transitioning to and from different sections. See diagrams below for details.
There are six tracks in total that follow the above system for combat. It’s possible to add more variation than what is depicted above. For example, the following scenario is possible through track branching;
Track 1 section 1 - Transition to Track 3 section 2 - Transition to Track 2 section 3 - Transition to Track 5 section 4.
It was decided to stay away from this method, as the initially composed themes were lost, and the music too random, and unmemorable. Therefore, instead, an arranged branching system was better suited in this scenario.
Team damaged / severely damaged: Combat in XCOM is between aliens and humans, one way to provide feedback to the player, is through an RTPC. The RTPC will be linked to the teams' health, as the team's health decreases it will activate a switch track. A switch track within wise is essentially streaming specific tracks within the switch, and the track playing is dependent on the variable.
Stingers that represent alien turns, player turns, and teammate downed: This will be implemented in the same way that stingers were implemented for exploration gameplay.
The music will need to transition back and forth between these states, in a way that’s musical, rewarding, and firmly in time to player actions. With the various state information now identified, it’s possible to compose and arrange music that can be implemented into a musical system, capable of providing feedback to the player without being repetitive. The player can also complete or lose the mission, which will also need separate transitions.
Composition
Having detailed the type of musical feedback that will be required, action can be taken in the composition stage by creating music that transitions without interrupting the flow of the game. In this project, a decision was made to use a consistent tempo of 120bpm, the scale D, and its various modes, then utilising two chords Dmin and E, with the same regular structure. Planning for WWISE middleware will enable more options for flexibility when making transitions happen at expeditious moments gameplay.
Preparing For Switch Tracks
In the game XCOM; Enemy Unknown, the battle is between humans and aliens, to try and convey this regarding music, organic instruments like strings and brass will represent the humans, while electronic, synthesised sounds will represent the aliens. In combat, by introducing the electronic instrumentation as the player team is damaged, could increase the intensity of gameplay, while also informing the player the need to play smarter. This is achieved by taking elements of the rhythm midi track, then applying the midi to a synthesiser. The further a player is damaged, a more intense version of the synthesiser can be achieved, by adding the midi theme to the synthesiser along with the rhythm. This can later be implemented via a switch track within WWISE. See below for file preparation within Cubase.
Previewing Arrangements
In exploration gameplay, the music needs to convey the mood mystery at regular intervals without sounding repetitive. To achieve this slight adjustments to the arrangement and musical phrasing of the theme have been composed within Cubase, which will later be improved upon within the WWISE environment. This will be achieved with playlists containers that will shuffle and randomise the tracks within a music container. To save time and preview results, Cubase has an arranger track, which enables users to experiment with the structure of a song. Here, the arranger track has helped in promptly testing the results of randomly stitching tracks together, without the need of exporting and importing into WWISE, finding a mistake and repeating the process. See below for details:
Naming Conventions
When working a project like this one, multiple files have many variations, different lengths, tempo and signatures. Without a naming convention system, searching for specific files could become difficult. In this project, a system based on the WWISE 201 tutorial was implemented (AudioKinetic, 2018); Track name, tempo, time-signature, measure. For further organisation, files are structured using the folders; Exploration, Combat, Stingers and Transitions.
Sampling Rate
WWISE converts files to a sampling rate of 48,000 (AudioKinetic, 2018), to prevent unnecessary file conversions when importing audio files to WWISE, the Cubase project was set to a sample rate of 48,000. Exported WAV files (Waveform Audio File Format) were also set to 48,000, 16 Bit.
WWISE Implementation
New Arrangements
Within the WWISE interactive music layout, fresh unplanned (or thought of at the composition stage), musical arrangements can be created. This is possible with the various track types that exist within WWISE; Normal, Random, Sequence and Switch. Normal tracks are simply that, it will playback audio like a regular DAW. A sequence track can store more than one track, and it will then play through each track in sequence. A random track works much the same way as a sequence; however, the fundamental difference being it will randomly select a track within its storage. A switch track can also house more than one track, with the ability to select a specific track depending on game variables.
A combination of these methods offers a considerable amount of diversity in music arrangements, minimising or completely removing repetitive music.
Random Track
Random track containers were utilised in different ways throughout this project for the previously mentioned reasons. See below for examples;
Sequence Track
In this project, sequence tracks have been used to synchronise rhythm tracks together, and to organise specific arrangements in instrumentation. See below for examples:
Switch Track
A switch track will be used to increase tension during combat. The track will only activate once the player team is injured, or severely injured. An RTPC variable will control this.
Real-time Parameter Controls (RTPC)
As seen above, there is an RTPC in control of the switch track. A specific switch is enabled during combat, depending on the player’s health. Enemy proximity was detailed in a previous diagram; the system will be implemented into exploration gameplay. The player is trying to seek and locate the aliens; this is an opportunity to influence the player to travel in the correct direction. To do this, an RTPC will be used; this will function by controlling the volume of a percussion layer with a parallel form approach, also referred to as vertical remixing (Phillips, W. (2014). See below for implementation;
Music Playlists
In the previous section music stems were imported into music segments, these segments allow the user to create random, sequence and switch tracks. Music playlists enable the user to create an arrangement with these newly formed segments, through the use of the Music Playlist Editor. Here, the music playlist will be used to develop a structured audio system that selects random segments.
The combat playlist had the potential for many more variables. However, the approach would have lost the originally composed melodies, with too much room for randomisation, it would be unlikely to remember any of the melodic content. For this reason, the music themes have been arranged to transition into the next phrase of the melody, after eight bars, which is equal to four transitions, the music will branch into a new random theme or rhythm section. See below for more details on the combat playlist.
States
States are crucial, as they allow the music to change to match the gameplay state. In this case, when the player is exploring the terrain exploration music can be heard. When the player enters combat, the music needs to adapt and switch to combat music. See below for details:
Stingers
Stingers will add some interactivity to the music, as the player confirms a move during exploration and combat, a short percussion stinger will be introduced on either the next beat or bar. As the enemy takes their turn, a short stinger of unusual percussion or texture will also be introduced on the next beat. There will also be subtle stingers added for teammate/alien down.
Transitions
Transitions help to connect two parts smoothly. In this project, the primary transition that’s frequently heard is from exploration to combat. To achieve a smooth transition, the cello, viola, and percussion section are playing the first two bars of the incoming track. Rather than start at full volume, it’s being vectorised over the exploration music, which is simultaneously fading out. See below for more details.
Soundcaster - Developer Testing
Soundcaster is a feature within WWISE that allows users to set up a custom layout for testing the music system. In this a case, a series of events have been set up to test the system. Events are used within WWISE for the game development software to call upon. If the game has called an event, anything can happen from something simple as triggering a stinger, or more complex like trigger intro music, delay exploration by eight seconds, then trigger texture stinger etc. This makes soundcaster particularly useful, as developers can quickly load and preview the various game events. See below for more information.
Conclusion
By the games analysed in this paper, which was limited to science fiction turn-based strategy games;
XCOM: Enemy Unknown (Firaxis Games, 2012)
Phantom Doctrine (CreativeForge Games, 2018)
Attack Of The Earthlings (Team Junkfish, (2018)
Xenonauts (Goldhawk Interactive, 2014)
It would appear that adaptive music is not the method of choice of implementation. Developers are tending to lean towards a more linear looping approach or a linear transitional approach. Video game Phantom Doctrine stood out with its approach to parallel remixing/vertical remixing but didn’t take advantage of introducing a system which could provide musical feedback while playing the core gameplay. Phantom Doctrine is also one of the latest games releases of the games analysed, which does appear to be taking a step away from competitors, with regards to music implementation.
This project experimented with adaptive music, to test if it was a viable option for the niche, but growing genre; Science fiction turn-based strategy. By using soundcaster to create events, and mirroring these events with gameplay taken from XCOM Enemy Unknown, it’s clear that adaptive music could serve as functional music. This project demonstrated it could provide directional information through parallel/vertical remixing, by increasing the volume of specific stems as the player gets closer to the enemies. This could be extended to include objectives, targets, items of interest and more.
The implementation of a switch track linked to an RTPC showcases how music can be used to increase the intensity of gameplay. In this instance, it was achieved by introducing new layers of electronic sounds during combat, the electronic sounds represent the alien invaders, while the organic sounds represent the players’ team. If the players’ team receives enough damage, the switch track is activated, if the team is severely damaged, then switch track will activate a more dramatic piece of music.
The transitional/horizontal resequencing approach taken in this project enabled the music to quickly change out from combative to explore, loss and win, without disrupting the musicality of the arrangement. When combat was completed and switched to the exploration state, the combat music ended musically, rather than abrupt, which is more satisfying to the player Bajakian, C. (2013). With more steps forward like Phantom Doctrine, it may be possible to see more games incorporate adaptive music that can help players obtain mastery game.
Lastly, by populating and structuring the soundcaster with events, developers are equipped to quickly test the music system, without the need for an extended knowledge on WWISE.
Recommendations
WWISE proved to be a tool that helps composers experiment with their music, while also helping to bridge the gap between game developer and composer. If time constraints is an issue, then this could be a possible reason why adaptive music is not commonly implemented, however with careful planning, it could be straightforward to implement the linear music that exists within these games. For example with a parallel/vertical remixing approach, with only one extra stem required to implement this system, it could add greater diversity and provide ludic feedback to players. The stinger based approach surprisingly worked well, which is demonstrated in this XCOM project. As turns become closer together, so do the stingers, becoming a battle amongst itself, fighting for space, and increasing the intensity of gameplay, it would be interesting to see this concept taken further, introducing melodic or dissonant content.
A small sample set of games were used from a specific genre in this report; it would be interesting to find out if adaptive music has been implemented when looking at the wider spectrum of titles within the genre. Or if other genres within the turn-based strategy style have implemented a form of adaptive music, then this too, could potentially crossover to this niche genre.
References
AudioKinetic (2018) Delivering The Assets [online] Available at <https://www.audiokinetic.com/courses/wwise201/?source=wwise201&id=delivering_assets> [Accessed 26/03/2018].
AudioKinetic (2018) Audio Output On Platforms [online] Available at <https://www.audiokinetic.com/library/edge/?source=SDK&id=audio__output__platform.html> [Accessed 26/04/2018].
Bajakian, C. (2013) Producing Music for AAA Video Games [Presentation] Available at <https://www.gdcvault.com/play/1017670/Audio-Bootcamp-Producing-Music-for> [Accessed 24/04/2018].
Collins, Karen. (2008) Game Sound: an introduction to the history, theory, and practice of video game music and sound design. eBook edn. Cambridge, Mass; London: MIT Press.
CreativeForge Games (2018) Phantom Doctrine. Computer Software: Good Shepherd Entertainment.
Firaxis Games (2012) XCOM Enemy Unknown. Computer Software: 2K Games.
Goldhawk Interactive (2014) Xenonauts. Computer Software : Goldhawk Interactive Publishing.
Phillips, W. (2014) Composer's guide to game music. eBook edn. Cambridge, Massachusetts: The MIT Press.
Sweet, M. (2015) Writing interactive music for video games. eBook edn. United States: Addison Wesley.
Team Junkfish (2018) Attack Of The Earthlings. Computer Software: Junkfish Limited.
Ubisoft Shanghai (2006) Tom Clancy's Splinter Cell: Double Agent. Computer Software: Ubisoft.