Procedural Audio is a misunderstood subject which has been around from the very beginning of game audio, in fact it used to be only the way. This post will explain what Procedural Audio is and to explore the history behind the term. We will also look at when to use Procedural Audio, and what issues may arise in doing so. Finally, the post will end with what’s in store for Procedural Audio.
What is Procedural Audio?
Procedural Audio is often mistaken for Procedural sound design, however where Procedural audio is capable of generating it's own sound, Procedural Sound Design relies entirely upon recorded sound, and is not capable of generating sound on its own. According to Andy Farnell who specialises in digital audio processing;
"Procedural audio is sound qua process, as opposed to sound qua product. Be- hind this statement lies a veritable adventure into semiotics, mathematics, computer science, signal processing and music. Let us reformulate our definition in verbose but more precise modern terms before moving on. Procedural audio is non-linear, often synthetic sound, created in real time according to a set of programmatic rules and live input. To further define it let’s explain it in terms of linear, recorded, interactive, adaptive, sequenced, synthetic, generative and AI audio" (Andy Farnell, 2007).
Andy's statement widens the umbrella of Procedural Audio to include both Procedural Sound Design and Procedural Music. Which are in their own right separate subjects that follow different processes. University professors Richard Stevens and Dave Raybould, co authors of "Game Audio Implementation" (Stevens and Raybould, 2016) have a full chapter dedicated to Procedural Sound Design, detailing the processes and steps that are necessary to setup a Procedural Sound Design System. The introduction to the book acknowledges Procedural Audio, but ultimately spots the divide between the two subjects.
"Procedural sound design is about sound design as a system, an algorithm, or a procedure that re-arranges, combines, or manipulates sound assets so they might:
a. produce a greater variety of outcomes (variation or non-repetitive design)
b. be more responsive to interaction (parameterisation).
This approach to sound design exists on a spectrum from procedural sound design, where we tend to be manipulating pre-existing assets, to procedural audio, a term more frequently used when systems of synthesis are used to generate the sounds themselves (with much in between that combine both approaches). Procedural audio is an increasingly important field, but one that requires its own tools and is worthy of its own study". (Stevens and Raybould, 2016)
In 2013 a research article published by the International Journal Of Computer Games Technology describes and explores the difference between Procedural Audio and Procedural Sound Design, although the article refers to Procedural Sound Design as Recorded Audio (Böttcher, Martínez,, and Serafin, 2018). The research involved participants interacting with games using motion controllers, the purpose of the research was to determine whether or not players were affected by procedural audio, vs pre-recorded audio (Procedural Sound Design), which again showcases the differences between the two subjects.
Nicolas Fournel is the Principle Audio Programmer for Sony Computer Entertainment Europe. On the GDC Vault Nicolas has defined procedural audio as Real-time Sound Synthesis (Fournel, 2017). From this research we can safely conclude that Procedural Audio is in fact an audio system that is capable of generating its own sound using electronic synthesis. It does not use recorded audio material.
History
In the early days of gaming, Procedural Audio was a necessary step to include sound in games. There simply wasn't enough RAM to even consider Procedural Sound Design.
Sound Chips
Game audio development started in 1972 with the Atari publishing the game Pong on the Magnavox Odyssey (Pong,1972). It's important to note that a similar version of the game also came standard with the Magnavox Odyssey. The difference being that the original version contained no sound. (Gamespot/Glenn McDonald, 2005). The Atari version generated sound with a chip known as the Television Interface Adapter (TIA). The TIA had two oscillators for generating sound waves.
This is the first case of Procedural Audio. The game contained three sounds which provided feedback to the player of successful pass backs, wall deflections and a buzz for missing the ball. Atari would later release a console in 1975 specifically for the game Pong (Chang and Kim, 2007).
There were other consoles developed in the late 1970s that were cable of Procedural Audio such as the Fairchild (1976), Atari 2600 (1977), Odyssey 2 (1978), and the Intellivision Mattel (1979). Each cycle bringing its own enhancement or take on game audio.
Entering the 1980s rapid advancements could be seen with the Vectrex (1983) and the Nintendo (1983). The Nintendo was able to produce five channels of audio, it was capable of pitch bends, and supported frequencies from 54Hz to 28kHz. Games like Mario set a new benchmark for Procedural Audio and Music, with unique and various sounds used for pick-ups, powers, pipe travelling, jumping, character deaths, time count downs and enemy sounds.
In 1986 Sega produced the Master System, which involved yet another advancement in game audio, this time making use of electronic synthesis and sampling. The sound chip used was the YM2413, which was designed by Yamaha and was based on their YM3812 chip found in their synthesisers. The Sega Mastersystem was capable of producing 4 channels of audio, three for music, and one for noise generation.
This trend would continue with Nintendo and Sega releasing consoles Super Nintendo and the Sega Mega Drive. Ending this generation of consoles was the Sega Saturn, which was impressive as it contained both a sound processor and sound chip. Meaning the console could support up to 16 channels of audio on its custom sound processor. The console had a sampling rate of 44100kHz, which is equal to CD quality audio. A big jump from the previous 16kHz sampling rate found on previous consoles the Sega Megadrive and Super Nintendo. (Chang and Kim, 2007).
Pre-Recored Music & Sound Effects
A big change in the industry happened in 1994 with the release of Sonys Playstation. The Playstation allowed up to 24 channels of audio, a sample rate up to 44100kHz equal to CD quality, which supported stereo sound. The sound chip provided support for audio looping and reverb effects. Music composers and sound designers could now use pre-recorded audio in terms of both music and sound effects.
Before Playstation, sound designers and composers would have required knowledge of audio programming to make music and sound effects. In other word Procedural Audio was a necessity if developers wanted sound in their games. The Playstation Era meant composers and sound designers no longer needed to know difficult coding to implement sound into games. The sound and music could now be recorded, and then passed to the developer for implementation, something that was impossible with the previous generation of consoles (The National, 2014).
The End of Procedural Audio
Not even close, it's true that most games post Playstation use Procedural Sound Design, however there have been examples of Procedural Audio used in critically acclaimed games. Take a look at the video game Spore for an example (Spore, 2008). The Audio programmers of Spore; Ken Jolly and Aaron McLeran implemented Procedural Audio Techniques for the music and environments of the game. The pair embedded an audio engine known as libpd, an adaption of Puredata which is a visual programming language. This allowed the team to make music and environmental sounds based on ever changing probabilities that exist within the game world (Plans and Morelli, 2012).
Other examples exist in popular games such as Elite Dangerous released in 2014 (Elite: Dangerous, 2014) this is a science fiction game which takes advantage of Procedural Audio to create engine and graphic interface sounds. DEFCON, Love, Elite and Sentinel are recent games to make use of Procedural Audio.
When To Use Procedural Audio
There are many ways to use Procedural Audio, here are a few examples taken from Nicolas Fournels GDC talk. Remove repetitive sounds such as footsteps, impacts and explosions. Create wind and oceans sounds. This feature exists within WWISE, helping to combine Procedural Sound Design and Procedural Audio together. Objects that require complex control, such as engine sounds and creature vocalisations. When memory is an issue, proper usage of Procedural Audio techniques will save a vast amount of computer memory(Fournel, 2017).
Issues
As Procedural Audio creates sound using synthesis, the results can be less realistic when compared to sounds in reality. Therefore if a game wants to create authentic sounds, then Procedural Audio is probably not going to work. There is a steep learning curve with Procedural Audio, were sound designers that are interested in the field normally end up building their own audio engine. This engine is then incorporated into the game engine. There is also a fear factor to Procedural Audio, were both sound designers and music composers worry that Procedural Audio could replace them, as the Procedural Audio system does not require any input, and can generate its own sound. Currently Procedural Audio systems still require direction by someone with knowledge of audio and music (Fournel, 2017).
Developments
Companies such as WWISE and Procedural Audio have been making advances in their middleware technology. These advancements are helping sound designers and composers get around the steep programming centric learning curve of Procedural Audio (Procedural Audio, 2018).
Conclusion
The recent advancements of Procedural Audio in middleware, make Procedural Audio more accessible than it has ever been before. It's looking more likely that Procedural Audio and Procedural Sound Design will share the same space, a combination between the two fields, giving all sound designers and composers, even those that lack the programming language, more freedom and creativity than what they've ever had before.
Bibliography
3DAudio (2017) Digital Sound & Music In Computer Games [online] Available at <http://3daudio.info/gamesound/history.html> [Accessed 27/02/2018].
Andy Farnell (2007) An Introduction to Procedural Audio [online] : . Available at <cs.au.dk/~dsound/DigitalAudio.dir/Papers/proceduralAudio.pdf> [Accessed 27/02/2018]
American History (2015) Magnavox Odyssey Video Game Unit, 1972 [online] Available at <http://americanhistory.si.edu/collections/search/object/nmah_1302004> [Accessed 27/02/2018].
Böttcher, N.B., Martínez,, H.M., and Serafin, S.S. (2018) 'Procedural Audio in Computer Games Using Motion Controllers: An Evaluation on the Effect and Perception'. International Journal of Computer Games Technology 2013, pp.1-16.
Chang, K.C., and Kim, T.K. (2007) ' Video Game Console Audio: Evolution and Future Trends '. CGIV 97 (102), pp.1.
Elite: Dangerous, 2014. (Video Game) Directed by David Barben, . Frontier Developments: Frontier Developments.
Fournel, N.F. (2017) Procedural Audio for Video Games: Are we there yet ? [Presentation] Available at <https://www.gdcvault.com/play/1012645/Procedural-Audio-for-Video-Games> [Accessed 27/02/2018].
Farnell, A.F. (2010) Desiging Sound. United States: Westchester Publishing.
Fournel, N.F. (2018) Procedural Audio for Video Games: Are we there yet ? [Presentation] Available at <https://www.gdcvault.com/play/1012645/Procedural-Audio-for-Video-Games> [Accessed 27/02/2018].
Gamespot/Glenn McDonald (2005) A History of Video Game Music [online] Available at <https://www.gamespot.com/articles/a-history-of-video-game-music/1100-6092391/> [Accessed 27/02/2018].
Ping Pong, 1972. (Arcade Game) Directed by Allan Alcorn. United States: Atari.
Plans, D.P., and Morelli, D.M. (2012) 'Experience-Driven Procedural Music Generation for Games'. IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES 4 (3), pp.192-197.
Stevens, R.S., and Raybould, D.R. (2016) Game Audio Implementation A Practical Guide Using The Unreal Engine . Book edn. UK: Focal Press.
Spore, 2008. (Video Game) Directed by Will Wright. Electronic Arts: Maxis.
The National (2014) How PlayStation changed the gaming world [online] Available at <https://www.thenational.ae/arts-culture/how-playstation-changed-the-gaming-world-1.633396> [Accessed 27/02/2018].
Procedural Audio (2018) The Procedural Audio Middleware [online] Available at <http://www.procedural-audio.com/> [Accessed 27/02/2018].