Introduction
This post will explore creative sound design techniques that can be applied to film and games, this will include Tape, Granular and Fast Fourier Transform techniques. To demonstrate this, the animated short film The Shark In The Park (Fabian Pross (2017) will be featured. The film will be stripped from its original audio and imported into Cubase, which will be the DAW of choice for this walkthrough. The animation Shark In The Park was selected as it contains many visuals that can benefit from creative sound design, especially the granular visuals, which could also benefit from using granular synthesis to enhance and embrace the scene with context.
The original short film is based on a recently discovered Korean strawberry plant known as Parco Pistris Cheolyun Lee (2018). The plant grows tentacles, teeth, and consumes bugs like worms. Even though this plant exists, it’s seems entirely alien, which will factor into the sound design in this project, which will implement a blend of unusual granular synthesis for the alien aspect, mixed with hyper-realistic sounds to keep the animation grounded in reality.
Video Reference - Sound Design Rescore
Project Planner
Through the use of markers the project was organised into various sections, these sections can be quickly looped and allow for quick previewing of sounds and aid in synchronisation.
Background Sounds (Keynote Sounds)
Before applying the creative sound design techniques, the general background audio was placed into the project to set the mood. This included distant cars, crickets and dogs. This was to showcase the setting, a park at night. Parks are usually close to public areas, which is the reason for distant cars and dogs barking. These sounds can be later manipulated with tape techniques like delay and reverbs.
FFT Techniques
During the opening of Shark In The Park, the organism is beginning to take form, as it oozes and bubbles from the top down. FFT (Fast Fourier Transform) synthesis was desirable during this scene, as it has the potential to convolve various audio signals together, generating a new unique sound. Miranda (2002) explains the mathematics of Fast Fourier Transform suggest, that composite of signal harmonics are identifiable by matching frequency occurrences, concurrently varying the frequency of the audio reference continuously. To achieve this a patch from Max MSP was customised, tweaked, and manipulated to suit the needs of the project, which works by analysing the audio into various windows, and applying a filter like process to merge the sounds. See illustration below for details;
This patch was then used to create some interesting sounds, like sizzling bubbles, and bubble stretched rubber sounds, bubbling squishiness. These sounds were inspired by the TV show Stranger Things (2016), which takes full advantage of disgusting and disturbing sounds that can make the audience feel uncomfortable. In season two of Stranger Things, episode six, time-code: 00:02:30, sounds like this can be heard as Steve enters the basement, and finds the shedded skin of the once pet Demogorgon. In season one episode six time-code 00:01:10, Nancy is trapped in the upside down, similar disturbing sounds can be heard as she escapes, and squeezes through the tree, which sounds like bubbling, sticky mud. Take a look at the example below, of how such sounds were created using the Max FFT Patch, to generate the Stranger Things inspired sound design.
This process was carried out on multiple sounds, and sometimes the newly created sound went through the same processing again, an example of this can be heard in this projects opening scene as Parco Pistris forms together, here the following convolution took place;
- Fried egg convolved with Blowing Bubbles Through Straw (in a glass of water) = Fried Bubbly.
- Fried Bubbly convolved with Dry Ice = Sizzling Ice.
As one can see, the naming can quickly get out of hand, and in this case, it was decided to name the sounds on how they sounded, with the prospect that they could provide to be useful for future endeavours/projects. Because of this, all files created were tagged with metadata, then added to a personal library collection.
A similar process was applied to recorded audio of a balloon, which when layered with the sizzling ice, created some unique and uncomfortable sounds, which synchronised well for Parco Pistris; again this is in the beginning as it bubbles and expands. In the case of layering, this is different from the convolution process, layering here means the sound has already been convolved, imported into Cubase, and then synced to the picture. Afterwards a second convolved sound was imported into Cubase, synchronised to picture, and layered to works with the audio that has already in place. See illustration below for more details on the synchronisation.
Granular Synthesis
Whittington (2007) discusses the use of unusual sounds to generate a perception of being in a strange place, in this project the camera cuts to inside Parco Pistris, giving insight on a granular scale to its inner workings. To match the granular visuals, it was also decided to use granular audio as this would generate the necessary sounds to create a strange world within Parco Pistris. As Miranda (2002) states, granular synthesis operates by generating rapid sound bursts called grains, which are usually less than 35ms in length, and are pieced together to create more extended events. Research of granular synthesis by Curtis Roads and Barry Truax in the 1970s led to the first computerised granular synthesis system. This project implemented Graincube for the creation of common sounds for the inner workings of Pacro Pistris. Graincube is made up of four oscillators, which randomly switch reference samples, while simultaneously adding random delays, filtering, reverb and panning. After five minutes of recording its randomness, the files were spliced at various sections that proved useful, then experimented with by syncing the spliced audio to the picture.
Although this worked for creating some unique sounds, almost all recorded audio was unusable and too random, costing too much time to capture something useable. With the goal to combat this uncontrollable randomness, some research was carried out into the development of granular synthesis, which led to Sound Particles (2018). Sound Particle has a unique audio engine, and works standalone, separate from the DAW; it has been used to design sounds for films like Guardians of the Galaxy, Ready Player One, The Dark Tower and Cars. Although it’s unclear what exactly this program did for these films, it was worth investigating its capabilities. After reaching out to the developer, an authorisation key was provided, allowing full access to the program. It was soon discovered Sound Particles Granular Synthesis Engine could produce randomly controlled results, which is highly desirable for accurate synchronisation when using granular.
By using Sound Particles, the sound results were much more predictable. If something was off regarding the sound, like too repetitive or machine gun like, which is something that can happen easily when working with granular synthesis, this is due to the small bursts of grains, which is sometimes a desirable effect. However, in this project, a smoother approach was desired, at 00:01:15 in this video, Sound Particles was used to create an underwater movement sound and an underwater explosion sound. This was achieved by first filtering the sound of water with a low pass filter, then importing it to Sound Particles. The sound of a melon being repeatedly squished was added. Take a look at the illustration below for more details:
Azimuth Rotation velocity was set at -60, this essentially controls the speed of movement, while Radius controls the proximity, in this case, each time the circle is complete, the circle will get smaller, which means the sound is getting closer. The time was set to 10 seconds, which was enough time to create a fade in and fade out movement that synchronises with the picture from 00:01:15 to 00:01:25 as the camera slowly zooms in. This program proved to be flexible, and extremely powerful, but not without its downsides. Depending on the task at hand, it’s hugely CPU / RAM hungry, to perform this underwater task, the program was consuming 4GB of RAM, and over 300% in CPU usage, then a further fifteen minutes to render the results. The program was only implemented when a desirable outcome was desired from granular synthesis; otherwise, the randomness of plugins like grain cube was fine.
The film Innerspace (1987) follows the character Tuck Pendleton as he is going through an experiment to be shrunk to microscopic size, and explore the insides of a rabbit within a microscopic ship. The experiment goes wrong, and Tuck ends up inside of a human. During the stomach scene, there is an underwater sound effect, lots of squish and slurp sounds; this is what initially led to the idea of an underwater sound effect for the insides of Parco Pistris.
As Parco Pistris is a form of predator, to showcase this, a sound effect was created inspired by the original Predator (1987) film, and video game The Last Of Us (Naughty Dog (2013). The Predator film revolves around an alien, which generates an unusual click sound from its mouth. In the Last Of Us, the clicker monster type is an enemy that can’t see and uses a click sound as a form of echolocation to find enemies. With that in mind, a clicker like sound was recorded, imported into Sound Particle, where it went through the following processing:
Granular Synthesis 1000 particles per second.
Random gain from 0 -12db or +12db
Random filters applied to each grain burst
Random time/Pitch Shift Between +1 and - 3 semitones, also affects the length, lowering
the pitch slightly increases the length, whereas increasing the pitch shortens the length.
Panning movement
This can be heard in this film at 00:00:35. The panning and reverb choice was influenced by Whittington William (2007), who discusses the use of reverb and space to confuse the audience as they try to distinguish sounds. The idea has been set up to sound as if the clicks are searching the area, waiting to detect something, it could be considered Point-of-audition from the perspective of Parco Pistris.
Tape Techniques
Tape techniques in this project helped to extend or shorten the length of specific audio through the use of pitch shifting (Russ, M. (2012). This helped with synchronization of sound to picture, subtly adjusting time until the audio was at the correct length, an example of this in the Pacro Pistris video, can be heard in the beginning as the bee flies past, the original audio was higher pitched, and shorter in length, thankfully after the pitch adjustments the sound still matched the visual.
A standard reverb was used to create the space, and bringing in the otherwise dry sounding samples, and making them sound like they are part of the world. However, creating a different spatial sound to glue everything together while the Granular Scene took place, took some inspiration. The website A Sound Effect interviewed the sound designers and audio director of Stranger Things; where they discussed the creation of the film world. What was in particular interest, was the discussion of the upside down place, an alternate and parallel reality which exists within the series, the team explained that the space needed to sound wet, but not reverberant, somewhere between delay and reverb (How the Outstanding (2016). This sentiment resonated with this project when the screen cuts to show within Parco Pistris, it’s like an alien world. Under this influence, a chorus effect bus was set up within Cubase, which automates to turn on, as the screen cuts to the inside of Parco Pistris, then off, as the screen cuts back to reality. See illustration below for project example.
Aside from creating the space and atmosphere, tape techniques were also crucial for general layering, like the sound of melons being sliced, squished and poked to develop goopy slush sounds, which were used to layer against the FFT and granular sounds. The sounds of cars driving by, have been placed in a way that they often synchronise with the visuals, or help with a transition by creating a rise, and then quickly fading off.
During the beginning of the film, the bee can be heard entering the centre pan, moving to the left, then to the right as it exits the screen, with use of panning and gain to tell its location and distance. Also used in this project was the sound of various percussion hits, which have been designed in various ways with EQ, reverb, distortion and tape reverse. This can be prominently heard at Timecode: 00:01:00.
Conclusion
This project demonstrates how effective creative sound design techniques can be in creating new and unique sounds. Granular synthesis proved to be particularly effective in creating alien-like sounds, which are capable of synchronising to granular visuals. The tape technique of applying chorus also worked well to glue the various sounds together, making them exist within the same space, although a similar result could have been achieved by reverb, the chorus effect added something strange to the sound, creating an alien-like spaciousness, which wasn't overly reflective. FFT techniques are useful in creating unique sounds, as was demonstrated by combing the sound of an egg frying with bubbles, which synchronised well, and sculpted the intro of Parco Pistris.
References
Cheolyun L.C (2018) Parco Pistris – Plant Structure & Growth Habit. Korean Strange Plant Research Institute, 20 April [Online blog]. Available from: <http://www.strangeplants.org/parco-pistris/>
Fabian Pross (2017) The Shark In The Park [Online video]. Available from: <https://www.youtube.com/watch?v=MzIz3kRpvVU>.
Innerspace (1987) Directed by Joe Dante. Warner Bros.
How the Outstanding Sound for ‘Stranger Things’ Is Made: (2016) A Sound Effect, 17 August [Online blog]. Available from: <https://www.asoundeffect.com/stranger-things-sound/>
Miranda, E. R. (2002) Computer Sound Design: Synthesis Techniques and Programming [Book]. Taylor & Francis Ltd.
Naughty Dog (2013) The Last Of Us [Playstation 3]. Sony Computer Entertainment. United States.
Predator (1987) Directed by John McTiernan. 20th Century Fox.
Russ, M. (2012) Sound Synthesis and Sampling: [Online]. Focal Press. Available from:
<https://www.dawsonera.com:443/abstract/9780080926957>.
Sound Particles (2018) [Online]. Available from: <http://soundparticles.com/index.html>
Stranger Things (2016) Directed by The Duffer Brothers. Netflix [Stream].
Whittington W.W (2007) Sound Design And Science Fiction. United States: University of Texas Press.