Text to Speech Player |
WithFeel VR Granulator
If you’ve read about the general development of WithFeel VR on this site you’ll know that it takes advantage of our embodied descriptions of timbre. In particular descriptors such as smooth sounding or rough sounding. The development description page mentions that 3D texture and morphology are mapped to timbral and pitch modulations but it doesn’t say how this is done.
The WithFeel VR Granulator is the means of audio synthesis.
The image above shows a standalone version of this plugin created using the JUCE C++ framework for audio development. Don’t worry, I won’t go into detail about the actual code that runs this but maybe an explanation of granulation might help an interested party to understand why it lies at the core of WithFeel VR. The unsung hero that no-one will ever see in VR but the hero making all the noise.
Have a look and listen to the video below. In this video I whistle a single note and then count aloud. You’ll hear the unfiltered whistle coming from my microphone but it is quickly dominated by the synthesis occurring in the standalone version of the WithFeel VR Granulator plugin. If you are using the provided text to speech then pause here if you wish to listen to the video.
What is Granulation?
Granulation is a modern audio synthesis technique but it was actually theorised back in 1947 by Denis Gabor, a Nobel laureate, physicist and engineer. It didn’t come into use as a musical tool until the 1980’s when pioneers such as Curtis Roads and Barry Truax produced the first compositions using computerised granulation (Iannis Xenakis composed granulated compositions previously by cutting small sections of magnetic tape).
The name tells a lot about this method. The original methods use by Roads and Truax combine tiny ‘grains’ of oscillators together. The result of this method of granulation often sounds like water so it is not surprising that the first real-time granular synthesis composition was Riverrun by Barry Truax. (A fantastic composition that I had the pleasure of analysing years ago. Go and find it.)
Barry Truax had to create his own computer language to achieve this amazing feat with 1980’s computer processing power and memory capacity. As I was born in the seventies, I can tell you we used to get excited about 64Kb of memory in the 1980’s not the Gigabytes that we are used to today.
A common approach to granular synthesis today does not use oscillators (though some do) but audio samples as the generator of grains. The WithFeel VR Granulator works this way. The standalone version works a little differently to the VR plugin but the principle is the same.
What you heard was the granulator storing my voice in a ring buffer. Think of it like a temporary storage space. When it records to the end of the buffer it goes back to the beginning to fill it in again. Hence ‘ring’. The granulator then selects a tiny ‘grain’ of sound from my recorded voice. Often at random positions within the buffer. In fact it collects lots of grains. It then adds those grains together to create the sound you hear.
By changing the length of the grains and the distance between them (delay) we can affect the timbral qualities of the sound. We can even change the pitch, (this system does not use PSOLA but simple resampling), of individual grains to create complex sounds.
How does that work in accessible VR development?
The image above shows WithFeel VR in development in Unity 3D. You might recognise things such as ‘Grain Duration’ or ‘Grain Delay’ on the right. That’s the WithFeel VR granulator in its Unity form. JUCE allows me to output for different platforms using the same shared code. The reason for that will become evident as I go through my PhD and can share more of my work. But it is very important to this project (hint: composition).
Unlike the standalone version the Unity version of the granulator works with whole samples and not ring buffers. (hint: composition again). This allows us to freely scrub through the audio as needed.
WithFeel VR measures the roughness of an object in VR and also whether it has bumps or dips. This data is then used to change the parameters on the granulator to make the sounds ‘rough’ or ‘smooth’. These are embodied descriptors of timbre that many of us share and therefore it makes the development of this software more accessible.
I’m the only person who needs to worry about granulators, C++, timbral modulation and so on. My co-designers need only consider how they perceive and describe music. and the granulator does the heavy lifting in the background unseen and under appreciated.