Using External Analysis to Process a Custom AudioSource Filter Chain

NOTE: Please read and understand Delegate Processing if you have not already.

There are many ways to work with audio data in Unity and SALSA. By default, SALSA uses the AudioClip buffer of an AudioSource as its standardized method of working with audio data. This is not always possible and sometimes requires additional steps to implement. How a developer deals with audio data is outside the scope of SALSA support, but SALSA's delegate processing opens up the doors to dealing with nearly any audio data situation.

A common situation can result when audio data is inserted into the AudioSource pipeline, bypassing the AudioClip buffer. When this occurs, there will be an active AudioSource that does not have an AudioClip. As such, SALSA cannot "see" the data and therefore cannot process it. In this instance, it may be possible to expose the data by using Unity's OnAudioFilterRead() callback functionality. Check the Unity documentation for more information on OnAudioFilterRead().

While the audio data could be processed (analyzed) each time OnAudioFilterRead() is called, SALSA has its own timing, governed by Salsa.audioUpdateDelay. The OnAudioFilterRead() callback will likely be called much more frequently than SALSA requires. For this reason, it is recommended to simply store the data as a more lightweight action and only process it when necessary. The general process would be to collect the data from the filter (consumed by the audio chain) and then allow SALSA to process it as needed (according to its audio delay pulse cycle) -- leveraging external analysis. This is also recommended so that the data analysis viewport size can be controlled -- the size of the OnAudioFilterRead() data chunk is variable based on platform and audio parameters.

The script example below leverages Unity's OnAudioFilterRead() callback functionality to collect data fed to the AudioSource chain where an AudioClip buffer is not used. Refer to Unity's OnAudioFilterRead() documentation for information and details.

NOTE: The example script component should be placed on the GameObject with the AudioSource and essentially becomes a filter for the audio data.

Example Script Operation Explanation

FYI: SALSA must be configured to use External Analysis!

In the example below, a simple circular buffer called analysisBuffer is used to collect data since the size of the data chunk presented via Unity's OnAudioFilterRead() is variable based on platform and audio data characteristics. The idea is to simply have the latest data available in the analysisBuffer and the buffer should always be full (depending on the analysis used). Since we are using a simple 'amplitude peak analysis', the buffer does not need to be full, but does need to be of adequate size and filled with enough data to represent the amplitude characteristics of the current data. The size of the buffer should be tweaked to handle the audio parameters of your project; higher frequency data = a general requirement for a larger buffer. Also, consider using a larger buffer for more channels, or cull additional channel data and process a single channel. The example below only utilizes the first channel's data (typically the left channel in a stereo file).

Each call-back to OnAudioFilterRead() adds data to the analysisBuffer. Only data from the first channel is added to the analysisBuffer.

SALSA then 'polls' the GetAnalysisValueLeveragingSalsaAnalyzer() as a delegate to Salsa.getExternalAnalysis which is set in Awake(). In this example, SALSA's analysis engine delegate is used to analyze the data stored in the analysisBuffer. Unless the analysis engine has been delegated to custom code as well, the default internal analysis engine will be used.

NOTE: This process can also be used with Unity's Timeline AudioTrack and also with AudioSource.PlayOneShot() calls. OneShot requires the filter callback script be added (dynamically) after the OneShot is called.

Script Example

FYI: SALSA must be configured to use External Analysis!

NOTE: This is only an example and may or may not meet your requirements. As such, it is your responsibility to modify this example for your own project's needs.

using UnityEngine;
using CrazyMinnow.SALSA;

namespace DemoCode
{
    public class UpstreamAudioFilterProcessing : MonoBehaviour
    {
        public Salsa salsaInstance;
        private float[] analysisBuffer = new float[1024];
        private int bufferPointer = 0;
        private int interleave = 1;

        private void Awake()
        {
            if (!salsaInstance)
                salsaInstance = GetComponent<Salsa>();
            if (salsaInstance)
                salsaInstance.getExternalAnalysis = GetAnalysisValueLeveragingSalsaAnalyzer;
            else
                Debug.Log("SALSA not found...");
        }

        private void OnAudioFilterRead(float[] data, int channels)
        {
            // Simply fill our buffer and keep it updated for ad-hoc analysis processing.

            // Only fill 'analysisBuffer' with channel 1 data. If you want
            // to store and keep track of additional channels, uncomment and
            // set 'interleave' to the number of channels of data passed into
            // this callback. Additionally adjust the for-loop as necessary.

            //interleave = channels;
            for (int i = 0; i < data.Length; i+=channels)
            {
                analysisBuffer[bufferPointer] = data[i];
                bufferPointer++;
                bufferPointer %= analysisBuffer.Length; // wrap the pointer if necessary
            }
        }

        // Utilize the built-in SALSA analyzer on your custom data.
        float GetAnalysisValueLeveragingSalsaAnalyzer()
        {
            // If you need more control over the analysis, process the buffer
            // here and then return the analysis. Since only the first channel of 
            // audio data is stored in the 'analysisBuffer' (in this example), the 
            // 'interleave' value is initialized as '1' -- we've already 
            // separated the data in the callback, so we want to analyze all of it.
            return salsaInstance.audioAnalyzer(interleave, analysisBuffer);
        }
    }
}