SALSA Delegate Processing

SALSA 2.4.0 was the first release to support delegates for plugging custom functionality into SALSA. SALSA 2.5.0 extends this capability significantly. By implementing delegate calls, some of SALSA's key internals can be re-routed and it is relatively simple to do.

General How-To Re-route SALSA Processing

1) Create a new custom process to replace SALSA's internal process -- match the parameter signature requirements for the delegate.
2) Ensure the new custom process returns the correct data type.
3) Connect the appropriate delegate to your custom method (usually in your custom code's Start() callback, but technically wherever/whenever you please). You can turn this on/off at any time.

Return SALSA to Default Processes

After having re-routed any of SALSA's available internal processes, if you need to re-map SALSA back to its default delegate actions, it requires two steps:

1) Set the delegate you previously mapped to null.
2) Then, simply call Salsa.InitializeDelegates() to re-map any null pointers back to the SALSA defaults.

For example: re-route back to default process for external analysis. (See example for mapping external analysis below for context).

// re-map back to default process for external analysis
private void RemoveExternalAnalysisMapping()
{
    CrazyMinnow.SALSA.Salsa _salsa = GetComponent<CrazyMinnow.SALSA.Salsa>();
    _salsa.getExternalAnalysis = null; // required for salsa to re-map to default.
    _salsa.InitializeDelegates();
}

NOTE: Salsa.InitializeDelegates() only re-maps delegates that are null. It will not re-map delegates pointing to valid methods.

While the Editor is in Play mode, the inspector will display the available delegates and the method name of their current mappings. Any delegates that are not mapped to the default process will be displayed in bold type. In the following image, note the Clip Pointer has been mapped to a method called GetMicRecordPointer.
delegate mapping display

Delegate Processes and Definitions

Analysis

Audio analysis has two functions...1) get the data to be analyzed, and 2) analyze the data. There are two available delegates that appear similar but have distinct differences. First is the AudioAnalyzer type and it redirects audio analysis to a custom function after SALSA has gathered the data to be analyzed. It requires no additional flags to be set. Second is the GetExternalAnalysis type and it expects that all data acquisition and analysis will be performed externally (in custom code). This option does require a flag to be set to switch SALSA to external analysis mode CrazyMinnow.SALSA.Salsa.useExternalAnalysis = true.

Audio Analysis

<AudioAnalyzer> float Salsa.audioAnalyzer(int channels, float[] audioSampleData)

  • SALSA has its own internal audio analysis engine; however, it is simple to replace this with your own custom code. Audio analysis can be customized and called during the normal SALSA processing tick cycle see further reading section: (Custom Audio Analyzers). Utilizing this delegate allows you to let SALSA take care of grabbing the data to be analyzed and lets you perform your own analysis.

NOTE: The difference between external analysis and the audio analyzer is what is being delegated away from SALSA internal code. By delegating external analysis, you are basically telling SALSA to not perform data fetching or analysis. Whereas by delegating the audio analyzer (above), you are telling SALSA to continue to fetch data but have a custom process analyze it.

CrazyMinnow.SALSA.Salsa salsaInstance;
salsaInstance.audioAnalyzer = MyAudioAnalyzer;

float MyAudioAnalyzer(int channelInterleave, float[] audioData)
{
    var dataAnalysis = 0.0f;
    // process data...
    return dataAnalysis;
}

External Analysis

<GetExternalAnalysis> float Salsa.getExternalAnalysis()

Previously, it was required to keep the CrazyMinnow.SALSA.Salsa.analysisValue field updated (usually in the Update() loop of your custom class), which meant every frame -- even though SALSA's operations operate far less frequently. Now, it is preferred to re-map this delegate process to your own code and SALSA can then fetch it on demand.

This delegate allows the Salsa.analysisValue to be retrieved from custom code instead of having to supply the value to SALSA. To enable this processing, it is required to turn on External Analysis in the SALSA configuration.
CrazyMinnow.SALSA.Salsa.useExternalAnalysis = true;

NOTE: The difference between external analysis and the audio analyzer is what is being delegated away from SALSA internal code. By delegating external analysis, you are basically telling SALSA to not perform data fetching or analysis. Whereas by delegating the audio analyzer (above), you are telling SALSA to continue to fetch data but have a custom process analyze it.

Example: Random Data Analysis Generation

private Start()
{
    Salsa _salsa = GetComponent<Salsa>();
    _salsa.getExternalAnalysis = MyCustomAnalysisMethod;
}

private float MyCustomAnalysisMethod()
{
    return Random.Range(0f,1f);
}

Using External Analysis to Process a Custom AudioSource Filter Chain

There are many ways to work with audio data in Unity and SALSA. By default, SALSA uses the AudioClip buffer of an AudioSource as its standardized method of working with audio data. This is not always possible and sometimes requires additional steps to implement. How a developer deals with audio data is outside the scope of SALSA support, but SALSA's delegate processing opens up the doors to dealing with nearly any audio data situation.

An increasingly common situation can result when audio data is inserted into the AudioSource pipeline, bypassing the AudioClip buffer. When this occurs, there will be an active AudioSource that does not have an AudioClip. As such, SALSA cannot "see" the data and therefore cannot process it. It may be possible to expose the data by using Unity's OnAudioFilterRead() callback functionality. Check the Unity documentation for more information on OnAudioFilterRead() The idea is to collect the data from the filter (consumed by the audio chain) and then allow SALSA to process it as needed (according to its audio delay pulse cycle) -- leveraging external analysis.

While the audio data could be processed (analyzed) each time OnAudioFilterRead() is called, SALSA has its own timing, governed by Salsa.audioUpdateDelay. The OnAudioFilterRead() callback will likely be called much more frequently than SALSA requires. Therefore, it is recommended to simply store the data as a more lightweight action and only process it when necessary. This is also recommended so that the data analysis viewport size can be controlled -- the size of the OnAudioFilterRead() data chunk is variable based on platform and audio parameters.

This script leverages Unity's OnAudioFilterRead() callback functionality to collect data fed to the AudioSource chain where an AudioClip buffer is not used. Refer to Unity documentation for OnAudioFilterRead() for details. This script component should be placed on the GameObject with the AudioSource and essentially becomes a filter for the audio data.

A simple circular buffer analysisBuffer is used to collect data since the size of the data chunk presented via OnAudioFilterRead() is variable based on platform and audio data characteristics. The idea is to simply have the latest data available in the analysisBuffer and the buffer should always be full (depending on the analysis used). Since we are using a simple amplitude peak analysis, the buffer does not need to be full, but does need to be of adequate size and filled with enough data to represent the amplitude characteristics of the current data. The size of the buffer should be tweaked to handle the audio parameters; higher frequency data = larger buffer. Also, use a larger buffer for more channels, or cull additional channel data and process a single channel. The example below only utilizes the first channel's data (typically the left channel in a stereo file).

As mentioned previously, SALSA must be configured to use External Analysis!

NOTE: This is only an example and may or may not meet your requirements. As such, it is your responsibility to leverage this example for your own project needs.

Example: External Analysis to Process AudioSource Filter Chain

using UnityEngine;
using CrazyMinnow.SALSA;

namespace DemoCode
{
    public class UpstreamAudioFilterProcessing : MonoBehaviour
    {
        public Salsa salsaInstance;
        private float[] analysisBuffer = new float[1024];
        private int bufferPointer = 0;
        private int interleave = 1;

        private void Awake()
        {
            if (!salsaInstance)
                salsaInstance = GetComponent<Salsa>();
            if (salsaInstance)
                salsaInstance.getExternalAnalysis = GetAnalysisValueLeveragingSalsaAnalyzer;
            else
                Debug.Log("SALSA not found...");
        }

        private void OnAudioFilterRead(float[] data, int channels)
        {
            // simply fill our buffer and keep it updated for ad-hoc analysis processing.
            interleave = channels;
            for (int i = 0; i < data.Length; i+=channels)
            {
                analysisBuffer[bufferPointer] = data[i];
                bufferPointer++;
                bufferPointer %= analysisBuffer.Length; // wrap the pointer if necessary
            }
        }

        // Utilize the built-in SALSA analyzer on your custom data.
        float GetAnalysisValueLeveragingSalsaAnalyzer()
        {
            // if you need more control over the analysis, process the buffer
            // here and then return the analysis. since only the first channel of 
            // audio data is stored in the analysisBuffer, we substitute the 
            // interleave value with '1' -- we've already separated the data,
            // so we want to analyze all of it.
            return salsaInstance.audioAnalyzer(1, analysisBuffer);
        }
    }
}

AudioClip Functions

In order to facilitate custom audio buffers, the following delegates are available if you want SALSA to analyze your data. These delegates use the AudioClip API by default. Utilizing a custom audio buffer requires all of the following hooks (modeled on the AudioClip API) to be implemented into your buffer API:

AudioClip Channel Count

<ClipChannels> int Salsa.clipChannels()

Analogous with AudioClip.channels

Requests an int describing the number of audio channels interleaved in the buffer data. Must be >=1.
NOTE: by default, SALSA only processes the first (left) channel data.


AudioClip Recording Frequency

<ClipFrequency> int Salsa.clipFrequency()

Analogous with AudioClip.frequency

Requests an int describing the sample playback frequency of the buffer data. Must be >=1.


AudioClip Buffer Size

<ClipSampleCount> int Salsa.clipSampleCount()

Analogous with AudioClip.samples

Requests an int describing the size of the buffer data. Must be >=1 and >Salsa.sampleSize.


AudioClip Data Fetch

<GetClipData> bool Salsa.getClipData(float[] audioData, int offsetPointer)

Analogous with AudioClip.GetData()

Requests a bool describing the success of data retrieval from the buffer data. Requires two parameters:
1) Reference to a Salsa float array to fill with [0.0f..1.0f] data values.
2) An int value representing the start (offset) pointer for data retrieval. This functionality should operate exactly the same way as AudioClip.GetData. Using the offsetPointer as the starting position, fill the supplied float[] audioData array with buffer data.

For example -- assume a buffer containing 10 float elements and a float array of size 3 called bufferData. Internally, Salsa will execute the following:

Salsa.getClipData(bufferData, 3);

Salsa will be expecting to retrieve the following data (marked by 'x'):

                    record head pointer
                    v
buffer: d d d X X X - - - -
              ^
              offsetPointer (buffer index 3)

Current Record Head Pointer

<ClipHeadPointer> int Salsa.clipHeadPointer()

The record head pointer is typically used when a streamed data source is supplied to an AudioSource.AudioClip circular buffer, such as a Unity Microphone clip. This value is only referenced when CrazyMinnow.SALSA.Salsa.autoAdjustMicrophone is true. When an AudioSource is playing a pre-recorded AudioClip file, there is no record head pointer and only a playback pointer and this value (clipHeadPointer) is not used.

Requests an int describing the current recording (or playback) position of the buffer data. AudioClip data is generally expected to be recorded into a circular buffer. This pointer is where Salsa will determine where to offset data retrieval from. In live recorded data, this will be the most recent data available and Salsa will need to retrieve data behind this pointer. It will make its own calculations based off of this pointer value.


Assign a Viseme Trigger

<GetTriggerIndex> int Salsa.getTriggerIndex()

Normally, when audio is processed and analyzed, SALSA will use its analysis to select the appropriate viseme trigger. By default, this is based on ascending trigger levels associated with rising/falling amplitude values. There may be times when this is not desirable, such as when analysis is not based on amplitude and trigger selection may need to be more deliberately implemented. In this instance, you can disable the use of audio analyzers or external analysis and get your own trigger value based on some other algorithm, such as realtime phoneme processing.

In order to properly implement this in SALSA, you will likely want to set CrazyMinnow.SALSA.Salsa.useAudioAnalysis = false; then set this delegate to your trigger selection custom code.

For example: re-map default delegate for random trigger selection:

CrazyMinnow.SALSA.Salsa salsa;

private Start()
{
    salsa = GetComponent<CrazyMinnow.SALSA.Salsa>();
    salsa.useAudioAnalysis = false;
    salsa.getTriggerIndex = MyCustomTriggerSelection;
}

private int MyCustomTriggerSelection()
{
    return Random.Range(0, salsa.visemes.Count);
}