SALSA Delegate Processing

SALSA 2.4.0 was the first release to support delegates for plugging custom functionality into SALSA. SALSA 2.5.0 extends this capability significantly. By implementing delegate calls, some of SALSA's key internals can be re-routed and it is relatively simple to do.

General How-To Re-route SALSA Processing

1) Create a new custom process to replace SALSA's internal process -- match the parameter signature requirements for the delegate.
2) Ensure the new custom process returns the correct data type.
3) Connect the appropriate delegate to your custom method (usually in your custom code's Start() callback, but technically wherever/whenever you please). You can turn this on/off at any time.

If you need to re-map SALSA's default delegate actions, it requires two steps:

1) Set the delegate you have re-mapped to null.
2) Then, simply call Salsa.InitializeDelegates() to re-map to the defaults.

For example: re-map default delegate for external analysis (see example for mapping external analysis below for context)

// re-map default delegate for external analysis
private void RemoveExternalAnalysisMapping()
{
    CrazyMinnow.SALSA.Salsa _salsa = GetComponent<CrazyMinnow.SALSA.Salsa>();
    _salsa.getExternalAnalysis = null; // required for salsa to re-map to default.
    _salsa.InitializeDelegates();
}

NOTE: Salsa.InitializeDelegates() only re-maps delegates that are null. It will not re-map delegates pointing to valid methods.

While the Editor is in Play mode, the inspector will display the available delegates and the method name of their current mappings. Any delegates that are not mapped to the default process will be displayed in bold type. In the following image, note the Clip Pointer has been mapped to a method called GetMicRecordPointer.
delegate mapping display

Delegate Processes and Definitions

Analysis

Audio analysis has two functions...1) get the data to be analyzed, and 2) analyze the data. There are two available delegates that appear similar but have distinct differences. First is the AudioAnalyzer type and it redirects audio analysis to a custom function after SALSA has gathered the data to be analyzed. It requires no additional flags to be set. Second is the GetExternalAnalysis type and it expects that all data acquisition and analysis will be performed externally (in custom code). This option does require a flag to be set to switch SALSA to external analysis mode CrazyMinnow.SALSA.Salsa.useExternalAnalysis = true.

Audio Analysis

AudioAnalyzer float Salsa.audioAnalyzer()

  • SALSA has its own internal audio analysis engine; however, it is simple to replace this with your own custom code. Audio analysis can be customized and called during the normal SALSA processing tick cycle see further reading section: (Custom Audio Analyzers). Utilizing this delegate allows you to let SALSA take care of grabbing the data to be analyzed and lets you perform your own analysis.

External Analysis

GetExternalAnalysis float Salsa.getExternalAnalysis()

This delegate allows the Salsa.analysisValue to be retrieved from custom code instead of having to supply the value to SALSA. To enable this processing, it is required to set CrazyMinnow.SALSA.Salsa.useExternalAnalysis = true.

Previously, it was required to keep the CrazyMinnow.SALSA.Salsa.analysisValue value updated (usually in the Update() loop of your custom class), which meant every frame -- even though SALSA's operations operate far less frequently. Now it is preferred to re-map this delegate field to your own code and SALSA can then fetch it on demand.

NOTE: The difference between external analysis and the audio analyzer is what is being delegated away from SALSA internal code. By delegating external analysis, you are basically telling SALSA to not perform data fetching or analysis. Whereas by delegating the audio analyzer (above), you are telling SALSA to continue to fetch data but have a custom process analyze it.

Example:

private Start()
{
    Salsa _salsa = GetComponent<Salsa>();
    _salsa.getExternalAnalysis = MyCustomAnalysisMethod;
}

private float MyCustomAnalysisMethod()
{
    return Random.Range(0f,1f);
}

AudioClip Functions

In order to facilitate custom audio buffers, the following delegates are available. These delegates use the AudioClip API by default. Utilizing a custom audio buffer requires all of the following hooks (modeled on the AudioClip API) to be implemented into your buffer API:

AudioClip Channel Count

ClipChannels int Salsa.clipChannels()

Analogous with AudioClip.channels

Requests an int describing the number of audio channels interleaved in the buffer data. Must be >=1.
NOTE: by default, SALSA only processes the first (left) channel data.


AudioClip Recording Frequency

ClipFrequency int Salsa.clipFrequency()

Analogous with AudioClip.frequency

Requests an int describing the sample playback frequency of the buffer data. Must be >=1.


AudioClip Buffer Size

ClipSampleCount int Salsa.clipSampleCount()

Analogous with AudioClip.samples

Requests an int describing the size of the buffer data. Must be >=1 and >Salsa.sampleSize.


AudioClip Data Fetch

GetClipData bool Salsa.getClipData(float[] audioData, int offsetPointer)

Analogous with AudioClip.GetData()

Requests a bool describing the success of data retrieval from the buffer data. Requires two parameters:
1) Reference to a Salsa float array to fill with [0.0f..1.0f] data values.
2) An int value representing the start (offset) pointer for data retrieval. This functionality should operate exactly the same way as AudioClip.GetData. Using the offsetPointer as the starting position, fill the supplied float[] audioData array with buffer data.

For example -- assume a buffer containing 10 float elements and a float array of size 3 called bufferData. Internally, Salsa will execute the following:

Salsa.getClipData(bufferData, 3);

Salsa will be expecting to retrieve the following data (marked by 'x'):

                    record head pointer
                    v
buffer: d d d X X X - - - -
              ^
              offsetPointer (buffer index 3)

Current Record Head Pointer

ClipHeadPointer int Salsa.clipHeadPointer()

The record head pointer is typically used when a streamed data source is supplied to an AudioSource.AudioClip circular buffer, such as a Unity Microphone clip. This value is only referenced when CrazyMinnow.SALSA.Salsa.autoAdjustMicrophone is true. When an AudioSource is playing a pre-recorded AudioClip file, there is no record head pointer and only a playback pointer and this value (clipHeadPointer) is not used.

Requests an int describing the current recording (or playback) position of the buffer data. AudioClip data is generally expected to be recorded into a circular buffer. This pointer is where Salsa will determine where to offset data retrieval from. In live recorded data, this will be the most recent data available and Salsa will need to retrieve data behind this pointer. It will make its own calculations based off of this pointer value.


Assign a Viseme Trigger

GetTriggerIndex int Salsa.getTriggerIndex()

Normally, when audio is processed and analyzed, SALSA will then use the analysis to select the appropriate viseme trigger based on ascending trigger levels such as rising/falling amplitude values. There may be times when this is not desirable, such as when analysis is not based on amplitude and trigger selection may need to be more deliberately implemented. In this instance, you can disable the use of audio analyzers or external analysis and get your own trigger value based on some other algorithm, such as realtime phoneme processing.

In order to properly implement this in SALSA, you will likely want to set CrazyMinnow.SALSA.Salsa.useAudioAnalysis = false; then set this delegate to your trigger selection custom code.

For example: re-map default delegate for trigger selection:

private Start()
{
    CrazyMinnow.SALSA.Salsa salsa = GetComponent<CrazyMinnow.SALSA.Salsa>();
    salsa.useAudioAnalysis = false;
    salsa.getTriggerIndex = MyCustomTriggerSelection;
}

private int MyCustomTriggerSelection()
{
    return Random.Range(0, salsa.visemes.Count);
}