SALSA Delegate Processing

SALSA 2.4.0 was the first release to support delegates for plugging custom functionality into SALSA. SALSA 2.5.0 extends this capability significantly. By implementing delegate calls, some of SALSA's key internals can be re-routed and it is relatively simple to do.

General How-To Re-route SALSA Processing

1) Create a new custom process to replace SALSA's internal process -- match the parameter signature requirements for the delegate.
2) Ensure the new custom process returns the correct data type.
3) Connect the appropriate delegate to your custom method (usually in your custom code's Start() callback, but technically wherever/whenever you please). You can turn this on/off at any time.

Return SALSA to Default Processes

After having re-routed any of SALSA's available internal processes, if you need to re-map SALSA back to its default delegate actions, it requires two steps:

1) Set the delegate you previously mapped to null.
2) Then, simply call Salsa.InitializeDelegates() to re-map any null pointers back to the SALSA defaults.

For example: re-route back to default process for external analysis. (See example for mapping external analysis below for context).

// re-map back to default process for external analysis
private void RemoveExternalAnalysisMapping()
{
    CrazyMinnow.SALSA.Salsa _salsa = GetComponent<CrazyMinnow.SALSA.Salsa>();
    _salsa.getExternalAnalysis = null; // required for salsa to re-map to default.
    _salsa.InitializeDelegates();
}

NOTE: Salsa.InitializeDelegates() only re-maps delegates that are null. It will not re-map delegates pointing to valid methods.

While the Editor is in Play mode, the inspector will display the available delegates and the method name of their current mappings. Any delegates that are not mapped to the default process will be displayed in bold type. In the following image, note the Clip Pointer has been mapped to a method called GetMicRecordPointer.

Delegate Processes and Definitions

Analysis

Audio analysis has two functions...1) get the data to be analyzed, and 2) analyze the data. There are two available delegates that appear similar but have distinct differences. First is the AudioAnalyzer type and it redirects audio analysis to a custom function after SALSA has gathered the data to be analyzed. It requires no additional flags to be set. Second is the GetExternalAnalysis type and it expects that all data acquisition and analysis will be performed externally (in custom code). This option does require a flag to be set to switch SALSA to external analysis mode CrazyMinnow.SALSA.Salsa.useExternalAnalysis = true.

Audio Analysis

<AudioAnalyzer> float Salsa.audioAnalyzer(int channels, float[] audioSampleData)

• SALSA has its own internal audio analysis engine; however, it is simple to replace this with your own custom code. Audio analysis can be customized and called during the normal SALSA processing tick cycle see further reading section: (Custom Audio Analyzers). Utilizing this delegate allows you to let SALSA take care of grabbing the data to be analyzed and lets you perform your own analysis.

NOTE: The difference between external analysis and the audio analyzer is what is being delegated away from SALSA internal code. By delegating external analysis, you are basically telling SALSA to not perform data fetching or analysis. Whereas by delegating the audio analyzer (above), you are telling SALSA to continue to fetch data but have a custom process analyze it.

CrazyMinnow.SALSA.Salsa salsaInstance;
salsaInstance.audioAnalyzer = MyAudioAnalyzer;

float MyAudioAnalyzer(int channelInterleave, float[] audioData)
{
    var dataAnalysis = 0.0f;
    // process data...
    return dataAnalysis;
}

---

External Analysis

<GetExternalAnalysis> float Salsa.getExternalAnalysis()

Previously, it was required to keep the CrazyMinnow.SALSA.Salsa.analysisValue field updated (usually in the Update() loop of your custom class), which meant every frame -- even though SALSA's operations operate far less frequently. Now, it is preferred to re-map this delegate process to your own code and SALSA can then fetch it on demand.

This delegate allows the Salsa.analysisValue to be retrieved from custom code instead of having to supply the value to SALSA. To enable this processing, it is required to turn on External Analysis in the SALSA configuration.
CrazyMinnow.SALSA.Salsa.useExternalAnalysis = true;

NOTE: The difference between external analysis and the audio analyzer is what is being delegated away from SALSA internal code. By delegating external analysis, you are basically telling SALSA to not perform data fetching or analysis. Whereas by delegating the audio analyzer (above), you are telling SALSA to continue to fetch data but have a custom process analyze it.

Example: Random Data Analysis Generation

private Start()
{
    Salsa _salsa = GetComponent<Salsa>();
    _salsa.getExternalAnalysis = MyCustomAnalysisMethod;
}

private float MyCustomAnalysisMethod()
{
    return Random.Range(0f,1f);
}

---

For a more advanced example, using a custom filter chain, please see this custom filter chain documentation.

---

AudioClip Functions

In order to facilitate custom audio buffers, the following delegates are available if you want SALSA to analyze your data. These delegates use the AudioClip API by default. Utilizing a custom audio buffer requires all of the following hooks (modeled on the AudioClip API) to be implemented into your buffer API:

AudioClip Channel Count

<ClipChannels> int Salsa.clipChannels()

Analogous with AudioClip.channels

Requests an int describing the number of audio channels interleaved in the buffer data. Must be >=1.
NOTE: by default, SALSA only processes the first (left) channel data.

---

AudioClip Recording Frequency

<ClipFrequency> int Salsa.clipFrequency()

Analogous with AudioClip.frequency

Requests an int describing the sample playback frequency of the buffer data. Must be >=1.

---

AudioClip Buffer Size

<ClipSampleCount> int Salsa.clipSampleCount()

Analogous with AudioClip.samples

Requests an int describing the size of the buffer data. Must be >=1 and >Salsa.sampleSize.

---

AudioClip Data Fetch

<GetClipData> bool Salsa.getClipData(float[] audioData, int offsetPointer)

Analogous with AudioClip.GetData()

Requests a bool describing the success of data retrieval from the buffer data. Requires two parameters:
1) Reference to a Salsa float array to fill with [0.0f..1.0f] data values.
2) An int value representing the start (offset) pointer for data retrieval. This functionality should operate exactly the same way as AudioClip.GetData. Using the offsetPointer as the starting position, fill the supplied float[] audioData array with buffer data.

For example -- assume a buffer containing 10 float elements and a float array of size 3 called bufferData. Internally, Salsa will execute the following:

Salsa.getClipData(bufferData, 3);

Salsa will be expecting to retrieve the following data (marked by 'x'):

                    record head pointer
                    v
buffer: d d d X X X - - - -
              ^
              offsetPointer (buffer index 3)

---

Current Record Head Pointer

<ClipHeadPointer> int Salsa.clipHeadPointer()

The record head pointer is typically used when a streamed data source is supplied to an AudioSource.AudioClip circular buffer, such as a Unity Microphone clip. This value is only referenced when CrazyMinnow.SALSA.Salsa.autoAdjustMicrophone is true. When an AudioSource is playing a pre-recorded AudioClip file, there is no record head pointer and only a playback pointer and this value (clipHeadPointer) is not used.

Requests an int describing the current recording (or playback) position of the buffer data. AudioClip data is generally expected to be recorded into a circular buffer. This pointer is where Salsa will determine where to offset data retrieval from. In live recorded data, this will be the most recent data available and Salsa will need to retrieve data behind this pointer. It will make its own calculations based off of this pointer value.

---

Assign a Viseme Trigger

<GetTriggerIndex> int Salsa.getTriggerIndex()

Normally, when audio is processed and analyzed, SALSA will use its analysis to select the appropriate viseme trigger. By default, this is based on ascending trigger levels associated with rising/falling amplitude values. There may be times when this is not desirable, such as when analysis is not based on amplitude and trigger selection may need to be more deliberately implemented. In this instance, you can disable the use of audio analyzers or external analysis and get your own trigger value based on some other algorithm, such as realtime phoneme processing.

In order to properly implement this in SALSA, you will need to set CrazyMinnow.SALSA.Salsa.useAudioAnalysis = false; then set this delegate to your trigger selection custom code.

NOTE: Using this method to change visemes does NOT support Advanced Dynamics. There is no analysis to support Advanced Dynamics so the net effect is an imperceptible animation. Turn off Advanced Dynamics.

To turn visemes off, set the index to -1. Internally, SALSA considers -1 to be SILENCE, no viseme active.

For example: re-map default delegate for random trigger selection:

CrazyMinnow.SALSA.Salsa salsa;

private void Start()
{
    salsa = GetComponent<CrazyMinnow.SALSA.Salsa>();
    salsa.useAudioAnalysis = false;
    salsa.useAdvDyn = false;
    salsa.getTriggerIndex = MyCustomTriggerSelection;
}

private int MyCustomTriggerSelection()
{
    return Random.Range(0, salsa.visemes.Count);
}