Overview for Salsa.cs

SALSA LipSync has a great custom Inspector, designed to facilitate all of SALSA's advanced lip-sync functionality. However, there may be times when more complex functionality is desired. For this reason, SALSA has an underlying API, allowing developers to programmatically configure and control SALSA's workflow. Since SALSA LipSync is a complete re-write of the original v1 code, the previous API has changed and is no longer valid.

The following properties and methods are available for: CrazyMinnow.SALSA.Salsa.cs

Properties (for SALSA configuration)

There are many public fields available in the SALSA LipSync library. Most of these are implemented for SALSA's runtime configuration and control.

References

QueueProcessor queueProcessor

A QueueProcessor is a required reference link needed to initiate priority-based animations. If setting SALSA up in a runtime environment, a QueueProcessor must be instantiated and SALSA must know where it is.


Emoter emoter

An optional link to an EmoteR component which SALSA will use to trigger emphasis emotes. If this is configured, SALSA will call the EmoteR method when the emphasiszerTrigger is exceeded.

[Range(0f,1f)] float emphasizerTrigger = 0.70f

When an EmoteR is linked to SALSA when this value is exceeded, SALSA will call Emoter.TriggerLipSyncEmphasis.


AudioSource audioSrc

SALSA's reference to an AudioSource for data analysis. SALSA requires an AudioSource to analyze unless external analysis is configured. Version 2 no longer provides its own interface to audio play/pause/stop functionality. Instead control of the playing clip is relegated to the AudioSource itself. You can; therefore, control which clip is playing via the AudioSource itself or via SALSA's reference to it.

    // controlling the AudioSource/Clip via SALSA's reference to the AudioSource.
    CrazyMinnow.SALSA.Salsa salsaInstance;          // reference to SALSA
    salsaInstance.audioSrc.clip = yourAudioClip;    // load your AudioClip
    salsaInstance.audioSrc.Play();                  // play the clip
    salsaInstance.audioSrc.Pause();                 // pause the clip
    salsaInstance.audioSrc.Stop();                  // stop the clip

bool waitForAudioSource

Enable this to have SALSA wait for an AudioSource to be instantiated at runtime. SALSA will wait for an AudioSource on the same object as itself or, if configured, will wait for an AudioSource on the audioSourceWaitTarget GameObject.

GameObject audioSourceWaitTarget

As mentioned above, if SALSA is configured to wait for an AudioSource, it will monitor this referenced GameObject (if specified) for an AudioSource. Otherwise, it will monitor the GameObject containing the SALSA component.


Operational Settings

bool useExternalAnalysis

Normally, SALSA will analyze audio from an AudioClip assigned to an AudioSource. However, there are several occasions where SALSA should respond to data analysis provided externally from its internal analysis engine. One key example is while using Text-to-Lipsync, where there is no audio to analyze. Enable this to switch SALSA into a passive mode that expects analysis data to be fed to it via the analysisValue field.

float audioUpdateDelay = 0.08f

SALSA operates on a tick-rate mechanism, specified by this value (in seconds). If this value is set too small (fast tick-rate), animations will appear jittery and will not allow animation dynamics to reach their full potential. Likewise, if this value is set too slow, animation dynamics will be allowed to reach their full potential, but the sampling periods may be to slow to pick up on sufficient speech nuance (less dynamic accuracy). A value between [0.07f and 0.09f] is recommended, but experimentation is encouraged to find the desired look-and-feel.


float loCutoff = 0.03f

Low cutoff value is used to provide an easy filter mechanism for the analysis noise floor. Values below this cutoff are treated as 0f by the SALSA engine and do not trigger any visemes.

float hiCutoff = 0.75f

The high cutoff serves as an adjustable upper end boundary of the analysis range. Instead of being a pure cutoff, the analysis range is scaled from the low cutoff (0.0f) to the high cutoff as (1.0f). This allows a linear amplification of the range from loCutoff to hiCutoff. For example, if the default values are chosen, SALSA's analysis engine will spread values from [0.03f to 0.75f] as [0.0f to 1.0f]. This allows an easily tunable range both in the design interface as well as programmatically to adjust for varying audio quality/levels without having to adjust trigger points. Tuning the high cutoff to a lower value will provide more dynamic range (amplification) across the viseme trigger settings.

bool scaleExternalAnalysis = false

This option tells SALSA whether or not it should apply Linear Scaling to externally provided analysis values.


[Range(-2048,9048)] int playheadBias = 2800

The amount of 'look-ahead' SALSA should perform on the audio samples. Higher values look further ahead of the playhead pointer, allowing analysis to occur before audio data is played, reducing/eliminating animation lag. Going too far ahead is just as bad as being behind. It is usually recommended to leave this value at defaults and enable the autoAdjustAnalysis flag.

bool autoAdjustAnalysis = true

Enables an auto-adjustment to the playheadBias and sampleSize. It is recommended to leave this flag enabled (true).

bool autoAdjustMicrophone

This flag enables a different algorithm for auto-adjusting the playheadBias. Microphone data has tighter tolerances and due to its live nature, it is not possible to look too far into the future. It is recommended to allow SALSA to control the best outcome for the playheadBias while using a microphone.

string microphone

SALSA needs to know the microphone name that is being used in order to read its telemetry data.

int sampleSize = 512

The amount of data SALSA reads on each tick. Larger amounts of data are recommended for higher recorded sample rates and smaller data sizes for lower recorded sample rates. It is recommended to let SALSA adjust this value by enabling the autoAdjustAnalysis flag.

int silencePulseThreshold (min 1)

Configure the number of (update delay) cycles SALSA uses to determine if silence is present. The lowest setting is (int) 1, indicating SALSA must find silence in at least 1 audio analysis pulse. SALSA uses updateDelay to specify the amount of time between audio analysis processing (a pulse) cycles. For example: If updateDelay is set to .08f and a pulse threshold of 3 is configured, SALSA must see silence for 3 processing cycles, encompassing 3 * 0.08f seconds [0.24f seconds]. Experiment with your audio to discover what value works well for your situation.
salsa silence threshold

bool IsSALSAing

Returns a state status of whether SALSA is processing non-silence analysis data, either from internal or externally provided sources. This value is computed regardless of the state of the AudioSource. If the AudioSource is playing and the dialogue has gaps of silence, depending on the silencePulseThreshold value, IsSALSAing will return false if appropriate gaps are detected. For passive detection of SALSA's processing state, please see the events section below.


List visemes

The list of viseme expression configurations.


bool useAdvDyn

Flag to enable Advanced Dynamics in SALSA processing. It is recommended to use Advanced Dynamics in most cases. Advanced Dyanmics should not be used in instances where there are no variations possible, such as single-frame animations.

[Range(0f,1f)] float advDynPrimaryBias = .50f

Specifies the minimum amount of animation to use if Advanced Dynamics is enabled (as a percentage).

bool useAdvDynJitter

Flag to enable the use of jitter variation in Advanced Dynamics. This value will further vary the dyanamics processed if Advanced Dynamics is enabled.

[Range(0f,1f)] float advDynJitterAmount = .25f

Maximum amount of jitter to apply if enabled and triggered.

[Range(0f,1f)] float advDynJitterProb = .25f

Probability jitter amount will be triggered if enabled.


bool useAdvDynSecondaryMix

Flag enables Secondary Mix dynamics.

[Range(0f, 1f)] float advDynSecondaryMix = 0f

When Secondary Mix is enabled, this value indicates the minimum amount of animation that should be used from the secondary visemes. It is recommended to leave the settings for secondary viseme timings/easings at default values; however, if you wish to change them, use the following:

float dynaVariOn = 0.08f
float dynaVariHold = 0.00f
float dynaVariOff = 0.06f
LerpEasings.EasingType dynaVariEasing = LerpEasings.EasingType.CubicOut


bool useAdvDynRollback

Enable rollback blending for switcher type controllers.

[Range(0f, 1f)] float advDynRollback = .3f

When enabled, rollback blending will use this value as a percentage of current position to rollback to.

[Range(0f, 1f)] float globalFrac = 1.0f

Global fractional value allows adjusting the overall amount or max value (as a percentage) for all visemes.


Processed Values (input/output)

float analysisValue (input field)

When useExternalAnalysis (see above) is enabled, the programmer can feed analysis data to SALSA via this variable. Analysis data should be in the normalized data format [0f..1f]. NOTE: SALSA will only check this value if useExternalAnalysis is true and only at intervals specified by audioUpdateDelay seconds.

float CachedAnalysisValue (output field)

Returns the post-analysis value SALSA is using for processing viseme triggers. This is the value SALSA uses regardless of whether data analysis was performed internally or was externally fed via analysisValue when useExternalAnalysis is true and SALSA is processing during its tick-rate.

useExternalAnalysis = true;
var someValue = Random.Range(0.0f, 1.0f);
analysisValue = someValue;

// The CachedAnalysisValue will be equal to someValue during SALSA's LateUpdate() cycle when audioUpdateDelay seconds has passed.
Debug.Log("Cached analysis value: " + CachedAnalysisValue);

int TriggeredIndex (output field)

Returns the trigger SALSA processed this tick. If the value was relative silence, TriggeredIndex will return a -1.


Methods

void AdjustAnalysisSettings()

Calculates and sets audio analysis settings based on the current available AudioSource and AudioClip information. Call this method when the AudioClip changes if clips are not using standardized sample frequencies. It is not necessary (or recommended) to call this method if autoAdjustAnalysis is enabled.

void UpdateExpressionControllers()

To implement the underlying data structure and also allow serialization to be maintained in the Inspector, we chose to implement a helper class for defining ExpressionComponent data. This allowed us to build a rich, custom inspector and also maintain a generic component implementation on the back-end where the QueueProcessor handled all objects the same way and in priority fashion. Due to this implementation, building an ExpressionComponent at runtime requires the associated controllers to be created and updated according to the associated data. Essentially, this bakes the controller helper data (used by the inspector or configured at runtime) into the controller data object(s), creating the necessary derived data and linkages to the proper ExpressionControllers. This should be called after the visemes and viseme components have been programmatically constructed. See the OneClickBoxhead.cs example for a detailed configuration example.

void SortLipsyncExpressionsOnTrigger()

If triggers have been programmatically configured on visemes and no consideration for value of the trigger order has been initiated, call this method to sort the visemes based on trigger value. For proper operation, triggers must be configured in ascending order.

void TurnOffAll()

Turn off all configured/registered visemes. This is normally called in OnEnable(), but it may be desirable to call it in other circumstances. NOTE: to smoothly turn visemes off, ensure Salsa.visemes[expressionIndex].[expressionIndex].expData.components[componentIndex].isSmoothDisable is true prior to calling this method. Setting isSmoothDisable to false will turn all visemes off instantly.

void DistributeTriggers(LerpEasings.EasingType easing)

Processes viseme order and distributes triggers based on easing type parameter. Call this method for a standard trigger distribution (based on easing). Should be called after viseme configuration unless triggers have been manually set (not recommended).

Events

SALSA has two available (C#) events that fire when Salsa.IsSALSAing is true and subsequently when it is false. These events provide a passive means of determining when SALSA is actively processing non-silence analysis. Please read the detailed document on SALSA Suite events for more information.

NOTE: for active detection of SALSA's processing state, please see the Operational Settings above.