Overview for CrazyMinnow.SALSA.Salsa.cs

SALSA LipSync has a great custom Inspector, designed to facilitate all of SALSA's advanced lip-sync functionality. However, there may be times when more complex functionality is desired. For this reason, SALSA has an underlying API, allowing developers to programmatically configure and control SALSA's workflow. Since SALSA LipSync is a complete re-write of the original v1 code, the previous API has changed and is no longer valid.

NOTE: This article details specific fields and methods available. Information on manipulating ExpressionComponent settings in code can be found in the Further Reading article on ExpressionComponents (API section).

Namespace: CrazyMinnow.SALSA

All of the main Suite modules reside in the CrazyMinnow.SALSA namespace. If your custom code cannot find Salsa ensure you have included a reference to the appropriate namespace. Please ensure you understand C# namespaces and their usage. There are a myriad of references available on the Internet discussing the subject. If you fail to reference the appropriate SALSA Suite namespaces, you will not be able to use the SALSA Suite API.

The following properties and methods are available for: CrazyMinnow.SALSA.Salsa.cs

Properties (for CrazyMinnow.SALSA.Salsa configuration)

There are many public fields available in the SALSA LipSync library. Most of these are implemented for SALSA's runtime configuration and control.

References

QueueProcessor queueProcessor

A QueueProcessor is a required reference link needed to initiate priority-based animations. If setting SALSA up in a runtime environment, a QueueProcessor must be instantiated and SALSA must know where it is.

---

Emoter emoter

An optional link to an EmoteR component which SALSA will use to trigger emphasis emotes. If this is configured, SALSA will call the EmoteR method when the emphasiszerTrigger is exceeded. Multiple EmoteRs could be configured for different emphasis moods and then switched based on the demeanor of the character (i.e. happy, sad, mad, etc.).

using CrazyMinnow.SALSA;

public class MyMoodSwitcher
{
    public Salsa salsa;
    public Emoter happyEmoter;
    public Emoter sadEmoter;
    public enum Mood{
        Happy,
        Sad
    }

    public void SwitchMood(Mood newMood)
    {
        switch (newMood)
        {
            case Mood.Happy:
                salsa.emoter = happyEmoter;
                break;
            case Mood.Sad:
                salsa.emoter = sadEmoter;
                break;
            default:
                salsa.emoter = happyEmoter;
        }
    }
}

---

[Range(0f,1f)] float emphasizerTrigger = 0.0f

When an EmoteR is linked to SALSA, when this value is exceeded, SALSA will call Emoter.TriggerLipSyncEmphasis(). Recommend this value be set to 0.0f and the corresponding Emoter.lipsyncEmphasisChance be set to 1.0f.

---

AudioSource audioSrc

SALSA's reference to an AudioSource for data analysis. SALSA requires an AudioSource to analyze unless external analysis is configured. Version 2 no longer provides its own interface to audio play/pause/stop functionality. Instead control of the playing clip is relegated to the AudioSource itself. You can; therefore, control which clip is playing via the AudioSource itself or via SALSA's reference to it.

// controlling the AudioSource/Clip via SALSA's reference to the AudioSource.
CrazyMinnow.SALSA.Salsa salsaInstance;          // reference to SALSA
salsaInstance.audioSrc.clip = yourAudioClip;    // load your AudioClip
salsaInstance.audioSrc.Play();                  // play the clip
salsaInstance.audioSrc.Pause();                 // pause the clip
salsaInstance.audioSrc.Stop();                  // stop the clip

bool waitForAudioSource

Enable this to have SALSA wait for an AudioSource to be instantiated at runtime. SALSA will wait for an AudioSource on the same object as itself or, if configured, will wait for an AudioSource on the audioSourceWaitTarget GameObject.

GameObject audioSourceWaitTarget

As mentioned above, if SALSA is configured to wait for an AudioSource, it will monitor this referenced GameObject (if specified) for an AudioSource. Otherwise, it will monitor the GameObject containing the SALSA component.

---

Operational Settings

bool useAudioAnalysis

This flag tells SALSA to process audio analysis. Generally speaking, this should always be true. However, one use case for not wanting audio analysis is if you are adjusting the dynamics of how triggers are selected and do not need internal or external audio analysis for this function. See Delegate Processing for an example of how/why this might be desirable.

bool useExternalAnalysis

Normally, SALSA will analyze audio from an AudioClip assigned to an AudioSource. However, there are several occasions where SALSA should respond to data analysis provided externally from its internal analysis engine. One key example is while using Text-to-Lipsync, where there is no audio to analyze. Enable this to switch SALSA into a passive mode that expects analysis data to be fed to it via the analysisValue field.

float audioUpdateDelay = 0.08f

SALSA operates on a tick-rate mechanism, specified by this value (in seconds). If this value is set too small (fast tick-rate), animations will appear jittery and will not allow animation dynamics to reach their full potential. Likewise, if this value is set too slow, animation dynamics will be allowed to reach their full potential, but the sampling periods may be to slow to pick up on sufficient speech nuance (less dynamic accuracy). A value between [0.07f and 0.09f] is recommended, but experimentation is encouraged to find the desired look-and-feel.

---

float loCutoff = 0.03f

Low cutoff value is used to provide an easy filter mechanism for the analysis noise floor. Values below this cutoff are treated as 0f by the SALSA engine and do not trigger any visemes.

float hiCutoff = 0.75f

The high cutoff serves as an adjustable upper end boundary of the analysis range. Instead of being a pure cutoff, the analysis range is scaled from the low cutoff (0.0f) to the high cutoff as (1.0f). This allows a linear amplification of the range from loCutoff to hiCutoff. For example, if the default values are chosen, SALSA's analysis engine will spread values from [0.03f to 0.75f] as [0.0f to 1.0f]. This allows an easily tunable range both in the design interface as well as programmatically to adjust for varying audio quality/levels without having to adjust trigger points. Tuning the high cutoff to a lower value will provide more dynamic range (amplification) across the viseme trigger settings.

bool scaleExternalAnalysis = false

This option tells SALSA whether or not it should apply Linear Scaling to externally provided analysis values.

---

[Range(-2048,9048)] int playheadBias = 2800

The amount of 'look-ahead' SALSA should perform on the audio samples. Higher values look further ahead of the playhead pointer, allowing analysis to occur before audio data is played, reducing/eliminating animation lag. Going too far ahead is just as bad as being behind. It is usually recommended to leave this value at defaults and enable the autoAdjustAnalysis flag.

bool autoAdjustAnalysis = true

Enables an auto-adjustment to the playheadBias and sampleSize. It is recommended to leave this flag enabled (true).

bool autoAdjustMicrophone

This flag enables a different algorithm for auto-adjusting the playheadBias. Microphone data has tighter tolerances and due to its live nature, it is not possible to look too far into the future. It is recommended to allow SALSA to control the best outcome for the playheadBias while using a microphone.

string microphone // DEPRECATED!!!

SALSA needs to know the microphone name that is being used in order to read its telemetry data.

int sampleSize = 512

The amount of data SALSA reads on each tick. Larger amounts of data are recommended for higher recorded sample rates and smaller data sizes for lower recorded sample rates. It is recommended to let SALSA adjust this value by enabling the autoAdjustAnalysis flag.

int silencePulseThreshold (min 1)

Configure the number of (update delay) cycles SALSA uses to determine if silence is present. The lowest setting is (int) 1, indicating SALSA must find silence in at least 1 audio analysis pulse. SALSA uses updateDelay to specify the amount of time between audio analysis processing (a pulse) cycles. For example: If updateDelay is set to .08f and a pulse threshold of 3 is configured, SALSA must see silence for 3 processing cycles, encompassing 3 * 0.08f seconds [0.24f seconds]. Experiment with your audio to discover what value works well for your situation.

bool IsSALSAing

Returns a state status of whether SALSA is processing non-silence analysis data, either from internal or externally provided sources. This value is computed regardless of the state of the AudioSource. If the AudioSource is playing and the dialogue has gaps of silence, depending on the silencePulseThreshold value, IsSALSAing will return false if appropriate gaps are detected. For passive detection of SALSA's processing state, please see the events section below.

---

List visemes

The list of viseme expression configurations.

---

bool useAdvDyn

Flag to enable Advanced Dynamics in SALSA processing. It is recommended to use Advanced Dynamics in most cases. Advanced Dyanmics should not be used in instances where there are no variations possible, such as single-frame animations.

[Range(0f,1f)] float advDynPrimaryBias = .50f

Specifies the minimum amount of animation to use if Advanced Dynamics is enabled (as a percentage).

bool useAdvDynJitter

Flag to enable the use of jitter variation in Advanced Dynamics. This value will further vary the dyanamics processed if Advanced Dynamics is enabled.

[Range(0f,1f)] float advDynJitterAmount = .25f

Maximum amount of jitter to apply if enabled and triggered.

[Range(0f,1f)] float advDynJitterProb = .25f

Probability jitter amount will be triggered if enabled.

---

bool useAdvDynSecondaryMix

Flag enables Secondary Mix dynamics.

[Range(0f, 1f)] float advDynSecondaryMix = 0f

When Secondary Mix is enabled, this value indicates the minimum amount of animation that should be used from the secondary visemes. It is recommended to leave the settings for secondary viseme timings/easings at default values; however, if you wish to change them, use the following:

float dynaVariOn = 0.08f
float dynaVariHold = 0.00f
float dynaVariOff = 0.06f
LerpEasings.EasingType dynaVariEasing = LerpEasings.EasingType.CubicOut

---

bool useAdvDynRollback

Enable rollback blending for switcher type controllers.

[Range(0f, 1f)] float advDynRollback = .3f

When enabled, rollback blending will use this value as a percentage of current position to rollback to.

[Range(0f, 1f)] float globalFrac = 1.0f

Global fractional value allows adjusting the overall amount or max value (as a percentage) for all visemes.

bool useTimingsOverride

In order to implement the globalDurON and globalDurOFF timings, set this value to true.

float globalDurON = 0.10f

This is a global override for the ON animation timing values for all visemes. This setting typically does not need to be changed once SALSA has been tuned to designer preference. However, it may be desirable to change this dynamically or set it up at runtime. REQUIRES: useTimingsOverride = true;

float globalDurOFF = 0.10f

This is a global override for the OFF animation timing values for all visemes. This setting typically does not need to be changed once SALSA has been tuned to designer preference. However, it may be desirable to change this dynamically or set it up at runtime. REQUIRES: useTimingsOverride = true;

---

Processed Values (input/output)

float analysisValue (input field)

When useExternalAnalysis (see above) is enabled, the programmer can feed analysis data to SALSA via this variable. Analysis data should be in the normalized data format [0f..1f]. NOTE: SALSA will only check this value if useExternalAnalysis is true and only at intervals specified by audioUpdateDelay seconds.

float CachedAnalysisValue (output field)

Returns the post-analysis value SALSA is using for processing viseme triggers. This is the value SALSA uses regardless of whether data analysis was performed internally or was externally fed via analysisValue when useExternalAnalysis is true and SALSA is processing during its tick-rate.

useExternalAnalysis = true;
var someValue = Random.Range(0.0f, 1.0f);
analysisValue = someValue;

// The CachedAnalysisValue will be equal to someValue during SALSA's LateUpdate() cycle when audioUpdateDelay seconds has passed.
Debug.Log("Cached analysis value: " + CachedAnalysisValue);

---

int microphoneRecordHeadPointer (input field)

When autoAdjustMicrophone is enabled, SALSA will read this value as the current data array position pointer and pull data from the clip using calculations based on this field. This field must be updated from a process external to SALSA. SALSA no longer contains a reference to the Unity Microphone class and can therefore, no longer get this pointer position on its own. As of SALSA v2.5.0, there is a new delegate function that looks for a pointer value. By default, it references this value microphoneRecordHeadPointer. Versions of SalsaMicInput prior to v2.5.0 actively injected a value into this field using custom code in an Update loop. Now, SalsaMicInput (v2.5.0) leverages the delegate mapping for Salsa.clipHeadPointer() and SALSA polls this value for the current pointer position as needed. It is still possible to manually update this value if desired.

CrazyMinnow.SALSA.Salsa salsa;
private void Start()
{
    salsa.autoAdjustMicrophone = true; // if setting up manually
}

// generally in an update loop...
private void Update()
{
    salsa.microphoneRecordHeadPointer = YourCustomPointerMethod();
}

private int YourCustomPointerMethod()
{
    int pointerValue;
    // calculate your pointerValue...
    return pointerValue;
}

NOTE: In version 2.5.0+ it is not necessary to manually feed a pointer value to SALSA as the calculation can be delegated so SALSA can actively poll the value as needed using the CrazyMinnow.SALSA.Salsa.clipHeadPointer() delegate value. See Salsa.clipHeadPointer in the following document for more information: Delegate Plugins

CrazyMinnow.SALSA.Salsa salsa;

// generally in Start
private void Start()
{
    salsa.autoAdjustMicrophone = true; // if setting up manually
    salsa.clipHeadPointer = YourCustomPointerMethod;
    // SALSA will now poll your custom method when it needs to fetch
    // the pointer value.
}

private int YourCustomPointerMethod()
{
    int pointerValue;
    // calculate your pointerValue...
    return pointerValue;
}

---

int TriggeredIndex (output field)

Returns the trigger SALSA processed this tick. If the value was relative silence, TriggeredIndex will return a -1.

---

Methods

void AdjustAnalysisSettings()

Calculates and sets audio analysis settings based on the current available AudioSource and AudioClip information. Call this method when the AudioClip changes if clips are not using standardized sample frequencies. It is not necessary (or recommended) to call this method if autoAdjustAnalysis is enabled.

void UpdateExpressionControllers()

To implement the underlying data structure and also allow serialization to be maintained in the Inspector, we chose to implement a helper class for defining ExpressionComponent data. This allowed us to build a rich, custom inspector and also maintain a generic component implementation on the back-end where the QueueProcessor handled all objects the same way and in priority fashion. Due to this implementation, building an ExpressionComponent at runtime requires the associated controllers to be created and updated according to the associated data. Essentially, this bakes the controller helper data (used by the inspector or configured at runtime) into the controller data object(s), creating the necessary derived data and linkages to the proper ExpressionControllers. This should be called after the visemes and viseme components have been programmatically constructed. See the SALSA API-Example for a detailed configuration example.

void SortLipsyncExpressionsOnTrigger()

If triggers have been programmatically configured on visemes and no consideration for value of the trigger order has been initiated, call this method to sort the visemes based on trigger value. For proper operation, triggers must be configured in ascending order.

void TurnOffAll()

Turn off all configured/registered visemes. This is normally called in OnEnable(), but it may be desirable to call it in other circumstances. NOTE: To smoothly turn visemes off, ensure Salsa.visemes[expressionIndex].[expressionIndex].expData.components[componentIndex].isSmoothDisable is true prior to calling this method. Setting isSmoothDisable to false will turn all visemes off instantly.

CrazyMinnow.SALSA.Salsa salsaInstance;
salsaInstance.TurnOffAll();

void DistributeTriggers(LerpEasings.EasingType easing)

Processes viseme order and distributes triggers based on easing type parameter. Call this method for a standard trigger distribution (based on easing). Should be called after viseme configuration unless triggers have been manually set (not recommended).

Static Expression Capture

In Development: expected v2.6

void CaptureStaticExpression()

Call this method to capture all non-zero blendshapes from all SkinnedMeshRenderer objects attached to child objects under SALSA. This operation is performed automatically, in the Awake() callback, if the option Salsa.captureStaticExpression == true. Components for each discovered blendshape will be created and registered persistently in the QueueProcessor 'head' queue. SALSA, EmoteR and Eyes will smoothly animate back to these components if conflicting components are registered.

void LoadStaticExpression(bool isSmooth = true)

Manually loads a previously captured static expression (automatically captured at startup if Salsa.captureStaticExpression == true). If a static expression has not been captured, a snapshot can be taken using Salsa.CaptureStaticExpression() (see associated documentation). If isSmooth == true, the expression will be smoothly animated ON using the preconfigured duration value.

CrazyMinnow.SALSA.Salsa salsaInstance;
salsaInstance.staticExpressionDuration = 0.3f; // 0.3 seconds
var useSmoothAnimation = true;
salsaInstance.LoadStaticExpression(useSmoothAnimation);

void RemoveStaticExpression(bool isSmooth = true)

Manually unloads a previously captured static expression. This method can be used to turn off a static expression, smoothly or instantly (isSmooth == false). If a static expression has not been captured, a snapshot can be taken using Salsa.CaptureStaticExpression() (see associated documentation). If isSmooth == true, the expression will be smoothly animated OFF using the preconfigured duration value.

CrazyMinnow.SALSA.Salsa salsaInstance;
salsaInstance.staticExpressionDuration = 0.3f; // 0.3 seconds
// smoothly animate the expression off...
salsaInstance.RemoveStaticExpression(true);
// or..instantly turn the expression off...
salsaInstance.RemoveStaticExpression(false);

Copy Viseme Configuration From Different SALSA Instance

In Development: expected v2.6

void CopyVisemeConfigurationFromSalsa(Salsa salsaFrom, Dictionary smrMap = null, GameObject skeletonRoot = null, string srcPrefix = "", string dstPrefix = "")

Copy the viseme configuration from another SALSA instance. This will copy the configuration and attempt to remap the corresponding blendshape and/or bone controllers to this SALSA's hierarchy. To attempt an auto-copy, leave smrMap and skeletonRoot as null. This should work fine assuming the underlying mesh and rig structures are named identically between SALSA hierarchies. If the underlying structures are named differently, you may supply a mapping of SkinnedMeshRenderer objects and a root starting point for the bone search. The dictionary should be created in Key=fromSMR, Value=toSMR fashion. Additionally, if bones are present and have consistent source and destination prefix mismatches, two strings can be supplied to replace the prefix in the hierarchy search. See Using documentation for more information.

Events

SALSA has several (C#) events that fire as SALSA processes. See the Event documentation for more details.

Delegates

SALSA has several delegate maps that allow for redirecting certain processing to your own custom code. See the Delegate Processing documentation for more details.

SALSAing (Lipsyncing) Status

SALSA's internal processing state is true while SALSAing (lipsyncing) and subsequently false when it is not SALSAing (silent). These events provide a passive means of determining when SALSA is actively processing non-silence analysis. Please read the detailed document on SALSA Suite events for more information.

NOTE: It is also possible to actively poll SALSA to detect SALSA's processing state by checking the Salsa.IsSALSAing boolean property, please see the Operational Settings above.

An additional event is now available (v2.4.1+) VisemeTriggered when a valid (non-silence) viseme trigger has been selected. This event occurs after audio processing. The viseme index is provided in the event arguments. Please read the detailed document on SALSA Suite events for more information.