ExpressionComponent (class)

Each main expression type (LipsyncExpression, EmoteExpression, EyeExpression, etc.) works hand-in-hand with a SALSA LipSync Suite component (SALSA, EmoteR, RandomEyes). Each is a collection of building-blocks called ExpressionComponents. While the expressions themselves have differing qualities and settings to support their particular type of expression, each ExpressionComponent is common across all expression types.

While some configuration details may be hidden in the components of each expression type, the underlying ExpressionComponent data structure is the same. For example, SALSA utilizes One-Way handlers to control its animations. One-Way handlers do not utilize the "Hold" animation timing, therefore, "Hold" is not displayed in the component settings.

In the inspector, an expression (Viseme, Emote, EyeMovement) requires at least one ExpressionComponent (component). Here is a component on a EmoteR emote (notice there is a "Hold" setting in the animation timings).

ExpressionComponent Breakdown

The base component elements are as follows:

  1. Enable/disable a component. Disable a component by removing the tick in the checkbox on the left of the header. This will disable the component on all "activating" calls. Deactivating calls will still initiate a deactivation on the component. This is to ensure all components are able to turn off if they were on when the component is deactivated. When deactivated, a component's box color will turn a light red color and the title will indicated "(disabled)". Below are two components configured on an EmoteR emote, the first is disabled.

  2. The Header Bar displays component information, the name and controller type for easy identification when collapsed. Name is simply for organization purposes, similar to the name field for the viseme itself. It is not used for any purpose by the SALSA engine. You may change the default name or leave it as-is.

  3. Indicator for Controller type used for the component.

  4. Copy/Paste action:

    • Copy mode: select a component to copy.
    • Paste mode: paste a previously copied component's values to another component. Or cancel the copy action on the copied component.
  5. Delete this component.

  6. Each component has a Name field that is strictly used for organizational purposes and is not used by any script component included within SALSA LipSync. Give it a friendly name for reference.

  7. Set animation timings. All options may not be available in all modules based on applicability. i.e. SALSA does not offer a HOLD timing because it controls the visemes in a one-way manner. There are a couple of caveats to consider for these timings. Regardless of the starting position, a full round-trip animation, if unimpeded, will take the combined time of On/Hold/Off. For example, assume animation timings of 1.0/0.5/1.0: if an animation begins and it is already half-way to its On extent (it is overridding another animation), it will still take 1.0 seconds to complete the On-animation. It will then hold this position for 0.5 seconds and subsequently take 1.0 seconds to transition to Off (2.5 seconds).

  8. Easing type for the animation - similar to web-based animation easings. Easing determines the attack/decay of the animation and there are several options available for use. These settings follow the same basic rules of the common web-based animation easing timings. For lip-synchronization, the more natural 'look' is a fast start, slow finish, an "out"-based easing. The recommended setting is cubicOut -- good fast to slow progress with a low calculation footprint.

  9. Controller Type: Shape (blendshape)

  10. Controller Type options

  11. SkinnedMeshRenderer reference for shape controller type.

  12. Blendshape selection

    Over-driving Under-driving Blendshapes
  13. Minimum shape value (animation OFF -- usually 0f).

    NOTE: As of version 2.3.0, Shape component types now support over/under-driving the shape values. This is only supported in Unity version 2018.3+ and requires disabling the legacy blendshape clamping option in Project Settings > Player. To support this functionality, the Inspector has been switched to a float value (with no clamping limits). Additionally, the display value has been changed to represent the blendshape value desired rather than the underlying fractional value (i.e. 53.4 vs 0.534). Under the hood the value remains fractional for backwards compatibility with existing projects. Therefore, it is still required to use the fractional value when scripting against the API.

  14. Maximum shape value at full (animation ON).

    NOTE: As of version 2.3.0, Shape component types now support over/under-driving the shape values. This is only supported in Unity version 2018.3+ and requires disabling the legacy blendshape clamping option in Project Settings > Player. To support this functionality, the Inspector has been switched to a float value (with no clamping limits). Additionally, the display value has been changed to represent the blendshape value desired rather than the underlying fractional value (i.e. 53.4 vs 0.534). Under the hood the value remains fractional for backwards compatibility with existing projects. Therefore, it is still required to use the fractional value when scripting against the API.

  15. Controller Type: Bone/Transform

  16. Transform reference for bone controller type.

  17. Return to initial stored transform settings. This value is saved immediately when the bone transform is linked to the reference slot (16).

  18. Shortcut to adjust the transform for (Min) or (Max).

  19. Minimum transform setting (animation OFF) -- usually the same as the initial transform settings (17).

  20. Maximum transform setting (animation ON)

  21. Select if externally controlled by another animator in the Update() loop (i.e. Mecanim). > NOTE: When selected, another option is available: Offset -- applies the configured animation as an offset to the externally controlled animation. Referential verses absolute.

  22. Transform constraint settings. Enable to apply, disable to ignore.

  23. Controller Type: Sprite (2D Switcher -- also applies to UGUI Sprite, Texture, Material)

  24. Renderer reference for 2D Switcher controller type.

  25. On State:

    • On When Active: non-rest 2D elements are enabled during animation ON, once animation begins towards OFF the 'rest' element is immediately displayed. This option will work for lip-sync, but should not be used when multiple animation frames are used per viseme since the off animation will simply, immediately snap to the OFF (rest) position.
    • On Active One: non-rest 2D element is enabled once animation ON has reached full normalized value (1.0f). OFF (rest) is then activated once the animation OFF has reached full normalized OFF value (0.0f). This option should not be used for lip-sync.
    • On Until Off: non-rest 2D elements are animated the full cycle from OFF to ON to OFF. This is the appropriate option to choose for lip-sync where multiple 2D elements (frames) are configured per viseme.
  26. Drag-n-drop area -- allows easy configuration of multi-frame animations.

  27. Option to display larger views.

  28. Null Rest is used to blank the 2D element while at rest. This is normally not used for lip-sync, but may be desired if an animation should only be present while it is being animated and then should disappear.

  29. When 'null Rest' is not enabled, this position indicates the OFF (rest) 2D element. Similar to Min in a blendshape.

  30. Indicates the full ON 2D element. Similar to Max in a blendshape.

  31. 2D element sort and delete options.

  32. Controller Type: UMA DCS proxy

  33. Reference link to the UmaUepProxy controller.

  34. UMA ExpressionPlayer expression (similar to a blendshape selection).

  35. Expression amount, similar to Max on a Shape controller. > NOTE: Most UMA expressions use a bidirectional amount setting. 0 > 1 indicates the primary name value and 0 > -1 indicates the secondary name value. For example: neckUp_Down; neckUp is the primary name value, therefore values [0 .. 1] will move the neck up. Values [-1 .. 0] will move the neck down.

  36. Fold this Expression's components. Once 'folded', this option changes to 'Un-Fold', which does the opposite.

  37. Add Component: add a new component to the current expression.

  38. Disable Smoothly: controls the manner in which expression components are reset to their minimum setting when the module (SALSA/EmoteR) is disabled. Smoothly -- registers the components OFF using the same timings they would normally receive during an animation. Without enabling the smoothing option, the components are reset to minimum immediately.

Specialty Component Controllers

Over time, new controllers may be added for more specific or advanced purposes.

AnimatorController

animator controller

  1. Controller Type: Animator

  2. Reference link to the desired Animator.

  3. Select a pre-configured parameter on the linked Animator. The parameter type is automatically detected and handled according to the following rules (refer to the Animation Phases diagram below):

    • Bool: Set true on animation ON start (ref point #1 in the Animation Phases diagram). Set false on Animation OFF start (ref point #3 in the Animation Phases diagram).

    • Int: Set to 1 on animation ON finish (ref point #2 in the Animation Phases diagram). Set to 0 on animation OFF finish (ref point #4 in the Animation Phases diagram).

    • Trigger: Set on animation ON start (ref point #1) and reset on animation ON finish (ref point #2). (Optionally, when 'Trigger both ways' is enabled -- see point 4 in the controller diagram above) Set on animation OFF start (ref point #3) and reset on animation OFF finish (ref point #4).

    • Float: Set throughout the animation cycle according to the current calculated animation lerper. The value will be between [0.0f .. 1.0f]. From 0.0f to 1.0f between points #1 and #2 in the Phases diagram. From 1.0f to 0.0f between points #3 and #4 in the Phases diagram.

  4. (ONLY applies to Trigger types) Enable trigger operation in both directions ON and OFF.

  5. Selected parameter information -- displays the type of the selected parameter. Hovering this box will provide information specific to the controller operation on this parameter type.

EventController

Please read the information on Events in SALSA LipSync Suite. This controller implements event triggers for certain points along the animation cycle. Please refer to the Animation Phases diagram below in the Notes on Animation Timings section.

NOTE: There are two template files in the Examples > Scripts folder (SALSA_Template_EventControllerSubscriber, SALSA_Template_SalsaEventSubscriber) that demonstrate how the events can be used. The demo boxhead scene also implements the SALSA events template, writing to the console when SALSA starts/stops SALSAing.
The SALSA event template component simply needs to be dropped on an object and linked to a SALSA component -- this simple template will then write to the console as the SALSA processing state changes. The EventController event template operates similarly -- add the component to any game object. To configure it, simply enter the EventIdentifier string that matches the controller configuration. It will subscribe to the EventController events and filter delegate processing according to the identifier string.

event controller

  1. Controller Type: (C#) Event

  2. The event identifier is a string element used to assist in filtering events during subscription processing. For example, in the EventController.AnimationStarting delegate method, events from this controller are filtered based on the identifier matching the string == "buddy" (case sensitive):

private void OnAnimationStarting(object sender, EventController.EventControllerNotificationArgs e)
{
    if (e.eventName == "buddy")
    {
        // do some stuff...
        Debug.Log("EventController fired OnAnimationStarting for: " + componentEventName);
    }
}

Notes on Animation Timings

animation cycle

SALSA LipSync 2 has switched to a time-based animation control. Animation Timings instruct the QueueProcessor how long each phase of the component's animation takes to complete (in seconds). There are three phase settings available, depending on the handling and activating mechanism used.

  • On timing applies to the component's animation from its starting point to a full "On" calculated position. Refer to points #1 and #2 in the above diagram. For example, assume a setting of 1.0. Once activated, the animation will take 1.0 seconds to complete its animation from rest/starting to full On.

  • Hold occurs after full On has been reached, assuming it is using a Round-Trip handler. Once the animation lerp has reached 1.0, the QueueProcessor waits for the "Hold" time to complete and then changes the activation direction state. Refer to the period denoted between points #2 and #3 in the above diagram.

  • Off follows "Hold" in Round-Trip handlers and, as expected, this is the time it takes to animate from full "On" to full "Off" -- again, considering a Round-Trip handler. However, if the "On" animation phase is interrupted/overridden by the QueueProcessor, the "Off" animation will begin at the point reached in the "On" animation and take the "Off" amount of time to complete. The OFF timeframe occurs between points #3 and #4 in the above diagram.

    Of particular note (and by design), the "On" and "Off" animations take their specified setting (in seconds) to complete regardless of where the "starting" position is. In the example of an "Off" setting of 0.5 seconds, it makes no difference where it begins. If the "On" animation completed its full cycle, the "Off" will begin at the full "On" position and will take 0.5 seconds to complete its path to the full "Off" position. However, if the "On" animation is interrupted at the halfway point, the "Off" animation will still take 0.5 seconds to complete animating to the full "Off" position. The same is true of an "On" animation that has overridden and "Off" animation (such as in the case of a reactivation), regardless of the starting position, the new "On" queue registration will take "On" seconds to complete.

API Examples for ExpressionComponents

Overview

An ExpressionComponent has a somewhat complex data structure. All Expression types (i.e. LipsyncExpression, EmoteExpression, etc.) are composed of one or more ExpressionComponent(s). For the code samples below, consider the following hypothetical EmoteExpression, "smile".

For example, consider an emote called "smile". In some models, this emote may be made up of a single blendshape called 'smile', while in others, it may have two blendshapes ('smile_l' and 'smile_r'). Additionally, you may wish to add some additional expression in the eyes or eyebrows, such as 'eyes_wide' and 'brow_l_up' and 'brow_r_up'. So our hypothedical EmoteExpression called "smile" has multiple ExpressionComponent(s):
'smile_l' 'smile_r' 'eyes_wide'
'brow_l_up' * 'brow_r_up'

Editor Setting Manipulation

Emotes

To manipulate the settings of this EmoteExpression in the Editor configuration and the settings of the individual ExpressionComponents contained in the emote, we need to have reference to it/them. First, let's change some settings on the EmoteExpression itself, these settings will affect the entire emote. First grab a reference to the emote we are looking for. Refer to the EmoteR API for more information on the settings available.

NOTE: runtime manipulation of expressions is a bit more complex than Editor configuration changes since the normal controller configuration process must be called after the settings are changed. While manipulating Editor configurations (i.e. when creating your own custom OneClick configurations), it is not necessary to update the controller data since it will be accomplished at runtime Awake/Start.

Emoter emoter = gameObject.GetComponent<Emoter>; // need a ref to Emoter on this GObj

// find the emote to work on...'smile'
var myemote = emoter.FindEmote("smile"); // from the Emoter API

// manipulate the EmoteExpression settings...
myemote.isPersistent = true; // make this emote persistent in the QueueProcessor
myemote.isRandomEmote = true; // add this emote to the random pool

Now, let's update the components of this emote to use a different easing from the default type. We will change the easing to 'linear'. Continuing on from the above example, we will need to loop through each component in the EmoteExpression. The components are held in the Expression's expData.components, which is a List of type: ExpressionComponent.

foreach (var component in myemote.expData.components)
    component.easing = LerpEasings.EasingType.Linear;

Now, consider the desire to change the animation timing for the ON duration (for all configured emotes). In this example we need to loop through all emotes and then apply the change to each ExpressionComponent in each emote.

Emoter emoter = gameObject.GetComponent<Emoter>;
foreach (var emote in emoter.emotes) // emotes is the List of emotes.
    foreach (var component in emote.expData.components) // loop through each ExpressionComponent.
        component.durationOn = 0.11f; // change the duration ON animation timing.

Visemes

Visemes are slightly different to reference, there is currently no helper method for locating individual visemes because it is typically not something we need to do. But it is nearly as easy to implement. Let's find the viseme called "t" and change the component easing types to CircleIn and animation OFF timing to 0.06f. Salsa visemes are maintained in List<LipsyncExpression> visemes. We will use a lambda expression to find the viseme called "t".

NOTE: If the viseme "t" is not found, nothing will happen.

Salsa salsa = gameObject.GetComponent<Salsa>;
int idx = salsa.visemes.FindIndex(v => v.expData.name == "t"); // find valid viseme index...
if (idx > -1)
{
    var myviseme = salsa.visemes[idx]; // create reference to viseme.
    foreach (var component in myviseme.expData.components) // loop through components and change settings...
    {
        component.durationOff = 0.06f;
        component.easing = LerpEasings.EasingType.CircleIn;
    }
}