Working With External Influencers (i.e. Mecanim)

Overview

Animations generally occur when certain properties of an object are manipulated, such as: a bone/transform's rotation or a blendshape's weight value. As with any other object property, when multiple systems attempt to modify a single value (or a related value) undesired effects can occur. Unless multiple systems are aware of each other, controlling animations is typically handled by a single system. Most "systems" are not aware of each other and simply make their changes to the object's properties as though they are the only one doing so.

For the purposes of this document and the SALSA Suite, systems other than SALSA Suite which manipulate object properties for animation (i.e. Mecanim) are considered external influencers. They are external to the SALSA Suite processes and are attempting to influence the object's properties.

How SALSA Suite Works with External Influencers

SALSA has been designed to work with Mecanim and other external influences as gracefully as it can. Since facial animations are very deliberate, SALSA Suite should be considered the highest-priority when multiple animation systems are used together. In other words, SALSA Suite expects it should override all other animations while it is doing its thing. Since other animation systems are generally not aware of SALSA Suite, an order of operations needs to be possible for SALSA Suite to 'take control'.

SALSA Suite v2.5.0+ includes more frequent and intelligent detection of external influences. First, when enabled, it dynamically checks for influence on each frame (see the QueueProcessor documentation for more details). Previously, the Suite utilized more passive checking, leaving the determination of influence processing to the designer. However, more and more designers are using external influencers in their projects, many unaware of the how or why of things, and this can be catastrophic to SALSA Suite animations. Subsequently, SALSA Suite has taken a more hard-line approach to this and enables dynamic detection by default. If you want to control the process yourself, please disable the automatic option.

Last in the Order of Operation

SALSA Suite assumes it is the final word in object property changes (animation). This happens by leveraging the LateUpdate() cycle to apply its changes to an object's properties -- it is as late in the Unity game-loop as it can get. This generally works well, but assumes that all other external influencers are operating in an earlier Unity loop phase (i.e. Update()). Of course, as can be assumed, if an external influence is applied during the same game-loop phase as SALSA Suite, the results will likely be less than desirable.

MergeBack

When it is necessary for SALSA Suite to work with other animation influences, depending on design implementation, it may be possible to make the Suite 'aware' of the external influence. When properly configured, an ExpressionComponent can attempt to smoothly merge its animation back towards an external influence.

NOTE: SALSA Suite will not blend the entire animation with an external influence. It will only attempt to smoothly return to the external process's current calculations during the animation OFF phase. There are two criteria where influence is NOT checked and MergeBack is NOT calculated: the animation ON/HOLD phases and when Persistence (see below) is enabled. If either of these are true, influence checks and MergeBack processing are bypassed.

There are two mechanisms for enabling detection of external influence. The global option (enabled in the QueueProcessor settings -- recommended for general use) and the (manual) per-component setting. Read about the QueueProcessor for more details.

NOTE: If the manual, per-component option is enabled (see below), SALSA Suite will assume the component is under influence (whether it is or not).

Once SALSA Suite has finished its animation ON and HOLD phases, it will try to MergeBack towards the external influence (if detected or manually enabled). As mentioned previously, automated detection is only possible when the specific bone or shape is in conflict. When the Merge with Influencer feature is enabled on the QueueProcessor the Suite tries to automatically determine whether a property has been modified by an external process. If manually enabled on a per-component basis, a mergeback operation is assumed necessary and forced.

QueueProcessor automatic MergeBack

NOTE: If SALSA Suite expects an external influence (manual, per-component option enabled) and there is none, the results will also be undesirable. SALSA Suite will attempt to animate back towards a phantom value that happens to be the last Suite-applied value and will result in a one-way animation towards the full ON position and will never turn OFF.

In some instances, SALSA Suite may not be able to correctly determine such a scenario and should be manually configured.

MergeBack manual configuration

Keep in mind, there are scenarios where conflict will result and cannot be mitigated due to design implementation. For example, in the case of bone/transform animations: if an upstream (parent) or downstream (child) bone is animated by an external process, the Suite-configured bone has no awareness of the other animations and the results will likely be undesirable. In this case, the best practice is to disable the external animation until the Suite animation is complete. Or modify the animation to not be in conflict with SALSA Suite animations. These are design problems, not SALSA Suite problems. The designer will need to correct the "design".

A similar situation can occur with blendshapes. Blendshapes can share vertices and if two "conflicting" blendshapes are animated together the results will be additive, exaggerating the resulting animation. These are design problems, not SALSA Suite problems. The designer will need to correct the "design".

NOTE: MergeBack and Persistence (described below) do not work together. Each tackles a specific requirement and works in lieu of the other. If Persistence is enabled, MergeBack is effectively disabled.

Persistence

As a design consideration, some Suite features should be considered Suite-only implementations, such as lipsync, eye/lid-movement, and probably emotes. As a general rule, if SALSA Suite is implemented, it should be left to control its functions autonomously.

As mentioned above, conflict arises when multiple systems try to influence an object property. Some Suite functions automatically assume they are the only one in control (i.e. the Eyes module) and persist the animation controller in the QueueProcessor throughout its operations. However, other modules take a more passive approach. When a lipsync or emote module registers an animation with the QueueProcessor, the processor fully controls the animation until it has completed and then releases the controller from the queue. In the case of a one-way ExpressionComponent handler, the QueueProcessor maintains control until the animation has completed its full ON or OFF direction (and then releases the Component controller). In the case of a round-trip handler, the QueueProcessor maintains control throughout the entire ON/HOLD/OFF cycle prior to releasing the Component controller.

Assuming the Order of Operation is implemented correctly, all queue calculations will be the final animation value written to an object's property. Persistence ensures an ExpressionComponent remains in the queue until told otherwise, writing the calculated value to the object property on each frame, even when the animation cycle has completed. This process maintains the animation state SALSA Suite is configured for (full ON or OFF) regardless of external influence.

Configuration of persistence is slightly different depending on the module.

  • Lipsync persistence is configured globally for visemes:

    SALSA persistence

  • EmoteR persistence is configured per emote, covering all components in an emote:

    EmoteR persistence

NOTE: Persistence and MergeBack do not work together. Each tackles a specific requirement and works in lieu of the other.

Best Practices for Animating with SALSA Suite

Ultimately, the best overall experience is to only use SALSA Suite to animate the features you wish it to control. All other influencers should be configured to avoid animating properties SALSA Suite is controlling.

If you must use additional animation tools or have created custom scripts as external infuencers, best practice for working with SALSA Suite is to implement these processes during the Update() phase. Mecanim already operates prior to LateUpdate() by default and generally works fine with SALSA Suite (assuming all configuration is correct). There are, however, some instances where it can be problematic. For example: assume Mecanim animates a child bone where SALSA Suite has taken control of the parent bone and the child animation is designed to complement a parent animation. Since the Suite is animating the parent in accordance with its configuration, the two animations may not "align" as desired.

If you are using Mecanim or other animation tools and the 'animations' indiscriminately set property values even if they are not used by the animation, they can cancel out Suite animations if SALSA Suite is not configured to handle external influence. Disable all animation configurations that apply a blanket value to an object's properties. For example: Reallusion animation exports can contain property values of 0 for blendshapes even if they are not used in the animations. To reduce processing requirements and eliminate conflicts with SALSA Suite, remove all animation keyframes for properties that are not part of the animation.

Key Points to Remember

There are a few key points to remember when designing animations for your character:

  1. SALSA Suite operates in the LateUpdate cycle in an attempt to be the last entity to update bone/shape positions.
  2. External influencers (like Mecanim) must operate earlier than the LateUpdate cycle (i.e. Update) so they can be overridden.
  3. SALSA Suite must be configured to work with external influencers where conflicts arise (i.e. enable persistence, mergeback, etc. if needed).
  4. Conflicts are not always simply a matter of operating on the same bone/shape. Externally influenced animations on parent or child objects can also cause visual conflicts and must be considered at design time.

We have tried to make the Suite work well in as many situations as possible, but there is no guarantee it can work in every situation. Sometimes the designer needs to make a choice on how much effort is put into making a particular situation work perfectly vs adjusting the design to work within the framework provided.