Working With External Influences (i.e. Mecanim, Animator, etc.)

Overview

Animations generally occur when certain properties of an object are manipulated over time, such as: a bone/transform's rotation or a blendshape's weight value. As with any other object property, when multiple systems attempt to modify a single value (or a related value) undesired effects can occur. Unless multiple systems are aware of each other, animations are typically handled by a single system. Most "systems" are not aware of each other and simply make their changes to the object's properties as though they are the only one doing so.

Technically, there can be only one winner in an animation conflict when different animation systems are operating on the same component. Conflicting animation processes are primarily a design issue. While animating processes (such as SALSA Suite) can attempt to work around other processes, the result is not necessarily guaranteed. The best way to avoid issues is to remove conflict in the design.

For the purposes of this document and the SALSA Suite, systems other than SALSA Suite which manipulate object properties for animation (i.e. Mecanim) are considered external influencers. They are external to the SALSA Suite processes and are attempting to influence the object's properties. Internally, SALSA Suite resolves conflict between modules very effectively within a module hierarchy.

How SALSA Suite Works with External Influences

SALSA has been designed to work with Mecanim and other external influences as gracefully as it can. Since facial animations are very deliberate, SALSA Suite should be considered the highest-priority when multiple animation systems are used together. In other words, SALSA Suite expects it should override all other animations while it is doing its thing. Since other animation systems are generally not aware of SALSA Suite, an order of operations needs to be possible for SALSA Suite to 'take control'.

SALSA Suite v2.5.0+ includes more frequent and intelligent detection of external influences. First, when enabled, it dynamically checks for influence on each frame (see the QueueProcessor documentation for more details). Previously, the Suite utilized more passive checking, leaving the determination of influence processing to the designer. However, more and more designers are using external influencers in their projects, many unaware of the how or why of things, and this can be catastrophic to SALSA Suite animations. Subsequently, SALSA Suite has taken a more hard-line approach to this and enables dynamic detection by default. If you want to control the process yourself, please disable the automatic option.

Last in the Order of Operation

SALSA Suite assumes it is the final word in object property changes (animation). This happens by leveraging the LateUpdate() cycle to apply its changes to an object's properties -- it is as late in the Unity game-loop as it can get. This generally works well, but assumes that all other external influencers are operating in an earlier Unity loop phase (i.e. Update()). Of course, as can be assumed, if an external influence is applied during the same game-loop phase as SALSA Suite, the results will likely be less than desirable. Order of operation cannot be guaranteed unless execution order is explicitly set within player settings.

Additionally, if a process operates in a differently timed loop, like the physics loop, order of operation can also not be guaranteed since these loops are not necessarily contiguous.

If SALSA Suite cannot be guaranteed to be the final operation in the overall Unity game loop, it is necessary for the designer to ensure conflicts do not result from more than one process trying to change the state of the components SALSA Suite is trying to animate. In other words, if you have animations controlled by the Unity Animator or Mecanim or some other process, those animations should not try to animate the same underlying components as SALSA Suite.

This is no different from any other philosophy in development. For example, assume there is a process that turns a component on a GameObject off if the score is 100. If another process turns the component on for some other reason and is not aware of the score check process, the component will likely be turned on when the other process is trying to turn it off. The result can be a flapping or toggling effect where the two processes fight to change the state, or it could be an ambiguous result where the state is not what is expected.

The same thing occurs when multiple processes attempt to animate the same underlying component. If there are other influences on a bone for instance, unless SALSA Suite is the last operation in the overall Unity game loop, SALSA Suite cannot force its own animation calculations.

MergeBack When It Can

When it is necessary for SALSA Suite to work with other animation influences, depending on design implementation, it may be possible to make the Suite 'aware' of the external influence. When properly configured, an ExpressionComponent can attempt to smoothly merge its animation back towards an external influence. This can only occur when SALSA Suite is the last animation operation on the underlying component.

NOTE: SALSA Suite will not blend the entire animation with an external influence. It will start where the current position or state is in the animation ON phase and continue to its configured end point. Once any HOLD time is complete, it will then attempt to smoothly return to the externally influencing process's current calculations during the animation OFF phase. There are two criteria where influence is NOT checked and MergeBack is NOT calculated: the animation ON/HOLD phases and when Persistence (see below) is enabled. If either of these are true, influence checks and MergeBack processing are bypassed.

There are two mechanisms for enabling detection of external influence. The global option (enabled in the QueueProcessor settings -- recommended for general use) and the (manual) per-component setting. Read about the QueueProcessor for more details.

NOTE: If the manual, per-component option is enabled (see below), SALSA Suite will assume the component is under influence (whether it is or not).

Once SALSA Suite has finished its animation ON and HOLD phases, it will try to MergeBack towards the external influence (if detected or manually enabled). As mentioned previously, automated detection is only possible when the specific bone or shape is in conflict. When the Merge with Influencer feature is enabled on the QueueProcessor the Suite tries to automatically determine whether a property has been modified by an external process. If manually enabled on a per-component basis, a mergeback operation is assumed necessary and forced.

QueueProcessor automatic MergeBack

NOTE: If SALSA Suite expects an external influence (manual, per-component option enabled) and there is none, the results will also be undesirable. SALSA Suite will attempt to animate back towards a phantom value that happens to be the last Suite-applied value and will result in a one-way animation towards the full ON position and will never turn OFF. It is usually best to let SALSA Suite automatically handle external influence.

In some instances, SALSA Suite may not be able to correctly determine such a scenario and should be manually configured. This is rare.

MergeBack manual configuration

NOTE: MergeBack and Persistence (described below) do not work together. Each tackles a specific requirement and works in lieu of the other. If Persistence is enabled, MergeBack is effectively disabled.

Persistence

As a design consideration, some Suite features should be considered Suite-only implementations, such as lipsync, eye/lid-movement, and probably emotes. As a general rule, if SALSA Suite is implemented, it should be left to control its functions autonomously.

As mentioned above, conflict arises when multiple systems try to influence an object property. Some Suite functions automatically assume they are the only one in control (i.e. the Eyes module) and persist the animation controller in the QueueProcessor throughout its operations. However, other modules take a more passive approach. When a lipsync or emote module registers an animation with the QueueProcessor, the processor fully controls the animation until it has completed and then releases the controller from the queue. In the case of a one-way ExpressionComponent handler, the QueueProcessor maintains control until the animation has completed its full ON or OFF direction (and then releases the Component controller). In the case of a round-trip handler, the QueueProcessor maintains control throughout the entire ON/HOLD/OFF cycle prior to releasing the Component controller.

Assuming the Order of Operation is implemented correctly, all queue calculations will be the final animation value written to an object's property. Persistence ensures an ExpressionComponent remains in the queue until told otherwise, writing the calculated value to the object property on each frame, even when the animation cycle has completed. This process maintains the animation state SALSA Suite is configured for (full ON or OFF) regardless of external influence.

Configuration of persistence is slightly different depending on the module.

Lipsync persistence is configured globally for visemes:
EmoteR persistence is configured per emote, covering all components in an emote:

NOTE: Persistence and MergeBack do not work together. Each tackles a specific requirement and works in lieu of the other.

Design Changes May Be Required to Eliminate Conflict

Keep in mind, there are scenarios where conflict will result and cannot be mitigated due to design implementation. These are design problems and require design changes. Generally speaking, there can be only one in control. Sometimes timing can help as with the case of LateUpdate vs Update, sometimes it cannot. The following is not all-inclusive of conflict issues where design choices must be made, but the fact remains, these are design problems, not SALSA Suite problems. It will be necessary for the designer to correct the "design".

Upstream Downstream Animations

This can result when the animation of another component indirectly affects the outcome of another component.

For example, in the case of bone/transform animations: if an upstream (parent) or downstream (child) bone is animated by an external process, the Suite-configured bone has no awareness of the other animations and the results will likely be undesirable. In this case, the best practice is to disable the external animation until the Suite animation is complete. Or modify the animation to not be in conflict with SALSA Suite animations.

A similar situation can occur with blendshapes. Blendshapes can share vertices and if two "conflicting" blendshapes are animated together the results will be additive (or subtractive), exaggerating the resulting animation. These are design problems, not SALSA Suite problems. The designer will need to correct the "design".

Animation Clip/File Locks Properties

Another common issue is when an animation exported by a character model system (i.e. Reallusion CC), where the export routine includes all blendshapes regardless of whether they actually have animations. Blendshapes or bones without curves/keys are locked when included in an animation in any Animator state, by Unity design. You can test this yourself and read about it in this thread: https://forum.unity.com/threads/animator-locking-animated-value-even-when-current-state-has-no-curves-keys-for-that-value.440363/

This list is not all-inclusive and, as mentioned, these are design problems, not SALSA Suite problems. It will be necessary for the designer to correct the "design".

Model Jaw Remains Open

A frequent issue we hear about, especially with Reallusion Character Creator models, is an incorrect jaw position during runtime. The cause for this is the same thing as the previous section. The jaw hangs open and, while SALSA and EmoteR deal with this fine while animating, once their components have finished, the jaw flaps open again. This is not a SALSA Suite issue, but rather an export issue out of Reallusion software. The jaw will remain open regardless of whether SALSA Suite is applied to the model. A quick fix is to remove the jaw from the avatar bone map. There should also be a setting in CC to properly include/exclude the jaw as well -- check Reallusion support for information. We do not use or support Reallusion products other than providing a OneClick configuration solution. To disconnect the jaw from the avatar bone map:

Select the model in the Project.
Switch to the 'Rig' tab.
Click 'Configure'.
Switch to 'Mapping' tab.
Switch to 'Head' mapping.
Remove the 'Jaw' link.

Best Practices for Animating with SALSA Suite

Ultimately, the best overall experience is to only use SALSA Suite to animate the features you wish it to control. All other influencers should be configured to avoid animating properties SALSA Suite is controlling.

If you must use additional animation tools or have created custom scripts as external infuencers, the best practice for working with SALSA Suite is to implement these processes during the Update() phase. Mecanim already operates prior to LateUpdate() by default and generally works fine with SALSA Suite (assuming all configuration is correct). There are, however, some instances where it can be problematic. For example: assume Mecanim animates a child bone where SALSA Suite has taken control of the parent bone and the child animation is designed to complement a parent animation. Since the Suite is animating the parent in accordance with its configuration, the two animations may not "align" as desired.

If you are using Mecanim or other animation tools and the 'animations' indiscriminately set property values even if they are not the currently active clip, they can cancel out or interfere with Suite animations if SALSA Suite is not configured to handle external influence. Disable or delete (preferred) all animation configurations that apply a blanket value to an object's properties. For example: Reallusion animation exports can contain property values of 0 for blendshapes even if they are not used in the animations. To reduce processing requirements and eliminate conflicts with SALSA Suite, remove all animation keyframes for properties that are not part of the animation.

And remember, even if an animation clip is not active, the Unity Animator will lock the values when present in a dormant clip. See this thread: https://forum.unity.com/threads/animator-locking-animated-value-even-when-current-state-has-no-curves-keys-for-that-value.440363/

Summary: Key Points to Remember

There are a few key points to remember when designing animations for your character:

SALSA Suite operates in the LateUpdate cycle in an attempt to be the last entity to update bone/shape positions.
External influencers (like Mecanim) must operate earlier than the LateUpdate cycle (i.e. Update) so they can be overridden.
SALSA Suite must be configured to work with external influencers where conflicts arise (i.e. enable persistence, mergeback, etc. if needed).
Conflicts are not always simply a matter of operating on the same bone/shape. Externally influenced animations on parent or child objects can also cause visual conflicts and must be considered at design time.

We have tried to make the Suite work well in as many conflicting situations as possible, but there is no guarantee it can work in every situation. Sometimes the designer needs to make a choice on how much effort is put into making a particular situation work perfectly vs adjusting the design to work within the framework provided.