SALSA LipSync Suite v2 was written for the Unity Game Engine development platform. Current version support is all stable, release versions v2017.4.1+ (unless otherwise stated). The Suite is not supported with pre-release Unity engines or packages.
NOTE: SALSA LipSync Suite v2 only works with Unity and there are no plans to port it to other game development platforms.
Please see the Asset Store page for currently supported deployment platforms.
SALSA, along with EmoteR and Eyes, has very few requirements. Since it is a Unity asset, it does not care about originating file format. SALSA Suite simply requires the originating format be recognized and imported by Unity into one of the following supported controller types:
- Shape (blendshapes)
- Bone (transform)
- Sprite (single image or array of animated frames)
- UGUI Sprite (single image or array of animated frames)
- Texture (single image or array of animated frames)
- Material (single image or array of animated frames)
- UMA ExpressionPlayer Poses
- Animator parameters (bool, float, int, trigger)
While 2D and 3D implementations are usually very different, with SALSA Suite v2, mixing 2D and 3D is quite easy. For both 2D and 3D, there simply needs to be a sufficient capability to individually manipulate the mouth (for SALSA lipsync), the face (for EmoteR), and the head/eyes (for Eyes).
The current, supported methods for manipulating 3D model features is using either bones (transforms) or blendshapes (also known as morphs, shapekeys, etc. in 3D modeling software).
NOTE: It is also possible to implement some 2D workflows in conjunction with 3D models, such as performing material texture swaps on 3D meshes.
Typically, blendshapes are best for facial animations, lip/mouth movement, brow movement, etc. Bones work best for eyes and head, and may also be used to manipulate the jaw. However, either bones or blendshapes may be used to manipulate all aspects of facial animation. SALSA and EmoteR can mix and match blendshapes and bones for viseme and emote expression configurations. For that matter, they can also include 2D-centric controllers and swap out textures, etc. The Eyes module has some stricter requirements due to the specific movement controls it implements -- it uses templates for head/eyes/eyelids, corresponding to the specific animation controller types (bones, shapes, sprites, textures, etc.).
When using blendshapes, the hard, simple rule is: Unity has to be able to utilize whatever morphing system your 3D modeler implements. When you import your model, if Unity creates a SkinnedMeshRenderer which contains sufficient blendshapes, your model will work with SALSA LipSync Suite. First and foremost, Unity has to recognize the blendshapes on your model. If it (Unity) does not, SALSA Suite's ShapeController cannot work with your model. Additionally, if there are not sufficient blendshapes to create dynamic mouth-movements, results will not be as fantastic as they could be. SALSA Suite is magical, but it is not magic, it cannot create blendshapes to animate your character.
Likewise, when using bones, it is necessary for there to be a sufficient number and placement of bones to independently manipulate the desired mesh object. If there are no bones to manipulate the facial (or other) part you are trying to animate, SALSA cannot create them for you.
Blendshapes -- What is required?
As mentioned above, a sufficient selection of shapes are required to perform dynamic animations. Since SALSA, EmoteR, and Eyes each work on different parts of the face and in different ways, they have differing design needs. Of course, this very much depends on your design requirements and decisions. SALSA Suite is very flexible and can produce a very wide range of design looks.
Generally speaking, the blendshapes required for any of the modules are such they need to be able to produce the look and feel you, as a designer, want to implement before SALSA Suite enters the equation. The more flexible and dynamic the shapes are on your character model, the more flexible and dynamic your end result can be. If the shapes are dull and muted, the results will likewise be dull and muted. What does this mean? If you cannot statically produce the look you are trying to achieve with your blendshapes, SALSA Suite will also not be able to facilitate your expectations.
Bones -- What is required?
This is a little less cut-and-dry. A bone/transform is required to move, rotate, or scale a weighted portion of a model's mesh object. Bones are recommended for eye and head movement but can become unwieldy for mouth and emote configuration due to the quantities of bones typically required for such movement. However, it is certainly possible to do so. Most model generation systems (other than UMA) do not use bones for facial animation; however, if you are creating your own setup, you may certainly use bones and configure the SALSA Suite to animate your character model. The suggestion is to read the section on blendshape requirements as well as the SALSA recommendations and configure a sufficient number of bones with specific weights to create the same representation of movement. If all you have is a jawbone, you can implement mouth-sync with your audio files, but it will not be compelling lipsync.
SALSA LipSync Suite v2 currently supports sprite, texture, and material manipulation. It uses a switching mechanism to swap out element frames to create animation, much like the cell-based cartoon animation techniques. SALSA v1 supported single-frame sprite configurations, swapping out one frame for each represented animation. SALSA Suite v2 also supports the same single-frame configuration as well as multi-frame configurations and has added additional support for textures, materials, and UGUI sprites.
Whether you are using sprites or textures, the golden rule for getting the most out of the SALSA Suite is to implement separate facial segments for animation. The main separation should be eyes animated separately from the mouth. However, if you wish to introduce emotes, you may have to get creative in your process since emotes frequently include eye and mouth movement. SALSA Suite has some mechanisms to deal with this such as priority overrides in the QueueProcessor following these basic rules:
- Lipsync overrides everything.
- Emotes override Eyes/Eyelids/Head
- and Eyes/Eyelids/Head override themselves
2D Eyes Requirements
Eyes has some specific requirements for 2D depending on whether you are using sector-based or bone/transform-based movement. Bone-based movement is the loosest and really just requires your model looks correct as the eye is moved around on its transform. For example, if manipulating an eye behind an eye-socket, the eye should not expose itself outside of the socket area and should also not expose blank space within the socket -- unless, of course, that is the look-and-feel you are going for.
Sector-based movement is a bit different. The Eyes module has a few templates to work with such scenarios:
- 2-way (left, right)
- 3-way (left, center, right)
- 5-way (center, top, right, bottom, left)
- 9-way (center, top, top-right, right, bottom-right, bottom, bottom-left, etc.)
If using sectors, your animated cells should adhere to one of the above template scenarios.