SALSA LipSync with SalsaMicInput

Overview

SalsaMicInput takes a very basic approach to adding real-time Unity microphone input to your SALSA lip-sync project and while still intended as an example code-base for real-time input, it does include some robust capabilities for both design-time and run-time usage. Currently, SalsaMicInput only requires a Unity AudioSource for microphone input operation. However, a linked SALSA component is required for real-time lip-synchronization. Additionally, SalsaMicInput can be selectively connected to any Unity AudioSource.

The simplicity of SalsaMicInput remains a priority and the same automated feature set exists as in previous versions, plus some new capabilities. At run-time, SalsaMicInput will attempt to gain reference to a local AudioSource component if one has not been specified (linked in the custom inspector). The process of dynamically getting the AudioSource component will operate in a non-blocking coroutine and will continue to look for an AudioSource on the gameObject until one is present.

NOTE: Without an AudioSource, this asset cannot function and will simply continue to wait for an AudioSource.

The benefit of this method is a flexible start-up solution, allowing programmatic implementations of an AudioSource (such as UMA2 character startup procedures). Once the AudioSource is connected, the asset will automatically complete the wire-up of the Unity microphone (using the default microphone if one has not been selected in the Custom Inspector). It will then start microphone recording (by default - selectable by the developer) and terminate the coroutine.

In addition to all of the automated processes, SalsaMicInput now has an option to disable the auto-start functionality. During startup, automatic wire-up of the Unity microphone will occur, but recording will not start automatically. This feature is very useful for implementations where microphone input is desired in an on-demand scenario. As expected, SalsaMicInput can be started and stopped on demand via the API calls, StartMicrophone() and StopMicrophone().

Support

We (Crazy Minnow Studio) are happy to provide SALSA LipSync Suite-related support. However, we do not provide support for third-party assets. If you have problems getting the third-party asset working, please contact the asset's publisher for assistance.

NOTE: Please remember, the source is included for this free add-on and this should be considered example code that you can use to jump-start your project. It is not intended to fit all scenarios or requirements -- you are free to update it as necessary for your needs. We do provide limited support for all of our products; however, we do not make development changes to support specific project needs.

For SALSA LipSync Suite-related support, please email (assetsupport@crazyminnow.com) us the following information:

Invoice number (support will not be provided without a Unity Invoice Number).
Operating System and version.
SALSA Suite version.
Add-on version (generally located in an associated readme or the script header comment).
Full details of your issue (steps to recreate the problem), including any error messages.
Full, expanded component screenshots (or video).
Full, expanded, associated hierarchy screenshots (or video).
Super helpful: video capture of issue in action if appropriate.

Requirements for This Example Implementation Tutorial

SALSA LipSync v2.5.0+
SalsaMicInput
Requires a known-working microphone.

ATTENTION: These instructions require you to download and install the appropriate assets and/or add-on scripts into your Unity project. If you skip this step, you will not find the applicable option in the menu and/or component library.

NOTE: While every attempt has been made to ensure the safe content and operation of these files, they are provided as-is, without warranty or guarantee of any kind. By downloading and using these files you are accepting any and all risks associated and release Crazy Minnow Studio, LLC of any and all liability.

Installation Instructions

NOTE: For information on how to import/install Unity AssetStore packages or unitypackage files, please read the Unity documentation.

Import SALSA LipSync into your project and please familiarize yourself with SALSA using the online documentation for SALSA LipSync.
Import this SALSA LipSync v2 integration add-on and familiarize yourself with these instructions.

NOTE: While every attempt has been made to ensure the safe content and operation of these files, they are provided as-is, without warranty or guarantee of any kind. By downloading and using these files you are accepting any and all risks associated and release Crazy Minnow Studio, LLC of any and all liability.

Usage Instructions (using SalsaMicInput with SALSA LipSync v2.5.0+):

SalsaMicInput v2.5.0 is the current compatible micInput version for SALSA LipSync v2.5.0+.
Ensure you have at least one microphone attached to your system (see caveats in the troubleshooting section below.
- Ensure it is enabled and working outside of Unity.
- It is also a good practice to make sure it is not being used by another application while trying to use it as an input source for lip synchronization.
NOTE: [on PC] the 'Default' microphone selection will attach to the operating system default microphone. [on Mac] the 'Default' microphone selection will attempt to attach to the input device currently selected in the operating system's 'Sound' utility applet.
Import/install SALSA LipSync Suite into your Unity lip-sync project. \

NOTE: Ensure the version requirements are met.
Download and import the SalsaMicInput unitypackage into your Unity lip-sync project.

NOTE: Ensure the version requirements are met.
Configure and test SALSA LipSync on a character model in your scene (without using SalsaMicInput) to ensure SALSA is configured properly and is working.
Add SalsaMicInput to a GameObject in your scene (preferably the same SALSA-configured GameObject). If you are not adding it to the same GameObject that contains the AudioSource linked to SALSA LipSync, it will be necessary to link SalsaMicInput to the AudioSource driving SALSA.

NOTE: You can also add the SalsaMicInput component to your GameObject from the Component menu: Crazy Minnow Studio > SALSA LipSync > Add-ons.
Configure SalsaMicInput as per the screenshot.

As of version 2.5.0, SalsaMicInput no longer utilizes a microphone pointer updater script. Instead, it leverages SALSA's ability to delegate certain processing modules to custom code. SalsaMicInput implements this by setting CrazyMinnow.SALSA.Salsa.clipHeadPointer to SalsaMicInput custom code which provides the microphone pointer value to SALSA upon request.

NOTE: Previous versions of SalsaMicInput (<2.5.0) added another component (SalsaMicPointerSync) to your game object when "Link with SALSA" was enabled. This helper script (or similar) is required for proper SALSA Core audio processing of the microphone buffer when the pointer location has not been delegated. It lets SALSA know where the current buffer record head position is. Please see the SALSA API information for the microphoneRecordHeadPosition field description for more information if you wish to write your own procedure to update SALSA Core. Again, this is only required for versions prior to 2.5.0.
Ensure SALSA and SalsaMicInput are linked to the same/appropriate AudioSource if they are not attached to the same GameObject.

NOTE: SALSA needs to connect to the same AudioSource component micInput will use. This can be wired up any way you desire, the AudioSource used by SALSA and micInput need to match.
Also ensure SALSA's "Data Analysis" settings are configured to auto-adjust and use microphone.

NOTE: If "Link with SALSA" is enabled in SalsaMicInput, these settings should already be enabled.
You might need to adjust your Trigger levels in SALSA's "Viseme Configuration" section. This depends on the sensitivity of your mic and how dynamic you want the lip sync movement to be. If your microphone produces lower input, you may need to adjust the sliders to the left to take advantage of the lower input levels (or turn up the microphone levels).
Hit [Play] and enjoy lip-syncing your model with your own voice!

API Information

AudioSource fields used by SalsaMicInput

SalsaMicInput makes use of the standard Unity Microphone class. As such, it also uses a standard UnityEngine.AudioSource. You may manipulate the AudioSource directly, or via the key fields used by SalsaMicInput.

public AudioSource audioSrc

Must be set to the same AudioSource as SALSA. By default, on start up, this will attempt to grab the AudioSource on the same object. If you have disconnected the AudioSource from this component, you will need to set it prior to calling Start/StopMicrophone().

public bool isMuted = false

Determines whether the AudioSource (and subsequently, the microphone) is muted during microphone capture.

Key Microphone Fields

public int DEFAULT_RATE = 11025

For Unity Editor Inspector use only -- configures the default rate to display in the Inspector.

public int sampleRate = 0

User-selectable sample rate (microphone dependent).\

public bool overrideSampleRate = false

Some microphones do not report the true capabilities. Enable this flag to override a false return when checking the caps of the selected microphone.

Key SalsaMicInput Control Properties

public bool linkWithSalsa = false

Primarily for use with the Unity Editor Inspector, this allows the Inspector to configured key SALSA fields for use with SalsaMicInput. It has little impact on runtime programmatic use, but does affect the OnEnable/Disable() callbacks.

public bool isAutoStart = true

Determines whether SalsaMicInput auto starts the microphone wire-up process during the Unity Start() callback.

public bool isDebug = false

Enables debug mode for detailed log output to the Editor console.

public string selectedMic = default(string)

Holds the currently set microphone. This field is primarily for historical use and keeps track of the microphone that was started using SalsaMicInput.StartMicrophone(). It is recommended not to alter this field.

Methods

Starting and stopping the microphone will be the most commonly used methods in the SalsaMicInput class.

public void StartMicrophone(string mic) : can be null for default microphone

Call this method with a microphone name to specify which microphone to start. Specifying null will start the operating system's default associated microphone. AudioSource must be set first otherwise this will fail and return without starting a microphone. NOTE: It is not necessary or recommended to first set SalsaMicInput.selectedMic. The supplied parm string mic will be stored in selectedMic via StartMicrophone. See below for examples of resetting and changing the microphone.

public void StopMicrophone(string mic) : can be null for default microphone

This is the complementary method to StartMicrophone() and will stop an actively recording microphone. This will also disconnect the AudioSource.AudioClip by setting it to null. It is generally advised to use SalsaMicInput.selectedMic when stopping the microphone.

CrazyMinnow.SALSA.SalsaMicInput micInput;
micInput.StopMicrophone(micInput.selectedMic); // stops specified microphone.

Examples

Restarting the Current Microphone

Example:

CrazyMinnow.SALSA.SalsaMicInput micInput;
micInput.StopMicrophone(micInput.selectedMic); // stop the current microphone to reset SalsaMicInput.  
micInput.StartMicrophone(micInput.selectedMic); // re-starts the current microphone..

Changing the Current Microphone

Example:

CrazyMinnow.SALSA.SalsaMicInput micInput;
string yourMicrophoneName = "some microphone";
micInput.StopMicrophone(micInput.selectedMic); // stop the current microphone to reset SalsaMicInput.  
micInput.StartMicrophone(yourMicrophoneName); // starts specified microphone and sets SalsaMicInput.selectedMic to 'yourMicrophoneName'.
// .
// .
// You may subsequently stop 'yourMicrophoneName' as needed...
micInput.StopMicrophone(micInput.selectedMic); // stops the microphone that was previously started.

Troubleshooting and Operational Notes:

Microphone Issues on MacOS

NOTE: If you are experiencing microphone issues with MacOS, you could be seeing permissions issues with Unity Hub and MacOS Privacy security. When Unity Hub is installed, you should be prompted for microphone permissions and that does not always appear to happen. Please see this thread for more detail. Be aware, we are NOT suggesting this as a fix; however, it does demonstrate the case and condition. We have confirmed this to be an issue ourselves and have successfully performed the resolution presented here on a test machine to validate the condition. You must determine your own course of action.
Microphone muting now works as would be expected in conjunction with SALSA LipSync v2+. It is no longer necessary to create attenuated mixer groups.
(OS X): Default microphone selection may not work reliably. Select the microphone explicitly if 'Default' does not work as expected.

IMPORTANT! (OS X): Selecting multiple microphones for input may not be supported on your platform. Save your project before testing this feature. In our testing, Unity didn't quite know what to do with itself and required a force-quit.
If microphone statuses change, use Refresh Mic List to update the Available Microphones list for selection.
If there are no microphones available, the Available Microphones list will display 'ERROR - no microphones available'.
Lip synchronization with a microphone also requires an AudioSource component to be present on your object or, if using an AudioSource on a different object, it must be linked to SalsaMicInput.
On mobile devices (Android), lip sync works well, but we have seen some anomalous issues when the application is task-switched to another application and then back. If the microphone seems to misbehave after doing this, try switching to the home screen and then back into the application.
As would be expected, microphone input will only work with one application asset at a time. Windows operating systems may allow more than one microphone input to be active at once on the same computer - this is not guaranteed behavior. If it is desireable to use microphone input for multiple assets/models from the same computer, it would be necessary to programatically disconnect one asset and then connect another.
Similar to the above caveat, SalsaMicInput cannot work in conjunction with other microphone "input" scripts/components (i.e. Dissonance, Photon Networking, etc.) on the same microphone.

Release Notes:

v2.5.0 - (2020-09-15):

NOTE: requires SALSA v2.5.0+

+ Basic co-routine startup protection -- there can be only one.
+ Preprocessor wraps for SALSA (CMS_SALSA).
+ Basic OnEnable/Disable logic -- automated teardown and setup to revert to/from normal AudioSource.AudioClip to AudioSource.(microphone)AudioClip.
+ Updated to take advantage of SALSA 2.5.0 delegate plugin.
~ Added general null-ref checks -- defensive coding.
! Wireup now appropriately detects missing AudioSource regardless of wireup status.

v2.3.0 - (2020-02-03):

NOTE: SALSA Core v2.3.0+ is required for this version of SalsaMicInput.

Updated for SALSA LipSync v2.3.0+: SALSA Core v2.3.0+ requires the virtual pointer for the microphone record head to be updated externally -- the Unity Microphone dependency was removed from the SALSA Core code-base. SalsaMicInput now includes a helper script to perform this action. You are free to write your own helper script if you do not wish to use this one.

v2.0.0-beta - (2019-06-22):
* Initial release for SALSA LipSync v2.0.0.