Character Customizer

Owned by Former user (Deleted)
Last updated: Nov 01, 2013 by Arno HartholtVersion comment
8 min read

Overview

The Character Customizer is a tool that allows for the quick setup of a single-character scene and a set of lines for the character to act out. It offers control over camera angles, backgrounds, voices and facial animation. The goal is to be able to render out movies of the character. Because of this special purpose we opted to create a separate tool. Depending on user feedback we may eventually fold in some of this functionality into VHBuilder.

Quick facts:

Location: /tools/CharacterCustomizer

Users 

Set up

The Toolkit is a collection of various components, so there is some manual work required to set everything up for creating movies using the Character Customizer. In general you will need to have the following elements up and running:

  • Unity (the renderer / game engine)

  • Character Customizer tool

  • Supporting VH modules

Instructions for setting up Unity:

  • Run the Launcher by double clicking 'run-launcher.bat' in the root of your installation directly ('c:\vhtoolkit' by default).

  • Click on the Advanced button in the lower right corner. This will expand the view to show all components.

  • In the Renderer row, change some of the start-up parameters:Launch Unity by clicking on the button with the green up arrow icon. Unity may take a while to load and will initially take over the screen before going back to window mode.

    • Set Fullscreen to 'false'
    • Set Intro to the empty value
    • Optionally, set the resolution to your preferred one

  • Once Unity comes up, select the window and hit the C key on your keyboard to bring up the debug menu (you may need to click away the Tips window, depending on your earlier start-up settings). In the debug menu, select 'Load Customizer'. This will load the scene specifically made for the Character Customizer.

    • Note that you can use the M key to toggle between having your mouse cursor and using the mouse button for speech interactions (indicated by the microphone icon in the lower right).

Instructions for setting up the Character Customizer:

  • In the Launcher, click on the button with the green up arrow in the Character Customizer row in the Tools section.

Instructions for setting up the supporting VH modules:

  • In the Character Customizer window, go to the System menu and select Launch All.

Install a codec for movie recording

The Character Customizer uses a Unity movie plug-in which in turn uses the video codecs present on your machine. You can try out different ones by selecting them from the Codecs int the Options menu of the Character Customizer. MJPEG may be available and is pretty good. Alternatively, you can install other codecs. The Lagarith video codec is a free lossless codec with good performance. It does produce quite large files, which the Character Customizer automatically converts to a more manageable format. You can find this codec here http://lags.leetcode.net/codec.html

Once you have the codec installed, you can select the codec from the Character Customizer by going to Options->Codec in the menu. Your movies will now be recorded using this codec. If a specified codec cannot be found, uncompressed video will be produced. (note that you can see a warning of this in the Unity console window when you select a non-existing codec; hit the ~ key in the Unity window to bring up the console).

For more detailed information on how to use the recording plugin for Unity, please refer to http://www.renderheads.com/portfolio/UnityAVProMovieCapture/docs/UnityAVProMovieCapture.pdf

Make changes globally to several or all lines in a project

Most settings are saved per utterance. You can select multiple utterances and change the setting for all selected ones at once. You can select multiple lines by:

  • Holding the mouse button and drag up and down over the lines.

  • Adding individual lines to your selection by holding the CTRL button and clicking lines with the mouse.

  • Adding a range of lines by selecting the first line with the mouse button, then select the last line while holding the Shift key.

You can select all lines by clicking a single line and then use CTRL + A.

Add new lines and speech

You can add new lines by typing them into the text box on the utterance list and hitting enter.

For each line you there are three ways of getting the character to speak the line:

  • Use text-to-speech. This is the default and you don't need to do anything for this. You can however change the voice from the Voice pull down menu in the Settings tab, and clicking the Set button

  • Record your own voice. Select the appropriate line. Hold down the Record button while speaking the line. Make sure to:

    • Start speaking immediately after pushing the button
    • Hold the button continuously while speaking
    • Release immediately after done speaking
    • Read the line accurately, since both the text and audio is used for creating a lip syncing schedule
    • Note that a window pops up indicating automatic analyzes of the text and audio to generate a lip syncing schedule; you can change the method from Lipsync under the Options menu.
  • Use an existing audio file:
    • Select the appropriate line; ensure that the text in this line exactly matches the text in the audio
    • Click the Add button in the Audio section
    • Select the file and click OK
    • Note that a window pops up indicating automatic analyzes of the text and audio to generate a lip syncing schedule; you can change the method from Lipsync under the Options menu.

Play a line

Either double click the line or click the Play button in the Line panel.

Change the background

There are several ways to change the background:

  • Select a pre-existing option from the Backdrop pull down menu at the top and click Set

  • Click Browse to choose one of your own pictures; these ideally are very wide in order to fill the entire backdrop without too much stretching; the aspect ratio is roughly 5:2

  • Click Color to open a color picker; this adds a hue to the existing picture. You can for instance use the existing Gradient01 background and select any color you want. Note that the default color selections are very bold; it is advised to click Define Custom Colors in the picker to select a more subtle one. This hue will also work on photo's.

Change the camera 

You can select a pre-defined camera from the Camera list in the Settings pane and click Set. Note that you can also directly move the camera around in Unity with the W, A, S & D keys. Any changes won't be saved, however. 

Change the character's gaze direction

By default the character looks at the Medium Center camera. To change the target, select one from the Gaze Target pull down menu and click Set. Alternatively, you can turn on Always Gaze At Camera in the Options menu.

Change nonverbal behavior

The Nonverbal Behavior Generator analyses the text a character says and dynamialy generates a behavior schedule for SmartBody to execute. You can turn off individual behaviors with the check boxes in the Settings pane. Furthermore, you can toggle whether the character will dislay ilde fidget behaviors (looking around, stretching. etc.) in the Options menu.

Record a movie

Make sure you have good codec installed first, like Lagarith, see above. After that, you can hit the Rec button in the Video panel at any time. Clicking it the first time will start the recording, clicking it a second time will stop the recording. These recorded movies by default are very large, so by default they are converted to a more manageable MP4 format. You can turn this behavior on and off in the Options menu.

You can also render all lines in a row as a batch from the Options menu. Note that this may take some time and that the subtitles may not fully line up. Also make sure that the first line is a text-to-speech line, as otherwise a bug will result in the same audio clip being used for all lines.

Record in-game audio and microphone audio at the same time

Currently this plugin only supports recording audio from a single Windows audio device. There is a “trick” you can use though. In Windows 7 (and perhaps Vista) you may be able to set your microphone to play through the speakers by going to: Control Panel ­> Sound ­> Recording ­> Select your microphone ­> right click ­>Properties ­> Listen ­> check "Listen to this device". You should then hear your microphone recording through your speakers. It’s recommended to use headphones during recording to prevent feedback from the speakers into the microphone.

In order for you to be able to see the 'Listen to this device' option, please make sure that you have installed the correct drivers for your soundcard.

Known Issues

  • For now, always explicitly use Save Project As, instead of just Save Project, since the latter may overwrite the wrong project file
  • When loading an existing project, the Camera list may be empty, which prevents you from adding more lines; set the camera as desired to fix this
  • The GUI does not always accurately reflect all settings properly when setting a new character or loading a new project
  • The background looks washed out; as a work around, you can play around with the color picker to see add a darker grey hue to the picture
  • At times the background turns black; using the color picker here should reset it
  • Text-to-speech may become unstable when changing the voice often on an existing line
  • The Always Gaze at Camera option may interfere with the saving of manually set gaze targets
  • If the first line uses a pre-recorded or user recorded audio file, batch video exporting does not work; as a work-around, you can make the first line text-to-speech
  • Subtitles in batch mode may look odd, timing wise

 

Message Flow

 

Setting character

renderer destroy Brad
renderer destroy Rachel
renderer create Brad Brad (on creating Brad)

 

Change Background

renderer background file Gradient01_UscShield.png

 

Change Background Color

renderer color 0 255 255

 

Add audio file

vhtspeechrecorder wizard_text Hello, I am a virtual human.
vhtspeechrecorder processaudiofile C:\Build4272-08-06-2013-ci\vhtoolkit\core\vhtoolkitUnity\Assets\StreamingAssets\Sounds\brad_askmeabout.wav Hello, I am a virtual human.
VHBuilder audiofile_created utterance_20130906121840_Iamavirtualhuman.
vhtspeechrecorder stop
render_text_overlay enable
nvbg_set_option disable_nvbg false

 

 

Flow of messages sent in order to record speech and generate audio file and bml:

vhtspeechrecorder start
120528396 acquireSpeech startSession
120528396 acquireSpeech set mic_utteranceFile.enabled true
120528396 nvbg_set_option disable_nvbg true
120528396 render_text_overlay disable
120528494 vhtspeechrecorder wizard_text I am having a great time
120528495 acquireSpeech startUtterance mic
120528503 vrSpeech start user0001 user
120528503 acquireSpeech startedListening mic 20130906115616,033 user0001 1378494328500
120528522 vrSpeech partial user0001 1 1.0 normal
120528665 vrSpeech partial user0001 2 1.0 normal WHAT
120528799 TO_ASR <start id="temp">
120528799 TO_ASR <stop id="temp">
120528800 TO_ASR <start id="temp">
120528800 TO_ASR <stop id="temp">
120528808 TO_ASR <start id="temp">
120528859 vrSpeech partial user0001 3 1.0 normal WHAT
120529103 vrSpeech partial user0001 4 1.0 normal DON'T
120529301 vrSpeech partial user0001 5 1.0 normal DON'T WHAT
120529496 vrSpeech partial user0001 6 1.0 normal DON'T WHAT IS
120529691 vrSpeech partial user0001 7 1.0 normal DON'T WHAT IS THE
120529706 acquireSpeech stopUtterance mic
120529707 acquireSpeech stopSession
120529707 acquireSpeech set mic_utteranceFile.enabled false
120529710 vrSpeech finished-speaking user0001
120529710 acquireSpeech stoppedListening mic 20130906115616,033 user0001 1378494329709
120529711 acquireSpeech stoppedSession null 20130906115616,033 1378494329711
120529885 vrSpeech partial user0001 8 1.0 normal DON'T WHAT IS THE
120529983 vrSpeech partial user0001 9 1.0 normal DON'T WHAT IS THE
120529995 vrSpeech interp user0001 1 1.0 normal DON'T WHAT IS THE
120529995 vrSpeech asr-complete user0001
120530048 TO_ASR <stop id="temp">
120535424 VHBuilder audiofile_created utterance_20130906120529_DON'TWHATISTHE
120535427 vhtspeechrecorder stop
120535427 render_text_overlay enable
120535427 NPCEditor <script target="user">document.getModel().getAnswers().getUtterances().get(document.getModel().getAnswers().findIndexOfWithID("utterance_20130206114108_IAMHAVINGAGREATTO")).setID("utterance_20130906120529_DON'TWHATISTHE");</script>
120535428 NPCEditor <script target="user">URL url = new File("C:/VHToolkit/data/VHBuilder/Scenarios/ExampleScenario/ExampleScenario_dialog.plist").toURI().toURL();document.saveToURLOfTypeForSaveOperation(url, document.getApplication().fileTypeForURL(url), com.leuski.af.Document.SaveOperation.kSaveAs);</script>
120535428 NPCEditor <script target="user">URL url = new File("C:/VHToolkit/data/VHBuilder/Scenarios/ExampleScenario/QNA.xml").toURI().toURL();document.writeToURL(url, new edu.usc.ict.npc.editor.io.ClassifierJAXBFileType());</script>
120535575 nvbg_set_option disable_nvbg false
120535575 acquireSpeech startSession
120535585 acquireSpeech startedSession null 20130906120535,575 1378494335585
120535675 acquireSpeech startUtterance mic
120535679 vrSpeech start user0002 user
120535679 acquireSpeech startedListening mic 20130906120535,575 user0002 1378494335678
120535717 vrSpeech partial user0002 1 1.0 normal

 

 

Play Utterance

vrExpress Brad all 1623190679 <?xml version="1.0" encoding="UTF-8" standalone="no" ?><act><participant id="Brad" role="actor" /><fml><turn start="take" end="give" /><affect type="neutral" target="addressee"></affect><culture type="neutral"></culture><personality type="neutral"></personality></fml><bml><speech id="sp1" ref="utterance_20130906121840_Iamavirtualhuman." type="application/ssml+xml">Hello, I am a virtual human.</speech></bml></act>

 

Record start/stop

renderer_record start
renderer_record stop

 

Set Camera

renderer setcamera set Camera01_mediumCt

Gaze at camera

sbm bml char Brad <gaze target="Camera01_mediumCt" sbm:joint-range="HEAD EYES NECK" sbm:joint-speed="500"/>')

Set Voice

sbm python scene.getCharacter("Brad").setVoiceBackupCode("Festival_voice_cmu_us_jmk_arctic_clunits")

 

Set Facial Expression

sbm char * viseme au_1_left 0 0
sbm char * viseme au_1_right 0 0
sbm char * viseme au_2_left 0 0
sbm char * viseme au_2_right 0 0
sbm char * viseme au_4_left 0 0
sbm char * viseme au_4_right 0 0
sbm char * viseme au_5 0 0
sbm char * viseme au_6 0 0
sbm char * viseme au_7 0 0
sbm char * viseme au_10 0 0
sbm char * viseme au_12_left 0 0
sbm char * viseme au_12_right 0 0
sbm char * viseme au_26 0 0
sbm char * viseme au_112 0 0
sbm char * viseme au_130 0 0
sbm char * viseme au_124 0 0
sbm char * viseme au_129 0 0
sbm char * viseme au_136 0 0
sbm char * viseme au_103 0 0
sbm char * viseme au_102 0 0
sbm char * viseme au_101 0 0
sbm char * viseme au_100 0 0
sbm char * viseme au_126 0 0
sbm char * viseme au_127 0 0
sbm char * viseme au_131 0 0
sbm char * viseme au_132 0 0
sbm char * viseme au_133 0 0
sbm char * viseme au_134 0 0
sbm char * viseme au_112 1 0.2


Generate Gestures checkbox

nvbg_set_option Brad speaker_gesture false

nvbg_set_option Brad speaker_gesture true

 

Generate Facial expressions/head movement checkbox

nvbg_set_option Brad nvbg_POS_rules false

nvbg_set_option Brad nvbg_POS_rules true

 

Toggle subtitles

render_text_overlay disable

render_text_overlay enable


Toggle Saccades

sbm bml char Brad <saccade finish="true"/>

sbm bml char Brad <saccade mode="listen"/>

sbm bml char Brad <saccade mode="talk"/>

sbm bml char Brad <saccade mode="think"/>



FAQ

See Main FAQ for frequently asked questions regarding the installer.  Please use the Google Groups emailing list for unlisted questions.