VHBuilder

Owned by Former user (Deleted)
Last updated: Oct 16, 2013 by Andrew LeedsVersion comment
10 min read

Overview

VHBuilder is a tool that allows you to quickly author and review verbal and nonverbal behaviors for one or more characters. It is aimed at quickly getting basic characters up and running; power users can then move on to using the NPCEditor or NVBG directly for more fine-grained control. 

It abstracts the end user from individual components of the Toolkit pipeline and offers a go-to place for easy authoring of characters. 

Quick facts:

  • Location: /tools/VHBuilder
  • Language: C#
  • Distribution: source
  • Platform(s): Windows

Users

Creating a new scenario

The VHBuilder application requires the user to either create or load a scenario before authoring the virtual human characters. A scenario is basically a sandbox which includes all the files and data associated with it. Click on the File menu option on the top left and select the 'Create Scenario' option as shown below

On selecting the 'Create Scenario' option, you will be presented with a dialog as shown below. Please type in the name of the new Scenario you want to create, select where you want to create it and select 'Save' as shown. On hitting 'Save' above, it will create a scenario and it will also launch all the components required to create the new scenario.

On creating the scenario, the VHBuilder tool will launch all required components and while it is doing so, it will indicate that background processes are running, using the indicator on the top right (as shown below)

Once it is done loading the scenario, it will indicate that it is ready, again using the indicator on the top right as shown below.

Adding/Querying characters to the scenario

You cannot begin editing the scenario unless you add new characters to the scene. In order to do this, you can use either the 'Query' or 'Add' button as shown below.

The 'Query' button works only if you have an instance of Unity running in the background with a scene consisting of Toolkit characters. The 'Query' button basically queries Unity for the characters that are currently loaded and it adds these characters to the VHBuilder's list of characters as shown below.

If you don't have a Unity instance running, another way to add the characters is hitting the 'Add' button next to the 'Query' button. This will pop up a window as shown below

You can type in the name of the character you want to add and it will get added to the VHBuilder's list of characters just like the 'Query' button does.

Applying changes

You need to hit the 'Apply' button in order to apply the changes that you make (characters, questions, answers, behaviors etc.). On hitting the 'Apply' button, the tool will create a language model, classify the NPCEditor and initialize NVBG with the current scenario. Every time you make a change to the scenario, the Apply button will turn red as a reminder that you need to press it to see your changes.

Adding a question and answer

In order to add a question/answer, first select the character that you want to add questions/answers for, from the character drop-down menu on the top left. Then type the question/answer in the respective text-box and hit enter. This will add the question/answer. 

NOTE: The newly added answer is spoken by the character that is currently selected in the CharacterCombo box on the top left. e.g. in the image below, 'Rachel' is the currently selected character, so any answer that is entered will be spoken by her.

Linking a question to an answer

Once you have added the question and answer, you can link them together by selecting them and clicking the 'Link' button as shown below. You can select multiple questions and answers at the same time and link them as shown.

Recording the audio for an answer

In order to record audio for a newly added answer, first select the answer by clicking on it. This will highlight the answer. Now hold down the 'Record' button and speak into the microphone and release the button when you are done speaking. 

On releasing the 'Record' button, you will see a window popup as shown below. This generates the bml with the lipsync for the audio you just recorded. Please wait until the tool indicates otherwise. Once the tool indicates it is done, the audio has been recorded and the lipsync generated. You should now be able to ask the corresponding question and hear the character respond in the recorded voice.

In case you don't have FaceFx installed on your machine, the window you see will look different as VHBuilder will default to using the VisemeSchedulerSOX tool instead which is our in-house tool for generating basic lipsync.

Selecting nonverbal behavior files

A 'behavior file', contains the mapping from the spoken words to animations. For each character, you can select the 'behavior file' that you want to use for that character from the list of available files in the drop-down list. e.g. in the example below, we have selected 'rule_input_Rachel_template.xml' for the selected character 'Rachel'. Once you select the behavior file for a character, the tool will remember it. 

Modifying nonverbal behavior rules

In order to modify a behavior rule, you select the 'Rule' that you want to modify and the 'Posture' that you want the rule to apply to, using the drop-down lists as shown below. The drop-down list will contain the available rules and postures for the character.

Now you can add words that will trigger this rule under the 'Words' section by typing the word in the text-box and hitting the enter key. You are also able to select the animations that will be triggered when the character speaks that particular word, by clicking on the animation name and selecting it. The list of selected animations for a particular rule is displayed in the bottom right as shown.

Notes:

a) The character only performs the animation when the words in the 'Words' list are spoken and he is in the specified posture.

b) You can select multiple animations from the 'Available animations' list by holding down the 'ctrl' key and clicking on them.

c) If you have Unity running with the characters loaded, you can double click on the animations (or select a posture) and see how they play on the character.

d) You can select and hit enter to add animations and hit delete in the 'Selected Animations' list to remove animations.

Adding a new nonverbal behavior rule

You can add a new rule by hitting the '+' button next to the Rule drop-down list. This will pop up a dialog box as shown below.

 

Enter the name of the new rule. Hitting OK will create a new rule and you can go ahead and add words to this rule and select animations for it. Rule names are mostly for organizational purposes allowing you to have different rule names for different set of words.

Loading an existing scenario

Click on the File menu option on the top left and select 'Load Scenario' as shown

A dialog box will pop up as shown below prompting you to select an existing scenario. Select a previously saved scenario folder to load up an existing scenario. e.g. In the toolkit trunk, under data/VHBuilder/Scenarios, there is an existing scenario called ExampleScenario. Select the scenario you want to load

On selecting the example scenario and clicking 'OK', you should see a few windows show up and minimize. This should include the NVBG, NPCEditor, VHTSpeechRecorder, AcquireSpeech, ElvinParser, PocketSphinx-sonic-server and TTSRelay.

During this interval, the tool should indicate that it is busy loading. It does this using a 'wait' image on the top right as shown below

Once the scene is loaded and all the components are initialized, the tool will indicate that it is ready as shown below.

At this point, the scene has been loaded and you can begin interaction with the characters. Note that currently the tool does not load up the characters in Unity automatically (yet to be implemented). So you will have to open up an empty scene in the Unity editor (e.g. CampusEmpty) and drag the listed characters in the scene and hit play in Unity. The characters that are in the scene can be seen in the combo-box as shown below. Once Unity loads up with the characters, then you can begin interacting with the characters just as you always do. The characters will respond to the questions with the answers as they are linked.

Saving the scenario

1) In order to save the scenario and all the changes made to it, select the File menu option and hit the 'Save Scenario' as shown below

Pinging background components

In case of unexpected functionality, a good way to check if all background components are functioning is to ping them and make sure they are running. This can be done using the 'Ping' menu option. If all components are running fine, then the VHBuilder tool will indicate so. If any component is missing, then you will be notified of this.

Relaunching background components

If for any reason, one of the components fails to function or exits you can relaunch the components using the 'Relaunch' menu option as shown below. This can be used to restart the system with all components functioning in the background.

 

Message API

 

Query Unity for characters : 

vht_get_characters

VHBuilder character_names Rachel Brad

Link question and answer : 

NPCEditor <script target="user">document.getModel().setLinkValue(document.getModel().getQuestions().findIndexOfWithID("Anybody-9"),document.getModel().getAnswers().findIndexOfWithID("utterance_20130206114053_AMYNAMEISITHAVEA"),6);</script>


Flow of messages sent in order to record speech and generate audio file and bml:

vhtspeechrecorder start
120528396 acquireSpeech startSession
120528396 acquireSpeech set mic_utteranceFile.enabled true
120528396 nvbg_set_option disable_nvbg true
120528396 render_text_overlay disable
120528494 vhtspeechrecorder wizard_text I am having a great time
120528495 acquireSpeech startUtterance mic
120528503 vrSpeech start user0001 user
120528503 acquireSpeech startedListening mic 20130906115616,033 user0001 1378494328500
120528522 vrSpeech partial user0001 1 1.0 normal
120528665 vrSpeech partial user0001 2 1.0 normal WHAT
120528799 TO_ASR <start id="temp">
120528799 TO_ASR <stop id="temp">
120528800 TO_ASR <start id="temp">
120528800 TO_ASR <stop id="temp">
120528808 TO_ASR <start id="temp">
120528859 vrSpeech partial user0001 3 1.0 normal WHAT
120529103 vrSpeech partial user0001 4 1.0 normal DON'T
120529301 vrSpeech partial user0001 5 1.0 normal DON'T WHAT
120529496 vrSpeech partial user0001 6 1.0 normal DON'T WHAT IS
120529691 vrSpeech partial user0001 7 1.0 normal DON'T WHAT IS THE
120529706 acquireSpeech stopUtterance mic
120529707 acquireSpeech stopSession
120529707 acquireSpeech set mic_utteranceFile.enabled false
120529710 vrSpeech finished-speaking user0001
120529710 acquireSpeech stoppedListening mic 20130906115616,033 user0001 1378494329709
120529711 acquireSpeech stoppedSession null 20130906115616,033 1378494329711
120529885 vrSpeech partial user0001 8 1.0 normal DON'T WHAT IS THE
120529983 vrSpeech partial user0001 9 1.0 normal DON'T WHAT IS THE
120529995 vrSpeech interp user0001 1 1.0 normal DON'T WHAT IS THE
120529995 vrSpeech asr-complete user0001
120530048 TO_ASR <stop id="temp">
120535424 VHBuilder audiofile_created utterance_20130906120529_DON'TWHATISTHE
120535427 vhtspeechrecorder stop
120535427 render_text_overlay enable
120535427 NPCEditor <script target="user">document.getModel().getAnswers().getUtterances().get(document.getModel().getAnswers().findIndexOfWithID("utterance_20130206114108_IAMHAVINGAGREATTO")).setID("utterance_20130906120529_DON'TWHATISTHE");</script>
120535428 NPCEditor <script target="user">URL url = new File("C:/VHToolkit/data/VHBuilder/Scenarios/ExampleScenario/ExampleScenario_dialog.plist").toURI().toURL();document.saveToURLOfTypeForSaveOperation(url, document.getApplication().fileTypeForURL(url), com.leuski.af.Document.SaveOperation.kSaveAs);</script>
120535428 NPCEditor <script target="user">URL url = new File("C:/VHToolkit/data/VHBuilder/Scenarios/ExampleScenario/QNA.xml").toURI().toURL();document.writeToURL(url, new edu.usc.ict.npc.editor.io.ClassifierJAXBFileType());</script>
120535575 nvbg_set_option disable_nvbg false
120535575 acquireSpeech startSession
120535585 acquireSpeech startedSession null 20130906120535,575 1378494335585
120535675 acquireSpeech startUtterance mic
120535679 vrSpeech start user0002 user
120535679 acquireSpeech startedListening mic 20130906120535,575 user0002 1378494335678
120535717 vrSpeech partial user0002 1 1.0 normal



Messages sent out on clicking Apply button

 


121025076 NPCEditor <script target="user">document.startTrainingAll();</script>
121025076 nvbg_create_character Rachel
121025077 nvbg_set_option Rachel rule_input_file rule_input_ChrRachel_template.xml
121025077 nvbg_set_option Rachel nvbg_POS_rules false
121025077 nvbg_set_option Rachel posture ChrBrad@Idle01
121025077 nvbg_set_option refresh_transform
121025077 sbm bml char Rachel posture ChrBrad@Idle01
121025077 nvbg_create_character Brad
121025078 nvbg_set_option Brad rule_input_file rule_input_ChrBrad_template.xml
121025078 nvbg_set_option Brad nvbg_POS_rules false
121025078 nvbg_set_option Brad posture ChrBrad@Idle01
121025078 nvbg_set_option refresh_transform
121025078 sbm bml char Brad posture ChrBrad@Idle01
121025078 acquireSpeech stopSession
121025078 vrKillComponent asr-server
121025080 vrProcEnd asr-server
121025081 NPCEditor <script target="user">URL url = new File("C:/VHToolkit/data/VHBuilder/Scenarios/ExampleScenario/ExampleScenario_dialog.plist").toURI().toURL();document.saveToURLOfTypeForSaveOperation(url, document.getApplication().fileTypeForURL(url), com.leuski.af.Document.SaveOperation.kSaveAs);</script>
121025081 NPCEditor <script target="user">URL url = new File("C:/VHToolkit/data/VHBuilder/Scenarios/ExampleScenario/QNA.xml").toURI().toURL();document.writeToURL(url, new edu.usc.ict.npc.editor.io.ClassifierJAXBFileType());</script>


Message sent out on posture change

sbm bml char Rachel posture ChrHarmony@IdleCrossedArms01

 

Message sent out on animation double click

sbm bml char Rachel anim ChrBrad@Idle01_BeatHighBt01

 

Message sent out on adding question

NPCEditor <script target="user">edu.usc.ict.npc.editor.model.Person domain = document.getModel().getDefaultSpeaker(); edu.usc.ict.npc.editor.model.EditorUtterance eu = new edu.usc.ict.npc.editor.model.EditorUtterance("oooga booga", "1987503103635144181104057941", domain, new Date()); document.getManagedObjectContext().insertObject(eu); document.getModel().getQuestions().getUtterances().add(eu);</script>
135510407 NPCEditor <script target="user">URL url = new

 

Message sent out on adding answer

NPCEditor <script target="user">edu.usc.ict.npc.editor.model.Person domain1 = document.getModel().getSpeakers().get(0); edu.usc.ict.npc.editor.model.EditorUtterance eu = new edu.usc.ict.npc.editor.model.EditorUtterance("ooga booga", "48309906635144181677535283", domain1, new Date());eu.addAnnotation(document.getModel().getCategoryWithID("speaker").tokenWithID("Rachel")); document.getManagedObjectContext().insertObject(eu); document.getModel().getAnswers().getUtterances().add(eu);</script>

Known Issues

  • All answers within a given scenario need to be unique, regardless of whether different characters speak them.

FAQ

See the Main FAQ. Please use the Google Groups emailing list for unlisted questions.