...
MultiSense is a perception framework that enables multiple sensing and understanding modules to inter-operate simultaneously, broadcasting data through the Perception Markup Language. MultiSense currently contains GAVAM, CLM FaceTracker and FAAST which you can use be used with a webcam or Kinect. The Toolkit provides an example of how to use the MultiSense framework (also known as multimodal framework, developed by Multicomp Lab).
...
- CLM (developed by Jason Saragih et .al)
- Gavam face-tracker (developed by Louis-Philippe Morency's Multicomp Lab) face-tracker
- Kinect
- FAAST (developed by Mixed Reality Group)
- ssi Library (developed by Johannes Wagner, University of Augsburg).
...
- ssi_vhmsger (\core\multisense\ssi_vhmsger\)
This is the core application that uses MultiSense . It will only work if you have a web cam and it only works if there is a webcam installed on the system. This application integrates many different MultiSense components like vision, audio and analysis into a single application that outputs PML. - PerceptionTest (\core\multisense\PerceptionApplication)
This application is a middle-man between the ssi_vhmsger and any other application which plans to use the PML for specific capabilities. For example, in the Toolkit, this application takes the PML as input (via the vrPerception message) and outputs messages to Smartbody to mimic the head orientation of the detected user.
...
MultiSense currently supports multiple modules, including a vision module (for tracking face features like smile, gaze, attention, activity, etc) and a speech recognition module. To use MultiSense, run the MultiSense application, and it . It will broadcast vrPerception messages (based on PML) generated by each module, which can be received by any other module or external component.
...
Two components are required to make full use of MultiSense, : the MultiSense Application , and the Perception Application.
...
To run the MultiSense Application, either:
- Run the application from the Launcher (go to the 'Perception Components' tab)
- Navigate to \bin\multisense\ssi_vhmsger\bin folder and run "run.bat".
By default, only the Gavam and CLM facetrackers are turned on (for the toolkitToolkit; otherwise only the Gavam is turned on).
...
- If you wish to change the configuration for running other modules, you can edit \bin\multisense\ssi_vhmsger\bin\config.ini and change the "useXXXX" parameter to "true" to make a module work.
- If the dependencies for the module are not setup correctly, the ssi_vhmsger will crash.
- If you want to run running FAAST, make sure the "useKinect" as well as "useFAAST" parameters are both "true" in the config.ini.
- Make sure to have a web cam A webcam must be connected to the computer.
- If the application fails to start, one of the dependencies might be missing.
- ssi_vhmsger will not exist when you click the "Kill" button from the launcher is clicked. Also, if you kill it's killed from the launcher, the ssi tool will turn yellow.
...
To run the PerceptionTest application, either:
- Run it from the Launcher (go to the 'Perception Components' tab)
- Navigate to \bin\multisense\PerceptionTest\ and run "PerceptionTest.exe".
Please contact either Unlicensed user or Unlicensed user if you have any issues in running the application.
Developers
Setting up MultiSense
To setup the MultiSense, the user will need to checkout the repository from -:
https://svn.ict.usc.edu/svn_vision/Projects/trunk/ssi-ict
After the user downloads the framework is downloaded, they will need to build a module which allows to send and receive VHMsgs (located in \tools\ssi_vhmsger; The the solution file is located at \tools\ssi_vhmsger\build\ssi_vhmsger.sln). Make "ssi_vhmsger_test as the "Startup project". The user will also need to set Set the working directory as "..\..\..\..\bin". Once the project builds successfully, the user will need to have a web cam connected connect a webcam to the computer to send vrPerception messages using activeMQ ActiveMQ (note that Note: this module has a dependency on activeMQ ActiveMQ and it will need to be installed. The latest installer for apache-activemq can be found at \\vhbuild\SASO-Installs).
You can check out the For a detailed description of how to setup the project at the following location -go to /wiki/spaces/VHTK/pages/14583447.
As an illustration of integration, the please check out the following link (integrates the toolkit with MultiSense)
...
- The MultiSense Application currently does not listen to the vrKillComponent message, which means if the user tries . The application must be manually closed; attempts to kill it from the Launcher , it will turn will cause it to highlight yellow and not close. The user has to manually close the application.
- If the application does not seem appear to start up, make sure your camera is working properly. Also, if there is camera.option file present at the location "\bin\multisense\ssi_vhmsger\bin", delete it and try to run the application again. If there is a problem with running MultiSense, please contact us at vh-support@ict.usc.edu.
- If the user enables Gavam and sees a big red X appears on the screen after Gavam is enable, it means that Gavam is not able to detect the face properly. Currently The application needs to be restarted; currently, there is no other way to reset (other than restarting the application). The user can restart CLM tracker can be restarted, though, by right clicking the output window.
...
See Main FAQ for frequently asked questions regarding the installer. Please use the Google Groups emailing list for unlisted questions.