A Demonstration of the Perception System in SimSensei, a Virtual Human Application for Healthcare Interviews

 

The success of video-sharing and social network websites has led to an increased posting of online multimedia content, with a large proportion of these videos being human-centric. The sheer amount of such data promotes research on behavior understanding that can discover the affective and social states within human-centric multimedia content.  We can model personality and social interaction via temporal modeling and multimodal fusion.

Rapport: Rapport is a harmonious relationship in which people are coordinated and understand each other. The power of rapport in social interactions has inspired us to develop the intelligent virtual agent that induces the subjective feeling and many of the behavioral benefits of the psychological concept of rapport. Moreover, we develop the system of automatic detection for remote peer tutoring.

Persuasiveness: Persuasiveness is a high-level personality trait that quantifies the influence a speaker has on the beliefs, attitudes, intentions, motivations, and behavior of the audience. With social multimedia becoming an important channel in propagating ideas and opinions, analyzing persuasiveness is very important. Inspired by the success of deep learning techniques, we study the persuasiveness prediction with deep multimodal fusion that combines signals from the visual, acoustic, and text modalities effectively.

 

Psychological Distress is a group of overlapping mental disorders which include depression, anxiety, and Post Traumatic Stress Disorder (PTSD). While depressed individuals often lose interest in activities and feel sad or overwhelmed, PTSD is mainly characterized by flashbacks to traumatic events.

We evaluated behaviors of a large group of individuals in interviews with a virtual human. In these interviews, people answered questions related to symptoms of depression and PTSD. During these interactions, we observed more downwards gaze behavior for self-reported depression, a more tense voice for self-reported PTSD and self-reported depression, and less smile intensity for self-reported PTSD. Currently, we focus on automatically estimating behavioral markers which are typically measured by therapists and clinicians.