Faculty
Louis-Philippe Morency
Louis-Philippe Morency is Assistant Professor in the Language Technology Institute at Carnegie Mellon University where he leads the Multimodal Communication and Machine Learning Laboratory (MultiComp Lab). He was formely research assistant professor in the Computer Sciences Department at University of Southern California and research scientist at USC Institute for Creative Technologies. Prof. Morency received his Ph.D. and Master degrees from MIT Computer Science and Artificial Intelligence Laboratory. His research focuses on building the computational foundations to enable computers with the abilities to analyze, recognize and predict subtle human communicative behaviors during social interactions. In particular, Prof. Morency was the lead investigator for the multi-institution effort that created SimSensei and MultiSense, two technologies to automatically assess nonverbal behavior indicators of psychological distress. He is currently chair of the advisory committee for ACM International Conference on Multimodal Interaction and associate editor at IEEE Transactions on Affective Computing.
Postdocs
Ph.D Students
Gaoussou Youssouf Kebe
Youssouf Kebe is a PhD student at the Language Technologies Institute of the School of Computer Science at Carnegie Mellon University. His research focuses on modeling social interactions and exploring the influence of individual differences such as personality traits on social communication and behavior across various modalities such as visual, verbal, and vocal. Youssouf aims to build interactive agents that can interpret social cues in real-time and respond appropriately in personalized ways, with potential applications in mental health, education, and human-robot interaction. He obtained his M.S in Computer Science from the University of Maryland, Baltimore County, where he worked on mitigating bias in language grounding models under the guidance of Cynthia Matuszek, and completed his B.S in Computer Engineering at Bursa Technical University in Turkey.
Leena Mathur
Leena Mathur is a PhD student at CMU’s School of Computer Science in the Language Technologies Institute. Her work is supported by the National Science Foundation Graduate Research Fellowship. Her long-term research goal is to advance virtual and embodied AI systems that can perceive, understand, and respond to multimodal human communication during social interactions. Her research focuses on foundations of multimodal learning and artificial social intelligence, as well as real-world applications of multimodal AI to enhance human health and well-being. Leena completed her B.S. in Computer Science, B.A. in Linguistics, and B.A. in Cognitive Science at the University of Southern California as a Goldwater Scholar, Astronaut Scholar, and CRA Outstanding Undergraduate Researcher Awardee. As an undergraduate, she worked with Maja Matarić and Khalil Iskarous at USC, Ralph Adolphs at Caltech, Michael Shindler at UC Irvine, and Rémi Lebret at the Ecole Polytechnique Fédérale de Lausanne.
Alex Wilf
Alex Wilf is a doctoral student in the Language Technologies Institute in the School of Computer Science. He is interested in multimodal representation learning, specifically for tasks involving how people express themselves – both when they are alone and in groups, and across both modalities and languages. He is currently interested in the promise of self-supervised learning and graph neural network architectures for use in designing novel multimodal networks. He completed his B.S. in Computer Science at the University of Michigan, where he worked with Emily Mower Provost on building robust and generalizable models for speech emotion recognition.
Martin Q. Ma
Martin Q. Ma is a Ph.D. student at the Language Technologies Institute in the School of Computer Science at Carnegie Mellon University. His research interests include deep representation learning, self-supervised learning, contrastive learning, and applications to problems in multimodal learning. Martin obtained his Master’s degree in the School of Computer Science, Carnegie Mellon University, and his Bachelor’s degree in math and computer science from Brandeis University.
Paul Pu Liang
Paul is Ph.D. student in the Machine Learning Department at CMU, advised by Louis-Philippe Morency and Ruslan Salakhutdinov. His long-term research goal is to build socially intelligent embodied agents with the ability to perceive and engage in multimodal human communication. As steps towards this goal, his research focuses on 1) the fundamentals of multimodal learning, specifically the representation, translation, fusion, and alignment of heterogeneous data sources, 2) human-centered language, vision, speech, robotics, and healthcare applications, as well as 3) the real-world deployment of socially intelligent agents by improving fairness, robustness, and interpretability in real-world applications. Previously, he received an M.S. in Machine Learning and a B.S. with University Honors in Computer Science from CMU
Victoria Lin
Victoria is a Ph.D. student in the Department of Statistics at Carnegie Mellon University. Her research interests include multimodal behavior analysis, uncertainty quantification, causal inference for high-dimensional and complex longitudinal data, and applications in mental health and medical diagnostics. Prior to joining CMU, Victoria was a researcher with Miguel Hernán in the Program for Causal Inference at the Harvard School of Public Health. She received her M.S. in data science from CMU and her joint A.B. in statistics and in molecular and cellular biology from Harvard University.
Torsten Wörtwein
Torsten Wörtwein is a Ph.D. student in the Language Technologies Institute. His research interests include affective computing, integrating statistical methods in deep learning, multimodal behavior analysis, and healthcare analytics with a focus on symptoms of depression and psychotic disorders. He is currently focusing on emotion recognition and its technical challenges including uncertainty estimation and personalization. Previously, he worked on public speaking anxiety and public speaking performance assessment as well as on several computer vision projects. He received his B.Sc. and M.Sc. in Informatics from the Karlsruhe Institute of Technology in Germany and a M.Sc. in Language Technologies from Carnegie Mellon University.
Alexandria Vail
Alexandria K. Vail is a Ph.D. student in Human-Computer Interaction. Her research interests include user modeling, affective computing, and multimodal behavior analysis, particularly within the context of healthcare and clinical decision support technologies. Her work recently received the Best Paper Award at the International Conference on User Modelling, Adaptation, and Personalization; this work has also been recognized with distinction at the International Conference on Intelligent Tutoring Systems and the International Conference on Educational Data Mining. Before joining CMU, she received the B.S. degree in Computer Science and the B.S. degree in Mathematics from North Carolina State University, with minor concentrations in Cognitive Science and Physics.
Master Students
Yiwei Lyu
Yiwei Lyu is a Master’s student in Machine Learning. His research focuses on multimodal machine learning, evaluation and interpretation of multimodal models, and multimodal pretraining/transfer learning. He has previously worked on text style transfer and automated software repair. He has earned a B.S. in Computer Science at Carnegie Mellon University.
Santiago Benoit
Santiago Benoit is an undergraduate student majoring in Artificial Intelligence at Carnegie Mellon University. He is currently working on generative modeling of multimodal speech from text, encoderless stochastic variational inference, and temporal sequence modeling, especially applied to music generation. His research interests are in both theoretical and applied artificial intelligence. Some of his favorite subfields in AI include multimodal machine learning, generative modeling, and reinforcement learning.
Research Engineer
Visiting Scholar
Lab Manager
Nicki Siverling
Nicki is the MultiComp Lab’s Coordinator. She has 12 years of experience in managing social/behavioral research labs and is currently providing support on several grants. Before joining CMU, she served as Coordinator of the Affect Analysis Group at the University of Pittsburgh. Nicki received her B.A. from the University of Pittsburgh.
Staff
Taylor Gautreaux
Taylor is a research associate in the MultiComp Lab. She provides support to several projects with a particular focus on data creation, processing and management. Her current research interests lie in helping intelligent systems understand human behaviors and social interactions. She previously completed a BA degree in Mass Communication with a focus on journalism at Nicholls State University.
Graduate Research Assistants
Yinghuan Zhang
Yinghuan is a graduate student in Master of Information Technology Strategy (MITS) program at CMU. Her research interests include cross-modal supervision, representation learning, and few-/zero-shot learning in multimodal learning. Prior to joining CMU, she worked on efficient neural architecture search at Shanghai Qi Zhi Institute for 1 year. She received a B.E in Geotechnical Engineering at Harbin Institute of Technology.
Sanika Natu
Sanika Natu is a Master’s student in the School of Computer Science at Carnegie Mellon University. Her research interests include multimodal learning, affective computing, computational ethics, and health/behavior analytics. In the MultiComp Lab, she is currently working on research for grounding text, image, and language in educational lecture videos. Before joining her Master’s program at CMU she received her B.S. in Information Systems at CMU and worked for a few years in industry.
Yuxin Xiao
Yuxin Xiao is a Machine Learning Master student at CMU. His long-term research goal is to safely automate decision-making in society with knowledge extracted from structured data. He is currently working on quantifying the uncertainty in pre-trained language models via empirical studies. He previously earned a B.S. in Computer Science and a B.S. in Statistics and Mathematics from UIUC.
Blake Martin
Blake is a master’s student in Machine Learning. His primary research interests are in deep learning for computer vision and natural language processing, with a current focus on multilingual machine translation. He previously received a B.S.E. is Data Science from the University of Michigan, where his research focused on machine learning applications in human health and behavior.
Nihal Jain
Nihal Jain is a Master’s in Machine Learning student at CMU. His research focuses on developing interpretability tools for analysis of multimodal machine learning models. He has previously worked on applications of deep learning in information retrieval when he was interning at Adobe Research. Before joining CMU, he obtained a B.E. in Computer Science at the Birla Institute of Technology & Science (BITS), Pilani, India.
Pratik Joshi
Pratik Joshi is a Master’s student in the Language Technologies Institute in the School of Computer Science at Carnegie Mellon University. His research interests revolve around multimodal learning, commonsense reasoning, and multilingual systems. He is currently working on implementing continual learning techniques for multimodal generation to accommodate domain shift. Prior to joining CMU, he worked at Google Research and Microsoft Research for 2 years on problems in natural language inference, semantic parsing, and low-resource language analysis. He graduated with a Bachelor’s in Computer Science from BITS Pilani.
Catherine Cheng
Catherine Cheng is a Master’s student in Machine Learning Department. Her research interests include multimodal co-learning, representation learning, and potential alignment between language and emotion representation. She has previously worked on the robustness evaluation of multimodal models. She received a B.S. in Computer Science and a B.S. in Mathematical Science at Carnegie Mellon University.
Undergraduate Research Assistants
Aneesha Sampath
Aneesha is an undergraduate student at Carnegie Mellon University majoring in Artificial Intelligence and minoring in Language Technologies. Her research interests lie in Natural Language Processing and Multimodal Deep Learning. Her current project involves uncertainty quantification.
Sheryl Mathew
Sheryl Mathew is an undergraduate computer science student at CMU. She is interested in exploring artificial intelligence, especially natural language processing and deep learning. Currently, she is working on extracting usable features for multimodal machine learning models in the social intelligence project.
Shuyu Zhang
Shuyu Zhang is an undergraduate in the BXA program studying computer science and art. She is currently working on the CMU-MOSEAS project by organizing data, processing datasets, and teaching others to do the same. Her research interests include multimodal machine learning, multimodal language models, and ethics of machine learning.
Edward Garemo
Edward Garemo is an undergraduate student in SCS studying computer science. He is interested in understanding and developing the theoretical foundations and practical implementations of automated generalizable learning models.
Evan Zhang
Hi, I’m Evan! I’m currently a freshman in SCS and plan on concentrating in machine learning. I became very interested in both software engineering and data science after completing various projects in both fields. In my free time, I like to play sports (ping pong, frisbee, basketball), listen to music (pop for sure), and hang out with friends.
Arvin Wu
Arvin is an undergraduate computer science student at CMU. He is interested in Multi-Modal Machine Learning, Computer Vision, and Natural Language Processing. He is currently working on a data mining project for gesture generation research.