Abstract
This framework utilizes the large-scale multimodal capabilities of GPT-4 to analyze speech or text input and employs the Facial Action Coding System (FACS) to synchronize text and mouth shape animations for 3D virtual characters (as illustrated in Figure 1). The results are encoded in JSON format through emotion analysis, with facial expressions driven by the Unreal Engine API, allowing direct mapping of FACS data to 3D digital humans. By implementing a state machine function, the framework modularizes facial expressions, ensuring calculations do not require a database configuration, thereby achieving a delay-free response and enhancing efficiency. Ultimately, this framework empowers users to rapidly generate facial AU data for visualizing nonverbal communication.
Keywords
GPT-4 sentiment analysis; FACS (Facial Action Coding System); 3D digital human; Unreal Engine; BlendShape
Reference
Yang, F., Fang, L., & Whang, M. (2024). A modular framework for data-driven facial expressions utilizing GPT-4 and FACS. 한국감성과학회 추계학술대회, 59-61.
Abstract
This framework utilizes the large-scale multimodal capabilities of GPT-4 to analyze speech or text input and employs the Facial Action Coding System (FACS) to synchronize text and mouth shape animations for 3D virtual characters (as illustrated in Figure 1). The results are encoded in JSON format through emotion analysis, with facial expressions driven by the Unreal Engine API, allowing direct mapping of FACS data to 3D digital humans. By implementing a state machine function, the framework modularizes facial expressions, ensuring calculations do not require a database configuration, thereby achieving a delay-free response and enhancing efficiency. Ultimately, this framework empowers users to rapidly generate facial AU data for visualizing nonverbal communication.
Keywords
GPT-4 sentiment analysis; FACS (Facial Action Coding System); 3D digital human; Unreal Engine; BlendShape
Reference
Yang, F., Fang, L., & Whang, M. (2024). A modular framework for data-driven facial expressions utilizing GPT-4 and FACS. 한국감성과학회 추계학술대회, 59-61.