Analysis of Head Gesture and Prosody Patterns for Prosody-Driven Head-Gesture Animation

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 30; no. 8; pp. 1330 - 1345
• Main Authors: Sargin, M.E., Yemez, Y., Erzin, E., Tekalp, A.M.
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01-08-2008; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Artificial intelligence; Audio input/output; Cluster Analysis; Computer science; control theory; systems; Euler angles; Exact sciences and technology; Face - physiology; Face and gesture recognition; Facial animation; Head - physiology; Hidden Markov models; Humans; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Imaging, Three-Dimensional - methods; Loudspeakers; Mapping; Mathematical models; Pattern analysis; Pattern Recognition, Automated - methods; Pattern recognition. Digital image processing. Computational geometry; Reproducibility of Results; Sensitivity and Specificity; Spatial databases; Speech; Speech - physiology; Speech analysis; Speech synthesis; Streaming media; Synthesis; Temporal logic; Testing; Transplants; Videos; Audio input/output; Pattern analysis; Face and gesture recognition; Multimedia; Euler equation; Head; Segmentation; Information system; Objective analysis; pattern analysis and recognition; Markov model; Pattern recognition; Computer animation; Modeling; Subjective evaluation; animation; face and gesture recognition; Prosody; Correlation analysis; Facies; Gesture; Hidden Markov model; Artificial intelligence; Speaker; Multimedia information systems; speech analysis
• ISSN: 0162-8828; 1939-3539
• DOI: 10.1109/TPAMI.2007.70797

We propose a new two-stage framework for joint analysis of head gesture and speech prosody patterns of a speaker towards automatic realistic synthesis of head gestures from speech prosody. In the first stage analysis, we perform hidden Markov model (HMM) based unsupervised temporal segmentation of head gesture and speech prosody features separately to determine elementary head gesture and speech prosody patterns, respectively, for a particular speaker. In the second stage, joint analysis of correlations between these elementary head gesture and prosody patterns is performed using Multi-Stream HMMs to determine an audio-visual mapping model. The resulting audio-visual mapping model is then employed to synthesize natural head gestures from arbitrary input test speech given a head model for the speaker. In the synthesis stage, the audio-visual mapping model is used to predict a sequence of gesture patterns from the prosody pattern sequence computed for the input test speech. The Euler angles associated with each gesture pattern are then applied to animate the speaker head model. Objective and subjective evaluations indicate that the proposed synthesis by analysis scheme provides natural looking head gestures for the speaker with any input test speech, as well as in ``prosody transplant" and ``gesture transplant" scenarios.