國立虎尾科技大學 |

Talking Human Synthesis : = Learning Photorealistic Co-Speech Motions and Visual Appearances From Videos.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Talking Human Synthesis :/
Reminder of title:	Learning Photorealistic Co-Speech Motions and Visual Appearances From Videos.
Author:	Zhang, Chenxu.
Description:	1 online resource (115 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-03, Section: A.
Contained By:	Dissertations Abstracts International85-03A.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798380342445

Talking Human Synthesis : = Learning Photorealistic Co-Speech Motions and Visual Appearances From Videos.
Zhang, Chenxu.

Talking Human Synthesis :Learning Photorealistic Co-Speech Motions and Visual Appearances From Videos. - 1 online resource (115 pages)

Source: Dissertations Abstracts International, Volume: 85-03, Section: A.

Thesis (Ph.D.)--The University of Texas at Dallas, 2023.

Includes bibliographical references

Talking video synthesis is a cutting-edge technology that enables the creation of highly realistic video sequences of individuals speaking. This technology has a wide range of applications in fields such as film-making, advertising, gaming, entertainment, social media, and is likely to continue to be an active area of research in the coming years. However, there are still many open questions and challenges in the field of talking video synthesis. In 3D talking face generation, most existing methods can only generate 3D faces with a static head pose, which is inconsistent with how humans perceive faces. Only a few works focus on head pose generation, but even these ignore the attribute of personality. In realistic talking face generation, it is still very challenging to generate photo-realistic talking faces that are indistinguishable from real captured videos, which not only contain synchronized lip motions, but also have personalized and natural head movements and eye blinks, etc. In full-body speech video synthesis, although substantial progress has been made in audio-driven talking video synthesis, there still remain two major difficulties: existing works 1) need a long sequence of training dataset (>1h) to synthesize co-speech gestures, which causes a significant limitation on their applicability; 2) usually fail to generate long sequences, or can only generate long sequences without enough diversity.To address those limitations, my research will be developed in a progressive manner, focusing on three main aspects. Firstly, we will delve into the generation of personalized head poses for 3D talking faces. Secondly, for realistic 2D talking faces, we propose a generation method that takes an audio signal as input and a short target video clip as a reference to synthesize a photo-realistic video of the target face with natural lip motions, head poses, and eye blinks that are synchronized with the input audio signal. Lastly, we propose a data-efficient ReMix learning method, which can be trained on monocular "in-the-wild" short videos to synthesize photo-realistic talking videos with full-body gestures.To generate personalized head poses for 3D talking faces, we propose a unified audio-driven approach to endow 3D talking faces with personalized pose dynamics. To achieve this goal, we establish an original person-specific dataset, providing corresponding head poses and face shapes for each video. To model implicit face attributes with input audio, we propose a FACe Implicit Attribute Learning Generative Adversarial Network (FACIAL-GAN), which integrates the phonetics-aware, context-aware, and identity-aware information to synthesize the 3D face animation with realistic motions of lips, head poses, and eye blinks. Finally, we make an audio-pose remixed latent space assumption to encourage unpaired audio and pose combinations, which results in diverse "one-to-many" mappings in pose generation. We also develop a dual-function inference scheme to regularize both the start pose and the general appearance of the next sequence, enhancing the long-term video generation of full continuity and diversity.Experimental results indicate that our methods can generate 1) person-specific head pose sequences that are in sync with the input audio and that best match with the human experience of talking heads, 2) realistic talking face videos with not only synchronized lip motions, but also natural head movements and eye blinks, 3) realistic synchronized full-body talking videos with training data efficiency with better qualities than the results of state-of-the-art methods.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798380342445Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

Audio-driven generationIndex Terms--Genre/Form:

554714
Electronic books.

Talking Human Synthesis : = Learning Photorealistic Co-Speech Motions and Visual Appearances From Videos.
LDR:05039ntm a22003857 4500 001 1142125
005 20240414211952.5
006 m o d
007 cr mn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798380342445
035 $a (MiAaPQ)AAI30740491
035 $a (MiAaPQ)0382vireo2198Zhang
035 $a AAI30740491
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Zhang, Chenxu. $0 (orcid)0000-0001-5014-6962 $3 1413543
245 1 0 $a Talking Human Synthesis : $b Learning Photorealistic Co-Speech Motions and Visual Appearances From Videos.
264 0 $c 2023
300 $a 1 online resource (115 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-03, Section: A.
500 $a Advisor: Guo, Xiaohu;Bhatia, Dinesh.
502 $a Thesis (Ph.D.)--The University of Texas at Dallas, 2023.
504 $a Includes bibliographical references
520 $a Talking video synthesis is a cutting-edge technology that enables the creation of highly realistic video sequences of individuals speaking. This technology has a wide range of applications in fields such as film-making, advertising, gaming, entertainment, social media, and is likely to continue to be an active area of research in the coming years. However, there are still many open questions and challenges in the field of talking video synthesis. In 3D talking face generation, most existing methods can only generate 3D faces with a static head pose, which is inconsistent with how humans perceive faces. Only a few works focus on head pose generation, but even these ignore the attribute of personality. In realistic talking face generation, it is still very challenging to generate photo-realistic talking faces that are indistinguishable from real captured videos, which not only contain synchronized lip motions, but also have personalized and natural head movements and eye blinks, etc. In full-body speech video synthesis, although substantial progress has been made in audio-driven talking video synthesis, there still remain two major difficulties: existing works 1) need a long sequence of training dataset (>1h) to synthesize co-speech gestures, which causes a significant limitation on their applicability; 2) usually fail to generate long sequences, or can only generate long sequences without enough diversity.To address those limitations, my research will be developed in a progressive manner, focusing on three main aspects. Firstly, we will delve into the generation of personalized head poses for 3D talking faces. Secondly, for realistic 2D talking faces, we propose a generation method that takes an audio signal as input and a short target video clip as a reference to synthesize a photo-realistic video of the target face with natural lip motions, head poses, and eye blinks that are synchronized with the input audio signal. Lastly, we propose a data-efficient ReMix learning method, which can be trained on monocular "in-the-wild" short videos to synthesize photo-realistic talking videos with full-body gestures.To generate personalized head poses for 3D talking faces, we propose a unified audio-driven approach to endow 3D talking faces with personalized pose dynamics. To achieve this goal, we establish an original person-specific dataset, providing corresponding head poses and face shapes for each video. To model implicit face attributes with input audio, we propose a FACe Implicit Attribute Learning Generative Adversarial Network (FACIAL-GAN), which integrates the phonetics-aware, context-aware, and identity-aware information to synthesize the 3D face animation with realistic motions of lips, head poses, and eye blinks. Finally, we make an audio-pose remixed latent space assumption to encourage unpaired audio and pose combinations, which results in diverse "one-to-many" mappings in pose generation. We also develop a dual-function inference scheme to regularize both the start pose and the general appearance of the next sequence, enhancing the long-term video generation of full continuity and diversity.Experimental results indicate that our methods can generate 1) person-specific head pose sequences that are in sync with the input audio and that best match with the human experience of talking heads, 2) realistic talking face videos with not only synchronized lip motions, but also natural head movements and eye blinks, 3) realistic synchronized full-body talking videos with training data efficiency with better qualities than the results of state-of-the-art methods.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Mass communications. $3 1335238
650 4 $a Information technology. $3 559429
653 $a Audio-driven generation
653 $a Cutting-edge technology
653 $a Visual appearances
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0489
690 $a 0708
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a The University of Texas at Dallas. $b Computer Science. $3 1182545
773 0 $t Dissertations Abstracts International $g 85-03A.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30740491 $z click for full text (PQDT)