國立虎尾科技大學 |

Towards Integrated Audio-Visual Learning : = from Vision-to-Audio Generation to a Unified Audio-Visual Framework.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Towards Integrated Audio-Visual Learning :/
Reminder of title:	from Vision-to-Audio Generation to a Unified Audio-Visual Framework.
Author:	Su, Kun.
Description:	1 online resource (158 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-10, Section: B.
Contained By:	Dissertations Abstracts International85-10B.
Subject:	Electrical engineering. -
Online resource:	click for full text (PQDT)
ISBN:	9798382215228

Towards Integrated Audio-Visual Learning : = from Vision-to-Audio Generation to a Unified Audio-Visual Framework.
Su, Kun.

Towards Integrated Audio-Visual Learning :from Vision-to-Audio Generation to a Unified Audio-Visual Framework. - 1 online resource (158 pages)

Source: Dissertations Abstracts International, Volume: 85-10, Section: B.

Thesis (Ph.D.)--University of Washington, 2024.

Includes bibliographical references

The interplay between audio and visual signals, rich in correlations across various scales, significantly impacts human perception and drives a consistent demand for audio-visual applications across fields such as video production, animation, and virtual reality. Historically, the creation and adaptation of audio content has been predominantly a manual process reliant on the expertise of Foley artists. Exploring automated systems capable of assisting with such tasks and achieving comparable proficiency suggests an intriguing prospect. In recent years, deep learning-based methodologies have shown considerable promise in handling image, video, and text data. Nonetheless, the task of integrating audio with visual inputs introduces distinct challenges that stem from the fundamental differences and complexities inherent to each modality. In particular, techniques for generating non-speech audio, such as music, object material impact sounds, natural sounds, and spatial sound effects, have received limited exploration.Audio and visual signals can be represented in various ways, with their connections varying across different scenarios. The research domain of learning to connect and relate audio and visual signals, termed audio-visual learning, has traditionally focused on either audio-visual representation learning or on generative modeling of one modality conditioned on the other. My research traverses audio-visual learning and connects audio-visual representation and generation from three distinct perspectives.In the first category, my investigation focuses on vision-to-audio generation through an intermediate representation, which serves as a bridge to link visual and audio domains. For instance, musical notes can translate piano keystrokes into their corresponding sounds, and the rhythm of movement can connect dance videos to their accompanying music. The intermediate representation could also be derived from pre-trained deep learning models and act as semantic bridges to facilitate the transition from more general video and background music where the audio-visual relationship is largely subjective.In the second category, my research delves into learning implicit representation through the vision-to-audio generative process. This angle seeks not only to achieve vision-to-audio conversion but also to construct meaningful representations therein. Innovations here include unsupervised models that deduce instrumental sounds from musicians' body movements and diffusion models capable of synthesizing impact sounds from visual cues of object-physical interaction. This exploration reveals the association between visual inputs and various timbre characteristics. Additionally, by mapping indoor scene geometry to room impulse responses at discrete locations, we can infer a continuous acoustic field that enables the rendering of high-fidelity audio for any emitter-listener locations and enhances the realism and immersion of auditory experiences.Finally, in the third part of work, I proposed a unified audio-visual framework that seamlessly merges representation learning and generative modeling. This general approach enables the efficient generation of high-fidelity audio from visual stimuli while constructing robust semantic audio-visual representations. Its applications are broad, ranging from audio-visual retrieval and event classification to audio-only classification, paving the way for more immersive and contextually rich audio-visual experiences.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798382215228Subjects--Topical Terms:

596380
Electrical engineering.
Subjects--Index Terms:

Audio-visual learningIndex Terms--Genre/Form:

554714
Electronic books.

Towards Integrated Audio-Visual Learning : = from Vision-to-Audio Generation to a Unified Audio-Visual Framework.
LDR:04871ntm a22003737 4500 001 1149695
005 20241022112616.5
006 m o d
007 cr bn ---uuuuu
008 250605s2024 xx obm 000 0 eng d
020 $a 9798382215228
035 $a (MiAaPQ)AAI30997482
035 $a AAI30997482
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Su, Kun. $3 1476021
245 1 0 $a Towards Integrated Audio-Visual Learning : $b from Vision-to-Audio Generation to a Unified Audio-Visual Framework.
264 0 $c 2024
300 $a 1 online resource (158 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-10, Section: B.
500 $a Advisor: Shlizerman, Eli.
502 $a Thesis (Ph.D.)--University of Washington, 2024.
504 $a Includes bibliographical references
520 $a The interplay between audio and visual signals, rich in correlations across various scales, significantly impacts human perception and drives a consistent demand for audio-visual applications across fields such as video production, animation, and virtual reality. Historically, the creation and adaptation of audio content has been predominantly a manual process reliant on the expertise of Foley artists. Exploring automated systems capable of assisting with such tasks and achieving comparable proficiency suggests an intriguing prospect. In recent years, deep learning-based methodologies have shown considerable promise in handling image, video, and text data. Nonetheless, the task of integrating audio with visual inputs introduces distinct challenges that stem from the fundamental differences and complexities inherent to each modality. In particular, techniques for generating non-speech audio, such as music, object material impact sounds, natural sounds, and spatial sound effects, have received limited exploration.Audio and visual signals can be represented in various ways, with their connections varying across different scenarios. The research domain of learning to connect and relate audio and visual signals, termed audio-visual learning, has traditionally focused on either audio-visual representation learning or on generative modeling of one modality conditioned on the other. My research traverses audio-visual learning and connects audio-visual representation and generation from three distinct perspectives.In the first category, my investigation focuses on vision-to-audio generation through an intermediate representation, which serves as a bridge to link visual and audio domains. For instance, musical notes can translate piano keystrokes into their corresponding sounds, and the rhythm of movement can connect dance videos to their accompanying music. The intermediate representation could also be derived from pre-trained deep learning models and act as semantic bridges to facilitate the transition from more general video and background music where the audio-visual relationship is largely subjective.In the second category, my research delves into learning implicit representation through the vision-to-audio generative process. This angle seeks not only to achieve vision-to-audio conversion but also to construct meaningful representations therein. Innovations here include unsupervised models that deduce instrumental sounds from musicians' body movements and diffusion models capable of synthesizing impact sounds from visual cues of object-physical interaction. This exploration reveals the association between visual inputs and various timbre characteristics. Additionally, by mapping indoor scene geometry to room impulse responses at discrete locations, we can infer a continuous acoustic field that enables the rendering of high-fidelity audio for any emitter-listener locations and enhances the realism and immersion of auditory experiences.Finally, in the third part of work, I proposed a unified audio-visual framework that seamlessly merges representation learning and generative modeling. This general approach enables the efficient generation of high-fidelity audio from visual stimuli while constructing robust semantic audio-visual representations. Its applications are broad, ranging from audio-visual retrieval and event classification to audio-only classification, paving the way for more immersive and contextually rich audio-visual experiences.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Electrical engineering. $3 596380
650 4 $a Computer science. $3 573171
653 $a Audio-visual learning
653 $a Audio-visual representation
653 $a Video-to-audio generation
655 7 $a Electronic books. $2 local $3 554714
690 $a 0544
690 $a 0800
690 $a 0984
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a University of Washington. $b Electrical and Computer Engineering. $3 1465511
773 0 $t Dissertations Abstracts International $g 85-10B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30997482 $z click for full text (PQDT)