國立虎尾科技大學 |

Temporal Learning for Video-Language Understanding and Generation.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Temporal Learning for Video-Language Understanding and Generation./
Author:	Zhang, Songyang.
Description:	1 online resource (249 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-03, Section: A.
Contained By:	Dissertations Abstracts International85-03A.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798380318648

Temporal Learning for Video-Language Understanding and Generation.
Zhang, Songyang.

Temporal Learning for Video-Language Understanding and Generation. - 1 online resource (249 pages)

Source: Dissertations Abstracts International, Volume: 85-03, Section: A.

Thesis (Ph.D.)--University of Rochester, 2023.

Includes bibliographical references

Vision-language studies jointly perceive, understand, and generate over the vision and language modalities to perform various tasks, such as retrieve an image/video by a given sentence or generate an image/video by a given sentence. Great success has been made in image-language studies, however, video-language studies still lag behind. Different from image studies that mainly focus on static objects or scenes, the core challenge in video studies is how to further learn dynamic changes. Time is an intrinsic attribute for both video and language. How to encode time has been studied in both CV and NLP communities, however, their alignments and interactions has been rarely studied until recently.In this thesis, we study temporal learning of video-language tasks from both video and language aspects. In order to interact between different modalities, it's natural to ask how to learn alignment between these two modalities? In the first part, we answer this question by studying a specific video-language alignment task, moment localization with natural language. This task aims to retrieve a specific moment from an untrimmed video by a query sentence. We study this problem from two aspects, video context modeling and temporal language modeling. If we have learnt a good video-language alignment, a following question would be could we leverage such knowledge and benefit some conventional NLP tasks? Or could we learn CV tasks with language as guidance?In the second part, we first investigate a conventional NLP problem, grammar induction. Grammar induction aims to find hierarchical syntactic structures from plain sentences. We found that leveraging the regularities between video and text can improve parser's performance. We further investigate the dataset limitation of this approach and propose a solution by leveraging instructional videos without any human efforts.In the third part, we learn video generation with language. We investigate this problem from dataset collection, spatial-temporal modeling and efficiency. We develop a dataset to enable focused advances in some of the core challenges of multimodal video research. We also leverage text-to-image models to learn the correspondence between text and the visual world, and uses unsupervised learning on unlabeled (unpaired) video data, to learn realistic motion. We also propose a novel temporal shift module to leverage a T2I model as-is for T2V generation without adding any new parameters.Upon these thesis works, we present several exciting research directions for future studies.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798380318648Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

Temporal learningIndex Terms--Genre/Form:

554714
Electronic books.

Temporal Learning for Video-Language Understanding and Generation.
LDR:03952ntm a22003977 4500 001 1147654
005 20240916065429.5
006 m o d
007 cr bn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798380318648
035 $a (MiAaPQ)AAI30572454
035 $a AAI30572454
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Zhang, Songyang. $3 1473421
245 1 0 $a Temporal Learning for Video-Language Understanding and Generation.
264 0 $c 2023
300 $a 1 online resource (249 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-03, Section: A.
500 $a Advisor: Luo, Jiebo.
502 $a Thesis (Ph.D.)--University of Rochester, 2023.
504 $a Includes bibliographical references
520 $a Vision-language studies jointly perceive, understand, and generate over the vision and language modalities to perform various tasks, such as retrieve an image/video by a given sentence or generate an image/video by a given sentence. Great success has been made in image-language studies, however, video-language studies still lag behind. Different from image studies that mainly focus on static objects or scenes, the core challenge in video studies is how to further learn dynamic changes. Time is an intrinsic attribute for both video and language. How to encode time has been studied in both CV and NLP communities, however, their alignments and interactions has been rarely studied until recently.In this thesis, we study temporal learning of video-language tasks from both video and language aspects. In order to interact between different modalities, it's natural to ask how to learn alignment between these two modalities? In the first part, we answer this question by studying a specific video-language alignment task, moment localization with natural language. This task aims to retrieve a specific moment from an untrimmed video by a query sentence. We study this problem from two aspects, video context modeling and temporal language modeling. If we have learnt a good video-language alignment, a following question would be could we leverage such knowledge and benefit some conventional NLP tasks? Or could we learn CV tasks with language as guidance?In the second part, we first investigate a conventional NLP problem, grammar induction. Grammar induction aims to find hierarchical syntactic structures from plain sentences. We found that leveraging the regularities between video and text can improve parser's performance. We further investigate the dataset limitation of this approach and propose a solution by leveraging instructional videos without any human efforts.In the third part, we learn video generation with language. We investigate this problem from dataset collection, spatial-temporal modeling and efficiency. We develop a dataset to enable focused advances in some of the core challenges of multimodal video research. We also leverage text-to-image models to learn the correspondence between text and the visual world, and uses unsupervised learning on unlabeled (unpaired) video data, to learn realistic motion. We also propose a novel temporal shift module to leverage a T2I model as-is for T2V generation without adding any new parameters.Upon these thesis works, we present several exciting research directions for future studies.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Information technology. $3 559429
650 4 $a Multimedia communications. $3 655342
653 $a Temporal learning
653 $a Vision-language studies
653 $a Video-language alignment
653 $a Video generation
653 $a Natural language processing
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0489
690 $a 0558
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a University of Rochester. $b Hajim School of Engineering and Applied Sciences. $3 1148592
773 0 $t Dissertations Abstracts International $g 85-03A.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30572454 $z click for full text (PQDT)