國立虎尾科技大學 |

Learning to Describe Images via Natural Language.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Learning to Describe Images via Natural Language./
Author:	Yan, An.
Description:	1 online resource (81 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-10, Section: B.
Contained By:	Dissertations Abstracts International85-10B.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798382223483

Learning to Describe Images via Natural Language.
Yan, An.

Learning to Describe Images via Natural Language. - 1 online resource (81 pages)

Source: Dissertations Abstracts International, Volume: 85-10, Section: B.

Thesis (Ph.D.)--University of California, San Diego, 2024.

Includes bibliographical references

Teaching machines to describe visual images is one of the most long-standing challenges in the field of Machine Learning. This thesis tackles the problem of describing images via natural language: how to build machine learning models to read visual images and describe their content as well as answer relevant questions. From the application perspective, strong image captioning systems can contribute to applications such as visual question answering, dialogue systems and visual-based robotics. For a long term goal, if we can build such systems (e.g., GPT-4V and beyond), they would be a crucial step towards building Artificial General Intelligence: computers that can perceive and explore the world as humans do.This thesis focus on neural models: building vision understanding and language generation models with deep neural networks. It mainly consists of three parts.First, we will introduce concept bottleneck models, a class of models that build concept layers for visual understanding. We will present our work on learning a concise concept space, and follow-up applications for medical imaging to gain robustness.In the second part of this thesis, we investigate how we can build practical image captioning systems based on different neural text generation architectures, from LSTM to transformers and pre-trained language models. In particular, we will cover four different tasks: 1) how we can describe the visual difference of two images; 2) how we can write medical reports given Chest X-rays to assist doctors; 3) how to generate personalized explanations for recommender systems; 4) how to augment text generation with visual imagination generated from vision diffusion models.In the third part, we will discuss recent advances, future directions and open questions in this field, focusing on aspects of datasets, models, and applications. We will also introduce some of our on-going attempts for these directions: for example, how to navigate phone screens and complete mobile tasks with GPT-4V.In summary, my research contributes to the field of vision and language, specifically visual understanding via natural language, from the aspects of data curation, algorithm designing, model training, as well as various downstream applications.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798382223483Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

Machine learningIndex Terms--Genre/Form:

554714
Electronic books.

Learning to Describe Images via Natural Language.
LDR:03653ntm a22004097 4500 001 1144977
005 20240617111732.5
006 m o d
007 cr mn ---uuuuu
008 250605s2024 xx obm 000 0 eng d
020 $a 9798382223483
035 $a (MiAaPQ)AAI30994065
035 $a AAI30994065
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Yan, An. $3 1470172
245 1 0 $a Learning to Describe Images via Natural Language.
264 0 $c 2024
300 $a 1 online resource (81 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-10, Section: B.
500 $a Advisor: McAuley, Julian.
502 $a Thesis (Ph.D.)--University of California, San Diego, 2024.
504 $a Includes bibliographical references
520 $a Teaching machines to describe visual images is one of the most long-standing challenges in the field of Machine Learning. This thesis tackles the problem of describing images via natural language: how to build machine learning models to read visual images and describe their content as well as answer relevant questions. From the application perspective, strong image captioning systems can contribute to applications such as visual question answering, dialogue systems and visual-based robotics. For a long term goal, if we can build such systems (e.g., GPT-4V and beyond), they would be a crucial step towards building Artificial General Intelligence: computers that can perceive and explore the world as humans do.This thesis focus on neural models: building vision understanding and language generation models with deep neural networks. It mainly consists of three parts.First, we will introduce concept bottleneck models, a class of models that build concept layers for visual understanding. We will present our work on learning a concise concept space, and follow-up applications for medical imaging to gain robustness.In the second part of this thesis, we investigate how we can build practical image captioning systems based on different neural text generation architectures, from LSTM to transformers and pre-trained language models. In particular, we will cover four different tasks: 1) how we can describe the visual difference of two images; 2) how we can write medical reports given Chest X-rays to assist doctors; 3) how to generate personalized explanations for recommender systems; 4) how to augment text generation with visual imagination generated from vision diffusion models.In the third part, we will discuss recent advances, future directions and open questions in this field, focusing on aspects of datasets, models, and applications. We will also introduce some of our on-going attempts for these directions: for example, how to navigate phone screens and complete mobile tasks with GPT-4V.In summary, my research contributes to the field of vision and language, specifically visual understanding via natural language, from the aspects of data curation, algorithm designing, model training, as well as various downstream applications.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Systematic biology. $3 1179695
650 4 $a Medical imaging. $3 1180167
653 $a Machine learning
653 $a Visual images
653 $a Concept bottleneck models
653 $a Concise concept space
653 $a Diagnostic datasets
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0423
690 $a 0574
690 $a 0800
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a University of California, San Diego. $b Computer Science and Engineering. $3 1189479
773 0 $t Dissertations Abstracts International $g 85-10B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30994065 $z click for full text (PQDT)