國立虎尾科技大學 |

Essay on Foundation Models and Reinforcement Learning.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Essay on Foundation Models and Reinforcement Learning./
Author:	Zhang, Yufeng.
Description:	1 online resource (344 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-02, Section: B.
Contained By:	Dissertations Abstracts International85-02B.
Subject:	Statistics. -
Online resource:	click for full text (PQDT)
ISBN:	9798380145268

Essay on Foundation Models and Reinforcement Learning.
Zhang, Yufeng.

Essay on Foundation Models and Reinforcement Learning. - 1 online resource (344 pages)

Source: Dissertations Abstracts International, Volume: 85-02, Section: B.

Thesis (Ph.D.)--Northwestern University, 2023.

Includes bibliographical references

In this dissertation, we aim to develop a theoretical understanding of foundation models and reinforcement learning. We delve into a comprehensive analysis of specific aspects within these domains. The focal points of our study are as follows:• Generative Adversarial Imitation Learning (GAIL) with Neural Networks: GAIL is poised to execute tasks based on expert demonstrations. By parameterizing both the reward function and policy using neural networks, we develop a gradient-based algorithm with alternating updates for GAIL. Through rigorous analysis, we establish that this algorithm converges to the global optimum at a sublinear rate.• Temporal-Difference (TD) Learning and Q-learning with Neural Networks: We dissect the fundamental reason behind the empirical success of deep TD learning and deep Q-learning: the learned feature representation. Utilizing mean-field analysis, we scrutinize the evolution of this representation. We demonstrate that, when implemented through an overparameterized two-layer neural network, both TD learning and Q-learning algorithms are capable of globally minimizing the mean-squared projected Bellman error at a sublinear rate.• Attention Mechanisms and Transformers: Analyzing attention mechanisms and transformers through the lens of exchangeability, we first establish the existence of a representation for input tokens that is sufficient and minimal. We then ascertain that the attention mechanism with the appropriate parameters is able to infer the latent posterior within a margin of approximation error that diminishes as input sizes increase. Additionally, we prove that employing either supervised or self-supervised objectives enables empirical risk minimization to learn the optimal parameters within a generalization error that remains independent of input sizes.• In-Context Learning (ICL): We execute an exhaustive investigation into ICL by addressing several pertinent questions. Firstly, from a Bayesian view, we show that the language models learn an ICL estimator by implementing Bayesian model averaging. Subsequently, we evaluate the performance of the ICL algorithm from an online learning standpoint and establish a regret bound decreasing with the length of the ICL input sequence. Then, we demonstrate that during pretraining, the total variation distance between the learned model and the underlying true model is constrained by a generalization error decreasing with the number of token sequences and the length of each sequence during pretraining, respectively. Finally, by combining this two results, we show that the learned model is capable in ICL.This dissertation aspires to enrich the academic discourse on foundation models and reinforcement learning by offering novel insights and rigorous proofs that may serve as building blocks for future research in these rapidly evolving fields.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798380145268Subjects--Topical Terms:

556824
Statistics.
Subjects--Index Terms:

Imitation leanringIndex Terms--Genre/Form:

554714
Electronic books.

Essay on Foundation Models and Reinforcement Learning.
LDR:04202ntm a22003857 4500 001 1144294
005 20240531083821.5
006 m o d
007 cr mn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798380145268
035 $a (MiAaPQ)AAI30632002
035 $a AAI30632002
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Zhang, Yufeng. $e editor. $3 1287732
245 1 0 $a Essay on Foundation Models and Reinforcement Learning.
264 0 $c 2023
300 $a 1 online resource (344 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-02, Section: B.
500 $a Advisor: Wang, Zhaoran.
502 $a Thesis (Ph.D.)--Northwestern University, 2023.
504 $a Includes bibliographical references
520 $a In this dissertation, we aim to develop a theoretical understanding of foundation models and reinforcement learning. We delve into a comprehensive analysis of specific aspects within these domains. The focal points of our study are as follows:• Generative Adversarial Imitation Learning (GAIL) with Neural Networks: GAIL is poised to execute tasks based on expert demonstrations. By parameterizing both the reward function and policy using neural networks, we develop a gradient-based algorithm with alternating updates for GAIL. Through rigorous analysis, we establish that this algorithm converges to the global optimum at a sublinear rate.• Temporal-Difference (TD) Learning and Q-learning with Neural Networks: We dissect the fundamental reason behind the empirical success of deep TD learning and deep Q-learning: the learned feature representation. Utilizing mean-field analysis, we scrutinize the evolution of this representation. We demonstrate that, when implemented through an overparameterized two-layer neural network, both TD learning and Q-learning algorithms are capable of globally minimizing the mean-squared projected Bellman error at a sublinear rate.• Attention Mechanisms and Transformers: Analyzing attention mechanisms and transformers through the lens of exchangeability, we first establish the existence of a representation for input tokens that is sufficient and minimal. We then ascertain that the attention mechanism with the appropriate parameters is able to infer the latent posterior within a margin of approximation error that diminishes as input sizes increase. Additionally, we prove that employing either supervised or self-supervised objectives enables empirical risk minimization to learn the optimal parameters within a generalization error that remains independent of input sizes.• In-Context Learning (ICL): We execute an exhaustive investigation into ICL by addressing several pertinent questions. Firstly, from a Bayesian view, we show that the language models learn an ICL estimator by implementing Bayesian model averaging. Subsequently, we evaluate the performance of the ICL algorithm from an online learning standpoint and establish a regret bound decreasing with the length of the ICL input sequence. Then, we demonstrate that during pretraining, the total variation distance between the learned model and the underlying true model is constrained by a generalization error decreasing with the number of token sequences and the length of each sequence during pretraining, respectively. Finally, by combining this two results, we show that the learned model is capable in ICL.This dissertation aspires to enrich the academic discourse on foundation models and reinforcement learning by offering novel insights and rigorous proofs that may serve as building blocks for future research in these rapidly evolving fields.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Statistics. $3 556824
653 $a Imitation leanring
653 $a Reinforcement leanring
653 $a Transformer
653 $a Bayesian model
653 $a Neural networks
655 7 $a Electronic books. $2 local $3 554714
690 $a 0463
690 $a 0796
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a Northwestern University. $b Industrial Engineering and Management Sciences. $3 1180832
773 0 $t Dissertations Abstracts International $g 85-02B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30632002 $z click for full text (PQDT)