國立虎尾科技大學 |

Principles and Applications of Discrete Deep Generative Models.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Principles and Applications of Discrete Deep Generative Models./
Author:	Fan, Ting-Han.
Description:	1 online resource (138 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 84-12, Section: B.
Contained By:	Dissertations Abstracts International84-12B.
Subject:	Electrical engineering. -
Online resource:	click for full text (PQDT)
ISBN:	9798379717513

Principles and Applications of Discrete Deep Generative Models.
Fan, Ting-Han.

Principles and Applications of Discrete Deep Generative Models. - 1 online resource (138 pages)

Source: Dissertations Abstracts International, Volume: 84-12, Section: B.

Thesis (Ph.D.)--Princeton University, 2023.

Includes bibliographical references

This thesis studies the principles and applications of discrete DGMs. Deep generative models (DGM) are deep neural networks capable of high-dimensional probability distribution modeling and random sample generation. Among the various applications of DGM, some involve inherently discrete components, which drives the need to model discrete random variables; for example, text modeling and control with discrete variables. The discreteness raises fundamental questions about the design of a discrete DGM. How to train a discrete DGM? What are the applications? How to perform large-scale discrete modeling and predictions?We study the training of discrete DGM from the perspective of reparameterization. Reparameterization is a gradient estimation method for a random variable modeled by DGM. It is challenging due to the high variance of the gradient estimation. Inspired by the essential properties of Straight-Through Gumbel-Softmax estimators, we propose a new reparameterization method called the Gapped Straight-Through estimator to reduce variance without incurring resampling overhead.We also present an application of discrete reparameterization in Reinforcement Learning (RL) for power system control where the control variables are integers. We contribute to this application in two aspects: an RL environment for power systems and an RL algorithm with an integer reparameterization scheme. The environment construction identifies the practical choices of the system. An open-source package for this environment has been released and used in the power research community. The RL algorithm for power systems includes a DDPG-style policy gradient and a reparameterization for integer actions.Lastly, we explore large-scale generative text modeling from a kernelized perspective of Transformers. We observe that relative positional embedding (RPE) has been essential for Transformers to perform well on long sequences. However, a theoretical framework for RPE is still lacking. Thus, we formulate a kernelized version of RPE through Conditionally Positive Definite (CPD) kernels. The diversity of CPD kernels allows us to derive various RPE that enable length extrapolation (train short, test long). Experiments demonstrate that the logarithmic variant achieves excellent extrapolation on three large language modeling datasets.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798379717513Subjects--Topical Terms:

596380
Electrical engineering.
Subjects--Index Terms:

Deep generative modelsIndex Terms--Genre/Form:

554714
Electronic books.

Principles and Applications of Discrete Deep Generative Models.
LDR:03653ntm a22003737 4500 001 1143592
005 20240517104607.5
006 m o d
007 cr mn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798379717513
035 $a (MiAaPQ)AAI30490390
035 $a AAI30490390
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Fan, Ting-Han. $3 1468325
245 1 0 $a Principles and Applications of Discrete Deep Generative Models.
264 0 $c 2023
300 $a 1 online resource (138 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 84-12, Section: B.
500 $a Advisor: Ramadge, Peter J.
502 $a Thesis (Ph.D.)--Princeton University, 2023.
504 $a Includes bibliographical references
520 $a This thesis studies the principles and applications of discrete DGMs. Deep generative models (DGM) are deep neural networks capable of high-dimensional probability distribution modeling and random sample generation. Among the various applications of DGM, some involve inherently discrete components, which drives the need to model discrete random variables; for example, text modeling and control with discrete variables. The discreteness raises fundamental questions about the design of a discrete DGM. How to train a discrete DGM? What are the applications? How to perform large-scale discrete modeling and predictions?We study the training of discrete DGM from the perspective of reparameterization. Reparameterization is a gradient estimation method for a random variable modeled by DGM. It is challenging due to the high variance of the gradient estimation. Inspired by the essential properties of Straight-Through Gumbel-Softmax estimators, we propose a new reparameterization method called the Gapped Straight-Through estimator to reduce variance without incurring resampling overhead.We also present an application of discrete reparameterization in Reinforcement Learning (RL) for power system control where the control variables are integers. We contribute to this application in two aspects: an RL environment for power systems and an RL algorithm with an integer reparameterization scheme. The environment construction identifies the practical choices of the system. An open-source package for this environment has been released and used in the power research community. The RL algorithm for power systems includes a DDPG-style policy gradient and a reparameterization for integer actions.Lastly, we explore large-scale generative text modeling from a kernelized perspective of Transformers. We observe that relative positional embedding (RPE) has been essential for Transformers to perform well on long sequences. However, a theoretical framework for RPE is still lacking. Thus, we formulate a kernelized version of RPE through Conditionally Positive Definite (CPD) kernels. The diversity of CPD kernels allows us to derive various RPE that enable length extrapolation (train short, test long). Experiments demonstrate that the logarithmic variant achieves excellent extrapolation on three large language modeling datasets.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Electrical engineering. $3 596380
650 4 $a Computer engineering. $3 569006
653 $a Deep generative models
653 $a Natural language processing
653 $a Reinforcement learning
655 7 $a Electronic books. $2 local $3 554714
690 $a 0544
690 $a 0464
690 $a 0800
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a Princeton University. $b Electrical and Computer Engineering. $3 1413580
773 0 $t Dissertations Abstracts International $g 84-12B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30490390 $z click for full text (PQDT)