國立虎尾科技大學 |

Deep Learning on a Diet : = An Error Landscape Perspective on Parameter and Data Efficiency in Deep Learning.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Deep Learning on a Diet :/
Reminder of title:	An Error Landscape Perspective on Parameter and Data Efficiency in Deep Learning.
Author:	Paul, Mansheej.
Description:	1 online resource (107 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-06, Section: B.
Contained By:	Dissertations Abstracts International85-06B.
Subject:	Deep learning. -
Online resource:	click for full text (PQDT)
ISBN:	9798381019674

Deep Learning on a Diet : = An Error Landscape Perspective on Parameter and Data Efficiency in Deep Learning.
Paul, Mansheej.

Deep Learning on a Diet :An Error Landscape Perspective on Parameter and Data Efficiency in Deep Learning. - 1 online resource (107 pages)

Source: Dissertations Abstracts International, Volume: 85-06, Section: B.

Thesis (Ph.D.)--Stanford University, 2023.

Includes bibliographical references

Example, for many applications on edge devices, networks need to have a small memory footprint to fit on-device. Current models on the other hand need entire servers to run on. The latency of current LLMs is also too large for many applications that require fast generation. Additionally, the large capital investment required to train current AI systems is a threat to the democratization of AI as only a few incumbents make all the decisions that go into developing these systems. It is thus imperative to develop methods that can be used to train models of similar capabilities under compute and data resource constraints.This leads us to the central question that motivates this thesis: when and how can we train small networks on small datasets that nevertheless perform as well as large, state of the art models. For network parameter eciency, we study sparse networks that share the same architectural structure as larger networks but have only a fraction of the parameters. For data eciency, we study how to prune datasets by identifying the subset of data most important for achieving good performance. Throughout this work, our focus will be on large scale empirical investigations that develop a scientific understanding of the principles that enable data and parameter eciency in deep learning. The goal is to not only build better methods for pruning network parameters and training datasets but to also build a foundational understanding about why pruning is possible in the first place.This intuition is critical for identifying the most important directions for developing future methods.To investigate these questions, we empirically study the training dynamics of deep networks from the perspective of the geometric structure of the error landscape. This approach has been used in recent works to scientifically understand many aspects of deep learning such as generalization, calibration, and the di↵erent phases of training [18, 20-22, 25, 45, 53]. We will use the tools developed in these works to study when and why pruning both network parameters and training examples is possible. In particular, in Chapter 2, we will show that the curvature of the error landscape around well performing dense networks governs our ability to find equally well performing sparse networks. In Chapter 3, we will develop a method that ranks examples by diculty and discover how example diculty impacts the large scale structure of the optimization landscape by shaping its error basins. Finally, in Chapter 4, we will use these insights to bias the training data distribution in the early phase of training to create an optimization landscape that is more amenable to eciently finding sparse deep networks (here we return to parameter sparsity). Through these investigations, we will uncover novel connections between training dynamics, the geometric properties of the error landscape, and eciency in deep learning.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798381019674Subjects--Topical Terms:

1127428
Deep learning.
Index Terms--Genre/Form:

554714
Electronic books.

Deep Learning on a Diet : = An Error Landscape Perspective on Parameter and Data Efficiency in Deep Learning.
LDR:04162ntm a22003257 4500 001 1148744
005 20240930100134.5
006 m o d
007 cr bn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798381019674
035 $a (MiAaPQ)AAI30726832
035 $a (MiAaPQ)STANFORDwh462kf8223
035 $a AAI30726832
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Paul, Mansheej. $3 1474787
245 1 0 $a Deep Learning on a Diet : $b An Error Landscape Perspective on Parameter and Data Efficiency in Deep Learning.
264 0 $c 2023
300 $a 1 online resource (107 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-06, Section: B.
500 $a Advisor: Ganguli, Surya.
502 $a Thesis (Ph.D.)--Stanford University, 2023.
504 $a Includes bibliographical references
520 $a Example, for many applications on edge devices, networks need to have a small memory footprint to fit on-device. Current models on the other hand need entire servers to run on. The latency of current LLMs is also too large for many applications that require fast generation. Additionally, the large capital investment required to train current AI systems is a threat to the democratization of AI as only a few incumbents make all the decisions that go into developing these systems. It is thus imperative to develop methods that can be used to train models of similar capabilities under compute and data resource constraints.This leads us to the central question that motivates this thesis: when and how can we train small networks on small datasets that nevertheless perform as well as large, state of the art models. For network parameter eciency, we study sparse networks that share the same architectural structure as larger networks but have only a fraction of the parameters. For data eciency, we study how to prune datasets by identifying the subset of data most important for achieving good performance. Throughout this work, our focus will be on large scale empirical investigations that develop a scientific understanding of the principles that enable data and parameter eciency in deep learning. The goal is to not only build better methods for pruning network parameters and training datasets but to also build a foundational understanding about why pruning is possible in the first place.This intuition is critical for identifying the most important directions for developing future methods.To investigate these questions, we empirically study the training dynamics of deep networks from the perspective of the geometric structure of the error landscape. This approach has been used in recent works to scientifically understand many aspects of deep learning such as generalization, calibration, and the di↵erent phases of training [18, 20-22, 25, 45, 53]. We will use the tools developed in these works to study when and why pruning both network parameters and training examples is possible. In particular, in Chapter 2, we will show that the curvature of the error landscape around well performing dense networks governs our ability to find equally well performing sparse networks. In Chapter 3, we will develop a method that ranks examples by diculty and discover how example diculty impacts the large scale structure of the optimization landscape by shaping its error basins. Finally, in Chapter 4, we will use these insights to bias the training data distribution in the early phase of training to create an optimization landscape that is more amenable to eciently finding sparse deep networks (here we return to parameter sparsity). Through these investigations, we will uncover novel connections between training dynamics, the geometric properties of the error landscape, and eciency in deep learning.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Deep learning. $3 1127428
650 4 $a Connectivity. $3 1468736
650 4 $a Sparsity. $3 1372730
655 7 $a Electronic books. $2 local $3 554714
690 $a 0800
710 2 $a Stanford University. $3 1184533
710 2 $a ProQuest Information and Learning Co. $3 1178819
773 0 $t Dissertations Abstracts International $g 85-06B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30726832 $z click for full text (PQDT)