國立虎尾科技大學 |

Towards Understanding the Challenges in Scaling Frontier Machine Learning Models.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Towards Understanding the Challenges in Scaling Frontier Machine Learning Models./
Author:	Sreenivasan, Kartik.
Description:	1 online resource (353 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-06, Section: A.
Contained By:	Dissertations Abstracts International85-06A.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798381185164

Towards Understanding the Challenges in Scaling Frontier Machine Learning Models.
Sreenivasan, Kartik.

Towards Understanding the Challenges in Scaling Frontier Machine Learning Models. - 1 online resource (353 pages)

Source: Dissertations Abstracts International, Volume: 85-06, Section: A.

Thesis (Ph.D.)--The University of Wisconsin - Madison, 2023.

Includes bibliographical references

Machine Learning (ML) research is going through arguably one of the most volatile periods in its history. While this is incredibly exciting has led to amazing discoveries, it has also exposed challenges and vulnerabilities that simply did not exist, or were not relevant, until now. This dissertation focuses on the challenges that have arisen with the effort to consistently scale machine learning models in the last few years. In some cases, we will expose unavoidable vulnerabilies associated with certain families of algorithms (e.g., Federated Learning) whereas in other cases, we will improve the state of the art and propose new algorithms (e.g., Pruning). In the last few chapters concerning Contrastive Learning (CL) and Large Language Models (LLMs), we believe that one of the most important challenges remaining is understanding how, why and when they work. This dissertation will make an effort to improve that understanding through theory as well as well-designed experiments.The first part of this dissertation focuses on Federated Learning (FL). ). McMahan et al. (2017a) showed that FL was able to resolve distributed ML's growing concerns regarding privacy. While it rapidly rose in popularity (Kairouz et al., 2019), it exposed a potential vulnerability to adversarial attacks (Bagdasaryan et al., 2020). We prove rigorously, through algorithmic reductions, that this vulnerability is inherent. We also develop the edge-case backdoor attack which bypasses all known defenses at the time and is provably computationally hard to detect. We show that these backdoors are easy to insert, persist for long periods, and are extremely hard to remove without damaging the model's accuracy.The second chapter focuses on addressing the growing number of parameters in state-of-the-art (SOTA) models. Frankle and Carbin (2018) re-ignited interest in sparsity through the Lottery Ticket Hypothesis (LTH). The LTH promised the existence of sparse networks that could be discovered at (or near) initialization, that could be trained to dense accuracy at a fraction of the cost. However, several works found that algorithms that found subnetworks at initialization could hardly surpass random pruning, and Iterative Magnitude Pruning (IMP) required the network to be retrained to full accuracy several times, making its cost impractical. We show that pruning at initialization is indeed possible, and it can be done computationally efficiently. Using intuitions from the so-called Strong LTH, we develop Gem-Miner - the first algorithm to prune at initialization while passing the sanity checks proposed by Frankle et al. (2020b) and being up to 19x faster than IMP. We also show that we can go one step further than merely pruning and show that pruning binary {+1/-1} networks is enough. Moreover, we prove that these binary networks need to be just a log factor larger than any target network we want to approximate.The final part of this dissertation focuses on better understanding models that are showing seemingly miraculous performance. While Contrastive Learning has enjoyed remarkable success, we show that mini-batch optimization in this setting has several challenges that cannot be easily addressed by the well-known SGD literature. We prove that despite being different from the full-batch problem even in expectation, they share the same global solution under certain conditions. Finally, we address the current family of frontier models - Transformers. With carefully designed experimental settings, we separate out the complexities of language from these models and explain properties like chain-of-thought and the emergence of arithmetic from the perspectives of sample complexity and function approximation.To the reader, while at first, this dissertation may seem like random meanderings through the fields of machine learning, I urge you to keep the following in mind. While the architectures, model scales and problems might have changed rapidly in the short period of time that this dissertation spans, there are certain recurring themes. While there are always challenges associated with scaling up solutions at the frontier of ML research, sometimes we need to look carefully to find them. However, there also exist simple solutions to these challenges, as long as we stare at the problems for long enough. I hope this dissertation reveals several such challenges, and simple solutions to some of them.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798381185164Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

Machine LearningIndex Terms--Genre/Form:

554714
Electronic books.

Towards Understanding the Challenges in Scaling Frontier Machine Learning Models.
LDR:05808ntm a22003857 4500 001 1147347
005 20240909100803.5
006 m o d
007 cr bn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798381185164
035 $a (MiAaPQ)AAI30820325
035 $a AAI30820325
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Sreenivasan, Kartik. $3 1473053
245 1 0 $a Towards Understanding the Challenges in Scaling Frontier Machine Learning Models.
264 0 $c 2023
300 $a 1 online resource (353 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-06, Section: A.
500 $a Advisor: Papailiopoulos, Dimitris.
502 $a Thesis (Ph.D.)--The University of Wisconsin - Madison, 2023.
504 $a Includes bibliographical references
520 $a Machine Learning (ML) research is going through arguably one of the most volatile periods in its history. While this is incredibly exciting has led to amazing discoveries, it has also exposed challenges and vulnerabilities that simply did not exist, or were not relevant, until now. This dissertation focuses on the challenges that have arisen with the effort to consistently scale machine learning models in the last few years. In some cases, we will expose unavoidable vulnerabilies associated with certain families of algorithms (e.g., Federated Learning) whereas in other cases, we will improve the state of the art and propose new algorithms (e.g., Pruning). In the last few chapters concerning Contrastive Learning (CL) and Large Language Models (LLMs), we believe that one of the most important challenges remaining is understanding how, why and when they work. This dissertation will make an effort to improve that understanding through theory as well as well-designed experiments.The first part of this dissertation focuses on Federated Learning (FL). ). McMahan et al. (2017a) showed that FL was able to resolve distributed ML's growing concerns regarding privacy. While it rapidly rose in popularity (Kairouz et al., 2019), it exposed a potential vulnerability to adversarial attacks (Bagdasaryan et al., 2020). We prove rigorously, through algorithmic reductions, that this vulnerability is inherent. We also develop the edge-case backdoor attack which bypasses all known defenses at the time and is provably computationally hard to detect. We show that these backdoors are easy to insert, persist for long periods, and are extremely hard to remove without damaging the model's accuracy.The second chapter focuses on addressing the growing number of parameters in state-of-the-art (SOTA) models. Frankle and Carbin (2018) re-ignited interest in sparsity through the Lottery Ticket Hypothesis (LTH). The LTH promised the existence of sparse networks that could be discovered at (or near) initialization, that could be trained to dense accuracy at a fraction of the cost. However, several works found that algorithms that found subnetworks at initialization could hardly surpass random pruning, and Iterative Magnitude Pruning (IMP) required the network to be retrained to full accuracy several times, making its cost impractical. We show that pruning at initialization is indeed possible, and it can be done computationally efficiently. Using intuitions from the so-called Strong LTH, we develop Gem-Miner - the first algorithm to prune at initialization while passing the sanity checks proposed by Frankle et al. (2020b) and being up to 19x faster than IMP. We also show that we can go one step further than merely pruning and show that pruning binary {+1/-1} networks is enough. Moreover, we prove that these binary networks need to be just a log factor larger than any target network we want to approximate.The final part of this dissertation focuses on better understanding models that are showing seemingly miraculous performance. While Contrastive Learning has enjoyed remarkable success, we show that mini-batch optimization in this setting has several challenges that cannot be easily addressed by the well-known SGD literature. We prove that despite being different from the full-batch problem even in expectation, they share the same global solution under certain conditions. Finally, we address the current family of frontier models - Transformers. With carefully designed experimental settings, we separate out the complexities of language from these models and explain properties like chain-of-thought and the emergence of arithmetic from the perspectives of sample complexity and function approximation.To the reader, while at first, this dissertation may seem like random meanderings through the fields of machine learning, I urge you to keep the following in mind. While the architectures, model scales and problems might have changed rapidly in the short period of time that this dissertation spans, there are certain recurring themes. While there are always challenges associated with scaling up solutions at the frontier of ML research, sometimes we need to look carefully to find them. However, there also exist simple solutions to these challenges, as long as we stare at the problems for long enough. I hope this dissertation reveals several such challenges, and simple solutions to some of them.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Information science. $3 561178
653 $a Machine Learning
653 $a Optimization
653 $a Adversarial attacks
653 $a Lottery Ticket Hypothesis
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0800
690 $a 0723
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a The University of Wisconsin - Madison. $b Computer Sciences. $3 1179878
773 0 $t Dissertations Abstracts International $g 85-06A.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30820325 $z click for full text (PQDT)