國立虎尾科技大學 |

Dynamic Sparsity for Efficient Machine Learning.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Dynamic Sparsity for Efficient Machine Learning./
Author:	Liu, Zichang.
Description:	1 online resource (145 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 86-02, Section: B.
Contained By:	Dissertations Abstracts International86-02B.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798383417386

Dynamic Sparsity for Efficient Machine Learning.
Liu, Zichang.

Dynamic Sparsity for Efficient Machine Learning. - 1 online resource (145 pages)

Source: Dissertations Abstracts International, Volume: 86-02, Section: B.

Thesis (Ph.D.)--Rice University, 2024.

Includes bibliographical references

Over the past decades, machine learning (ML) models have delivered remarkable accomplishments in various applications. For example, large language models usher in a new wave of excitement in artificial intelligence. Interestingly, these accomplishments also unveil the scaling law in machine learning: larger models, equipped with more parameters and trained on more extensive datasets, often significantly outperform their smaller counterparts. However, the trends of increasing model size inevitably introduce unprecedented computation resource requirements, creating substantial challenges in model training and deployments.This thesis aims to improve the efficiency of ML models through algorithmic advancements. Specifically, we exploit the dynamic sparsity pattern inside ML models to achieve efficiency goals. Dynamic sparsity refers to the subset of parameters or activations that are important for a certain data, and different data may have a different dynamic sparsity pattern. We advocate identifying the dynamic sparsity pattern for each data set and focusing computation and memory resources on it.The first part of this thesis centers around the inference stage. We verify the existence of dynamic sparsity in trained ML models, namely, within the classification layer, attention mechanism, and transformer layers of trained models. Further, we demonstrate that such dynamic sparsity can be cheaply predicted and leveraged for each data to improve the inference efficiency goals. The subsequent part of the dissertation will shift its focus to the training stage, where dynamic sparsity emerges as a tool to mitigate the problem of catastrophic forgetting or data heterogeneity in federated learning to improve training efficiency.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798383417386Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

Machine learningIndex Terms--Genre/Form:

554714
Electronic books.

Dynamic Sparsity for Efficient Machine Learning.
LDR:03089ntm a22003977 4500 001 1150536
005 20241028051517.5
006 m o d
007 cr bn ---uuuuu
008 250605s2024 xx obm 000 0 eng d
020 $a 9798383417386
035 $a (MiAaPQ)AAI31532859
035 $a (MiAaPQ)0187rice5035Liu
035 $a AAI31532859
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Liu, Zichang. $3 1477029
245 1 0 $a Dynamic Sparsity for Efficient Machine Learning.
264 0 $c 2024
300 $a 1 online resource (145 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-02, Section: B.
500 $a Advisor: Shrivastava, Anshumali.
502 $a Thesis (Ph.D.)--Rice University, 2024.
504 $a Includes bibliographical references
520 $a Over the past decades, machine learning (ML) models have delivered remarkable accomplishments in various applications. For example, large language models usher in a new wave of excitement in artificial intelligence. Interestingly, these accomplishments also unveil the scaling law in machine learning: larger models, equipped with more parameters and trained on more extensive datasets, often significantly outperform their smaller counterparts. However, the trends of increasing model size inevitably introduce unprecedented computation resource requirements, creating substantial challenges in model training and deployments.This thesis aims to improve the efficiency of ML models through algorithmic advancements. Specifically, we exploit the dynamic sparsity pattern inside ML models to achieve efficiency goals. Dynamic sparsity refers to the subset of parameters or activations that are important for a certain data, and different data may have a different dynamic sparsity pattern. We advocate identifying the dynamic sparsity pattern for each data set and focusing computation and memory resources on it.The first part of this thesis centers around the inference stage. We verify the existence of dynamic sparsity in trained ML models, namely, within the classification layer, attention mechanism, and transformer layers of trained models. Further, we demonstrate that such dynamic sparsity can be cheaply predicted and leveraged for each data to improve the inference efficiency goals. The subsequent part of the dissertation will shift its focus to the training stage, where dynamic sparsity emerges as a tool to mitigate the problem of catastrophic forgetting or data heterogeneity in federated learning to improve training efficiency.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Systems science. $3 1148479
653 $a Machine learning
653 $a Large language model
653 $a Sparsity
653 $a ML models
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0800
690 $a 0790
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a Rice University. $b Computer Science. $3 1184788
773 0 $t Dissertations Abstracts International $g 86-02B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31532859 $z click for full text (PQDT)