國立虎尾科技大學 |

Towards Efficient Deep Learning in Computer Vision via Network Sparsity and Distillation.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Towards Efficient Deep Learning in Computer Vision via Network Sparsity and Distillation./
Author:	Wang, Huan.
Description:	1 online resource (251 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-10, Section: B.
Contained By:	Dissertations Abstracts International85-10B.
Subject:	Computer engineering. -
Online resource:	click for full text (PQDT)
ISBN:	9798382318530

Towards Efficient Deep Learning in Computer Vision via Network Sparsity and Distillation.
Wang, Huan.

Towards Efficient Deep Learning in Computer Vision via Network Sparsity and Distillation. - 1 online resource (251 pages)

Source: Dissertations Abstracts International, Volume: 85-10, Section: B.

Thesis (Ph.D.)--Northeastern University, 2024.

Includes bibliographical references

Artificial intelligence (AI) empowered by deep learning, has been profoundly transforming the world. However, the excessive size of these models remains a central obstacle that limits their broader utility. Modern neural networks commonly consist of millions of parameters, with foundation models extending to billions. The rapid expansion in model size introduces many challenges including training cost, sluggish inference speed, excessive energy consumption, and negative environmental implications such as increased CO2 emissions.Addressing these challenges necessitates the adoption of efficient deep learning (EDL). The dissertation focuses on two overarching approaches, network sparsity (a.k.a. pruning) and knowledge distillation, to enhance the efficiency of deep learning models in the context of computer vision. Network pruning focuses on eliminating redundant parameters in a model while preserving the performance. Knowledge distillation aims to enhance the performance of the target model, referred to as the "student", by leveraging guidance from a stronger model, known as the "teacher". This approach leads to performance improvements in the target model without reducing its size.In this dissertation, I will start with the background and motivation for more efficient deep learning models in the past several years in the context of the arising foundation models. Then, the basic concepts, goals, and challenges of EDL will be introduced along with the major sub-methods. After that, the major part of this dissertation will be dedicated to elaborating on the proposed efficiency algorithms based on pruning and distillation in a variety of applications.For the pruning part, the dissertation first presents an effective pruning algorithm GReg in image classification, by tapping into a growing regularization strategy. Then, in order to understand the real progress of network pruning, a fairness principle is introduced to fairly compare different pruning methods. The investigation leads us to the central role of network trainability in pruning, which has been largely overlooked by prior works. A trainability-preserving pruning approach, TPP, is then proposed to show the merits of maintaining trainability during pruning. A short survey on an emerging new pruning paradigm, pruning at initialization, is then presented to discuss its potential and the connections with the conventional pruning after training. The GReg algorithm is further extended to a low-level vision task, single image super-resolution (SR), to explore the difference of utilizing pruning in low-level vision (SR) vs. high-level vision (image classification). Three efficient SR approaches (ASSL, GASSL,SRP) are introduced.For the distillation part, the dissertation first focuses on the interaction between knowledge distillation and data augmentation in image classification, a proved proposition presented to rigorously understand what defines the "goodness" of a data augmentation scheme in distillation. Next, the dissertation showcases how to employ distillation to significantly improve the inference efficiency for novel view synthesis in 3D vision. Both static scenes and dynamic scenes are considered. Finally, SnapFusion is presented to demonstrate a systematic efficiency optimization of deep models by jointly utilizing pruning and distillation, towards an unprecedentedly fast speed of text-to-image generation based on diffusion models.Finally, a comprehensive summary along with takeaways and outlooks of the future work will conclude the dissertation. Major takeaways include (1) there is no panacea towards efficient deep learning for all tasks; solution is usually case-by-case; (2) there is a clear trend that the efficiency solution for future models (especially the large models) will feature a systematical optimization and co-design in many axes (e.g., hardware, system, and algorithm); (3) profiling is always a good start to understand the problem so as to build the right efficiency portfolio.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798382318530Subjects--Topical Terms:

569006
Computer engineering.
Subjects--Index Terms:

Computer visionIndex Terms--Genre/Form:

554714
Electronic books.

Towards Efficient Deep Learning in Computer Vision via Network Sparsity and Distillation.
LDR:05411ntm a22003977 4500 001 1149731
005 20241022112625.5
006 m o d
007 cr bn ---uuuuu
008 250605s2024 xx obm 000 0 eng d
020 $a 9798382318530
035 $a (MiAaPQ)AAI31241458
035 $a AAI31241458
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Wang, Huan. $3 1236832
245 1 0 $a Towards Efficient Deep Learning in Computer Vision via Network Sparsity and Distillation.
264 0 $c 2024
300 $a 1 online resource (251 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-10, Section: B.
500 $a Advisor: Fu, Yun.
502 $a Thesis (Ph.D.)--Northeastern University, 2024.
504 $a Includes bibliographical references
520 $a Artificial intelligence (AI) empowered by deep learning, has been profoundly transforming the world. However, the excessive size of these models remains a central obstacle that limits their broader utility. Modern neural networks commonly consist of millions of parameters, with foundation models extending to billions. The rapid expansion in model size introduces many challenges including training cost, sluggish inference speed, excessive energy consumption, and negative environmental implications such as increased CO2 emissions.Addressing these challenges necessitates the adoption of efficient deep learning (EDL). The dissertation focuses on two overarching approaches, network sparsity (a.k.a. pruning) and knowledge distillation, to enhance the efficiency of deep learning models in the context of computer vision. Network pruning focuses on eliminating redundant parameters in a model while preserving the performance. Knowledge distillation aims to enhance the performance of the target model, referred to as the "student", by leveraging guidance from a stronger model, known as the "teacher". This approach leads to performance improvements in the target model without reducing its size.In this dissertation, I will start with the background and motivation for more efficient deep learning models in the past several years in the context of the arising foundation models. Then, the basic concepts, goals, and challenges of EDL will be introduced along with the major sub-methods. After that, the major part of this dissertation will be dedicated to elaborating on the proposed efficiency algorithms based on pruning and distillation in a variety of applications.For the pruning part, the dissertation first presents an effective pruning algorithm GReg in image classification, by tapping into a growing regularization strategy. Then, in order to understand the real progress of network pruning, a fairness principle is introduced to fairly compare different pruning methods. The investigation leads us to the central role of network trainability in pruning, which has been largely overlooked by prior works. A trainability-preserving pruning approach, TPP, is then proposed to show the merits of maintaining trainability during pruning. A short survey on an emerging new pruning paradigm, pruning at initialization, is then presented to discuss its potential and the connections with the conventional pruning after training. The GReg algorithm is further extended to a low-level vision task, single image super-resolution (SR), to explore the difference of utilizing pruning in low-level vision (SR) vs. high-level vision (image classification). Three efficient SR approaches (ASSL, GASSL,SRP) are introduced.For the distillation part, the dissertation first focuses on the interaction between knowledge distillation and data augmentation in image classification, a proved proposition presented to rigorously understand what defines the "goodness" of a data augmentation scheme in distillation. Next, the dissertation showcases how to employ distillation to significantly improve the inference efficiency for novel view synthesis in 3D vision. Both static scenes and dynamic scenes are considered. Finally, SnapFusion is presented to demonstrate a systematic efficiency optimization of deep models by jointly utilizing pruning and distillation, towards an unprecedentedly fast speed of text-to-image generation based on diffusion models.Finally, a comprehensive summary along with takeaways and outlooks of the future work will conclude the dissertation. Major takeaways include (1) there is no panacea towards efficient deep learning for all tasks; solution is usually case-by-case; (2) there is a clear trend that the efficiency solution for future models (especially the large models) will feature a systematical optimization and co-design in many axes (e.g., hardware, system, and algorithm); (3) profiling is always a good start to understand the problem so as to build the right efficiency portfolio.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer engineering. $3 569006
650 4 $a Computer science. $3 573171
653 $a Computer vision
653 $a Efficient deep learning
653 $a Knowledge distillation
653 $a Network pruning
653 $a Network sparsity
655 7 $a Electronic books. $2 local $3 554714
690 $a 0464
690 $a 0800
690 $a 0984
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a Northeastern University. $b Electrical and Computer Engineering. $3 1182193
773 0 $t Dissertations Abstracts International $g 85-10B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31241458 $z click for full text (PQDT)