國立虎尾科技大學 |

Statistical and High-Dimensional Perspectives on Machine Learning.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Statistical and High-Dimensional Perspectives on Machine Learning./
Author:	Lee, Donghwan.
Description:	1 online resource (270 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-12, Section: A.
Contained By:	Dissertations Abstracts International85-12A.
Subject:	Applied mathematics. -
Online resource:	click for full text (PQDT)
ISBN:	9798382830889

Statistical and High-Dimensional Perspectives on Machine Learning.
Lee, Donghwan.

Statistical and High-Dimensional Perspectives on Machine Learning. - 1 online resource (270 pages)

Source: Dissertations Abstracts International, Volume: 85-12, Section: A.

Thesis (Ph.D.)--University of Pennsylvania, 2024.

Includes bibliographical references

In the first chapter, we consider the problem of calibration. While the accuracy of modern machine learning techniques continues to improve, many models exhibit mis-calibration, wherein the probability scores produced by the models fail to align with the actual frequencies of the labels. This discrepancy can lead to unreliable predictions and hinder the practical application of these models. To address this issue, we frame the task of detecting mis-calibration as a hypothesis testing problem. Drawing inspiration from nonparametric hypothesis testing, we propose T-Cal, a minimax optimal test for calibration based on a debiased plug-in estimator of the ℓ2-Expected Calibration Error (ECE). T-Cal offers a principled and statistically sound approach to assess the calibration of machine learning models.The second chapter focuses on out-of-distribution performance estimation. Evaluating model performance under distribution shift is particularly challenging when we are only given unlabeled data from the target domain. Recent work suggests the notion of disagreement, the degree to which two models trained with different randomness differ on the same input, can be used as a proxy for the accuracy. We establish a theoretical foundation for analyzing disagreement in high-dimensional random features regression. Our analysis shows that there is a linear relationship between source and target disagreement, which we can leverage to estimate the out-of-distribution performance.The third chapter studies feature learning in two-layer neural networks, which is considered one of the fundamental reasons behind the success of deep neural networks. Despite its significance, existing theoretical frameworks do not fully explain the mechanism of feature learning, even in the simplest case of two-layer neural networks. In this work, we enrich our understanding of feature learning by considering a general setting where the learning rate grows with the sample size. Under this setting, we demonstrate that a single step of gradient descent introduces multiple rank-one components to the feature matrix, each corresponding to a specific polynomial feature. Furthermore, we prove that the limiting training and test errors of the updated neural networks are fully characterized by these spikes. By precisely analyzing the improvement in the training and test errors, we illustrate how these non-linear features can enhance the learning process. Through this comprehensive analysis, we shed light on the intricate dynamics of feature learning and its crucial role in the performance of neural networks.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798382830889Subjects--Topical Terms:

1069907
Applied mathematics.
Subjects--Index Terms:

Deep learning theoryIndex Terms--Genre/Form:

554714
Electronic books.

Statistical and High-Dimensional Perspectives on Machine Learning.
LDR:04068ntm a22004217 4500 001 1147576
005 20240909103816.5
006 m o d
007 cr bn ---uuuuu
008 250605s2024 xx obm 000 0 eng d
020 $a 9798382830889
035 $a (MiAaPQ)AAI31237254
035 $a AAI31237254
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Lee, Donghwan. $3 1181857
245 1 0 $a Statistical and High-Dimensional Perspectives on Machine Learning.
264 0 $c 2024
300 $a 1 online resource (270 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-12, Section: A.
500 $a Advisor: Dobriban, Edgar;Hassani, Hamed.
502 $a Thesis (Ph.D.)--University of Pennsylvania, 2024.
504 $a Includes bibliographical references
520 $a In the first chapter, we consider the problem of calibration. While the accuracy of modern machine learning techniques continues to improve, many models exhibit mis-calibration, wherein the probability scores produced by the models fail to align with the actual frequencies of the labels. This discrepancy can lead to unreliable predictions and hinder the practical application of these models. To address this issue, we frame the task of detecting mis-calibration as a hypothesis testing problem. Drawing inspiration from nonparametric hypothesis testing, we propose T-Cal, a minimax optimal test for calibration based on a debiased plug-in estimator of the ℓ2-Expected Calibration Error (ECE). T-Cal offers a principled and statistically sound approach to assess the calibration of machine learning models.The second chapter focuses on out-of-distribution performance estimation. Evaluating model performance under distribution shift is particularly challenging when we are only given unlabeled data from the target domain. Recent work suggests the notion of disagreement, the degree to which two models trained with different randomness differ on the same input, can be used as a proxy for the accuracy. We establish a theoretical foundation for analyzing disagreement in high-dimensional random features regression. Our analysis shows that there is a linear relationship between source and target disagreement, which we can leverage to estimate the out-of-distribution performance.The third chapter studies feature learning in two-layer neural networks, which is considered one of the fundamental reasons behind the success of deep neural networks. Despite its significance, existing theoretical frameworks do not fully explain the mechanism of feature learning, even in the simplest case of two-layer neural networks. In this work, we enrich our understanding of feature learning by considering a general setting where the learning rate grows with the sample size. Under this setting, we demonstrate that a single step of gradient descent introduces multiple rank-one components to the feature matrix, each corresponding to a specific polynomial feature. Furthermore, we prove that the limiting training and test errors of the updated neural networks are fully characterized by these spikes. By precisely analyzing the improvement in the training and test errors, we illustrate how these non-linear features can enhance the learning process. Through this comprehensive analysis, we shed light on the intricate dynamics of feature learning and its crucial role in the performance of neural networks.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Applied mathematics. $3 1069907
650 4 $a Statistics. $3 556824
650 4 $a Computer science. $3 573171
650 4 $a Information science. $3 561178
653 $a Deep learning theory
653 $a Machine learning theory
653 $a Hypothesis testing
653 $a Two-layer neural networks
653 $a Out-of-distribution performance
655 7 $a Electronic books. $2 local $3 554714
690 $a 0364
690 $a 0463
690 $a 0984
690 $a 0800
690 $a 0723
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a University of Pennsylvania. $b Applied Mathematics and Computational Science. $3 1180480
773 0 $t Dissertations Abstracts International $g 85-12A.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31237254 $z click for full text (PQDT)