國立虎尾科技大學 |

Multi-Tenant Machine Learning Model Serving Systems on GPU Clusters.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Multi-Tenant Machine Learning Model Serving Systems on GPU Clusters./
Author:	Chen, Lequn.
Description:	1 online resource (99 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-10, Section: B.
Contained By:	Dissertations Abstracts International85-10B.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798382211862

Multi-Tenant Machine Learning Model Serving Systems on GPU Clusters.
Chen, Lequn.

Multi-Tenant Machine Learning Model Serving Systems on GPU Clusters. - 1 online resource (99 pages)

Source: Dissertations Abstracts International, Volume: 85-10, Section: B.

Thesis (Ph.D.)--University of Washington, 2024.

Includes bibliographical references

In an era where GPUs are both costly and scarce, efficiently serving machine learning models has become a critical challenge. Assuming that serving one model requires k GPUs, serving n models would seemingly require kn GPUs. In the multi-tenant setting, we can pool the whole cluster's GPUs to serve the n models collectively, thus requiring far fewer GPUs. This talk addresses how to optimize cluster-wide GPU utilization in a multi-tenant setting. Key challenges addressed include: (1) batching efficiency under latency constraints, (2) bursty requests and GPU consolidation, (3) GPU cluster auto-scaling.This dissertation discusses two projects that address the above research problems. The first project, Symphony, focuses on serving DNN models. With a novel Deferred Batch Scheduling algorithm and a system design supporting it, Symphony makes high-quality batching decisions and enables robust auto-scaling. Symphony achieves 6x goodput given the same number of GPUs, saves 60% GPUs when serving the same request rate, and is capable to handle 15 million requests per second. The second project, Punica, creates a new paradigm of serving multiple LoRA fine-tuned large language models at the cost of one. Punica improves throughput by 12x without latency sacrifice.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798382211862Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

InferenceIndex Terms--Genre/Form:

554714
Electronic books.

Multi-Tenant Machine Learning Model Serving Systems on GPU Clusters.
LDR:02589ntm a22003737 4500 001 1144980
005 20240617111733.5
006 m o d
007 cr mn ---uuuuu
008 250605s2024 xx obm 000 0 eng d
020 $a 9798382211862
035 $a (MiAaPQ)AAI30997236
035 $a AAI30997236
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Chen, Lequn. $3 1470175
245 1 0 $a Multi-Tenant Machine Learning Model Serving Systems on GPU Clusters.
264 0 $c 2024
300 $a 1 online resource (99 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-10, Section: B.
500 $a Advisor: Krishnamurthy, Arvind.
502 $a Thesis (Ph.D.)--University of Washington, 2024.
504 $a Includes bibliographical references
520 $a In an era where GPUs are both costly and scarce, efficiently serving machine learning models has become a critical challenge. Assuming that serving one model requires k GPUs, serving n models would seemingly require kn GPUs. In the multi-tenant setting, we can pool the whole cluster's GPUs to serve the n models collectively, thus requiring far fewer GPUs. This talk addresses how to optimize cluster-wide GPU utilization in a multi-tenant setting. Key challenges addressed include: (1) batching efficiency under latency constraints, (2) bursty requests and GPU consolidation, (3) GPU cluster auto-scaling.This dissertation discusses two projects that address the above research problems. The first project, Symphony, focuses on serving DNN models. With a novel Deferred Batch Scheduling algorithm and a system design supporting it, Symphony makes high-quality batching decisions and enables robust auto-scaling. Symphony achieves 6x goodput given the same number of GPUs, saves 60% GPUs when serving the same request rate, and is capable to handle 15 million requests per second. The second project, Punica, creates a new paradigm of serving multiple LoRA fine-tuned large language models at the cost of one. Punica improves throughput by 12x without latency sacrifice.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Computer engineering. $3 569006
653 $a Inference
653 $a Large language model
653 $a Model serving systems
653 $a GPU clusters
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0464
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a University of Washington. $b Computer Science and Engineering. $3 1182238
773 0 $t Dissertations Abstracts International $g 85-10B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30997236 $z click for full text (PQDT)