國立虎尾科技大學 |

Improve Language Model Serving Efficiency With Fine-Grained and Stateful Scheduling.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Improve Language Model Serving Efficiency With Fine-Grained and Stateful Scheduling./
Author:	Yu, Lingfan.
Description:	1 online resource (115 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-12, Section: A.
Contained By:	Dissertations Abstracts International85-12A.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798383161364

Improve Language Model Serving Efficiency With Fine-Grained and Stateful Scheduling.
Yu, Lingfan.

Improve Language Model Serving Efficiency With Fine-Grained and Stateful Scheduling. - 1 online resource (115 pages)

Source: Dissertations Abstracts International, Volume: 85-12, Section: A.

Thesis (Ph.D.)--New York University, 2024.

Includes bibliographical references

The world has witnessed the remarkable success of large language models (LLMs), led by the fast-growing popularity of ChatGPT. However, it is challenging to serve these language models and deliver both high throughput and low latency due to the iterative nature of language models. This thesis identifies two key issues impacting the performance of existing systems: (1) Coarse-grained batching at the request level results in wasteful computation for requests with variable input and output lengths; (2) The lack of stateful context management results in duplicate computation for applications that engage in multi-turn interactions with the LLM model. Two systems, \uD835\uDC35\uD835\uDC4E\uD835\uDC61\uD835\uDC50ℎ\uD835\uDC40\uD835\uDC4E\uD835\uDC58\uD835\uDC52\uD835\uDC5F and \uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52, are then presented to address these issues.\uD835\uDC35\uD835\uDC4E\uD835\uDC61\uD835\uDC50ℎ\uD835\uDC40\uD835\uDC4E\uD835\uDC58\uD835\uDC52\uD835\uDC5F proposes a technique called cellular batching to improve the throughput and latency of language model inference. Existing systems use batch execution of the dataflow graphs of a fixed set of requests. By contrast, \uD835\uDC35\uD835\uDC4E\uD835\uDC61\uD835\uDC50ℎ\uD835\uDC40\uD835\uDC4E\uD835\uDC58\uD835\uDC52\uD835\uDC5F makes finer-grained batching decisions at each token processing step, and dynamically assembles a batch for execution as requests join and leave the system.\uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52 is a system optimized for multi-turn conversation LLM serving. It maintains the conversation state across requests from the same conversation by caching previously processed history to avoid duplicate processing. \uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52's multi-tier caching strategy utilizes both GPU and CPU memory to store and retrieve cached data efficiently. \uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52 also generalizes the recent Paged-Attention kernel to support attention between multiple input tokens whose KV cache is spread over non-contiguous GPU memory.Experiments on various workloads show that \uD835\uDC35\uD835\uDC4E\uD835\uDC61\uD835\uDC50ℎ\uD835\uDC40\uD835\uDC4E\uD835\uDC58\uD835\uDC52\uD835\uDC5F improves throughput by 25-80% while reducing latency by 18-90% latency, and \uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52 improves 33-100% throughput and reduces latency by 40-77%.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798383161364Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

Cache managementIndex Terms--Genre/Form:

554714
Electronic books.

Improve Language Model Serving Efficiency With Fine-Grained and Stateful Scheduling.
LDR:04033ntm a22003857 4500 001 1150154
005 20241022111601.5
006 m o d
007 cr bn ---uuuuu
008 250605s2024 xx obm 000 0 eng d
020 $a 9798383161364
035 $a (MiAaPQ)AAI31139782
035 $a AAI31139782
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Yu, Lingfan. $3 1476589
245 1 0 $a Improve Language Model Serving Efficiency With Fine-Grained and Stateful Scheduling.
264 0 $c 2024
300 $a 1 online resource (115 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-12, Section: A.
500 $a Advisor: Li, Jinyang.
502 $a Thesis (Ph.D.)--New York University, 2024.
504 $a Includes bibliographical references
520 $a The world has witnessed the remarkable success of large language models (LLMs), led by the fast-growing popularity of ChatGPT. However, it is challenging to serve these language models and deliver both high throughput and low latency due to the iterative nature of language models. This thesis identifies two key issues impacting the performance of existing systems: (1) Coarse-grained batching at the request level results in wasteful computation for requests with variable input and output lengths; (2) The lack of stateful context management results in duplicate computation for applications that engage in multi-turn interactions with the LLM model. Two systems, \uD835\uDC35\uD835\uDC4E\uD835\uDC61\uD835\uDC50ℎ\uD835\uDC40\uD835\uDC4E\uD835\uDC58\uD835\uDC52\uD835\uDC5F and \uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52, are then presented to address these issues.\uD835\uDC35\uD835\uDC4E\uD835\uDC61\uD835\uDC50ℎ\uD835\uDC40\uD835\uDC4E\uD835\uDC58\uD835\uDC52\uD835\uDC5F proposes a technique called cellular batching to improve the throughput and latency of language model inference. Existing systems use batch execution of the dataflow graphs of a fixed set of requests. By contrast, \uD835\uDC35\uD835\uDC4E\uD835\uDC61\uD835\uDC50ℎ\uD835\uDC40\uD835\uDC4E\uD835\uDC58\uD835\uDC52\uD835\uDC5F makes finer-grained batching decisions at each token processing step, and dynamically assembles a batch for execution as requests join and leave the system.\uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52 is a system optimized for multi-turn conversation LLM serving. It maintains the conversation state across requests from the same conversation by caching previously processed history to avoid duplicate processing. \uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52's multi-tier caching strategy utilizes both GPU and CPU memory to store and retrieve cached data efficiently. \uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52 also generalizes the recent Paged-Attention kernel to support attention between multiple input tokens whose KV cache is spread over non-contiguous GPU memory.Experiments on various workloads show that \uD835\uDC35\uD835\uDC4E\uD835\uDC61\uD835\uDC50ℎ\uD835\uDC40\uD835\uDC4E\uD835\uDC58\uD835\uDC52\uD835\uDC5F improves throughput by 25-80% while reducing latency by 18-90% latency, and \uD835\uDC43\uD835\uDC52\uD835\uDC5B\uD835\uDC60\uD835\uDC56\uD835\uDC52\uD835\uDC63\uD835\uDC52 improves 33-100% throughput and reduces latency by 40-77%.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Information science. $3 561178
653 $a Cache management
653 $a Fine-grained scheduling
653 $a Language model serving
653 $a Cellular batching
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0723
690 $a 0800
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a New York University. $b Computer Science. $3 1184382
773 0 $t Dissertations Abstracts International $g 85-12A.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31139782 $z click for full text (PQDT)