國立虎尾科技大學 |

Human-Centric Reward Design.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Human-Centric Reward Design./
Author:	Du, Yu Qing.
Description:	1 online resource (118 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-08, Section: B.
Contained By:	Dissertations Abstracts International85-08B.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798381704136

Human-Centric Reward Design.
Du, Yu Qing.

Human-Centric Reward Design. - 1 online resource (118 pages)

Source: Dissertations Abstracts International, Volume: 85-08, Section: B.

Thesis (Ph.D.)--University of California, Berkeley, 2023.

Includes bibliographical references

How can we elicit the behaviors we want from artificial agents? One way of guiding behaviors of intelligent systems is through reward design. By specifying reward functions to optimize, we can use reinforcement learning (RL) to enable agents to learn from their own experience and interactions. Thus, RL has seen great success in settings where it is feasible to hand-specify reward functions that are well-aligned with the intended behaviors (e.g., using scores as rewards for games). However, as we progress to developing intelligent systems that have to learn more complex behaviors in the rich, diverse real world, reward design becomes increasingly difficult-and crucial. To address this challenge, we posit that improving reward signals will require new ways of incorporating human input.This thesis comprises two main parts: reward design directly using human input or indirectly using general knowledge we have about people. In the first part, we propose a framework for building robust reward models from direct human feedback. We present a reward modeling formulation that is amenable to large-scale pretrained vision-language models, leading to more generalizable multimodal reward functions under visual and language distribution shifts. In the second part, we use broad knowledge about humans as novel forms of input for reward design. In the human assistance setting, we propose using human empowerment as a task-agnostic reward input. This enables us to train assistive agents that circumvent limitations of existing goal inference based methods, while also aiming to preserve human autonomy. Finally, we study the case of eliciting exploratory behaviors in artificial agents. Unlike prior work that indiscriminately optimizes for diversity in order to encourage exploration, we propose leveraging human priors and general world knowledge to design intrinsic reward functions that lead to more human-like exploration. To better understand how intrinsic objectives guiding human behavior can inform agent design, we also compare how well human and agent behaviors in an open-ended exploration setting align with commonly-proposed information theoretic objectives used as intrinsic rewards. We conclude with some reflections on reward design challenges and directions for future work.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798381704136Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

Artificial agentsIndex Terms--Genre/Form:

554714
Electronic books.

Human-Centric Reward Design.
LDR:03591ntm a22003737 4500 001 1146410
005 20240812064608.5
006 m o d
007 cr bn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798381704136
035 $a (MiAaPQ)AAI30693527
035 $a AAI30693527
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Du, Yu Qing. $3 1471797
245 1 0 $a Human-Centric Reward Design.
264 0 $c 2023
300 $a 1 online resource (118 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-08, Section: B.
500 $a Advisor: Abbeel, Pieter.
502 $a Thesis (Ph.D.)--University of California, Berkeley, 2023.
504 $a Includes bibliographical references
520 $a How can we elicit the behaviors we want from artificial agents? One way of guiding behaviors of intelligent systems is through reward design. By specifying reward functions to optimize, we can use reinforcement learning (RL) to enable agents to learn from their own experience and interactions. Thus, RL has seen great success in settings where it is feasible to hand-specify reward functions that are well-aligned with the intended behaviors (e.g., using scores as rewards for games). However, as we progress to developing intelligent systems that have to learn more complex behaviors in the rich, diverse real world, reward design becomes increasingly difficult-and crucial. To address this challenge, we posit that improving reward signals will require new ways of incorporating human input.This thesis comprises two main parts: reward design directly using human input or indirectly using general knowledge we have about people. In the first part, we propose a framework for building robust reward models from direct human feedback. We present a reward modeling formulation that is amenable to large-scale pretrained vision-language models, leading to more generalizable multimodal reward functions under visual and language distribution shifts. In the second part, we use broad knowledge about humans as novel forms of input for reward design. In the human assistance setting, we propose using human empowerment as a task-agnostic reward input. This enables us to train assistive agents that circumvent limitations of existing goal inference based methods, while also aiming to preserve human autonomy. Finally, we study the case of eliciting exploratory behaviors in artificial agents. Unlike prior work that indiscriminately optimizes for diversity in order to encourage exploration, we propose leveraging human priors and general world knowledge to design intrinsic reward functions that lead to more human-like exploration. To better understand how intrinsic objectives guiding human behavior can inform agent design, we also compare how well human and agent behaviors in an open-ended exploration setting align with commonly-proposed information theoretic objectives used as intrinsic rewards. We conclude with some reflections on reward design challenges and directions for future work.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
653 $a Artificial agents
653 $a Reinforcement learning
653 $a Intelligent systems
653 $a Reward functions
655 7 $a Electronic books. $2 local $3 554714
690 $a 0800
690 $a 0984
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a University of California, Berkeley. $b Computer Science. $3 1179511
773 0 $t Dissertations Abstracts International $g 85-08B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30693527 $z click for full text (PQDT)