國立虎尾科技大學 |

Towards Understanding the Role of Knowledge in Improving Transformer-Based Language Models.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Towards Understanding the Role of Knowledge in Improving Transformer-Based Language Models./
Author:	Pal, Kuntal Kumar.
Description:	1 online resource (283 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-02, Section: B.
Contained By:	Dissertations Abstracts International85-02B.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798380109390

Towards Understanding the Role of Knowledge in Improving Transformer-Based Language Models.
Pal, Kuntal Kumar.

Towards Understanding the Role of Knowledge in Improving Transformer-Based Language Models. - 1 online resource (283 pages)

Source: Dissertations Abstracts International, Volume: 85-02, Section: B.

Thesis (Ph.D.)--Arizona State University, 2023.

Includes bibliographical references

In natural language processing, language models have achieved remarkable success over the last few years. The Transformers are at the core of most of these models. Their success can be mainly attributed to an enormous amount of curated data they are trained on. Even though such language models are trained on massive curated data, they often need specific extracted knowledge to understand better and reason. This is because often relevant knowledge may be implicit or missing, which hampers machine reasoning. Apart from that, manual knowledge curation is time-consuming and erroneous. Hence, finding fast and effective methods to extract such knowledge from data is important for improving language models. This leads to finding ideal ways to utilize such knowledge by incorporating them into language models. Successful knowledge extraction and integration lead to an important question of knowledge evaluation of such models by developing tools or introducing challenging test suites to learn about their limitations and improve them further. So to improve the transformer-based models, understanding the role of knowledge becomes important.In the pursuit to improve language models with knowledge, in this dissertation I study three broad research directions spanning across the natural language, biomedical and cybersecurity domains: (1) Knowledge Extraction (KX) - How can transformer-based language models be leveraged to extract knowledge from data? (2) Knowledge Integration (KI) - How can such specific knowledge be used to improve such models? (3) Knowledge Evaluation (KE) - How can language models be evaluated for specific skills and understand their limitations?I propose methods to extract explicit textual, implicit structural, missing textual, and missing structural knowledge from natural language and binary programs using transformer-based language models. I develop ways to improve the language model's multi-step and commonsense reasoning abilities using external knowledge. Finally, I develop challenging datasets which assess their numerical reasoning skills in both in-domain and out-of-domain settings.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798380109390Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

EvaluationIndex Terms--Genre/Form:

554714
Electronic books.

Towards Understanding the Role of Knowledge in Improving Transformer-Based Language Models.
LDR:03506ntm a22003977 4500 001 1143620
005 20240517104614.5
006 m o d
007 cr mn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798380109390
035 $a (MiAaPQ)AAI30531226
035 $a AAI30531226
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Pal, Kuntal Kumar. $3 1468360
245 1 0 $a Towards Understanding the Role of Knowledge in Improving Transformer-Based Language Models.
264 0 $c 2023
300 $a 1 online resource (283 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-02, Section: B.
500 $a Advisor: Baral, Chitta.
502 $a Thesis (Ph.D.)--Arizona State University, 2023.
504 $a Includes bibliographical references
520 $a In natural language processing, language models have achieved remarkable success over the last few years. The Transformers are at the core of most of these models. Their success can be mainly attributed to an enormous amount of curated data they are trained on. Even though such language models are trained on massive curated data, they often need specific extracted knowledge to understand better and reason. This is because often relevant knowledge may be implicit or missing, which hampers machine reasoning. Apart from that, manual knowledge curation is time-consuming and erroneous. Hence, finding fast and effective methods to extract such knowledge from data is important for improving language models. This leads to finding ideal ways to utilize such knowledge by incorporating them into language models. Successful knowledge extraction and integration lead to an important question of knowledge evaluation of such models by developing tools or introducing challenging test suites to learn about their limitations and improve them further. So to improve the transformer-based models, understanding the role of knowledge becomes important.In the pursuit to improve language models with knowledge, in this dissertation I study three broad research directions spanning across the natural language, biomedical and cybersecurity domains: (1) Knowledge Extraction (KX) - How can transformer-based language models be leveraged to extract knowledge from data? (2) Knowledge Integration (KI) - How can such specific knowledge be used to improve such models? (3) Knowledge Evaluation (KE) - How can language models be evaluated for specific skills and understand their limitations?I propose methods to extract explicit textual, implicit structural, missing textual, and missing structural knowledge from natural language and binary programs using transformer-based language models. I develop ways to improve the language model's multi-step and commonsense reasoning abilities using external knowledge. Finally, I develop challenging datasets which assess their numerical reasoning skills in both in-domain and out-of-domain settings.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Linguistics. $3 557829
653 $a Evaluation
653 $a Knowledge extraction
653 $a Language model
653 $a Reasoning
653 $a Transformers
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0290
690 $a 0800
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a Arizona State University. $b Computer Science. $3 845377
773 0 $t Dissertations Abstracts International $g 85-02B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30531226 $z click for full text (PQDT)