國立虎尾科技大學 |

Inducing Symbolic Characteristics in Neural Question-Answering Systems via Data Interventions.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	Inducing Symbolic Characteristics in Neural Question-Answering Systems via Data Interventions./
Author:	Dua, Dheeru.
Description:	1 online resource (210 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 85-02, Section: B.
Contained By:	Dissertations Abstracts International85-02B.
Subject:	Computer science. -
Online resource:	click for full text (PQDT)
ISBN:	9798380129763

Inducing Symbolic Characteristics in Neural Question-Answering Systems via Data Interventions.
Dua, Dheeru.

Inducing Symbolic Characteristics in Neural Question-Answering Systems via Data Interventions. - 1 online resource (210 pages)

Source: Dissertations Abstracts International, Volume: 85-02, Section: B.

Thesis (Ph.D.)--University of California, Irvine, 2023.

Includes bibliographical references

Question Answering (QA) is an innate way in which humans converse, understand abstract concepts, and reason about complex interactions among different objects. This makes question answering a great way to test the natural language understanding of an artificial intelligence system. The recent advances in model architectures and large-scale datasets have led to the development of neural QA systems that surpass human performance on simple question answering - which requires looking up a single fact in given a document. The reason behind the success of neural systems lies in their ability to directly learn features to extract answers from data. In contrast, symbolic systems encounter notable difficulties in scaling due to their restricted applicability to semi-structured or symbol-grounded data. Despite their reliance on structured data, symbolic systems demonstrate proficiency in executing deterministic operations and performing reasoning tasks. Conversely, neural systems exhibit limitations in reasoning, as they are (1) inconsistent, (2) unable to compose simple facts and perform complex reasoning, and (3) sensitive to changes in domain distribution.In this dissertation, we present a range of data intervention schemes that facilitate in building consistent, decomposable, and generalizable neural QA systems. In the first part, we show that purely neural systems are inconsistent and biased because most training and data collection procedures for neural systems make the independence assumptions. We explore two ways to address this problem in the context of question answering. Firstly, we introduce a way to curate related QA pairs and a contrastive loss function that takes bundles of related questions as input and learns to jointly answer them. Such related or minimally different data enable us to capture the relationship across examples that would otherwise have remained unseen or, at the very least, not seen simultaneously at the time of training. This method improves the performance over baseline methods by ∼14% in F1. Secondly, we take a generative approach to passage selection in multi-hop QA, where we learn a prior over interactions between the current passage and the subsequent passage for hopping along with a conditional question generation model, which ensure that parts of the question are not overlooked. This improves the performance over baseline on adversarial set by ∼ 5% in passage selection accuracy. In the second part, we first introduce a compositional QA dataset, DROP, that requires extracting multiple facts from a given document and then performing discrete (symbolic) operations on these facts while solving a question. We show that purely neural and symbolic methods perform poorly on this task - even commercial systems like GPT-4 are unable to reach human performance. Then, we show that prompting-based neural QA methods lack decomposability for two reasons: (1) the intermediate steps are not specified in a common format across different complex questions, (2) the model ignores valuable sub-problem demonstrations associated with complex questions that do not appear similar on the surface to the question at hand, even though they provide much better supervision signal. We propose a method that leverages synthetically generated data to break down a complex question into simpler sub-question answer pairs in a consistent format. This data intervention not only promotes transfer but also enables us to search for pertinent QA pairs at each step especially tailored to solving that step, further improving supervision signal and overall performance in a zero-shot setting by ∼5%.Finally, we dissect the complex interactions among questions, answers, and documents learned by a neural QA system to assess their effectiveness towards generalization under a range of different data distributions. We demonstrate three distinct data intervention techniques that aid in improving generalization capabilities. The first data intervention approach involves zero-shot and few-shot adaptation methods that use none to little examples from unseen domain to effectively adapt an existing QA model to those unseen domains. Our few-shot method improves out-of-domain generalization performance by up to ∼24%. This style of intervention is typically employed for adapting to a specific target domain. However, in real-world scenarios, a single model needs to possess the capability to learn from multiple domains. To this end, we introduced an open reading evaluation benchmark (ORB) that tests a single model performance on a range of reasoning abilities. We show that the models have a tendency to forget how to solve examples that they were previously capable of solving, when trained on multiple datasets. Our second intervention technique revolves around strategic sampling of examples from various datasets to alleviate catastrophic forgetting. This method, called dynamic sampling improves multi-task performance by ∼13%. In the final data intervention technique, we introduce intermediate reasoning steps when solving compositional problems. This approach reduces label bias by up to ∼61%. Overall, we demonstrate how data interventions can be utilized to induce characteristics of symbolic systems into neural QA systems.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9798380129763Subjects--Topical Terms:

573171
Computer science.
Subjects--Index Terms:

Question AnsweringIndex Terms--Genre/Form:

554714
Electronic books.

Inducing Symbolic Characteristics in Neural Question-Answering Systems via Data Interventions.
LDR:06690ntm a22003977 4500 001 1146399
005 20240812064605.5
006 m o d
007 cr bn ---uuuuu
008 250605s2023 xx obm 000 0 eng d
020 $a 9798380129763
035 $a (MiAaPQ)AAI30569619
035 $a AAI30569619
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Dua, Dheeru. $3 1471784
245 1 0 $a Inducing Symbolic Characteristics in Neural Question-Answering Systems via Data Interventions.
264 0 $c 2023
300 $a 1 online resource (210 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 85-02, Section: B.
500 $a Advisor: Singh, Sameer.
502 $a Thesis (Ph.D.)--University of California, Irvine, 2023.
504 $a Includes bibliographical references
520 $a Question Answering (QA) is an innate way in which humans converse, understand abstract concepts, and reason about complex interactions among different objects. This makes question answering a great way to test the natural language understanding of an artificial intelligence system. The recent advances in model architectures and large-scale datasets have led to the development of neural QA systems that surpass human performance on simple question answering - which requires looking up a single fact in given a document. The reason behind the success of neural systems lies in their ability to directly learn features to extract answers from data. In contrast, symbolic systems encounter notable difficulties in scaling due to their restricted applicability to semi-structured or symbol-grounded data. Despite their reliance on structured data, symbolic systems demonstrate proficiency in executing deterministic operations and performing reasoning tasks. Conversely, neural systems exhibit limitations in reasoning, as they are (1) inconsistent, (2) unable to compose simple facts and perform complex reasoning, and (3) sensitive to changes in domain distribution.In this dissertation, we present a range of data intervention schemes that facilitate in building consistent, decomposable, and generalizable neural QA systems. In the first part, we show that purely neural systems are inconsistent and biased because most training and data collection procedures for neural systems make the independence assumptions. We explore two ways to address this problem in the context of question answering. Firstly, we introduce a way to curate related QA pairs and a contrastive loss function that takes bundles of related questions as input and learns to jointly answer them. Such related or minimally different data enable us to capture the relationship across examples that would otherwise have remained unseen or, at the very least, not seen simultaneously at the time of training. This method improves the performance over baseline methods by ∼14% in F1. Secondly, we take a generative approach to passage selection in multi-hop QA, where we learn a prior over interactions between the current passage and the subsequent passage for hopping along with a conditional question generation model, which ensure that parts of the question are not overlooked. This improves the performance over baseline on adversarial set by ∼ 5% in passage selection accuracy. In the second part, we first introduce a compositional QA dataset, DROP, that requires extracting multiple facts from a given document and then performing discrete (symbolic) operations on these facts while solving a question. We show that purely neural and symbolic methods perform poorly on this task - even commercial systems like GPT-4 are unable to reach human performance. Then, we show that prompting-based neural QA methods lack decomposability for two reasons: (1) the intermediate steps are not specified in a common format across different complex questions, (2) the model ignores valuable sub-problem demonstrations associated with complex questions that do not appear similar on the surface to the question at hand, even though they provide much better supervision signal. We propose a method that leverages synthetically generated data to break down a complex question into simpler sub-question answer pairs in a consistent format. This data intervention not only promotes transfer but also enables us to search for pertinent QA pairs at each step especially tailored to solving that step, further improving supervision signal and overall performance in a zero-shot setting by ∼5%.Finally, we dissect the complex interactions among questions, answers, and documents learned by a neural QA system to assess their effectiveness towards generalization under a range of different data distributions. We demonstrate three distinct data intervention techniques that aid in improving generalization capabilities. The first data intervention approach involves zero-shot and few-shot adaptation methods that use none to little examples from unseen domain to effectively adapt an existing QA model to those unseen domains. Our few-shot method improves out-of-domain generalization performance by up to ∼24%. This style of intervention is typically employed for adapting to a specific target domain. However, in real-world scenarios, a single model needs to possess the capability to learn from multiple domains. To this end, we introduced an open reading evaluation benchmark (ORB) that tests a single model performance on a range of reasoning abilities. We show that the models have a tendency to forget how to solve examples that they were previously capable of solving, when trained on multiple datasets. Our second intervention technique revolves around strategic sampling of examples from various datasets to alleviate catastrophic forgetting. This method, called dynamic sampling improves multi-task performance by ∼13%. In the final data intervention technique, we introduce intermediate reasoning steps when solving compositional problems. This approach reduces label bias by up to ∼61%. Overall, we demonstrate how data interventions can be utilized to induce characteristics of symbolic systems into neural QA systems.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Computer science. $3 573171
650 4 $a Linguistics. $3 557829
653 $a Question Answering
653 $a Natural language
653 $a Datasets
653 $a QA systems
653 $a Reasoning abilities
655 7 $a Electronic books. $2 local $3 554714
690 $a 0984
690 $a 0800
690 $a 0290
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a University of California, Irvine. $b Computer Science. $3 1179555
773 0 $t Dissertations Abstracts International $g 85-02B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30569619 $z click for full text (PQDT)