國立虎尾科技大學 |

How embedded memory affects the performance of recurrent neural networks.

Record Type:	Language materials, manuscript : Monograph/item
Title/Author:	How embedded memory affects the performance of recurrent neural networks./
Author:	Lin, Tsung-Nan.
Description:	1 online resource (163 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 58-08, Section: B.
Contained By:	Dissertations Abstracts International58-08B.
Subject:	Electrical engineering. -
Online resource:	click for full text (PQDT)
ISBN:	9780591171402

How embedded memory affects the performance of recurrent neural networks.
Lin, Tsung-Nan.

How embedded memory affects the performance of recurrent neural networks. - 1 online resource (163 pages)

Source: Dissertations Abstracts International, Volume: 58-08, Section: B.

Thesis (Ph.D.)--Princeton University, 1996.

Includes bibliographical references

Dynamical neural networks are fascinating with respect to their capacity of modeling nonlinear system dynamics from input/output behavior. They differ from static models in the sense that dynamical models have memory elements, which are the key components for preserving their states and making them able to handle time dependent problems. A rigorous understanding of how the memory elements affect the performance of dynamical neural networks is thus very important in practice. Since different architectures may be theoretically equivalent in terms of representation power yet there might be some practical advantages to use one architecture over another. This dissertation attempts to take a step in this direction. To achieve the goal, we study carefully the effects of embedded memory on the performance of a class of neural networks with observable states (NARX neural networks); then extend to other dynamical networks with hidden states. First, we show that NARX networks are able to simulate a class of finite state machines. Of particular interest is the result that NARX networks are able to "learn" very large state machines. Then, an extensive exploration toward the effect of propagating the gradient information through the embedded memory in NARX networks on the problem of long-term dependencies is presented. The third, we extend the similar improvement in reducing the sensitivity to long-term dependency learning to other types of recurrent neural networks. Fourth, the effects of embedded memory on generalization for NARX networks are also critically examined. Increasing the order of embedded memory will increase the learning performance, while the generalization performance will also strongly depend on the memory orders. Therefore, choosing a proper memory architecture to balance the learning ability and generalization performance becomes a crucial topic to be explored. With an ambitious attempt to provide an efficient algorithm, we propose a pruning-based algorithm to determine the optimal memory architecture. The final memory architectures always result in sparse connections which will make useful information more explicit and help the networks more robustly model the long-term behavior of the underlying system.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2024

Mode of access: World Wide Web

ISBN: 9780591171402Subjects--Topical Terms:

596380
Electrical engineering.
Subjects--Index Terms:

artificial intelligenceIndex Terms--Genre/Form:

554714
Electronic books.

How embedded memory affects the performance of recurrent neural networks.
LDR:03542ntm a22003617 4500 001 1144365
005 20240531083836.5
006 m o d
007 cr mn ---uuuuu
008 250605s1996 xx obm 000 0 eng d
020 $a 9780591171402
035 $a (MiAaPQ)AAI9709698
035 $a AAI9709698
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Lin, Tsung-Nan. $3 1469360
245 1 0 $a How embedded memory affects the performance of recurrent neural networks.
264 0 $c 1996
300 $a 1 online resource (163 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 58-08, Section: B.
500 $a Publisher info.: Dissertation/Thesis.
502 $a Thesis (Ph.D.)--Princeton University, 1996.
504 $a Includes bibliographical references
520 $a Dynamical neural networks are fascinating with respect to their capacity of modeling nonlinear system dynamics from input/output behavior. They differ from static models in the sense that dynamical models have memory elements, which are the key components for preserving their states and making them able to handle time dependent problems. A rigorous understanding of how the memory elements affect the performance of dynamical neural networks is thus very important in practice. Since different architectures may be theoretically equivalent in terms of representation power yet there might be some practical advantages to use one architecture over another. This dissertation attempts to take a step in this direction. To achieve the goal, we study carefully the effects of embedded memory on the performance of a class of neural networks with observable states (NARX neural networks); then extend to other dynamical networks with hidden states. First, we show that NARX networks are able to simulate a class of finite state machines. Of particular interest is the result that NARX networks are able to "learn" very large state machines. Then, an extensive exploration toward the effect of propagating the gradient information through the embedded memory in NARX networks on the problem of long-term dependencies is presented. The third, we extend the similar improvement in reducing the sensitivity to long-term dependency learning to other types of recurrent neural networks. Fourth, the effects of embedded memory on generalization for NARX networks are also critically examined. Increasing the order of embedded memory will increase the learning performance, while the generalization performance will also strongly depend on the memory orders. Therefore, choosing a proper memory architecture to balance the learning ability and generalization performance becomes a crucial topic to be explored. With an ambitious attempt to provide an efficient algorithm, we propose a pruning-based algorithm to determine the optimal memory architecture. The final memory architectures always result in sparse connections which will make useful information more explicit and help the networks more robustly model the long-term behavior of the underlying system.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2024
538 $a Mode of access: World Wide Web
650 4 $a Electrical engineering. $3 596380
650 4 $a Computer science. $3 573171
653 $a artificial intelligence
653 $a finite memory machine
655 7 $a Electronic books. $2 local $3 554714
690 $a 0544
690 $a 0984
690 $a 0800
710 2 $a ProQuest Information and Learning Co. $3 1178819
710 2 $a Princeton University. $2 fast $3 1008168
773 0 $t Dissertations Abstracts International $g 58-08B.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=9709698 $z click for full text (PQDT)