distributed representations, simple recurrent networks, and grammatical structure（分布式表示,简单的复发性网络,和语法结构）.pdf

Machine Learning, 7, 195-225 (1991) © 1991 Kluwer Academic Publishers, Boston. Manufactured in The Netherlands. Distributed Representations, Simple Recurrent Networks, and Grammatical Structure JEFFREY L. ELMAN (ELMAN@CRL.UCSD.EDU) Departments of Cognitive Science and Linguistics, University of California, San Diego Abstract. In this paper three problems for a connectionist account of language are considered: 1. What is the nature of linguistic representations? 2. How can complex structural relationships such as constituent structure be represented? 3. How can the apparently open-ended nature of language be accommodated by a fixed-resource system? Using a prediction task, a simple recurrent network (SRN) is trained on multiclausal sentences which contain multiply-embedded relative clauses. Principal component analysis of the hidden unit activation patterns reveals that the network solves the task by developing complex distributed representations which encode the relevant grammatical relations and hierarchical constituent structure. Differences between the SRN state representations and the more traditional pushdown store are discussed in the final section. Keywords. Distributed representations, simple recurrent networks, grammatical structure 1. Introduction In recent years there has been considerable progress in developing connectionist models of language. This work has demonstrated the ability of network models to account for a variety of phenomena in phonology (e.g., Gasser Lee, 1990; Hare, 1990; Touretzky, 1989; Touretzky Wheeler, 1989), morphology (e.g., Hare, Corina, Cottrell, 1989; MacWhinney et