We propose a method to generate large-scale encyclopedic knowledge , which is valuable for much NLP research , based on the Web . We first search the Web for pages containing a term in question . Then we use linguistic patterns and HTML structures to extract text fragments describing the term . Finally , we organize extracted term descriptions based on word senses and domains . In addition , we apply an automatically generated encyclopedia to a question answering system targeting the Japanese Information-Technology Engineers Examination . Reflecting the growth in utilization of the World Wide Web , a number of Web-based language processing methods have been proposed within the natural language processing ( NLP ) , information retrieval ( IR ) and artificial intelligence ( AI ) communities . A sample of these includes methods to extract linguistic resources ( Fujii and Ishikawa , 2000 ; Resnik , 1999 ; Soderland , 1997 ) , retrieve useful information in response to user queries ( Etzioni , 1997 ; McCallum et al . , 1999 ) and mine / discover knowledge latent in the Web ( Inokuchi et al . , 1999 ) . In this paper , mainly from an NLP point of view , we explore a method to produce linguistic resources . Specifically , we enhance the method proposed by Fujii and Ishikawa ( 2000 ) , which extracts encyclopedic knowledge ( i.e. , term descriptions ) from the Web . In brief , their method searches the Web for pages containing a term in question , and uses linguistic expressions and HTML layouts to extract fragments describing the term . They also use a language model to discard non-linguistic fragments . In addition , a clustering method is used to divide descriptions into a specific number of groups . On the one hand , their method is expected to enhance existing encyclopedias , where vocabulary size is relatively limited , and therefore the quantity problems has been resolved . On the other hand , encyclopedias extracted from the Web are not comparable with existing ones in terms of quality . In hand-crafted encyclopedias , term descriptions are carefully organized based on domains and word senses , which are especially effective for human usage . However , the output of Fujii 's method is simply a set of unorganized term descriptions . Although clustering is optionally performed , resultant clusters are not necessarily related to explicit criteria , such as word senses and domains . To sum up , our belief is that by combining extraction and organization methods , we can enhance both quantity and quality of Web-based encyclopedias . Motivated by this background , we introduce an organization model to Fujii 's method and reformalize the whole framework . In other words , our proposed method is not only extraction but generation of encyclopedic knowledge . Section 2 explains the overall design of our encyclopedia generation system , and Section 3 elaborates on our organization model . Section 4 then explores a method for applying our resultant encyclopedia to NLP research , specifically , question answering . Section 5 performs a number of experiments to evaluate our methods . The World Wide Web has been an unprecedentedly enormous information source , from which a number of language processing methods have been explored to extract , retrieve and discover various types of information . In this paper , we aimed at generating encyclopedic knowledge , which is valuable for many applications including human usage and natural language understanding . For this purpose , we reformalized an existing Web-based extraction method , and proposed a new statistical organization model to improve the quality of extracted data . Given a term for which encyclopedic knowledge ( i.e. , descriptions ) is to be generated , our method sequentially performs a ) retrieval of Web pages contain-ing the term , b ) extraction of page fragments describing the term , and c ) organizing extracted descriptions based on domains ( and consequently word senses ) . In addition , we proposed a question answering system , which answers interrogative questions associated with what , by using a Web-based encyclopedia as a knowledge base . For the purpose of evaluation , we used as test inputs technical terms collected from the Class II IT engineers examination , and found that the encyclopedia generated through our method was of operational quality and quantity . We also used test questions from the Class II examination , and evaluated the Web-based encyclopedia in terms of question answering . We found that our Webbased encyclopedia improved the system coverage obtained solely with an existing dictionary . In addition , when we used both resources , the performance was further improved . Future work would include generating information associated with more complex interrogations , such as ones related to how and why , so as to enhance Webbased natural language understanding .

Challenge: Existing methods that extract encyclopedic knowledge from the Web output unorganized clusters of term descriptions not necessarily related to explicit criteria while clustering is performed.
Approach: They propose to use word senses and domains for organizing extracted term descriptions on questions extracted from the Web to improve the quality.
Outcome: The generated encyclopedia is applied to a Japanese question and answering system and improves over a system which solely depends on a dictionary.