Robert Steele

Techniques for Specialized Search Engines

Robert Steele
University of Technology, Sydney

Abstract

It is emerging that it is very difficult for the major search engines to provide a comprehensive and up-to-date search service of the Web. Even the largest search engines index only a small proportion of static Web pages and do not search the Web's backend databases that are estimated to be 500 times larger than the static Web. The scale of such searching introduces both technical and economic problems. What is more, in many cases users are not able to retrieve the information they desire because of the simple and generic search interface provided by the major search engines. A necessary response to these search problems is the creation of specialized search engines. These search engines search just for information in a particular topic or category on the Web. Such search engines will have smaller and more manageable indexes and have a powerful domain-specific search interface. This paper discusses the issues in this area and gives an overview of the techniques for building specialized search engines.