Searching the Invisible Web
© 2000 Walt Howe and Hope Tillman
(last update 27 May 2006)
While the major search engines are steadily growing in size, the Internet and the web are growing even faster. Major stores of information on the web are never found by the search engines. Some of this is on Intranets, which are not open to the public. Some of it is deep in web sites below the levels which search engines typically search and index. Still more is contained, not in web pages, but in databases that are served up as web pages on the fly when someone requests information. delphiforums.com message boards are an example of this. Their content is searchable by delphiforums.com's own internal search engine, but not to the web as a whole. These large databases, searchable with their own tools, but not to the major search engines, and the other generally unreachable sources of information have come to be known as the Invisible Web.
Until a few years ago, files prepared in Adobe PDF format were not searchable by the major engines. Google has added this capability. These are typically documents that live in print form, also, and the PDF format provides a format that looks just like the printed page when viewed onscreen.
If the major search engines cannot reach the information in databases, it does not mean that you cannot. If you realize that such databases exist, the first step in the problem is to find the databases, not the information itself. Once you locate a database, you can use its own search tools to find the information. There are several good collections of these databases you can use as a starting point:
- FindIt, a short, select list of frequently used specialized searches and databases.
- Direct Search, a similar collection maintained by Gary Price
- Price's List of Lists, a collection started by Gary Price of lists of information rather than databases of information.
- CompletePlanet, a large index of databases and search engines. It can access the "Deep Web," which ordinary search engines cannot reach, using its LexiBot technology.
If you cannnot find what you are looking for in one of these collections, try using the major search engines to search for databases of the general type of information for which you are looking. Search for the word "database" and general terms which would describe the whole list, rather than the specific information you are looking for.
Questions or comments? Visit the Navigating the Net Forum Message Board or send e-mail to Walt Howe
Return to SearchTips