WebCrawler uses Excite's technology to index
web pages. Although they mainly index index pages of a site, WebCrawler will also crawl deeper into some sites, particularly those with high "link popularity" which means other sites which have links to yours.
WebCrawler indexes each word on each page up to 1 megabyte of text.
The description on their results page will be the keywords under which your page is found.
WebCrawler suggests that you use a title which truly
describes your page. Their relevance alogrithm gives more weight to titles than to body text. If you use "Welcome to My Site" or "So-and-So's Homepage" as a title, it will get buried with the other
sites with "Welcome to" and "Homepage" in the title.
Describe as fully as possible what the site is about. Use as many of your keywords as possible as close to the top of the document as possible.
Embedded user IDs in the URL such as the ones generally used in many shopping cart sites are not indexed, nor does the spider recognize CGI directories, so pages generated on the fly are not indexed.
Your best bet
for WebCrawler is to set up a static HTML page as the front door of your site and submit that one
