How Do Spiders Work?
Search engines use spiders or crawlers to determine which content to include. Spiders crawl through your Web content and index relevant content. It is important to know how this is done so that you know how to design your site so that it will be optimized for inclusion.
First, you should know that spiders do not like slow-loading sites and will give up indexing a page or site if they encounter slow-loading Flash or graphics pages. Also, large graphics and Flash elements on your Web pages take up space that could be used for more textual content that would raise your site's rating with the search engines. That's a good reason to include fewer graphics and more relevant textual content on your Web site.
A common problem with the recent emergence of server-side functionality on Web sites is that if some pages on your site require session IDs/cookies or parameters to be passed from page to page, these pages will likely not be indexed because spiders cannot create sessions or cookies. Where server-side functionality is required, try to ensure that you have a few content pages that do not require session IDs and the like that will be accessed before a user logs in or places items into shopping carts.
Spiders take a lot of notice of the title of a Web page and use it as the title in the listing of your site on the search engine's listing page. Make sure that you use the title tag for each Web page and give each page a meaningful name, using words that your customers might be looking for in search engines. Another tip is to use capital letters in the titles of your pages: Search engines rank words containing capital letters higher than those in lower case.
The final thing to be aware of is that, when linking to other pages within your site, it is important not to use images or JavaScript to generate the links if you want these links to be "spidered." Spiders and robots cannot traverse these types of links. It is also a good ploy to use textual descriptive text within the <a> tags of any link so that the spider can use that to determine the relevance of the page linked to.