You build your website and want potential customers to see it. In order to be seen, your website will have already been crawled and indexed by the search engines spiders. Once the search engines have your information cached/stored from your website, they rank it based on relevance and authority and match it to the appropriate search queries.
Wait! Did you say Cash?
No, I said cache! When a search engine crawls your website, it captures the information 
and stores all of the text and links for later. When a person types a query into the search engine field, it goes back to that cached info and based on the relevance and authority of your information with that search and some algorithm mumbo jumbo determines the best information to provide to the searcher.
Cache and SEO
Remember that I said that your information will be cached for later? This is a big part of why search engine optimization takes some time to see results because what you may have changed or added onto your website today may not have gotten indexed for a search query that is run two weeks, a month or more from now.
What do Spiders look for?
They collect information from your website such as your url, the title, meta tags, the content on the page and even links on the page. It takes it to the search engine and is cached until needed by the search engine to provide that information back to the search engine results page when someone enters a search.
Search engines come and crawl on a website that is fresh with content and easy to crawl. You don’t typically change the main pages of your website often. However, your base content should have a once over at least once every 6 months to a year depending on the size of your website and the meta tags should be revised based on the changes in your industry and target market. With that being said, the next best thing is having a blog on your website. This is a great way to provide further value to your consumers but also providing that fresh updated content to keep the search engine spiders coming back to crawl and index your site.
Keep in mind….
The search engines spiders can’t get to every page on the web, let alone every day so they find websites with heavy traffic, are more popular websites (links to them), have the best information and are the most accessible.
The search engine spiders crawl the internet via links so if your website has login information or requires you to type something in an internal search box, know that the spiders won’t be crawling there. This goes on to prove how important internal links are for SEO as well as the importance of having a sitemap.
A sitemap is basically the hierarchy, navigation, but in links of your website all on one page. This surely would make it easier to navigate through all of your pages on your website and provide less chances of the spider going down a black hole of no return. See the value of having a sitemap? 
Spider Traps are Black Holes (of no return)
It is important to be sure your website loads quickly, that there isn’t any script, flash or other code that is slowing things down or getting in the way of the search engines crawling. Be sure that all of your links are working; each page has one URL and even be sure that you have properly working redirects.
One last thing on this is to be sure you have your robots.txt files set up correctly. Real quick (glazing over yet?) a robots.txt file simply tells the spiders which pages you don’t want them to crawl and the meta robots tag tells search engines not to index a page or follow certain links.
How many pages do you have indexed?
You can check to see how your website is doing and how many pages that a search engine has indexed by searching on a specific search engine with this – site:yoursite.com – like mine would be site:avisualbusiness.com. Once you find out the number of pages that are indexed, verify that is the total number of pages on your site? This is helpful to see what pages are missing out on being indexed and shown to searches and figure out why and how to get them indexed.
I hope that you have found this information to be helpful and informative. If you would like further information on search engine optimization services or maybe schedule a 1-on-1 consultation session so you can Do It Yourself, I would be happy to provide further information and/or pricing for you. Contact me and I look forward to your comments, questions and any help I may provide to see your business grow.
~Kristen
Kristen, that’s a great explanation of how the critters handle websites. Very useful information for people setting up their business sites.
Thanks Beth.. glad it was helpful for you and easier to understand.
OH yes fresh content is the key for those spiders and seo rankings! Thanks for all the tips!
You are welcome and thank you!
Did you read my min? Spent 1 hour yesterday trying to explain to a friend about spiders and crawlers. I’m sending her this excellent explanation and I pinned it for reference. Thank you. So easy to understand this process now.
Actually Roslyn, I do believe I did! LOL Thanks for sharing the post and hope it helps explain it to her better. Glad to help, thanks!
Woo-hoo! 443 pages indexed on my site. I have no idea if that’s right or not, but it’s one heck of a lot more than I have in my menu! Super information, girlfriend. Thanks for sharing.
Wow, that IS great Jackie because I know you don’t have but less than 25 pages, ish… so that 443 is due to your white papers, blogs and such. Just think of the additional reach your blogs have given you!
Loved this blog post!! I checked my site and glad to report there are pages and pages indexed. Lots of my pages and tons of my products.
That’s awesome Sonya.. I wonder how many are indexed on Google.com versus Google.uk? Glad you enjoyed it and I look forward to working with you.
Thank you for the information. I am still learning how all this work and what is the proper way to get her site out there. Any information is great information!
Great information! Thanks for sharing!
I can actually imagine spiders crawling on the web while I’m reading your article. =) Well, putting that one aside, I specifically like your suggestion on using site:yoursite.com in tracking the website’s performance.
Thanks Edmund… hoping I didn’t give you the heeby jeebies! LOL Glad you found a nugget of helpful info!