Crawl on Over to My Website

You build your website and want potential customers to see it.  In order to be seen, your website will have already been crawled and indexed by the search engines spiders.  Once the search engines have your information cached/stored from your website, they rank it based on relevance and authority and match it to the appropriate search queries.

Wait! Did you say Cash?

No, I said cache! When a search engine crawls your website, it captures the information and stores all of the text and links for later.  When a person types a query into the search engine field, it goes back to that cached info and based on the relevance and authority of your information with that search and some algorithm mumbo jumbo determines the best information to provide to the searcher.

Cache and SEO

Remember that I said that your information will be cached for later?  This is a big part of why search engine optimization takes some time to see results because what you may have changed or added onto your website today may not have gotten indexed for a search query that is run two weeks, a month or more from now.

What do Spiders look for?      

They collect information from your website such as your url, the title, meta tags, the content on the page and even links on the page.  It takes it to the search engine and is cached until needed by the search engine to provide that information back to the search engine results page when someone enters a search.

Keep them Spiders Crawling

Search engines come and crawl on a website that is fresh with content and easy to crawl.  You don’t typically change the main pages of your website often.  However, your base content should have a once over at least once every 6 months to a year depending on the size of your website and the meta tags should be revised based on the changes in your industry and target market.  With that being said, the next best thing is having a blog on your website.  This is a great way to provide further value to your consumers but also providing that fresh updated content to keep the search engine spiders coming back to crawl and index your site.

Keep in mind….

The search engines spiders can’t get to every page on the web, let alone every day so they find websites with heavy traffic, are more popular websites (links to them), have the best information and are the most accessible.

The search engine spiders crawl the internet via links so if your website has login information or requires you to type something in an internal search box, know that the spiders won’t be crawling there.  This goes on to prove how important internal links are for SEO as well as the importance of having a sitemap.

A sitemap is basically the hierarchy, navigation, but in links of your website all on one page.  This surely would make it easier to navigate through all of your pages on your website and provide less chances of the spider going down a black hole of no return.  See the value of having a sitemap?    

Spider Traps are Black Holes (of no return)

It is important to be sure your website loads quickly, that there isn’t any script, flash or other code that is slowing things down or getting in the way of the search engines crawling.  Be sure that all of your links are working; each page has one URL and even be sure that you have properly working redirects.

One last thing on this is to be sure you have your robots.txt files set up correctly.  Real quick (glazing over yet?) a robots.txt file simply tells the spiders which pages you don’t want them to crawl and the meta robots tag tells search engines not to index a page or follow certain links.

How many pages do you have indexed?

You can check to see how your website is doing and how many pages that a search engine has indexed by searching on a specific search engine with this –  site:yoursite.com – like mine would be site:avisualbusiness.com. Once you find out the number of pages that are indexed, verify that is the total number of pages on your site? This is helpful to see what pages are missing out on being indexed and shown to searches and figure out why and how to get them indexed.

I hope that you have found this information to be helpful and informative. If you would like further information on search engine optimization services or maybe schedule a 1-on-1 consultation session so you can Do It Yourself, I would be happy to provide further information and/or pricing for you.  Contact me and  I look forward to your comments, questions and any help I may provide to see your business grow.

~Kristen