• Home
  • Why a Website?
  • What Is SEO?
    • Social Media
      • Social Audit
  • What is PPC?
  • Tools
    • Referrals
  • Blog
  • Contact
    • About AVB
    • Podcasts & Guest Spots
    • Schedule w/ Me
    • Testimonials
      • Leave a Testimonial

A Visual Business

You are here: Home / blog / Crawl on Over to My Website

Crawl on Over to My Website

in blog, search engine optimization on 09/16/14

search engines crawl your website for content, links & meta tagsYou build your website and want potential customers to see it.  In order to be seen, your website will have already been crawled and indexed by the search engines spiders.  Once the search engines have your information cached/stored from your website, they rank it based on relevance and authority and match it to the appropriate search queries.

Wait! Did you say Cash?

No, I said cache! When a search engine crawls your website, it captures the information cache, not cash is a way the search engines store your website infoand stores all of the text and links for later.  When a person types a query into the search engine field, it goes back to that cached info and based on the relevance and authority of your information with that search and some algorithm mumbo jumbo determines the best information to provide to the searcher.

Cache and SEO

Remember that I said that your information will be cached for later?  This is a big part of why search engine optimization takes some time to see results because what you may have changed or added onto your website today may not have gotten indexed for a search query that is run two weeks, a month or more from now.

What do Spiders look for?      

They collect information from your website such as your url, the title, meta tags, the content on the page and even links on the page.  It takes it to the search engine and is cached until needed by the search engine to provide that information back to the search engine results page when someone enters a search.

Keep them Spiders Crawling

spiders crawl optimized websites bestSearch engines come and crawl on a website that is fresh with content and easy to crawl.  You don’t typically change the main pages of your website often.  However, your base content should have a once over at least once every 6 months to a year depending on the size of your website and the meta tags should be revised based on the changes in your industry and target market.  With that being said, the next best thing is having a blog on your website.  This is a great way to provide further value to your consumers but also providing that fresh updated content to keep the search engine spiders coming back to crawl and index your site.

Keep in mind….

The search engines spiders can’t get to every page on the web, let alone every day so they find websites with heavy traffic, are more popular websites (links to them), have the best information and are the most accessible.

The search engine spiders crawl the internet via links so if your website has login information or requires you to type something in an internal search box, know that the spiders won’t be crawling there.  This goes on to prove how important internal links are for SEO as well as the importance of having a sitemap.

A sitemap is basically the hierarchy, navigation, but in links of your website all on one page.  This surely would make it easier to navigate through all of your pages on your website and provide less chances of the spider going down a black hole of no return.  See the value of having a sitemap?    spiders get trapped and sent into a black hole

Spider Traps are Black Holes (of no return)

It is important to be sure your website loads quickly, that there isn’t any script, flash or other code that is slowing things down or getting in the way of the search engines crawling.  Be sure that all of your links are working; each page has one URL and even be sure that you have properly working redirects.

use robot.txt files to command the spiders where not to goOne last thing on this is to be sure you have your robots.txt files set up correctly.  Real quick (glazing over yet?) a robots.txt file simply tells the spiders which pages you don’t want them to crawl and the meta robots tag tells search engines not to index a page or follow certain links.

How many pages do you have indexed?

You can check to see how your website is doing and how many pages that a search engine has indexed by searching on a specific search engine with this –  site:yoursite.com – like mine would be site:avisualbusiness.com. Once you find out the number of pages that are indexed, verify that is the total number of pages on your site? This is helpful to see what pages are missing out on being indexed and shown to searches and figure out why and how to get them indexed.

I hope that you have found this information to be helpful and informative. If you would like further information on search engine optimization services or maybe schedule a 1-on-1 consultation session so you can Do It Yourself, I would be happy to provide further information and/or pricing for you.  Contact me and  I look forward to your comments, questions and any help I may provide to see your business grow.

~Kristen

14 Comments

About Kristen Day

Kristen is a small business owner and thus dedicated to other small business owners by helping them improve their online presence with having a professional and mobile responsive website, then implementing search engine optimization and pay per click advertising all of which are necessary 'evils' in the digital marketing sphere but enables small business owners the opportunity to compete with larger companies. When not in the digital arena, you can find her hanging with her girlfriends, husband, drinking a margarita or shooting competition (not in that order).

Subscribe for Your Free Report!

10 INSANELY Easy SEO Tips YOU Can Do Right Now

« How do you Rank?
The Skinny on SEO »

Comments

  1. Beth Niebuhr says

    09/17/2014 at 9:16 AM

    Kristen, that’s a great explanation of how the critters handle websites. Very useful information for people setting up their business sites.

    Reply
    • Kristen Wilson says

      09/17/2014 at 5:47 PM

      Thanks Beth.. glad it was helpful for you and easier to understand.

      Reply
  2. Danielle Brogna says

    09/17/2014 at 10:06 AM

    OH yes fresh content is the key for those spiders and seo rankings! Thanks for all the tips!

    Reply
    • Kristen Wilson says

      09/17/2014 at 5:47 PM

      You are welcome and thank you!

      Reply
  3. Roslyn Tanner Evans says

    09/17/2014 at 10:24 AM

    Did you read my min? Spent 1 hour yesterday trying to explain to a friend about spiders and crawlers. I’m sending her this excellent explanation and I pinned it for reference. Thank you. So easy to understand this process now.

    Reply
    • Kristen Wilson says

      09/17/2014 at 5:46 PM

      Actually Roslyn, I do believe I did! LOL Thanks for sharing the post and hope it helps explain it to her better. Glad to help, thanks!

      Reply
  4. Jackie Harder says

    09/17/2014 at 4:08 PM

    Woo-hoo! 443 pages indexed on my site. I have no idea if that’s right or not, but it’s one heck of a lot more than I have in my menu! Super information, girlfriend. Thanks for sharing.

    Reply
    • Kristen Wilson says

      09/17/2014 at 5:46 PM

      Wow, that IS great Jackie because I know you don’t have but less than 25 pages, ish… so that 443 is due to your white papers, blogs and such. Just think of the additional reach your blogs have given you!

      Reply
  5. SonYa says

    09/17/2014 at 4:21 PM

    Loved this blog post!! I checked my site and glad to report there are pages and pages indexed. Lots of my pages and tons of my products.

    Reply
    • Kristen Wilson says

      09/17/2014 at 5:45 PM

      That’s awesome Sonya.. I wonder how many are indexed on Google.com versus Google.uk? Glad you enjoyed it and I look forward to working with you.

      Reply
  6. Yakini Pierce says

    09/17/2014 at 7:00 PM

    Thank you for the information. I am still learning how all this work and what is the proper way to get her site out there. Any information is great information!

    Reply
  7. michael kawula says

    09/18/2014 at 7:41 AM

    Great information! Thanks for sharing!

    Reply
  8. Edmund S. Lee says

    09/18/2014 at 4:08 PM

    I can actually imagine spiders crawling on the web while I’m reading your article. =) Well, putting that one aside, I specifically like your suggestion on using site:yoursite.com in tracking the website’s performance.

    Reply
    • Kristen Wilson says

      09/24/2014 at 10:02 AM

      Thanks Edmund… hoping I didn’t give you the heeby jeebies! LOL Glad you found a nugget of helpful info!

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search My Site

AVB Blog Categories

4 FREE SEO TIPS YOU CAN DO

Get 4 Easy Peasy SEO Tips you can do right now!

Managed Hosting for your WordPress Website

Siteground Hosting

Back Up your Business Data

Back Blaze backups

Join My Mailing List!

Get updates on SEO, PPC, website readability and small business in our monthly email! (double opt in so look for the email)

Don't forget to look for the confirmation email to double opt in for my email list.

Buy SEO Title & Meta Description eBook

Learn how to optimize your SEO Title & Meta Description

Known for SEO

Top Seo Agency in League City

Known for PPC Advertising

Top Pay-Per-Click (PPC) Agency in League City

Follow AVB!

A Visual Business on FacebookFollow Kristen on TwitterFollow AVB on Pinterest
Connect with Kristen on Linked In Follow AVB on Instagram

Learn About My Other Business

WTF Kristen

Managed Hosting for your WordPress Website

Create Images Quickly & Easily w/ Get Stencil

Get Stencil

Stock Photo Subscription Service

Depositphotos

Learn About My Other Business

Copyright © 2023 · A Visual Business