What Is a Sitemap? Do I Need It?

By Daniel Scocco

questions and answersThis post is part of the weekly Q&A section. Just use the contact form if you want to submit a question.

Rajasekharan asks:

I couldn’t find an XML sitemap in your blog. Do you have one? If no, why are you reluctant to include an XML sitemap?

First things first, what is a sitemap? According to Sitemaps.org:

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.

Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata. Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.

The key message is in the second paragraph: “Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data.”

Google and the other search engines rely mainly on their bots and web crawlers to discover new pages around the web. Sitemaps help them in some situations, but they are not compulsory nor necessarily beneficial.

Here is what Google itself says about Sitemaps:

Sitemaps are particularly helpful if:

  • Your site has pages that aren’t easily discovered by Googlebot during the crawl process – for example, pages featuring rich AJAX or Flash.
  • Your site is new and has few links to it. (Googlebot crawls the web by following links from one page to another, so if your site isn’t well linked, it may be hard for us to discover it.)
  • Your site has a large archive of content pages that are not well linked to each other, or are not linked at all.

Unless you fall into one of those cases, therefore, having a Sitemap will not be “particularly helpful” for your site.

In fact, some people claim that it might even be the opposite. There are cases where the addition of a Sitemap actually reduced the crawl rate of the website, possibly because Google no longer need to crawl the site completely to discover its internal pages.

On most of my sites I don’t use Sitemaps because I always try to have an efficient internal link structure in place. For example on Daily Blog tips every single page of the site is accessible within 2 clicks of the homepage.

If your website is having indexation problems, has a poor internal link structure, or falls in one of the three situations described by Google, then using a Sitemap would probably be a good idea.



Related Articles

Please install the YARPP plugin

Share

24 Responses to “What Is a Sitemap? Do I Need It?”

  • Paul

    I agree that sitemap is especially useful for new sites. I can tell this from my own experience. Anyways, google do take it into account, so if you want to improve your seo results it is a thing to do!

  • du hoc

    If you wanted to place that list of links into the look of your site you could “split” the HTML code of contractindex-results.html between a HEADER and README and upload those to the same directory. You split the code right after:
    Code:

    Everything after that point you would put in the README file and that and everything above it you would put in the HEADER file. Hope that makes sense to you.

  • Brian Christner

    Sitemaps are a huge help for not only Google but Bing and ask.com as well. With WordPress you can have plugins notify the search engines every time you publish a new post and update the sitemap at the same time. I found this very useful for some of my pages.

  • ravi

    I don’t have a sitemap myself, but I should get to work. Is a sitemap very important? Or just a little difference?

  • medyum

    Well, I’d like to say that the XML Sitemap is really very useful to a blog, I have used it for nearly one year, the most pages of my first blog are indexed by Google, all thanks by Google XML Sitemap plugin for WordPress.

  • The Medical Plus

    Building a good site map is also critical for web pages which use JavaScript menus, because most search engines cannot follow JavaScript links a Site Map will help those web pages to be indexed by the search engines.

  • Cristian

    That’s a great point about the sitemap possibly being a negative. I think there are three big benefits to a sitemap that I didn’t see mentioned here.

    1) The last modified date should encourage the googlebot to crawl the page – even for small pages – to at least see what changed.

    2) The sitemap priority to reduce the importance placed on less important pages (like a contact page) and also influence the links google will display in your authority listing (I think they call them sitelinks).

    3) Having a sitemap allows you to ping Google, Yahoo, and MSN when you’ve updated your site. They may not crawl your site immediately, but it should get them back crawling your site sooner than if you had waited for the bots to come on their own.

    Most good content management systems will take care of all three of these for you.

  • Rob

    I will always start a site with a sitemap now, but I have seen the negative effect to adding one with an older site.

  • diabetes man

    yes sitemap very important for search engine and also for visitor….I always use frr sitemap generator

  • Rajbir D

    I don’t have a sitemap myself, but I should get to work. Is a sitemap very important? Or just a little difference?

  • Skaizer

    I always make sitemap for my sites. I’t realy help crawlers with indexing my webpages.

  • Mark Harrison

    Using Google sitemaps is vastly underrated but it is still the quickest way to get your site indexed by Google. One tip…if your update is set to monthly for a sitemap update, you can also resubmit manually every time you write a new post/article.

  • zyva

    I am considering that site map is very important for our site, to make web crawlers easily crawl our site.

  • Daniel Scocco

    @Clarky, I am not sure if your pages are indexed by Google thanks for the Sitemap or thanks for the backlinks that your site has. Probably the latter 🙂 .

  • Clarky

    Well, I’d like to say that the XML Sitemap is really very useful to a blog, I have used it for nearly one year, the most pages of my first blog are indexed by Google, all thanks by Google XML Sitemap plugin for WordPress.

  • Destination Infinity

    Oh thanks for the info, as long as I don’t have to do something like this, I am happy 🙂 But looking at the way search engines have brought traffic, I am amazed at the terms that they catch. I guess they go well into the post (much beyond the tags) and crawl the entire content and bookmark them to recommend to the readers. There is one more thing I have noticed, If I write a story and immediately write something on two mobile phones, still the search engines know how to differentiate between the two posts and whom to refer what. That’s amazing!

    Destination Infinity

  • Sheila Atwood

    I got that it is important to make sure that you create your links as you go so that you create an internal map.

    I had not even started to look at what a site map is. Now I know it should be one of the important first steps for a new blog.

    Thank you
    Sheila

  • Denis

    Having a sitemap has one positive side effect. Actually the sitemap should be submitted to Google Webmaster Tool.

    Here is a post from John Mueller (Googler) about hard-to-detect website exploits that silently redirect Googlebot to third-party sites.
    http://johnmu.com/pagerank-hacker-using-suomi-co-in/
    “There’s another way to spot this kind of hack with Google Webmaster Tools: When you submit a Sitemap file, Google will show warnings for URLs that redirect. By design, you should be listing the final URL in your Sitemap file, so if the URL is redirecting for our crawlers (as in this case), we’ll show a warning in your account.”

    And when I talked with a webmaster of one of the hacked sites (who lost his PR and search engine positions:
    http://oldtenspeedgallery.com/other-things/otsg-hacked/), it appeared that he actually saw the warnings but didn’t know how to interpret them.

  • SEO Tips

    Excellent post Daniel you definitely brought to my attention what could be a negative point of having a sitemap. Its tempting to actually get rid of my sitemap now however I think I shall keep it for now.

  • Daniel Scocco

    @Rajesekharan, we would to know what was happening on the site when that re-submission was carried out. For example, they could have been having some crawling issues.

  • Rajasekharan

    Thanks dude for picking that question up.

    I removed my sitemap also just for the reasons you said and after noticing that you don’t use XML sitemaps.

    However, I recently heard tweets from Jason Calacanis that soon after he “resubmitted” sitemap of Mahalo to Google, he had heavy traffic to knock his server down couple of times! (Read it here: http://tinyurl.com/bfgyqj) Strange coincidence that you picked this question soon after that.

    Note the word “resubmitted.” Strange! Mysterious! Or any other hidden reason?

  • Chung Bey Luen

    It is no harm to include a sitemap. Just in case Google couldn’t crawl your content due to some issue, sitemap will help.

    And we can another kind of HTML page sitemap which is useful to human being.

  • bansama

    I use a sitemap for second reason listed by Google. And it certainly has helped. However, from personal experience I also think it can be useful to offer a sitemap formatted more for human readers too (although I still haven’t had time to make one for my site).

  • Jaan Kanellis

    I still feel is you have a relatively new website or one that just doesn’t have many back links you should be using a site map.

    Another one of the best uses for it is during a redesign. making sure that one is in place before and after so the bots can see exactly what is happening with your website.

Comments are closed.