When and How Will Google Index My Website?

Daniel Scocco

questions and answersThis post is part of the weekly Q&A section. Just use the contact form if you want to submit a question.

Rick Regan asks:

I have a two-part question about how Google indexes sites:

1) Is Google supposed to index an entire site at once or does it do so incrementally? My blog (on my own domain) appears to be getting indexed incrementally, to the point where it’s taken a month to index all 30+ of my pages. Most of those pages were present before I manually submitted my URL to Google. I’m wondering if I’m doing something wrong or not doing something I should be doing. Does this have anything to do with my blog being new, or having no external links pointing to it?

2) Does Google eventually drop noindex and 404 pages? I have archive pages that got indexed but I have since added to them the “noindex” tag. I also deleted an empty category that Google now gets a 404 on. Will Google eventually remove those pages as it re-crawls my site?

I will answer to each question individually.

1) Most of the times Google will index a new website gradually, yes. That is at least what I have observed with most of my websites. The speed at which Google will index all your internal pages will depend on different factors though.

If you get some very trusted and relevant backlinks, and on top of that you also have a very efficient internal link structure, all your internal pages will get indexed fast. If, on the other hand, you have very few backlinks and a poor link structure, it might take a while before you get to see all your pages indexed.

Keep in mind that using the manual URL submission to Google will have a small impact upon the speed and breadth of your indexation. In fact many people recommend that if you want to get a site indexed fast you should NOT use that feature, and rather focus on getting some trusted backlinks to your site.

2) Yes, eventually Google will fix those issues. New sites don’t get crawled very often, so that is certainly the reason. As soon as Google finds out about the “noindex” tag, for example, it will remove those pages from their index.



Browse all articles on the Q & A category or check the recommended articles for you below:

124 Responses to “When and How Will Google Index My Website?”

  • nathan

    What does everyone mean by backlinks?
    I’m a newbie so give me some leeway

  • FourPx

    Hey,

    That’s a nice piece of info….

    Is there a specific time frame that Google follows for refreshing the Google’s search positioning… If yes, please let me know about it.

  • Terence Milbourn

    Thanks, guys, there’s some really helpful stuff here. I will try it out and report back what works for me.

  • medyum

    I also submit sitemaps to Google, Yahoo and Live.I also use Google Webmaster Tools. I have some anecdotal evidence that when you have errors on your site like duplicate titles and meta-descriptions that this seems to have an adverse affect on the pages that are indexed.

  • saeedbhatti

    since last one year i build this web site,and also verified by webmaster,after this i can’t understand after the expiry one year my web is not working properly,being an affiliate with click bank,having nickname and with other sites,my web site makes not a single dollar,what is the short falls in my site i could not under stand.please help to make this site in full functioned

  • Melayu Boleh

    My site is already indexed by google. When I serach using site:mysiteurl, its ok, but when I google link:mysiteurl it does not show eventhough I have done alot of backlinking. Why?

  • Lowongan

    in this month i lost half of my google index and that was suck, i don’t know what happen …
    is google filtering the site content? and deleting the duplicate index?

  • Karl

    Thanks alot, this article is definately worth taking note of.

    Socially bookmarked to bring awareness 🙂

    Karl

  • sachin

    yes dear this is true .
    Good backlinks from google trusted sites always will get your site indexed more quickly. We have had performed many experiements to find out what gives the best approach.

  • webfool

    Good backlinks from google trusted sites always will get your site indexed more quickly. We have had performed many experiements to find out what gives the best approach.
    (1) manual site submission – can be slow, google will typically send the bot round within day to check that the site has indexable content, and returning a 200.Then over a period of weeks gradually more pages are crawled and indexed.
    (2) Links on High PR / High volume sites ( we believe that google puts more priorty on traffic that is flowing to the site rather than page rank). Typically you get index inclusion after a couple of days.
    (3) Now here is the 24 carret nugget, Site registration period – pay a bit extra and get your domain registered for 10 years and you will get almost immediate indexing. Google have it in their patent – just think about it – with yearly domains around $3 for bulk registrations, google sandbox yearlings but mark 10 year regies as more likely to web sites with conviction behind them rather than spam / squeeze sites. We registered WebTrafficAgents.com and on the second day were at page 4 of 27Million results.

  • vikiraj

    Now Google put the new site in the “Sandbox”. Hence it takes much time to index. Better to use the XML sitemap for gradual indexing.

  • the english guy

    Putting a sitemap in Google’s webmaster tools also speeds up inclusion. Well worth the (short) time it takes to set up there.

  • Web Hosting

    YES,backlinks are very important to a new blog.
    Also you will be indexed quickly by adding blog to some social bookmarks.

  • Folly Beach Trip

    I agree with the trusted backlinks approach. It’s probably the fastest way to get your domain indexed. Aged domains are also good if you can get them.

  • love japan

    Daniel,
    i have a new blog and hope that google will index the domain soon before i put more post. Yes, i am working on link building. Thanks for your articles. Useful for me.
    Good 2 know you
    Aiko

  • Steve Hall

    Very Good Post.

    I have a number of websites that I have created and had indexed quite quickly.

    I use a few different methods to do this. Submit content to sites that Google loves like ezinearticles and digg.

    I also submit sitemaps to Google, Yahoo and Live.

    I also use Google Webmaster Tools. I have some anecdotal evidence that when you have errors on your site like duplicate titles and meta-descriptions that this seems to have an adverse affect on the pages that are indexed.

    You can also use Google Webmaster Tools to see how often Google crawls your site.

  • Rob O.

    On a somewhat related note, I’m really confounded about the disparity of the search results between Google, Yahoo, and Live (MSN). I submit sitemaps to each of these and they are crawling my site, but the return can be radically different.

    For example, if I use the image search feature of each to search for “site:www.2dolphins.com logo” I get:

    Live – 112 results

    Google – 72 results

    Yahoo – 22 results

    And the funny thing is that there’s very little overlap – not many of the results from Yahoo also show up in the Live image search results.

  • Eric

    Yes, #2 is usually just a matter of time before it all gets cleaned up. You just have to be patient because there’s nothing you can do to speed up the process. Consider it a lesson learned I guess!

  • loephie

    if we have used manual URL submission, what should we do in order to be better indexed by Google?

  • Rick Regan

    @Cristian: Webmaster tools says that tool is for “urgent” removal (mine was not urgent – just wanted to remove duplicate content). They recommend noindex or robots.txt otherwise.

    @Calvin: I have a similar problem. The number of URLs Webmaster tools says it’s indexed from my sitemap doesn’t match the URLs returned in a “site: ” search for my site. I don’t know why.

  • Ben

    Wouldn’t a site map help? If you loaded a sitemap xml from day one, wouldn’t that help Google easily find all your pages?

  • Calvin Loh

    Daniel,

    A number of my sites seem to have dropped off Google. Even when I use “site:mysubdomain.mydomain.com”, Google says it can’t find anything.

    But when I look at Google Webmaster Tools, it said my sites are in the index.

    Any idea what’s going on?

  • Nikhil

    Excellent article…
    Just DIGG it…..

  • Karl Staib – Work Happy Now

    I’m glad I stopped by today. I’ve always been curious about the “noindex” tag. I’ve been meaning to research this, but I’ve procrastinated this forever. Maybe I knew I would stop by Daily Blog Tips and find the answer. Thanks!

  • Cristian Graziano

    Rick: For question #2, Google Webmaster Tools now allows you to submit URLs for removal. I submitted a set and had them all removed within 4-5 days.

  • laptop lover

    I think the best way to let google index your sites quickly is submit your web pages to some social bookmark sites, like digg, mixx, reddit, and then you can get a lot of traffic.

  • Daniel Scocco

    @Matej, yeah social bookmarks that pass link juice are a good option to get indexed.

  • Rick Regan

    Daniel, thanks for answering my question.

    In the time that has passed since I asked the question Google did eventually index all my pages (well, all but one for some reason) . And yes, the noindexes and 404s eventually got removed as you said they would.

    I have a few backlinks now and I see from my stats that Google crawls my site more frequently now. Most of my posts seem to get indexed within an hour of posting.

  • aaron

    Good timing for this post.

    Last night I had a couple hours to kill so I registered http://www.ovechkin-goals.com, because “ovechkin goal” and “ovechkin goals” are popular search terms whenever the great NHL player scores a beauty.

    I’ve been reading your blog for months, but only my second or third comment. I got the site to where it is now from nothing in about 4 hours last night, and as I was finishing it up, I noticed you made the WordPress theme, so thanks for that.

    Now it’s a little experiment to see how quickly/high I can rank for those two search terms with minimal time commitment. Since the domain only costs $10-15 a year, I’ll be happy to get $10-$15/month from AdSense. I love watching hockey highlight videos anyways, and I only plan on spending about an hour a month adding videos unless it takes off more than expected.

    Aaron

  • Matej

    Usually my sites get indexed in a couple of hours. I usually submit my site on Digg with the exact keywords. It helps me to get traffic ( google-digg-my site ) and I get indexed really fast.

Comments are closed.