When and How Will Google Index My Website?

by Daniel Scocco

questions and answers This post is part of the weekly Q&A section. Just use the contact form if you want to submit a question.

Rick Regan asks:

I have a two-part question about how Google indexes sites:

1) Is Google supposed to index an entire site at once or does it do so incrementally? My blog (on my own domain) appears to be getting indexed incrementally, to the point where it’s taken a month to index all 30+ of my pages. Most of those pages were present before I manually submitted my URL to Google. I’m wondering if I’m doing something wrong or not doing something I should be doing. Does this have anything to do with my blog being new, or having no external links pointing to it?

2) Does Google eventually drop noindex and 404 pages? I have archive pages that got indexed but I have since added to them the “noindex” tag. I also deleted an empty category that Google now gets a 404 on. Will Google eventually remove those pages as it re-crawls my site?

I will answer to each question individually.

1) Most of the times Google will index a new website gradually, yes. That is at least what I have observed with most of my websites. The speed at which Google will index all your internal pages will depend on different factors though.

If you get some very trusted and relevant backlinks, and on top of that you also have a very efficient internal link structure, all your internal pages will get indexed fast. If, on the other hand, you have very few backlinks and a poor link structure, it might take a while before you get to see all your pages indexed.

Keep in mind that using the manual URL submission to Google will have a small impact upon the speed and breadth of your indexation. In fact many people recommend that if you want to get a site indexed fast you should NOT use that feature, and rather focus on getting some trusted backlinks to your site.

2) Yes, eventually Google will fix those issues. New sites don’t get crawled very often, so that is certainly the reason. As soon as Google finds out about the “noindex” tag, for example, it will remove those pages from their index.

Got Your Free eBook?

Subscribe to the Daily Blog Tips newsletter and you will be able to download the "Make Money Blogging" eBook for free.
You will also receive tips to improve your blog, strategies to make money and useful resources from around the web.

69 Responses to “When and How Will Google Index My Website?”

Matej on March 24th, 2009 4:30 pm

Usually my sites get indexed in a couple of hours. I usually submit my site on Digg with the exact keywords. It helps me to get traffic ( google-digg-my site ) and I get indexed really fast.
aaron on March 24th, 2009 4:58 pm

Good timing for this post.

Last night I had a couple hours to kill so I registered http://www.ovechkin-goals.com, because “ovechkin goal” and “ovechkin goals” are popular search terms whenever the great NHL player scores a beauty.

I’ve been reading your blog for months, but only my second or third comment. I got the site to where it is now from nothing in about 4 hours last night, and as I was finishing it up, I noticed you made the Wordpress theme, so thanks for that.

Now it’s a little experiment to see how quickly/high I can rank for those two search terms with minimal time commitment. Since the domain only costs $10-15 a year, I’ll be happy to get $10-$15/month from AdSense. I love watching hockey highlight videos anyways, and I only plan on spending about an hour a month adding videos unless it takes off more than expected.

Aaron
Rick Regan on March 24th, 2009 5:07 pm

Daniel, thanks for answering my question.

In the time that has passed since I asked the question Google did eventually index all my pages (well, all but one for some reason) . And yes, the noindexes and 404s eventually got removed as you said they would.

I have a few backlinks now and I see from my stats that Google crawls my site more frequently now. Most of my posts seem to get indexed within an hour of posting.
Daniel Scocco on March 24th, 2009 9:50 pm

@Matej, yeah social bookmarks that pass link juice are a good option to get indexed.
laptop lover on March 24th, 2009 10:04 pm

I think the best way to let google index your sites quickly is submit your web pages to some social bookmark sites, like digg, mixx, reddit, and then you can get a lot of traffic.
Cristian Graziano on March 25th, 2009 12:49 am

Rick: For question #2, Google Webmaster Tools now allows you to submit URLs for removal. I submitted a set and had them all removed within 4-5 days.
Karl Staib - Work Happy Now on March 25th, 2009 1:26 am

I’m glad I stopped by today. I’ve always been curious about the “noindex” tag. I’ve been meaning to research this, but I’ve procrastinated this forever. Maybe I knew I would stop by Daily Blog Tips and find the answer. Thanks!
Nikhil on March 25th, 2009 4:25 am

Excellent article…
Just DIGG it…..
Calvin Loh on March 25th, 2009 7:24 am

Daniel,

A number of my sites seem to have dropped off Google. Even when I use “site:mysubdomain.mydomain.com”, Google says it can’t find anything.

But when I look at Google Webmaster Tools, it said my sites are in the index.

Any idea what’s going on?
Ben on March 25th, 2009 10:48 am

Wouldn’t a site map help? If you loaded a sitemap xml from day one, wouldn’t that help Google easily find all your pages?
Rick Regan on March 25th, 2009 10:57 am

@Cristian: Webmaster tools says that tool is for “urgent” removal (mine was not urgent – just wanted to remove duplicate content). They recommend noindex or robots.txt otherwise.

@Calvin: I have a similar problem. The number of URLs Webmaster tools says it’s indexed from my sitemap doesn’t match the URLs returned in a “site: ” search for my site. I don’t know why.
loephie on March 25th, 2009 1:16 pm

if we have used manual URL submission, what should we do in order to be better indexed by Google?
Eric on March 25th, 2009 2:45 pm

Yes, #2 is usually just a matter of time before it all gets cleaned up. You just have to be patient because there’s nothing you can do to speed up the process. Consider it a lesson learned I guess!
Rob O. on March 25th, 2009 7:15 pm

On a somewhat related note, I’m really confounded about the disparity of the search results between Google, Yahoo, and Live (MSN). I submit sitemaps to each of these and they are crawling my site, but the return can be radically different.

For example, if I use the image search feature of each to search for “site:www.2dolphins.com logo” I get:

Live – 112 results

Google – 72 results

Yahoo – 22 results

And the funny thing is that there’s very little overlap – not many of the results from Yahoo also show up in the Live image search results.
Steve Hall on March 26th, 2009 7:44 am

Very Good Post.

I have a number of websites that I have created and had indexed quite quickly.

I use a few different methods to do this. Submit content to sites that Google loves like ezinearticles and digg.

I also submit sitemaps to Google, Yahoo and Live.

I also use Google Webmaster Tools. I have some anecdotal evidence that when you have errors on your site like duplicate titles and meta-descriptions that this seems to have an adverse affect on the pages that are indexed.

You can also use Google Webmaster Tools to see how often Google crawls your site.
love japan on March 26th, 2009 5:27 pm

Daniel,
i have a new blog and hope that google will index the domain soon before i put more post. Yes, i am working on link building. Thanks for your articles. Useful for me.
Good 2 know you
Aiko
Folly Beach Trip on March 30th, 2009 9:42 pm

I agree with the trusted backlinks approach. It’s probably the fastest way to get your domain indexed. Aged domains are also good if you can get them.
Web Hosting on March 31st, 2009 7:46 am

YES,backlinks are very important to a new blog.
Also you will be indexed quickly by adding blog to some social bookmarks.
the english guy on March 31st, 2009 6:43 pm

Putting a sitemap in Google’s webmaster tools also speeds up inclusion. Well worth the (short) time it takes to set up there.
vikiraj on April 1st, 2009 5:19 am

Now Google put the new site in the “Sandbox”. Hence it takes much time to index. Better to use the XML sitemap for gradual indexing.
webfool on April 4th, 2009 6:27 am

Good backlinks from google trusted sites always will get your site indexed more quickly. We have had performed many experiements to find out what gives the best approach.
(1) manual site submission – can be slow, google will typically send the bot round within day to check that the site has indexable content, and returning a 200.Then over a period of weeks gradually more pages are crawled and indexed.
(2) Links on High PR / High volume sites ( we believe that google puts more priorty on traffic that is flowing to the site rather than page rank). Typically you get index inclusion after a couple of days.
(3) Now here is the 24 carret nugget, Site registration period – pay a bit extra and get your domain registered for 10 years and you will get almost immediate indexing. Google have it in their patent – just think about it – with yearly domains around $3 for bulk registrations, google sandbox yearlings but mark 10 year regies as more likely to web sites with conviction behind them rather than spam / squeeze sites. We registered WebTrafficAgents.com and on the second day were at page 4 of 27Million results.
sachin on April 7th, 2009 10:45 am

yes dear this is true .
Good backlinks from google trusted sites always will get your site indexed more quickly. We have had performed many experiements to find out what gives the best approach.
Karl on April 28th, 2009 9:46 am

Thanks alot, this article is definately worth taking note of.

Socially bookmarked to bring awareness

Karl
Lowongan on May 9th, 2009 1:17 pm

in this month i lost half of my google index and that was suck, i don’t know what happen …
is google filtering the site content? and deleting the duplicate index?
Melayu Boleh on May 18th, 2009 1:20 am

My site is already indexed by google. When I serach using site:mysiteurl, its ok, but when I google link:mysiteurl it does not show eventhough I have done alot of backlinking. Why?
saeedbhatti on July 10th, 2009 10:12 am

since last one year i build this web site,and also verified by webmaster,after this i can’t understand after the expiry one year my web is not working properly,being an affiliate with click bank,having nickname and with other sites,my web site makes not a single dollar,what is the short falls in my site i could not under stand.please help to make this site in full functioned
medyum on July 17th, 2009 9:07 am

I also submit sitemaps to Google, Yahoo and Live.I also use Google Webmaster Tools. I have some anecdotal evidence that when you have errors on your site like duplicate titles and meta-descriptions that this seems to have an adverse affect on the pages that are indexed.
Terence Milbourn on August 5th, 2009 2:16 am

Thanks, guys, there’s some really helpful stuff here. I will try it out and report back what works for me.
FourPx on August 11th, 2009 4:15 am

Hey,

That’s a nice piece of info….

Is there a specific time frame that Google follows for refreshing the Google’s search positioning… If yes, please let me know about it.
nathan on August 31st, 2009 3:42 am

What does everyone mean by backlinks?
I’m a newbie so give me some leeway
aman on August 31st, 2009 8:49 am

Yes, Make sure you dont have major errors in your code, validate your html, link building will also help.
Edward on September 1st, 2009 11:40 pm

Backlinks means high traffic sites having a higher page rank than you linking to your site. Simply means they have your URL somewhere on their site pointing to you.
Edward on September 5th, 2009 11:58 pm

How do I find a list of my indexed URLs by Google?
Thanks.
GV Marketing on September 14th, 2009 12:01 pm

The idea of going with tools other than direct Google submission seems to make a lot of sense.
We are just starting out and will look into backlinking and possibly creating sitemaps for our sites.

GVMarketing
Tooey on November 7th, 2009 7:24 pm

I have set up several different sites and is often an annoyance to get them listed up quickly I agree with everyone else though the fastest way is backlinks usually asking a couple of friends to link to me in there blogs and such gets your page up a lot quicker especially if they are a regular blogger.
I’ve found submit xml sitemaps helps but I experimented with one of my domains and the backlinks had a much quicker effect.
Ed on November 9th, 2009 3:13 pm

Another way is to try to submit to as many web directories as possible.
morphusis on November 18th, 2009 7:37 am

My website markmystep.com got listed on the Google fairly quickly. But when I started developing the website and put it under maintenance, it somehow got lost and is not showing up on Google search.

I have put it online now and still it is not showing up on search results. As per your suggestions, I am now working of creating some backlinks. Hope this works out well.

Thanks for your post. If you could provide any other suggestion, which would speed up the process, that would be helpful.
Ed on November 18th, 2009 4:47 pm

Always better to not put it under maintenance for a while as you never know when the bots will arrive. Use a web directory submission software to get more backlinks.
Jamie on November 20th, 2009 4:11 am

I’m having the most frustrating time trying to get my site indexed! GRRrrrrr!

I have a 30 page website built on x site pro 2 full of unique content, i have at least 200 backlinks, I have my articles and videos popping up on the front page of google from sites like hub pages, squidoo, scribd, youtube, metacafe, dailymotion, veoh, go articles, article city, digg, reddit, stumbleupon…. and many more; however its been 3 days since i finished my site and published it to the web but its still not indexed! Grrrrrr!

I know its not indexed because when i use google page rank tracker it says its not indexed!! I have always used this tool and I know it works, so why so long this time google???

Does anyone have any idea why its taking so long??

You can check for yourself, pregnancy-central.com (make sure you include the hyphen!)
Ed on November 23rd, 2009 3:06 pm

It can take several days or weeks for Google to index your site. Have you submitted it to Google using an XML sitemap also?
Thanks.
strewais on November 24th, 2009 5:35 pm

good thoughts but I still have problem with Indexing my my website on google
Ed on November 24th, 2009 5:50 pm

Indexing can take a while as Google has billions of web pages to crawl. But submitting an XML sitemap does help as I have seen them download it in a few minutes after submitting it to them.
Paul G on December 1st, 2009 6:13 pm

Some great ideas here.

Googlebot visits one of my sites everyday but does not cache it very often. It seems that even though the bot is crawling the site they are not indexing the pages. Is this only done on the days that they cache the site?

Is this correct?

If anyone can help I would appreciate it.
Ed on December 2nd, 2009 2:56 pm

Only Google can answer that question for you Paul G. They index pages that are of interest to them. Depends on Keywords you use.
palahubog on December 9th, 2009 5:43 am

how do I get google to update the cached information on my website? I am talking about the description of my website found in the search results of google.
Paul G on December 9th, 2009 11:52 am

That is called your meta description. You can change that within your website controls. If you don’t google will just take part of your page. You have about 150 characters and its also a good idea to use your keyword in there.
Ed on December 9th, 2009 4:32 pm

If you use wordpress, the headers on the pages can be used as meta descriptions too.
C. John Menskil on December 11th, 2009 4:18 pm

@Edward I’m guessing you already figured this out, but there are a few ways to learn how many of your pages are indexed by Google. I’ll mention a couple:

1) In Google’s search page type in “site:www.domain.com” The results will show you how many pages have been indexed by Google.

2) Use websites such as Alexa and Web Grader to find out how many times Googles has indexed your website’s pages.
Ed on December 14th, 2009 7:02 pm

@JohnMenskil I know how to do that. I was not the one looking for that information anyway. I use Google Webmaster tools for that.
Tom on January 5th, 2010 4:45 am

I want Google to index my blog more frequently. My site has been indexed and I post new content every day. However the last time Google visited the site was 5 days ago.

I use the site:URL command to check in Google if my new content has been indexed. Not sure if there is another method.
Andrew Reynolds on January 5th, 2010 3:37 pm

That is probably the best way to check. If your site is new Google does not visit it so often.

Get some good quality links and that should help. If you keep adding content Google will return more often.
Ed on January 5th, 2010 4:28 pm

@Tom when you build your XML Sitemap, change the frequency of pages you update daily to “daily”. This way Google knows of it and will come to it on a daily basis. No guarantees though.
Ed on January 6th, 2010 3:17 pm

@Tom, getting quality backlinks like @Andrew mentioned is a good way too. It makes you more popular and so you might get more Google visits.
Jokomodo on February 6th, 2010 11:23 am

Does anyone else have a problem, that Google tend to index irrelevant pages like tags and categories (if we are talking about blogs) instead of actual pages?
Ed on February 6th, 2010 10:30 pm

They do that as there might be something in the URL or the page it connects to of interest to them and worth indexing.
Dave on February 12th, 2010 6:00 pm

For the first time, I am have a hard time getting indexed on Google.

I usually get indexed within just a few days. Both KalamazooRealEstate.com and KalamazooHomesForSale.com will not index.

They both were for sale and I did pay more than the standard rate.

Is this one of the reasons why they will not index?
Ed on February 15th, 2010 6:17 pm

What were for sale? You do not need to pay anything to get indexed. Have you submitted a sitemap to Google yet?
Dave on February 15th, 2010 8:19 pm

I do not pay to have a website indexed. I have had many websites indexed over the years, but this one will not index. I paid more than a normal reg. fee for the name and it has some history going back 6 or 7 years. This is one of three that will not index on Google.

I did receive a report that the Nameservers for the geoname.com that may be poisoning DNS caches. Replies from these nameservers contain bogus Referrals to other nameservers for top-level Zones. If the poisoning succeeds, future queries for names in the Zone Poisoned would go to one of the Referral nameservers instead of its truly authoritative server.

I do not know if this is the problem or what it all means.

As far as a site map, Google has to look at it first and I have a tracker and google analytics and they have not been on the site yet.
Ed on February 16th, 2010 7:03 pm

@Dave that seems to be the problem. I would call upi the hosting service to dig into this more.
Dave on February 21st, 2010 1:21 am

I called my hosting service and everything looked correct. I have several websites hosted with them and they all have indexed with no problems.

I am now using adwords for securing traffic. I have back links from high traffic sites and regular websites as well. We will have to wait it out and see what happens.

I still think it has something to do with having it for sale and information on the website listing it for sale.
Ed on February 21st, 2010 11:20 pm

@Dave having it for sale should not matter with it being indexed. They are two different issues.
Shaishav on February 23rd, 2010 11:27 am

I have my website hosted a week back. i have added the google code to my website. But when i search the google i cant find my website. I have added keywords also on my site. What else do i need to do to top it up on Google. Its a corporate website hence cant host it on other websites or cross link it. Please help.
Ed on February 23rd, 2010 5:59 pm

Where is your site hosted? What is the name? It can take anywhere from a day to a week or even a month for Google to index your site.
inupinu on March 6th, 2010 2:51 pm

Many times I observed that Google indexes pages almost instantly.
I am exploring what processes are involved in fastest google indexing
vivek on March 7th, 2010 10:23 am

One Question i want to ask you that when i post a new post in my blog and when i search my blog with blog name i shows only few old and some time aonly formal pages it never shows latest post on search engine can u suggest me what the matter i am as begnnier wnt some tips from u r side,
Ed on March 7th, 2010 10:39 pm

They normally do index it pretty quick. Only if there is a problem, it might take longer.
Juanita on March 15th, 2010 4:30 pm

Hi. Il ike your advice. I have my first blog up and running. But don’t know when google will index it. I hope it will be soon. Please visit and leave some comments and advice. Thanks
MrsD on March 18th, 2010 11:46 pm

Hi Juanita, don’t give up. If you already register yourself to Google webmaster tools, and google analytics, it will need a while. I also almost gave up, because i havent found any datas of my blog, but tonight, Google finally found me
Ed on March 19th, 2010 1:32 pm

It takes time. But Google does find everyone eventually.