How can I tell how many of my webpages are indexed by a search engine?
What are the step by step instructions to check how many of my website’s pages are indexed by a search engine?
If my site has 1000 pages and all are indexed (taking up lots of server space and thusly money) can I block some of those pages with a robots.txt file and still have the same search results listing? I want to drop the number of pages a crawler looks at to save me money but I want to remain high on the search listings…is this posisble-how?
Alright, I have found out how many pages are indexed…and you guys are not going to believe me but the answer is 67,000. This site is a gigantic corporate site who hosts their own stuff…now the main sys admins guy says the site is getting crawled by google bots at regular intervals (he says as frequently as 5 minutes).
So you can see how this truly could affect the bottom dollar by several thousand a month.
So now what….what do you recommend?
3 Responses
pixelfused
29 Jan 2010
liverpoolscouser
29 Jan 2010
You can certainly disallow bots from indexing certain pages and, providing your home page and the main content pages are indexed, your overall rating in the search engines is unlikely to suffer.
Randy Moss
29 Jan 2010
I agree with the previous answer. The more pages you have indexed the better. Google should not be sending so many bots to your site that you go over the badnwidth limit.
Go to google.com/sitemaps and create an account if don’t already. Follow the steps and it will allow you to track how often Google goes to your site and and what pages. You can submit a request to Google to unlink a certain page if you want…though I would not sugget this.
To find out how many of your pages are indexed type site:yourwebsite.com into Google.
I think you may be going about this the wrong way, though. Google Indexing your site doesn’t use up any server space on your hosting account. The only way it would cost you money is if Google is pushing you over your bandwidth limit. Unless your site is getting 10,000+ visits a day or your pages are >1MB each you shouldn’t be anywhere near your limit.
Restricting indexing of content pages on your website won’t do anything but hurt you. You could be losing long tail searches on every page you delist. I’d be glad to take a quick look at your site if you give me the link.