How can I tell how many of my webpages are indexed by a search engine?
What are the step by step instructions to check how many of my website’s pages are indexed by a search engine?
If my site has 1000 pages and all are indexed (taking up lots of server space and thusly money) can I block some of those pages with a robots.txt file and still have the same search results listing? I want to drop the number of pages a crawler looks at to save me money but I want to remain high on the search listings…is this posisble-how?
Alright, I have found out how many pages are indexed…and you guys are not going to believe me but the answer is 67,000. This site is a gigantic corporate site who hosts their own stuff…now the main sys admins guy says the site is getting crawled by google bots at regular intervals (he says as frequently as 5 minutes).
So you can see how this truly could affect the bottom dollar by several thousand a month.
So now what….what do you recommend?




