Saving bandwidth by preventing spider access to your wordpress blog.

Aus Salespoint

I have a couple of internet sites, and just not too long ago (this past month), I've noticed that a single of them is utilizing what I would think about excessive bandwidth. Digging a small deeper, it appears that bots are employing all the bandwidth.

This is just my personal website – I have a travel weblog in the principal domain and a loved ones history site in a subdomain. In the past, I have used on average 400mb per month of bandwidth. This month I've had to improve the bandwidth to 1.5GB, but it really is possibly going to go more than (it really is at present at 1.38GB). These two websites aren't large, and don't get stacks of hits, mostly just friends and family members. The household history internet site has a few huge images, but nothing excessive.

Looking at awstats this month, the blog internet site has employed 150mb of bandwidth + 480mb of bot bandwidth (380mb of that is msnbot). The loved ones history website has employed 55mb of bandwidth + 650mb of bot bandwidth (620mb of that is googlebot).

Most robots determine themselves by a custom user agent in the request headers. Which can effortlessly be blocked with htaccess.

There are a quantity of very good articles on this. Let me know if you have any problems, as it is a matter of identifying the offending bots/crawlers and banning them as per your want.

I have utilised google tools to inform it to not frequent the web site as a lot about a week ago, but it does not seem to have produced a diverse. Short of telling the robots to bugger off completely by means of the robots.txt file (it really is just my individual website, wordpress spider management but it is still nice to be listed in google!), is there something else I can do?

I presently use this on all my internet sites , basically it blocks all negative user agents , poor bots and scrappers, Not only can it save your content material from getting mass harvested but will also save you a tiny bandwidth due to the fact of much less bots operating around your website. Hope it assists

can tell you that Google drags in a fantastic several spiders due to advertizing, specifically if you are making use of Adsense on your site along with ising the a variety of advertisements from the Google ad network wordpress spiderspanker partners - these partners also send their bots to test your traffic sources and what adverts to place on your internet site - Google has been hitting hard lately simply because of the algorithm tweaks and Adsense obtaining had a lucrative month in the terms of the quantity of new advertisers on board.

Even if you slow down the crawl rate, you will still see a large chunk of bandwidth disappear. The bots are way also intermittent to make accurate adjustments unless you wish to block them.

There's a new wordpress plugin that can support with this! I've gotten a couple of emails re a item being sold to remove or "spank" the poor spiders that are taking up lots of bandwidth and not adding worth to your organization, freeing up space for actual guests and not causing a issue with hosting limits. It really is referred to as Spyder spanker at that name .com if you want to see the sales page.

Anyway, I'm not positive if this is a thing beneficial that I need to have or not. I do see a lot of spider activity in my stats, but I constantly thought that was sort of excellent b/c it implies they are crawling my websites and hopefully indexing them.

The huge danger is stealing your bandwidth. Some of the spiders sent filter spiders wordpress by spammers will hammer your site as fast as they can, slowing down response for your human visitors.