Wazoo Posted June 29, 2007 Share Posted June 29, 2007 http://www.ipsbeyond.com/forums/index.php?showtopic=24698 (IPB customer only) IPSBeyond _ IPB Assistance _ Increased Guests Online? Posted by: bfarber Jun 29 2007, 09:06 AM This topic highlights a large reason why you may be seeing an increased number of users (or more specifically, Slurp/Yahoo spiders) online recently. http://www.ipsbeyond.com/forums/index.php?...st&p=145039 (IPB customer only) To highlight the important parts 1) Yahoo is now spidering sites from multiple IPs at the same time, which causes most guest tracking (including IPB's) to treat the requests as different sessions. As such whereas you may have seen 100 users online previously, you may now see 1000. This is actually working as IPB is intended, because you indeed have 1000 different computers accessing your site (and your server resources ARE being used to serve 1000 requests) however it is "new" because no other spiders do this, and yahoo did not used to do this. 2) Yahoo IS actually sending out hundreds (and sometimes thousands) of spiders to sites at once, which is crippling many people's servers. Our company forums have 1500 guests online last night when I looked, and I estimated about 1200 were Yahoo spiders. It's not fair for users who get 50 real visitors, and maybe 50 guest visitors to have to pay and accomodate 1000 users because of Yahoo spiders. Please see this Digg article if you wish to voice your support: http://www.digg.com/software/Yahoo_Spider_..._guest_tracking and PLEASE vote for this suggestion for Yahoo to fix their problem if you too are unhappy with these recent changes: http://suggestions.yahoo.com/detail/?prop=...r&fid=31431 I mean, just thinking of OUR shared hosting customers, once they hit a certain number of users online we have to recommend to upgrade their hosting plan, and I think it's entirely unfair to them that they should do this (and pay more, of course) just because of Yahoo. There are suggestions that can help with this issue. One involves throttling Yahoo's requests, and others involve banning their IP/spider so they do not index you at all. While it is never recommended to ban a search engine spider, at this rate that may be the only thing users can do to protect their sites. Posted by: bfarber Jun 29 2007, 09:07 AM I am pinning this because I think it deserves a higher level of attention. I urge everyone to visit the suggestion thread at yahoo's board and click the "Rate this" icon. Show Yahoo that you don't want your server crippled so that they can add a few more pages to their index. Posted by: Axel Wers Jun 29 2007, 09:50 AM If somebody thinks that Yahoo is too active at your boards you can try http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html Posted by: bfarber Jun 29 2007, 01:57 PM I noticed the link to that suggestion on Yahoo was posted on vbulletin's forums today too. Their users are reporting the same problems with Yahoo. Yahoo's latest response was something to the effect of "We're looking at some documentation and we'll be replying soon.". Heh, in the mean time, I say everyone just ban Yahoo's spiders. Perhaps they will wake up when their spiders are unable to index anything because they're being resource hogs. We just banned Yahoo on one of our server setups. I cleared out 817 out of 917 guest sessions from Yahoo and load dropped from 22 to 9 in about 60 seconds. I mean load of 22 (higher yesterday) down to 2.54???? Yahoo == evil Posted by: bfarber Jun 29 2007, 03:39 PM We did it in iptables today iptables -I INPUT -m iprange --src-range 74.6.0.0-74.6.255.255 -j DROP This ENTIRE range is owned by Inktomi (Yahoo) so you won't be banning any legit users. The load on the box went from 22 (it was closer to 40 yesterday) to 2 in about 5-10 minutes. Guests online went from over 1000 to 169. Link to comment Share on other sites More sharing options...
Thor Posted October 10, 2007 Share Posted October 10, 2007 http://www.ipsbeyond.com/forums/index.php?showtopic=24698 (IPB customer only) Our site has experienced a problem similar, but the one thing that we don't understand is the fact that some of the spiders are printing out parts of the forum, or so the forum software says they're printing things off. Is this a glitch in the forum software or some other reason the spiders appear to be printing?? Link to comment Share on other sites More sharing options...
DavidT Posted October 10, 2007 Share Posted October 10, 2007 Wazoo - I'd say you should consider blocking Yahoo, as did someone in the thread you posted. The current board visitor stats are: 35 guests, 2 members 0 anonymous membersI'm guessing that perhaps there are multiple Yahoo spiders among the guests? DT Link to comment Share on other sites More sharing options...
Wazoo Posted October 10, 2007 Author Share Posted October 10, 2007 Our site has experienced a problem similar, but the one thing that we don't understand is the fact that some of the spiders are printing out parts of the forum, or so the forum software says they're printing things off. Is this a glitch in the forum software or some other reason the spiders appear to be printing?? The actual situation depends on your software. Technically, a 'search bot' won't really be printing stuff off. On the other hand, your 'print command/function' may result in a 'printable page' that may end up being indexed. Most search engines will respect a DISALLOW in robots.txt if you put the 'print command line' in that section. Link to comment Share on other sites More sharing options...
Wazoo Posted October 10, 2007 Author Share Posted October 10, 2007 Wazoo - I'd say you should consider blocking Yahoo, as did someone in the thread you posted. ... I'm guessing that perhaps there are multiple Yahoo spiders among the guests? That line reads as 28 guests, 2 members 0 anonymous members at present. Looking at the 'sessions' Table ... Yahoo is there maybe 70+ times, MSN about 20+ times, Ask Jeeves twice, in addition to 40+ entries not associated with users or identifying themselves as bots ..... Numbers look strange as the seesions Table isn't actually real-time in that it tracks data for a (configured) amount of time but the displayed line you're looking at only reaches back 10 minutes. This server has been keeping up thus far ... between the search bots, users, and the constant stream of hacking attempts. The previous server showed its problems when the bots all hit a high-activity mode at the same time, basically running into an issue where it couldn't keep up with writing out log-file entries. Link to comment Share on other sites More sharing options...
DavidT Posted October 11, 2007 Share Posted October 11, 2007 Looking at the 'sessions' Table ... Yahoo is there maybe 70+ times, MSN about 20+ times I still say "off with their heads"! Who needs those two when everyone is using Google. DT Link to comment Share on other sites More sharing options...
Wazoo Posted October 11, 2007 Author Share Posted October 11, 2007 Who needs those two when everyone is using Google. Heh! Remebering the awe in the "only a Tradeshow demo" that once was king .. Altavista ... A lot of the folks that I deal with really don't 'know' Google actually. Those that use MSN Explorer, those that end up with toolbaritis .... had one a while back that had less than half the (vertical) browser window actually available due to all the toolbars that had been installed (most of them malware associated) Link to comment Share on other sites More sharing options...
DavidT Posted October 11, 2007 Share Posted October 11, 2007 Heh! Remebering the awe in the "only a Tradeshow demo" that once was king .. Altavista ... That was my fav for a very long time. I still use it to do "link" lookups, because it's better than Google for that purpose. A lot of the folks that I deal with really don't 'know' Google actually. Those that use MSN Explorer, those that end up with toolbaritis .... had one a while back that had less than half the (vertical) browser window actually available due to all the toolbars that had been installed (most of them malware associated) I've seen the same, and quickly and quietly uninstalled all and added the Google toolbar....although I avoid their Desktop and Updater. DT Link to comment Share on other sites More sharing options...
StevenUnderwood Posted October 11, 2007 Share Posted October 11, 2007 Heh! Remebering the awe in the "only a Tradeshow demo" that once was king .. Altavista ...Ran on VMS and I had my entire company using them as the original home page. P.S. I am a VMS bigot http://www.openvmshobbyist.com/vmsbigot.html Link to comment Share on other sites More sharing options...
DavidT Posted October 11, 2007 Share Posted October 11, 2007 This server has been keeping up thus far ...Maybe not....I just opened a browser tab to the main Forum page and it took a lot longer than usual. I had also noticed some new slowness over the last few days. While I know that there could be many other factors betwixt here and there that might be causing this, if this is a known issue "out there," and it's having negative effects on other forum sites/servers, then please take another (serious) look at banning the misbehaving Yahoo bots. DT Link to comment Share on other sites More sharing options...
Farelf Posted October 11, 2007 Share Posted October 11, 2007 Behaving OK for me at the moment - but only 37 guests. Whatever, I don't use Yahoo anyway. On the gripping hand, some of those Google features are actually useful. I love the "convert" thing. And webpage translations (but not the translator on the toolbar). Heh, others like the convert facility as well, but as always "computation without mentation" is an ambush awaiting. The correct expression for convering precious metals production being convert x kilograms to troy ounces (or vice versa) NOT convert x kilograms to ounces. No names, no pack drill. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.