Farelf Posted July 8, 2012 Share Posted July 8, 2012 Back to square one!...I submitted one about an hour previous to yours with no problem. OK, I don't get many. With average processing running at around 50-55% for the past week (and significantly degraded before - stats charts) it is evidently "the luck of the draw" in getting reports through at any given time (re-trying is definitely an option if timed out). Webform reporting is currently for the lucky and the dedicated - and indications of lost or hopelessly delayed e-mail submissions are a worry. Not sure about quick reporting within those e-mail submissions. Saw one comment in the newsgoups recently that most seemed to be going through - I suppose that is believable in a fluctuating capacity regime since those must use a little less capacity and maybe could be channeled a little more productively. Dunno - but not available to everyone anyway for various reasons. Link to comment Share on other sites More sharing options...
petzl Posted July 8, 2012 Share Posted July 8, 2012 I submitted one about an hour previous to yours with no problem. OK, I don't get many. With average processing running at around 50-55% for the past week (and significantly degraded before - stats charts) it is evidently "the luck of the draw" in getting reports through at any given time (re-trying is definitely an option if timed out). Webform reporting is currently for the lucky and the dedicated - and indications of lost or hopelessly delayed e-mail submissions are a worry. Not sure about quick reporting within those e-mail submissions. Saw one comment in the newsgoups recently that most seemed to be going through - I suppose that is believable in a fluctuating capacity regime since those must use a little less capacity and maybe could be channeled a little more productively. Dunno - but not available to everyone anyway for various reasons. I always use full reporting from web interface even email submissions. Link to comment Share on other sites More sharing options...
dra007 Posted July 8, 2012 Share Posted July 8, 2012 I don't know if the statistics is indicative of what's happening. There are large spikes in reporting/submission followed by significant drop downs. The total whiteouts that we had when this started are missing. Link to comment Share on other sites More sharing options...
gnarlymarley Posted July 8, 2012 Share Posted July 8, 2012 This tracker goes with all of the following: http://www.spamcop.net/sc?id=z5366876323z2...e538cb80f83f3cz -> 2600:6:340:1::41ab:a77a Gateway Timeout The proxy server did not receive a timely response from the upstream server. Reference #1.84a7ab41.1341737067.41941d40 Gateway Timeout The proxy server did not receive a timely response from the upstream server. Reference #1.76a7ab41.1341739096.1300861f Gateway Timeout The proxy server did not receive a timely response from the upstream server. Reference #1.76a7ab41.1341739299.1300f576 Gateway Timeout The proxy server did not receive a timely response from the upstream server. Reference #1.76a7ab41.1341739551.130158b2 Parsing header: got sigalarm, taking too long to process, aborted. Perhaps you can wait a few minutes and reload? 0: Received: from 127.0.0.1 (EHLO mail5.zrffluouledrh.com) (130.255.230.115) by mta1304.mail.gq1.yahoo.com with SMTP; Sat, 07 Jul 2012 21:45:50 -0700 No unique hostname found for source: 130.255.230.115 YahooMain received mail from sending system 130.255.230.115 got sigalarm, taking too long to process, aborted. Perhaps you can wait a few minutes and reload? Link to comment Share on other sites More sharing options...
dra007 Posted July 8, 2012 Share Posted July 8, 2012 Whatever is happening these guys have managed to cripple us for longer than a week now. I would think someone is thinking of serious countermeasures before there is irreversible damage to the reputation of this service. I really don't want to see the bad guys win this battle.. Link to comment Share on other sites More sharing options...
csouter Posted July 8, 2012 Share Posted July 8, 2012 Still down, as of now. Link to comment Share on other sites More sharing options...
dra007 Posted July 8, 2012 Share Posted July 8, 2012 Maybe there should be a temporary ban on new free user registrations until this clears. I can imagine that the bad guys flood the system with new requests as well as spam submitals.. Link to comment Share on other sites More sharing options...
hok Posted July 8, 2012 Share Posted July 8, 2012 Maybe there should be a temporary ban on new free user registrations until this clears. I can imagine that the bad guys flood the system with new requests as well as spam submitals..I agree ... Link to comment Share on other sites More sharing options...
kamot Posted July 8, 2012 Share Posted July 8, 2012 Been really bad for several days. This is all I get now. Gateway Timeout The proxy server did not receive a timely response from the upstream server. Reference #1.af4d1160.1341762251.3d51101d Now forwarding spam to spam[at]uce.gov At least it is something. Link to comment Share on other sites More sharing options...
Lodewijk Posted July 8, 2012 Share Posted July 8, 2012 I know nothing about Internet technique, but had these 2 ideas come up in my mind that might be all wrong and useless. Still, just in case: Would it be possible to terminate all new accounts that were made since a little time before this problem started? With apologies (and refunds to payers) to all those new SC users, as there will be innocents among them. Then in case there would be a way to verify the legitimacy of new applicants -including of those terminated ones- allow only those in. I don't know if the latter would be possible. The other idea I already posted, and it might also be useless, although it looks like these guys are really getting into all the different ways a website can be hacked or somehow misused, and not only how to prevent it, but also how to repair it. And they have a forum, I suppose with some free expert help: "How to prevent your site from getting hacked. How to repair a damaged site. Website security precautions.": http://25yearsofprogramming.com/blog/20070705.htm Link to comment Share on other sites More sharing options...
Dee Bee Posted July 8, 2012 Share Posted July 8, 2012 As explained to me by a SpamCop administrator, this is a back end issue with something to do with their database server. They are very much aware of the issue and I hope they will resolve the issue as soon as possible. My best guess is that their database servers and or the connections to them are having issues. It could be any number of things: failing hard drives, not enough physical memory, tables to large, cloud server issues, fragmented storage system, failing network cards, network switch issues, router issues, firewall issues, bad updates, and so on. As an administrator for a number of large installations, I have encountered all the listed miladies. Locating and eliminating these issues can be sometime be quite elusive. Link to comment Share on other sites More sharing options...
SpamCopAdmin Posted July 8, 2012 Share Posted July 8, 2012 Not to worry. Spammers are not coming here in droves and signing up for accounts. If anyone abuses an account, it gets shut down and I get a notice. - Don D'Minion - SpamCop Admin - - Service[at]Admin.SpamCop.net - . Link to comment Share on other sites More sharing options...
danorton Posted July 8, 2012 Share Posted July 8, 2012 Why do I have to dig down into this forum to see the status? Why isn't it included in the main status pages? I've been experiencing this for a week or so. What's the status and what's the expected time to resolve? Thanks. Link to comment Share on other sites More sharing options...
efa Posted July 8, 2012 Share Posted July 8, 2012 Not to worry. Spammers are not coming here in droves and signing up for accounts. did you understood if they use payd account too, or only free accounts? Link to comment Share on other sites More sharing options...
A.J.Mechelynck Posted July 8, 2012 Share Posted July 8, 2012 Why do I have to dig down into this forum to see the status? Why isn't it included in the main status pages? I've been experiencing this for a week or so. What's the status and what's the expected time to resolve? Thanks. To see the status (the amount of spam recently reported per unit of time) you can do any of the following: Click the white-and-green thumbnail near top-right of any forum page. Go to the URL linked by that thumbnail, for instance after bookmarking it in your browser. For the individual graphs, you can go directly to one of the daily, weekly, monthly or yearly stats, each of which can also be bookmarked. And for you Internet Explorer users, what I call "bookmarks" on my Mozilla (Firefox or SeaMonkey) browser, is also sometimes known elsewhere as "favourites". As for the expected time to resolve, it is: when the problem will be fixed, or in other words, no one knows yet. If you see a large upward spike on the daily graph linked above, or if your "Report Now" link starts working, it will mean that (for the time being) you can report spam. Link to comment Share on other sites More sharing options...
CPUjohn Posted July 8, 2012 Share Posted July 8, 2012 What happened on MONTHLY GRAPH Week 25 that Tuesday through Thursday had NO ACTIVITY ? Is this the result of spammers attacking ? If there is anything that you need, please put it out there for us to help you. Sincerely, CPUjohn Illinois, USA Link to comment Share on other sites More sharing options...
gnarlymarley Posted July 8, 2012 Share Posted July 8, 2012 FYI, once I get my first gateway timeout, it is about 30 minutes for it to work. From what I see, it times out at each mail message hop. The database could be affecting the whois lookups too. Gateway Timeout The proxy server did not receive a timely response from the upstream server. Reference #1.76a7ab41.1341778670.13a8e432 Link to comment Share on other sites More sharing options...
jas88 Posted July 8, 2012 Share Posted July 8, 2012 As a paying SC customer, it's a pain. The front-end we all connect to is hosted by Akamai (EdgeSuite), so the ability to ping or connect to that is meaningless: it just means your nearest Akamai edge node is live, which should always be the case whatever is happening to Spamcop. One way or another, this attack is taking out the back end servers that do the actual analysis and DNS lookups for spam reporting. My suspicion would be the DNS side of things, unless it's a pure brute force crapflood or an old load balancer in front of the servers - although with the current codebase, junk submissions with lots of slow-resolving hostnames in might be enough to hit concurrency limits or disk thrashing. Link to comment Share on other sites More sharing options...
Farelf Posted July 8, 2012 Share Posted July 8, 2012 What happened on MONTHLY GRAPH Week 25 that Tuesday through Thursday had NO ACTIVITY ? ... That was just something to do with the data feed/processing to the charts CPUjohn - reporting was normal throughout that period. It has been commented on that the present degraded service started rearing its ugly head shortly after it was restored but if there is a causal link and the engineers missed it I guess they would have been on the next boat back to Elbonia and their replacements would have rectified it before now. Link to comment Share on other sites More sharing options...
danorton Posted July 8, 2012 Share Posted July 8, 2012 Okay, so why do I have to dig down into this forum to find out where to find out the status? I would expect to see something on the “Report spam†page, which has a link to a report that hasn't been updated in over a month. Thanks. Link to comment Share on other sites More sharing options...
gnarlymarley Posted July 8, 2012 Share Posted July 8, 2012 Okay, so why do I have to dig down into this forum to find out where to find out the status? I would expect to see something on the “Report spam†page, which has a link to a report that hasn't been updated in over a month. Thanks. You mean the news page http://mail.spamcop.net/news.php. The status page is in the upper right corner in the link found by the white, blue, and green image: http://forum.spamcop.net/forums/index.php?...&page=stats Link to comment Share on other sites More sharing options...
Ex_Brit Posted July 8, 2012 Share Posted July 8, 2012 That page is for Spamcop Email news only, not the reporting side of things. They are 2 separate things apparently. Link to comment Share on other sites More sharing options...
gnarlymarley Posted July 8, 2012 Share Posted July 8, 2012 As a paying SC customer, it's a pain. The front-end we all connect to is hosted by Akamai (EdgeSuite), so the ability to ping or connect to that is meaningless: it just means your nearest Akamai edge node is live, which should always be the case whatever is happening to Spamcop. One way or another, this attack is taking out the back end servers that do the actual analysis and DNS lookups for spam reporting. My suspicion would be the DNS side of things, unless it's a pure brute force crapflood or an old load balancer in front of the servers - although with the current codebase, junk submissions with lots of slow-resolving hostnames in might be enough to hit concurrency limits or disk thrashing. Ping is not completely a good test with the default settings. I have worked with switches that only let through 122 bytes. Ping by default is 80 bytes. The diagnosis too longer until we made the ping packet size closer to the MTU. I can say that the front-end server are responding perfect, but seem to be timing out when trying to contact the backend servers. Both the spam emails and the whois information will be stored and cached in the database. The issue could be either along the lines of the database, or also could be when it tries to go out and perform an external query such as whois or DNS from abuse.net. I believe we would see something different if this was a brute force on the Akamai servers. Link to comment Share on other sites More sharing options...
hok Posted July 9, 2012 Share Posted July 9, 2012 Forwarding spam via e-mail got response with delay but still acceptable. On the other hand, unable to report them due to time out for past 24 hours. Gateway Timeout The proxy server did not receive a timely response from the upstream server. Reference #1.6f961645.1341795416.741077e Link to comment Share on other sites More sharing options...
gnarlymarley Posted July 9, 2012 Share Posted July 9, 2012 This is new. Maybe we are getting somewhere. SpamCop.net 500 Internal Server Error Sorry, your request could not be processed. Typically, this is the result of a temporary problem. Please re-try the operation which caused this error in a minute or two. You can even leave this error screen open and use your browser's "reload" button to re-try. Please do not press reload repeatedly though. Pressing the button more often does not resolve the problem faster. It just makes a bad situation worse. If you believe this is a bug which occurs only under specific circumstances or have observed the problem frequently, please contact us with a full description of the problem. Copyright © 1998-2007, IronPort Systems, Inc. All rights reserved. HTML4 / CSS2 Firefox recommended - Policies and Disclaimers Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.