Jump to content
Sign in to follow this  
mrmaxx

Reporting problems today?

Recommended Posts

Back to square one!...

I submitted one about an hour previous to yours with no problem. OK, I don't get many. With average processing running at around 50-55% for the past week (and significantly degraded before - stats charts) it is evidently "the luck of the draw" in getting reports through at any given time (re-trying is definitely an option if timed out). Webform reporting is currently for the lucky and the dedicated - and indications of lost or hopelessly delayed e-mail submissions are a worry.

Not sure about quick reporting within those e-mail submissions. Saw one comment in the newsgoups recently that most seemed to be going through - I suppose that is believable in a fluctuating capacity regime since those must use a little less capacity and maybe could be channeled a little more productively. Dunno - but not available to everyone anyway for various reasons.

Share this post


Link to post
Share on other sites

I submitted one about an hour previous to yours with no problem. OK, I don't get many. With average processing running at around 50-55% for the past week (and significantly degraded before - stats charts) it is evidently "the luck of the draw" in getting reports through at any given time (re-trying is definitely an option if timed out). Webform reporting is currently for the lucky and the dedicated - and indications of lost or hopelessly delayed e-mail submissions are a worry.

Not sure about quick reporting within those e-mail submissions. Saw one comment in the newsgoups recently that most seemed to be going through - I suppose that is believable in a fluctuating capacity regime since those must use a little less capacity and maybe could be channeled a little more productively. Dunno - but not available to everyone anyway for various reasons.

I always use full reporting from web interface even email submissions.

Share this post


Link to post
Share on other sites

I don't know if the statistics is indicative of what's happening. There are large spikes in reporting/submission followed by significant drop downs. The total whiteouts that we had when this started are missing.

Share this post


Link to post
Share on other sites

This tracker goes with all of the following: http://www.spamcop.net/sc?id=z5366876323z2...e538cb80f83f3cz -> 2600:6:340:1::41ab:a77a

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.84a7ab41.1341737067.41941d40

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.76a7ab41.1341739096.1300861f

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.76a7ab41.1341739299.1300f576

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.76a7ab41.1341739551.130158b2

Parsing header:

got sigalarm, taking too long to process, aborted.

Perhaps you can wait a few minutes and reload?

0: Received: from 127.0.0.1 (EHLO mail5.zrffluouledrh.com) (130.255.230.115) by mta1304.mail.gq1.yahoo.com with SMTP; Sat, 07 Jul 2012 21:45:50 -0700

No unique hostname found for source: 130.255.230.115

YahooMain received mail from sending system 130.255.230.115

got sigalarm, taking too long to process, aborted.

Perhaps you can wait a few minutes and reload?

Edited by gnarlymarley

Share this post


Link to post
Share on other sites

Whatever is happening these guys have managed to cripple us for longer than a week now. I would think someone is thinking of serious countermeasures before there is irreversible damage to the reputation of this service. I really don't want to see the bad guys win this battle..

Share this post


Link to post
Share on other sites

Still down, as of now. :(

Share this post


Link to post
Share on other sites

Maybe there should be a temporary ban on new free user registrations until this clears. I can imagine that the bad guys flood the system with new requests as well as spam submitals..

Share this post


Link to post
Share on other sites

Maybe there should be a temporary ban on new free user registrations until this clears. I can imagine that the bad guys flood the system with new requests as well as spam submitals..

I agree ...

Share this post


Link to post
Share on other sites

Been really bad for several days. This is all I get now.

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.af4d1160.1341762251.3d51101d

Now forwarding spam to spam[at]uce.gov

At least it is something.

:(

Share this post


Link to post
Share on other sites

I know nothing about Internet technique, but had these 2 ideas come up in my mind that might be all wrong and useless. Still, just in case:

Would it be possible to terminate all new accounts that were made since a little time before this problem started? With apologies (and refunds to payers) to all those new SC users, as there will be innocents among them.

Then in case there would be a way to verify the legitimacy of new applicants -including of those terminated ones- allow only those in. I don't know if the latter would be possible.

The other idea I already posted, and it might also be useless, although it looks like these guys are really getting into all the different ways a website can be hacked or somehow misused, and not only how to prevent it, but also how to repair it. And they have a forum, I suppose with some free expert help:

"How to prevent your site from getting hacked. How to repair a damaged site. Website security precautions.":

http://25yearsofprogramming.com/blog/20070705.htm

Edited by Lodewijk

Share this post


Link to post
Share on other sites

As explained to me by a SpamCop administrator, this is a back end issue with something to do with their database server. They are very much aware of the issue and I hope they will resolve the issue as soon as possible.

My best guess is that their database servers and or the connections to them are having issues. It could be any number of things: failing hard drives, not enough physical memory, tables to large, cloud server issues, fragmented storage system, failing network cards, network switch issues, router issues, firewall issues, bad updates, and so on. As an administrator for a number of large installations, I have encountered all the listed miladies. Locating and eliminating these issues can be sometime be quite elusive.

Edited by Dee Bee

Share this post


Link to post
Share on other sites

Not to worry. Spammers are not coming here in droves and signing up for accounts.

If anyone abuses an account, it gets shut down and I get a notice.

- Don D'Minion - SpamCop Admin -

- Service[at]Admin.SpamCop.net -

.

Share this post


Link to post
Share on other sites

Why do I have to dig down into this forum to see the status?

Why isn't it included in the main status pages?

I've been experiencing this for a week or so. What's the status and what's the expected time to resolve?

Thanks.

Share this post


Link to post
Share on other sites

Not to worry. Spammers are not coming here in droves and signing up for accounts.

did you understood if they use payd account too, or only free accounts?

Share this post


Link to post
Share on other sites

Why do I have to dig down into this forum to see the status?

Why isn't it included in the main status pages?

I've been experiencing this for a week or so. What's the status and what's the expected time to resolve?

Thanks.

To see the status (the amount of spam recently reported per unit of time) you can do any of the following:

  • Click the white-and-green thumbnail near top-right of any forum page.
  • Go to the URL linked by that thumbnail, for instance after bookmarking it in your browser.
  • For the individual graphs, you can go directly to one of the daily, weekly, monthly or yearly stats, each of which can also be bookmarked.

And for you Internet Explorer users, what I call "bookmarks" on my Mozilla (Firefox or SeaMonkey) browser, is also sometimes known elsewhere as "favourites".

As for the expected time to resolve, it is: when the problem will be fixed, or in other words, no one knows yet. If you see a large upward spike on the daily graph linked above, or if your "Report Now" link starts working, it will mean that (for the time being) you can report spam.

Share this post


Link to post
Share on other sites

What happened on MONTHLY GRAPH Week 25 that Tuesday through Thursday had NO ACTIVITY ?

Is this the result of spammers attacking ?

If there is anything that you need, please put it out there for us to help you.

Sincerely,

CPUjohn

Illinois, USA

Share this post


Link to post
Share on other sites

FYI, once I get my first gateway timeout, it is about 30 minutes for it to work. From what I see, it times out at each mail message hop. The database could be affecting the whois lookups too.

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.76a7ab41.1341778670.13a8e432

Edited by gnarlymarley

Share this post


Link to post
Share on other sites

As a paying SC customer, it's a pain.

The front-end we all connect to is hosted by Akamai (EdgeSuite), so the ability to ping or connect to that is meaningless: it just means your nearest Akamai edge node is live, which should always be the case whatever is happening to Spamcop.

One way or another, this attack is taking out the back end servers that do the actual analysis and DNS lookups for spam reporting. My suspicion would be the DNS side of things, unless it's a pure brute force crapflood or an old load balancer in front of the servers - although with the current codebase, junk submissions with lots of slow-resolving hostnames in might be enough to hit concurrency limits or disk thrashing.

Share this post


Link to post
Share on other sites
What happened on MONTHLY GRAPH Week 25 that Tuesday through Thursday had NO ACTIVITY ?

...

That was just something to do with the data feed/processing to the charts CPUjohn - reporting was normal throughout that period. It has been commented on that the present degraded service started rearing its ugly head shortly after it was restored but if there is a causal link and the engineers missed it I guess they would have been on the next boat back to Elbonia and their replacements would have rectified it before now.

Share this post


Link to post
Share on other sites

Okay, so why do I have to dig down into this forum to find out where to find out the status?

I would expect to see something on the “Report spam†page, which has a link to a report that hasn't been updated in over a month.

Thanks.

Share this post


Link to post
Share on other sites

Okay, so why do I have to dig down into this forum to find out where to find out the status?

I would expect to see something on the “Report spam†page, which has a link to a report that hasn't been updated in over a month.

Thanks.

You mean the news page http://mail.spamcop.net/news.php.

The status page is in the upper right corner in the link found by the white, blue, and green image: http://forum.spamcop.net/forums/index.php?...&page=stats

Share this post


Link to post
Share on other sites

That page is for Spamcop Email news only, not the reporting side of things. They are 2 separate things apparently.

Share this post


Link to post
Share on other sites

As a paying SC customer, it's a pain.

The front-end we all connect to is hosted by Akamai (EdgeSuite), so the ability to ping or connect to that is meaningless: it just means your nearest Akamai edge node is live, which should always be the case whatever is happening to Spamcop.

One way or another, this attack is taking out the back end servers that do the actual analysis and DNS lookups for spam reporting. My suspicion would be the DNS side of things, unless it's a pure brute force crapflood or an old load balancer in front of the servers - although with the current codebase, junk submissions with lots of slow-resolving hostnames in might be enough to hit concurrency limits or disk thrashing.

Ping is not completely a good test with the default settings. I have worked with switches that only let through 122 bytes. Ping by default is 80 bytes. The diagnosis too longer until we made the ping packet size closer to the MTU. I can say that the front-end server are responding perfect, but seem to be timing out when trying to contact the backend servers. Both the spam emails and the whois information will be stored and cached in the database. The issue could be either along the lines of the database, or also could be when it tries to go out and perform an external query such as whois or DNS from abuse.net. I believe we would see something different if this was a brute force on the Akamai servers.

Edited by gnarlymarley

Share this post


Link to post
Share on other sites

Forwarding spam via e-mail got response with delay but still acceptable. On the other hand, unable to report them due to time out for past 24 hours.

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.6f961645.1341795416.741077e

Share this post


Link to post
Share on other sites

This is new. Maybe we are getting somewhere.

SpamCop.net

500 Internal Server Error

Sorry, your request could not be processed.

Typically, this is the result of a temporary problem. Please re-try the operation which caused this error in a minute or two. You can even leave this error screen open and use your browser's "reload" button to re-try. Please do not press reload repeatedly though. Pressing the button more often does not resolve the problem faster. It just makes a bad situation worse.

If you believe this is a bug which occurs only under specific circumstances or have observed the problem frequently, please contact us with a full description of the problem.

Copyright © 1998-2007, IronPort Systems, Inc. All rights reserved. HTML4 / CSS2 Firefox recommended - Policies and Disclaimers

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×