Unable to access http://www.spamcop.net

[Jim Pivonka]

This is new behavior, today, Saturday Feb. 10. I have not modified my machine since my last reports were completed yesterday, and those went through fine.

I am unable to access spamcop.net, either directly or through the emails I recieve from spamcop[at]devnull.spamcop.net which contain reporting links/URL's.

The error message I get is this:

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.e8497d8.1171147000.284aaefe

The last line changes with each error, I believe, as in:

Reference #1.e8497d8.1171147000.284aaefe

Reference #1.e8497d8.1171147841.28670025

Reference #1.e8497d8.1171149326.289775e3

Reference #1.e8497d8.1171147641.286002ce

Reference #1.e8497d8.1171147700.28622780

etc....

Any suggestions?

Note: This may reprise material at http://forum.spamcop.net/forums/lofiversio....php/t6547.html which I do not understand, so please do not think that pointing me to that discussion will be of any help.

Also, I do not know "traceroute", but would be glad to learn and get (it, one, whatever?) if someone helps me learn how to do that, and what tools to use. I gather that the "Gateway" and "Upstream Server" references are to internet infrastucture behavior, not to Spamcop website behavior, and that the "traceroute" thingy might help to parse that.

[DavidT]

Same thing is happening to me also. The reporting/parsing system is having problems. I saw this a little while ago, then the system recovered for a while, and now it is not responding at the moment. I'm surprised nobody else has reported it, but maybe everyone else has something better to do on a Saturday night. :-)

You'll note that you and I are the only two people logged into the Forums at the moment....there's nothing that can be done. Wazoo is always very busy on Saturdays, or maybe he could "page" someone....the Deputies....whoever. In any case, the system is unstable, so I'd wait a few hours, or maybe until tomorrow until trying to use it.

DT

[StevenUnderwood]

Same thing is happening to me also. The reporting/parsing system is having problems. I saw this a little while ago, then the system recovered for a while, and now it is not responding at the moment. I'm surprised nobody else has reported it, but maybe everyone else has something better to do on a Saturday night. :-)

You'll note that you and I are the only two people logged into the Forums at the moment....there's nothing that can be done. Wazoo is always very busy on Saturdays, or maybe he could "page" someone....the Deputies....whoever. In any case, the system is unstable, so I'd wait a few hours, or maybe until tomorrow until trying to use it.

DT

As Don would say, when you experience problems with the reporting system, please shut down your machine, grab your favorite beverage and relax on the couch for a while.

That being said, I just logged into all the available ways without problem, though I have no spam to report at the moment:

http://www.spamcop.net using email account

http://members.spamcop.net using free ccount

http://mailsc.spamcop.net using email account

[Jim Pivonka]

As Don would say, when you experience problems with the reporting system, please shut down your machine, grab your favorite beverage and relax on the couch for a while.

This pretty dramatically goes beyond the reporting system. And based on the prior discussion referenced in my edit to the original posting it may not be a problem with the reporting system, or with the www.spamcop.net site per se, but with internet infrastucture supporting access to those.

While the problem may disappear in a few hours, that may be due to a change in routing through the internet, not because the underlying issue has been identified and corrected. Which would mean that it might have migrated to affect other users, or might have gone dormant, to be back the next day, or next week.

As for myself, I have some Lagunitas Czech Style Pilsener here, and I think I'll relax for a while. ;-)

[DavidT]

I just logged into all the available ways without problem, though I have no spam to report at the moment:

http://www.spamcop.net using email account

http://members.spamcop.net using free ccount

http://mailsc.spamcop.net using email account

Actually, logging in to any of the systems wasn't the issue yesterday. It was only upon requesting a parsing action by the system that things went south. If you take a look at the "Reporting Server Status" from late yesterday, you'll see an interesting anomaly where reports trended downward and then suddenly spiked. This might indicate some sort of intermittent timeouts/problems, followed by a reboot of the server or the like.

DT

[Jim Pivonka]

Still no joy here. Attempting to use a reporting link like http://www.spamcop.net/sc?id=z1221123929zc...4de450c8257328z results, after about 3 minutes, in this message:

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.e8497d8.1171208763.2c33568a

The same is still occuring on login attempts. It's now 18 hours since I first detected the behavior.

[Wazoo]

Still no joy here.

The same is still occuring on login attempts. It's now 18 hours since I first detected the behavior.

You'll note that you aren't seeing this while connecting to 'here' .. which is not involving the Akamai services.

Yes, any number of folks could advise, suggest, explain how to do/use a 'traceroute' command thing, but .... you have yet to follow any of the suggested guidance on "How to post a question ..." and provide any data on what you might be using to make this connection. You'd do it one way if you were using Win-98, another way if you were using Win-XP, another way if using OS-X, a different way if using a Linux distribution/install .....

I have checked for you .. agree, 'we' haven't put in a definition/description int the Dictionary, Glossary, or the Wiki. On the other hand, SpamCop Discussion > Discussions & Observations > How to use .... Instructions, Tutorials > Research Tools does contain a post provided by another SpamCop.net user (Merlyn) that offers some specific data, provides a link to a web-site with an interesting name .. take a look at Lookup tools

There are multple web-sites out there that offer a 'traceroute' function from their servers, but ... what is needed is the traceroute data between your computer and the attempted SpamCop.net connection.

[DavidT]

Wazoo, I don't think any tracerouting by end-users is going to shed much light on this situation, but I could be wrong. This is something that more than one of us is experiencing/reporting, and it has happened before. Here's a link:

http://forum.spamcop.net/forums/index.php?showtopic=6547

Here's what Ellen wrote last November about this very issue:

There have been various problems which cause the akamai timeout

situation. And engineering has been working on the issues so, in

general, you should be seeing fewer of them and optimistically none at

all. "the akamai timeout" may be the result of hardware or software

issues or a combination of same either at akamai or at SC. Akamai has

been responsive when we have opened tickets with them.

On the SC side -- we have made some hardware changes and upgrades and

the SC code base is regularly updated. The amount of spam that the

system has been handling has been increasing steadily and, of course,

that has put a strain on both the hardware and software; sometimes in

unanticipated ways. It's an ongoing process and also frustrating to

track down the cause a lot of the time, because a number of things have

to happen together at exactly the wrong time to cause the timeout and

recreating that to be able to ensure that all the causes have been found

tends to be impossible. So it is not as simple as a light bulb burns out

and when it is replaced then the akamai timeout goes away forever.

I just experienced the problem again, with three different attempts at using the system. First, I tried reloading an IP check I've been watching:

http://mailsc.spamcop.net/sc?track=212.247.178.236

result:

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.48497d8.1171221270.3846d98b

So then I tried going to the Tracking URL provided above by Jim, and here's the result:

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.e8497d8.1171221260.2d57ddd3

Then, I tried to simply reach http://mailsc.spamcop.net/, and here's the result:

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.48497d8.1171221348.38492e31

I'd say there's a bit of a problem. I'm probably not the best one to be contacting the Deputies at the moment:

http://forum.spamcop.net/forums/index.php?...ost&p=54107

http://forum.spamcop.net/forums/index.php?...ost&p=50031

etc.

DT

[Wazoo]

Wazoo, I don't think any tracerouting by end-users is going to shed much light on this situation, but I could be wrong.

Won't say right or wrong either, just going with the historical on this error message ...

02/11/07 14:15:50 Slow traceroute mailsc.spamcop.net

Trace mailsc.spamcop.net (8.7.41.6) ...

...

12.123.6.65 RTT: 20ms TTL: 0 (ggr2-p310.cgcil.ip.att.net bogus rDNS: host not found [authoritative])

192.205.33.186 RTT: 23ms TTL: 0 (No rDNS)

4.68.101.1 RTT: 30ms TTL: 0 (ae-1-51.bbr1.Chicago1.Level3.net ok)

64.159.1.66 RTT: 50ms TTL: 0 (so-1-0-0.mp1.Weehawken1.Level3.net ok)

4.68.97.254 RTT: 40ms TTL: 0 (ge-11-1.hsa1.Weehawken1.Level3.net ok)

8.7.41.6 RTT: 57ms TTL: 53 (No rDNS)

My first thought is that this is not a SpamCop.net/IronPort IP address .....

02/11/07 14:20:40 Slow traceroute spamcop.net

Trace spamcop.net (205.161.6.6) ...

...

12.123.4.249 RTT: 19ms TTL: 0 (ggr3-ge90.cgcil.ip.att.net bogus rDNS: host not found [authoritative])

192.205.33.158 RTT: 34ms TTL: 0 (No rDNS)

144.232.20.90 RTT: 26ms TTL: 0 (sl-bb22-chi-5-0.sprintlink.net ok)

144.232.20.129 RTT: 42ms TTL: 0 (sl-bb21-kc-3-0.sprintlink.net ok)

144.232.8.63 RTT: 43ms TTL: 0 (sl-bb26-fw-13-0.sprintlink.net ok)

144.232.11.29 RTT: 45ms TTL: 0 (sl-bb22-fw-11-0.sprintlink.net ok)

144.232.11.226 RTT: 43ms TTL: 0 (sl-bb20-fw-15-0.sprintlink.net ok)

144.232.20.81 RTT: 46ms TTL: 0 (sl-st21-dal-13-0.sprintlink.net ok)

205.161.6.6 RTT: 44ms TTL: 52 (No rDNS)

02/11/07 14:25:26 Slow traceroute mail.spamcop.net

Trace mail.spamcop.net (216.154.195.50) ...

...

12.117.136.42 RTT: 53ms TTL: 0 (No rDNS)

66.35.174.102 RTT: 55ms TTL: 0 (pos3-0.suwangaeq00w.cr.deltacom.net ok)

66.35.174.122 RTT: 55ms TTL: 0 (suw02.gig1-0.edeltacom.com bogus rDNS: host not found [authoritative])

216.154.207.13 RTT: 61ms TTL: 0 (suwC1-gig3-5.qualitytech.com bogus rDNS: host not found [authoritative])

216.154.207.22 RTT: 54ms TTL: 0 (suwE1N-gig1-1-4.qualitytech.com bogus rDNS: host not found [authoritative])

216.154.195.50 RTT: 55ms TTL: 50 (mail.spamcop.net ok)

The only one that counts is the first one, but just trying to show that the mailsc address is not a direct connection to JT's hardware in Georgia ... it is an Akamai 'serviced' address.

Here's what Ellen wrote last November about this very issue:

Yeah, but .... I (and I repeat) I have to factor in the language used in a Newsgroup posting by Don that the "engineering reports are all greek to me" .... Ellen, to the best of my knowledge, is not a programmer either. Going to leave that right there.

I just experienced the problem again, with three different attempts at using the system. First, I tried reloading an IP check I've been watching:

and as usual, all three links work fine from here ....

I'd say there's a bit of a problem. I'm probably not the best one to be contacting the Deputies at the moment:

heh! I would believe you know my status also ...

[DavidT]

heh! I would believe you know my status also ...

You bet. In any case the timing out issue is very sporadic, but it's happening to more than one of us and is a previously-documented issue. Don't waste any of your time on it Wazoo, but if anyone else has problems with the "Gateway Timeout" issue, please drop a note to: deputies[at]admin.spamcop.net

DT

[Jim Pivonka]

I've had some other problems with the Internet recently, Earthlink Web Hosting outages, etc., so I thought this problem might be related to a general attack or breakdown on the web, and called brother, who is a pro in such matters. He checked and said no. I mentioned the Akamai involvement. Then he walked me through the surprisingly simple traceroute process... sheesh..

Start>Run>Cmd>tracert www.spamcop.net It is easier to specify the command than to say it should be done. Results:

Tracing route to a369.g.akamai.net [216.151.132.18]

over a maximum of 30 hops:

1 29 ms 37 ms 35 ms h-67-101-213-1.snfccasy.dynamic.covad.net [67.101.213.1]

2 49 ms 83 ms 39 ms 192.168.36.101

3 44 ms 39 ms 28 ms ge-8-0-181.ipcolo1.SanFrancisco1.Level3.net [63.211.150.25]

4 39 ms 30 ms 46 ms so-6-1-0.mp2.SanFrancisco1.Level3.net [4.68.96.145]

5 29 ms 40 ms 31 ms ae-0-0.bbr1.SanJose1.Level3.net [64.159.1.129]

6 42 ms 39 ms 40 ms ae-14-55.car4.SanJose1.Level3.net [4.68.123.142]

7 38 ms 43 ms 38 ms gblx-level3-te.SanJose1.Level3.com [4.68.111.162]

8 57 ms 41 ms 43 ms 67.17.193.138

9 39 ms 42 ms 41 ms 216.151.132.18

Trace complete.

D:\Documents and Settings\Jim>tracert

Usage: tracert [-d] [-h maximum_hops] [-j host-list] [-w timeout] target_name

Options:

-d Do not resolve addresses to hostnames.

-h maximum_hops Maximum number of hops to search for target.

-j host-list Loose source route along host-list.

-w timeout Wait timeout milliseconds for each reply.

D:\Documents and Settings\Jim>nslookup www.spamcop.net

Server: ns1.mindspring.com

Address: 207.69.188.185

Non-authoritative answer:

Name: a369.g.akamai.net

Addresses: 216.151.132.18, 216.151.132.50

Aliases: www.spamcop.net, www.spamcop.net.edgesuite.net

Since the tracert had not failed I tried to browse to www.spamcop.net. That worked. So did the links to the spam reporting, which had been out 24 hours. Mike speculates that running tracert may have refreshed cache somewhere along the line, but whatever was blocking access to spamcop is now gone away. I think the thread can be closed, though this seems to be an intermittent and persistent issue, worthy of attention to get a permanent clarification about cause and possible resolution - and maybe a FAQ item.

Thanks for your attention and support. I learned a bit and any day that happens, without utter disaster being involved, is not a bad day.

[DavidT]

The problem still exists, according to my tests. I've had a tab open in my browser for a few days pointing to the SC database lookup on a particular IP, and I refresh it every once in a while to see the latest status. Right now, I'm getting a Gateway Timeout error once again. These have happened sporadically ever since you started this topic yesterday. IOW, there's a problem that needs looking into.

DT

[Jim Pivonka]

The problem still exists, according to my tests. I've had a tab open in my browser for a few days pointing to the SC database lookup on a particular IP, and I refresh it every once in a while to see the latest status. Right now, I'm getting a Gateway Timeout error once again. These have happened sporadically ever since you started this topic yesterday. IOW, there's a problem that needs looking into.

If there were a chunk of the Akamai infrastructure that was not handling your access requests properly then some action that caused your access to be routed to a different Akamai facility might help.

If tracert had not found www.spamcop.com and so indicated that I had access (it may in fact have forced renewal of some cached routing information, and so fixed the problem) then getting a new IP from my ISP might have forced a change in the Akamai facility handling my www.spamcop.net requests.

My next steps, from a suggestion by Don at admin, would have been to clear browser cache and shut down and restart my browser, test; shut down and restart my connection, test; shut down and restart the machine, test.

Good luck with your connection. Time for a Pilsener, here.

[Wazoo]

Start>Run>Cmd>tracert www.spamcop.net It is easier to specify the command than to say it should be done. Results:

Tracing route to a369.g.akamai.net [216.151.132.18]

over a maximum of 30 hops:

1 29 ms 37 ms 35 ms h-67-101-213-1.snfccasy.dynamic.covad.net [67.101.213.1]

2 49 ms 83 ms 39 ms 192.168.36.101

3 44 ms 39 ms 28 ms ge-8-0-181.ipcolo1.SanFrancisco1.Level3.net [63.211.150.25]

4 39 ms 30 ms 46 ms so-6-1-0.mp2.SanFrancisco1.Level3.net [4.68.96.145]

5 29 ms 40 ms 31 ms ae-0-0.bbr1.SanJose1.Level3.net [64.159.1.129]

6 42 ms 39 ms 40 ms ae-14-55.car4.SanJose1.Level3.net [4.68.123.142]

7 38 ms 43 ms 38 ms gblx-level3-te.SanJose1.Level3.com [4.68.111.162]

8 57 ms 41 ms 43 ms 67.17.193.138

9 39 ms 42 ms 41 ms 216.151.132.18

Just noting that my traceroute took me to 205.161.6.6 via Sprintlink servers as compared to your path through Level3 servers ....

As stated elsewhere, the Akamai scenario deals with geographicly situated servers (and paths) .. thus the "works from here" / "doesn't work from there" situations.

[Farelf]

Thanks for your attention and support. I learned a bit and any day that happens, without utter disaster being involved, is not a bad day.

Thanks Jim, glad it worked for you, you may be right about the FAQ (there again, these forums are long-lived and searchable and provide many answers too). But in any case

...Start>Run>Cmd>tracert www.spamcop.net It is easier to specify the command than to say it should be done. ...

can't slip by without a comment/correction. Do you remember?

...Yes, any number of folks could advise, suggest, explain how to do/use a 'traceroute' command thing, but .... you have yet to follow any of the suggested guidance on "How to post a question ..." and provide any data on what you might be using to make this connection. You'd do it one way if you were using Win-98, another way if you were using Win-XP, another way if using OS-X, a different way if using a Linux distribution/install .....

It was a fairly sure bet the DOS command-line method would work but "we" can waste a lot of time in similar instances before finding out that ain't the case and totally bamboozling the poster in the process. At the end of the day it was neat that you got your own help if you couldn't play along on that and thanks for reporting back. Maybe we need to add a prominent FAQ on how to obtain system specifications - that first step is possibly the killer for many. Now to see about those who haven't made it through to the spamcop.net pages yet ... maybe Don's advice that you have thoughtfully passed on (many thanks for that) would do the trick for DavidT? David?

[Wazoo]

Start>Run>Cmd>tracert www.spamcop.net It is easier to specify the command than to say it should be done.

As I stated prior, Farelf talks to .... this sequence wouldn't work on 90%+ of the computers currently surrounding me ..... as a matter of fact, trying to peer through the maze and stacks of stuff, that sequence would only work on one system out of five towers, three desktops, four laptops currently powered up .... the other two systems it would work on are actually running Live-CD Linux distros at present ..... only one system running Win-XP .. thus my remarks about your not providing enough data on your setup.

Only the XP machine has a "Start" button ....

Win-98 doesn't know "Cmd"

OS-X doesn't know "Cmd"

Linux doesn't know "Cmd"

Microsoft spells it "tracert"

Most others spell it "traceroute"

I've got it aliased on a couple of systems to "trt"

[Jim Pivonka]

As I stated prior, Farelf talks to .... this sequence wouldn't work on 90%+ of the computers currently surrounding me ..... as a matter of fact, trying to peer through the maze and stacks of stuff, that sequence would only work on one system out of five towers, three desktops, four laptops .... the other two systems it would work on are actually running Live-CD Linux distros at present ..... only one system running Win-XP .. thus my remarks about your not providing enough data on your setup.

I see said the blind man, as he picked up his hammer and saw.... I am far to Windows centered, for sure, and should have provided that critical OS information. I have Aida here, and recommend it and similar programs to others, but erred in not realizing the OS information was needed to help me run internet diagnostics.

Thanks again for your time and assistance.

[DavidT]

DavidT? David?

Using a cleared browser cache didn't do it...I'll try the other two options eventually, but....the problem is obviously "out there" and not "right here" at my PC's connection. I shouldn't have to jump through hoops to make a connection to a remote site work properly....it's just that simple.

DT

[Wazoo]

Using a cleared browser cache didn't do it...I'll try the other two options eventually, but....the problem is obviously "out there" and not "right here" at my PC's connection. I shouldn't have to jump through hoops to make a connection to a remote site work properly....it's just that simple.

The flip side to that is the amount of money flowing to Akamai to provide that 'service' .....

In Akamai's defense, thousands upon thousands of installed, co-located, leased, whatever servers scattered all around the world .. I certainly have no real idea just how one would/could monitor 'all' of them every second of the day ... the last 'major' issue seemed to be UK connection, to which Ellen had asked for user input to pass data back to Akamai ... definitely going back to an issue with oversight/monitoring of 'all' Akamai assets ....

In IronPort's defense, the actual SpamCop.net servers are (typically/generically) running just fine .. they don't/can't see that an Akamai server in another part of the country/world isn't passing bits to a user elsewhere ...

The bad part to me is that the magic 'number' that shows up in the oft-posted error message has to mean something .... else why are they all so long and different ?? My queries about those numbers have never been answered by Staff, Akamai won't talk to me about it, I never found this sort of data on Akamai's site ... so I'm again left with the sensation that "it's all Greek" to the Deputies ... or maybe it really doesn't mean squat, going back to Ellen asking for traceroute data from affected users ????

[DavidT]

And now things seem to be working again from here, without any action on my part. However, a few minutes or hours from now, it could crap out again...that's what been going on most of the weekend. The "Reporting Server Status" graph is suspiciously "flat" over the last 24 hours. There's usually more variation. Seems that would be indicative of some issue with people's ability to interact with the system.

DT

[Wazoo]

The "Reporting Server Status" graph is suspiciously "flat" over the last 24 hours. There's usually more variation. Seems that would be indicative of some issue with people's ability to interact with the system.

Yeah, but ..... look at it a bit closer and start conjecturing on the how-comes of the recently increased white-space gap between submittals and reports ...... yet another area not open to any useful discussion ...

BTW: someone just posted into the newsgroups about a small issue of not being able to connect .... pointed to 'here' in my reply ... spamcop.help if I remember right ...

[DavidT]

Looking at the "Weekly" graph this morning shows an even more pronounced abnormal pattern for the weekend. On any given day, there should be a bit of a "mountain" in the middle, given when the majority of the users of the system are awake and processing their emails. However, starting Friday night, you'll see an odd pattern where the lines trend down and then there's an almost instantaneous jump (in some cases, a tripling of the number) and then a gradual trend downward, until the next "jump." So, it would seem that the connectivity issues were more pronounced than just the few people who have reported problems here and in the ".help" NG.

DT

[Mike C.]

BTW: someone just posted into the newsgroups about a small issue of not being able to connect .... pointed to 'here' in my reply ... spamcop.help if I remember right ...

And I made over 'here', I've got to remember that these forums run on different servers than www.spamcop.net. Thanks for the redirect...

[shinerweb]

The flip side to that is the amount of money flowing to Akamai to provide that 'service' .....

As an incidental note to this, over the weekend, large parts of Akamai servers were down all over Australia.

Worldwide there was also major problems with Akamai specifically effecting Flickr (part of Yahoo, but still using Akamai).

And one more thing, doing a traceroute to Akamai servers isn't as straight forward all the time.

At my last company we had a deal with Akamai which allowed the to place a few co-located servers in one of our data centers. If you pinged 'that specific' Akamai machine, it would have appeared to have come from company X address space, not Akamai. Only if the server identified itself would you know it was an Akamai server. They would use these machines for control rather than data routing.

All I know that during the period you guys were talking about this problem, there were major problems on the other parts of Akamai servers.

(I'm off to have a lay down after my foray into the Forum... I feel weak now )