Webmail down?

[DavidT]

Good analysis, Wazoo...SpamCop.net is not being maintained properly, which is infortunate. It would be the ideal place for posting a system status issue regarding SpamCop Mail, Forum, etc., but that will probably never happen....it's just too logical.

Regarding the newsgroups....I discovered that just ".help" is apparently mirrored (?) to a GoogleGroups-accessible Usenet server:

http://groups.google.com/group/spamcop.hel...s?hl=en&lnk

When everything was inaccessible from here, I found that resource and posted to it, but then suddenly realized that I had done so from my "day job" identity, potentially exposing my work email address to harvesting, so I issued a cancel. The message showed up in the JT-hosted-NNTP version of the group, and the cancel shows up in "control.cancel" so I'm hoping that little long-term harm was done to the spam exposure of my work address.

p.s. - Steven -- I was composing this while you posted....would you please edit out my ".help" post from your CODE section? Trying to protect that address, and that doesn't help!

DT

[StevenUnderwood]

p.s. - Steven -- I was composing this while you posted....would you please edit out my ".help" post from your CODE section? Trying to protect that address, and that doesn't help!

Done. Actually pulled out all of the email addresses as they are really not needed. Sorry.

[DavidT]

Thanks. As for why my experience was different here...that's hard to say. If it were only a DNS issue, then my browser shouldn't have been able to resolve the addresses, and the error messages should have reflected that. During the outage, I also attempted command-line email deliveries to the MX servers, but I did so from a web-hosting box on the East coast, and those attempts also timed out, indicating not DNS issues, but actual connectivity issues.

DT

[jefft]

Thanks. As for why my experience was different here...that's hard to say. If it were only a DNS issue, then my browser shouldn't have been able to resolve the addresses, and the error messages should have reflected that. During the outage, I also attempted command-line email deliveries to the MX servers, but I did so from a web-hosting box on the East coast, and those attempts also timed out, indicating not DNS issues, but actual connectivity issues.

In fact, the problem is much stranger and obscure than that.

We have very good DNS. spamcop.net DNS is handled by Akamai globally. cesmail.net DNS is handled by a set of four different nameservers, in four datacenters, in both North America and Europe. There was no problem with the internet at large reaching our DNS.

The real problem was with our servers internally reaching these nameservers. We use them ourselves to resolve names used internally within the system. We have resolving cache nameservers but they ultimately look up our names on the same servers you guys do. For a time Friday evening, our caches couldn't get to our own nameservers and couldn't look up IP addresses, like the IP address of the database server.

We've been informed now that upstream from our data center, one provider had problems reaching a broken connection to Level3. Apparently they were accepting traffic but it was disappearing. The upstreams went back and forth for a while before our data center did an emergency failover and took all traffic away from those guys. At that point, the problem instantly went away.

My guess is that inbound traffic came in a different route and that reply packets went back out that route. So, we were mostly reachable. However, transactions that we initiated (like DNS queries) mostly went out that broken route.

Anyway, this all happened several layers upstream of us so I don't have any more visibility into the problem. This is probably about all the info I'll ever get and I wasn't able to do any real network debugging while it was going on.

JT

[DavidT]

Thanks for the details, JT. Thankfully, the "disturbance in the force" was pretty short-lived.

DT