eric Posted July 22, 2004 Share Posted July 22, 2004 For a few hours now the MX servers for email filtering have not been accepting email from us. Both mx.cesmail.net and mx2.cesmail.net either don't connect on port 25, or when they do the transaction times out. Our email queues are getting filled up with message queued to be filtered by SpamCop email filtering. This is happening for our servers on both East Coast and West Coast, so it looks like a problem at the SpamCop email end. Any clues? Link to comment Share on other sites More sharing options...
Wazoo Posted July 22, 2004 Share Posted July 22, 2004 Can you provide some error messages, log entries, something ???? That you say "a few hours" would seem to suggest that there'd be all kinds of entries in the newsgroups and possibly "here" ... but you've the only post thus far ... Link to comment Share on other sites More sharing options...
DavidT Posted July 22, 2004 Share Posted July 22, 2004 Eric is correct....there is clearly a current problem with the Mail Exchangers for "spamcop.net" and "cesmail.net" mailboxes. I just tried a command-line SMTP connection to port 25 of mx.cesmail.net from a webserver and Spamcop isn't answering. Here's what I'm seeing: $ telnet mx.cesmail.net 25 Trying 216.154.195.36... I initiated that command before starting this response and there's still no response. No wait....it just officially timed out: telnet: Unable to connect to remote host: Connection timed out Now I'm trying mx2.cesmail.net: $ telnet mx2.cesmail.net 25 Trying 216.154.195.44... Same thing...no response. This is URGENT and JT should be paged. I guess I'll post to the Usenet groups and send email directly to JT. David T. Link to comment Share on other sites More sharing options...
Merlyn Posted July 22, 2004 Share Posted July 22, 2004 Yup looks like email is down, Webmail is down. Just have a little patience everyone Link to comment Share on other sites More sharing options...
DavidT Posted July 22, 2004 Share Posted July 22, 2004 Same thing...no response. This is URGENT and JT should be paged. I guess I'll post to the Usenet groups and send email directly to JT. 13714[/snapback] Oops....can't send email to JT or even "deputies <at> spamcop.net" because obviously, the message won't get through at the moment. Link to comment Share on other sites More sharing options...
eric Posted July 22, 2004 Author Share Posted July 22, 2004 Can you provide some error messages, log entries, something ???? That you say "a few hours" would seem to suggest that there'd be all kinds of entries in the newsgroups and possibly "here" ... but you've the only post thus far ... 13710[/snapback] Sure! The first failure in our mail log is at 9:06 PDT (NOTE: I've munged the spamcop subscriber address, and wrapped the lines, the actual log entry is all one long line): Jul 22 09:06:26 mail sendmail[8345]: i6MG4Nn08344: to=munged[at]spamcop.net, delay=00:02:02, xdelay=00:02:00, mailer=esmtp, pri=90761, relay=mx2.cesmail.net. [216.154.195.44], dsn=4.0.0, stat=Deferred: Connection timed out with mx2.cesmail.net. Since then every connection attempt to forward email for filtering generates a similar log entry. In the 2.5 hours since that first failure, there have been 76 failed attempts while running the queue (sendmail -bd -q30m). Even telnet doesn't work: eric$ telnet mx.cesmail.net 25 Trying 216.154.195.36... telnet: connect to address 216.154.195.36: Operation timed out telnet: Unable to connect to remote host eric$ telnet mx2.cesmail.net 25 Trying 216.154.195.44... telnet: connect to address 216.154.195.44: Operation timed out telnet: Unable to connect to remote host eric$ When I try to run the queue manually, I get this: /usr/lib/sendmail -q -v Running /var/spool/mqueue/i6MHeVn08827 (sequence 1 of 12) munged[at]spamcop.net... Connecting to mx.cesmail.net. via esmtp... 220 mailgate.cesmail.net ESMTP >>> EHLO mail.abcstuff.com 250-mailgate.cesmail.net 250-PIPELINING 250-8BITMIME 250 SIZE 0 >>> MAIL From:<x> SIZE=2536 250 ok >>> RCPT To:<munged[at]spamcop.net> 250 ok >>> DATA 354 go ahead >>> . 451 connection to mail server timed out (#4.4.1) munged[at]spamcop.net... Connecting to mx2.cesmail.net. via esmtp... munged[at]spamcop.net... Deferred: Connection timed out with mx2.cesmail.net. Running /var/spool/mqueue/i6LLxAn04832 (sequence 2 of 12) munged[at]spamcop.net... Deferred: Connection timed out with mx.cesmail.net. munged[at]spamcop.net... Deferred: Connection timed out with mx2.cesmail.net. ^C I really think there's a problem! Link to comment Share on other sites More sharing options...
eric Posted July 22, 2004 Author Share Posted July 22, 2004 Oops....can't send email to JT or even "deputies <at> spamcop.net" because obviously, the message won't get through at the moment. 13719[/snapback] Yes, exactly why I posted here instead of sending email! -Eric Link to comment Share on other sites More sharing options...
Wazoo Posted July 22, 2004 Share Posted July 22, 2004 Yes, agreed ... only know do the others start chiming in with reports <g> ... again, I was keying in on the "two hours" and nary a peep ... a bit strange that it took so long for complaints to start showing up ... Link to comment Share on other sites More sharing options...
eric Posted July 22, 2004 Author Share Posted July 22, 2004 Yup looks like email is down, Webmail is down. Just have a little patience everyone 13716[/snapback] I dunno about webmail, I think there are other threads scattered about regarding gateway timeouts and webmail login problems. This is a different symptom, although depending on the network topology and distribution of workload among servers, the two symptoms might very well be related. The MX server failing to converse is not webmail-specific at all. It's email filtering: a user's email is forwarded (.forward, alias, or other mechanism) to a spamcop address. The spamcop address is set up (using webmail for configuration) to filter the mail, and holds potential spam but forwards the ham to a secret email address back at the user's ranch. To process the held potential spam does require either webmail or VER, but the forward-filter-return loop is what's broken here. No webmail involved other than the fact that it's the SpamCop email side of the SC corpus callosum. Link to comment Share on other sites More sharing options...
Wazoo Posted July 22, 2004 Share Posted July 22, 2004 Not sure how many are aware of this factoid, but from a few days ago .... Jul 19, 2004 [18:59 EDT] We have started the process of moving user email onto a new set of fileservers. If you keep mail stored on our servers, it might appear that some of your mail has disappeared for a bit while your mail is being moved. This should last only a minute or so for each user, depending on how much they have stored here. As found at http://mail.spamcop.net/news.php So, there might be some teething problems going on also ...??? Link to comment Share on other sites More sharing options...
eric Posted July 22, 2004 Author Share Posted July 22, 2004 Yes, agreed ... only know do the others start chiming in with reports <g> ... again, I was keying in on the "two hours" and nary a peep ... a bit strange that it took so long for complaints to start showing up ... 13723[/snapback] Many would not notice it until some sender got a 4-hour non-delivery notice, and contacted the recipient by another channel. Someone here who does not use SC filtering got email that I did not receive, yet I was CC'd on it. In investigating, I saw that my copy was caught in the clogged outgoing sendmail queue, on its way to be filtered by SC. To a non-admin, the only observable symptom probably would be a couple of hours with no incoming email, and that's not necessarily abnormal. Link to comment Share on other sites More sharing options...
eric Posted July 22, 2004 Author Share Posted July 22, 2004 Not sure how many are aware of this factoid, but from a few days ago .... Jul 19, 2004 [18:59 EDT] We have started the process of moving user email onto a new set of fileservers. If you keep mail stored on our servers, it might appear that some of your mail has disappeared for a bit while your mail is being moved. This should last only a minute or so for each user, depending on how much they have stored here. As found at http://mail.spamcop.net/news.php So, there might be some teething problems going on also ...??? 13729[/snapback] Yes, I did see that, but this is 3 days later, and it's not clear that the description "moving user email" would apply to MX servers. It sounded to me more like they are/were moving stored user mailboxes to a bigger disk farm. Could be continued teething problems -- maybe someone kicked out a cable while rummaging around behind a rack! Link to comment Share on other sites More sharing options...
Wazoo Posted July 22, 2004 Share Posted July 22, 2004 For a bit of explanation, I don't use the e-mail side of the house, so the only clues I get are the newsgroup traffic and the posts in here ... and of course, keeping up with all that is mixed in with the rest of real life <g> .... I should be outside mowing the yard right now, actually ... well, either that ot get cracking on the three systems dropped off in the last hour and see just how screwed up they are <g> That "won't turn on" and "doesn't work" sure explains a lot <g> Link to comment Share on other sites More sharing options...
eric Posted July 22, 2004 Author Share Posted July 22, 2004 Hey, let's hear it for pagers! (no, I don't mean /usr/bin/less) Looks like the cesmail.net MX servers are no longer contemplating their own navels. Mail is going through, although a bit slowly -- several seconds per transaction, instead of a fraction of a second. We may never know what the problem was, or if it just healed itself. But it does appear to be working better now. Link to comment Share on other sites More sharing options...
DavidT Posted July 22, 2004 Share Posted July 22, 2004 I've re-routed my two major forwards so that they bypass SpamCop for now, because I can't afford to have messages go into limbo. However, I see that someone has created a new (emergency?) MX for "spamcop.net" addresses. Previously, the MX list was: spamcop.net preference = 5, mail exchanger = mx.cesmail.net spamcop.net preference = 10, mail exchanger = mx2.cesmail.net but one of my web servers now knows about a new one: spamcop.net preference = 5, mail exchanger = mx.cesmail.net spamcop.net preference = 10, mail exchanger = mx2.cesmail.net spamcop.net preference = 2, mail exchanger = c60.cesmail.net You'll note that the "preference" value on the new one is the lowest, meaning that connections should be first tried to that server. David T. Link to comment Share on other sites More sharing options...
DavidT Posted July 22, 2004 Share Posted July 22, 2004 Regarding the DNS for the Spamcop.net mail servers....there seem to be some problems. Take a look here: DNS Report for SpamCop.net You'll note that there are actually too many name servers for Spamcop.net at the present time. I've tried sending DNS queries to some of them, and they're not responding with valid information, IMO. David T. Link to comment Share on other sites More sharing options...
Wazoo Posted July 22, 2004 Share Posted July 22, 2004 Regarding the DNS for the Spamcop.net mail servers....there seem to be some problems. Take a look here: DNS Report for SpamCop.net You'll note that there are actually too many name servers for Spamcop.net at the present time. I've tried sending DNS queries to some of them, and they're not responding with valid information, IMO. 13735[/snapback] Actually, the "error" is a bit of fallout from using the Akamai service, and also this is for Julian's side of the house. for the most part, albeit members/mailsc/etc.spamcop.net would flow through these servers .... there's nothing really exciting about JT's side in that report. Link to comment Share on other sites More sharing options...
eric Posted July 22, 2004 Author Share Posted July 22, 2004 I see that someone has created a new (emergency?) MX for "spamcop.net" addresses. Previously, the MX list was: [...] spamcop.net preference = 2, mail exchanger = c60.cesmail.net Ah, time to refresh the cache on our DNS server! (or add a temporary alias for 'mx.cesmail.net' in our /etc/hosts file pointing to c60's IP address). Link to comment Share on other sites More sharing options...
eric Posted July 22, 2004 Author Share Posted July 22, 2004 Sigh. Now the SpamCop MX (c60.cesmail.net) is slurping up emails forwarded for filtering, but is not passing anything through. I'll bet the outgoing queue at the SpamCop end looks bad! As much as I hate to do it, I have to bypass the SpamCop filtering of our incoming email, and let all the spam through along with the real mail. Maybe it will work better tomorrow. Anyone else having better luck today with SpamCop pass-through email filtering? (specifically this is *not* the mailbox filters available with the webmail web interface) -Eric Link to comment Share on other sites More sharing options...
DavidT Posted July 22, 2004 Share Posted July 22, 2004 Now the SpamCop MX (c60.cesmail.net) is slurping up emails forwarded for filtering, but is not passing anything through. I'll bet the outgoing queue at the SpamCop end looks bad!13745[/snapback] This is also what I'm seeing....the three MX servers are accepting incoming SMTP sessions -- I've even done some command-line (aka "fakemail") SMTP sessions, talking directly to the servers -- but the mail isn't generally making it on to our POP boxes (or in your case, being forwarded on elsewhere?). Where the heck are the system admins, and why hasn't there been any sort of announcement posted anywhere????? This sucks. :angry: David T. Link to comment Share on other sites More sharing options...
StevenUnderwood Posted July 23, 2004 Share Posted July 23, 2004 Where the heck are the system admins, and why hasn't there been any sort of announcement posted anywhere? There are a total of 1 system admins (JT) for the email side of the house. And as part of your second question, I would prefer that he work on getting the systems back in operation to posting updates here. Link to comment Share on other sites More sharing options...
DavidT Posted July 23, 2004 Share Posted July 23, 2004 There are a total of 1 system admins (JT) for the email side of the house. Then that seems to be part of the problem here. We don't even know if he's on the job or not, do we? If the system relies on only one person, does the one person not sleep or take days off? From the scarcity of JT posts in these forums (especially during the last few months), he must have a lot of other irons in the fire besides the SC email system. And as part of your second question, I would prefer that he work on getting the systems back in operation to posting updates here. I don't think it's an either/or situation, Steven. I didn't ask for "updates" -- a simple message acknolwledging the severity of the problems would suffice, and surely the 30 seconds or so that it would take to post it wouldn't significantly impact the restoration of email functionality...if anyone is actually even working on it. Some people assume that JT is working on it, but we've little to no proof of that. David T. Link to comment Share on other sites More sharing options...
DavidT Posted July 23, 2004 Share Posted July 23, 2004 Somone just now posted the following at: http://mail.spamcop.net/news.php Jul 22, 2004 * [21:14 EDT] During the day today, we were under a large spam attack. This caused mail delays that aren't over yet. Overnight, we'll deliver the rest of the mail that is queued. Also, expect more spam than usual to get through today. Link to comment Share on other sites More sharing options...
turetzsr Posted July 23, 2004 Share Posted July 23, 2004 David, ...Although I'm not a user of the SpamCop e-mail system, I'm in general agreement with you. However, you may want to consider that, at best, you are preaching to the choir as no one here can do anything about the problem -- we're all users. Also, I believe that the "scarcity of JT posts in these forums" does not necessarily justify the (apparent; I apologize if I jumped to an incorrect assumption about your point) conclusion that JT doesn't frequent this forum to identify potential issues and problems that he needs to address. ...Another leap I'll make about your postings here (remember, I'm in agreement with you) is that they seem to imply that you believe there is a certain level of service that you feel entitled to expect and that this level is not being met. Is there something that has led you to this conclusion or are you just saying that a few hours' problem is not acceptable to you? Link to comment Share on other sites More sharing options...
StevenUnderwood Posted July 23, 2004 Share Posted July 23, 2004 If the system relies on only one person, does the one person not sleep or take days off? I don't own my own business and I rarely take a vacation without my pager. I doubt if JT get to (though I hope he does). That is part of life as a computer person in a small company. he must have a lot of other irons in the fire besides the SC email system. he things I know about are... the email storage server(s) (which were either just upgraded or are still being upgraded), the webmail server(s), the newsgroup server(s) and server(s) for this forum (which wer just upgraded) surely the 30 seconds or so that it would take to post it wouldn't significantly impact the restoration of email functionality Sometimes you think it is going to be a quick fix and suddenly you realize you have been working on something for several hours. Some people assume that JT is working on it, but we've little to no proof of that. You are right...we have no proof. We also assumed he was working on it the whole weekend of the pool incident. After that problem, there is supposed to be a backup person that JT also helps out at times in a work swap. As it turns out, someone was woking on it but we have no way to know that. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.