derrick.hansen Posted August 25, 2006 Posted August 25, 2006 We are a smallish cable ISP, our mail server 208.98.210.10 has been on and off the blacklist for the last week like 5 times. we are listed again after we thought we resolved the issue. We thought one of our customers who is running an exchange server and using ours for outbound was having some bounceback messages nabbed by the spamtrap. We pinned that down. We have also found a couple virused hosts. they have been killed. was wondering if any of the paid members can see anything more detailed about what was getting us listed?
Wazoo Posted August 25, 2006 Posted August 25, 2006 I'm very appreciative of the way you posted the question. http://www.spamcop.net/w3m?action=checkblo...p=208.98.210.10 Causes of listing System has sent mail to SpamCop spam traps in the past week (spam traps are secret, no reports or evidence are provided by SpamCop) SpamCop users have reported system as a source of spam less than 10 times in the past week http://www.senderbase.org/?searchBy=ipaddr...g=208.98.210.10 Volume Statistics for this IP Magnitude Vol Change vs. Average Last day ........ 3.7 ... 15% Last 30 days .. 3.3 .. -45% Average ........ 3.6 That's as far as I can go right now ....
derrick.hansen Posted August 25, 2006 Author Posted August 25, 2006 Thanks. We think we found a virused host that at times had over 200 socks connections at once. We think this customer is the cause of our issues. I really hope that shutting this guy off keeps us off the blacklist. our customers are starting to get annoyed Yeah I was afeared that it might have been hiting the secret spam traps. Problem is that because they are secret there aint too much info provoided. IF this virused host was not doing it, i guess you will be hearing from me. Thanks for the help.
Telarin Posted August 25, 2006 Posted August 25, 2006 If it was indeed spamtrap hits, the users here aren't going to be too much help other than general suggestions, as we don't have access to any more spamtrap data than you do. Your best bet for spamtrap hits is to fire an email to deputies[at]spamcop.net. Just out of curiosity, do you have access to the abuse[at]sunwave.net account? It looks like that is where the (non-spamtrap) reports are being sent. If you aren't receiving anything there, then spamtraps are your most likely culprit. Some ISPs have taken to blocking port 25 connections to addresses outside their IP space unless a customer specifically requests it. Kind of a big hammer, but as long as you make your support guys aware of it, and don't make it too big a hassle for customers to get it opened up, I think most customers understand. This helps combat many of the viruses, as most of them do direct to MX connections rather than looking for an SMTP server to relay through. As far as the non-spamtrap items go, check the abuse[at]sunwave.net mailbox as it should be receiving current reports. As far as past reports, one of the paid reporters will give us a list of them shortly I am sure. And I would have to second Wazoo's response. Thank you for coming here with a good attitude, asking an intelligent question, and providing us with the information to try our best to help. There are so many people who come here with nothing other than a rant as to why they don't like spamcop, they forget that spamcop has no control over how their list is used or misused by ISPs.
derrick.hansen Posted August 25, 2006 Author Posted August 25, 2006 yes, we do recieve non-spamtrap reports to our abuse address which i have access to. I also have reports being sent to my personal address. we currently do not accept port 25 to our mail server from outside our network. It has been that way since before i was here. It helps. In this case I think this virus just hijacked this guys mail client and managed to work around our anti-spam rules eg. no more than 40 messages at a time, no more than 1000 per day.
StevenUnderwood Posted August 25, 2006 Posted August 25, 2006 yes, we do recieve non-spamtrap reports to our abuse address which i have access to. I also have reports being sent to my personal address. we currently do not accept port 25 to our mail server from outside our network. It has been that way since before i was here. It helps. In this case I think this virus just hijacked this guys mail client and managed to work around our anti-spam rules eg. no more than 40 messages at a time, no more than 1000 per day. Most recent report is showing as Tuesday. The last bounce to a spamtrap (we think that is what the uube[at]devnull.spamcop.net reports are) is also Tuesday. Sorry not much help there. Currently, non-spamtrap reports are only being sent to: Reporting addresses:abuse[at]sunwave.net The routing sees your support[at]sunwave from abuse.net but ignores it for some reason. Report History: -------------------------------------------------------------------------------- Submitted: Tuesday, August 22, 2006 5:51:39 PM -0400: Bachelors, Masters, MBA, PhD can be yours in 4 weeks if you qualify. 1887857680 ( 208.98.210.10 ) To: spamcop[at]imaphost.com 1887857671 ( 208.98.210.10 ) To: abuse[at]sunwave.net -------------------------------------------------------------------------------- Submitted: Tuesday, August 22, 2006 2:03:29 PM -0400: Undelivered Mail Returned to Sender 1887585552 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net -------------------------------------------------------------------------------- Submitted: Wednesday, August 09, 2006 1:44:29 PM -0400: Delivery Status Notification (Failure) 1870365870 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net -------------------------------------------------------------------------------- Submitted: Tuesday, August 08, 2006 10:24:39 PM -0400: Undelivered Mail Returned to Sender 1869483845 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net -------------------------------------------------------------------------------- Submitted: Tuesday, August 08, 2006 6:11:04 PM -0400: Undelivered Mail Returned to Sender 1869284938 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net -------------------------------------------------------------------------------- Submitted: Monday, July 24, 2006 10:06:06 PM -0400: Undelivered Mail Returned to Sender 1850467640 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net -------------------------------------------------------------------------------- Submitted: Sunday, June 11, 2006 10:48:21 AM -0400: Returned mail: User unknown 1789903165 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net Older Reports Moderator Edit: added the prefix "non-" to prevent some confusion .....
Telarin Posted August 25, 2006 Posted August 25, 2006 Hmm, yep, not much help in the manual reports then. So that pretty much leaves spamtraps. I would fire an email over to the deputies (deputies[at]spamcop.net) and see if they can tell you what kind of traffic they are seeing at the spamtraps.
Wazoo Posted August 25, 2006 Posted August 25, 2006 Plenty of evidence at http://psbl.surriel.com/evidence?ip=208.98...=Check+evidence The disturbing thing though are the lines; spam and removal history for 208.98.210.10 (times in UTC): 2006-08-20 01:21:42.990509 received spamtrap mail 2006-08-20 02:16:50.474946 received spamtrap mail 2006-08-20 02:17:01.565076 received spamtrap mail 2006-08-21 13:13:49.919842 received spamtrap mail 2006-08-21 15:35:34.51922 removed through website Someone knows about this site ....????? However, this collection may be from the bad stuff already removed ....
Telarin Posted August 25, 2006 Posted August 25, 2006 One good thing I see about all those message is that their SMTP server is stamping a Received line with the IP its picking them up from, so should be easy to trace them back to a particular customer. Don't see anything there newer than Aug 20th, which I believe was an issue the OP said they already addressed. I am noticing something odd though... Received: from [208.98.210.10] (helo=lithium.sunwave.net) by angband.surriel.com with esmtp (Exim 4.43) id 1GEgcD-00022W-Jw for victim[at]smtp.example; Sun, 20 Aug 2006 02:16:58 -0400 Ok, last hop from SMTP server to its destination as we would expect Received: from 208-98-202-239.cable.dynamic.sunwave.net (208-98-202-239.cable.dynamic.sunwave.net [208.98.202.239]) by lithium.sunwave.net (Postfix) with SMTP id 498E343F0B for <victim[at]smtp.example>; Sat, 19 Aug 2006 22:56:50 -0700 (PDT) I assume this is the SMTP server receiving the email from 208.98.202.239, apparently a dynamic IP address in the ISPs range, again normal for anyone using the SMTP server as a smarthost. Received: from [208.98.202.239] (port=5374 helo=208-98-202-239.cable.dynamic.sunwave.net) by mail.asiafreeport.com with esmtp id 1a2lC2-ycP765-87 for victim[at]smtp.example; Sat, 19 Aug 2006 22:56:37 --800 Received-SPF: none (mail.asiafreeport.com: 208.98.202.239 is neither permitted nor denied by domain of asiafreeport.com) client-ip=208.98.202.239; envelope-from=victim[at]smtp.example; helo=208-98-202-239.cable.dynamic.sunwave.net; This is where things get squirly. It appears 208.98.202.239 (mail.asiafreeport.com) is receiving mail from itself. That means either some kind of odd relay attack from a compromised machine to its own SMTP service, or more likely, abuse of some kind of webform hosted on the same server. There are certainly other explanations, and as Wazoo pointed out, this may be referring to the issues that have already been taken care of, but if not, this might be the first place to look.
derrick.hansen Posted August 25, 2006 Author Posted August 25, 2006 wow you guys are good. That was the customer we shut down that was our prime suspect. This just validates our suspicion. Thanks for the help!
Wazoo Posted August 25, 2006 Posted August 25, 2006 wow you guys are good. That's odd .... I'm so much more accustomed to the "you are a bunch of &^%$#, *&^%[at], or even worse .. &***%$[at](&!!!" <g> That was the customer we shut down that was our prime suspect. This just validates our suspicion. Thanks for the help! On the flip side, kudos on being there and doing a good job.
derrick.hansen Posted August 25, 2006 Author Posted August 25, 2006 Being a support person, who soaks up abuse from users daily, I would never dare sling hostility toward people that are in a position to help. Well, ok i might, But only if they really deserve it. *cough* AT&T blacklists *cough* NTL world.
mason Posted August 25, 2006 Posted August 25, 2006 Hello, I'm the systems administrator at the ISP Derrik.Hansen works at. I wanted to throw a few more questions out there and clarify a few things. Hopefully someone on this list has some suggestions. Our smarthost (lithium.sunwave.net - 208.98.210.10) has been blacklisted a few times over the past few days. The spam has not originated from our smarthost, it has simply been the relay for systems on our network that are permitted to relay through our smarthost. The first instance was from an exchange box that had been configured to relay through us some time ago, but just this week, their antispam filter died. The result was exchange doing asynchronus bouncing and the bounced spam messages being relayed through our smarthost.... The problem was identified and the customer was asked not to send their email on their own rather than through us. They took a little too long to comply (got us blacklisted again) and I ended up blacklisting them - they are now sending their own mail The second instance was from a host that got infected with a bot and began relaying through our smarthost. This has happened in the past and I always have to quickly identify the culprit, have one of our support guys shut them down and then we start sending delist requests to the blacklists we have found ourselves on. So, that's what we have been dealing with. I have two questions: 1 - Does spamcop intend to block ISP smarthosts? I was reading through the spamcop faq today and saw this faq entry http://www.spamcop.net/fom-serve/cache/99.html It suggests that relays shouldn't get blacklisted, that this is "relay rape". I fail to see why that would be; if my relay is forwarding spam, the only way to block the spam using a RBL is to block the relay. Seems fair to me. However, if spamcop doesn't intend to block relays and there is some way that I can reconfigure the system to take advantage of that, I will.... No sense in making myself go grey if someone hands me a get out of jail free card I have a feeling I'm just misinterpreting what was stated in that faq, which brings me to my next question. 2 - What can be done about bots relaying through a smarthost? I have given this particular issue a great deal of thought and actually have a plan for dealing with the problem, I just need to find time to implement my plan. However, if anyone in this forum has any fresh ideas that I may have missed, I'd love to hear them. I'd also welcome any debate concerning problems with my plan. So, here's the plan: -------------------------- Note: We have very effective inbound filtering through Postini, so, as you'll note in the plan, I don't waste any CPU cycles on inbound spam filtering on our mailserver. I'm going to start implementing the following steps. It's quite a big list and I expect that I'll implement everything up to step 8, but beyond that I'm not sure. I'll almost certainly implement step 12 on our dynamic networks, but I'll leave our business networks alone. Step 13 is already done. Feedback from this list may help me decide what steps I skip. 1 - Setup new mail server with 2 instances of the latest postfix (2.3) http://www.advosys.ca/papers/postfix-instance.html Also check out the postfix docs, I seem to remember code being added to make the dual instance setup much easier to implement - one for inbound connections from the net and one for outbound connections from our customers. - inbound will do basic postfix UCE checks, but nothing fancy - absolutely no SA/ClamAV filtering - well, maybe I could be convinced to do a little bit of SA on inbound. This ruleset would be so nice to have and couldn't do any harm to legit email http://www.timj.co.uk/linux/bogus-virus-warnings.cf - outbound will be much more intensive (see below for details) - change our MXes to point to lithium.sunwave.net (new IP) - have all clients point at mail.sunwave.net (current IP) for outbound smtp and pop3/imap - this will make it much easier to have different filter rules and SA configs for inbound vs outbound mail. As well, this makes it a bit easier to break the mail system into two hosts, in the future, if need be (likely due to scanning software overhead or imap overhead) - having a separate postfix instance for outbound (on it's own IP) also makes it easier to buy myself some time, by just switching IPs if we get blacklisted again (this is what Telus does) 2 - Install local authoritative and caching name servers so that RBL checks are much faster. As well, this will ensure that DNS changes for domains we host are seen by the mail server immediately. 3 - Implement postfix *outbound* rate limiting controls on a per client basis http://www.postfix.org/anvil.8.html /etc/postfix/samples/sample-rate.cf - intention is to slow down bot spam so that the inline scanner doesn't have to work so hard and we also reduce the risk of getting black listed 4 - Tighten up postfix a fair bit - especially outbound! - http://jimsun.linxnet.com/misc/postfix-anti-UCE.txt - Do what is realistic using postfix native UCE controls so that less load need be placed on the inline scanner. - I would love to do this http://www.postfix.org/postconf.5.html#smt...unlisted_sender but then we would have the problem of non local users being rejected by our relay. Is there a way around this? - Could I enforce strict_rfc821_envelopes or would this break some clients? Try using a warn_if_reject to see. 5 - publish spf records for all our domains (sunwave, sunlite, shuswap) - start simple and conservative http://spf.pobox.com/wizard.html sunwave.net. IN TXT "v=spf1 ip4:64.141.16.1/27 mx a:lithium a:mail ?all" lithium.sunwave.net. IN TXT "v=spf1 a -all" mail.sunwave.net. IN TXT "v=spf1 a -all" - put in some logging so that I can see who is using servers other than what I have specified Page 15 - http://spf.pobox.com/whitepaper.pdf - test them http://spf.pobox.com/faq.html#checkers 6 - Implement inline transparent spam filters on *OUTBOUND* mail Use spampd or smtpprox and spamassassin - return an error code and message I think that spampd is probably the better option http://www.postfix.org/SMTPD_PROXY_README.html http://www.worlddesign.com/Content/rd/mta/...xspecific_notes Don't forget to use XFORWARD!!! If for some reason, spampd is not logging the client IP, then consider either patching it or using smptpprox (patching should be dead easy as spampd uses the same code as a base as smtpprox) http://bent.latency.net/smtpprox/index.html - this is going to take a fair bit of tuning to get right, because we will be focusing on outbound rather than inbound; this is a non standard approach. Because we are focusing on outbound, I think that we can give more points to matches from the blacklists (see below for RBLs to lean on). I'm sure there will also be other settings that will make sense to lean heavier on due to the outbound approach (perhaps spf?) 7 - Implement post queue antivirus scanning on *OUTBOUND* mail Use amavisd-new and clam - drop viruses on the floor with no notifications Perfect! - http://workaround.org/articles/ispmail-sarge/#amavis 8 - Implement Intra domain routing through postini http://www.postini.com/admindoc/intra_domain.html - this setup increases the effectiveness of postini for user to user and gives us free user to user email virus scanning! - basically, don't let the outbound smtp server think it's authoritative for local domains 9 - Start generating outbound usage stats so that I can see/be notified of suspicious trends. Specifically on a per user basis. - Parse the outbound server's logs looking for: - lots of blocks by spampd/spamassassin - lots of blocks by amavisd-new/clamav - # of connections / recipeints per min - probably be good to do this every 5-15min or so - statistically large increases in mail output from a given IP (would require software that tracks over much longer periods of time) # if software exists, great, if not, then don't go here, we don't have the time for this sort of thing - If the parser sees something that looks nasty, it will send an email to the ticket system and the ticket system will email the customer. This should allow me to better tune anvil as well. 10 - Optimize postfix to make it more friendly to our customers - Protect customers against "back scatter" http://www.postfix.org/BACKSCATTER_README.html - create useful error responses - look in the /etc/postfix/samples directory for lots of great ideas http://www.mailsbroadcast.com/email.bolts&...codes.error.htm - run postfix 2.3 so that I can do DSN http://www.postfix.org/DSN_README.html http://archives.neohapsis.com/archives/pos...05-03/2016.html 11 - Tune amavis/clam/spamassassin performance http://wiki.apache.org/spamassassin/FasterPerformance http://wiki.apache.org/spamassassin/SitewideServerSpec http://www.ijs.si/software/amavisd/README.performance 12 - Ingress/Egress port 25 block - for all dynamic networks - leave static networks alone 13 - Change our reverse dns so that dynamic ranges are clearly identified http://www.onlymyemail.com/company/press_m...Threat_RBL.html http://www.spamhaus.org/faq/answers.lasso?...am%20Issues#129 - ex - 64-141-16-142.dialup.dynamic.sunwave.net - ex - 64-141-19-234.cable.dynamic.sunwave.net - ex - 64-141-18-234.business.static.sunwave.net 14 - SMTP AUTH http://spamlinks.net/prevent-secure-relay-auth.htm http://postfix.state-of-mind.de/patrick.ko...auth/index.html use port 587, tls and any auth method or port 465, no tls with crammd5 14.5 - SMTP AUTH would be hard as it would require us to contact all customers and have them change their outgoing mail settings (have you heard the term "kicking dead whales down the beach"? - it would apply..... So, rather than go with SMTP AUTH, I could setup something like camram, TMDS, or another challenge response tool (I hate challenge response, but this might work). Again, this would be for outbound traffic only - not inbound. Now, rather than just set the challenge response tool loose, I would preload the tool with all our valid customer addresses. That means that the only people that will get challenges will be those that are our customers, are using our smarthost, but do not have an email address on our system. This should be fairly pain free. 15 - Address security for all hosts on the network for all services, not just mail - Create an automated lepper colony that tosses known infected users into it and directs all traffic to a webpage, sends them an email - start doing regular vulnerability scans of the network using nessus or better yet setup ossim. Create a monthly top 10 report and have TSRs contact people and try to educate them - use adaptive response on the shaper to create tickets for users that have constant socks traffic - Do the big media blitz and get people up to speed with the big four defences - firewall, anti-virus, anti-spyware, windows updates. Get as many people as possible to upgrade to XP SP2, - move people to firefox and thunderbird. Also encourage people to tell their friends. - seriously look at tools such as the fortinet appliance - focus on antispyware - consider other near term gateway approaches such as using the new spyware RBL that someone at bleeding snort is working on. Or start blocking known spyware user-agent strings at our cache or shaper. -- Mason Schmitt
Wazoo Posted August 25, 2006 Posted August 25, 2006 I was supposed to be elsewhere an hour ago, so short answer here .... Item 6 (the client's IP address bit) is the most 'functional' item bit there .... No, the SpamCopDNSBL's intent is not to snag your smarthost .. however, if the 'chain-test' of the parsing engine cannot proceed beyond that Received: line, that's where it has to stop .... the nastiest of these examples are found in the GMail servers blocked massive Topic, Yahoo servers blocked massive Topic .... the same 'problem' exists there .. the source of the spew cannot be tracked beyond the output server due to the non-existent Recieved: lines that show the 'real' source of the problem e-mail .....
StevenUnderwood Posted August 25, 2006 Posted August 25, 2006 Note: We have very effective inbound filtering through Postini, so, as you'll note in the plan, I don't waste any CPU cycles on inbound spam filtering on our mailserver. Since you are already a customer, have you investigated Postini's outbound service and how it works? My company is also a Postini customer for incoming only so I do not know how this works exactly, but we are a single server shop with less "problem areas" to address.
Telarin Posted August 28, 2006 Posted August 28, 2006 From the couple of sample mails we looked at in the evidence posted earlier, it looks like your server is already stamping the "source" IP address, or customer IP, of the message. I believe all spamcop needs to do is add your MX address to its list of trusted addresses so that it will parse up to the next received line. I would suggest emailing the deputies and asking them if they can do this, as it is beyond what anyone here has access to. I don't know what their requirements are for this, but I know there are a number of MXs that I have seen it "trust" and proceed on to the next Received header for the source.
mason Posted August 28, 2006 Posted August 28, 2006 From the couple of sample mails we looked at in the evidence posted earlier, it looks like your server is already stamping the "source" IP address, or customer IP, of the message. I believe all spamcop needs to do is add your MX address to its list of trusted addresses so that it will parse up to the next received line. I would suggest emailing the deputies and asking them if they can do this, as it is beyond what anyone here has access to. I don't know what their requirements are for this, but I know there are a number of MXs that I have seen it "trust" and proceed on to the next Received header for the source. Ah, thanks! I didn't realize that was an option. I'll get an email out to spamcop asap. Since you are already a customer, have you investigated Postini's outbound service and how it works? My company is also a Postini customer for incoming only so I do not know how this works exactly, but we are a single server shop with less "problem areas" to address. Postini does not provide outbound filtering for their ISP service. We are however investigating upgrading to Enterprise edition in order to get this functionality. I'd rather rely on someone else playing the cat and mouse spam game rather than having to attempt to keep up on my own...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.