Jump to content

recurring blacklist issues.


derrick.hansen

Recommended Posts

We are a smallish cable ISP, our mail server 208.98.210.10 has been on and off the blacklist for the last week like 5 times. we are listed again after we thought we resolved the issue. We thought one of our customers who is running an exchange server and using ours for outbound was having some bounceback messages nabbed by the spamtrap. We pinned that down. We have also found a couple virused hosts. they have been killed.

was wondering if any of the paid members can see anything more detailed about what was getting us listed?

Link to comment
Share on other sites

I'm very appreciative of the way you posted the question.

http://www.spamcop.net/w3m?action=checkblo...p=208.98.210.10

Causes of listing

System has sent mail to SpamCop spam traps in the past week (spam traps are secret, no reports or evidence are provided by SpamCop)

SpamCop users have reported system as a source of spam less than 10 times in the past week

http://www.senderbase.org/?searchBy=ipaddr...g=208.98.210.10

Volume Statistics for this IP

Magnitude Vol Change vs. Average

Last day ........ 3.7 ... 15%

Last 30 days .. 3.3 .. -45%

Average ........ 3.6

That's as far as I can go right now ....

Link to comment
Share on other sites

Thanks. We think we found a virused host that at times had over 200 socks connections at once. We think this customer is the cause of our issues. I really hope that shutting this guy off keeps us off the blacklist. our customers are starting to get annoyed :P

Yeah I was afeared that it might have been hiting the secret spam traps. Problem is that because they are secret there aint too much info provoided.

IF this virused host was not doing it, i guess you will be hearing from me. Thanks for the help.

Link to comment
Share on other sites

If it was indeed spamtrap hits, the users here aren't going to be too much help other than general suggestions, as we don't have access to any more spamtrap data than you do. Your best bet for spamtrap hits is to fire an email to deputies[at]spamcop.net.

Just out of curiosity, do you have access to the abuse[at]sunwave.net account? It looks like that is where the (non-spamtrap) reports are being sent. If you aren't receiving anything there, then spamtraps are your most likely culprit.

Some ISPs have taken to blocking port 25 connections to addresses outside their IP space unless a customer specifically requests it. Kind of a big hammer, but as long as you make your support guys aware of it, and don't make it too big a hassle for customers to get it opened up, I think most customers understand. This helps combat many of the viruses, as most of them do direct to MX connections rather than looking for an SMTP server to relay through.

As far as the non-spamtrap items go, check the abuse[at]sunwave.net mailbox as it should be receiving current reports. As far as past reports, one of the paid reporters will give us a list of them shortly I am sure.

And I would have to second Wazoo's response. Thank you for coming here with a good attitude, asking an intelligent question, and providing us with the information to try our best to help. There are so many people who come here with nothing other than a rant as to why they don't like spamcop, they forget that spamcop has no control over how their list is used or misused by ISPs.

Link to comment
Share on other sites

yes, we do recieve non-spamtrap reports to our abuse address which i have access to. I also have reports being sent to my personal address. we currently do not accept port 25 to our mail server from outside our network. It has been that way since before i was here. It helps. In this case I think this virus just hijacked this guys mail client and managed to work around our anti-spam rules eg. no more than 40 messages at a time, no more than 1000 per day.

Link to comment
Share on other sites

yes, we do recieve non-spamtrap reports to our abuse address which i have access to. I also have reports being sent to my personal address. we currently do not accept port 25 to our mail server from outside our network. It has been that way since before i was here. It helps. In this case I think this virus just hijacked this guys mail client and managed to work around our anti-spam rules eg. no more than 40 messages at a time, no more than 1000 per day.

Most recent report is showing as Tuesday. The last bounce to a spamtrap (we think that is what the uube[at]devnull.spamcop.net reports are) is also Tuesday. Sorry not much help there.

Currently, non-spamtrap reports are only being sent to: Reporting addresses:abuse[at]sunwave.net

The routing sees your support[at]sunwave from abuse.net but ignores it for some reason.

Report History:

--------------------------------------------------------------------------------

Submitted: Tuesday, August 22, 2006 5:51:39 PM -0400:

Bachelors, Masters, MBA, PhD can be yours in 4 weeks if you qualify.

1887857680 ( 208.98.210.10 ) To: spamcop[at]imaphost.com

1887857671 ( 208.98.210.10 ) To: abuse[at]sunwave.net

--------------------------------------------------------------------------------

Submitted: Tuesday, August 22, 2006 2:03:29 PM -0400:

Undelivered Mail Returned to Sender

1887585552 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net

--------------------------------------------------------------------------------

Submitted: Wednesday, August 09, 2006 1:44:29 PM -0400:

Delivery Status Notification (Failure)

1870365870 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net

--------------------------------------------------------------------------------

Submitted: Tuesday, August 08, 2006 10:24:39 PM -0400:

Undelivered Mail Returned to Sender

1869483845 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net

--------------------------------------------------------------------------------

Submitted: Tuesday, August 08, 2006 6:11:04 PM -0400:

Undelivered Mail Returned to Sender

1869284938 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net

--------------------------------------------------------------------------------

Submitted: Monday, July 24, 2006 10:06:06 PM -0400:

Undelivered Mail Returned to Sender

1850467640 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net

--------------------------------------------------------------------------------

Submitted: Sunday, June 11, 2006 10:48:21 AM -0400:

Returned mail: User unknown

1789903165 ( 208.98.210.10 ) To: uube[at]devnull.spamcop.net

Older Reports

Moderator Edit: added the prefix "non-" to prevent some confusion .....

Link to comment
Share on other sites

Hmm, yep, not much help in the manual reports then. So that pretty much leaves spamtraps. I would fire an email over to the deputies (deputies[at]spamcop.net) and see if they can tell you what kind of traffic they are seeing at the spamtraps.

Link to comment
Share on other sites

Plenty of evidence at http://psbl.surriel.com/evidence?ip=208.98...=Check+evidence

The disturbing thing though are the lines;

spam and removal history for 208.98.210.10 (times in UTC):

2006-08-20 01:21:42.990509 received spamtrap mail

2006-08-20 02:16:50.474946 received spamtrap mail

2006-08-20 02:17:01.565076 received spamtrap mail

2006-08-21 13:13:49.919842 received spamtrap mail

2006-08-21 15:35:34.51922 removed through website

Someone knows about this site ....?????

However, this collection may be from the bad stuff already removed ....

Link to comment
Share on other sites

One good thing I see about all those message is that their SMTP server is stamping a Received line with the IP its picking them up from, so should be easy to trace them back to a particular customer. Don't see anything there newer than Aug 20th, which I believe was an issue the OP said they already addressed. I am noticing something odd though...

Received: from [208.98.210.10] (helo=lithium.sunwave.net)

by angband.surriel.com with esmtp (Exim 4.43)

id 1GEgcD-00022W-Jw

for victim[at]smtp.example; Sun, 20 Aug 2006 02:16:58 -0400

Ok, last hop from SMTP server to its destination as we would expect

Received: from 208-98-202-239.cable.dynamic.sunwave.net (208-98-202-239.cable.dynamic.sunwave.net [208.98.202.239])

by lithium.sunwave.net (Postfix) with SMTP id 498E343F0B

for <victim[at]smtp.example>; Sat, 19 Aug 2006 22:56:50 -0700 (PDT)

I assume this is the SMTP server receiving the email from 208.98.202.239, apparently a dynamic IP address in the ISPs range, again normal for anyone using the SMTP server as a smarthost.

Received: from [208.98.202.239] (port=5374 helo=208-98-202-239.cable.dynamic.sunwave.net)

by mail.asiafreeport.com with esmtp

id 1a2lC2-ycP765-87

for victim[at]smtp.example; Sat, 19 Aug 2006 22:56:37 --800

Received-SPF: none (mail.asiafreeport.com: 208.98.202.239 is neither permitted nor denied by domain of asiafreeport.com) client-ip=208.98.202.239; envelope-from=victim[at]smtp.example; helo=208-98-202-239.cable.dynamic.sunwave.net;

This is where things get squirly. It appears 208.98.202.239 (mail.asiafreeport.com) is receiving mail from itself. That means either some kind of odd relay attack from a compromised machine to its own SMTP service, or more likely, abuse of some kind of webform hosted on the same server. There are certainly other explanations, and as Wazoo pointed out, this may be referring to the issues that have already been taken care of, but if not, this might be the first place to look.

Link to comment
Share on other sites

wow you guys are good.

That's odd .... I'm so much more accustomed to the "you are a bunch of &^%$#, *&^%[at], or even worse .. &***%$[at](&!!!" <g>

That was the customer we shut down that was our prime suspect. This just validates our suspicion. Thanks for the help!

On the flip side, kudos on being there and doing a good job.

Link to comment
Share on other sites

Hello,

I'm the systems administrator at the ISP Derrik.Hansen works at. I wanted to throw a few more questions out there and clarify a few things. Hopefully someone on this list has some suggestions.

Our smarthost (lithium.sunwave.net - 208.98.210.10) has been blacklisted a few times over the past few days. The spam has not originated from our smarthost, it has simply been the relay for systems on our network that are permitted to relay through our smarthost.

The first instance was from an exchange box that had been configured to relay through us some time ago, but just this week, their antispam filter died. The result was exchange doing asynchronus bouncing and the bounced spam messages being relayed through our smarthost.... The problem was identified and the customer was asked not to send their email on their own rather than through us. They took a little too long to comply (got us blacklisted again) and I ended up blacklisting them - they are now sending their own mail ;)

The second instance was from a host that got infected with a bot and began relaying through our smarthost. This has happened in the past and I always have to quickly identify the culprit, have one of our support guys shut them down and then we start sending delist requests to the blacklists we have found ourselves on.

So, that's what we have been dealing with. I have two questions:

1 - Does spamcop intend to block ISP smarthosts?

I was reading through the spamcop faq today and saw this faq entry

http://www.spamcop.net/fom-serve/cache/99.html

It suggests that relays shouldn't get blacklisted, that this is "relay rape". I fail to see why that would be; if my relay is forwarding spam, the only way to block the spam using a RBL is to block the relay. Seems fair to me. However, if spamcop doesn't intend to block relays and there is some way that I can reconfigure the system to take advantage of that, I will.... No sense in making myself go grey if someone hands me a get out of jail free card :) I have a feeling I'm just misinterpreting what was stated in that faq, which brings me to my next question.

2 - What can be done about bots relaying through a smarthost?

I have given this particular issue a great deal of thought and actually have a plan for dealing with the problem, I just need to find time to implement my plan. However, if anyone in this forum has any fresh ideas that I may have missed, I'd love to hear them. I'd also welcome any debate concerning problems with my plan.

So, here's the plan:

--------------------------

Note:

We have very effective inbound filtering through Postini, so, as you'll note in the plan, I don't waste any CPU cycles on inbound spam filtering on our mailserver.

I'm going to start implementing the following steps. It's quite a big list and I expect that I'll implement everything up to step 8, but beyond that I'm not sure. I'll almost certainly implement step 12 on our dynamic networks, but I'll leave our business networks alone. Step 13 is already done. Feedback from this list may help me decide what steps I skip.

1 - Setup new mail server with 2 instances of the latest postfix (2.3)

http://www.advosys.ca/papers/postfix-instance.html Also check out the

postfix docs, I seem to remember code being added to make the dual

instance setup much easier to implement

- one for inbound connections from the net and one for outbound

connections from our customers.

- inbound will do basic postfix UCE checks, but nothing fancy -

absolutely no SA/ClamAV filtering

- well, maybe I could be convinced to do a little bit of SA

on inbound. This ruleset would be so nice to have and

couldn't do any harm to legit email

http://www.timj.co.uk/linux/bogus-virus-warnings.cf

- outbound will be much more intensive (see below for details)

- change our MXes to point to lithium.sunwave.net (new IP)

- have all clients point at mail.sunwave.net (current IP) for

outbound smtp and pop3/imap

- this will make it much easier to have different filter rules and SA

configs for inbound vs outbound mail. As well, this makes it a bit

easier to break the mail system into two hosts, in the future, if

need be (likely due to scanning software overhead or imap overhead)

- having a separate postfix instance for outbound (on it's own IP) also

makes it easier to buy myself some time, by just switching IPs if we

get blacklisted again (this is what Telus does)

2 - Install local authoritative and caching name servers so that RBL

checks are much faster. As well, this will ensure that DNS changes

for domains we host are seen by the mail server immediately.

3 - Implement postfix *outbound* rate limiting controls on a per client

basis

http://www.postfix.org/anvil.8.html

/etc/postfix/samples/sample-rate.cf

- intention is to slow down bot spam so that the inline scanner

doesn't have to work so hard and we also reduce the risk of

getting black listed

4 - Tighten up postfix a fair bit - especially outbound!

- http://jimsun.linxnet.com/misc/postfix-anti-UCE.txt

- Do what is realistic using postfix native UCE controls so that less

load need be placed on the inline scanner.

- I would love to do this

http://www.postfix.org/postconf.5.html#smt...unlisted_sender

but then we would have the problem of non local users being

rejected by our relay. Is there a way around this?

- Could I enforce strict_rfc821_envelopes or would this break some

clients? Try using a warn_if_reject to see.

5 - publish spf records for all our domains (sunwave, sunlite, shuswap)

- start simple and conservative

http://spf.pobox.com/wizard.html

sunwave.net. IN TXT "v=spf1 ip4:64.141.16.1/27 mx a:lithium a:mail ?all"

lithium.sunwave.net. IN TXT "v=spf1 a -all"

mail.sunwave.net. IN TXT "v=spf1 a -all"

- put in some logging so that I can see who is using servers

other than what I have specified

Page 15 - http://spf.pobox.com/whitepaper.pdf

- test them

http://spf.pobox.com/faq.html#checkers

6 - Implement inline transparent spam filters on *OUTBOUND* mail

Use spampd or smtpprox and spamassassin - return an error code and

message

I think that spampd is probably the better option

http://www.postfix.org/SMTPD_PROXY_README.html

http://www.worlddesign.com/Content/rd/mta/...xspecific_notes

Don't forget to use XFORWARD!!!

If for some reason, spampd is not logging the client IP, then consider

either patching it or using smptpprox (patching should be dead easy as

spampd uses the same code as a base as smtpprox)

http://bent.latency.net/smtpprox/index.html

- this is going to take a fair bit of tuning to get right, because

we will be focusing on outbound rather than inbound; this is a non

standard approach. Because we are focusing on outbound, I think

that we can give more points to matches from the blacklists (see

below for RBLs to lean on). I'm sure there will also be other

settings that will make sense to lean heavier on due to the

outbound approach (perhaps spf?)

7 - Implement post queue antivirus scanning on *OUTBOUND* mail

Use amavisd-new and clam - drop viruses on the floor with no

notifications

Perfect! - http://workaround.org/articles/ispmail-sarge/#amavis

8 - Implement Intra domain routing through postini

http://www.postini.com/admindoc/intra_domain.html

- this setup increases the effectiveness of postini for user to user

and gives us free user to user email virus scanning!

- basically, don't let the outbound smtp server think it's

authoritative for local domains

9 - Start generating outbound usage stats so that I can see/be

notified of suspicious trends. Specifically on a per user basis.

- Parse the outbound server's logs looking for:

- lots of blocks by spampd/spamassassin

- lots of blocks by amavisd-new/clamav

- # of connections / recipeints per min

- probably be good to do this every 5-15min or so

- statistically large increases in mail output from a given

IP (would require software that tracks over much longer

periods of time) # if software exists, great, if not,

then don't go here, we don't have the time for this sort

of thing

- If the parser sees something that looks nasty, it will send an

email to the ticket system and the ticket system will email the

customer. This should allow me to better tune anvil as well.

10 - Optimize postfix to make it more friendly to our customers

- Protect customers against "back scatter"

http://www.postfix.org/BACKSCATTER_README.html

- create useful error responses

- look in the /etc/postfix/samples directory for lots of great

ideas

http://www.mailsbroadcast.com/email.bolts&...codes.error.htm

- run postfix 2.3 so that I can do DSN

http://www.postfix.org/DSN_README.html

http://archives.neohapsis.com/archives/pos...05-03/2016.html

11 - Tune amavis/clam/spamassassin performance

http://wiki.apache.org/spamassassin/FasterPerformance

http://wiki.apache.org/spamassassin/SitewideServerSpec

http://www.ijs.si/software/amavisd/README.performance

12 - Ingress/Egress port 25 block

- for all dynamic networks

- leave static networks alone

13 - Change our reverse dns so that dynamic ranges are clearly identified

http://www.onlymyemail.com/company/press_m...Threat_RBL.html

http://www.spamhaus.org/faq/answers.lasso?...am%20Issues#129

- ex - 64-141-16-142.dialup.dynamic.sunwave.net

- ex - 64-141-19-234.cable.dynamic.sunwave.net

- ex - 64-141-18-234.business.static.sunwave.net

14 - SMTP AUTH

http://spamlinks.net/prevent-secure-relay-auth.htm

http://postfix.state-of-mind.de/patrick.ko...auth/index.html

use port 587, tls and any auth method

or port 465, no tls with crammd5

14.5 - SMTP AUTH would be hard as it would require us to contact all customers

and have them change their outgoing mail settings (have you heard the term

"kicking dead whales down the beach"? - it would apply.....

So, rather than go with SMTP AUTH, I could setup something like camram,

TMDS, or another challenge response tool (I hate challenge response, but

this might work). Again, this would be for outbound traffic only - not inbound.

Now, rather than just set the challenge response tool loose, I would preload

the tool with all our valid customer addresses. That means that the only

people that will get challenges will be those that are our customers, are using

our smarthost, but do not have an email address on our system. This should

be fairly pain free.

15 - Address security for all hosts on the network for all services, not

just mail

- Create an automated lepper colony that tosses known infected

users into it and directs all traffic to a webpage, sends

them an email

- start doing regular vulnerability scans of the network using

nessus or better yet setup ossim. Create a monthly top 10

report and have TSRs contact people and try to educate them

- use adaptive response on the shaper to create tickets for users

that have constant socks traffic

- Do the big media blitz and get people up to speed with the big

four defences - firewall, anti-virus, anti-spyware, windows

updates. Get as many people as possible to upgrade to XP SP2,

- move people to firefox and thunderbird. Also encourage people

to tell their friends.

- seriously look at tools such as the fortinet appliance

- focus on antispyware

- consider other near term gateway approaches such as using the

new spyware RBL that someone at bleeding snort is working on. Or

start blocking known spyware user-agent strings at our cache or

shaper.

--

Mason Schmitt

Link to comment
Share on other sites

I was supposed to be elsewhere an hour ago, so short answer here ....

Item 6 (the client's IP address bit) is the most 'functional' item bit there ....

No, the SpamCopDNSBL's intent is not to snag your smarthost .. however, if the 'chain-test' of the parsing engine cannot proceed beyond that Received: line, that's where it has to stop .... the nastiest of these examples are found in the GMail servers blocked massive Topic, Yahoo servers blocked massive Topic .... the same 'problem' exists there .. the source of the spew cannot be tracked beyond the output server due to the non-existent Recieved: lines that show the 'real' source of the problem e-mail .....

Link to comment
Share on other sites

Note:

We have very effective inbound filtering through Postini, so, as you'll note in the plan, I don't waste any CPU cycles on inbound spam filtering on our mailserver.

Since you are already a customer, have you investigated Postini's outbound service and how it works? My company is also a Postini customer for incoming only so I do not know how this works exactly, but we are a single server shop with less "problem areas" to address.

Link to comment
Share on other sites

From the couple of sample mails we looked at in the evidence posted earlier, it looks like your server is already stamping the "source" IP address, or customer IP, of the message. I believe all spamcop needs to do is add your MX address to its list of trusted addresses so that it will parse up to the next received line. I would suggest emailing the deputies and asking them if they can do this, as it is beyond what anyone here has access to. I don't know what their requirements are for this, but I know there are a number of MXs that I have seen it "trust" and proceed on to the next Received header for the source.

Link to comment
Share on other sites

From the couple of sample mails we looked at in the evidence posted earlier, it looks like your server is already stamping the "source" IP address, or customer IP, of the message. I believe all spamcop needs to do is add your MX address to its list of trusted addresses so that it will parse up to the next received line. I would suggest emailing the deputies and asking them if they can do this, as it is beyond what anyone here has access to. I don't know what their requirements are for this, but I know there are a number of MXs that I have seen it "trust" and proceed on to the next Received header for the source.

Ah, thanks! I didn't realize that was an option. I'll get an email out to spamcop asap.

Since you are already a customer, have you investigated Postini's outbound service and how it works? My company is also a Postini customer for incoming only so I do not know how this works exactly, but we are a single server shop with less "problem areas" to address.

Postini does not provide outbound filtering for their ISP service. We are however investigating upgrading to Enterprise edition in order to get this functionality.

I'd rather rely on someone else playing the cat and mouse spam game rather than having to attempt to keep up on my own...

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...