Update to blacklist options needed

[jongrose]

I don't remember SORBS being a BL option, but I did have some back and forth with a few of you (I can pull up the thread if necessary) about it, and as I recall, SORBS is pretty slow just when trying to look up something through it's web based lookup tool, so I believe one of the assumptions was that it, along with some of the other DNS/RBLs, might be too slow and that's the reason that emails (which, in some cases, have IPs that are listed in one ore more of the BLs enabled) get skipped by them. That's one of the reasons I have both the CBL and SpamHaus's XBL enabled, even though that is technically redundant.

[DavidT]

I had the impression you used to be critical.

Yes, I've been both "hot" and "cold" regarding SORBS, because for a while, they were listing my cable provider's email relay servers, which was causing too many false positives. However, I'm using the following BLs on my server at work:

ZEN.SPAMHAUS.ORG

BL.SPAMCOP.NET

DNSBL.NJABL.ORG

LIST.DSBL.ORG

DNSBL.SORBS.NET

I just did a quick check of the last 253 items held due to RBL hits, and here are the stats:

ZEN.SPAMHAUS.ORG = 50

BL.SPAMCOP.NET = 54

DNSBL.NJABL.ORG = 3

LIST.DSBL.ORG = 5

DNSBL.SORBS.NET = 141

Assuming that the BLs are checked in the order I've got them listed, then SORBS is sure doing a lot of "cleanup" on stuff the other four allow on through. The system is currently configured to only check two levels of Received headers in each message, which is another factor in the mix (SpamCop's system checks all Received headers).

So, although I had stopped using SORBS among the SC Email BL offerings myself, I still wonder why and when it dropped off the list.

DT

[trevorb]

DavidT,

For the e-mail you linked to that we didn't block even though it had an IP in the PBL, that is actually correct behavior. The Barracuda doesn't reject messages that are in blocklists like we do, it just scores them higher. Since we outright block them, we have to be more careful about what IP addresses we consider.

The very first hop of the e-mail chain does not get checked against the PBL list. The first address is usually the IP address of the computer you are sending from, which is very likely to be your own dynamic IP assigned from your ISP. Since the PBL includes IP blocks that ISPs designate as their dynamic IP range, there is a very high chance that the first source IP in an e-mail will be on the PBL list even though the message is valid. We check every IP except the first hop (which is actually the last one listed in the 'Received' headers) against the PBL.

-Trevor

[DavidT]

The very first hop of the e-mail chain does not get checked against the PBL list. The first address is usually the IP address of the computer you are sending from, which is very likely to be your own dynamic IP assigned from your ISP.

Basically logical, but if that's the case, how would you block "direct to MX" spew coming from machines that aren't putting mail through the "normal" relays? Also, here's a TU of one that the SC system actually put into my Held due to a PBL hit:

http://www.spamcop.net/sc?id=z1501079132z3...eaad67a755118az

Let's look at the Received headers before it hits the Barracuda:

Received: from eqm39.neoplus.adsl.tpnet.pl (eqm39.neoplus.adsl.tpnet.pl [83.20.80.39])

by filter.spry.com (spam Firewall) with ESMTP id 7679AD00B983

for <x>; Sun, 28 Oct 2007 15:32:51 -0700 (PDT)

Received: from [83.20.80.39] by ns6.dowjones.com; Sun, 28 Oct 2007 22:20:03 +0000

The first one (starting at the bottom) is obviously bogus, inserted there by the spammer to fool systems and/or users that aren't sophisticated enough to see that "dowjones.com" isn't involved in the relaying of the spam. So, this spam was actually "direct to MX" to the server that accepts mail for my address. Here's what the Barracuda found about the only valid IP in the chain:

0.80 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL

[83.20.80.39 listed in zen.spamhaus.org]

And the SpamCop system agreed:

X-SpamCop-Disposition: Blocked pbl.spamhaus.org

Now let's look at the same info from the false negative:

Received: from p54BE5D60.dip.t-dialin.net (p54BE5D60.dip.t-dialin.net [84.190.93.96])

by filter.spry.com (spam Firewall) with SMTP id 2DAC0D01E188

for <x>; Sun, 28 Oct 2007 09:22:47 -0700 (PDT)

and:

0.80 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL

[84.190.93.96 listed in zen.spamhaus.org]

But "blade6" only found some "CHICKENPOX" in the message:

X-spam-Status: hits=0.6 tests=J_CHICKENPOX_73 version=3.2.3

Both of those messages were transmitted to my original email address in a similar manner, with the exception that the latter doesn't have a bogus Received header. Surely the SC system is sophisticated enough to tell the difference between a real, meaningful Received line and a bogus one? I know that the parsing system used for Reporting can tell the difference. If not, then wouldn't a lot of spam get through that otherwise shouldn't?

DT

[Telarin]

Maybe the PBL check needs to be changed so it only checks the connecting IP, rather than all the received lines?

[DavidT]

Maybe the PBL check needs to be changed so it only checks the connecting IP, rather than all the received lines?

Hmmmm...but how would that affect those of us who are forwarding lots of different accounts to our SC email addresses?

DT

[michaelanglo]

Maybe the PBL check needs to be changed so it only checks the connecting IP, rather than all the received lines?

Counterproductive. If a extra and bogus Received: line causes a spam to be blocked when it otherwise would not have been then good (and LOL).

The requirements for parsing for the true source are different from doing blocking.

Unless someone has an example where a dynamic IP address could properly occur other than as the last (lowest) such line ?

Have to be something like running a mailing list server on a dynamic IP, but with all outgoing mails relayed though a proper mail server.

[Telarin]

Have to be something like running a mailing list server on a dynamic IP, but with all outgoing mails relayed though a proper mail server.

Interestingly enough, that is exactly how my email at home is setup I run my own server because I use a number of outgoing domains, but then route everything through a legitimate smarthost to keep emails from being rejected due to their Dynamic nature. Of course, this should still work, as the mail server would be the first IP seen in the Received lines, since Exchange doesn't stamp an IP address from the outlook client.

[michaelanglo]

The very first hop of the e-mail chain does not get checked against the PBL list. The first address is usually the IP address of the computer you are sending from, which is very likely to be your own dynamic IP assigned from your ISP. Since the PBL includes IP blocks that ISPs designate as their dynamic IP range, there is a very high chance that the first source IP in an e-mail will be on the PBL list even though the message is valid. We check every IP except the first hop (which is actually the last one listed in the 'Received' headers) against the PBL.

Here an example where the spammer's extra bogus line gets the message blocked, GOOD.

http://www.spamcop.net/sc?id=z1501812963z4...3ad8ee8b035cbfz

I have had two more very similar ones today.

BTW

SpamCop-Checked: 192.168.1.108 216.154.195.53 212.74.100.190 89.243.115.187 89.243.115.187 89.243.115.187

X-SpamCop-Disposition: Blocked pbl.spamhaus.org

Why the repetition of 89.243.115.187 ?

I assume cacheing means that there is no performance hit even if the test is triplicated.

[DavidT]

...the spammer's extra bogus line gets the message blocked, GOOD.

Yes, I had two such examples in the last 24 hours myself. However, if the messages did *not* have the bogus extra line, then I suppose they would have bypassed the test and hit my inbox....NOT GOOD.

We're all aware that a lot of spam is sent "direct-to-MX," yes? Here's a refresher from "Rick's spam Digest" -

How spam is transmitted

On that page, scroll down to the "Direct-to-MX" section, where you'll see:

The so-called “direct-to-MX†model is now used for the vast majority of spam.

So, regarding the application of the PBL to direct-to-MX spam (without a bogus extra line), I suppose wer're basically SOL?

DT

[DavidT]

The very first hop of the e-mail chain does not get checked against the PBL list.

So...pursuing this a bit more...does the IP of the first hop get checked against *any* of the BLs offered in our options? How about the SCBL?

For those of us with other email addresses that forward to our CES accounts, I'd think that having defined Mailhosts could serve as a method of identifying "direct-to-MX" spam to the filtering system. Perhaps the Mailhosts system doesn't apply to the filtering/analysis of messages arriving at our SpamCop email accounts, but wouldn't that be a good way of catching more direct-to-MX spam?

DT

[trevorb]

For every other blacklist we check all IPs. It's only for the PBL that the last IP is skipped. It is *not* skipped if it is the only IP, so direct-to-MX should be caught. I fixed the duplicate IP bug today, so each IP address should only be checked once.

There is nothing we can do about the spam that gets by even though the first IP is in the PBL, since the first IP is usually in the PBL whether it is spam or not.

-Trevor

[michaelanglo]

Interestingly enough, that is exactly how my email at home is setup I run my own server because I use a number of outgoing domains, but then route everything through a legitimate smarthost to keep emails from being rejected due to their Dynamic nature. Of course, this should still work, as the mail server would be the first IP seen in the Received lines, since Exchange doesn't stamp an IP address from the outlook client.

Some mailing list software keeps all the Received: line for a post all the way from the listee/poster to the mailing list server and back out to the mailing list, that was what I had in mind.

My example is from Mailman.

http://www.spamcop.net/sc?id=z1503442867z1...2529e12a6843e6z

[DavidT]

It is *not* skipped if it is the only IP, so direct-to-MX should be caught.

I'm thinking that statement only applies to messages sent directly to our spamcop.net/cesmail.net addresses? Some of us don't ever use those addresses, but rather have other addresses that forward to our SC mailboxes. IIUC then, the system won't be able to tell that there's really only one received header and one IP involved in the initial delivery, correct? However, if we could define the Mailhosts that receive mail on our behalf before the mail is forwarded to SC, then those "direct-to-MX" items could also be caught. Am I thinking clearly about this, Trevor?

DT

[StevenUnderwood]

However, if we could define the Mailhosts that receive mail on our behalf before the mail is forwarded to SC, then those "direct-to-MX" items could also be caught. Am I thinking clearly about this, Trevor?

Once again, mailhosts is a reporting side (west coast, Ironport owned) thing. I don't believe the email side of the house has any access to that information.

[DavidT]

Once again, mailhosts is a reporting side (west coast, Ironport owned) thing.

Yes, Steve, I think I acknowledged that in linear post #36 above, where I wrote:

Perhaps the Mailhosts system doesn't apply to the filtering/analysis of messages arriving at our SpamCop email accounts, but wouldn't that be a good way of catching more direct-to-MX spam?

I shouldn't have used "perhaps" -- I know the difference between what happens in California vs. Georgia, but what I'm proposing is a similar system for SC Email users. If the SC email system knew about the mailhosts that receive mail before forwarding it to our SC email accounts, then false negatives such as the one I've recently cited wouldn't leak through to our mailboxes, because they could be treated in the same manner as if they had been sent directly to our actual SC email addresses.

Oh...and please let's not start any discussion of moving this to the "New Feature Request" forum...we've been around that block before.

on edit: Let me add some new statistics....I just checked the overnight accumulation in my wife's Held mail. All messages there were sent to a single, non SC email address and then forwarded to her SC mailbox. Of the 20 items (all put there due to their SA scores), 12 came from IP addresses listed on the Spamhaus PBL. Yet, IIUC, because those messages are forwarded/relayed to her SC mailbox by another host, the "direct-to-MX" items from PBL-listed IPs would *not* result in them being put into Held mail if their SA scores fell below her threshhold (unless, of course, that the IP was on one of the other selected BLs).

Similarly, I've got 114 message in my own Held mail right now, and although a Barracuda flagged 74 of them as coming from PBL-listed IPs, the way things are currently configured, they'd only get put into Held by the SC system if the spammers happen to add a bogus initial Received line (or the IP happens to be on another selected BL). I think that a Mailhost definition system for SC email customers would solve that problem.

DT

[michaelanglo]

For every other blacklist we check all IPs. It's only for the PBL that the last IP is skipped. It is *not* skipped if it is the only IP, so direct-to-MX should be caught.

Problem with "only IP" is that what with smart hosts, "only IP" doesn't occur for most of us.

Thanks, though no more PBL hits so far that SA didn't catch.

But now the "skip the last IP"code is present it you can now consider adding an optional "dynamic IP" block list if a suitable one is suggested.

And the same for a virtual block list that looks at the reverse DNS and 'blocks' if it doesn't exist or contains dynamic or modem or ADSL (etc.).

[DavidT]

Update: no further spam messages have been put into my Held due to the addition of the PBL to our options, even when the PBL-listed IP is in the second Received header. Here's a false negative:

http://www.spamcop.net/sc?id=z1506910322zd...afca8df955aecbz

When Trevor first implemented the PBL option, the message referenced above probably *would* have been flagged and held, so it appears that the "fix" that he mentioned here:

I fixed the duplicate IP bug today

means that the PBL option is no longer doing any good for those of us whose mail is forwarded to our SC email accounts.

Therefore, the need for my proposed solution (an option of defined Mailhosts for SC email customers) seems even more needed, because I believe it would allow the SC email system to better detect and trap direct-to-MX spam when the source is listed on the PBL....wouldn't it? In fact, I think that if the system determined that a given message had been sent direct to one of our defined Mailhosts (not the existing ones for reporting), there are other BL choices that list dynamic IP ranges that might be offered...but at least the PBL would work better.

DT

[michaelanglo]

Update: no further spam messages have been put into my Held due to the addition of the PBL to our options, even when the PBL-listed IP is in the second Received header. Here's a false negative:

[snip]

When Trevor first implemented the PBL option, the message referenced above probably *would* have been flagged and held, so it appears that the "fix" that he mentioned here:

means that the PBL option is no longer doing any good for those of us whose mail is forwarded to our SC email accounts. [...]

Here's today's example to show that PBL is now (or still) working on the second Received: line (POP not Forward but does it matter ?)

http://www.spamcop.net/sc?id=z1508432580z2...2df3bc5d1aca24z

[DavidT]

Here's today's example to show that PBL is now (or still) working on the second Received: line (POP not Forward but does it matter ?)

Yes, it appears to matter, because I've got more false negatives in my inbox today that are from PBL-listed IPs in the second Received line. I use the forwarding method, non only because it's the preferred method, but you don't get (up to) 15-minute delays in the receipt of messages. I can provide more Tracking URLS of messages that are slipping through the SC email system because of the change Trevor made to the system. However, instead of simply undoing that change, I'm going to lobby for a better method, as I've described above.

DT

[DavidT]

It's only for the PBL that the last IP is skipped.

Simply skipping the last (actually the initial) IP to avoid problems using the PBL isn't good enough, IMO. I've just posted an example of this going terribly wrong here:

http://forum.spamcop.net/forums/index.php?...amp;#entry61122

To quote the SpamHaus folks about using the PBL:

PBL policy is based on ranges which should not directly deliver e-mail, so any use beyond that will be riskier and subject to more false interceptions

So, in order to use the PBL, especially for those of us who have mail forwarded to our SC addresses (which many of us would like to keep secret), what's really needed is for the system to know about the mailhosts that are supposed to receive mail for our various addresses. Then, if a PBL-listed IP tries to deliver to one of those mailhosts, the message should get flagged/filtered/held.

DT