Jump to content

PBL incorrectly used by SC - blocks web-mail submission


Firefly

Recommended Posts

I recently enabled the Spamhaus PBL list in my list of blocklists, but have found that when people use a web-mail system, such as SpamCop's own, SC incorrectly looks up the user's own IP in the PBL where it is likely to be found. For example:

Return-Path: <me[at]example.com>

Delivered-To: spamcop-net-My.Name[at]spamcop dot net

Received: (qmail 6870 invoked from network); 8 Nov 2007 01:43:08 -0000

X-spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blade2.cesmail.net

X-spam-Level: ***

X-spam-Status: hits=3.3 tests=ALL_TRUSTED,HTML_MESSAGE,MIME_QP_LONG_LINE,

NORMAL_HTTP_TO_IP,TVD_SPACE_RATIO version=3.2.3

Received: from unknown (HELO c60.cesmail.net) (192.168.1.105)

by blade2.cesmail.net with SMTP; 8 Nov 2007 01:43:08 -0000

Received: from unknown (HELO epsilon2) ([192.168.1.60])

by c60.cesmail.net with ESMTP; 07 Nov 2007 20:43:08 -0500

Received: from pool-71-181-32-168.cncdnh.fios.verizon.net

(pool-71-181-32-168.cncdnh.fios.verizon.net [71.181.32.168]) by

webmail.spamcop.net (Horde MIME library) with HTTP; Wed, 07 Nov 2007

20:43:08 -0500

Message-ID: <20071107204308.8njl34bq8kook448[at]webmail.spamcop.net>

Date: Wed, 07 Nov 2007 20:43:08 -0500

From: My Name <me[at]example.com>

To: My.Namel[at]spamcop dot net

Subject: UPS

MIME-Version: 1.0

Content-Type: multipart/mixed;

boundary="=_4t3dfs5ttmss"

Content-Transfer-Encoding: 7bit

User-Agent: Internet Messaging Program (IMP) H3 (4.1.4)

X-SpamCop-Checked: 71.181.32.168

X-SpamCop-Disposition: Blocked pbl.spamhaus.org

Spamhaus's description of the PBL says:

Caution: Because the PBL lists normal customer IP space, do not use PBL on smarthosts or SMTP AUTH outbound servers for your own customers (or you risk blocking your own customers if their dynamic IPs are in the PBL). Do not use PBL in filters that do any ‘deep parsing’ of Received headers, or for other than checking IP addresses that hand off to your mailservers.

I maintain that SpamCop is doing exactly what it shouldn't - deep parsing the headers.

I have therefore disabled the PBL for my account. I'd wonder how effective it would be anyway for anyone who gets mail forwarded.

Steve

Link to comment
Share on other sites

...I have therefore disabled the PBL for my account. I'd wonder how effective it would be anyway for anyone who gets mail forwarded.
Thanks for the analysis Steve, hopefully some of the mail users (or even staff) can comment - but have you been following the discussion at http://forum.spamcop.net/forums/index.php?showtopic=8869? That has developed into a discussion on the use of the PBL in SC filtering in some depth (but is outside my own area of familiarity/knowledge).
Link to comment
Share on other sites

I'd wonder how effective it would be anyway for anyone who gets mail forwarded.

Not at all, unfortunately. I have a growing folder full of "PBL false negatives" that all slipped through to my inbox, despite the actual sending IPs all being listed on the PBL. The way things are currently configured, SC can't properly handle it. However, if we had a system in which the SC servers knew about the mailhosts that receive our mail before forwarding it to our SC addresses (as I mentioned in that other topic), then the PBL could probably be used to help us.

DT

Link to comment
Share on other sites

I maintain that SpamCop is doing exactly what it shouldn't - deep parsing the headers.

I have therefore disabled the PBL for my account. I'd wonder how effective it would be anyway for anyone who gets mail forwarded.

I'm tentatively inclined to agree... although I'd like to see more evidence to confirm the exact issue.

But, FWIW, as I noted in the SMTP AUTH thread, the PBL is picking up my home IP if I send a copy to my SpamCop account - rather than stopping at the outgoing SMTP server address.

Like you I find the PBL is unhelpful at this point.

Andrew

Link to comment
Share on other sites

I'm tentatively inclined to agree... although I'd like to see more evidence to confirm the exact issue.

But, FWIW, as I noted in the SMTP AUTH thread, the PBL is picking up my home IP if I send a copy to my SpamCop account - rather than stopping at the outgoing SMTP server address.

Like you I find the PBL is unhelpful at this point.

Andrew

I use my spamcop address as my primary email address. I have seen nothing get caught that should not have, but have not seen a marked decrease in my Inbox (though I do not get many to begin with). I do not generally watch the listed messags to see why they are held.
Link to comment
Share on other sites

I use my spamcop address as my primary email address. I have seen nothing get caught that should not have, but have not seen a marked decrease in my Inbox (though I do not get many to begin with). I do not generally watch the listed messags to see why they are held.

My situation is that I send my outgoing mail through smtp.cesmail.net

I sent a message to my SC Email address.

This mail item was sent because the SC mailbox consulted the PBL and identified the mail item as originating from my home cable IP, not the originating mail server - exactly contrary to the way the PBL says things should be configured.

Andrew

Link to comment
Share on other sites

My situation is that I send my outgoing mail through smtp.cesmail.net

I sent a message to my SC Email address.

This mail item was sent because the SC mailbox consulted the PBL and identified the mail item as originating from my home cable IP, not the originating mail server - exactly contrary to the way the PBL says things should be configured.

And contrary to how Trevor SAID it was supposed to be working here (skipping first header). Hopefully he will comment here.

Link to comment
Share on other sites

I had seen references to PBL in that other thread, but I stopped reading it when it seemed to degenerate into an argument about feature requests. I see I should have persevered.

I have one other test case (which I will post tonight as I don't have it handy) and this one too is from a web-form-mail submission. You can see all of the IPs that SC checked and it is the user's IP that triggered the PBL block. I have not seen any other false-positives due to PBL and have not noticed false-negatives (the negatives I tend to get are all Russian spam with no links - darned if I know what they are trying to sell me.)

As I see it, the Spamhaus instruction that only "sending servers" that connect directly to SpamCop's MX should be checked against the PBL - this will block bot-spam using its own SMTP server, but will work only for mail sent directly to a SC address and not for forwarded mail. Since all my mail is forwarded (or picked up), the PBL is not an option for me, even if SC's implementation was correct. Unless I can somehow register my form-mail servers, I don't see how I could make use of the PBL.

Steve

Link to comment
Share on other sites

I had seen references to PBL in that other thread, but I stopped reading it when it seemed to degenerate into an argument about feature requests. I see I should have persevered.

[...]

up), the PBL is not an option for me, even if SC's implementation was correct. Unless I can somehow register my form-mail servers, I don't see how I could make use of the PBL.

If you check the other thread (Zen then PBL) you will see an example of the Spamcop Mail implementation working with POP'd spam.

Testing all IP's except the earliest IP (that in last, lowest, Received line in the header) should work without false postives if the implementation is as described. The LOL feature is that the forged header added by spammers makes it work better.

Link to comment
Share on other sites

Ok, well it's not doing that. Here's my other example.

Delivered-To: spamcop-net-my.spamcopid[at]spamcop.net

Received: (qmail 6268 invoked from network); 2 Nov 2007 16:36:37 -0000

X-spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blade1

X-spam-Level:

X-spam-Status: hits=0.0 tests=CONFIRMED_FORGED version=3.2.3

Received: from unknown (192.168.1.107)

by blade1.cesmail.net with QMQP; 2 Nov 2007 16:36:37 -0000

Received: from mx53.cesmail.net (216.154.195.53)

by mx70.cesmail.net with SMTP; 2 Nov 2007 16:36:37 -0000

X-Original-To: me[at]example.org

Delivered-To: m4455123[at]spunkymail-mx7.g.dreamhost.com

Received: from mail.example.com [208.97.132.20]

by mx53.cesmail.net with POP3 (fetchmail-6.2.1)

for my.spamcopid[at]spamcop.net (single-drop); Fri, 02 Nov 2007 12:36:37 -0400 (EDT)

Received: from smarty.dreamhost.com (sd-green-bigip-81.dreamhost.com [208.97.132.81])

by spunkymail-mx7.g.dreamhost.com (Postfix) with ESMTP id 1AAE4BE6D

for <me[at]example.org>; Fri, 2 Nov 2007 09:29:24 -0700 (PDT)

Received: from jawbreaker.dreamhost.com (jawbreaker.dreamhost.com [208.113.141.20])

by smarty.dreamhost.com (Postfix) with ESMTP id AAF5EEE2B7

for <me[at]example.org>; Fri, 2 Nov 2007 09:29:09 -0700 (PDT)

Received: by jawbreaker.dreamhost.com (Postfix, from userid 858040)

id ACF9B845DB; Fri, 2 Nov 2007 09:29:08 -0700 (PDT)

Received: from [64.136.49.229]

by www.example.org (NMS FormMail 3.14m1)

with HTTP; Fri, 2 Nov 2007 16:29:08 GMT

(scri_pt-name /cgi-bin/MySiteFormMail.pl)

(http-host www.example.org)

(http-x-forwarded-for 4.154.245.44, 64.136.48.58)

X-Mailer: NMS FormMail 3.14m1

To: me[at]example.org

From: Jane Doe<sender[at]example.com>

Subject: Re: Subject goes here

Message-Id: <20071102162908.ACF9B845DB[at]jawbreaker.dreamhost.com>

Date: Fri, 2 Nov 2007 09:29:08 -0700 (PDT)

X-SpamCop-Checked: 216.154.195.53 208.97.132.20 208.97.132.81 208.113.141.20 64.136.49.229 4.154.245.44 64.136.48.58

X-SpamCop-Disposition: Blocked pbl.spamhaus.org

And what is that CONFIRMED_FORGED about, anyway?

Link to comment
Share on other sites

Ok, well it's not doing that. Here's my other example.

[...]

Received: from [64.136.49.229]

by www.example.org (NMS FormMail 3.14m1)

with HTTP; Fri, 2 Nov 2007 16:29:08 GMT

(scri_pt-name /cgi-bin/MySiteFormMail.pl)

(http-host www.example.org)

(http-x-forwarded-for 4.154.245.44, 64.136.48.58)

X-Mailer: NMS FormMail 3.14m1

To: me[at]example.org

From: Jane Doe<sender[at]example.com>

[...]

X-SpamCop-Checked: 216.154.195.53 208.97.132.20 208.97.132.81 208.113.141.20 64.136.49.229 4.154.245.44 64.136.48.58

X-SpamCop-Disposition: Blocked pbl.spamhaus.org

And what is that CONFIRMED_FORGED about, anyway?

Need to look at the SpamAssassin website for that I think.

For the pbl problem, maybe the scanner got confused and took the "last" to be the http-x-forwarded-for

?

Anyway suggest you forward a TRACKING URL to Spamcop Mail support as a bug.

Link to comment
Share on other sites

I'll append the following post to this topic, although it would probably better fit with the posts found in:

Update to blacklist options needed...

or maybe I should start a new topic altogether....anyway...

The PBL simply is not properly implemented. I just got a false positive in my Held mail:

http://www.spamcop.net/sc?id=z1525034255z2...1bc1f7b322d7a9z

Here's the "X-SpamCop-Checked" IP list:

66.249.23.241

209.40.203.117

208.72.237.5

207.8.214.6

208.210.124.77

208.180.40.74

75.108.208.114 p

7.5.2.3

The one marked with a "p" above is the only one on the on the SpamHaus PBL, and it appears in the very first Received header in the "chain of custody" of the message (the check of "7.5.2.3" came from a mistaken parsing of "InterMail vM.7.05.02.03"). However, the next Received line is the mail relay server for the user at the "75..." IP, followed by other handoffs before it reached the mailhost responsible for receiving my mail. The reporting parser properly identified [208.72.237.5] as the "message source" that delivered the message to my address. If that IP were on the PBL, then the message should be blocked. That's the way the PBL is supposed to be used.

However, with the installation on the SC email server, too many Received hops are being parsed for PBL listings, and this resulted in the false positive. The PBL (or better yet Zen) lists at SpamHaus are good, but the only way we'll be able to see them used properly for our SC email accounts, IMO, is if a system of defining mailhosts is put in place for SC email users (or better yet, if the email side could simply use the mailhosts many of us have already defined for reporting purposes...and yes, I understand they're on different systems).

Where the heck did that nice and helpful Trevor person go, anyway???

DT

Link to comment
Share on other sites

The PBL simply is not properly implemented. I just got a false positive in my Held mail:

http://www.spamcop.net/sc?id=z1525034255z2...1bc1f7b322d7a9z

Here's the "X-SpamCop-Checked" IP list:

66.249.23.241

209.40.203.117

208.72.237.5

207.8.214.6

208.210.124.77

208.180.40.74

75.108.208.114 p

7.5.2.3

The one marked with a "p" above is the only one on the on the SpamHaus PBL, and it appears in the very first Received header in the "chain of custody" of the message (the check of "7.5.2.3" came from a mistaken parsing of "InterMail vM.7.05.02.03"). However, the next Received line is the mail relay server for the user at the "75..." IP, followed by other handoffs before it reached the mailhost responsible for receiving my mail. The reporting parser properly identified [208.72.237.5] as the "message source" that delivered the message to my address. If that IP were on the PBL, then the message should be blocked. That's the way the PBL is supposed to be used.

However, with the installation on the SC email server, too many Received hops are being parsed for PBL listings, and this resulted in the false positive. The PBL (or better yet Zen) lists at SpamHaus are good, but the only way we'll be able to see them used properly for our SC email accounts, IMO, is if a system of defining mailhosts is put in place for SC email users (or better yet, if the email side could simply use the mailhosts many of us have already defined for reporting purposes...and yes, I understand they're on different systems).

Where the heck did that nice and helpful Trevor person go, anyway???

DT

Actually it looks like it acted the way Trevor said it should work (checking all but the last IP address found). The issue with this message is that the scanner found a second bogus "IP address" in a received line which messed up that part of the code. If you email this information, including the Tracking URL, (or pointer to this topic) to support[at]spamcop.net, Trevor may be able to fix the coding.

Link to comment
Share on other sites

Actually it looks like it acted the way Trevor said it should work (checking all but the last IP address found).

Did you miss my point? That's not how SpamHaus says it should be used. You're only supposed to check the IP of the machine delivering the message to you...period...which is NOT what's being done here.

DT

Link to comment
Share on other sites

Did you miss my point? That's not how SpamHaus says it should be used. You're only supposed to check the IP of the machine delivering the message to you...period...which is NOT what's being done here.

DT

No, I got your point. Perhaps I did not fully explain mine. Trevor has hacked the system to try and incorporate this blocklist to work for a majority of users who have messages forwarded from other accounts into spamcop. If configured how SpamHaus says it should be used, the message you posted earlier should NOT be blocked by this list because the connecting server (66.249.23.241) is not listed on that blocklist. You require a non-standard configuration for this list to work properly for you.

We are going around in circles here and all of this has been stated before. Because of forwarding, how SpamHaus says it should be used would only work for spam sent directly to your SpamCop account because that is the server receiving the message. Or you would need to have ALL your servers configured to use this blocklist. Or perhaps this blocklist is no good for your configuration.

It is not simple to scri_pt finding a specific received header and then working this back to where it got the message. The SpamCop parser is an entire system designed to do that. This is one reason why I believe your solution will not work very well. Changing or multiple servers adds to the complexity on a "real-time" system, unlike the parser which can take a bit longer to process each parse.

My wish is that they would configure it how SpamHaus says it should be used, scanning only the connecting server, and just tell everyone that it will NOT block any messages which are originally received at other accounts and forwarded into spamcop. However, that would likely make it useless for a majority of their customers. Perhaps, this blocklist should simply be dropped as a failed experiment.

It is working great for me, but again, I am not a typical user where all of my real emails come directly to my spamcop account. Only spam comes in through my forwarded accounts (Yahoo and GMail). This makes me not the best to judge what SpamCop should do for their customers.

Link to comment
Share on other sites

Trevor has hacked the system to try and incorporate this blocklist to work for a majority of users who have messages forwarded from other accounts into spamcop.

Please explain that point a bit, Steven. How does the PBL implementation "work for a majority" whose mail is forwarded to SC? I'm sure you're aware that a LOT of spam is sent "direct to MX" from dynamic IPs, many of which are listed on the PBL. Because the implementation skips/ignores that first IP, "direct to MX" spam is NOT going to be detected/held for people whose mail is forwarded. Out of the 95 items currently in my Held mail, none of them were put there due to the PBL. I have various addresses on hosted domains (IOW, not Yahoo, Gmail, etc., but private domains) forwarded to my SC address. I doubt that my profile is very unusual.

Awaiting your explanation,

DT

Link to comment
Share on other sites

Please explain that point a bit, Steven. How does the PBL implementation "work for a majority" whose mail is forwarded to SC? I'm sure you're aware that a LOT of spam is sent "direct to MX" from dynamic IPs, many of which are listed on the PBL. Because the implementation skips/ignores that first IP, "direct to MX" spam is NOT going to be detected/held for people whose mail is forwarded. Out of the 95 items currently in my Held mail, none of them were put there due to the PBL. I have various addresses on hosted domains (IOW, not Yahoo, Gmail, etc., but private domains) forwarded to my SC address. I doubt that my profile is very unusual.

Awaiting your explanation,

DT

I said try. It works better for a majority of users than if it were only picking up the IP address which connected with SpamCop's servers as in the "proper" configuration. In that configuration, it would only work properly for direct to spamcop.net addresses.

I just scanned my spamcop trash (all reported spams for last 2+ days). 2 out of 87 messages were stopped by the PBL. The first was direct to my spamcop mailbox and only had the one IP address and would have been blocked in the suggested configuration. The other came through Yahoo popgate and would not have been blocked because there is no connecting IP address. 80.130.219.187 is listed on pbl.

http://www.spamcop.net/sc?id=z1523328503z7...80cbd9b75b25bfz

Link to comment
Share on other sites

I said try.

Yes, I saw that, but I contend that it is *not* working well "for a majority of users who have messages forwarded from other accounts into spamcop." The two examples you gave weren't messages forwarded to your SC email account. While the current configuration might occasionally work on a given forwarded message if the message happens to have a bogus first Received header inserted, I don't think this should be considered a finished project.

If the SC email system could become aware of specifically designated mail hosts, it could be much better at catching "direct-to-MX" spam, which accounts for a LOT of the spam currently hitting our addresses.

DT

Link to comment
Share on other sites

The PBL simply is not properly implemented. I just got a false positive in my Held mail:

http://www.spamcop.net/sc?id=z1525034255z2...1bc1f7b322d7a9z

[snip]

75.108.208.114 p

7.5.2.3

The one marked with a "p" above is the only one on the on the SpamHaus PBL, and it appears in the very first Received header in the "chain of custody" of the message (the check of "7.5.2.3" came from a mistaken parsing of "InterMail vM.7.05.02.03").

So have you forwarded that TRACKING URL to Spamcop Mail support as a bug ?

Two bugs in fact as will be clear if ip=7.5.2.3 is ever on a Blacklist.

Link to comment
Share on other sites

So have you forwarded that TRACKING URL to Spamcop Mail support as a bug ? Two bugs in fact as will be clear if ip=7.5.2.3 is ever on a Blacklist.

Actually, I reported the misinterpretation of "InterMail" Received lines to JT a little over 9 months ago, and he responded that they would look into it. It appears that didn't result in fixing the problem, as it was clearly the cause of the false positive mentioned above. I suppose I could try again. A month or so ago, TrevorB was actually reading these forums and respoonding to problems like this. It's too bad that he seems to have stopped doing that.

DT

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...