Stats: my RBL verses SCBL

[Mabu]

First and foremost let me say I love Spamcop's RBL and I'm one of the service's biggest boosters to the Internet community. However, I'm beginning to believe it just isn't as effective as it used to be.

A little background: I run a dedicated mail server for several hundred clients who include everything from individual/non-commercial users to large corporations, government agencies and several prominent media groups. I consider my customers to be a very good mix of a wide variety of demographics for typical online users. I am very anal about making sure legit mail is not blocked and would prefer to err on the side of caution. I employ no content-based filtering, only RBLs, and I stop about 95% of spam this way.

I haven't done a lot of research but my impression is that the current criteria for SCBL is that after x amount of time, blacklisted entries expire from the database. This would explain the ineffectiveness of the RBL over time, as spammers now operate from ever-rotating, very large chunks of DUL IP space that they can move through without much problems from Spamcop. In an attempt to deal with this, I've set up my own Sendmail access-based RBL and here are the stats for the past week:

Date, received mail, invalid users, Spamcop caught, Spamhaus caught, my own RBL caught

Jan 26 00:00:00, 4648, 1084, 2201, 528, 16132

Jan 27 00:00:00, 4634, 1488, 2280, 608, 16011

Jan 28 00:00:00, 2634, 1086, 2622, 535, 14516

Jan 29 00:00:00, 2654, 1284, 2465, 624, 16172

Jan 30 00:00:00, 5173, 1401, 2577, 645, 15844

Jan 31 00:00:00, 5370, 1490, 3493, 628, 17387

Feb 1 00:00:00, 4997, 1197, 2290, 617, 15640

Feb 2 00:00:00, 5130, 1129, 2782, 778, 18250

Feb 3 00:00:00, 4478, 1119, 2967, 649, 17198

Feb 4 00:00:00, 2823, 788, 2596, 532, 13251

According to the figures above...

* This server receives 24,756 daily e-mails on average, of which 19,281 (77.9%) are confirmed spam.

* If you consider most invalid user e-mails to be spam or spam bounces, then the UCE rate jumps to around 82%

* Spamcop's RBL catches 2627 spams a day or 13.63% of confirmed spam

* Spamnaus's RBL catches 614 spams per day

* My homebrew RBL catches 16040 spams a day or 83.18% of confirmed spam

It looks like my RBL is much more effective than either Spamcop or Spamhaus's efforts.

What am I doing differently? Well, I am not removing IPs from the database unless I'm specifically asked. If I'm asked, then I do so without any additional questions. In about five years, I've had maybe 1-2 dozen reports of legitimate mail being blocked, and in every case, I fixed this quickly. In the same period of time, I've probably had a similar amount of legit mail blocked complaints involving both SCBL and Spamhaus. I sometimes do wholesale IP range blocks when I run across a spam I can identify as coming from DUL space. I also have aggressively identified "rogue spam nations" like China, Korea and others and have most of their class Bs RBL'd. If I ever get any requests for removal, I whitelist IP blocks upon request.

In fairness, it may be possible that my access-based RBL might be checked first, before Spamcop or Spamhaus (can someone verify this) and this might be a factor in my system statistically seeming far superior to Spamcop. I guess the best way to test this would be to remove my RBL and tabulate some figures alone based on the existing RBLs, but I'm still pretty certain that my system is at least, just as effective and most likely more than SCBL.

My feeling is that, it is now no longer an option, but a NECESSITY to PERMANENTLY RBL DUL SPACE that should not have outbound SMTP traffic. AOL and a few responsible ISPs have finally decided to filter port 25 and this has made a tremendously positive impact on the reduction of spam, but others like Mindspring, Verizon, TDE, AT&T, Comcast and others have not taken action. As a result, I think responsible ISPs should just stop accepting mail from their DUL IP space. We need to force these systems into policing their own users' illegal activities. If Spamcop is expiring DUL IP RBL entries, then the service is nowhere near as useful as it needs to be.

Comments?

[Wazoo]

First comment: what does this have to do with "Reporting Help" which is where this Topic was posted? A more likely spot may have been the Block List Forum section, but again, that's a Help section for dealing with the SpamCopDNSBL, and I don't see that you're asking for help.

As this Topic doesn't fit the description for either of those Forum sections, it is being moved to the Lounge.

[StevenUnderwood]

Comments?

40070[/snapback]

Any percentages will be affected by the order in which the blocks are applied unless you are applying all the blocklists to every message (even if they are already marked positive). SpamCop's own email system used to have fairly low percentage of SpamAssassin hits until they changed and have that check first...now it looks like the RBL's are less effective.

[Mabu]

First comment: what does this have to do with "Reporting Help" which is where this Topic was posted?  A more likely spot may have been the Block List Forum section, but again, that's a Help section for dealing with the SpamCopDNSBL, and I don't see that you're asking for help.

As this Topic doesn't fit the description for either of those Forum sections, it is being moved to the Lounge.

40077[/snapback]

Wow... forum Nazis here.... You people need to relax a little bit.

Your guess is as good as mine. It seems a discussion about my experience with SCBL would fit in somewhere.

The operative issue here is:

1. Any Sendmail experts know if an access-based (i.e. connect:x.x.x.x 550 REJECT) takes precedence over the FEATURE() enabling of RBLs? Then we can put that issue to rest.

2. Is Spamcop still auto-expiring IP entries, even for confirmed IP space that shouldn't have SMTP traffic?

If so, then the way the Spammers are now operating, Spamcop's RBL isn't that useful.

I am willing to make my RBL data available to anyone that wants it.

[petzl]

I am willing to make my RBL data available to anyone that wants it.

40084[/snapback]

There are more advantages to SpamCop than blocking. SpamCop also attempts to report the spam sent to the ISP concerned meaning another spam hole is closed

I could just "block all" except whitelisted email which would catch more than your RBL (I believe Whitelisted IP's will eventually be the only servers accept email (IMO)

Been using the SCBL for years and do not get false positives (I use whitelisting)how many of numbers you list are false positives.

Because of SpamCop I also have the only email address I will ever need and have used that address openly since well into last century, It is my main email used for all business purposess and contact with friends

SpamCop will list an IP in seconds stopping a spam run while it is happening. Not after, releasing that IP when the spam stops being sent

With a SpamCop email account all spam trapped is easily reported which then attempts to notify the ISP that they have a problem so it can be corrected (and this is almost always the case) By having that spam source stopped it eliminates spammers closing the hole they crawl through

This all said if you have a better mousetrap I'm sure some may bother to test it out and advise further

[Wazoo]

Wow... forum Nazis here....  You people need to relax a little bit.

Your guess is as good as mine.  It seems a discussion about my experience with SCBL would fit in somewhere.

40084[/snapback]

Actually, it would be more appreciated that one would recognise that this Forum has been set-up to allow discussions on a number of different subjects and follow that bit of organization. As stated, there was nothing in your post that dealt with any issues involved with the parsing and reporting of spam via the SpamCop.net tool-set. This bit of outburst has led to an initial warning against your account.

[Miss Betsy]

I am not a server admin so my comments will not be very useful to you.

However, blocking Comcast and those who allow continued spam sounds like a good idea to me.

From other topics, IIUC, a combination of blocklists is the most effective in stopping spam. Depending on various factors, how blocklists are applied is a highly individual one.

As petzl said, a good thing about the scbl system is that it has become an early warning system for admins with problems because of the reports - if they pay attention to the reports, they can avoid being put on rbls such as yours where they have to ask to be removed. For blocking of entrenched spammers, other lists are more effective.

IMHO, there is no such thing as a false positive - whoever is knowingly using an email service that allows spamming is contributing to the spam problem. Ideally, any interruption of email due to scbl blocking would be short in duration since the admin would stop the source of spam immediately upon receiving reports.

Miss Betsy

ps

Wow... forum Nazis here.... You people need to relax a little bit.

Your guess is as good as mine. It seems a discussion about my experience with SCBL would fit in somewhere.

And it does - in the Lounge. The reason that we try to get things organized is so that people looking for help don't get exasperated by looking at topics that have nothing to do with their problem.

Unfortunately, there is no way to explain the reasons for the organization (or, at least, we haven't found it yet) that everyone sees when they get to the forum.

[petzl]

I am not a server admin so my comments will not be very useful to you.

40092[/snapback]

The answer to blocking using DUL listings (as this guy claims he does) is that everyone sends from them (a Dail Up Listing) and means ones/a blocking list would be very high with false positives using this DUL list!

Only from a properly configured servers that do list the destination chain, or IP (source) of the computer that sent the email/spam

If all ISP's configured their email severs listing the actual IP that sent the email (as they should). SpamCop's job in blocking reported spam would be more easy and correct. If this IP is not recorded SpamCop will block the misconfigured email server (SpamCop stops at last positive chain in spam)

[Mabu]

Well, those that use Spamcop for their main e-mail address.. my comments are not relevant to that situation..

People who like me, run mail servers that use SCBL, can chime in and let me know what they think. Notwithstanding the two-bit attitude from the moderator here... go ahead and warn me. If thinking you're a wanker is grounds for getting me banned, then go ahead and ban me. I'm only trying to help, and I've been working with spamcop and loyally reporting spam since the beginning, so pardon me if I feel I am entitled to drop in and express my undiplomatic opinion. We're battling an army of scumbags who steal peoples resources and I don't have patience for them, and I don't have patience for power-tripping moderators who want to "warn" me for my "outburst" even though I am on topic and trying to point out an issue. Sheesh.

With that out of the way, the issue here is, shouldn't there be a version of the SCBL where IP ranges do not expire? This seems to make a lot of sense. Is there another RBL that works like this? The DUL space for most broadband ISPs like Comcast, Bellsouth, Verizon and others should be permanently RBL'd and then whitelisted on an IP-by-IP basis IMO. This may shut out people who want to run their own rogue SMTP server legitimately, but in the long run it's better off IMO.

Anyway, I guess if you don't see another post from me, it's because I've offended Wazoo's ego. C'est la vie. Just remember, banning me doesn't reverse the fact that I think you're a dork for your heavy-handed control crap. You do this community and old-timers who helped make Spamcop successful in the first place, a profound disservice. It would be wise for you to loosen up.

[Jeff G.]

shouldn't there be a version of the SCBL where IP ranges do not expire?  This seems to make a lot of sense.

40520[/snapback]

No, that would be outside the SCBL's stated dynamic scope

Is there another RBL that works like this?

40520[/snapback]

Yes, certainly.

The DUL space for most broadband ISPs like Comcast, Bellsouth, Verizon and others should be permanently RBL'd and then whitelisted on an IP-by-IP basis IMO.

40520[/snapback]

There's no need to re-invent the wheel - there are multiple DUL lists available.

Also, "wanker" is a pejorative term that shouldn't be used here.

[Wazoo]

People who like me, run mail servers that use SCBL, can chime in and let me know what they think.  Notwithstanding the two-bit attitude from the moderator here... go ahead and warn me.  If thinking you're a wanker is grounds for getting me banned, then go ahead and ban me.

You'll note that the intentional goading has yet to accomplish what you seem to be asking for.

I'm only trying to help, and I've been working with spamcop and loyally reporting spam since the beginning, so pardon me if I feel I am entitled to drop in and express my undiplomatic opinion.  We're battling an army of scumbags who steal peoples resources and I don't have patience for them, and I don't have patience for power-tripping moderators who want to "warn" me for my "outburst" even though I am on topic and trying to point out an issue.  Sheesh.

On Topic????? Nothing to do with a Reporting issue, which is where you dropped this Topic originally. And as stated, your posting did not actually include a question on the use and/or implementation of the SpamCopDNSBL. Where / How do you see "On Topic" in there anywhere? Had your original posting been made in the Forum section set aside for things not being a "Help" type query, none of this bit of silliness would have come up in the first place.

With that out of the way, the issue here is, shouldn't there be a version of the SCBL where IP ranges do not expire?  This seems to make a lot of sense.

If you'll read the FAQ on What is on the SpamCopDNSBL, perhaps you can explain your logic...???

Is there another RBL that works like this?

Like what? Like the SpamCOpDNSBL? No, this BL is unique as are most of the others, which also helps to explain why there are so many available.

The DUL space for most broadband ISPs like Comcast, Bellsouth, Verizon and others should be permanently RBL'd and then whitelisted on an IP-by-IP basis IMO.  This may shut out people who want to run their own rogue SMTP server legitimately, but in the long run it's better off IMO.

There are BLs that do exactly this.

Anyway, I guess if you don't see another post from me, it's because I've offended Wazoo's ego.  C'est la vie.  Just remember, banning me doesn't reverse the fact that I think you're a dork for your heavy-handed control crap.  You do this community and old-timers who helped make Spamcop successful in the first place, a profound disservice.  It would be wise for you to loosen up.

40520[/snapback]

This silly bit affecting my ego? Surely you jest in addition to showing some foolishness. "On-Topic" has a specific definition, thus the movement of your original post to a place where it could be considered as such. You seem to suggest that the action taken should have been to delete the post (and now, deleting the follow-on dialog .. and your choice of words seems to make it seem like this is what happened..??) Note that it's still in place. But yes, you are trying the patience of a few folks. You can continue to discuss your opinions, but there is no need for this ridiculous bit of name-calling on the Moderating actions taken.

You have been provided a platform, made your statements, even have some dialog going on .... I fail to see what your problem can yet be, short of the apparent lack of research.

[turetzsr]

<snip>

You have been provided a platform, made your statements, even have some dialog going on .... I fail to see what your problem can yet be, short of the apparent lack of research.

40523[/snapback]

...Methinks it's more a matter of the pot calling the kettle black (or, in this case, egotistical):

<snip>

Anyway, I guess if you don't see another post from me, it's because I've offended Wazoo's ego.

<g>

[Miss Betsy]

People who like me, run mail servers that use SCBL, can chime in and let me know what they think.

With that out of the way, the issue here is, shouldn't there be a version of the SCBL where IP ranges do not expire?  This seems to make a lot of sense.  Is there another RBL that works like this?  The DUL space for most broadband ISPs like Comcast, Bellsouth, Verizon and others should be permanently RBL'd and then whitelisted on an IP-by-IP basis IMO.  This may shut out people who want to run their own rogue SMTP server legitimately, but in the long run it's better off IMO.

40520[/snapback]

Basically, if you are aware of the scbl, you should be aware of the 400 other bl's out there. Posts don't convey the 'body language' but I am sure that a lot of people who are server admins are wondering how come you don't know about spamhaus and SPEWS (though I am not exactly sure of the criteria for either one, I do know that IP addresses don't expire from them. In fact, the scbl is the /only/ bl that is automatic, I believe.)

Most server admins use a combination of bls to do their filtering.

And since server admins have the most contact with spam, they are a very suspicious lot. One has to /prove/ that s/he is not a spammer-in-disguise before they will even listen to what you have to say - at least from my observation. They also tend to be blunt in their dialogue and rarely take offense at blunt language from another server admin.

The world is changing though - I don't know what 'wanker' means, but it seems that it is not a nice way of expressing yourself and that even server admins (at least on this forum) prefer less color in their blunt observations.

Miss Betsy

[Jeff G.]

I don't know what 'wanker' means

40542[/snapback]

Please see http://en.wiktionary.org/wiki/wanker. Thanks!

[turetzsr]

Please see http://en.wiktionary.org/wiki/wanker.  Thanks!

40543[/snapback]

...But only if you're not offended by standard English sexual expressions.

...As a PM, I provided Miss Betsy a sanitized (but perhaps too obscure) definition.

[turetzsr]

<snip>

The world is changing though - I don't know what 'wanker' means, but it seems that it is not a nice way of expressing yourself and that even server admins (at least on this forum) prefer less color in their blunt observations.

40542[/snapback]

...My guess is that the objection was to borderline (or is it actually explicitly?) profanity in a public forum, not its bluntness.

[Farelf]

...My guess is that the objection was to borderline (or is it actually explicitly?) profanity in a public forum, not its bluntness.

40545[/snapback]

The word (merchant banker is the rhyming slang version) is intended to be offensive - Wazoo would have every right to be duly offended, therefore Jeff too, on his own account, on Wazoo's behalf and on behalf of all the "readership".

In the pre-internet days a magistrate from around here (Geraldton or Carnarvon, I forget which), in a fine display of "judicial innocence", ruled against the police prosecutor in a disorderly conduct case when the unfortunate man refused to disturb the dignity of the court by defining its meaning for His Worship. "If you don't know what it means, you have no right to be offended by it." was the sense of the ruling. Even then, that ruling was defective, not only because it failed to take into account the intent of the "offender" as might be expected under our (English) legal system but particularly because the magistrate suppressed evidence, he relied on the decency of a public official to manipulate him - simply to avoid, for the moment, adding to the lexicon of "forbidden utterances" (courts hate creating precedent, even at the expense of public order in this case).

The point is, in these days of the wiki and other instant references His Worship would have a hard time pulling a similar stunt (that and the fact he's long since gone to his reward). In this profane age new terms of obscenity are ameliorated to the point of innocuousness virtually overnight (cf dork) - the elder words take a little longer. IMO, and severely OT. The word is offensive and derisory in current usage, the way it was used and (evidently) the way it was understood.

[Miss Betsy]

In this profane age new terms of obscenity are ameliorated to the point of innocuousness virtually overnight......

I hate to spoil my image, but I am aware of what obscene terms generally refer to.

While it is good to know that a word is not considered innocuous (so that one doesn't use it inappropriately), it is not necessary to use such words in order to be derisive. And, conversely, some people use them so frequently that the words no longer have any connotations of offensive intent. However, in some venues, certain terms are offensive whether or not there is an intent. That's what my point was - though obviously too subtle.

Miss Betsy

[Farelf]

... However, in some venues, certain terms are offensive whether or not there is an intent.  That's what my point was - though obviously too subtle. ...

40550[/snapback]

Not too subtle at all - (for myself) just didn't agree the point was entirely applicable in this instance.

[Jeff G.]

The word is offensive and derisory in current usage, the way it was used and (evidently) the way it was understood.

40547[/snapback]

Certainly in the way it was understood by me (I actually did learn something on that trip to the UK).

I also "am aware of what obscene terms generally refer to", but this is a public forum that could easily be read by minors, so I was acting in accord with the original intent to keep it clean.