Jump to content
Sign in to follow this  
jboure

drastic decline in mail blocked

Recommended Posts

I subscribe to the blocking list for my company, which gets just under a million (attempted) emails inbound from the internet per day. About 95% of it is spam.

Historically, the spamcopbl has blocked about half (or more) of it. Over the last month I've seen a steady decline in the amount of email that spamcop is catching. I've graphed the last two weeks worth here (don't worry, it's not evil):

http://www.nansi.org/wtf.gif

I've spoken to a couple other mail admins at other sites and heard similar complaints.

Is there anything to what I'm seeing? Are others having the same experience? I'm comfortable with the fact that maybe I screwed something up, but I'm pretty sure I haven't.

Any ideas?

Thanks,

James.

Share this post


Link to post
Share on other sites

Yes if you check you will see almost all the spam is coming from trojanned machines. There are millions of them out there and it is hard for all the block lists to keep up with them.

Share this post


Link to post
Share on other sites
Yes if you check you will see almost all the spam is coming from trojanned machines.

I'm not sure I have a way of doing that. We get around 700k spam per day.

There are millions of them out there and it is hard for all the block lists to keep up with them.

I understand. I don't know if you saw the link to the graph I posted, but the decline has been pretty linear, pretty recent, and pretty drastic. I wonder if others are seeing the same thing?

Share this post


Link to post
Share on other sites
I'm not sure I have a way of doing that. We get around 700k spam per day.

I understand. I don't know if you saw the link to the graph I posted, but the decline has been pretty linear, pretty recent, and pretty drastic. I wonder if others are seeing the same thing?

28266[/snapback]

Not for me (I use the Spamcop mail service filters). My only current concern is a leakage of spam from China which ought to be caught by cn.rbl.cluecentral.net but somehow isn't.

It looks like you need a way of sampling and getting an analysis of the spam that is getting through by origin and type.

Can you tell us what filters and Block lists (besides Spamcop) you use ?

Share this post


Link to post
Share on other sites
Not for me (I use the Spamcop mail service filters).

That's good to know. Got any stats?

It looks like you need a way of sampling and getting an analysis of the spam that is getting through by origin and type.

That would be tricky. The layer after the Spamcop BL is Sophos Puremessage. The spam gets stored in a quarantine, the guts of which I am unfamiliar with.

I guess I could analyze the IPs of all inbound mail and sort it by country or something, but I'm pretty sure I would not get away with just dumping non-ARIN email, for instance. Any other ideas? I've got millions of spams to sort through.

Sophos is doing a good job quarantining what spamcop is missing, but the added load is putting a strain on it.

Can you tell us what filters and Block lists (besides Spamcop) you use ?

I'm not rejecting mail with any other blacklist. My end user community has a low tolerance for false positives, and over the years I've only gotten a small handful of complaints when using spamcopbl.

It might be okay to try out a DUL. If anyone knows of a good free one that would let me rsync, that could be helpful.

Share this post


Link to post
Share on other sites
That's good to know. Got any stats?

Sorry for responding to my own post, buy I just noticed that you weren't using the BL. Sorry about that.

Share this post


Link to post
Share on other sites
Sorry for responding to my own post, buy I just noticed that you weren't using the BL. Sorry about that.

No one but that user knows what filters are in use with a SpamCop e-mail account. As asked before, you've only mentioned two filters/BLs in use ... see http://forum.spamcop.net/forums/index.php?...indpost&p=26820 for a list of other BLs available for use with a SpamCop e-mail account. Most folks use a combination of these, but also noting that there are zillions of other BLs out there, though most not a good idea for your production environment.

Share this post


Link to post
Share on other sites

From my own spam load, what you're seeing isn't something you have done. It's to do with changes to the listing criteria.

Six months or so ago, the mail that ended up in my held mail folder was well over 95% due to hits on the SCBL, with most of the rest being caught by SpamAssassin. Now it's somewhere below 50% getting caught by the SCBL with the rest being caught by pretty much a random distribution of the other lists available to mail users (I have them all active).

spam that actually makes it through to my real inbox has gone from about 1-2 a week to around 2-3 a day. Though 99% of that gets tagged as spam by SA running locally on my mail server.

Changes were made to the listing criteria at around the time that the ability to do the one time remove of a server from the list was added. While the changes (whatever they were) significantly reduced false positives, they also made the SCBL much less effecive. It's a real balancing act which I'm pretty sure Julian reviews regularly.

If you're not using the SBL and XBL filters available to mail users, you might want to add them. They're low on false positives and catch a significant amount of spam.

Edited by GraemeL

Share this post


Link to post
Share on other sites

James,

The drop indicated by your graph is certainly pretty dramatic. I'm not a system admin, but I do wonder if something else technical-related is going on to cause that. Does your system make realtime queries to the BL servers? If so, could system-load related timeouts be a factor? In other words, maybe the BL server itself has gotten slower to respond, and your system gives up and cancels the query. Just an uninformed guess on my part.

I have several SpamCop email accounts, and I've not seen this kind of decline in the effectiveness of the SCBL portion of my filtering.

DT

Share this post


Link to post
Share on other sites

Adding any of the other blocklists or blacklists listed here should be safe (with the exception of the country-specific ones if you do business with people or companies in those countries, noting that bigfoot.com's mailservers are in Korea).

Share this post


Link to post
Share on other sites
Sorry for responding to my own post, buy I just noticed that you weren't using the BL. Sorry about that.

28270[/snapback]

I do use the Spamcop BL and others and finally SpamAssassin set to 3

SpamCop Blacklist bl.spamcop.net www.spamcop.net/bl.shtml

DSBL open relays list.dsbl.org dsbl.org

Spamhaus Blacklist sbl.spamhaus.org www.spamhaus.org/sbl/

South Korea (the country) korea.services.net korea.services.net

China (the country) cn.rbl.cluecentral.net www.cluecentral.net/rbl/

Argentina argentina.blackholes.us www.blackholes.us

Brazil brazil.blackholes.us www.blackholes.us

Composite Blocking List cbl.abuseat.org cbl.abuseat.org

Spamhaus XBL xbl.spamhaus.org www.spamhaus.org/xbl/

About 150,000 spams in two years, one or two a day are not caught and sent to the 'held' folder.

One extra technique is to reject mail from servers with no rDNS and/or a rDNS which indictates a ADSL or dial up IP this should catch zombies anyway.

Share this post


Link to post
Share on other sites
James,

The drop indicated by your graph is certainly pretty dramatic. I'm not a system admin, but I do wonder if something else technical-related is going on to cause that. Does your system make realtime queries to the BL servers? If so, could system-load related timeouts be a factor? In other words, maybe the BL server itself has gotten slower to respond, and your system gives up and cancels the query.  Just an uninformed guess on my part.

That is for sure possible (that I have screwed something up). I have a smokeping DNS probe running against the zone, and it goes off infrequently.

I'm using a local (to the mail server) copy of the smapcopbl zone running in named. It seems to be responding pretty well. I'm rsyincing the zone every 5 minutes. The scri_pt that downloads it, rewrites it BIND-style, verifies it w/ named-checkzone, and then loads it. This is pretty painless, as the zone itself is relatively ( to other blacklists ) small.

I'm not sure how to check if postfix is giving up on the name service. I doubt that is the culprit, because he can still do recursive reverse lookups using the name server that is auth for the spamcopbl.

Share this post


Link to post
Share on other sites
From my own spam load, what you're seeing isn't something you have done. It's to do with changes to the listing criteria.

[snip]

Changes were made to the listing criteria at around the time that the ability to do the one time remove of a server from the list was added. While the changes (whatever they were) significantly reduced false positives, they also made the SCBL much less effecive. It's a real balancing act which I'm pretty sure Julian reviews regularly.

Hmmm, that's a bummer. Spamcop has always been the best list out there in my experience. I've used most of them. I still use most of them as part of Sophos' Puremessage SpamAssassin implementation.

Spamcop has always been my first line of defense, as it's the only list I've ever trusted not to do crazy stuff. I get an astoundingly low number of false positives from it. That's why it's the only list I've trusted to outright reject mail for my end users. Up until a month ago, it was rejecting 500,000 emails a day, and I was getting less than one complaint a month. It's a huge cost savings to me to reject mail rather than quarantine it.

The amount of spam getting to my end users has not increased, thanks to Sophos. My real worry is that the amount of mail getting past spamcop has gone way up, and is really stressing my Sophos implementation. So I'm not looking for advice on other ways to catch spam, I'm mostly curious why my spamcop BL has lost effectiveness so drastically.

Share this post


Link to post
Share on other sites
No one but that user knows what filters are in use with a SpamCop e-mail account.  As asked before, you've only mentioned two filters/BLs in use ... see http://forum.spamcop.net/forums/index.php?...indpost&p=26820 for a list of other BLs available for use with a SpamCop e-mail account.  Most folks use a combination of these, but also noting that there are zillions of other BLs out there, though most not a good idea for your production environment.

28272[/snapback]

I'm not using the e-mail account product, I'm using the blacklist itself.

My second line of defense after the spamcopbl is spamassassin (as sold by Sophos), which is using a slew of BLs and works well.

My question is: is anyone else seeing the same dropoff in effectiveness of just the BL, and if not, any thoughts on what I could be doing wrong? I'm rsyncing the data down and loading it into named (after re-writing it). It's been working very well for years. It just started getting less and less effective over the last few weeks.

Thanks to everyone for their posts,

James.

Share this post


Link to post
Share on other sites
My question is: is anyone else seeing the same dropoff in effectiveness of just the BL, and if not, any thoughts on what I could be doing wrong? I'm rsyncing the data down and loading it into named (after re-writing it). It's been working very well for years. It just started getting less and less effective over the last few weeks.

28319[/snapback]

As most people here do not use the spamcop BL as their sole blocking list (which is not reccommended, as you are probably aware) most people will not be in a position to answer that question.

I use the spamcop email system and over the last 2+ days, bl.spamcop.net has caught 60% of the spam in my Held Mail box. Unfortunately, I can not compare that to any older numbers because I need to perform a specific filter search in order to calculate the number.

Share this post


Link to post
Share on other sites

Reply from Ellen on this topic:

Hi -- I can tell you that we are processing as much spam as we ever have

*but* there are millions and millions of compromised machines out there and

more being compromised every day. For example the sober.q German spam of the

last week -- so many of those appear to have been sent thru IPs no one had

seen before. Even the mail account of mine that runs behind draconian

filters had some of those sliding thru.

I looked at the raw number of IPs listed for the last 7 weeks and while it

varies from week to week, I don't see anything alarming. Last week we had

more IPs than in any of the other weeks I looked at.

I have no idea what other lists the user has activated on their server and

whether they are running in front of or behind the SCbl. If the user isn't

using the CBL, blitzed and maybe a couple of others, then I suggest he

consider doing that also. Considering the amount of spam being sent today

and the number of compromised machines, no one list is going to be able to

stop all the spam.

If the user can identify IP ranges that are getting past the SCbl it would

be interesting to know that and altho I can't guarantee that we could do

something about it at least it would give us a good idea of something to

investigate.

Ellen

SpamCop

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×