jboure Posted May 19, 2005 Share Posted May 19, 2005 I subscribe to the blocking list for my company, which gets just under a million (attempted) emails inbound from the internet per day. About 95% of it is spam. Historically, the spamcopbl has blocked about half (or more) of it. Over the last month I've seen a steady decline in the amount of email that spamcop is catching. I've graphed the last two weeks worth here (don't worry, it's not evil): http://www.nansi.org/wtf.gif I've spoken to a couple other mail admins at other sites and heard similar complaints. Is there anything to what I'm seeing? Are others having the same experience? I'm comfortable with the fact that maybe I screwed something up, but I'm pretty sure I haven't. Any ideas? Thanks, James. Link to comment Share on other sites More sharing options...
Merlyn Posted May 20, 2005 Share Posted May 20, 2005 Yes if you check you will see almost all the spam is coming from trojanned machines. There are millions of them out there and it is hard for all the block lists to keep up with them. Link to comment Share on other sites More sharing options...
jboure Posted May 20, 2005 Author Share Posted May 20, 2005 Yes if you check you will see almost all the spam is coming from trojanned machines. I'm not sure I have a way of doing that. We get around 700k spam per day. There are millions of them out there and it is hard for all the block lists to keep up with them. I understand. I don't know if you saw the link to the graph I posted, but the decline has been pretty linear, pretty recent, and pretty drastic. I wonder if others are seeing the same thing? Link to comment Share on other sites More sharing options...
michaelanglo Posted May 20, 2005 Share Posted May 20, 2005 I'm not sure I have a way of doing that. We get around 700k spam per day. I understand. I don't know if you saw the link to the graph I posted, but the decline has been pretty linear, pretty recent, and pretty drastic. I wonder if others are seeing the same thing? 28266[/snapback] Not for me (I use the Spamcop mail service filters). My only current concern is a leakage of spam from China which ought to be caught by cn.rbl.cluecentral.net but somehow isn't. It looks like you need a way of sampling and getting an analysis of the spam that is getting through by origin and type. Can you tell us what filters and Block lists (besides Spamcop) you use ? Link to comment Share on other sites More sharing options...
jboure Posted May 20, 2005 Author Share Posted May 20, 2005 Not for me (I use the Spamcop mail service filters). That's good to know. Got any stats? It looks like you need a way of sampling and getting an analysis of the spam that is getting through by origin and type. That would be tricky. The layer after the Spamcop BL is Sophos Puremessage. The spam gets stored in a quarantine, the guts of which I am unfamiliar with. I guess I could analyze the IPs of all inbound mail and sort it by country or something, but I'm pretty sure I would not get away with just dumping non-ARIN email, for instance. Any other ideas? I've got millions of spams to sort through. Sophos is doing a good job quarantining what spamcop is missing, but the added load is putting a strain on it. Can you tell us what filters and Block lists (besides Spamcop) you use ? I'm not rejecting mail with any other blacklist. My end user community has a low tolerance for false positives, and over the years I've only gotten a small handful of complaints when using spamcopbl. It might be okay to try out a DUL. If anyone knows of a good free one that would let me rsync, that could be helpful. Link to comment Share on other sites More sharing options...
jboure Posted May 20, 2005 Author Share Posted May 20, 2005 That's good to know. Got any stats? Sorry for responding to my own post, buy I just noticed that you weren't using the BL. Sorry about that. Link to comment Share on other sites More sharing options...
Wazoo Posted May 20, 2005 Share Posted May 20, 2005 Sorry for responding to my own post, buy I just noticed that you weren't using the BL. Sorry about that. No one but that user knows what filters are in use with a SpamCop e-mail account. As asked before, you've only mentioned two filters/BLs in use ... see http://forum.spamcop.net/forums/index.php?...indpost&p=26820 for a list of other BLs available for use with a SpamCop e-mail account. Most folks use a combination of these, but also noting that there are zillions of other BLs out there, though most not a good idea for your production environment. Link to comment Share on other sites More sharing options...
GraemeL Posted May 20, 2005 Share Posted May 20, 2005 From my own spam load, what you're seeing isn't something you have done. It's to do with changes to the listing criteria. Six months or so ago, the mail that ended up in my held mail folder was well over 95% due to hits on the SCBL, with most of the rest being caught by SpamAssassin. Now it's somewhere below 50% getting caught by the SCBL with the rest being caught by pretty much a random distribution of the other lists available to mail users (I have them all active). spam that actually makes it through to my real inbox has gone from about 1-2 a week to around 2-3 a day. Though 99% of that gets tagged as spam by SA running locally on my mail server. Changes were made to the listing criteria at around the time that the ability to do the one time remove of a server from the list was added. While the changes (whatever they were) significantly reduced false positives, they also made the SCBL much less effecive. It's a real balancing act which I'm pretty sure Julian reviews regularly. If you're not using the SBL and XBL filters available to mail users, you might want to add them. They're low on false positives and catch a significant amount of spam. Link to comment Share on other sites More sharing options...
DavidT Posted May 20, 2005 Share Posted May 20, 2005 James, The drop indicated by your graph is certainly pretty dramatic. I'm not a system admin, but I do wonder if something else technical-related is going on to cause that. Does your system make realtime queries to the BL servers? If so, could system-load related timeouts be a factor? In other words, maybe the BL server itself has gotten slower to respond, and your system gives up and cancels the query. Just an uninformed guess on my part. I have several SpamCop email accounts, and I've not seen this kind of decline in the effectiveness of the SCBL portion of my filtering. DT Link to comment Share on other sites More sharing options...
Jeff G. Posted May 20, 2005 Share Posted May 20, 2005 Adding any of the other blocklists or blacklists listed here should be safe (with the exception of the country-specific ones if you do business with people or companies in those countries, noting that bigfoot.com's mailservers are in Korea). Link to comment Share on other sites More sharing options...
michaelanglo Posted May 20, 2005 Share Posted May 20, 2005 Sorry for responding to my own post, buy I just noticed that you weren't using the BL. Sorry about that. 28270[/snapback] I do use the Spamcop BL and others and finally SpamAssassin set to 3 SpamCop Blacklist bl.spamcop.net www.spamcop.net/bl.shtml DSBL open relays list.dsbl.org dsbl.org Spamhaus Blacklist sbl.spamhaus.org www.spamhaus.org/sbl/ South Korea (the country) korea.services.net korea.services.net China (the country) cn.rbl.cluecentral.net www.cluecentral.net/rbl/ Argentina argentina.blackholes.us www.blackholes.us Brazil brazil.blackholes.us www.blackholes.us Composite Blocking List cbl.abuseat.org cbl.abuseat.org Spamhaus XBL xbl.spamhaus.org www.spamhaus.org/xbl/ About 150,000 spams in two years, one or two a day are not caught and sent to the 'held' folder. One extra technique is to reject mail from servers with no rDNS and/or a rDNS which indictates a ADSL or dial up IP this should catch zombies anyway. Link to comment Share on other sites More sharing options...
jboure Posted May 21, 2005 Author Share Posted May 21, 2005 James, The drop indicated by your graph is certainly pretty dramatic. I'm not a system admin, but I do wonder if something else technical-related is going on to cause that. Does your system make realtime queries to the BL servers? If so, could system-load related timeouts be a factor? In other words, maybe the BL server itself has gotten slower to respond, and your system gives up and cancels the query. Just an uninformed guess on my part. That is for sure possible (that I have screwed something up). I have a smokeping DNS probe running against the zone, and it goes off infrequently. I'm using a local (to the mail server) copy of the smapcopbl zone running in named. It seems to be responding pretty well. I'm rsyincing the zone every 5 minutes. The scri_pt that downloads it, rewrites it BIND-style, verifies it w/ named-checkzone, and then loads it. This is pretty painless, as the zone itself is relatively ( to other blacklists ) small. I'm not sure how to check if postfix is giving up on the name service. I doubt that is the culprit, because he can still do recursive reverse lookups using the name server that is auth for the spamcopbl. Link to comment Share on other sites More sharing options...
jboure Posted May 21, 2005 Author Share Posted May 21, 2005 From my own spam load, what you're seeing isn't something you have done. It's to do with changes to the listing criteria. [snip] Changes were made to the listing criteria at around the time that the ability to do the one time remove of a server from the list was added. While the changes (whatever they were) significantly reduced false positives, they also made the SCBL much less effecive. It's a real balancing act which I'm pretty sure Julian reviews regularly. Hmmm, that's a bummer. Spamcop has always been the best list out there in my experience. I've used most of them. I still use most of them as part of Sophos' Puremessage SpamAssassin implementation. Spamcop has always been my first line of defense, as it's the only list I've ever trusted not to do crazy stuff. I get an astoundingly low number of false positives from it. That's why it's the only list I've trusted to outright reject mail for my end users. Up until a month ago, it was rejecting 500,000 emails a day, and I was getting less than one complaint a month. It's a huge cost savings to me to reject mail rather than quarantine it. The amount of spam getting to my end users has not increased, thanks to Sophos. My real worry is that the amount of mail getting past spamcop has gone way up, and is really stressing my Sophos implementation. So I'm not looking for advice on other ways to catch spam, I'm mostly curious why my spamcop BL has lost effectiveness so drastically. Link to comment Share on other sites More sharing options...
jboure Posted May 21, 2005 Author Share Posted May 21, 2005 No one but that user knows what filters are in use with a SpamCop e-mail account. As asked before, you've only mentioned two filters/BLs in use ... see http://forum.spamcop.net/forums/index.php?...indpost&p=26820 for a list of other BLs available for use with a SpamCop e-mail account. Most folks use a combination of these, but also noting that there are zillions of other BLs out there, though most not a good idea for your production environment. 28272[/snapback] I'm not using the e-mail account product, I'm using the blacklist itself. My second line of defense after the spamcopbl is spamassassin (as sold by Sophos), which is using a slew of BLs and works well. My question is: is anyone else seeing the same dropoff in effectiveness of just the BL, and if not, any thoughts on what I could be doing wrong? I'm rsyncing the data down and loading it into named (after re-writing it). It's been working very well for years. It just started getting less and less effective over the last few weeks. Thanks to everyone for their posts, James. Link to comment Share on other sites More sharing options...
StevenUnderwood Posted May 21, 2005 Share Posted May 21, 2005 My question is: is anyone else seeing the same dropoff in effectiveness of just the BL, and if not, any thoughts on what I could be doing wrong? I'm rsyncing the data down and loading it into named (after re-writing it). It's been working very well for years. It just started getting less and less effective over the last few weeks. 28319[/snapback] As most people here do not use the spamcop BL as their sole blocking list (which is not reccommended, as you are probably aware) most people will not be in a position to answer that question. I use the spamcop email system and over the last 2+ days, bl.spamcop.net has caught 60% of the spam in my Held Mail box. Unfortunately, I can not compare that to any older numbers because I need to perform a specific filter search in order to calculate the number. Link to comment Share on other sites More sharing options...
StevenUnderwood Posted May 23, 2005 Share Posted May 23, 2005 Reply from Ellen on this topic: Hi -- I can tell you that we are processing as much spam as we ever have *but* there are millions and millions of compromised machines out there and more being compromised every day. For example the sober.q German spam of the last week -- so many of those appear to have been sent thru IPs no one had seen before. Even the mail account of mine that runs behind draconian filters had some of those sliding thru. I looked at the raw number of IPs listed for the last 7 weeks and while it varies from week to week, I don't see anything alarming. Last week we had more IPs than in any of the other weeks I looked at. I have no idea what other lists the user has activated on their server and whether they are running in front of or behind the SCbl. If the user isn't using the CBL, blitzed and maybe a couple of others, then I suggest he consider doing that also. Considering the amount of spam being sent today and the number of compromised machines, no one list is going to be able to stop all the spam. If the user can identify IP ranges that are getting past the SCbl it would be interesting to know that and altho I can't guarantee that we could do something about it at least it would give us a good idea of something to investigate. Ellen SpamCop Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.