Why does SpamCop release so much spam to me?

[shull2805@spamcop.net]

Lately, I've been concerned about the number of spams that slip past SpamCop (as opposed to being held for my review). Yesterday, within a 24-hr period, 192 spams reached my Inbox (that's AFTER passing through the SpamCop filters). I looked a little closer, and was puzzled by the fact that some of the spam that got caught and held appeared to be very similar to spam that was being forwarded. Here are some examples of what I'm talking about:

These emails were caught by SpamCop, and held for my review. I forwarded them to my submit address (i.e., NOT Quick Reported):

1) Subject: Re: [66:HHY]-Meddications

http://www.spamcop.net/sc?id=z742090570z0a...292d0c8774e7e5z

2) Subject: Re: [15:WCJ]-Medicattions

http://www.spamcop.net/sc?id=z742090572z4e...9c182e99d360d4z

3) Subject: Re: [73:TVO]-Medicattions

http://www.spamcop.net/sc?id=z742090573z05...9d8d458ca3dad7z

The following 3 emails were NOT caught by SpamCop and ended up in my Inbox. I also forwarded these to my submit address:

4) Subject: Re: [91:CKY]-Mediccations

http://www.spamcop.net/sc?id=z742096268zdd...574e73bbd00d69z

5) Subject: Re: KF94[Meddications]

http://www.spamcop.net/sc?id=z742096271z08...4d2d76e64b6ad3z

6) Subject: Re: -QM:52-Medicationns

http://www.spamcop.net/sc?id=z742096273z41...26cc112482e9eaz

Note that in all 6 instances, SpamCop was unhappy with the format of my submissions; although it could parse the headers, it thought the body of the email was missing (this problem has already been reported).

It seems obvious that all 6 of these spams are from the same source, although they may have taken different routes to get to me. Do the Black Lists ONLY look at the headers? Would it be correct to say that if spam 'A' took route 'A', spam 'B' took route 'B', and only route 'A' uses email servers on one of the Black Lists, that spam 'A' would be held and spam 'B' would sail right through SpamCop? I have received hundreds of spams identical to these 6 over the past couple of weeks, and have reported them, but obviously many still slip through the SpamCop filters.

Is SpamAssassin the only filter available at SpamCop that can examine message content? I had it configured at level 5, just changed it to 4 to see if that helps. In looking at the headers for example 6, does the "X-SpamLevel: **" mean I would have to lower my SpamAssassing threshold to 2 in order for it to flag this email as spam? Example 1 also had an X-SpamLevel of '**', so apparently SpamAssassin was not a contributing factor in flagging Ex. 1 as a spam.

Any suggestions for how I can reduce the amount of spam still getting through would be appreciated. Thanks!

[Jeff G.]

#1 was held because it came through IP Address 200.97.120.17 brazil.blackholes.us (in Brazil).

#2 was held because it came through IP Address 200.106.14.130 on list.dsbl.org.

#3 was held because it came through IP Address 4.244.102.172 on dnsbl.sorbs.net.

#4 was not held because none of the IP Addresses in its Received Header Lines were in any of the blocklists you have selected, and because the 2.9 hits on SpamAssassin did not meet or exceed your SpamAssassin threshold.

#5 was not held because none of the IP Addresses in its Received Header Lines were in any of the blocklists you have selected, and because the 2.3 hits on SpamAssassin did not meet or exceed your SpamAssassin threshold.

#6 was not held because none of the IP Addresses in its Received Header Lines were in any of the blocklists you have selected, and because the 2.3 hits on SpamAssassin did not meet or exceed your SpamAssassin threshold.

Yes, the blocklist examination methodology only looks at the Received Header Lines.

Yes, it would "be correct to say that if spam 'A' took route 'A', spam 'B' took route 'B', and only route 'A' uses email servers on one of the Black Lists, that spam 'A' would be held and spam 'B' would sail right through SpamCop", but only if the number of hits on SpamAssassin did not meet or exceed your SpamAssassin threshold.

Yes, SpamAssassin is the only filter available at SpamCop that can examine message content.

Yes, you would have to lower your SpamAssassin threshold to 2 in order for it to flag all of those emails as spam (3 wouldn't suffice for any of them).

I would suggest that you lower your SpamAssassin threshold to 2 and keep a close eye out for false positives in your Held Mail mailbox/Folder.

I have had to resort to manually applying filters to my Held Mail mailbox/Folder before reporting its contents in order to more consistently rescue some false positives.

[shull2805@spamcop.net]

Thanks for the prompt reply, Jeff. I just checked out the SpamAssassin web site & FAQ. I noticed that SpamAssassin has a Bayes filter option. Does SpamCop enable that option? How does that work for multiple users? For instance, I've been receiving large quantities of the previously mentioned spam. So you would think that SpamAssassin would know by now that it's spam. But, if I'm the only SpamCop user getting this particular spam, even though it's a significant source of spam to me, it may be an insignificant fraction of the email going through your servers each day. I guess I'm asking if there's anything I can do to train SpamAssassin to get a better handle on which of my emails are spam vs. legit.

Thanks again.

[StevenUnderwood]

There were several discussions about this about a year ago, the most recent I could find that also contained some kind of answer is:

http://forum.spamcop.net/forums/index.php?...findpost&p=3240

and http://forum.spamcop.net/forums/index.php?showtopic=534

Try performing a search with +SpamAssassin +Bayes and read some of those threads.

[Jeff G.]

IIRC, the end result of the Bayes experiment was that required too much CPU power and provided too little benefit. In retrospect, perhaps a much larger dictionary would have helped, due to the wide variety of spam spewed by the wide variety of spammers that have goaded us SpamCop Email System Customers into plonking down our hard-earned money to become Customers. The number of different ways spammers spell Viagra and Cialis in spam is positively frightening.

[shull2805@spamcop.net]

OK, so SpamCop (or SpamAssassin) isn't using the Bayesian filter option. I've dropped my SpamAssassin threshold to 2 per Jeff's suggestion. Thanks again.

- Steve

[Derek T]

spamcop.net,Mar 15 2005, 03:33 PM]Lately, I've been concerned about the number of spams that slip past SpamCop (as opposed to being held for my review).  Yesterday, within a 24-hr period, 192 spams reached my Inbox (that's AFTER passing through the SpamCop filters).  I looked a little closer, and was puzzled by the fact that some of the spam that got caught and held appeared to be very similar to spam that was being forwarded.  Here are some examples of what I'm talking about:

Any suggestions for how I can reduce the amount of spam still getting through would be appreciated.  Thanks!

25472[/snapback]

You don't say how many of the blocklists offered in 'options' you have 'ticked': mind you, if the IP is not on ANY of them they will still 'sail through'. I have all ticked, Assassin set to 4 and get only 2 or 3 false negatives a week and only 1 or so false positive a month. I fear SA at level 2 may create a lot of false positives (unless no-one you know sends you HTML mail!)

PS comfort yourself with th thought that your reporting will help put those unlisted IPs on the blocklist pdq.

[shull2805@spamcop.net]

I had all the Block Lists checked. As mentioned, SA threshold was 5, have changed to 2 and will try that for a while. I would be happy as a pig in sunshine if I only got 2 or 3 false negatives a week; with all lists checked and SA threshold of 5, I got 192 false negatives yesterday.

I'm not so sure how quickly an IP gets added to the lists. In the past two weeks, I'll bet I have received over 1,000 spams identical to the six I reported in the first message of this thread. Maybe 400 or so got to my Inbox, so their IP address wasn't on any of the Block Lists. Some of the 600 were blocked because they originated in Korea, Brazil, etc., and not because they were reported to SpamCop. Of the 400 that made it to my Inbox, I doubt very seriously that each of them was sent from a unique server, so at least some of them had to be sent from servers I had already reported.

I suspect there's probably a minimum number of spams that has to be reported before anything is done. (To be technically correct, there has to be a minimum, even if it's 1. However, I'm willing to be it takes more than one spam to get put on one of the lists.)

[StevenUnderwood]

As I understand it, tt also takes more than one reporter, so if this person is targeting you specifically, your reports alone will not get it listed either. Basically, an IP makes the list for passing a specific threshold percentage of spam/total messages.

[Wazoo]

spamcop.net,Mar 15 2005, 02:07 PM]I suspect there's probably a minimum number of spams that has to be reported before anything is done.  (To be technically correct, there has to be a minimum, even if it's 1.  However, I'm willing to be it takes more than one spam to get put on one of the lists.)

Suspect no more ... Follow the FAQ and end up reading the data provided at http://www.spamcop.net/fom-serve/cache/297.html

[shull2805@spamcop.net]

Suspect no more ... Follow the FAQ and end up reading the data provided at http://www.spamcop.net/fom-serve/cache/297.html

25535[/snapback]

OK, I understand what's going on, but I disagree with the decision to ignore Spamvertised URLs. I just opened my client email software, and downloaded 66 almost identical spams. All were from "Online Pharmacy-Wholesale"<some fake email address>, all had a few lines of nonsense and a link to rxchoices4sure.net. I have been reporting the heck out of these folks for weeks, but their spam still passes through SpamCop.

Here are some links to spam I just reported:

http://www.spamcop.net/sc?id=z746873325z78...b8170839c3879az

http://www.spamcop.net/sc?id=z746873326z77...af7422ffdfbd2dz

http://www.spamcop.net/sc?id=z746873327zfc...f6581f533fbe85z

http://www.spamcop.net/sc?id=z746873328z8f...7796048e43fb22z

http://www.spamcop.net/sc?id=z746873329zea...82978f8ac6e017z

Is there anything I can do to prevent this stuff from getting to my Inbox? Thanks!

[StevenUnderwood]

spamcop.net,Mar 28 2005, 03:14 PM]OK, I understand what's going on, but I disagree with the decision to ignore Spamvertised URLs.

26043[/snapback]

I don't quite understand your complaint here...your reports show that the URLs are not be ingnored. Reports are being sent to the admin of that site.

Re: http://rxchoices4sure.com: (Administrator of network hosting website referenced in spam)

   Reportid: 1391070376 To: root#dns.fz.fj.cn[at]devnull.spamcop.net

   Reportid: 1391070382 To: abuse[at]fjdcb.fz.fj.cn

spamcop.net,Mar 28 2005, 03:14 PM]I just opened my client email software, and downloaded 66 almost identical spams.  All were from "Online Pharmacy-Wholesale"<some fake email address>, all had a few lines of nonsense and a link to rxchoices4sure.net.  I have been reporting the heck out of these folks for weeks, but their spam still passes through SpamCop.

Is there anything I can do to prevent this stuff from getting to my Inbox?  Thanks!

26043[/snapback]

Keep reporting. I have received only 3 in the last 3 days, but 2 of them were held by the spamcop bl. Other than that there is not much you can do. They have so little text that spamcop's spamassassin sets it at 1.7. Much to low to use for filtering.

X-spam-Level: *

X-spam-Status: hits=1.7 tests=EXTRA_MPART_TYPE,HTML_90_100,HTML_IMAGE_ONLY_16,

HTML_MESSAGE version=3.0.0

You could setup a filter on whatever client you use to delete/move these to an alternate folder. The spamcop filters should work if you log into webmail at all. Mine all had the text "Online Pharmacy!" in the body so that filter worked for me. It seems the filters do not search on the text portion (or at least my quick test did not work) of the sender address.

[shull2805@spamcop.net]

I don't quite understand your complaint here...your reports show that the URLs are not be ingnored.  Reports are being sent to the admin of that site.

26045[/snapback]

My reports were sent to the web site admin because I manually submitted the spam for full reporting...after it had already passed thru SpamCop and ended up in my Inbox.

My complaint has two parts:

1) Why does this stuff keep passing through SpamCop and ending up in my Inbox, and,

2) Even if I go to the trouble to do a full report, the SpamCop FAQ says it will not use Spamvertised websites as a determining factor as to whether a particular email is spam.

So it appears that I am wasting my time reporting the spam that makes it to my Inbox.

[DavidT]

spamcop.net,Mar 15 2005, 11:03 AM]I've dropped my SpamAssassin threshold to 2 per Jeff's suggestion.

Oh my....you will surely see an uptick in false positives. For that many fase negatives to be getting through, your email address(es) must be seriously compromised on the web. For example, at your "steve-hull.com" website, I see that you have an unprotected "mailto" link at the bottom of the page. That leads to spam, due to harvesting by spammers. Let's hope that you're not also using a "catch-all" function to receive anything addressed to "randomaddress <at> steve-hull.com" -- that's a luxury that most of us have long since given up due to spam sent to random and typical role addresses.

For mailto protection, try this java scri_pt:

<scri_pt LANGUAGE="java scri_pt" TYPE="text/java scri_pt">

<!--

name="username"

at="[at]"

domain="domain.ext"

document.write("<A HREF='mailto:"+name+""+at+""+domain+"'>"+name+""+at+""+domain+"</A>")

// -->

</scri_pt>

replacing "username" and "domain.ext" with actual values.

DT

[shull2805@spamcop.net]

Oh my....you will surely see an uptick in false positives. For that many fase negatives to be getting through, your email address(es) must be seriously compromised on the web. For example, at your "steve-hull.com" website, I see that you have an unprotected "mailto" link at the bottom of the page. That leads to spam, due to harvesting by spammers. Let's hope that you're not also using a "catch-all" function to receive anything addressed to "randomaddress[at]steve-hull.com" -- that's a luxury that most of us have long since given up due to spam sent to random and typical role addresses.

For mailto protection, try this java scri_pt:

<scri_pt LANGUAGE="java scri_pt" TYPE="text/java scri_pt">

<!--

name="username"

at="[at]"

domain="domain.ext"

document.write("<A HREF='mailto:"+name+""+at+""+domain+"'>"+name+""+at+""+domain+"</A>")

// -->

</scri_pt>

replacing "username" and "domain.ext" with actual values.

DT

26051[/snapback]

Thanks for your suggestion regarding protecting my email address. While that will help prevent future address harvesting, I'm afraid after having that link on my website for over 10 years, the damage has already been done. And, yes, I am using a catch-all account, which does make things much worse. But despite whatever things I might have done to get myself into this predicament, SpamCop should be doing a better job of filtering out the spam.

I'm really not experiencing any problems as a result of lowering my spam Assassin threshold to 2. I had already whitelisted quite a few email addresses before I lowered the threshold. In fact, I'm thinking of lowering the threshold to 1, making the necessary additions to the whitelist, and using the whitelist to set up forwarding rules with my web host/email provider.

Thanks again for your code snippet.

[StevenUnderwood]

spamcop.net,Mar 28 2005, 10:31 PM]I'm really not experiencing any problems as a result of lowering my spam Assassin threshold to 2.  I had already whitelisted quite a few email addresses before I lowered the threshold.  In fact, I'm thinking of lowering the threshold to 1, making the necessary additions to the whitelist, and using the whitelist to set up forwarding rules with my web host/email provider.

26052[/snapback]

If you are going to go that far, why not configure the block all and only use your whitelist?

[shull2805@spamcop.net]

If you are going to go that far, why not configure the block all and only use your whitelist?

26053[/snapback]

Not all of my legitimate correspondents have been added to the whitelist. I still get a lot of legitimate email from people who have never emailed me before. I can lose business if people get turned off by a challenge-response type system. I'm OK with forwarding email from new correspondents through SpamCop, but what's the point if SpamCop is just going to forward spam back to my Inbox?

If SpamCop considered URLs that had been reported for spamvertising, this would be a moot point.

[dra007]

I have to agree with shull, but I also see how that would require an entire new approach/program. It is also not clear to me how many people go thrrough the trouble of full reporting. In addition, as pointed out by many here, a lot of those sites have dns and possibly other issues which make their identification unreliable.

[DavidT]

IMO, life's too short to spend countless hours whitelisting everyone with whom I correspond, but YMMV. That's why I keep my SA set at "5" and yet despite my addresses having been exposed here and there on the 'Net for a long time, I'm only seeing a few false negatives a day.

I hope you're not going to batch report all the extra stuff that winds up in your Held Mail, because I'm still thinking you'll wind up with false positives from people whom you've not yet whitelisted.

DT

[shull2805@spamcop.net]

IMO, life's too short to spend countless hours whitelisting everyone with whom I correspond, but YMMV. That's why I keep my SA set at "5" and yet despite my addresses having been exposed here and there on the 'Net for a long time, I'm only seeing a few false negatives a day.

I hope you're not going to batch report all the extra stuff that winds up in your Held Mail, because I'm still thinking you'll wind up with false positives from people whom you've not yet whitelisted.

DT

26077[/snapback]

Just went thru 914 held emails and batch reported the lot of 'em. Not one was legit.

I don't have to "spend countless hours whitelisting everyone with whom I correspond". As mentioned previously, I'm running spam Assassin at level 2, and I'll bet I have fewer than a dozen email addresses in my whitelist. Yet, there are hundreds of addresses that make it through just fine, primarily because they don't send emails that look enough like spam to upset spam Assassin. Heck, yesterday, the spammers even managed to get 66 identical emails through without SA complaining.

[DavidT]

spamcop.net,Mar 29 2005, 01:57 PM]Heck, yesterday, the spammers even managed to get 66 identical emails through without SA complaining.

That sounds like fallout from using a catchall address. If you seach these forums a bit, I think you'll find multiple accounts from those of us who have gone through the process of de-activation of catchall functions. However, if you're dead set on keeping the catchall, you'll certainly have to take more drastic steps, such as the extremely low SA threshhold.

DT

[shull2805@spamcop.net]

That sounds like fallout from using a catchall address. If you seach these forums a bit, I think you'll find multiple accounts from those of us who have gone through the process of de-activation of catchall functions. However, if you're dead set on keeping the catchall, you'll certainly have to take more drastic steps, such as the extremely low SA threshhold.

DT

26092[/snapback]

Wait a minute! You seem to imply that it's OK for SpamCop to pass spam on to my Inbox because I have a catch-all account. You have indicated that you are willing to tolerate a low number of false negatives. I would be as happy as you are if I had the same number of false negatives that you do. On a percentage basis, we probably still get approximately the same number of false negatives. But I get orders of magnitude more spam than you do, so the absolute number of false negatives hitting my Inbox is much higher than what you get. I acknowledge the fact that using a catch-all is the main reason that I get more spam than you. That's not the point.

My point is that regardless of whether I use a catch-all, or even if I openly post my email address(es) everywhere for spammers to harvest them, SpamCop is not doing a thorough job of filtering out the spam. By only looking at the headers, it's ignoring a vital weapon in the war against spam.

I willingly acknowledge that SpamCop does a very good job of catching incoming spam; it's taken header-only analysis to the state-of-the-art level. Maybe a 95% catch rate is as good as it can get for programs that only analyze email headers. But, I'm getting close to 3000 emails a day. How happy would YOU be if you had 150 spams in your Inbox each day?

The spam reaching my Inbox has innocuous headers (or they wouldn't get past SpamCop). This spam has a little bit of random text, one or two URLs, and maybe an embedded .gif file. Unless SpamCop starts looking at these URLs, we might as well pack up and head home- the spammers are gonna win this one.

I already have Baysian software running on my email client that does a better job than SpamCop at figuring out whether incoming mail is spam or not. When I get the crap that passes through SpamCop, my email client redirects it to my junk mail folder. (In case you're asking, "If he's already got software that he says does a better job than SpamCop, why is he using SpamCop?", the answer is simple. I don't want to download 3000 spam emails. I don't want to download 150, either. )

Am I being unreasonable in asking why SpamCop can't do a better job of figuring out what is spam and what isn't?

[StevenUnderwood]

Am I being unreasonable in asking why SpamCop can't do a better job of figuring out what is spam and what isn't?

Perhaps. I get anywhere between 1-3% false negative (a couple a day, about 30 real emails, and a couple hundred spam every day). If you are receiving 3000 messages every day, your numbers are about right. There is no perfect system. You could help greatly by limiting the numbers of messages being sent to the account for filtering (i.e. turning off the catchall and only configuring the addresses you have actually used.

[DavidT]

spamcop.net,Mar 29 2005, 08:25 PM]You seem to imply that it's OK for SpamCop to pass spam on to my Inbox because I have a catch-all account.

I didn't mean to. Ideally, SpamCop could be improved to the point where your needs would also be met. However, I do think that having a catch-all these days is asking for trouble.

By only looking at the headers, it's ignoring a vital weapon in the war against spam.

Wait a moment...IIUC, SpamAssassin *is* looking at the body of the messages. For example, here's a sample result from a message that got through to my mailbox:

X-spam-Status: hits=3.3 tests=EXCUSE_3,FORGED_RCVD_HELO,HTML_60_70,

HTML_IMAGE_ONLY_24,HTML_MESSAGE,HTML_TITLE_EMPTY,HTML_WEB_BUGS,

MIME_BOUND_NEXTPART,URIBL_OB_SURBL version=3.0.0

Many of those codes have to do with the body, not the headers. BTW, this is a message I wanted to receive, but I don't want to bother whitelisting this sender, so my setting of 5 is appropriate for me.

Sometime last year, we had a long discussion regarding the implementation of more Bayesian methods in SpamCop's SA implementation, but now I don't remember what ever happened. I think that JT was reluctant, but when he did an upgrade to SA 3.0, I think that some sort of Bayes stuff started being applied by default...but I might be wrong.

DT

[shull2805@spamcop.net]

Here is a follow-up in case anyone is interested. 2 days ago, I changed my spam Assassin threshold from 2 to 1. Since then, I have had no spam forwarded from SpamCop to my Inbox. I gave away a new email address to an online vendor and they sent me a confirmation which was not held by SpamCop (I didn't want it to be held); it ended up in my Inbox. 1 email (a Microsoft newsletter) was held by SpamCop; I whitelisted and released it.

I'm very happy with the current status quo. I can continue to use my catch-all account, I don't have to waste time reporting spam, whitelisting false positives isn't a big deal, and everybody who spams me gets reported. Best of all, when I go on the road and retrieve my email via my cell phone, I'm not paying by the minute to download spam.

FWIW, I have 15 addresses on my whitelist. I see that a couple of them can be eliminated if I use wildcards in the address field.