I'm seeing a lot of spam which comes in the form of PDF attachments to otherwise empty or nonsensical messages. I have done some tweaking of my filters, but it's not working nearly well enough to catch even the majority of the PDF spam.

Is there any way that you guys (i.e. JT) could add an option to filter on the type of attachment, or just add a filter option to hold messages that have any attachment? The vast majority of email that I get has no need for attachments, and I don't mind going hunting on the rare occasion that I need an attachment. I also have alternate email addresses which do not go through SC that I can give out. So, please... add an option to automatically hold messages with PDF attachments or ANY attachment if that's too much trouble.

Another approach to the query .. ask the Horde/IMP developers .... contact/FAQ data available in the SpamCop FAQ 'here'

Historically, lots of issues. Recall the days when ISPs took to rejecting anything with a number of file extensions, some still do .. then users took to complaining .... some ISPs still don't allow some file types at all .... some accept the e-mail but strip the attachmanets, some simply drop the e-mail, some really 'help' by killing the e-mail itself, but then generating a note to let you know that you once had an e-mail from somebody, but it's not available anymore ... on and on ... Typically, this wasn't the actual e-mail (server) application, but some filtering after the e-mail had been accepted, and this is where the variances came into play.

Remember, e-mail was originally designed around text only. This frigging HTML and MIME-type extension crap is what started all this mess ....

Oh, yeah... And I'm having trouble now because I created a filter to filter out messages with the word "postcard" in the subject. Now that is causing me grief...filtering legit messages. *sigh* Sometimes you just can't win.

It would be really nice if there would be some sort of option in the filters to catch messages with a specific mime-type attachment. I tried to create a filter like that using the "user-defined headers" and it really doesn't seem to work.

Yep, that little problem has been a round since day one .. those great-big-companies that provided software filters to protect the kids in school .. who then couldn't finish their homework because so many web-sites were locked-out ....

The catch is that a .PDF file isn't actually defined in the headers .... it's just one of the many possible attatchment/file types available under that all-inclusive line Content-Type: multipart/alternative; ... the file itself is "MIME Encapsulated' within the body .... and to all (normal?) e-mail server software, there is no .PDF file at all, it's just seen as (really wierd) text within the body contruct of the e-mail itself.

I wonder... Somewhere in the BODY of the message, should be the boundary for that MIME part, which DOES contain some information about the file type I believe. Can you create a body filter in the spamcop webmail? (I don't use it myself, so don't know what options are available). If so, does anyone know if said filters on the body text are compared to the original body source, or just rendered "readable" version of the body that actually gets shown?

Guess I'll have to do a little more research on MIME encoding and just what actually exists in the unencoded body of the message when there are file attached....

Ok, some followup... Here is what I see in files with a pdf attachment:



So, if you can do regular expressions in your filters, a filter for 'filename="*.pdf"' SHOULD catch any emails with a pdf attachment. Unfortunately, I'm willing to bet this is beyond the capability of the filters available in most mail applications. Also, it would require that the mail application do its search on the ENTIRE MIME encoded body, rather than just the text or html parts.

I took your advice and posted an "enhancement" bug on the bugs.horde.org website. Ticket #5643. Chuck Hagenbach had a question to which I don't know the answer. Can you please advise the answer to this question and then I can post the response, or maybe you (or someone else affiliated with SC webmail service) can follow up??? Here's Chuck's question:

Anyway, here's the URL for the Ticket

We use the "Ingo" filter plugin for Horde (http://www.horde.org/ingo/), with a standard IMAP backend. You may need to take this request to the Ingo developers instead of the Horde/IMP guys.

We looked into adding PDF spam detection earlier in the mail delivery chain, but the only known methods to detect it reliably are extremely CPU intensive, and would slow all mail delivery to a crawl. There is a lot of talk about this in the spam blocking channels, and hopefully a good solution will be available soon.

-Trevor

Thanks. I have passed the info along to the Horde people. If they say it's something to take to the Ingo developers, I'll pass it along to them as well.

Just got a response:

Any chance of that happening?

The SpamAssassin rule is what I said would be too processor intensive. We can block *all* incoming attached PDFs at SpamAssassin without any trouble, but it would be unfair to people who actually use PDFs often. There is a method to scan them for spam, but it involves converting the PDF, extracting the image from it, running OCR software on the image, then passing the text from the image into SpamAssassin... but that's just too much to do on all the traffic we receive.

It's very likely that a more advanced SpamAsssassin rule will come along soon that can filter a good chunk of the PDF spam out without having to interpret the embedded images, and when it does we'll certainly get it installed.

-Trevor

Maybe I'm missing something, but I never said anything about the content of the PDF, I merely wanted to filter any emails with PDF attachments to the spam folder. Is that extensively resource intensive? Maybe I'm not a typical SC user, but I rarely get any email with attachments here, and if a few legitimate emails get filtered, it wouldn't be the end of the world as I can always go retrieve them from the held mail folder.

I would guess the majority of users actually do receive attachments they want without having to go through their Held Mail. Out of 15 current messages in my inbox, I have 4 that have attachments I have been waiting for. I also rarely see the PDF spam in my inbox, maybe 2 per week.

I don't know how hard it would be to implement attachment blocking on a per user basis.

Could SpamAssassin be configured to add an X-Header when a PDF is present? And then a filtering option added in the spamcop web interface that would simply drop anything containing that particular X-Header into Held Mail if the user so wanted? Would require a bit of coding I think, but it might be handy for other attachments types as well where some users want to filter them, and some don't.

That sounds like an excellent idea... makes it easier to filter for those that want to block this type of spam.

I'm not sure this would be ideal - except for those that don't mind either opening their webmail interface to trigger the filter or don't mind downloading the spam and having a filter in their mail program.

Perhaps a better SA option would be to have this as an optional setting in SA and add an SA score of, say, 4 or 5 for those that wanted it.

That said, personally I have no interest in blocking pdf attachments. Most that I receive are legitimate and the majority of those which are spam get caught because they come from sources already listed in the SCBL or one of the other rbls. Like Steven I get one or two of these per week through to my mailbox.

Andrew

Be glad. I have about 5 or 6 email addresses that find their way back to my Spam Cop account. I don't really know how many I get any more, because I can identify them pretty much off the bat, but I know I was getting a handful of them per day! I, for one, typically leave a browser window open 24/7 to SpamCop at home, and most of the time when I'm at work, I have a window open there as well. Unfortunately SC will log me off automagically after a certain amount of "idle time" so I have to log back in, but that's OK. As soon as I log in, the filters kick in and some of the spam gets moved. Unfortunately not all of it.

I already have my spam settings set to "1" and I'm still getting a ton of spam through to my inbox. I'd like it if we could make SA more aggressive in it's tagging of things, because I'm getting a lot of stuff that's immediately identifiable as spam, but it's not being caught by the filters.

Are you only using SA? I have my SA settings to 5 but also select all of the available BL's and, as stated, rarely see a PDF spam in the Inbox, without any client side filtering. Like you, I have about 6 accounts all forward into this account and I check my account at least every hour, though not non-stop.

Nope. I have every option for filtering enabled, and I'm still getting tons of spam. Now, granted, the PDF spam may be gone, for now, I don't know... haven't actually read any spam lately, but I'm still getting tons of spam through the filters.

Could you give some numbers please ?

My own (SA=3.0) for July are

4305 spams, 110 leakers (=2.6 %), 0 false positive(s),

The only 'cute trick' I use is to add bankofamerica.com, ru, tw, and such to my personal blacklist.
See the thread "how to use Spamcop Mail'.

Frankly, I haven't been keeping count. I'll see what I can do tomorrow to count how many I get. However, I do know that just in a couple hours, I got 70 more in my "held" folder, not to mention another 5 that got through the filters, including bounces and auto-replies to non-existent email addresses on one of my three domains.

Sounds like you are using a catch-all address on your domain. If so, your numbers are going to be MUCH higher than typical, as you're going to pick up every piece of junk from every dictionary attack that goes on.

Since last night, I've had 32 messages get through the filters. Granted about 2/3 of those were unsolicited bounce messages for emails *I* didn't send. In my held email folder I have almost 1200 messages overnight. In my inbox, the non-spam messages are 288.

Yeah... I started out with my domains back before there was a huge problem with spam. Thus, I was using the "catchall" address to handle "targeted" email addresses. Doesn't work very well now.

Well that does explain the volume of junk you're receiving.

My brother's small company used a catchall mailbox since it allowed them to add users on their local system without having to configure the mail server. They still receive between 2,000 and 3,000 messages a day which are junk but they are now dropped at the SMTP stage by our mail server and only messages explicitly addressed to one of five members of staff are allowed into the 'system'.

It might be worth your while to make the switch.

Andrew

1520 total messages
288 non-spam messages 18.9%
1200 spam messages 78.9%
32 false negative 2.1%

That sounds typical for any spam filtering system. Cut down on the total messages and your numbers of false negative will drop as well.

For the 14 domains I have access to, for the month of August - no stats available on false negative, but our users complain about everyting and we receive relatively few complaints about spam:
995,882 total messages
72,497 passed messages 7.3%
663,952 blocked messages 67.0%
255,087 quarantined messages 25.7%

Seems to be about the same percent leaks that I am getting.

I suggest you take advantage of what spammers do.

This if the To: is something[at]posterswondersite.com then lots of spammers would forge the From: as the same or similar.
Assuming you don't send to yourself, setting personal blacklist posterswondersite.com might chop 10 % of the leakers.

It would be possible to accumulate a list of all the users names legitimately in use @posterswondersite.com
Your provider might then be able to flag or drop the others

Or In your mail client send the others and all those where posterswondersite doesn't occur in To: or CC: to a junk folder.

HTH