Bayesian Filtering?

[Jeff G.]

Does the SpamAssassin on the SpamCop Email System use Bayesian Filtering?

If not, can it please?

If so, how can we train it?

Thanks!

[jefft]

Does the SpamAssassin on the SpamCop Email System use Bayesian Filtering?

If not, can it please?

If so, how can we train it?

Thanks!

It does not at the present time.

JT

[ob1db]

Many thanks to Jeff for taking my request from the newsgroup and getting it rolling here.

JT, I see that Bayesian filtering is set to off and training is not enabled. As I understand it, we are wasting most of the horsepower of SpamAssassin by doing so. The whole reason I asked that this topic be brought up is as follows:

Lately, I noticed that the amount of spam getting through SpamAssassin is rising DAILY. With it still set to "5", I get 10-30 a day for Viagra, Penis Enlargers, Canadian Pharmacy, Breast enlargers (and my breasts are QUITE large enough, thank you, LOL!) etc. All OBVIOUS spam, but SpamAssassin gives them ratings like 2.6 or lower. (And still manages to false positive some!) I sent a bunch onto the .spam newsgroup, anyone need me to repost them in the forum area? And if so WHERE, TEST ?

Just so my setup is clear, I now have nearly ALL the bl lists checked. This has not helped much but HAS increased the false positives greatly.

I was just chatting with a local sysop who commented "I have used SpamAssassin for a while now, and very little gets through. Are you not training it?"

This stunned me! When I first enabled SpamAssassin, the spam really dropped dramatically. Now it is getting nearly intolerable. I was wondering "Is it possible the system is set up with no training enabled ?" Now I see that is the case.

My sysop friend is running one version OLDER SpamAssassin than SpamCop is, so that is not the issue. It would appear that something that needs to be changed in how SpamAssassin is set up. We don't have individual access to its settings besides level, correct ?

My Sysop buddy says says: "You collect mails that it missed that you know to be spam, and run a program over them "sa-learn". You also run it over mails it got right, utilizing the --ham switch. This basically adds data to the Bayesian classifier, I think.Those using it on a network might want to have a "this is spam, damnit" drop so that users can report it. It's smart enough to not register a mail twice."

I think this argues for getting the Bayesian filtering on AND setting up traing ASAP. Any reason NOT to at least turn the filtering on ?

How to implement training ? I think we need a column for SpamAssassin score on the HeldMail page (webmail) and a checkbox for missed scoring spams. Maybe the same on the inbox pages as well ?And maybe a special maildrop for TRULY egregious failures, e.g. "scored a 2.6 for Viagra with breast enlargement, all shouting and HTML, etc." Likewise a drop for ones that WERE flagged that were clean ?

Waddya say ?

We gotta get the levels down.

Drowning again. This was why I signed up for mail filtering. Now it is failing AND requiring as much time as I was spending on weeding through spam manually.

David

PS: Sorry if I confused thing by starting a new thread.

[PeterJ]

[mainly in response to ob1db]

Being no expert on SpamAssassin by any means, hopefully what I have to offer here is correct. I think you are misunderstanding the capabilities of SpamAssassin somewhat. Looking at their website one can see that SpamAssassin can utilize many different methods of detection in a variety of combinations. It is up to a given system administrator to decide what combinations of these methods they would like to implement. SpamAssassin's bayesian capabilities are only one of the variety of methods that can be implemented.

That being said, I still think it is worthwhile to consider adding bayesian methods to SpamCop's current implementation of SpamAssassin. It may not be as simple as just turning "something on", because one has to consider the storage and maintenance of the corpus that will be used. Also, will the corpus be shared among all SpamCop users or would each user have an individual corpus (the latter would be more complicated.)

SpamCop's current SpamAssassin implementation is effective for me, but that does not mean it cannot be improved. I think I keep my setting on "2" or "3", is there any reason yours is set to "5"? Have you received too many false positves when you set it lower?

[ob1db]

Mr. Newbie, I wonder if you read ALL my post!

Whatever SpamAssassin is set to: blatant spams are registering UNDER 3 with some consistentcy. That is just totally wrong. When it was introoduced, 5 was an adequate setting. That was not so very long ago (November ?)!

And I am solely quoting a VERY experienced Sysop when I state that SpamAssassin MUST have Bayesian filtering and training enabled to maintain effectiveness. In a nutshell, spammers appear to study SpamAssassin to see what is will pass. The Bayesian filtering with training ensures that the filters stay ahead of the spammers.

SpamAssassin, sadly, uses a VERY slow and cumbersome bug reporting system, (Bugzilla), so the developers appear to be in some blissful state of ignorance as to the current efficacy of the system. I would LOVE to have a direct reporting link to SpamAssassin so I could send THEM failed results directly.

In the meanwhilw, IMNSHO, we heed to turn on Bayesian filtering and hopefully implement training to get the spam filtering back to where it was a month ago.

In case you didn't get this first time: when SpamAssassin was first added to SC, my spams dropped to a few a day. At that time I only had 2 Blocklists enabled. Now I have ALL the on and I get 20-30 sneaking through a day, most of them SpamAssassin failures...

David

[ob1db]

Here, I just received this, makes my case pretty well:

Subject: FWD: Order V+a+lium , XA+n+ax \ V1[at]gRa < At|:v[at]n # Som+a+ . Pnt:e:rmin S2YQU

Date: Sat, 14 Feb 2004 11:42:34 -0200

MIME-Version: 1.0

Content-Type: multipart/mixed;

boundary="--9281174198244924"

X-Mailer: Mutt/1.4i00

X-IP: 206.170.156.6

X-Bogosity: Yes, tests=bogofilter, spamicity=0.990277, version=0.11.1.3

X-spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on blade6

X-spam-Level: *

X-spam-Status: hits=1.1 tests=BIZ_TLD,HTML_50_60,HTML_MESSAGE,MIME_HTML_ONLY

version=2.60

Scores 1.1 ??????

[Wazoo]

Mr. Newbie, I wonder if you read ALL my post!

You know, I think I've read it a number of times. As a matter of fact, I was surprised to see it here ... after the newsgroup stuffing, then your remarks that you'd "think about it a bit and re-postulate your thoughts here", I was amazed to see what appeared to me to be pretty much a copy of the newsgroup posting. The sarcasm of your "Mr. Newbie" definitly got my attention .. a bit hard to believe, coming from you ...

Whatever SpamAssassin is set to: blatant spams are registering UNDER 3 with some consistentcy. That is just totally wrong. When it was introoduced, 5 was an adequate setting. That was not so very long ago (November ?)!

Rumours are that spammers keep changing things .. something to do with getting around filters, blockages, and such .. not sure where I might have heard that one ...

And I am solely quoting a VERY experienced Sysop when I state that SpamAssassin MUST have Bayesian filtering and training enabled to maintain effectiveness.

Well, as we're all newbies on the bus in life here (and the newsgroups also, apparently) it's good that you have someone with experience to talk to. Guess it's too bad that there isn't any one here on par with your circle of experienced friends .. sorry ...

In a nutshell, spammers appear to study SpamAssassin to see what is will pass. The Bayesian filtering with training ensures that the filters stay ahead of the spammers.

Oh, you've heard that rumour also?

SpamAssassin, sadly, uses a VERY slow and cumbersome bug reporting system, (Bugzilla), so the developers appear to be in some blissful state of ignorance as to the current efficacy of the system. I would LOVE to have a direct reporting link to SpamAssassin so I could send THEM failed results directly.

Yep, just another group of no-experience idiots without a clue. Tis a shame there's so many of us out there.

In the meanwhilw, IMNSHO, we heed to turn on Bayesian filtering and hopefully implement training to get the spam filtering back to where it was a month ago.

the "NOT SO" has been demontrated rather well. But it's the "we" thing that I'd ask for clarification on at this point. I seem to recall that JT is the one guy that makes the call, handles the changes, lives with the complaints/results ....

In case you didn't get this first time: when SpamAssassin was first added to SC, my spams dropped to a few a day. At that time I only had 2 Blocklists enabled. Now I have ALL the on and I get 20-30 sneaking through a day, most of them SpamAssassin failures...

Nope, hard to miss when you get to re-read it a dozen times. On the other hand, I don't see that you made any note of reading Larry Lillgallen's response that he didn't have this issue with his SpamAssisin setting of "1" back over in the newsroups.

And so you don't get the wrong idea, the only reason I responded at all was your "Mr. Newbie" remark. ...sheesh ...

[Jeff G.]

Please refrain from ad-hominem attacks. Thanks!

[RogerB]

Okay, either way, Bayesian filtering needs to be activated and SA trained. Currently, I am using 1 as my SA setting and all the blacklists and I am now up to 5-10 OBVIOUS spams a day.

Example from this morning:

Message-ID: <thhfkjkwaq.03659732qtpblyhhnn[at]Trosefymgmgzmtx>

From: "Trose" <UtopLostatewide[at]adexec.com>

Date: Mon, 16 Feb 2004 09:17:29 -0200

To: tripper[at]atl.mindspring.com

Subject: ENLARGE Y0RU PE|NS Sanskritic acquire

MIME-Version: 1.0

Content-Transfer-Encoding: 8bit

Content-Type: text/html; charset=iso-8859-1

X-spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on blade1

X-spam-Level:

X-spam-Status: hits=0.8 tests=HTML_40_50,HTML_FONTCOLOR_UNKNOWN,HTML_FONT_BIG,

HTML_MESSAGE,MIME_HTML_ONLY version=2.63

X-SpamCop-Checked: 192.168.1.101 207.69.200.241 207.69.200.237 207.69.200.154 212.248.143.213

Kinda blatent, if you ask me.

[Firefly]

I have to admit that the amount of spam making its way past the blacklists and SpamAssassin (5) has grown significantly in the past couple of weeks. It used to be I could go 4-5 days without seeing a single one, and now it's not unusual to get 4-5 per day. The spammers are learning...

[PeterJ]

Assuming that SpamCop's implementation of SpamAssassin is not optimal, what I was trying to suggest earlier (probably did a poor job at this) is that there are other ways that SpamAssassin can be tweaked besides relying on "bayesian filtering" to be the holy grail.

The first thing that comes to mind is that perhaps updated or new rule sets can be added by JT to the SpamAssassin configuration. These may already be in use by SpamCop, but some popular rule sets (from perusing the SpamAssassin NG) are blackhair, popcorn, chickenpox, and weeds. Recently the blackhair and popcorn rule sets were combined into one, more info on these here: Jennifer's Sets

A quick look into SpamAssassin history shows that it has been around for about 3 years and has had bayesian capabilites since v2.50 about a year ago. With this in mind I am merely suggesting if SpamCop's SpamAssassin is "broken" then the best way to "fix" it is to improve all areas possible, not "just" by using bayesian based filtering.

Now, lets assume that JT has time and interest in setting up Bayesian filtering via SpamAssassin, then I see 3 main methods available after reading a bit about SpamAssassin at their Wiki and other places:

1) [easiest] Turn on bayesian filtering and enable autolearning. This would allow SA to autolearn ham and spam based on it's own scoring. The tolerances that are used as the defaults are <0.1=ham and >6.0=spam. As I understand the documentation this "setup" is on by default, so perhaps JT decided against this at some point in the past (JT?) These defaults oF 0.1 and 6 can be modifed by the administrator if desired, but given that it is "autolearning" this method is not as effective as "training."

2) [harder] Turn on bayesian filtering and disable autolearning, instead rely on training of a "shared corpus" by all SpamCop users. This has promise as being effective and also not excessively hard to implement. I liked the comments at end of this one page of the SpamAssassin Wiki (referenced: SiteWideBayesFeedback)

Another option, and one that's easier for most users to use, is to set up two public IMAP folders on your IMAP server, one for MissedSpam, one for NotSpam.

Then ask your users to move messages that SpamAssassin misses into the MissedSpam folder, and move messages that SpamAssassin marked incorrectly as spam into the NotSpam folder.

You can then run sa-learn from a cron job over those folders to update the Bayesian databases.

I like this idea, because I can picture where it could be mostly be automated from the administration side and users get most of the benefit of bayesian filtering even if they do not each have their own corpus and their own "ham" and "spam" tokens. Users would not have to spend too much time training, only on false-positives and false-negatives related to "SpamAssassin"

3) [hardest] Turn on bayesian filtering and disable autolearning, instead each user would have their own corpus and any training they conduct will affect only their corpus. This should be the most effective, but also the hardest to implement. Perhaps each user could have IMAP folders named "SA MissedSPam" and "SA NotSpam" Or name them without "SA" if one wanted them to be more universal. Seems complicated....well it is over my head.

Last thoughts: As a long time SpamCop Email user I expect my account to provide me with reporting and filtered mail by the SpamCop BL, but have for the most part considered JT's implementation of SpamAssassin as a bonus. I am not knocking it, I guess I am just saying thanks to JT for even bothering to set it up at all...he did not have to. Hopefully this thread can continue to bring more opinions on bayesian filtering and SpamAssassin from the SpamCop mail user base as we can only benefit.

JT - is the SpamAssassin setup using any of the custom lists I mentioned above? If not please consider at least the new "blackhair+popcorn" rule set. Thanks much.

[yourbuddy]

There's always POPFile (free) and K9 (free, and even better than POPFile)

that can be used - until Spamcop refines their version of SpamAssassin.

After all, it's spam we're trying to get rid of - right

[PeterJ]

I agree that PopFile and K9 are great programs, but briefly on the topic of K9 being better than PopFile, it is worth taking a look at what Robin Keir's own FAQ has to say on the matter: Robin Keir's - K9 - FAQ

To anyone who may be interested in using either of the two programs it is relevant that K9 is only designed for use on Windows.

[yourbuddy]

I agree that PopFile and K9 are great programs, but briefly on the topic of K9 being better than PopFile, it is worth taking a look at what Robin Keir's own FAQ has to say on the matter: Robin Keir's - K9 - FAQ <snip>

Thank you for supporting my point

[ob1db]

Mr. Wazoo, speaking of sarcasm, I think your unwarranted personal attack just took the prize away from me.

If you reread my post * your comments are clearly out of line. Aside from my first line, which could be contrued as FAINTLY sarcastic , it is sarcasm free (and I note that HE did not take offense). I was pointing out GENTLY to Mr. Newbie that he appeared to have missed half my points in his reponse.

I has no need to slam him, nor does the group need you slamming me. If you have a personal issue, please take it off-group. As an "advanced member" you should know better.

Despite your snide inferences, my intent is to get action on a problem that is obviously getting serious. Reading other replies it appears that I am not alone.

May I suggest that if you try some constructive criticism, ala Larry Kilgallen and Mike Easter, that you might see my contributions come more in line with what you would like to see?

I am all ears, believe it or not.

Whatever you read into my intentions, let me TRY to correct some of them:

My desire is to add to community and help others, not some form of self-aggrandizement. I am well enough known and regarded in my professional community, I don't need my ego fed here, thank you.

I likewise have no interest in taking credit for other's thinking. I quoted some which agrees with my own experience.

I don't pretend to be a know-it-all. I am more than a newbie but just so. The sysop I quoted was one I was trying to pass on some of my recent problems too. He offered some simple and direct advice that I quoted VERBATIM.

I have been trying to contact SpamAssassin to see if there is some helpful contribution we can make with our current problems. Obviously, I don't know more than they do or pretend to. Something IS wrong. I think we need to figure out if it is on OUR end or THEIR end.

I will not respond further in forum: you already know my email.

David

*(which IS substantially the same as newsgroup content, but reframed, I acknowledge. This is a sin ? I did what I said I would)

[jefft]

Mr. Newbie, I wonder if you read ALL my post!

Whatever SpamAssassin is set to: blatant spams are registering UNDER 3 with some consistentcy. That is just totally wrong. When it was introoduced, 5 was an adequate setting. That was not so very long ago (November ?)!

And I am solely quoting a VERY experienced Sysop when I state that SpamAssassin MUST have Bayesian filtering and training enabled to maintain effectiveness. In a nutshell, spammers appear to study SpamAssassin to see what is will pass. The Bayesian filtering with training ensures that the filters stay ahead of the spammers.

SpamAssassin, sadly, uses a VERY slow and cumbersome bug reporting system, (Bugzilla), so the developers appear to be in some blissful state of ignorance as to the current efficacy of the system. I would LOVE to have a direct reporting link to SpamAssassin so I could send THEM failed results directly.

In the meanwhilw, IMNSHO, we heed to turn on Bayesian filtering and hopefully implement training to get the spam filtering back to where it was a month ago.

In case you didn't get this first time: when SpamAssassin was first added to SC, my spams dropped to a few a day. At that time I only had 2 Blocklists enabled. Now I have ALL the on and I get 20-30 sneaking through a day, most of them SpamAssassin failures...

David

"Mr. Newbie" is right. SpamAssassin catches a TON of spam without Bayesian filtering. We may be able to do that at some point in the future. We can't now.

And, if you had looked at the SpamAssassin site more, you'd see that they're well aware of exactly how well their software works.

BTW, Bayesian filtering works on exact matches, while most of the SA rules are based on regular expressions. It's not clear to me that Bayes filtering would really be that much more accurate. The spammers are constantly changing their stupid little expressions for the bad words. And nothing else in the spam is enough to key it in as a spam.

As I said before, please post the percentage of spam that's getting through. That goes for everyone. I'm willing to hear what everyone's results are, but please count, don't just say "well, it's about such-and-such".

JT

[jefft]

Here, I just received this, makes my case pretty well:

Subject: FWD: Order V+a+lium , XA+n+ax \ V1[at]gRa < At|:v[at]n # Som+a+ . Pnt:e:rmin S2YQU

Date: Sat, 14 Feb 2004 11:42:34 -0200

MIME-Version: 1.0

Content-Type: multipart/mixed;

boundary="--9281174198244924"

X-Mailer: Mutt/1.4i00

X-IP: 206.170.156.6

X-Bogosity: Yes, tests=bogofilter, spamicity=0.990277, version=0.11.1.3

X-spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on blade6

X-spam-Level: *

X-spam-Status: hits=1.1 tests=BIZ_TLD,HTML_50_60,HTML_MESSAGE,MIME_HTML_ONLY

version=2.60

Scores 1.1 ??????

Yes, 1.1.

What makes you think it should be otherwise?

There are lots and lots of silly words that are recognizable to humans but can't be recognized by computers. V1[at]gRa? Sure, we could filter for that. You could train a Bayes filter on that. Tomorrow, they'll send V|[at]gRa and the day after tomorrow it will be \/îåg®ä. While that may be instantly recognizable to you, it's not to a computer. And that's just one character set. You can send HTML emails with different character sets specified and hexadecimal coding inside the email.

Pattern matching isn't the answer to spam. Bayes is just fancier pattern matching. The spammers who can install trojan horse proxies on thousands of innocent computers can come up with tricky patterns faster than your Bayes filter can keep up.

JT

[PeterJ]

JT --

Did you have any comments on custom rule sets?, specifically the ones I mentioned here:

The first thing that comes to mind is that perhaps updated or new rule sets can be added by JT to the SpamAssassin configuration. These may already be in use by SpamCop, but some popular rule sets (from perusing the SpamAssassin NG) are blackhair, popcorn, chickenpox, and weeds. Recently the blackhair and popcorn rule sets were combined into one, more info on these here: Jennifer's Sets

The reason I bring it back up is that if one wanted to stop a spam (using SpamAssassin) containing:

V+a+lium , XA+n+ax \ V1[at]gRa < At|:v[at]n # Som+a+ . Pnt:e:rmin

The "chickenpox" rule set is designed to do so. Here is the desription from the "Jennifer's Set" link above:

This set l.ooks f|or th.is kind of garb+age, but excludes a huge sample of legitimate punctuation/wierd character use.  This set may be a little more aggressive than the other sets in that you will get some hits on ham, but they are generally random and for the most part don't push the message into spam. You may want to lower these scores to a "testing mode" score (something non-threatening like 0.05) while you test them out in your environment. See how they react, then adjust them to your needs based on your spam threshold. Don't let me scare you away from this set, I'm only telling you that it is not as dead on as the others, but it *is* a controlled assault and we expect there to be some misfires here and there.  It does a fantastic job with this type of spam and with several types of eleete spelling.  I'm growing to love this little beast.  Adam and I are tweaking it hard in attempts to make it more reliable, and many members of the SA-Talk group are providing me with great feedback, language assistance, corpus testing, and ideas.

Can you elaborate on the current rule sets that are being used (I know I have seen "big evil" somewhere in a message I received) so that as users report what percentage of spam is slipping through we can get a clearer picture?

All --

Just to stick up for bayesian filtering a little, I do think it is effective. Many people are getting 99% filtering success by using it and some are getting 99.9%. As I stated in a previous post I do not necessarily expect to get SpamAssassin or "bayesian filtering" as a feature of the email service I pay for. It does work especially well at the client level, however it has no repercussion on the Spammers themselves.

Recently I enjoyed watching several hours worth of discussion on bayesian filtering and other spam fighting techniques at the MIT spam Conference website(videos here), if anyone cares to check it out, it is pretty interesting. There were several presenters at the conference who can back up the fact that "bayesian filtering" is not being greatly defeated by spammers currently. Spammers use of many extra words in spam (some call it "word salad") has had little effect to date on bayesian filters. John Graham-Cumming (primary author of POPFile) has an interesting presentation on bayesian filtering that can be found about 1 hour and 16 minutes into the "Afternoon 1" video at the above link.

Regarding SpamAssassin's efficiacy, many suggest (at least on the SpamAssassin general mailing list) that a combination of custom rule sets (many using Jennifer's Sets) and bayes is best.

NOTE: Jennifer's site appears to be down at the moment

[Wazoo]

just reading a few lines of your quote of Jenny's description ofissues, it looks like the complaints would turn from "too much spam getting through" to others complaining about the false positives .... not sure that poor JT would gain too much as far as that goes ... not arguing against your suggestions, just pointing out that maybe you picked the wrong one to plead your case <g>

[Jeff G.]

just reading a few lines of your quote of Jenny's description ofissues, it looks like the complaints would turn from "too much spam getting through" to others complaining about the false positives ....  not sure that poor JT would gain too much as far as that goes ...  not arguing against your suggestions, just pointing out that maybe you picked the wrong one to plead your case <g>

Of course, if it was optional for each Customer, that would reduce the possibility of complaints about it.

[cerij101]

As I said before, please post the percentage of spam that's getting through. That goes for everyone. I'm willing to hear what everyone's results are, but please count, don't just say "well, it's about such-and-such".

JT

I have a SpamAssassin level of 4. In the last 24 hours I received 55 spam e-mails, of which 2 slipped through to my regular inbox. That's 3.6%, which I guess is acceptable. I'd be interested to know if other people are worse off.

[PeterJ]

I think it could be used safely, just follow her suggestions:

<snip> You may want to lower these scores to a "testing mode" score (something non-threatening like 0.05) while you test them out in your environment. See how they react, then adjust them to your needs based on your spam threshold. <snip>

I'm not stuck on the "chickenpox" rule set by any means, but I wanted to bring up that one because clearly SpamCop's implementation of SpamAssassin missed a spam that contained this content as ob1db posted.

JT was suggesting that percentages of spam slipping through would be useful for people to post, but also useful (assuming JT is interested {and has time} in tweaking SpamAssassin on our behalf) would be what kinds of spam our slipping past SpamAssassin. Then educated decisions could be made on what rule sets may need to be added or modified to SpamAssassin.

Either way, it would be cool to hear what rule sets are currently in place for our mail setup.

[jefft]

JT --

Did you have any comments on custom rule sets?, specifically the ones I mentioned here:

The first thing that comes to mind is that perhaps updated or new rule sets can be added by JT to the SpamAssassin configuration. These may already be in use by SpamCop, but some popular rule sets (from perusing the SpamAssassin NG) are blackhair, popcorn, chickenpox, and weeds. Recently the blackhair and popcorn rule sets were combined into one, more info on these here: Jennifer's Sets

I probably don't have a problem with those. I'll look into that. That said, it's my intention that the rules we use are very safe. We use the standard rules, plus Big Evil. Big Evil is intended to be a zero-false-positive list. It's all URLs and if you show legit email with a URL in it, he'll remove the URL from Big Evil.

We are obviously a large and diverse community. So, if someone came up with a "custom" ruleset which assigned lots of points to certain "dirty" words, then our doctors or historians or people in different countries might start to get a lot of false positives. So, we need to balance that out and the official rules try to do that. It will take research on all of the unofficial rulesets like those you mention to determine if they're appropriate for the general population.

And, before you ask, at the present time we don't have the infrastructure for individuals to choose their own rulesets and that kind of thing.

JT

[PeterJ]

I probably don't have a problem with those. I'll look into that.

Excellent. Thanks for devoting some time towards this.

And, before you ask, at the present time we don't have the infrastructure for individuals to choose their own rulesets and that kind of thing.

I would not have even dreamed of it. Having read up a bit on SpamAssassin these days and not noticing it anywhere as a common implementation, I would not have brought it up.

[ob1db]

Given the number of people (including our own Ellen!!) reporting that the Bayesian filtering pushes their results up to the 99% range, why is it that we cannot try implementing it ? Is it really an issue that it will require too much CPU power ? You have as yet to say WHY.

I don't hear anyone saying that it pushes false positives up in a bad way either.

And yes, it would be good to add some rules based thinking but that will take a while to implement. Bayesian would appear to be just turning on a software switch, as would training.

Even if we accept your argument that 80% of spam is being blocked, that is WAY too much for most of us. In my case, I see up to 200 spams a day, so 20% would mean 30-40 getting through. I WILL try to get you a hard count. I really don't want to hear that 80 or even 90% is "good" blocking when I was STEADILY getting 97-98 % a month ago!

Thanks

David