We have just added a new spam-blocking feature called greylisting to our mailgates. When enabled, greylisting delays messages from unseen e-mail addresses for a short time (configured to 30 minutes right now). Messages from addresses that have been seen before are allowed through immediately.

"Good" mail relays, like your friendly neighborhood ISP, are set to automatically retry delayed messages periodically, so all of your good e-mail will still get through. Spammers, however, often use relays that don't automatically retry, so a lot of spam will simply never be delivered. This is all transparent to both you and the people e-mailing you, and the only side effect is a short delay the first time someone e-mails you. Our testing over the last few days has shown as much as 75% of spam to a specific e-mail address being rejected by the greylist before it ever hits our spam filters. Spam blocked by the greylist is not delivered at all, and will not show up in your Held Mail folder.

More information about greylisting is available at greylisting.org.

We have done some testing and the results have been very positive, but obviously results are not always the same when escalated from a few accounts to thousands. This feature should be considered "beta" for now.

To enable greylisting on your account:
1) Login to webmail
2) Click "Options" on the top menu bar
3) Click "Spamcop Tools"
4) Click "Manage your email forwarding, password, mail report, and greylist settings. "
5) Click the "Enable greylisting" checkbox, and press Submit

Please use this forum to discuss your results. We are interested in hearing about how well this feature works for you. If you encounter any problems, send an e-mail to support[at]spamcop.net.

***UPDATE***
We have added management pages so you can view the messages that are pending in your greylist, and the messages that have been permanently blocked in the past 72 hours. Click Options->Spamcop Tools->Manage Greylist - ... to view your greylist entries. From these pages you can manually unblock senders.

~Trevor

Wow, this sounds good.

But I pop down 99% of my mail from other accounts. Won't greylisting
create hassles with my other ISP's ? The only mail I get to my SpamCop
email address is spam from idiots ... or my Held Mail digest, etc. and mail
from the deputies after I screw up something ...

amenex

George Langford

Any mail that is POPed by our servers does not get greylisted. Enabling greylisting shouldn't affect it.

If you *forward* e-mail to your Spamcop account from another service, it *will* be greylisted, but it will also always be allowed through whether it is spam or not since your ISP is relaying it. If the majority of your mail is forwarded to your Spamcop account, enabling greylisting is probably more harmful than helpful.

-Trevor

To clarify this for me and all, does that mean you are using POP to move mail off other servers and into this one? That's what I understand it to mean. So you are reading Email on Webmail.

The other method would be to use POP to move Email off here and read it on another Email client like Outlook.

To clarify how greylisting affects you based on how you read your e-mail:

If you are using our system to POP mail from your ISP (i.e. move it from your ISP to your Spamcop account), greylisting will do nothing. If you are using POP to download e-mail from Spamcop (i.e. move it from your Spamcop account to your ISP or home machine), greylisting *will* work. Greylisting also works if you are using IMAP to read your e-mail without removing it from the servers, either by using Webmail or your own e-mail client.

Greylisting only benefits you by blocking spam originally sent to your Spamcop account. If the spam was sent to another e-mail account originally and redirected to your Spamcop account (either through ISP forwarding or POP), greylisting does nothing for you.

-Trevor

Is the "seen" list initially populated from our whitelist, or do we have to experience the delay once for every one of those, too?

Ok, If I enable greylisting, do I in effect not help the SPAMCOP system?
That is, if 'greylisted' mail is never seen, it never can be reported on....

I do 'pop' all of my mail. What I am using this for is bascially 'greylisting' people who send to my @spamcop.net address.

I use Sneakemail and that service has had greylisting for some time; I love it. However, they also have three views of greylisted mail: pendings, windows, and giveups. All views are useful, but the giveups in particular is VERY useful -- it is the only way to see if legitimate mail was bounced because the sender's servers are misconfigured.

I didn't see an equivalent being offered here, but perhaps I just missed it. How will SpamCop users know if legitimate mail has been rejected due to server that didn't follow the RFCs and make the window?

Thanks and regards,
--appyface

I'm interested in the answer to this as well.

I've had the same question about the Held Mail folder, but haven't had time to look for it in the FAQ or on the forum. I've often wondered if I need to report spam in the Held Mail folder, but so far I've decided that if it's held then it still might not be spam, so I report it anyway.

I also get most of my mail by POP'ing it over from my ISP, but spammers are starting to spam me at my spamcop email address. The greylisting will at least help with that.

This is definitely a question I would like SpamCop to address.

We need more controls over greylist settings. I would want a greylist to automatically use my whitelist to clear some e-mail addresses automatically, and I would want to be able to see and change the e-mail addresses that are cleared or uncleared that are on the greylist.

I'm concerned about this, too. There have been times when a user's email gets thrown into Held Mail simply because I chose to remove a full domain from my whitelist, or their email changed slightly (e.g. from my-friend[at]mail.friendlyISP.com to my-friend[at]friendlyISP.com, or even the rarer change of top-level domains), or they simply moved to another ISP.

There are two big advantages to the Spamcop system over others like Earthlink:

1. Nothing gets lost without my knowing it;
2. I get to do something about spammers, beyond simply ignoring them.

I fear greylisting removes much of that.

Nonetheless, thanks for offering it, and thanks for making it optional

I fear I'll be switching off the gray list for the time being. I tried it today, and it sadly failed my simplest of tests.

Eight hours ago, I sent an email from a whitelisted address to a spamcop filtered email account and it still hasn't arrived.

I love the idea of the gray filter but until it has a few safeguards to ensure legit mail hasn't been lost or delayed beyond a reasonable time, I'll wait to implement.

JRS

There are two major types of SpamCop users:
1) Those who want to see every mail coming in, decide if it is spam or not, and report the spam by hand
2) Those who just want little spam with no input on their side

SpamCop has traditionally focused on the first group, which is why spam is always allowed in and stored in your Held Mail folder. It is not reported until you actually click the "report as spam" button or forward it to a spam reporting address.

Recently, many of our users have been requesting a way to just *block* spam. Greylisting is the first of those methods that we are going to implement. The spam is not reported and doesn't help get spammers shut down or blacklisted... the mail just vanishes. Any system that makes the decision to delete spam without human interaction (i.e. what group 2 wants) *will* result in a tiny percentage of good mail being lost. That is why we have always focused on group 1. But demand is high enough now that we are offering this service to people who want to reduce their spam, and can survive losing a very small amount of good e-mail.

A page to monitor your own greylist entries was considered, and is still on the list of potential improvements. We decided to roll it out now to see how well it works, but a way to manage your own greylist and view statistics is planned.

Currently, the greylist does not consider your SpamCop whitelist. Even addresses on your whitelist will be delayed the first time they are received.

-Trevor

I was wondering if the filters are also going to be improved. Currently I need to log-in and manually apply the filters...it would be such a big improvement if the filters could be applied automatically as mail is coming in and/or being retrieved via POP3.

[at]trevor

Your description of how greylisting would work is exactly how Sneakemail does it. The difference is, IF I CHOOSE TO DO SO, I can look at the three views in Sneakemail and see what is happening with any mail at each stage of the greylisting process. That approach won't bother your people that don't care what happens to the mail.

And that approach doesn't mean I won't lose legitimate mail, it just means I will be able to KNOW that I lost it. That gives me an opportunity to contact the mail owner and let them know I didn't receive it, and make other arrangements. The sender probably can't contact me, when he/she receives the bounce, because all they have is an email address for me and it isn't going through. I have to be able to contact THEM.

In the case of Sneakemail which uses multiple receiving email addresses, I can simply turn off greylisting for that receiving email address until my legitimate sender's server issues are resolved. A good point raised by UltraJoe is to have the SpamCop greylisting ignore addresses already on the whitelist. This would serve pretty much the same function as being able to selectively turn off the greylisting processing.

The other bells and whistles mentioned such as editing email addresses, etc. would certainly be nice, but if I could just have the same views as Sneakemail and know that a greylisted mail failed (and why) and then whitelist the sender, that would be good enough control for me to use the feature.

Thanks and regards,
--appyface

We're getting, roughly, 45% of mails POP'ed down from other accounts, 45% forwarded from other accounts and 10% directly to the Spamcop account's address. Most of these 10% are messages between my partner and I (exclusively) when we're at different locations, as well as the occasional admin mail, but also and worryingly (where did the suckers get the address from?) some spam in the recent past. Usually things gets looked at in the webmail interface, spams reported, unnecessary mails deleted, etc. then real reading, answering and archiving take place in our mail application Entourage, after fetching what's left in the webmail.

If I understand the whole discussion well, we don't need to activate greylisting, especially as we're already losing mails (probably not due to Spamcop) because we correspond in all sorts of languages with people all over the world. Correct, Trevor?

But what we very much would appreciate (and I've asked Spamcop a couple of times already) is to make the whitelist (and the blacklist, while you're at it) more manageable. I've got 18 pages at this point, and I've been through hell a couple of times trying to clean the mess up after clicking (sheer distraction) on "Release and Whitelist" instead of "Report as Spam"

Michel

Can't imagine anyone reading email on a client like Outlook. For my part, I use Thunderbird. Which is a GOOD email client.

Not complaining - Just pulling your chain.

But I will definitely give greylisting a shot. Sounds like a useful and desired improvement.

You will experience the delay for those, too. The vast majority of users will not notice or even realize that their mail was delayed. However, if it is important to you that all of your email is received instantly after it is sent, greylisting may not be a great option for you.

JT

As time goes on, I think the majority of users don't ever report their spam, they just want it removed. This removes a lot of spam (and viruses) without relying on particular keywords or blacklists. If you want all your spam, though, you shouldn't enable it.

We're going to be working to add some additional information. Greylisting should "just work" though. It's really not intended for you to have to go in and fiddle with.

We are working on allowing addresses in your personal whitelist to pass without being delayed. That feature isn't available right now, though.

That isn't on our side. That's your mail server losing your email. I'd be very interested in looking in the logs to see what happened. Can you email the address that you were emailing from and to to me at support[at]

Thanks
JT

Most of the usefulness of greylisting comes from email sent directly to your spamcop.net account. If it is forwarded or we POP it for you, greylisting won't help much.

JT

FWIW, I forward all of my email accounts to my SpamCop mail and since I started graylisting, I've seen only 10-15% of the normal spam levels. This could be coincidence, but it's been consistent.

<i>Won't help at all</i> ITYM so switching it off for trusted relays and other forwarders may save trouble.

I have enabled greylisting and will report.

About 30-40 % of my spam is direct to spamcop mail (rest is POP and forward) so there should be some useful benefit.

It does seem to me that server IP addresses rather than or as well as "From:" should be placed on the good list
This would cut down volume of items to remember and save both forwarded and normal mail being delayed for mail from each new correspondent.

I have also come across a mailing list which used a different "From:" for every item (to keep the threading in order).

And I presume you're getting all of your good mail.

This is interesting. I'd like to see more data. If this holds up, there is a possible explanation. Email from new, unknown users forwarded by your ISP will all get greylisted and delayed. Your ISP will retry, of course, so all of this spam will eventually be delivered by us. However, during this time interval all of the blacklists that we use have had time to update. Delaying delivery of your spam by 30-60 minutes might make a real difference in how much the blacklists can catch.

If you actually aren't getting the spam at all, either to your inbox or the Held Mail, it might be that a lot of your spam was actually being sent directly to your SpamCop account. Greylisting will help remove a lot of that spam, even if 100% of your legitimate mail is forwarded to us by another ISP.

Well, no, see my other post about delaying delivery of spam. This is theoretical. I honestly don't know how much difference it makes. I do know that the SpamCop blacklist is very real-time and new spam sources are often detected within minutes.

JT

http://www.spamcop.net/sc?id=z1414845532zd...2875ec5dc920dbz
IP source 84.229.49.9
http://www.spamcop.net/sc?id=z1414845542zd...f8f63cbb327ff4z
IP source 218.7.192.70
Turned on GreyListing and leave spam filters active
There are still some getting to my held mail? None seem to be mail servers? Why are they getting through
I only use SpamCop email no forwarding or POP
Spam is greatly reduced however
Others
http://www.spamcop.net/sc?id=z1414845551z1...3a6a3abfabd784z
http://www.spamcop.net/sc?id=z1414845559zc...6b7eff90311223z
http://www.spamcop.net/sc?id=z1414845571za...16f3a6da5cb47ez
http://www.spamcop.net/sc?id=z1414845580zd...088ae7057b4b07z

Not sure yet if "innocent' but incompetent providers are getting bounced but it is worth a try

Interesting to note ... I turned on greylisting last night, and from that moment on I have received NO spam at all ... I've had nothing in my held mail all day.

I think it's interesting because I use spamcop to POP my mail from my ISP. Then pull it down into Outlook on my home computer.

From what I've read in this forum so far, this shouldn't be the case ... Whatever the case ... I'm doing a test by turning greylisting OFF on just one of my accounts.

I'll let you all know what happens after the next 24 hours!

Very interesting service!

mrcj

BTW ... I wouldn't be without Spamcop ... but I'm also part of that group who likes to report as much spam as I can, so it's entirely possible that greylisting "may not be for me" ... On the otherhand ... I'll never leave Spamcop! You guys ROCK!

I suspect the way for spammers to get through GreyListing is to simply send two (or more) spams
The first will be sent for "retry". The spammers second spam (from same IP) is then passed by SpamCop as a "response" to first spam and whitelisted

Yes but there is that set time interval before SpamCop graylisting will let any more with the same "From:" through.

I happened to do this as part of my initial check and both first and second emails, sent minutes apart, were delayed by 50 minutes indicating a '400' response followed by a retry for both.

There is also the point that if the second shot is sent from the same IP address then an hour later this address may be on a blocklist due to the earlier spams.

And if it's sent from a different IP address then an greylist enhancement to look at sending IP addresses (which might be a good idea anyway, see previous post) would nullify any benefit.

Spam I see being sent is multiple of same spams sent again again again etc (e.g Canadian pharm)

It is though then trapped by SpamCop emails spam filters.

The spam getting held is in greatly reduced numbers than before turning GeyListing on

Just wondering how this is getting past Greylisting in first place?

As GreyListing stops the reporting of that spam it "retries" without response I see these lists getting weakened (So my/the idea is to make SpamCops GreyListing 100% perfect and better than the rest)

If however GreyListing stops spammers without need for reporting this becomes a good thing

Although initially slow but less than a hour before I received test emails from colleagues?

Is there also a over time limit for GreyList reply?). Importantly no test emails sent have disappeared (no false positives)

I wonder if the GreyListing "whitelist" can be compared to the SCBL with entries removed periodically or even immediately. So far all IP's I checked and have made it through to my Spam folder were listed on the SCBL and not mail servers(the SCBL is reluctant to list mail servers)

Can SpamCop email customers add to the "SpamCop GreyList whitelist" in WebMail options (with email "from" field) Ideally my existing whitelist

I think I may have spoken too soon.. the spam rate went up to 50% of normal(which is still VERY good). For a day or so, it was really low and I did see the test messages I sent were delayed, but they WERE delivered, so that's a positive for me. I was a little hesitant to try the gray, because I've had trouble with getting mail from a few of my accounts (no fault of SC, though).

I understand SpamCop greylisting to be working like I'm used to (a la Sneakemail), which means if your test mails are being sent from a correctly-configured server they will not fail the greylisting, ever.

The 'false positives' come in to play when a legitimate sender unfortunately has a mail server which is misconfigured and doesn't follow the re-try/re-send per the RFC. I've seen this happen at large and small companies, private or otherwise, as well as email giants like Yahoo or MSN.

The legitimate sender is generally not in a position to know that his mail server is misconfigured. All he knows is that his mail to you bounces. If he has only an email address as a contact for you, then it's over... You don't know (without the views Jeff has indicated are coming) that he sent mail to you, and he has no way to tell you it never reached you.

--appyface

Following up to myself, I have now read the greylisting white paper
http://greylisting.org/articles/whitepaper.shtml
and this does use sending server IP addresses as well as From:

Can we have a reference to the details of the actual implementation ?

I also note that there was only one greylisting delay to my forwarded mail, perhaps because
the forwarding ISP inserts the same Return-Path: irrespective of the actual From:

Which is good but may provide a loophole.

Yes, we have an implementation very much like the reference implementation. There is a triplet of sender email address, connecting IP address, and recipient email address that we use to make decisions. We are currently using a 30 minute delay for newly discovered triplets.

Petzl asked why spam is still getting through and it is simply because the spammers are retrying. For spammers willing to retry, this method doesn't help at all. However, there is evidence that a large number of spammers do not retry.

JT

Thanks for the info

One feature, now disabled, was for spamcop VER reporting was to add to a "From" Blacklist. Can this be re-enabled for "SpamCop Greylist"? Idealy to make a fingerprint of both "From and IP address"?

If you're asking for a blacklist to be created automatically from your reported spam, this will never happen. Spammers rarely reuse the same email address, so there is really no point. And, how are you going to manage this data? Over months, we'd end up with millions of email addresses on this "blacklist". Your own blacklist would have thousands of entries (assuming you report spam a lot), pretty much none of which will ever email you again.

JT

For those of you who are interested in this sort of thing, we have some rough statistics captured since we announced the greylist feature:

Greylist entries allowed: 44966
Greylist entries waiting for a retry: 1877
Greylist entries rejected: 57362
Users with greylisting enabled: 270

Approximately 56% of incoming mail to those 270 account is being rejected as spam.

We have been working on an addition that lets the whitelisted e-mail addresses by without greylisting them. It is entering beta testing now (behind the scenes), and we will roll it out in a couple of days.

~Trevor

The same email address is reused for a few hours though (probably the same spam run) Where the spammer is sending again and again? I'm suggesting a blacklist be made and then reset/expire every two/three hours. Greylisting can maybe still send "try again"?

Just trying to get spam passing down to zero It may be in too hard basket but just suggesting. I do see you are improving things as we speak (trevorb's post). If whitelisting can be succesfully implemented the retry time can be increased/doubled meaning even less spam should get past

I tried the "greylist" & it did cut down the spam significantly. However the reason I use SpamCop is to report spam, not just delete/block the stuff (I can get that done for free by other providers). So as long as I'm paying for your service, I prefer to report all the *!@#!!* stuff.
Thanks.
bcstones

We added greylist management pages. See the first post in this thread for more information.

~Trevor

Moderator Edit: single paragraph content added to this post to remove the need to backtrack a page to locate this new data.

***UPDATE***
We have added management pages so you can view the messages that are pending in your greylist, and the messages that have been permanently blocked in the past 72 hours. Click Options->Spamcop Tools->Manage Greylist - ... to view your greylist entries. From these pages you can manually unblock senders.

I promised to report on my experience with graylisting

Summary, overall Spam received is 35 % down on the average for last month (August).

My first full day wih greylisting was 2007-08-31
The results for the following 11 days were 996 spams 90/day 14 leakers (=1.4 %)

This is a reduction on the 140/day average for August.
(4369 spams, 80 leakers (=1.8 %), 0 false positive(s) )

The improvement in leakage may be due to the new release of SpamAssassin 2007-08-28

I should explain that my SpamCop Mail account usage is about 40% direct to account mail (nearly all semi-dictionary spam). On the rest, half POP from a legacy account and a little forwarded from elsewhere, greylisting could have no effect.

I therefore hoped for a reduction of about 40% and got 35% because out of an expected 200 direct to spamcop spams, 26 still got through so used relays (or other servers configued to retry), proofing them against greylisting.

In other news 53 % of spam received during those 11 days had a source reportable to a Chinese ISP (since I use quick reporting I didn't realise this before - I analysed the text of the emails that quick reporting sends).

Is there anyway the SMTP "HELO" command can be enhanced to block spam or use with a greylist? All the spam I see getting through has
"Received: from unknown (HELO gwlrtjk) (201.38.214.16)"

I would like to know if one could also get an option to reject/hold email on a improper HELO response?
Or are there to many incompetent providers out there (A reason not to auto accept an email account from a ISP?)

Perhaps at first one should just tag such email as ???

Since mail servers typically retry every 15 minutes would it not be better to delay only say 10 -14 minutes instead of 30? In 10 minutes due to spam traps etc. a sender is likely been reported to many blacklists and possibly razor. This would be less delay to wait also.

I have sent several emails from my email address at work over last couple days. They have never gotten through yet. Its running on Exim with Clamav and spamassassin. I am not sure why but I suspect its not retrying every 15 minutes like its supposed to by looking at the log files. In exim.conf it looks to be setup to try every 15 minutes though. I suspect maybe a high load due to spamassissin is delaying processing the queue and the large amount of messages in the queue.

Perhaps the windows can be extended from 4 hours to 12 hours to fix things like this?

I just noticed that in 'rejected entries' that greylisting is blocking a few gmail addresses trying to email my account. Looking at the source IP they are indeed coming from gmail servers. This clearly is not good. I think the timeouts need expanded or something. Untill then I have turned greylisting off. I think its an excellent idea but the minimum of 30 minutes of wait and timeout of 4 hours is just too narrow. It should be 10 minutes to like 12 hours in my opinion. Perhaps even just 5 minutes. Even 5 minutes will give new spam sources time to hit a few spam traps and get listed so blacklist and/or spamassassin can catch them.

Greylisting works for me (and a major majority of users) that is not to say it will be everyone's cup of tea

As for Gmail I only see it as good for recieving email not sending. (Hotmail is a better choice as it is competently set-up)
That said I did a test from Gmail and recieved it in 30 minutes

The emails that end up on the Greylist are supposed to be from "unseen addresses". If I email myself from any of various email addresses I see that they all get stopped by Greylist and I have to go in and allow them. I would expect those to only be stopped by the regular SC filters and be in my Held Mail only.
Not that I email myself that often, but as I triple boot I sometimes do in order to receive an email in one of my other systems.
Now that I have allowed them once, will they always be treated that way?

I had hoped that with the new Email interface that we would at last be able to go to specific pages in Greylist/Whitelist/Blacklist - i.e. instead of just <Prev Next> we would have [1] [2] [3] [Last] like most other applications.

Here is a 'crude' workaround until the problem is fixed (or to phrase it another way, the enhancement is made).....

When you want to go to a specific page
APPEND ?page=2
to the URL while reviewing...
You can put whatever page number you would like...

(Incidently, this work great too, when review your whitelist/blacklist enteries for addresses)

Thanks for the tip ;-)

How is this greylist arranged? I see no particular logic to how each entry is filed. I just went in to check that nothing legit was caught in the rejected entries by accident and if the most recent items were filed on page 1 onwards then it would have been a lot easier to check.

My understanding is that Greylisting is configurable around a whitelisting of email address (or domain) and mail server IP/s (?)

A blacklist of email addresses ad IP can also be configured
(I think in a time out of 2 hours?)

As everything in greylisting is configurable

A main disadvantage of greylisting is the 30 minute wait that becomes part of greylisting. SpamCop Email no doubt are using what is the best set-up, but have to keep cards to chest as spammers read this newsgroup
JeffT last update and is still looking

Configurable? I don't see how. I obviously need to take a closer look at it. So far it's caught 19 legitimate items (18 from me to me - I keep my own email addresses off the whitelist purposely) and 1 from the server of someone in my address book - a "mail box full" rejection. Then 23 pages of legit items. That's just in a couple of days.

Another thing, I'm still hazy on how it is decided that this one goes to greylist and another one doesn't.

I think I may have to turn it off and just report/release them all manually as before.

It is only configurable in the sense that you can turn it on and off.

I updated the greylist management pages so they are sorted by from address. That should make it a little easier to find false positives.

Only e-mails that you wanted that are listed on the "blocked" page are actually false positives. *All* addresses e-mailing you for the first time will show up on the "pending" page for between 30 minutes and 5 hours. If you find a lot of e-mails that you wanted to receive listed on your "blocked" page, please e-mail me as soon as possible at trevorb[at]cesmail.net (while they are still listed).

The idea, again, is that when you receive an e-mail it has a "from" address, a "to" address, and the IP address of the server that sent it. We look at see if the combination from/to/IP has ever been seen before. If it hasn't, we send the ISP that sent it a "temporary failure" message, which tells them to try again in half an hour. If the ISP tries again in >30 minutes and <5 hours, the e-mail is allowed and all future e-mails with that from/to/IP combination are allowed. The "pending" list is those e-mails that have been received once, but haven't been retried yet. The "blocked" list is a list of e-mails that were received once and never retried in the 5 hour window. If they mail you again, they will be greylisted again and the process will start over.

The theory is that a lot of spammers send a message, and if it fails they retry constantly for about 5-15 minutes and then they never retry again. "Good" servers, however, usually try once every half hour for days before giving up.

Also, the greylist has been updated to use your personal whitelist. If an address is listed in your personal whitelist, it shouldn't be delayed by the greylist anymore.