New Feature: Greylisting *UPDATED*

[Ex_Brit]

Thanks for the explanation. I'll look into it's workings over the next few days and see how it goes.

[petzl]

It is only configurable in the sense that you can turn it on and off.

Configurability I were referring to was what administration can do not the user

Also, the greylist has been updated to use your personal whitelist

And keep up the good work Thanks

[silentlarry]

Also, the greylist has been updated to use your personal whitelist. If an address is listed in your personal whitelist, it shouldn't be delayed by the greylist anymore.

Excellent. That was what I was waiting for.

So far SC greylisting has been kicking spamass on my aunt's account. This is a happy thing as she's UNhappy about each one that leaks through. So far, none have made it past greylisting to face the regular old filters. No problems with false positives (albeit it's set up throughly whitelisted, so not the most rigorous test).

I probably won't be using it as I like the fun of reporting (of course it's only "fun" 'cause my volume has been low). Good to know it's available when needed.

Anyway, just wanted to say thanks very much and good work.

Larry

[jongrose]

The idea, again, is that when you receive an e-mail it has a "from" address, a "to" address, and the IP address of the server that sent it. We look at see if the combination from/to/IP has ever been seen before...If they mail you again, they will be greylisted again and the process will start over.

Is the triplet combination saved permanently once it has been resent and bypasses the graylist or is there a time frame when this information expires and has to be passed through the graylist again? Or to put it another way, does the graylist have it's own internal whitelist (separate from a user's personal whitelist) for the triplet information and, if so, does the information in that whitelist ever come off it for whatever reason?

My second question may have already been answered. In the Graylist pending entries under Options->SpamCop Tools, I see there is a button where you can "Allow Checked Entries". If you select a pending email and hit this button, will the email come directly into your Inbox (or other folder) or will this only allow it to be received once it is resent by the other mail server? I see under Rejected Entries is mentions that the emails listed there are "Unrecoverable".

To be completely clear on how the graylisting feature works, does SC's SMTP server just check the triplet and send a bounce or does it fully receive the message, then check the triplet and bounce if it isn't recognized? I presume the latter option would be better suited for the users in a case where a legitimate email message were fully rejected, then the user could still view the message.

2) Those who just want little spam with no input on their side

For users of this type, the rejected email(s) could be set to automatically be deleted after a set time period in the scenario I mentioned above. If that kind of implementation is possible, I think it would be helpful in making graylisting more suitable for everyone - emails wouldn't be lost (unless they are not checked) and they still wouldn't show up in the mail folders.

Finally, since graylisting bounces spam messages, would it work in the same vein as MailWasher in that since the email bounced, *some* spammers would automatically purge the address from their list? Or is the bounce message not of the same ilk that would be used to remove an address from a list?

Graylisting kind of strikes me as similar to the Telezapper or anonymous call blocker for defeating telemarketing calls - it will block out many telemarketers but also stop some legitimate calls from coming through.

[michaelanglo]

I should explain that my SpamCop Mail account usage is about 40% direct to account mail (nearly all semi-dictionary spam). On the rest, half POP from a legacy account and a little forwarded from elsewhere, greylisting could have no effect.

I therefore hoped for a reduction of about 40% and got 35% because out of an expected 200 direct to spamcop spams, 26 still got through so used relays (or other servers configued to retry), proofing them against greylisting.

Here is the data for a full month of greylisting.

September '07 2684 spams (89/d), 59 leakers (=2.2 %), 0 false positive(s)

(August was 4369 spams (140/d), 80 leakers (=1.8 %), 0 false positive(s))

So a 37 % reduction, excellent.

Leakage rate still the same , any effect of the new SA was short lived

[trevorb]

Is the triplet combination saved permanently once it has been resent and bypasses the graylist or is there a time frame when this information expires and has to be passed through the graylist again? Or to put it another way, does the graylist have it's own internal whitelist (separate from a user's personal whitelist) for the triplet information and, if so, does the information in that whitelist ever come off it for whatever reason?

The triplet is saved for 36 days after the last time a message from that triplet was received.

My second question may have already been answered. In the Graylist pending entries under Options->SpamCop Tools, I see there is a button where you can "Allow Checked Entries". If you select a pending email and hit this button, will the email come directly into your Inbox (or other folder) or will this only allow it to be received once it is resent by the other mail server? I see under Rejected Entries is mentions that the emails listed there are "Unrecoverable".

Allowing an entry will allow the e-mail to pass through the next time it is sent, but the message will not be immediately available, for the reason explained after the next question:

To be completely clear on how the graylisting feature works, does SC's SMTP server just check the triplet and send a bounce or does it fully receive the message, then check the triplet and bounce if it isn't recognized? I presume the latter option would be better suited for the users in a case where a legitimate email message were fully rejected, then the user could still view the message.

The messages are blocked at the SMTP server. We don't store them at all.

Finally, since graylisting bounces spam messages, would it work in the same vein as MailWasher in that since the email bounced, *some* spammers would automatically purge the address from their list? Or is the bounce message not of the same ilk that would be used to remove an address from a list?

That depends on how the spammer treats bounces. It is likely that some spammers will remove you from their list if they get a greylist bounce.

-Trevor

[jongrose]

Alright, I've been using the greylisting feature for about a week now and it's working very well. However, I would like to make a couple suggestions:

1) Change the Manage Greylist pages, for both pending and rejected, to have the list sorted by date - preferably received, but at least one or the other because it's hard to manage, especially the rejected page, since the messages don't seem to be in any discernible order from what I can tell. This would make it much easier to organize the list by pages and tell what day you're on and see if you need to approve any emails that were rejected.

2) One other thing I would like to see for either of the greylist management pages is the subject line of the email if that is possible. I realize that the object is not to download the entire mail to prevent the mail system from consuming resources, but the subject line would be very helpful in determining if the email may or may not be legit in case we don't know the sender, and since the IP address doesn't help very much in this instance either.

3) I think it would also be very helpful to incorporate a whitelist button within the Horde Inbox console, like there is for the Held Mail console. This way a user can whitelist email addresses permanently that might have gotten stuck in the graylist, and not have to go to the options, SpamCop tools, etc. every time a user needs to add to the list.

That's all for now. Keep up the great work!

[proski]

I've enabled the greylisting, and it's working great. The spam level is perhaps 20% of what it used to be. That's still a lot, but at least the legitimate mail (including mailing list traffic) outnumbers spam now for the first time in years. It's also less likely that I would misreport a legitimate message as spam, as most spam that comes through is either in Asian languages or has unambiguous subject lines (all capitals, Viagra and all such stuff).

The greylisting block most "newsletters" I never subscribed to from companies I had business with. Those often use benign or attention-grabbing subject lines. How ironic is that the messages purporting to be legitimate are sent in the spam-like fashion and never retried! I'm glad they trust the delivery of their wares to the real companies, that don't give up after the first attempt

One thing I'm feeling a bit uneasy about is that if I were still receiving those "newsletters", perhaps I would refrain from dealing with the companies sending them. But on the other hand, it would be great if I ignored all companies who leaked my address, and I cannot identify them, because I'm using my Spamcop address directly, without sneakemail. And if I had a TV, maybe I would not buy something advertised too aggressively. It's just not practical that I expose myself to extra advertising so that I can make better choices.

When I first subscribed to the Spamcop webmail system, I decided to use my spamcop address for most of my business needs, hoping that smart spammers won't spam Spamcop addresses, and dumb spammers would be caught or neutralized in some way. Unfortunately, I was wrong. The spam quickly dwarfed the legitimate e-mail, even though I was reporting all the spam coming to me.

I think lessons should be learned from that. Reporting alone doesn't harm spammers enough. They spam Spamcop accounts directly with no fear, month after month. Something else needs to be done. Spammers and those who pay for their services should be prosecuted. It may not be the core mission of Spamcop, but if nobody is doing it, we shouldn't be thinking that every our spam report increases our karma and makes the world a better place.

I can understand Spamcop users who want to limit the amount of spam they get. We all have to choose our battles. I would rather limit what I receive and report only the spam that comes through despite all automatic measures.

For me, spammers are like mosquitoes. You don't go to the woods to fight mosquitoes, you fight those in your house.

[Ex_Brit]

I'm a little mystified with the greylist workings. I deal a fair amount with Tigerdirect and they email me quite a lot. Despite having in my personal whitelist *[at]tigeronline.com & *[at]promo.tigeronline.com listed, some emails coming from those addresses seem to get caught in the greylist and I have to constantly allow them.

For now I've disabled greylisting as I'm about to go on vacation and I will only be accessing my email about twice a week remotely (from an Internet Cafe) and don't want the hassle of having to check extra filters.

Edited December 7, 2007 by Ex_Brit

[petzl]

Despite having in my personal whitelist *[at]tigeronline.com & *[at]promo.tigeronline.com listed, some emails coming from those addresses seem to get caught in the greylist and I have to constantly allow them.

[at]tigeronline.com is not whitelisted

REMOVE THE [at] symbol and it would be

eg

tigeronline.com will whitelist

[Ex_Brit]

I live and I learn, thanks very much.

[bradfuller1959]

hi - I'm just trialling the greylisting feature and it seems a handy tool.

just wondering what the "# Blocked" column is denotiong?

I was assuming it represented the number of rejected/greylisted emails "from" that address so far?

But I think that's a mistake on my part... if the same message was resent from the same user that would be the trigger to "pass it on" wouldn't it...

it's just that I got sent a valid email overnight that got stuck in the greylist ..

Once I manually "allowed" it I got the next "resent email" fine....

any comments? :^)

cheers

brad

[Farelf]

...just wondering what the "# Blocked" column is denotiong?...

Hi Brad, your query has drawn no response so far so to get something happening ...

Merged with this lengthy topic - have you skimmed through it already?

Have you looked at http://www.greylisting.org/forums/index.php ?

As far as I can see your query is not specifically covered here or in the greylisting forum (I've not looked that closely) - though I would be surprised if the general discussion of the way it all works doesn't answer you. Hopefully an actual user can step in and point you in the right direction if it continues to elude you. Let's know how you're getting on, either way.

[StevenUnderwood]

Hi Brad, your query has drawn no response so far so to get something happening ...

I just looked in my account and found only 1 sender (alerts[at]live.com) which had multiple blocks. What I take it to mean is that their server does not re-send the message within the time window spamcop is configured for. They may not ever re-send the message as they may be multiple messages. I have allowed that entry to see what that message is. Don't remember signing up for live.com alerts, but it is possible.

[Ex_Brit]

I've had to stop using it because it occasionally catches legitimate emails and, unless I can pounce on that within 30 minutes, I'm doomed and have lost that email forever. The worst part of that scenario is that one can't even read the email in the greylist interface, like one can in the SC Webmail interface so have no way really of following it up.

The other aspect of that is for emails one is not sure of, it's impossible to read them to make a decision if they are good or bad.

I can't be constantly monitoring it for errors.

So, unless the designers can come up with a better system, I'm avoiding it for now, sorry guys.

Edited April 18, 2009 by Ex_Brit

[StevenUnderwood]

I've had to stop using it because it occasionally catches legitimate emails and, unless I can pounce on that within 30 minutes, I'm doomed and have lost that email forever.

Your choice, of course.

Do you mean the sender will refuse to resend it? After the sending server timesout its retries, it should be sending a failure to the originator explaining the issue. I know that was happening on one of friends emails (Lotus Notes user). He contacted me through another friend and I whitelisted him, no problem.

The issue is that case was that their retry was set to something longer than the SpamCop window was (something like an hour) so every retry was seen as a new message.

[Ex_Brit]

I have no idea and I don't see any way of getting them to do that from this end. Most people wouldn't know what to do anyway I would imagine.

If one could only read the emails and/or the headers for items caught in the greylist it might help.

[StevenUnderwood]

I have no idea and I don't see any way of getting them to do that from this end. Most people wouldn't know what to do anyway I would imagine.

If one could only read the emails and/or the headers for items caught in the greylist it might help.

That is the whole point of greylisting, the email/headers have not been accepted yet, so less traffic has been transferred. Apparently it will not work for you.

The first things sent during an SMTP transaction are:

EHLO/HELO (telling the receiver who the senders server says they are which can attempt to be confirmed by an IP lookup)

MAIL FROM: (telling the receiver what the senders email address is)

RCPT TO: (telling the receiver who the message is addressed to)

That information, along with the IP address which the receiver sees is the information that is rejected the first time it is seen and stored for the retry. When it sees the same pairing a second time within its programmed window, it will allow the message to pass.

[Ex_Brit]

Thanks for clarifying it. In my case I wan't seeing any attempt at resending in the allotted period so the messages were, I assume, lost.

I'll give it another try eventually no doubt.

[btech]

Quick question on something I saw today:

MY IP HERE SPOOFEDEMAIL[at]spam.COM 2009-07-15 15:44:00 2009-07-15 19:44:00 3

So I see where "MY IP HERE" is, that it's the IP of my host, where my email account is hosted. I don't see that IP on the "Rejected Entries", so it seems the graylisting is only holding the messages relayed from that IP, but inevitably delivering them to my inbox. Normal?

[agsteele]

MY IP HERE SPOOFEDEMAIL[at]spam.COM 2009-07-15 15:44:00 2009-07-15 19:44:00 3

So I see where "MY IP HERE" is, that it's the IP of my host, where my email account is hosted. I don't see that IP on the "Rejected Entries", so it seems the graylisting is only holding the messages relayed from that IP, but inevitably delivering them to my inbox. Normal?

I'm not completely sure what your question is but to say that grey listing doesn't do an IP check. GL simply delays delivery pending a second attempt by the relaying server. Since the vast majority of spammers do not keep attempting delivery, any mail item attempted once only is discarded as spam. Any item where a delivery attempt is retried is then passed onto your mailbox. So if the relaying IP tries a second time then it will be delivered. Once an Email address is seen as correctly delivered it is entered onto an OK list and the next message is delivered without delay.

Grey listing is one part of SpamCop EMAIL where the sending Email address is the key factor rather than the IP address - just to confuse us all further.

Andrew

[DavidT]

...it's the IP of my host, where my email account is hosted.

Are you using the greylisting feature on an an account to which you're having messages forwarded from the actual receiving address/host? IIUC, the greylisting is primarily useful only for messages sent *directly* to your SpamCop/CESMail email address.

Is that correct, email account customers?

DT

[rconner]

I could be wrong about the details, but as I understand, graylisting is used to discourage deliveries to MX hosts. If you forward mail (including spam) from somewhere else to the SpamCop MX, then I don't think you would get the benefit of graylisting (because your outgoing host is presumably not going to fall for the graylist trick). Only if the spam were coming straight to the SpamCop MX would it be subject to graylist blocking.

-- rick

[agsteele]

IIUC, the greylisting is primarily useful only for messages sent *directly* to your SpamCop/CESMail email address.

Is that correct, email account customers?

Yes, that's correct. Anything forwarded from another Email address should be delivered because the forwarding mail server will retry.

Andrew

[btech]

Yes, that's correct. Anything forwarded from another Email address should be delivered because the forwarding mail server will retry.

That explains that. I forward 7 email addresses to by cesmail acct, so graylisting is just delaying those forwards. I turned on boxtrapper [at] my host for some of my most hit accounts, so that should take care of some of this.