Sign in to follow this  
Followers 0
trevorb

New Feature: Greylisting *UPDATED*

81 posts in this topic

I suspect the way for spammers to get through GreyListing is to simply send two (or more) spams

The first will be sent for "retry". The spammers second spam (from same IP) is then passed by SpamCop as a "response" to first spam and whitelisted

Share this post


Link to post
Share on other sites
I suspect the way for spammers to get through GreyListing is to simply send two (or more) spams

The first will be sent for "retry". The spammers second spam (from same IP) is then passed by SpamCop as a "response" to first spam and whitelisted

Yes but there is that set time interval before SpamCop graylisting will let any more with the same "From:" through.

I happened to do this as part of my initial check and both first and second emails, sent minutes apart, were delayed by 50 minutes indicating a '400' response followed by a retry for both.

There is also the point that if the second shot is sent from the same IP address then an hour later this address may be on a blocklist due to the earlier spams.

And if it's sent from a different IP address then an greylist enhancement to look at sending IP addresses (which might be a good idea anyway, see previous post) would nullify any benefit.

Share this post


Link to post
Share on other sites
Yes but there is that set time interval before SpamCop graylisting will let any more with the same "From:" through.

spam I see being sent is multiple of same spams sent again again again etc (e.g Canadian pharm)

It is though then trapped by SpamCop emails spam filters.

The spam getting held is in greatly reduced numbers than before turning GeyListing on

Just wondering how this is getting past Greylisting in first place?

As GreyListing stops the reporting of that spam it "retries" without response I see these lists getting weakened (So my/the idea is to make SpamCops GreyListing 100% perfect and better than the rest)

If however GreyListing stops spammers without need for reporting this becomes a good thing

Although initially slow but less than a hour before I received test emails from colleagues?

Is there also a over time limit for GreyList reply?). Importantly no test emails sent have disappeared (no false positives)

I wonder if the GreyListing "whitelist" can be compared to the SCBL with entries removed periodically or even immediately. So far all IP's I checked and have made it through to my spam folder were listed on the SCBL and not mail servers(the SCBL is reluctant to list mail servers)

Can SpamCop email customers add to the "SpamCop GreyList whitelist" in WebMail options (with email "from" field) Ideally my existing whitelist

Share this post


Link to post
Share on other sites

I think I may have spoken too soon.. the spam rate went up to 50% of normal(which is still VERY good). For a day or so, it was really low and I did see the test messages I sent were delayed, but they WERE delivered, so that's a positive for me. I was a little hesitant to try the gray, because I've had trouble with getting mail from a few of my accounts (no fault of SC, though).

Share this post


Link to post
Share on other sites
Importantly no test emails sent have disappeared (no false positives)

I understand SpamCop greylisting to be working like I'm used to (a la Sneakemail), which means if your test mails are being sent from a correctly-configured server they will not fail the greylisting, ever.

The 'false positives' come in to play when a legitimate sender unfortunately has a mail server which is misconfigured and doesn't follow the re-try/re-send per the RFC. I've seen this happen at large and small companies, private or otherwise, as well as email giants like Yahoo or MSN.

The legitimate sender is generally not in a position to know that his mail server is misconfigured. All he knows is that his mail to you bounces. If he has only an email address as a contact for you, then it's over... You don't know (without the views Jeff has indicated are coming) that he sent mail to you, and he has no way to tell you it never reached you.

--appyface

Share this post


Link to post
Share on other sites
And if it's sent from a different IP address then an greylist enhancement to look at sending IP addresses (which might be a good idea anyway, see previous post) would nullify any benefit.

Following up to myself, I have now read the greylisting white paper

http://greylisting.org/articles/whitepaper.shtml

and this does use sending server IP addresses as well as From:

Can we have a reference to the details of the actual implementation ?

I also note that there was only one greylisting delay to my forwarded mail, perhaps because

the forwarding ISP inserts the same Return-Path: irrespective of the actual From:

Which is good but may provide a loophole.

Share this post


Link to post
Share on other sites
Following up to myself, I have now read the greylisting white paper

http://greylisting.org/articles/whitepaper.shtml

and this does use sending server IP addresses as well as From:

Can we have a reference to the details of the actual implementation ?

I also note that there was only one greylisting delay to my forwarded mail, perhaps because

the forwarding ISP inserts the same Return-Path: irrespective of the actual From:

Which is good but may provide a loophole.

Yes, we have an implementation very much like the reference implementation. There is a triplet of sender email address, connecting IP address, and recipient email address that we use to make decisions. We are currently using a 30 minute delay for newly discovered triplets.

Petzl asked why spam is still getting through and it is simply because the spammers are retrying. For spammers willing to retry, this method doesn't help at all. However, there is evidence that a large number of spammers do not retry.

JT

Share this post


Link to post
Share on other sites
Petzl asked why spam is still getting through and it is simply because the spammers are retrying. For spammers willing to retry, this method doesn't help at all. However, there is evidence that a large number of spammers do not retry.

Thanks for the info

One feature, now disabled, was for spamcop VER reporting was to add to a "From" Blacklist. Can this be re-enabled for "SpamCop Greylist"? Idealy to make a fingerprint of both "From and IP address"?

Share this post


Link to post
Share on other sites
One feature, now disabled, was for spamcop VER reporting was to add to a "From" Blacklist. Can this be re-enabled for "SpamCop Greylist"? Idealy to make a fingerprint of both "From and IP address"?

If you're asking for a blacklist to be created automatically from your reported spam, this will never happen. Spammers rarely reuse the same email address, so there is really no point. And, how are you going to manage this data? Over months, we'd end up with millions of email addresses on this "blacklist". Your own blacklist would have thousands of entries (assuming you report spam a lot), pretty much none of which will ever email you again.

JT

Share this post


Link to post
Share on other sites

For those of you who are interested in this sort of thing, we have some rough statistics captured since we announced the greylist feature:

Greylist entries allowed: 44966

Greylist entries waiting for a retry: 1877

Greylist entries rejected: 57362

Users with greylisting enabled: 270

Approximately 56% of incoming mail to those 270 account is being rejected as spam.

We have been working on an addition that lets the whitelisted e-mail addresses by without greylisting them. It is entering beta testing now (behind the scenes), and we will roll it out in a couple of days.

~Trevor

Share this post


Link to post
Share on other sites
If you're asking for a blacklist to be created automatically from your reported spam, this will never happen. Spammers rarely reuse the same email address, so there is really no point. And, how are you going to manage this data? Over months, we'd end up with millions of email addresses on this "blacklist". Your own blacklist would have thousands of entries (assuming you report spam a lot), pretty much none of which will ever email you again.

The same email address is reused for a few hours though (probably the same spam run) Where the spammer is sending again and again? I'm suggesting a blacklist be made and then reset/expire every two/three hours. Greylisting can maybe still send "try again"?

Just trying to get spam passing down to zero It may be in too hard basket but just suggesting. I do see you are improving things as we speak (trevorb's post). If whitelisting can be succesfully implemented the retry time can be increased/doubled meaning even less spam should get past

Edited by Wazoo

Share this post


Link to post
Share on other sites
We have just added a new spam-blocking feature called greylisting to our mailgates. When enabled, greylisting delays messages from unseen e-mail addresses for a short time (configured to 30 minutes right now). Messages from addresses that have been seen before are allowed through immediately.

Please use this forum to discuss your results. We are interested in hearing about how well this feature works for you. If you encounter any problems, send an e-mail to support[at]spamcop.net.

I tried the "greylist" & it did cut down the spam significantly. However the reason I use SpamCop is to report spam, not just delete/block the stuff (I can get that done for free by other providers). So as long as I'm paying for your service, I prefer to report all the *![at]#!!* :angry: stuff.

Thanks.

bcstones

Share this post


Link to post
Share on other sites

We added greylist management pages. See the first post in this thread for more information.

~Trevor

Moderator Edit: single paragraph content added to this post to remove the need to backtrack a page to locate this new data.

***UPDATE***

We have added management pages so you can view the messages that are pending in your greylist, and the messages that have been permanently blocked in the past 72 hours. Click Options->Spamcop Tools->Manage Greylist - ... to view your greylist entries. From these pages you can manually unblock senders.

Share this post


Link to post
Share on other sites

I promised to report on my experience with graylisting

Summary, overall spam received is 35 % down on the average for last month (August).

My first full day wih greylisting was 2007-08-31

The results for the following 11 days were 996 spams 90/day 14 leakers (=1.4 %)

This is a reduction on the 140/day average for August.

(4369 spams, 80 leakers (=1.8 %), 0 false positive(s) )

The improvement in leakage may be due to the new release of SpamAssassin 2007-08-28

I should explain that my SpamCop Mail account usage is about 40% direct to account mail (nearly all semi-dictionary spam). On the rest, half POP from a legacy account and a little forwarded from elsewhere, greylisting could have no effect.

I therefore hoped for a reduction of about 40% and got 35% because out of an expected 200 direct to spamcop spams, 26 still got through so used relays (or other servers configued to retry), proofing them against greylisting.

In other news 53 % of spam received during those 11 days had a source reportable to a Chinese ISP (since I use quick reporting I didn't realise this before - I analysed the text of the emails that quick reporting sends).

Share this post


Link to post
Share on other sites
We added greylist management pages. See the first post in this thread for more information.

Click Options->Spamcop Tools->Manage Greylist - ... to view your greylist entries. From these pages you can manually unblock senders.

Is there anyway the SMTP "HELO" command can be enhanced to block spam or use with a greylist? All the spam I see getting through has

"Received: from unknown (HELO gwlrtjk) (201.38.214.16)"

I would like to know if one could also get an option to reject/hold email on a improper HELO response?

Or are there to many incompetent providers out there (A reason not to auto accept an email account from a ISP?)

Perhaps at first one should just tag such email as ???

Share this post


Link to post
Share on other sites
We have just added a new spam-blocking feature called greylisting to our mailgates. When enabled, greylisting delays messages from unseen e-mail addresses for a short time (configured to 30 minutes right now). Messages from addresses that have been seen before are allowed through immediately.

Since mail servers typically retry every 15 minutes would it not be better to delay only say 10 -14 minutes instead of 30? In 10 minutes due to spam traps etc. a sender is likely been reported to many blacklists and possibly razor. This would be less delay to wait also.

I have sent several emails from my email address at work over last couple days. They have never gotten through yet. Its running on Exim with Clamav and spamassassin. I am not sure why but I suspect its not retrying every 15 minutes like its supposed to by looking at the log files. In exim.conf it looks to be setup to try every 15 minutes though. I suspect maybe a high load due to spamassissin is delaying processing the queue and the large amount of messages in the queue.

Perhaps the windows can be extended from 4 hours to 12 hours to fix things like this?

Share this post


Link to post
Share on other sites

I just noticed that in 'rejected entries' that greylisting is blocking a few gmail addresses trying to email my account. Looking at the source IP they are indeed coming from gmail servers. This clearly is not good. I think the timeouts need expanded or something. Untill then I have turned greylisting off. I think its an excellent idea but the minimum of 30 minutes of wait and timeout of 4 hours is just too narrow. It should be 10 minutes to like 12 hours in my opinion. Perhaps even just 5 minutes. Even 5 minutes will give new spam sources time to hit a few spam traps and get listed so blacklist and/or spamassassin can catch them.

Edited by mh88

Share this post


Link to post
Share on other sites
I just noticed that in 'rejected entries' that greylisting is blocking a few gmail addresses trying to email my account. Looking at the source IP they are indeed coming from gmail servers.

Greylisting works for me (and a major majority of users) that is not to say it will be everyone's cup of tea

As for Gmail I only see it as good for recieving email not sending. (Hotmail is a better choice as it is competently set-up)

That said I did a test from Gmail and recieved it in 30 minutes

Share this post


Link to post
Share on other sites

The emails that end up on the Greylist are supposed to be from "unseen addresses". If I email myself from any of various email addresses I see that they all get stopped by Greylist and I have to go in and allow them. I would expect those to only be stopped by the regular SC filters and be in my Held Mail only.

Not that I email myself that often, but as I triple boot I sometimes do in order to receive an email in one of my other systems.

Now that I have allowed them once, will they always be treated that way?

I had hoped that with the new Email interface that we would at last be able to go to specific pages in Greylist/Whitelist/Blacklist - i.e. instead of just <Prev Next> we would have [1] [2] [3] [Last] like most other applications.

Edited by Ex_Brit

Share this post


Link to post
Share on other sites
I had hoped that with the new Email interface that we would at last be able to go to specific pages in Greylist/Whitelist/Blacklist - i.e. instead of just <Prev Next> we would have [1] [2] [3] [Last] like most other applications.

Here is a 'crude' workaround until the problem is fixed (or to phrase it another way, the enhancement is made).....

When you want to go to a specific page

APPEND ?page=2

to the URL while reviewing...

You can put whatever page number you would like...

(Incidently, this work great too, when review your whitelist/blacklist enteries for addresses)

Share this post


Link to post
Share on other sites

Thanks for the tip ;-)

Share this post


Link to post
Share on other sites

How is this greylist arranged? I see no particular logic to how each entry is filed. I just went in to check that nothing legit was caught in the rejected entries by accident and if the most recent items were filed on page 1 onwards then it would have been a lot easier to check.

Share this post


Link to post
Share on other sites

How is this greylist arranged? I see no particular logic to how each entry is filed. I just went in to check that nothing legit was caught in the rejected entries by accident and if the most recent items were filed on page 1 onwards then it would have been a lot easier to check.

My understanding is that Greylisting is configurable around a whitelisting of email address (or domain) and mail server IP/s (?)

A blacklist of email addresses ad IP can also be configured

(I think in a time out of 2 hours?)

As everything in greylisting is configurable

A main disadvantage of greylisting is the 30 minute wait that becomes part of greylisting. SpamCop Email no doubt are using what is the best set-up, but have to keep cards to chest as spammers read this newsgroup

JeffT last update and is still looking

Share this post


Link to post
Share on other sites

Configurable? I don't see how. I obviously need to take a closer look at it. So far it's caught 19 legitimate items (18 from me to me - I keep my own email addresses off the whitelist purposely) and 1 from the server of someone in my address book - a "mail box full" rejection. Then 23 pages of legit items. That's just in a couple of days.

Another thing, I'm still hazy on how it is decided that this one goes to greylist and another one doesn't.

I think I may have to turn it off and just report/release them all manually as before.

Edited by Ex_Brit

Share this post


Link to post
Share on other sites

Configurable? I don't see how. I obviously need to take a closer look at it. So far it's caught 19 legitimate items (18 from me to me - I keep my own email addresses off the whitelist purposely) and 1 from the server of someone in my address book - a "mail box full" rejection. Then 23 pages of legit items. That's just in a couple of days.

Another thing, I'm still hazy on how it is decided that this one goes to greylist and another one doesn't.

I think I may have to turn it off and just report/release them all manually as before.

It is only configurable in the sense that you can turn it on and off.

I updated the greylist management pages so they are sorted by from address. That should make it a little easier to find false positives.

Only e-mails that you wanted that are listed on the "blocked" page are actually false positives. *All* addresses e-mailing you for the first time will show up on the "pending" page for between 30 minutes and 5 hours. If you find a lot of e-mails that you wanted to receive listed on your "blocked" page, please e-mail me as soon as possible at trevorb[at]cesmail.net (while they are still listed).

The idea, again, is that when you receive an e-mail it has a "from" address, a "to" address, and the IP address of the server that sent it. We look at see if the combination from/to/IP has ever been seen before. If it hasn't, we send the ISP that sent it a "temporary failure" message, which tells them to try again in half an hour. If the ISP tries again in >30 minutes and <5 hours, the e-mail is allowed and all future e-mails with that from/to/IP combination are allowed. The "pending" list is those e-mails that have been received once, but haven't been retried yet. The "blocked" list is a list of e-mails that were received once and never retried in the 5 hour window. If they mail you again, they will be greylisted again and the process will start over.

The theory is that a lot of spammers send a message, and if it fails they retry constantly for about 5-15 minutes and then they never retry again. "Good" servers, however, usually try once every half hour for days before giving up.

Also, the greylist has been updated to use your personal whitelist. If an address is listed in your personal whitelist, it shouldn't be delayed by the greylist anymore.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0