Declining effectiveness of blocklists

[mshalperin]

While your statement is technically correct, Wazoo's frustration with that person did not appear to start with that Topic - it appears to have started that person's posts to the

Whatever provoked it, Wazoo's "frustration" threashold is a lot lower than it should be and not in the same ballpark as any other moderator Ive seen here. This is unfortunate as he othewise is very knowlegeable and helpful. I was particularly irritated by his insistance that my "inattention" to the content of that useless, annoying and mostly off-topic discussion was an issue. The newsgroup discussions, which weren't elaborated on in the Spamcop.net thread, did have interesting info, but wasn't really relevant to the issue I posed.

[Wazoo]

I was particularly irritated by his insistance that my "inattention" to the content of that useless, annoying and mostly off-topic discussion was an issue.

Try again. I did not make the original reference to that Topic. My additional comments used "the first referenced link" ... this link was the "second referenced link" in one post .. I did not start by pointing to that specific item until you made your complaints about reading it.

The newsgroup discussions, which weren't elaborated on in the Spamcop.net thread, did have interesting info, but wasn't really relevant to the issue I posed.

35472[/snapback]

Once again, you'll note .. I did not point you there, this additional data brought up by you in further posts.

Bottom line, continue the personal commentary and there will be a solution provided.

[Miss Betsy]

Whatever provoked it, Wazoo's "frustration" threashold is a lot lower than it should be and not in the same ballpark as any other moderator Ive seen here.  This is unfortunate as he othewise is very knowlegeable and helpful. I was particularly irritated by his insistance that my "inattention" to the content of that useless, annoying and mostly off-topic discussion was an issue. The newsgroup discussions, which weren't elaborated on in the Spamcop.net thread, did have interesting info, but wasn't really relevant to the issue I posed.

35472[/snapback]

The primary mode of support here is peer-to-peer, meaning users helping other users. (please remember this at all times!)

I am glad you recognize that Wazoo is very knowledgeable and helpful. I am repeating the quote at the top of the screen because we /are/ all volunteers. If you don't like the way that Wazoo phrases his replies, then skip over them. If there were several mechanics standing around looking at your engine and making comments, none of whom were paid by you to do so, you might not like one's manner of speaking. but I doubt if you would make a comment on his manner - because he obviously was being helpful and knew what he was talking about. This is the online version of that scene.

Hotmail (no filtering). Unless there is some silent upstream bl filtering, before getting to Spamcop email, and my true spam volume is much higher than I see, it would appear that the spammers are effectively defeating the Spamcop bl (and others). If the primary (really only) value of reporting spam to Spamcop is to support their bl, what is it now accomplishing?

From your first post: Hotmail filters aggressively, BTW, even if you choose the lowest level of filtering.

I am not an email customer and I cannot use blocklists to filter spam so I haven't posted. However, most of the unrepentent spamming ISPs are more or less perpetually blocked by the less aggressive blocklists (which apparently you are using). If the spam that gets by those blocks is primarily open proxies or trojanned machines, the SpamCop report alerts many ISPs to problems that they, then, scramble to correct before they get on the less aggressive blocklists. It seems to me that there are more people who are posting trying to find the problem than complaining about being blocked in error. To me that shows effectiveness.

There are a few (like Comcast) who ignore reports and do nothing to help their customers. In some thread recently, there was a discussion about blocking them.

The SpamCop bl is not only used by the email service, but many other ISPs. Again, since I don't use blocklists, I can't go into detail, but essentially there is a balance among various filters that people tweak in order to get the lowest rate of incoming spam uncaught and legitimate email caught. For many ISPs the scbl works well as part of their overall strategy and none of them have complained about declining effectiveness.

Also, it is the IP address from which the spam is sent that is added to the scbl. The same spamvertised sites may come from many IP addresses. Some people do block by spamvertised site (including perhaps spamassassin). IIRC, this was part of the confusion in previous posts, but maybe I have threads mixed up.

IOW, if you are interested in learning about how blocklists and other filters are used, you could continue this discussion. If all you want is an answer to your question, the answer is "No, the effectiveness of the scbl has not declined no matter what your personal use of filters seems to indicate."

Hope this helps

Miss Betsy

[mshalperin]

I am glad you recognize that Wazoo is very knowledgeable and helpful.  I am repeating the quote at the top of the screen because we /are/ all volunteers.  If you don't like the way that Wazoo phrases his replies, then skip over them.If there were several mechanics standing around looking at your engine and making comments, none of whom were paid by you to do so, you might not like one's manner of speaking.  but I doubt if you would make a comment on his manner - because he obviously was being helpful and knew what he was talking about.

I appreciate that you are volunteers and that this is a peer to peer site. If I didn't think Wazoo had potentially something valuable to say I would have ignored his posts. Also, if I thought he was genuinely trying to be helpful, I would have been unconcerned with an unpleasant manner. His dismissive attitude was negating that possibility, however. To use your auto mechanic analogy - he would be telling you that the evidence of your engine stalling was irrelevant and that because you were unmotivated and inattentive, you hadn't sufficiently researched the service manual for you to be knowledgeable enough to even know whether or not a problem exists...

There are a few (like Comcast) who ignore reports and do nothing to help their customers.  In some thread recently, there was a discussion about blocking them.

The SpamCop bl is not only used by the email service, but many other ISPs. 

Blocking Comcast and numerous off-shore domains would be great. Is that legal or practical?

Again, since I don't use blocklists, I can't go into detail, but essentially there is a balance among various filters that people tweak in order to get the lowest rate of  incoming spam uncaught and legitimate email caught.  For many ISPs the scbl works well as part of their overall strategy and none of them have complained about declining effectiveness.

I haven't seen any changes in overall effectiveness of the Spamcop email service blocking - only a sudden shift from the majority being blocked by various BL's to almost all by SpamAssassin. Since the BL's use origin IP addresses which have to be continually updated, I (wrongly) speculated that this might be due The BL's being defeated by more aggressive spammer activity through larger and larger botnets obscuring or rapidly changing the apparent origin, beyond the capacity of the BL's to keep up. In reality, it was due to a new ordering of filtering using SpamAssassin first. Apparently, my spam is especially susceptible to SpamAssassin.

Some people do block by spamvertised site (including perhaps spamassassin).  IIRC, this was part of the confusion in previous posts, but maybe I have threads mixed up.

From what I understand, SpamAssassin heuristically analyzes the content of the email, assigning a "level" of probability that it is spam and doesn't look at the origin or spamvertised IP addresses. It therefore is "static" and doen't depend on reporting to update the blocked IP address list. I'm not aware of any BL's that use spamvertised site addresses, but that would seem like a good idea.

IOW, if you are interested in learning about how blocklists and other filters are used, you could continue this discussion

.

Yes, I would like to hear more about this. I'm sure all of the staff and many of the participants are far more knowledgeable about this than me. Thanks for all your input.

[mshalperin]

Try again.  I did not make the original reference to that Topic.  My additional comments used "the first referenced link" ... this link was the "second referenced link" in one post .. I did not start by pointing to that specific item until you made your complaints about reading it.

No, you did make specific reference to it unrelated to my "complaints" when you stated:

"There was a reference to 'another' Topic .... Are Country blacklists working? ... there a re a number of "other" issues involved in that 'conversation. Take a hint and notice that it was moved to the Lounge for starters. Take some time to read it, note references there to additional newsgroup traffic, PMs involved, on and on .... "

I'm not conviced that there ever was any real confusion between the "first" and "second referenced link". My characterization of the content made it abundantly clear (especially to you as a participant in both) which one I was referring to in my early posts.

Once again, you'll note .. I did not point you there, this additional data brought up by you in further posts.

If you're referring to the newgroup posts, you did point me there - see above... In fact, that was the only place referenced in that link that had any real information about BL's (though not specifically relevant to me). I occasionally read the newgroups, but the discussions there are usually very technical and esoteric - oriented more to IP professionals.

[dbiel]

I am beginning to think that this entire topic should be moved to the Lounge as it has become nothing more than a shouting match accomplishing absolutely nothing!

[turetzsr]

...You said it, brother! <g>

[Miss Betsy]

I appreciate that you are volunteers and that this is a peer to peer site.  If I didn't think Wazoo had potentially something valuable to say I would have ignored his posts. Also, if I thought he was genuinely trying to be helpful, I would have been unconcerned with an unpleasant manner. His dismissive attitude was negating that possibility, however. To use your auto mechanic analogy - he would be telling you that the evidence of your engine stalling was irrelevant and that because you were unmotivated and inattentive, you hadn't sufficiently researched the service manual for you to be knowledgeable enough to even know whether or not a problem exists...

I will be a little blunter. If you don't like Wazoo's manner, keep it to yourself. Although you indicate that you might like to learn a little in response to my post, you followed that by going directly against my advice by bickering with Wazoo's terminology again. Following are some comments. If you are serious about learning what is happening, then you will read them carefully. Otherwise, your persoanl observations about your email are aneccotal and belong in the Lounge.

Blocking Comcast and numerous off-shore domains would be great.  Is that legal or practical?

Comcast is not a domain. It is an ISP with many servers that send email.

If you want to understand the problem, you need to look at the glossary. if you own a server, then you can block whatever you want - including all people who use 'Miss Betsy' as a signature. Whether it is practical is another question. I would, if I owned a server, block all Comcast even if I had correspondents (like my daughter) who used it. (You can look up what I have posted on the subject of blocklists and Miss Manners)

From what I understand, SpamAssassin heuristically analyzes the content of the email, assigning a "level" of probability that it is spam and doesn't look at the origin or spamvertised IP addresses. It therefore is "static" and doen't depend on reporting to update the blocked IP address list.  I'm not aware of any BL's that use spamvertised site addresses, but that would seem like a good idea.

I am technically non-fluent. However, I have used spamassassin and it does utilize blocking lists as part of its criteria on how to tag email as spam. It doesn't contribute to any blocklist, AFAIK, since its primary function is to tag email as spam. Whether it uses spamvertized sites as a criteria, I don't know. There are people who do use spamvertized sites to block, possibly using spamcop reports as a source, but I haven't followed that development very closely. Julian is not interested in that aspect.

[mshalperin]

Comcast is not a domain.  It is an ISP with many servers that send email.

If you want to understand the problem, you need to look at the glossary.  if you own a server, then you can block whatever you want - including all people who use 'Miss Betsy' as a signature.  Whether it is practical is another question.  I would, if I owned a server, block all Comcast even if I had correspondents (like my daughter) who used it.  (You can look up what I have posted on the subject of blocklists and Miss Manners)

35508[/snapback]

I was unclear. I was referring to off-shore domains and ISP's that transmit and/or generate spam or provide the "spamvertised" sites. I agree that any individual server owner or ISP can block whatever they want but I was referring to doing this with a widely distributed BL's...

There are people who do use spamvertized sites to block, possibly using spamcop reports as a source, but I haven't followed that development very closely.  Julian is not interested in that aspect.

It probably would require a lot of overhead to parse the body of each incoming email to check for reported spamvertised sites.

[Jeff G.]

It probably would require a lot of overhead to parse the body of each incoming email to check for reported spamvertised sites.

35644[/snapback]

It does, but some domains and their mail admins and executives consider it worth the investment in software and fast hardware to do that checking.

[Miss Betsy]

There are widely used bl's that cover different aspects than the scbl. There are ways to block countries that seem to harbor spamvertized sites and allow spammers to operate with impunity. There are, I believe, over 400 public bl's. Only a few are widely used, but the scbl is not the only bl out there.

Since I am not a server admin, I can't answer about the overhead to block spamvertized sites. However, it is not a matter of 'overhead' IIUC, but a matter of keeping up with the list of spamvertized sites to make an effective list. Spammers do all kinds of redirection and take sites up and down to avoid that. Spamassassin does give points if there is a 'biz' domain mentioned in an email, for instance, so there are certain criteria possible depending on how aggressive you want to be. Since I am not an email customer, I don't know exactly what filters are available to you, but I do know that there has been a lively discussion on country blocklists recently on the forum.

If one has a server, then one can tweak blocklists and content filters to achieve a pretty low percentage of spam. An end user has less to work with because s/he is dependent on whatever hir ISP allows/uses at the server level and has to use content filters which are not nearly as reliable at blocking spam since there are always legitimate emails that get caught. The problem is that if a legitimate email is caught by a content filter, then the receiver has to identify it manually - which is not easy to do amongst all the spam. Rejection by the server gives the sender a heads up that there is a problem. Non-identification by the end user means that the email disappears.

The reason that spam has not been eliminated is that not only are the technical details not widely known or understood by end users, but that there are varying attitudes toward the control.

Miss Betsy

[mshalperin]

There are ways to block countries that seem to harbor spamvertized sites and allow spammers to operate with impunity.  There are, I believe, over 400 public bl's.  Only a few are widely used, but the scbl is not the only bl out there.

35652[/snapback]

I haven't encountered the more agressive bl's in my limited experience. The Spamcop Email service uses multiple bl's in addition to the Spamcop bl which can be activated as desired (I use all available). Some are country specific types, but I don't know how broad handed they are.

The problem is that if a legitimate email is caught by a content filter, then the receiver has to identify it manually - which is not easy to do amongst all the spam.  Rejection by the server gives the sender a heads up that there is a problem.  Non-identification by the end user means that the email disappears.

I see your point, but can the server bounce backs to the the sender be misidentified as spam by the senders filters or server? The Spamcop email doesn't do this because a main function is to filter the spam in order to report it. This requires manual review to catch the false positives (and negatives) which aren't very frequent.

[Miss Betsy]

The way I understand rejection at the server level (bouncing is not used any longer because it is ambiguous about what it means) is that the server that rejects the email sends back a code with it that the sender's server then translates to create an email to send to the end user. It is returned as a standard non-delivered email message however that ISP does it.

IOW, it is not coming back as an email message that would be filtered as spam by the ISP and the sender's filter would recognize it as the standard non-delivery message.

I think that your original question was about the effectiveness of the spamcop bl filter. Again, since I haven't used the email service and don't use filters that I can control (except spamassassin), I can only make a guess.

The scbl is dynamic - that means that spam has to be coming from an IP address and being reported and that the first reports were ignored (or came from spam traps) for an IP address to be listed. The purpose is not only to block spam, but to give the server admin an opportunity to stop it. Like the rest of the problem of spam, the assumption in the beginning was that admins would take action quickly, and that users of the bl would spared dealing with spam from that IP address for a short time. And that does still happen - particularly when an admin does something dumb like forget to turn off relaying or a spammer slips under the wire, or someone gets an infection. The amount of time that spam is happening and senders from that IP would have their email blocked is minimal - not as long as the effects of a backhoe or thunderstorm. Naturally, someone has to receive that email to have it reported so that it can be fixed.

Other blocklists that are used are not so 'sensitive' as the scbl. They wait until it is obvious that spam is continuing to come from that IP address. When the address goes on their lists, it doesn't come off without some kind of procedure initiated by the admin of that IP address. One list, for instance, demands a charitable contribution (not to them, but to someone). Some admins refuse to pay the 'extortion' and so are always on that list - particularly ones with lots of servers. If a client complains, they will switch him to a 'clean' server.

If you are using country specific bl's, then a lot of spam will be caught by that blocklist (as well as any legitimate email from that country - I remember someone telling me that he actually got a real communication from Brazil! And then there are those who have family living in those countries who don't want to use the country blocklists). If you don't have any reason to be receiving email from that country, then every time you report what is caught it lengthens the scbl time those IP addresses are on the bl.

The reason that email service users have spamassassin is that the scbl works well enough so that responsible admins now are very careful about spammers and people didn't care about blocking the irresponsibly administered IP addresses permanently. Therefore, people wanted a way to use other criteria than the scbl to catch spam and filter it to a place where they could deal with it separately from their incoming email.

Some people who do not want to review hundreds of reports, simply report the spam that makes it past the filters (spammers are constantly trying to find ways to evade detection and blocking). And often, those reports go to admins who, unlike Comcast, do not have as many infected customers or will deal with the problem quickly.

As I said before, there are ways of tweaking filters that reduce the amount of spam you see in your inbox. Depending on your purpose in filtering, one allows either more or less. I personally would rather remove spam from my inbox than to have to hunt for a legitimate email in my held mail, but then I have the luxury of being able to change email addresses so that I no longer get spam by the hundreds. In fact, the only reason I get spam at all is due to a correspondent who got a virus, I believe.

Nigerian scam artists were here before email and they will be here as long as there are gullible people. Even if spam were only sent by criminals such as these, then the scbl would be a valuable tool in early detection and removal for responsible admins.

Miss Betsy