Test Case; Which IP gets blacklisted from these header?

[snowman]

Using the example below; assuming an IP is getting blacklisted by spamcop, which IP is it? and why? How do the headers point to the guilty server?

These are real; except I masked the guilty servers out of course. I am confused as to which IP is the problem source. The first? or the last who touched the email before arriving at my server?

===============================================

Microsoft Mail Internet Headers Version 2.0

Received: from a-server.somewhere.com ([55.66.77.88]) by nowhere.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 11 Oct 2006 12:45:28 -0700

Received: from b-server.somewhere.com (b-server-34ds.somewhere.com [33.44.55.66]) by a-server.somewhere.com (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Apr 15 2004)) with ESMTP id <2A35FAS2LMJJHG0[at]a-server.somewhere.com> for me[at]nowhere.com; Wed, 11 Oct 2006 13:47:07 -0600 (MDT)

Received: from bill ([11.22.33.44]) by b-server.somewhere.com (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Apr 15 2004)) with ESMTP id <2A35FA32LKJLL99[at]b-server.somewhere.com > for me[at]nowhere.com; Wed, 11 Oct 2006 13:47:07 -0600 (MDT)

Date: Wed, 11 Oct 2006 12:47:06 -0700

From: bill <bill[at]somewhere.com>

Subject: You just got a raise!

To: <me[at]nowhere.com>

[StevenUnderwood]

There is no way to tell because the parser includes a trust factor which you have eliminated.

That being stated, ff b-server is trusted to accurately stamp the headers, I believe it would be the machine bill ([11.22.33.44]) which would be tagged as the source by spamcop (see how the source is calculated through Yahoo email right now). This method is not the simplest for servers using DNSBL's to test for (they need to check more than the connecting IP like spamcop email service does), but it is the machine sending the spam and would protect against that machine sending directly to MX. It also protects the most innocent victims. That is a decision made by the people at spamcop.

I have attached a parse of a message sent to my spamcop account from my yahoo account which reports my home IP address. http://www.spamcop.net/sc?id=z1100108181z4...43b7f0a011be61z

[snowman]

Actually your wrong. But I would have guessed that same answer. So we both got fooled .

Spamcop set 55.66.77.88 as blacklisted. It is the last server to touch the email before it was delivered to email system that rejected it using spamcop.

I have not removed any trust factors or text to play games here. I just replaced the IP and server names to hide the ISP. I also replaced the email server with an Microsoft Exchange Server - Microsoft SMTPSVC to hide the owenership out of courtesy. We are just solving a technical problem here not starting a war!

55.66.77.88 is a firewall/bridgehead server that has 7 or so large email servers behind it. Over 8,000 emails an hour go through the 55.66.77.88 server and it got listed because a ISP customer somewhere behind the two layers of email servers sent a spam messages. Thus this message was blocked by a-server. Maybe there was lots of spam through 55.66.77.88 who knows to be honest, but is 10, 100 or 1000 lots if you send over 100,000 good ones? I don't think so.

However the point is why list the firewall/bridgehead IP. Should not spamcop list the actual source and be more accurate? Nail spamer instead of a firewall/bridgehead?

Can anybody tell me why this is the case? Is there something wrong with these headers that the ISP servers made? Format maybe that caused such a problem in the parsing?

[Wazoo]

StevenUnderwood tried to be nice.

I'll be blunt.

You "are" playing games.

For the "real and technical" answer, start with submiting an actual e-mail through the parser with 'full / technical' details turned on, read the parsing results. If you wish assistance, provide the Tracking URL of that parse here. If you want to argue, please locate the various listings of 'how to contact someone official' ...

Your 'numbers' argument doesn't carry much weight. GMail and Yahoo servers pass tons of traffic, but due to the way they are configured, they manage to get listed a lot. The easy assumption at this point is that your servers are configured similarly, not showing the real, technically correct, etc. actual handoffs between systems/servers such that the "real spammer" can't be tracked because the 'chain-test' fails .. (just one of the possibilities) .. How the SpamCopDNSBL works and what data is involved is linked to from the SpamCop FAQ here, the Wiki, the 'official' FAQ ... redefined, re-explained, reworded over and over and over and over in so many previous Topics/Discussions here ....

That no one here really likes to waste their volunteered time playing with 'doctored data' should be pretty obvious .... you want 'real' answers, you need to provide 'real' data ...

[snowman]

What parser? Is there a tool that answers my question about who gets listed in the blacklist using the headers? Can you post the URL for me and I will go play with it for a while.

About the games, just because its a computer it does not mean you or I can slander the owner by posting thier ip addresses all over the place. Some respect is required even if the system owner has made a configuration mistake - or maybe they did not and it is the spamcop code that is incorrect. Could be, it has happened before. Why you so defensive anyway? It is just a computer.

[Wazoo]

You managed to push my buttons. Ignored all the BIG BOLD BLACK words all over the screen, ignored the direct references to things like the SpamCop FAQ (some of those BIG BOLD thngs) ....

How about the Start Here .. before you make your first post? thing?

"slander someone by posting an IP address" .... where did you come up with this?

"respect is required ..." ??????? ...... How about respecting all the work that's already been done "here" to provide available answers, definitions, explanations that are freely available to anyone to simply read at their leisure?

Me defensive?????? I think you are sorely confused ... I'm trying to point out that you are wasting a lot of folks' time by the game-playing with 'constructed' data, assumedly wanting to save your precious time by not having to do any research on your own.

If you aren't "in charge" of that server farm, then send whoever is "here" to ask the real questions ... or have them contact the folks that get paid to answer questions directly ..... there's a lot of folks here willing to "help" .. but ..... I don't care to see any of them waste their precious time playing guessing games ...

[snowman]

Amazing assumptions. I actually read for 2 hours before I posted today. ALL the FAQ's I could find. That is not counting countless hours of studying the subject over the last year.

I don't understand why you want to argue about the header data. It is exactly the same as the email except for the ip and server names.

Why can't somebody explain the spamcop logic of determining what server sent the email that caused the complaint. Is this a big secret or something? Are you mad at me for asking a quesitons or at the question itself?

This logic has to be well understood by somebody. If you Google the topic "spamcop parser" you will read two opinions. The first is to report the first server to touch the data from the sender and the second the last server to send it to the receipient server both ignoring the middle relay servers.

I don't know which is best, I just need to know what the programmers at spamcop think is best. What do they do and why if that is possible.

It will help me determine if the headers are formated correctly and spamcop's parser can handle them correctly.

[dbiel]

Actually it is NOT possible to answer your question here; at least as I understand it.

The reason any IP address get added to the list depends on numerous factors that are based on that address.

Is the address properly registered

Does it append correct headers that can be verified and trusted.

How much traffic goes through it

What is the nature of the bad traffic (spamtrap hits / user reported)

SpamCop will consider the source of the problem to be the last IP address that can be trusted to not have been forged.

So at this point the best guess is that the firewall IP address is not properly registered and is therefor considered the last trusted address.

The headers alone are not good enough to answer your question.

Each header needs to be verified with what is actually registered and the ONLY way to do that is with the real complete data.

The handoff for server to server must be clear and verifiable.

Read the following and ask specific questions http://forum.spamcop.net/scwik/SpamCopBlockingList

[snowman]

Thanks. That gives me more more information to review and things to ask quesitons about. I will go look at the dns entries and see how the system as a whole is setup.

One thing you wrote surprises me. That the last address considered NOT to be forged is the one that spamcops picks.

It would seem to me (in a general way of course) to be two kinds of spam messages.

One that is purposely forged and overtly designed to fool a server into delivery. The second, one that is really 100% correct in the headers and come from real servers but just has spam in the message because some computer has been compromised on a otherwise valid network.

Would the second kind be most likely to trigger a false-positive?

[dbiel]

One thing you wrote surprises me. That the last address considered NOT to be forged is the one that spamcops picks.

That is not to say that it is forged. It is just to say that there is no way to tell if has or has not been forged. It is the last point that we know with full certainty that the message came from. Every thing after that is simply a matter of blind trust which has proven in way too many cases to represent forged data.

[StevenUnderwood]

I don't understand why you want to argue about the header data. It is exactly the same as the email except for the ip and server names.

As stated several times now (both by myself and others here) those IP addresses and server names have a history associated with them. Some have been shown to be trustworthy, others have not.

[Telarin]

AFAIK, it works like this:

Received: from a-server.somewhere.com ([55.66.77.88]) by nowhere.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 11 Oct 2006 12:45:28 -0700

Two things can happen:

A ) If spamcop knows that 55.66.77.88 belongs to XYZ ISP and that it correctly stamps it received: from headers, then it trusts this header line and goes on to the next.

or

B ) On the other hand, if 55.66.77.88 is a known open relay that will send anything it is given, then anything beyond this line is POTENTIALLY forged and cannot be trusted, therefore 55.66.77.88 is considered the source of the message and if enough instances are seen, is added to the SCBL.

If option A occurs, then spamcop goes on to the next received line:

Received: from b-server.somewhere.com (b-server-34ds.somewhere.com [33.44.55.66]) by a-server.somewhere.com (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Apr 15 2004)) with ESMTP id <2A35FAS2LMJJHG0[at]a-server.somewhere.com> for me[at]nowhere.com; Wed, 11 Oct 2006 13:47:07 -0600 (MDT)

and repeats the same process as above. If 33.44.55.66 can be trusted, it moves on, otherwise it considers 33.44.55.66 to be the source.

Received: from bill ([11.22.33.44]) by b-server.somewhere.com (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Apr 15 2004)) with ESMTP id <2A35FA32LKJLL99[at]b-server.somewhere.com > for me[at]nowhere.com; Wed, 11 Oct 2006 13:47:07 -0600 (MDT)

If all prior received lines are trusted, then the final line is processes and 11.22.33.44 is considered the source. Again, this does not mean it is automatically added to the SCBL after one message. A number of other factors are taken into account, like:

How many messages is this server sending?

How many reports are there against it?

How many spamtrap hits are there against it?

Does it have a prior listing history?

Those are the ones that we know of, and I'm sure there are other criteria that are taken into account before the SCBL listing shows up.

Now, if you are actually asking about the FILTERING process at a receiving ISP, then that is a bit different.

In your above example, most ISPs will not look at the headers at all when considering filtering, as that means they have to commit to receiving the message and paying for the bandwidth. Most ISPs will simply look at the IP address of the server that is trying to send the message, and will make a delivery decision based on that IP address.

I believe the spamcop email system is an exception, it examines all header lines in a message for "bad" IPs and considers those in its weighing of the message as potential spam. There are other mail systems that work like this as well, however, I believe they are the exception rather than the rule.

[Miss Betsy]

One that is purposely forged and overtly designed to fool a server into delivery. The second, one that is really 100% correct in the headers and come from real servers but just has spam in the message because some computer has been compromised on a otherwise valid network.

Would the second kind be most likely to trigger a false-positive?

Neither one would trigger a false-positive. If spam (lower case since uppercase spam is the Hormel meat product) is coming from an IP address and is reported, then it will go on the scbl.

The problem with forged headers and where the spamcop parser stops is that the parser is software and doesn't 'think'. If the 'real' headers are configured properly, the parser stops at the last header line that can be verified by its rules. Usually it is correct, but occasionally a human reading the same headers will get a different answer because the human doesn't have to use the exact same rules every time.

I don't know what you are trying to accomplish, but whatever it is you will need to know a lot about headers before you can even begin to understand why the parser does what it does.

Miss Betsy

[turetzsr]

What parser? Is there a tool that answers my question about who gets listed in the blacklist using the headers? Can you post the URL for me and I will go play with it for a while.

<snip>

...http://www.spamcop.net/. You will need to sign up for a reporting account. See the link labeled "Register Now" under the heading "REPORT spam."

<snip>

I don't understand why you want to argue about the header data. It is exactly the same as the email except for the ip and server names.

...But the point is that the SpamCop parser works is very precise and may generate results very differently on manipulated data than it does on the exact data in the real internet headers of an e-mail.

Why can't somebody explain the spamcop logic of determining what server sent the email that caused the complaint. Is this a big secret or something? <snip>

This logic has to be well understood by somebody. If you Google the topic "spamcop parser" you will read two opinions. The first is to report the first server to touch the data from the sender and the second the last server to send it to the receipient server both ignoring the middle relay servers.

I don't know which is best, I just need to know what the programmers at spamcop think is best. What do they do and why if that is possible.

<snip>

...The "opinions" you have found in your Google results are totally irrelevant to this discussion because there are only a limited number of people (perhaps only one person) who really know(s) -- the SpamCop parser programmer(s). IIUC, there are at least two very good reasons that no one here (or anywhere else on the internet) will authoritatively explain to you the SpamCop parser logic:

1. We don't know, because we haven't been "clued in" by the programmer(s).
   
2. If we did know, we wouldn't publish the information in a public forum like this because then it could be "gamed" by the spammers.
   

[snowman]

So I gather that the right answer depends on soley on what spamcop knows about the various IP's along a messages path at the time the message is being transmitted.

That makes sense now that you have all explained it. Also it is clear that an IP of a server can sometimes be clean with no reports and sometimes be listed because the user reports.

Another question if I may:

If a server that relays >7,000 of non-spam, not-reported-as-spam messages an hour, and maybe a few <3 reported as spam messages in the same hour how does one make sure that spamcop system knows about the good?

It is being told about the bad reported-as-spam by humans somewhere, and they may not be real spam, just that somebody labeled them as spam with one of the email tools?

If the overwellming majority of the messages go to other domains that do not report good messages, then the 3 can look like a lot if they all went to some poor soul at a single domain.

How does one make sure spamcop knows about the good messages?

[turetzsr]

<snip>

It is being told about the bad reported-as-spam by humans somewhere, and they may not be real spam, just that somebody labeled them as spam with one of the email tools?

If the overwellming majority of the messages go to other domains that do not report good messages, then the 3 can look like a lot if they all went to some poor soul at a single domain.

<snip>

...Not sure what you mean by "somebody labeled them as spam with one of the email tools" but IIUC your question is addressed by the SpamCop FAQ (see link near top left of page) item labeled "How can I be de-listed" under the label "User Error."

[snowman]

That is not what I am asking excatly.

My question centers around the fact that if a server has lots and lots of good-not-reported email and a few bad-reported emails how does that server make sure the spamcop knows about the good when it does its calculations about listing. Clearly 3 bad out of 7,000 good does not mean much of a problem.

Just because somebody labels a message as spam using a email tool does not mean it really is. Some stuff is obviously spam, and everybody knows that, but there is a lot that can go either way depending on the receiver attitudes I am sure we can all agree.

[dbiel]

So I gather that the right answer depends on soley on what spamcop knows about the various IP's along a

How does one make sure spamcop knows about the good messages?

SpamCop uses http://www.senderbase.org/ but you need the real IP address - not fake data.

[Telarin]

Senderbase provides spamcop with the statistics for overall email volume going out of a given IP address. I believe they primarily get this information from the numerous email filtering appliances that they have installed all over the world.

If a server sends only 3 spam messages, the chances of one of those ending up in the hands of a spamcop reporter are very very small.

Some stuff is obviously spam, and everybody knows that, but there is a lot that can go either way depending on the receiver attitudes I am sure we can all agree.

I disagree. Either you consented to receive messages from the sender, or you did not.

If someone reports something that is not spam, the sender can contest the report. Each report sent out contains a link for doing this. However, they would need to have proof that the reported really did consent to receive it. This is one of the reasons it is important to confirm email addresses and save important information like the IP it was confirmed from, when it was confirmed, what website was used to sign up the receiver, etc.

[turetzsr]

That is not what I am asking excatly.

My question centers around the fact that if a server has lots and lots of good-not-reported email and a few bad-reported emails how does that server make sure the spamcop knows about the good when it does its calculations about listing. Clearly 3 bad out of 7,000 good does not mean much of a problem.

...Yep, I understood that. It wasn't what I was suggesting. My intent was only to address the matter of someone reporting non-spam as spam.

Just because somebody labels a message as spam using a email tool does not mean it really is.

<snip>

...This is probably a bit orthogonal to your question but: what "email tool" are we talking about, here? SpamCop does not get any of its information, as far as I know, from any email tool (except indirectly from those reporters who are also subscribers to the SpamCop e-mail system and use its automated capabilities to report spam -- many, many spam reports are not made in that way).

[StevenUnderwood]

Just because somebody labels a message as spam using a email tool does not mean it really is. Some stuff is obviously spam, and everybody knows that, but there is a lot that can go either way depending on the receiver attitudes I am sure we can all agree.

spam is NOT defined by the CONTENT of the message but rather by the CONSENT given to send it to that address.

[snowman]

this is what I meant: (http://email.about.com/cs/spamfightingtips/qt/et061201.htm)

Report spam with SpamCop

To submit a correct and efficient spam report using SpamCop:

1. Open the source of the junk email in your email program.

2. Highlight the full source and press Ctrl-C (Windows), Command-C (Mac) or Alt-C (Unix) to copy.

3. Paste the source of the spam you received in the SpamCop input field.

4. Press Process spam.

5. Click Send spam Report(s) Now.

[snowman]

I am used to building and deploying large secured and trusted systems. It seems to me that from personal experience and the discussion here that spamcop relies on untrusted data and possible inacurate data in a lot of cases where the site in question is not participating in the collection of bad emails.

We all know spamcop has a hair trigger and is really good at catching obvious spam. But this hair trigger is probably also its greatest weakness when it lists servers that if it had more information about the volume and quality of the outgoing email it would not.

My guess, and it is only a guess, is that if you statisticly analysed the spamcop algoritm you would find that people who participate in the collection of data about email on their servers get the best results from the spamcop blacklist. This is because you contribute to the bad email profiles.

Conversly, if you do not participate in collection of data about email you can get blacklisted for false-postives more easily.

So the quesitons remains, how does a legitemate email relay server like my example at the start of the tread send to spamcop data about the good emails it processed so when a bad one comes along it is reviewed in the proper volume context??

[dbiel]

So the quesitons remains, how does a legitemate email relay server like my example at the start of the tread send to spamcop data about the good emails it processed so when a bad one comes along it is reviewed in the proper volume context??

How do you know that they do not already have that information?

Did you bother to look as SenderBase?

And since you still refuse to post any real data, it is impossible for us to give you any real answers.

Does SenderBase reflect the traffic levels you think are being sent?

SenderBase IS where SpamCop gets its data regarding server mail traffic levels.

[Farelf]

... if you do not participate in collection of data about email you can get blacklisted for false-postives more easily.

So the quesitons remains, how does a legitemate email relay server like my example at the start of the tread send to spamcop data about the good emails it processed so when a bad one comes along it is reviewed in the proper volume context??

Interesting conjecture. I had sort of thought the IronPort reputation score would be related somehow (and the monitoring receivers are supposedly extensive enough to give a statistically reliable/objective ham:spam estimate). See the topic http://forum.spamcop.net/forums/index.php?...ost&p=43606 - see Wazoo's comments there on SenderBase and note that you can probe for IronPort reputation score readouts by emailing to the test address (one of my posts there).

You may also be interested in the Netblock reports in the stats section http://www.spamcop.net/spamstats.shtml