IPv6 Routing Support

[StevenUnderwood]

The "Right" way of doing this would have been to add the IPv6 detection code, and have it log to a file for debugging. This doesn't change the existing behavior from the user perspective.

Since you know the spamcop code so well, why don't you write your own parser with IPv6 working and we can all use your code and leave SPamCop behind.

Sorry, but I am tired of everyone coming in here "knowing" how easy it would be to fix this problem when they have no idea what the code looks like, code which was self-described by the original developer as "spaghetti".

[turetzsr]

...Steven, while I share your frustration and agree with your point, I think it may be misplaced to direct it at techie. IIUC, techie is making a suggestion about how to code for IPv6 in general, not how specifically to do it in SpamCop and has made no public judgment about whether the time it has taken is appropriate. On the other hand, techie's "The wrong way is to simply add a check for IPv6, and puke if found, which is what currently happens, as of the update in March 2011" comment seems to assume something that isn't true -- that the current code was intended to address IPv6, which (IIUC) it was not!

[gnarlymarley]

The "Right" way of doing this would have been to add the IPv6 detection code, and have it log to a file for debugging. This doesn't change the existing behavior from the user perspective.

The detection code should include checks for the IPv6 loopback address [::1], and treat it the same as 127.0.0.1.

We should face the facts that IPv6 is not human, nor code friendly. IPv4 was easy to code for because it HAD three periods. IPv6 can have any number of colons, but not more than eight. Code that matches IPv6 will always be complex and never as simple as IPv4, as seen below. I suspect this is partly why SpamCop has not fully implemented it yet.

m/^([0-9A-Fa-f]{1,4}:){1,7}([0-9A-Fa-f]{1,4}){1,1}|([0-9A-Fa-f]{1,4}:){1,6}(:[0-9A-Fa-f]{1,4}){1,1}|([0-9A-Fa-f]{1,4}:){1,5}(:[0-9A-Fa-f]{1,4}){1,2}|([0-9A-Fa-f]{1,4}:){1,4}(:[0-9A-Fa-f]{1,4}){1,3}|([0-9A-Fa-f]{1,4}:){1,3}(:[0-9A-Fa-f]{1,4}){1,4}|([0-9A-Fa-f]{1,4}:){1,2}(:[0-9A-Fa-f]{1,4}){1,5}|([0-9A-Fa-f]{1,4}:){1,1}(:[0-9A-Fa-f]{1,4}){1,6}$/

...Steven, while I share your frustration and agree with your point, I think it may be misplaced to direct it at techie. IIUC, techie is making a suggestion about how to code for IPv6 in general, not how specifically to do it in SpamCop and has made no public judgment about whether the time it has taken is appropriate. On the other hand, techie's "The wrong way is to simply add a check for IPv6, and puke if found, which is what currently happens, as of the update in March 2011" comment seems to assume something that isn't true -- that the current code was intended to address IPv6, which (IIUC) it was not!

These are some generic questions, but I believe they get to the root of the problem. How do you go about reporting IPv6 to the upsteam, which is to say the ISP? How do you add the IPv6 address to a block list?

I believe the main reasons for the delay is that they do not need to just match IPv6, but they also need to get other underlying code updated as well. We know that they are able to find the IPv6 address now, because the page says it found IPv6 and stops. I believe that SpamCop is working on the whois, reverse DNS, blacklisting servers, and also working with the abuse.net DB to get all of it IPv6 compatible. SpamCop needs to get all of their code updated so it handles IPv6 in all of the code, not just the detector portion.

Edited April 10, 2012 by gnarlymarley

[A.J.Mechelynck]

We should face the facts that IPv6 is not human, nor code friendly. IPv4 was easy to code for because it HAD three periods. IPv6 can have any number of colons, but not more than eight. Code that matches IPv6 will always be complex and never as simple as IPv4, as seen below. I suspect this is partly why SpamCop has not fully implemented it yet.

m/^([0-9A-Fa-f]{1,4}:){1,7}([0-9A-Fa-f]{1,4}){1,1}|([0-9A-Fa-f]{1,4}:){1,6}(:[0-9A-Fa-f]{1,4}){1,1}|([0-9A-Fa-f]{1,4}:){1,5}(:[0-9A-Fa-f]{1,4}){1,2}|([0-9A-Fa-f]{1,4}:){1,4}(:[0-9A-Fa-f]{1,4}){1,3}|([0-9A-Fa-f]{1,4}:){1,3}(:[0-9A-Fa-f]{1,4}){1,4}|([0-9A-Fa-f]{1,4}:){1,2}(:[0-9A-Fa-f]{1,4}){1,5}|([0-9A-Fa-f]{1,4}:){1,1}(:[0-9A-Fa-f]{1,4}){1,6}$/

These are some generic questions, but I believe they get to the root of the problem. How do you go about reporting IPv6 to the upsteam, which is to say the ISP? How do you add the IPv6 address to a block list?

I believe the main reasons for the delay is that they do not need to just match IPv6, but they also need to get other underlying code updated as well. We know that they are able to find the IPv6 address now, because the page says it found IPv6 and stops. I believe that SpamCop is working on the whois, reverse DNS, blacklisting servers, and also working with the abuse.net DB to get all of it IPv6 compatible. SpamCop needs to get all of their code updated so it handles IPv6 in all of the code, not just the detector portion.

Yes, and considering all the ways to abbreviate an IPv6 address, just converting the address-as-text (as found in the mail headers) to the address-as-128-bits (the unique value which can be meaningfully compared for equality, and so blacklisted or not) is an unobvious task, albeit well-defined.

At least now (and unlike what happened when the first IPv6 spam appeared), when SpamCop chokes on an IPv6 spam it gives a clear message that it currently doesn't support IPv6. One could always wish for full IPv6 support; but let's not forget that even after the purchase by IronPort and the latter's purchase by Cisco, SpamCop is still cruelly understaffed and, AFAIK, the only person who more or less masters the SpamCop source is Julian, and I'm not sure how much time he can dedicate to finding (in the “mass of spaghetti†mentioned a few posts ago, and without neglecting his other duties) the relevant code for this particular problem (which probably is neither the only problem nor the most urgent one), fixing it, testing it, checking for side-effects, etc. etc. etc.

I don't know about y'all, but the IPv6 spam messages that fall into my inbox are (still) few and far between; I believe that we still have several years before they become the majority, and I'm confident that the SpamCop code will be suitably modified in time to handle them satisfactorily before they become an unbearable nuisance.

[turetzsr]

...Latest update: Scheduled Service Outage - Thursday May 10, 2012.

[gnarlymarley]

...Latest update: Scheduled Service Outage - Thursday May 10, 2012.

Good news to all the IPv6 folks out there.

http://www.spamcop.net/sc?id=z5324131362z4...47234ee1c0da1fz

There is some stuff still lacking like this tracking link I found couldn't find a person to report to, but that should be a minor fix. It is something about "Cannot find ip range in whois output".

turetzsr, I think you have connections with Julian. Can you pass on the congratulations?

[turetzsr]

<snip>

turetzsr, I think you have connections with Julian. Can you pass on the congratulations?

...Actually, no, everyone here has the same connections with SpamCop staff that I have -- you can just send an e-mail to deputies[at]admin.spamcop.net and a SpamCop deputy will (hopefully) let the relevant folks know.

[Farelf]

Looks like work continues - gnarlymarley's link just at the moment has the "Unable to process header. IPv6 addresses are not supported.

No source IP address found, cannot proceed." lines.

<!-- SpamCop::Web::Look $Revision: #15 $ produced by prod-sc-www1 -->

See http://forum.spamcop.net/forums/index.php?...amp;#entry80735

The analysis should find (I guess):

2a02:8420:4e46:2e00:211:32ff:fe11:32f5

tracks to SFR-BROADBAND-USER (FR)

inet6num: 2a02:8420::/32

...

remarks: ***********************************

remarks: * Abuse e-mail: abuse[at]gaoland.net *

remarks: ***********************************

...

role: LDCOM Legal Contact

...

abuse-mailbox: abuse[at]gaoland.net

...

role: LDCOM Networks Tech Contact

...

abuse-mailbox: abuse[at]gaoland.net

...

% This query was served by the RIPE Database Query Service version 1.8.13 (WHOIS2)

[A.J.Mechelynck]

Hm, yes, trying to re-parse with that same tracking URL gives me the same errors, but SC also tells me that "reports have already been sent" — albeit to nomaster[at]devnull.spamcop.net — concerning that IPv6 address.

Parsing header:

Received: from [2a02:8420:4e46:2e00:211:32ff:fe11:32f5] (helo=favre.fr) by kaysville.yaritz.net with esmtp (Exim 4.66 (FreeBSD)) (envelope-from <admin[at]megaleecher.net>) id 1SSIGO-0007ol-Dg for me[at]yaritz.net; Wed, 09 May 2012 19:33:29 -0600

Unable to process header. IPv6 addresses are not supported.

No source IP address found, cannot proceed.

Add/edit your mailhost configuration

Finding full email headers

Submitting spam via email (may work better)

Example: What spam headers should look like

Reports regarding this spam have already been sent:

Re: 2a02:8420:4e46:2e00:211:32ff:fe11:32f5 (Administrator of network where email originates)

Reportid: 5765414954 To: nomaster[at]devnull.spamcop.net

^{______________________________________________________________________}

Report another spam?

[Farelf]

Yes Tony - trial, some IPv6 spam parsed, bugs to be fixed so new version retracted, fixed version to be tried next week as advised in that other post by email_support. There was an opportunity while the new version was operating to add a report (if a paying user) to the correct abuse desk, if you were able to determine it. I use IPNetInfo.exe with the "Detect IPv6 addresses" box checked for those determinations since I am congenitally confused when it comes to using RIPE lookups unaided

Steve

[linux__rulez]

Adding another e-mail address to my account I discovered this error:

Sorry, SpamCop has encountered errors:

Headers not found.

It appears you did not provide all email headers. Please consult this FAQ

for more information on getting full headers from your software, then try

again.

Could this also be an effect of IPv6 addresses in the headers, because I'm sure I've included all headers.

I hope the IPv6 routing support will also include support for setting up e-mail addresses and Mail eXchanges that have and therefor by default use IPv6.

[Farelf]

Adding another e-mail address to my account I discovered this error:...

You were amending your mailhosting on a reporting account? Write to SpamCop Admin (Don D'Minion) with all the details:

service[at]admin.spamcop.net

He will know the answer and may(??) be able to help sort you out.

[turetzsr]

...Latest announcement: Scheduled Service Outage - Tuesday August 7, 2012.

[turetzsr]

...Bad news, folks! Steve's (Farelf) latest post in SpamCop Forum topic "Scheduled Service Outage - Tuesday August 7, 2012."

[Sven Golly]

I am seeing more and more IPv6 addresses -- bogus and real -- in my spam submissions.

[turetzsr]

Hi, Sven,

...Seems that for now you'll have to either just delete them or manually report them.

[techie]

...Steven, while I share your frustration and agree with your point, I think it may be misplaced to direct it at techie. IIUC, techie is making a suggestion about how to code for IPv6 in general, not how specifically to do it in SpamCop and has made no public judgment about whether the time it has taken is appropriate. On the other hand, techie's "The wrong way is to simply add a check for IPv6, and puke if found, which is what currently happens, as of the update in March 2011" comment seems to assume something that isn't true -- that the current code was intended to address IPv6, which (IIUC) it was not!

IIRC, the Spamcop update of March 2011 was advertised as providing for future support for IPv6.

Immediately after this update, I started receiving bounces for spam containing IPv6 loopback addresses, which had never been rejected prior to this update. I went back through my spam archives, and verified that I had submitted spam containing IPv6 loopback addresses several time during the previous month, none of which had been rejected.

My assumption, based on observed Spamcop behavior before and after that update, is that code to detect IPv6 was added during that update, and that instead of using that code to gather data, or as a step-by-step sanity check, it was instead used as a cop-out.

My comments on the "right way" and the "wrong way" are based on one assumption about the parsing process, namely that the process works from the recipient back towards the sender.

This is the logical method, since Spamcop needs to eliminate hosts within your MX path, so as not to report your receiving server as the spam source, which of course it is not.

The host immediately upstream of your receiving server is the last host which can be properly verified, since all headers upstream of that point can be forged. If Spamcop trusts the header upstream of that point, it is only because Spamcop trusts the upstream ISP (such as Yahoo/Google/AOL/etc), and has decided that the internal headers added by those ISP's are trustworthy.

Given that the parsing process needs to work backwards, it is logical that sanity check for IPv6 headers should be done at each step in the parsing process, which would eliminate all of the problems with IPv6 loopback addresses in the first hop, forged IPv6 headers, IPv6 tunnels and 6-to-4 gateways prior to the upstream server. This would not eliminate the problems with IPv6 use at the receiving site, or actual IPv6 transit between the receiving server and the host immediately upstream, but it would eliminate the vast majority of the current set of problems, since the spam source would be reached, and parsing terminated, prior to an IPv6 address being encountered.

Instead, what I have observed is that the parser looks for IPv6 addresses anywhere in the header as the first step, and pukes if any IPv6 addresses are found, instead of actually attempting to parse the headers in proper sequence, and only puking if an IPv6 address is encountered prior to determining the spam source.

Edited August 13, 2012 by techie

[turetzsr]

<snip>

My comments on the "right way" and the "wrong way" are based on one assumption about the parsing process, namely that the process works from the recipient back towards the sender.

<snip>

...The assumption is, from what I understand, correct. What we don't know is whether that fact allows us to conclude that it is necessarily feasible and "easy" for the parser to ignore all IPv6 addresses that appear prior to the point it decides that it has found the location at which it can stop trusting the headers. I think that is not an unreasonable conclusion but it is, nevertheless, a conclusion the validity of which we will never know because the SpamCop engineers won't ever reveal the answer to us. And since it's their product, they get to define the rules, so I for one feel unworthy to challenge it and can't imagine that anyone else outside the SpamCop staff circle is able to do so for certain, either.

[techie]

...The assumption is, from what I understand, correct. What we don't know is whether that fact allows us to conclude that it is necessarily feasible and "easy" for the parser to ignore all IPv6 addresses that appear prior to the point it decides that it has found the location at which it can stop trusting the headers. I think that is not an unreasonable conclusion but it is, nevertheless, a conclusion the validity of which we will never know because the SpamCop engineers won't ever reveal the answer to us. And since it's their product, they get to define the rules, so I for one feel unworthy to challenge it and can't imagine that anyone else outside the SpamCop staff circle is able to do so for certain, either.

Well, if we also assume two other things, namely that the IPv6 detection code is written as a subroutine, and that the parsing process works on the headers one hop at a time, then the real question is: why did they call the subroutine at the beginning, instead of as a conditional during the parsing process?

[mrmaxx]

Anyone know the ETA on IP V6 reporting? Even guesstimate would be nice. I have a couple nice, juicy spams waiting to report, but SC wont' accept them because they are IP V6.

[Farelf]

I guess if I was getting a number of them I would be tempted to manually report - but, in general, that would be an irresponsible thing to recommend to the general membership and, with most such reports ignored (and some maybe ending up in the hands of the spammers), fairly pointless or dangerous. Yes, it will be great to get the ease and security of the parser and reporting system in train on these, and the leverage of the SCbl behind it.

Can't help but reflect that some of what was learned in the last attempt will be forgotten with the passage of time - nobody documents so well that it could be avoided. Oh well, obviously that will be more than offset by what the engineers' community has learned in the interim, when the next attempt succeeds - as it must.

[SpamCopAdmin]

The next SpamCop version supports IPv6 parsing.

The release should be out within the next 30 days. No promises. It could easily be 60 days, or even 90 days, but we're hopeful that it will be sooner rather than later.

- Don D'Optimistic - SpamCop Admin -

- Service[at]Admin.SpamCop.net -

.

[keef]

The release should be out within the next 30 days.

Any update on this please?

"Here is your TRACKING URL - it may be saved for future reference:

http://www.spamcop.net/sc?id=z5429514652z9...53d6963524de78z

No source IP address found, cannot proceed."

[keef]

Here is your TRACKING URL - it may be saved for future reference:

http://www.spamcop.net/sc?id=z5431077784z4...60117291d5a987z

No source IP address found, cannot proceed.

[keef]

Here is your TRACKING URL - it may be saved for future reference:

http://www.spamcop.net/sc?id=z5431383623z3...7d46855936f863z

No source IP address found, cannot proceed.