Extracted from http://forum.spamcop.net/forums/index.php?...indpost&p=27119

From this side of the screen, one knows not all the stuff going on from Julian's perspective, so the following is simple observation / opinion. The SpamCop parsing and reporting tool was developed by Julian for his own purposes. he then offered it up for public usage. The prime concept was to report to the source of the spam with the intent that a caring ISP would resolve the problem. As time went on, more options added, more capabilities added, more functions introduced. In the meantime, some spammers got smarter (the dumb ones giving up after having account after account cancelled by those caring ISPs)

These days you've got Julian working his magic, and you've got spammers working individually and collectively trying to defeat the SpamCop tool set. There's now enough money floating around (thanks to the gullible) that even the dumb spammers can now afford to hire knowledgable folks to work the 'net' to their own advantage. (old data, the 'net' was originally built by and for the U.S. Government, thus there was not the concept that looters and thieves would be part of the user base. Thus, the entire network was built based on all users being trusted.)

This 'current issue' is just that. Last year it was rotating DNS, the year before that it was .... on and on. Two years ago, it took weeks to get a DNS change propagated. Now, in some case, it's just a matter of minutes. Some spammers are sending spam that includes links that won't actually be activated for hours/days after the spam goes out. Some spam goes out with included links of a site that was squashed days before. Some include links that never were and never will be active. As seen in the numerous complaints about "links not reported" .. a lot of this would be discovered by minimal research. Some research done results in the URL being found active, yet that's done from a system/browser that's designed to allow some lengthy timeout variables, as compared to the parsing tool trying to handle thousands of look-ups a minute. That DNS lookups are just another bit of web traffic that can be denied by a bit of code on a server also seems to be overlooked by some folks (i.e., referrer data can be evaluated, querying IP can be evaluated, and certain items can be ignored/blocked/dropped by that DNS server) ... a bit of 'for instance' ... there's an individual in the newsgroups that makes a repeated complaint that the SpamCop reporting results that send output to a /dev/null (though still feeding the statistics table) account (due to past e-mail bouncing) must be in error, because his e-mail to that address does not bounce ... somehow not relating his use of filtering of his e-mail to an ISP's capability to also filter e-mail coming from a certain address ..???

Getting back to the above, let's go back to the beginning, at which time the focus was to shut down the spew. I don't believe that this focus has changed. The reporting of spamvertized wsb-sites was an additional capability added along the way, but it's still a secondary item of interest. There has never been anything in place to stop someone from reporting things themselves (99%+ of my spam complaints I do myself as I'm much more brutal than the SpamCop parsing/reporting tool), so it's not like the world of complaints has stopped. I can tell you that Julian is working on the codebase, that's almost a constant, but again, it's him against the numerous spammer collective out there. In example, the SpamCopDNSBL has lost a bit of 'power' based on the merging of some spammer / virus/trojan writer activity, compromising the multitudes of end-user computers to send the spew ... spammer just moves to a new compromised machine when the SCBL kicks in. The majority of those IP addresses are already found in other BLs that contain DUL (dial-up IPs) .. but once again, the reports do go out, but to ISPs that either can't, won't, or are very slow to handle the spew issue from their customer base. So the continuing levels of spew from these sources aren't a failure on SpamCop's part ...

Well, getting massive here, just hoping to toss some useful thoughts out ...

Stolen from the spamcop newsgroup;

However important anyone may think that disrupting the relationships
between the spamsites and their providers may be, realize that the
various spamfighting tools are designed for specific purposes. If you
start trying to drive a nail with a screwdriver, you're going to find
that it doesn't work as well as a hammer -- likewise some other tool
related examples.

In the case of spamcop, its parser is designed to determine spamsources
*primarily* [IMO] and secondarily do things like feed possible relays to
the relay testers for 'handling' like testing/listing and to notify
providers for spamsources and spamvertisers.

But, while it /notifies/ providers for source and spamvertiser, the
notification business is totally toothless except for the toothiness of
the provider -- that is, the result of a notify of a whitehat vs grayhat
vs blackhat vs pinkhat provider has a very wide range of outcomes --
some of which are better for the spammer/spammersupport than the
spammee/notifier.

OTOH -- besides a parser/notifier, SC is something else. SC is
maintainer of the SCbl, the blocklist of spamsources; which for various
reasons has turned out to be a very powerful blocklist. Powerful
because it is popular. Popular because it is unique in its mechanism of
listing and delisting compared to the many other db/s. So, the SCbl is
nothing to be sneezed at. It is a blocklist to be respected.

But, the SCbl is simply a blocklist of spamsources. SC doesn't make any
kind of list of spamvertisers of similar import. The only thing that SC
does with its spamvertisers is to put them on a page. It happens that
from that page, a different blocklisting service, sc-surbl 'scrapes' the
SC scraped spamvertisers and makes its own list from that. The sc-surbl
is *not* a powerful list like the SCbl; but it /is/ a list. There are
a lot of lists.

So, what all of that comes down to is that the business which SC
performs of finding the spamvertisers in the body isn't as important as
the business of SC finding the spamsource -- because the spamsource
determination feeds the SCbl, whereas the spamvertiser discoveries tends
to notify blackhat providers of things about spamcop reporters and
doesn't feed anything very potent at all.

If you want to get into taking action against the business of
spamsupport, which is what spamvertiser providers are doing, then you
will have to appreciate blocklists which put leverage against them, such
as spews and to a lesser extent spamhaus.

What spews does is spews business. What SC does is SC's business. The
two lists are very very different and SC's doesn't do anything about
spamvertisers or spam support.

--
Mike Easter
kibitzer, not SC admin

E-mail from Don ... thus now offering "something from somebody at SpamCop" ...

Date: Fri, 06 May 2005 15:26:09 -0600
To: "Wazoo"
From: SpamCop Admin
Subject: Re: Your name has been invoked <g>

Please substitute this revised text into the message that Ellen sent you
and then you can post it as evidence that you're not nuts. :-)

- Don -


There are many reasons why a URL that a sentient being can see in a spam
may not be seen by the parser. The reasons range from problems with
boundaries, mime parts, how the URL is presented in the spam when it is
seen by the parser (i.e. does it agree with the content specification) thru
creative attempts by spammers to avoid programmatic recognition of a
link. Obviously the latter leads us to try to make code changes to
accommodate those when possible.

Specifically, the parser has code to attempt to avoid non-resolution by
nameservers which recognize that the query is coming from SpamCop servers
and this code may work better at some times than others. It would not be
beneficial to go into more detail on this. Also the URL's nameservers may
be working slowly or not at all. It is not at all strange to see spam URLs
with multiple nameservers, some of which are working and some of which
aren't at any point in time. The code tries to strike a balance between
hanging on a query for an unacceptably long period of time and the failure
to resolve, but when you think about the fact that the parser is trying to
handle huge numbers of spam with a limited amount of resource then it becomes
understandable that the parser cannot wait interminably for an answer to a
query. Most browsers will wait longer than the parser. Some browsers also
accept mal-formed URLs which should never bring up a page.

It is frustrating to everyone when there is not consistent parsing of URLs,
but I am not entirely sure that there is a way to solve this problem for
every URL and every instance of parsing. Of course we do try to improve
the overall situation within the limitations that we have.

As an aside -- the number of things that go into whether a URL can be
"seen" in the spam, whether SpamCop can get it to resolve, whether some
browser will display a page, etc, is so complex that you would probably
overrun the capability of the forum disk space trying to explain it all :-)
I know it frustrates people, but I am afraid that there isn't a whole lot that
can be done. In the ideal world we would have limitless resources which
would allow for no restrictions on parses, DNS servers, sneaky code to poke
at this and to poke at that. In the real world we do what we can. And we are
skewed towards injection sources rather than URLs. Not saying that URLs are
less important but it is what it is ...

NOTE: This was not a spur-of-the-moment response <g>
Minor editing on format of a couple of lines ...
mail-formed changed to mal-formed

Thanks ever so much!

...Ooh, this looks like a very good candidate for a link from the FAQ! <g>

Was put there the day I posted the excerpts from here and the newsgroup ... and a posting is headed 'over there' to answer the call for "someone from SpamCop" to "say something" .... Look for the entry under;
SpamCop Parsing and Reporting Service
New! SpamCop reporting of spamvertized sites - some philosophy

There are some very good reasons, then, for some sort of distributed computing technique to be used.

1) There would be no way for the DNS servers to know that they were being queried by SpamCop.

2) There would be no limit to the processing power available. It would be scaled by the number of users each adding their own processing power.

3) There could be some provision for 'human readable' data to be recongnised and translated into machine recongnisable form.

4) Users could focus on issues like Spamvertised sites without preventing SpamCop from concentrating on sending sites.

Maybe others can add their ideas, too.

Paul

Yes, it might help, but it's probably not likely to happen soon.

BTW, I referenced this Topic in my FAQ Entry: The Link Analysis Process.

I am learning slowly as I get into this reporting of Spam business.

Despite all the efforts being made by a growing number of organisations the amount of Spam is increasing rapidly.

The resources of Spammers is growing. There is now an established infrastructure that supports Spammers. Viz:

* Tens of thousands of compromised computers that can be used to relay Spam.
* ISPs who are willing to turn a blind eye to spammers activities, for large payments.
* Domain name suppliers who have contracted out the supply of names and their subies are making some fast bucks by selling an endless supply of Domain names to Spammers and to Spamvertised sites as re-directors.
* have I missed some item(s)?

It seems to me that, although the Internet was built originally on trust - everyone is trusted until they prove to be untrustworthy - it is now time to turn that around to the way normal business is done. [Imagine if anyone could open a bank account without a credit check or proper identification!] That is, no one should be trusted unless there is good reason to trust them and they can be held accountable if they fail to measure up to the level of trust placed in them.

In effect, instead of maintaining Black Lists, what we need to have is White Lists. ISPs, DNS servers, Web servers, etc. would need to apply to be on the White List. When they apply they would have to agree to certain policies, rules and procedures and they would be accountable if they failed to keep to this agreement. Rogue ISPs would find themselves outside of the Internet Community and would have to work to get themselves let back in rather than the community having to spend billions of dollars trying to keep then out.

Along with this change of distrust until trust is earned, there would need to be mechanisms for authentication. This would mean an overhead in terms of packets sent, constantly and permanently, however it would not be as onerous as the present burden of Spam.

The authentication bit is being worked on currently, but the change of 'trust until found untrustworthy' to 'trust when found trustworthy' does not yet seem to be much in focus. Yet this is the key to stopping Spam and a lot of other abuses on the Internet.

Paul

The only "fact" I am not sure of is the large payments, please provide your sources.

And there are many "trust" efforts our there. SenderBase by Ironport (they also currently own the parsing and reporting side of Spamcop) is one of them. Microsoft has made a proposal as have others.

A major problem I see with any vast change in the way email is handled throughout the world is getting the billions of legitimate email servers in use right now to become compliant. Spammers have shown the capability to adapt much quicker to new methods to stop them, making them look like the valid senders.

The ISPs who are making money by hosting spammer sites and not listening to reports are more or less 'outside' already by being blocked.

The problem of compromised computers is different. ISPs and online merchants want people to have access to the Internet and they don't want to make it difficult for the ordinary end user. OTOH, it is expensive to hand hold lots of customers who don't know what they are doing and so get infected.

There have been suggestions of licensing users so that they, at least, have knowledge of basic safe practices. However, the cost of licensing would drive up the cost of connecting as well as getting not being consistent.

The basic problem is that if you turn the trusted assumption around, you lose the unique quality of the internet which is open communication and commerce without any censorship and with cooperative regulation (netiquette). Spam is really a small price to pay for the possibilities of the Internet.

If ISPs utilized trade associations, they could form just the kind of system that you have described - only with a positive basis. They could support blocklists and advertise the advantages, educate end users, and have a whitelist system for members - and even with competing trade associations.

My solution is one that the techies say won't work, but IMHO, the major problem is the one that blocklists encountered when they were first used: it would take too much education of all the server admins in the world what the criteria were.

Bulk email is the problem - both for the recipients who have not asked for it and for the bandwidth it consumes. It is possible to indicate in the headers that this is a bulk email. My solution is that bulk email is always blocked unless whitelisted by the end user. It would be one more step in the confirmation process already used (and is suggested by many bulk emailers already). Any mail without a bulk email tag in the headers would be accepted. If it is unsolicited, the end user would report it to his ISP who would determine that it was bulk (from a system akin to nanas) and that IP address would go on a blocklist similar to spamcop's that is automatic and will drop off when no longer reported. ISPs can keep their customers from sending illegitimate bulk email by monitoring how many emails are being sent. Those who want to send a lot of emails would have to have a special account and be required to use a program that inserts the bulk email tag to send their emails.

Those who want to receive all bulk email would have to pay more for an account and for the filters to separate out the spam. Since that is usually businesses, it would just be part of the cost of doing business. The average end user would be glad to have the cheaper account and have all their bulk email (except what they have whitelisted) blocked and be able to receive unsolicited email from individuals like longlost cousins. The ISPs who get blocked for allowing users to send bulk email without the tag would not be scroundrels, but incompetent and irresponsible. And that gets rid of the 'innocents' who get their email blocked. It puts the responsibility for choosing a competent provider on the sender and also gives the sender an incentive to learn how to be a competent internet user (firewalls, anti-virus, etc.) because of the cheaper prices a competent user would get for connectivity.

My $.02 USD

Miss Betsy

POP QUIZ:
... let's see how well you understand the content of this thread:

FIRST TOSS-UP:

Your kid's playground is surrounded by dark shadowy figures in
trench coats giving away drugs and prono -- trying to steal their
the identities to break into your bank accounts.

What do you do?

ANSWER: (pick one)

[_] Punish the kids... they should protect themselves better
[_] Punish the school... they should build better fences
[_] Close the playground... kids have no business there anyway
[_] Lock up the dark shadowy figures in trench coats

OKAY... ROUND 2

There's a local terrorist organization who rents trucks to deliver bombs
to the subway and various government buildings.
WHAT DO YOU DO?

ANSWERS: (pick one)

[_] Send complaints to the truck rental agency?
[_] Lock up the trucks
[_] Close the streets leading to the bombed targets
[_] Make the bombed targets buy better security
[_] Lock up the persons who rented the truck?

SUMMARY:

How come your answers don't match the thread?

Is something wrong with this scenario?

Just asking.

In the case of this thread, the dark shadowy figures are no where in sight. Most of the spamvertized sites are hosted on victims machines and those victims only know that their machine is running slow.

It is perfectly legal to rent trucks and host web sites. IANAL, but I believe unless it is known in advance what the trucks would be used for, there is no liability or reason for suspect.

It can also be perfectly legal to run a web site, even for selling drugs or whatever, as long as you are not breaking any laws doing it. It is also perfectly legal to send out emails advertizing that site. The only issue with spamvertizing is how those emails are directed. If you blanket the community, you are spaming. If the emails are directed to people who have asked for them, they are not even spam, therefore no spamvertizing has happened.

to expand on SU's post:

And in most countries of the world, there are no laws against spam. There are laws against fraud, etc., but in spite of that, I still occasionally get a 419 or lottery scam in snail mail - the lottery ones are usually mailed from abroad. So, even if law enforcement did lock up criminals who send spam and maintain websites that are fraudelent, there would still be a need to block those who escape prosecution.

One of the reasons that many people prefer blocking spam (and not caring about spamvertised sites except to use on blocklists) is that the specter of censorship is worse than the spam. If email and websites were censored by anyone other than the receiver or person browsing, then there would be a much greater problem in many people's minds.

In the offline world, one can't simply say I don't want to be a victim and have it hold because criminals can physically force their way into your house or physically assault you. Online, one can say No without fear of being forced to accept spam, whether criminal or not. One does not have to visit websites that are questionable to one's morals. If my neighbor wants to open spam and order from it or visit websites I think are immoral, then that's his business. If he wants to email me, though, he had better do it from an IP address that does not tolerate spam.

Miss Betsy

Just a slight nit-pick
Undoubtedly true, though the "list" is changing all the time. The "spam industry" is much more than the individual who originates/initiates the spam but even within any expanded definition of spam-complicit activity you may be surprised (and it is supremely ironic) to see that all the major players do have (anti)spam laws - ref Anti-Spam Laws (not all are listed in the same place on the page) including:

• USA
• Many US States
• Brazil
• China
• Israel
• Canada
• Russia
• Germany
• France
• United Kingdom
• Netherlands
• Spain
• Italy
• (European Union generally)
• Romania

Just to list the main ones I see in whatever guise - originator, originator's host, "business", website or website host. India is about the only significant absentee, from my point of reference. These (laws) have some effect within their jurisdictions (not as much as most "here" would wish) but there are sovereignty, jurisidictional (obviously), evidencial and procedural issues in the international arena - where spam actually operates.

There is a tendency (which I mostly see in the UK) for TOS/AUPs to prohibit activity which is illegal in the recipient's country. Perhaps this is common to EU members. Anyway, it points to a mechanism which extends the scope of the laws and the effect of the laws to a possibly useful extent. (And, perhaps, to the provision of a source of income for lawyers and their descendents for all of the forseeable future but I prefer not to think about that.)

As said, a nit-pick, but something to be aware of IMO. FWIW I too see blocking as the here and now solution for the terminally bedevilled - with some ISPs needing to provide a lot more outwards blocking (but not on my email, thanks ).

And a nitpick back: I said 'most' of the countries - not limiting it to developed countries or to 'significant' ones.

I thought about it when I wrote that sentence and thought I was probably safe because there are a lot of countries in the world - some may not have enough computers, etc. to be significantly part of the online scene, but it is only a matter of time - and perhaps, a matter of time until there are laws in 'most' countries.

However, the laws that are there seem to be only effective in getting ISPs to include those prohibitions in their TOS/AUP - if they haven't already - since to be reliable and not get on blocklists, they need to have control over spammers on their networks.


and that's the rub - even though IMHO, the sender should be the one to take care of any problems resulting from blocklists, I have enough correspondents who are clueless that I shudder to think about trying to educate them if my ISP did use blocklists the way I think they should be used!

There have been numerous suggestions, including SPF, licensing, and other approaches, all of which would work at /reducing/ spam and creating two networks - one mostly spamfree with no restrictions for 'trusted' senders and one where anything goes - IF most people would use them. However, most people who are server admins are highly independent (my server, my rules) and not much interested in persuading others to do the 'right' thing. Few others understand how it all works enough to convince the server admins that his rules for his server should be like others to make things work better. Look at all the admins who argue about accepting email and then sending an email non-delivery notice. In fact, spamcop admins made the same arguments when I suggested that it was time to stop using that method because spam had spoiled it. The /only/ reason that server admins don't continue is because other admins are blocking them.

And, blocking is 'natural' to the internet where netiquette reigns. Etiquette ostracizes; it doesn't force the way legislation does.

Miss Betsy