URL Shrinking Website != Spam

[TheLeggett]

This morning, I was threatened to be put onto SpamCop's BL.

I hate spam just as much as the next guy. I have nothing to do with sending out spam related emails or anything of that nature. This is all beside the point though.

I run a very small website which performs a simple, free, service. Users can access the site, enter in their URL, and shorten it. Essentially, it acts as a URL redirect platform to make long URL's much shorter.

ie:

http://www.somesite.com/path/to/the/page/somepage.html

becomes

http://mysite.com/page/

If you're familiar with TinyURL, you already know what this service does. My site is one of hundreds of similar websites, does nothing unique, and I honestly don't really care if I end up tossing it to protect my IP's. It's just a simple, experimental project I threw together.

Why is it MY responsibility to go in and delete redirects to sites that SpamCop considers spam. What have I DONE wrong? Is it then my responsibility to manually check every link on one of my community message boards, and make sure that none of those links are misleading? Is it my responsibility to manually check the 2000 rows of URL redirects to make sure that none of them lead to websites I consider to be spam?

Even if I am to delete the bad redirects in the database, what's to stop them from simply making another one? It literally takes 5 seconds. And for their bad intentions, I am the one being accused of spam, just for offering a quick, free service with good intentions.

Note: I come off sounding a bit edgy about all of this, but these are real questions and concerns of mine. If there is something I need to know, by all means, please correct me.

[rconner]

This morning, I was threatened to be put onto SpamCop's BL.

My understanding (strictly as a SpamCop user with no inside knowledge) is that you get on the SCBL by sending lots spam mail, not by hosting websites. Are you sure that the message THREATENED to put you on a BLOCKING LIST, or did it just notify you that a URL in your domain was used in spam? Would you care to post some or all of that message so that we could see what you're talking about?

Why is it MY responsibility to go in and delete redirects to sites that SpamCop considers spam. What have I DONE wrong?

I appreaciate your public-spirited offer to shorten everyone's URLs, but you surely must know that by offering a public service that can be abused for profit by spammers and other crooks, you bear a responsiblity to do your part to keep that abuse under control. Do you really enjoy being considered a spammer by people who open mailings and find your URL inside them?

Is it then my responsibility to manually check every link on one of my community message boards, and make sure that none of those links are misleading? Is it my responsibility to manually check the 2000 rows of URL redirects to make sure that none of them lead to websites I consider to be spam?

I don't think it would be fair to require you to break links proactively, without knowing whether they are actually going to appear in spam. However, if you do receive reports from SpamCop (or elsewhere) indicating that a particular link is showing up in spam, what's the big deal about breaking it? You can furthermore post a policy statement indicating that you will break links used in spam (your competitors at TinyURL do).

Even if I am to delete the bad redirects in the database, what's to stop them from simply making another one? It literally takes 5 seconds. And for their bad intentions, I am the one being accused of spam, just for offering a quick, free service with good intentions.

What I'd like to know is why you would persist in this if you know your service is being abused? If I leave the keys in my car, I should not be surprised if someone takes it for a joyride, and complaining that I wasn't responsible for the theft isn't likely to get much traction. If I buy a pit bull and let it roam the neighborhood and bite people, the authorities won't be impressed with my argument that I should not be required to keep it on a leash and under my control.

-- rick

[agsteele]

Why is it MY responsibility to go in and delete redirects to sites that SpamCop considers spam. What have I DONE wrong?

As Rick noted, the SpamCop reporting service tells 'owners' of URLs when theirs is being quoted in spam so that they can, if they desire, remove the offending message, improve their practice in relation to spamming or advise a customer who is spamming.

If you don't wish to receive these advisory Emails regarding so-called spamvertised URLs then you can request that they be disabled.

The only time you will be listed is if spam is being sent through your mail server.

So, as Rick observed, you need to be certain what the warning message actually said. If spam is being sent from your IP then you will want to find out why and take action. If, though, you are just receiving these courtesy messages you can have them stopped or route them to trash if they are not helpful.

Andrew

[Miss Betsy]

Spamcop's philosophy is that by identifying the source of the spam - the IP address from which the email is sent (which would be your mail server if that is the message you got) so that server admins can either tag (recommended) or block emails from that server so that they do not receive spam in their inbox - is the best way to control spam.

Other people think that by shutting down the website (or redirect) spam will be stopped. In the beginning, spamcop notified the website that was being advertised in the spam because many website owners didn't understand that sending unsolicited email was not a good idea. Now there aren't very many legitimate website owners who send unsolicited email to advertise their product. Spamcop continues to identify spamvertized sites, however, for those who think that sites advertising via spam should be shut down and also, for those who tag or block email by those sites in the email. One server admin estimated that he caught 25% of the spam by identifying the website it was advertising. I don't quite know how that works, but if your redirect is often redirecting to a spam site, it might be possible that those who tag spam by website will tag all email with your redirects as spam if it often does redirect to a spam site.

Since most spammers are now either criminals, or just this side of the law, you are aiding and abetting them in either criminal or unethical activities by allowing them to use your redirect when you have been informed by spamcop that it is linked to a spam message. You do have the responsibility to make sure that it really does go to a spam site before you break the link. The spamcop message is simply advisory.

Spamcop adheres to netiquette in that it does not dictate to you what you 'should' do. If you are not interested in spamcop reports, then you can request spamcop not to send them. However, you can't stop spamcop from identifying your redirect in spam email for those who use spamvertized sites to identify spammers.

Miss Betsy

[Merlyn]

Well if you do not care if spammers abuse your site for their junk then no one cares if you get your site unplugged. Pick a less abusive business model.

BTW: Tiny URL is very proactive about removing those spam links.

[Telarin]

You might want to do a search of these forums for shorten.ws

The owner of that site, also a URL shortening service, has a similar problem a while back. He was able to setup an automated system where any complaints about his URLs were forwarded to him, and automatically processed. It was win-win, because it gave him an automated means of removing spammed URLs, and it meant that complaints about spammed URLs were acted on much more quickly than would be possible by having an abuse desk handle each one manually.

[showker]

Also, be further advised:

Domain length shortening services have come under the scrutiny of many "public" entities like libraries and public schools.

In an effort to curb the use of "TinyURL" (and others) being used to send and spread porno content, many public school systems merely block all TinyURL domains.

How did we first discover this?

One of our sister sites sends an email technology newsletter each Monday morning. (have since 1994) However we began getting lots and lots of complaints from subscribers that the "Links don't work"

We had begun using TinyURL as a method of shortening rediculously long domains into managable lengths so they wouldn't get truncated in email.

Turns out the readers who's links stoped working were receiving the newsletter through a system which had begun blocking all TinyURL domains.

One administrator said:

"We started blocking after we discovered the kids were using TinyURL to mask game, pirate software and porno site links in their forums and web pages ... strictly against our policy -- so we block."

Food for thought.

:-)

[rconner]

One administrator said:

"We started blocking after we discovered the kids were using TinyURL to mask game, pirate software and porno site links in their forums and web pages ... strictly against our policy -- so we block."

Seems unfortunate, since tinyURL seems to be one service that shows some diligence in routing out spam links. Indeed, I get very little spam personally that contains tinyURL links anymore. I do more frequently get spam using other proxies that have either been set up just for spam purposes, or may have been set up by well-intentioned folks like our original poster.

-- rick

[snover]

Is there a reason that SpamCop doesn’t make HEAD requests against the spamvertised URLs to ensure that the original host also gets informed of the spam instead of just the redirection service?

I’ve been receiving “Critical Microsoft Update†spam all day that use king.cd to redirect to rapidshare.com, but SpamCop only reports about king.cd (and Comcast doesn’t seem to be liking the reports; I keep getting form responses that they need more information).

(Refs:)

http://members.spamcop.net/sc?id=z31906593...19d5a53b7dd242z

http://members.spamcop.net/sc?id=z31922196...3b5bef2f4dc5ccz

http://members.spamcop.net/sc?id=z31922212...b316f04b65e337z

http://members.spamcop.net/sc?id=z31931161...33c440a5eff2bbz

http://members.spamcop.net/sc?id=z31931828...0f3d872a0b0120z

[turetzsr]

Is there a reason that SpamCop doesn’t make HEAD requests against the spamvertised URLs to ensure that the original host also gets informed of the spam instead of just the redirection service?

<snip>

...That isn't the service SpamCop sees itself as providing (although it does make some effort to identify sources of spamvertizing). See SpamCop FAQ (link near upper left of each SpamCop Forum page) link labeled "SpamCop reporting of spamvertized sites - some philosophy," especially "linear" post number three (Wazoo quoting Don, aka "SpamCop Admin").

Edit: Corrected text from "See SpamCop Forum ..." to "See SpamCop FAQ ...."

[Wazoo]

Is there a reason that SpamCop doesn’t make HEAD requests against the spamvertised URLs to ensure that the original host also gets informed of the spam instead of just the redirection service?

In addition to Steven's reply, please also check out the graphic/link at the top right of this page. The system is already kicking out approximately 80 reports-a-second .. which doesn't include all the traffic from input that doesn't get a Report, i.e., spamtraps .... let's not even start with the DNS/Resolver games played by so many spammers these days. The alternate question might be .. just how much would you like to donate to cover the costs of the additional programming, resources, and bandwidth needed to carry the load of additional tasks like these?

[snover]

See SpamCop Forum (link near upper left of each SpamCop Forum page) link labeled "SpamCop reporting of spamvertized sites - some philosophy," especially "linear" post number three (Wazoo quoting Don, aka "SpamCop Admin").

I don’t know if it’s really worth mentioning or not but I had a really, really hard time finding that page. I couldn’t find it by following your instructions; even doing a search for the word “spamvertized†only got me there thanks to a link in “FAQ Entry: The Link Analysis Processâ€. (The danger of “important†topics, especially when not organised really well, is that as you keep adding things, eventually there is too much and the relevant bits get lost. )

The alternate question might be .. just how much would you like to donate to cover the costs of the additional programming, resources, and bandwidth needed to carry the load of additional tasks like these?

Well, in the very specific case of resolving domain shortening services, it’s fortunately not a particularly vexing problem to solve, at least from a programming perspective.

The spam is already being processed for links. There will only need to be a couple of extra steps.

1. Match the link domain against a list of known shortening domains

2. Optionally, opportunistically check an unknown domain to see if it is a URL shortening service if the domain part is <= 9 characters or so — this will help so that nobody has to manually add domains to the list, but it may add too much overhead or garbage in the lookup table, slowing things down

3. Do a HEAD request and cache the resulting Location header

Unlike the problem of rotating DNS records, shortening service URLs are idempotent, so they really only need to be looked up once. Obviously, of course, this proposed method won’t catch ANY redirect, but it ought to take care of most of the shortening services.

Anyway — I completely understand if it’s not something that wants to be implemented, as turetzsr said. But it was a thought.

[Farelf]

...The spam is already being processed for links. There will only need to be a couple of extra steps.

1. Match the link domain against a list of known shortening domains

2. Optionally, opportunistically check an unknown domain to see if it is a URL shortening service if the domain part is <= 9 characters or so — this will help so that nobody has to manually add domains to the list, but it may add too much overhead or garbage in the lookup table, slowing things down

3. Do a HEAD request and cache the resulting Location header

Unlike the problem of rotating DNS records, shortening service URLs are idempotent, so they really only need to be looked up once. Obviously, of course, this proposed method won’t catch ANY redirect, but it ought to take care of most of the shortening services.

Anyway — I completely understand if it’s not something that wants to be implemented, as turetzsr said. But it was a thought.

And not a bad thought to be adding in as much detail in a new topic to the New Feature Request forum here IMO. Not that there's any evidence of SC noting/looking for inspiration in that forum but posting there will at least open up any further discussion on the topic without going too O/T in this one and will probably prove easier to find/research/resurrect 'there' than 'here' as time goes by, in this huge Lounge area (you've already experienced the difficulty of locating past discussion/data on these boards). Yeah, I would be quite pessimistic about SC picking it up right now but no way to guess how much use might be made of such suggestions in the future, within SC or even elsewhere.

[turetzsr]

See SpamCop Forum (link near upper left of each SpamCop Forum page) link labeled "SpamCop reporting of spamvertized sites - some philosophy," especially "linear" post number three (Wazoo quoting Don, aka "SpamCop Admin").

I don’t know if it’s really worth mentioning or not

...Well, I certainly think it is! <g>

but I had a really, really hard time finding that page. I couldn’t find it by following your instructions;

<snip>

...In both MS IE and Mozilla Firefox, Ctrl-F will bring up a "Find" dialog that allows you to search for specific text on the page -- in this case, a search for "SpamCop FAQ" will find the link to the FAQ; repeating the process on the FAQ page but with the text "SpamCop reporting of spamvertized sites - some philosophy" will find that link.

...Are you using one of these browsers and, if so, can you verify? If not, what browser are you using? Perhaps a fellow user will be kind enough to post analogous instructions.

[snover]

in this case, a search for "SpamCop FAQ" will find the link to the FAQ; repeating the process on the FAQ page but with the text "SpamCop reporting of spamvertized sites - some philosophy" will find that link.

...Are you using one of these browsers and, if so, can you verify? If not, what browser are you using? Perhaps a fellow user will be kind enough to post analogous instructions.

If you wanted me to search for SpamCop FAQ, you should have said that instead of SpamCop Forum.

The FAQ is exactly what I’m talking about in regards to too much information, though. It’s 11 pages long!

[turetzsr]

If you wanted me to search for SpamCop FAQ, you should have said that instead of SpamCop Forum.

<snip>

...BLUSH. So I should! Fixed -- thank you.

The FAQ is exactly what I’m talking about in regards to too much information, though. It’s 11 pages long!

...That's why it has a sort of table of contents and links. If you can come up with a way of improving it, please jump in and do so! <g>

[Miss Betsy]

The danger of “important” topics, especially when not organised really well, is that as you keep adding things, eventually there is too much and the relevant bits get lost.

The problem here is that there are techies who demand that every situation be covered and technically non-fluent people who feel that they are being told a lot more than they need to know in any FAQ. (from a technically non-fluent person who helped develop the FAQ)

The Spamcop Wiki was developed to kind of mediate between the two views because it is very easy to include in-depth and detailed information by linking. You might also look at the Wiki and see if it is easier to navigate. There are several articles in the FAQ that I would love to edit! However, many of the people who read this forum are techies. It is, in a way, easier to interpret for the technically non-fluent when they post - except that one of our number feels that everyone should know how to search! Not that I disagree with him exactly, but I have a lot of sympathy with those who can't find anything because I lack the search talent myself. It's probably a right brain/left brain thing.

Miss Betsy

[Farelf]

...I have a lot of sympathy with those who can't find anything because I lack the search talent myself. ...

Research skills have their place but you're right, there are limits (even with wikis) ... http://xkcd.com/333/