Jump to content
RobiBue

forum spam handling

Recommended Posts

I just had a brainfart (pardon my French)...

Sooo, we have these pesky little 💩 that think that the readers of these forums are interested in their spew 🤮

Well, here is my proposal to alleviate the problem:

  1. Reported posts receive a mark/counter (see below: 1 reported...)
  2. Posts that are less than 24 hours old and reported more than 3 times get hidden (can be unhid[sic] by the user if he/she so desires)
  3. A user with a post reported 4 times would be prevented from posting in the forum (reading is ok, and pm an admin to ask for unblocking)
  4. Eventually a forum  admin can do some garbage collection (GC) the way they usually do it ;)

this would be the forum view with all topics displayed (the two marked "4 reported" would be hidden by default)

1781011158_Screenshot_2019-07-18HowtouseInstructionsTutorials(1).png.a4063d1c18dccd63199fd44dfe3a9d47.png

This would be the "Unread" topics view (hey, no spam ;) but only if 4 reported them beforehand) 
in Content Types, the user could choose to see the spam (unless the forum admin already done the GC)

1304148149_Screenshot_2019-07-18UnreadContent-SpamCopDiscussion.thumb.png.3ed126b4f74f9d6b575533835926a526.png

 

Suggestions or ideas (or the other way around) are always welcome.

 

 

Share this post


Link to post
Share on other sites

Rob,  Thanks for your concern and active efforts to control of this forurm's spam.

I think your suggestions is overly complicated. 

Currently:

  • I review each new post to this forum.
  • Hide the spam
  • Restrict the poster from posting - Indefinitely
  • Send a warning email
  • Report the spam to:

In the morning after getting required quantities carbs, sugar and caffeine I start reviewing post.  Also randomly during the day and last thing at night. The timing, depending on the time of year, is UTC -6/7h so some may see the spam longer than others.

Rob it is apparent that many mornings you get to the forum before I do.

Currently there are some 4,450 member accounts ban from posting. Banning vs deleting an account prevents spammers from reusing an email address or user name. 

 

Share this post


Link to post
Share on other sites
On 7/18/2019 at 5:35 AM, RobiBue said:

Well, here is my proposal to alleviate the problem:

Rob, I like your solutions, but I don't think they apply here.  As near as I can tell all accounts used in the spam appear to be mostly one and done.  I have noticed that they have two posts at most, but most of them are created to post single content.  Sometimes the same exact thing is posted twice, but it seems to be by different accounts.  The account creation and the post appears to be within about ten minutes.  The reason why I see only one post could be because Lking limits their posting, so they appear to create a new account and move on.  I think Richard might be getting it late at night, while Lking has the daytime.

The forum admins have also changed the captia, so I don't think it is automated spam.  It appears to be fully human since they get through all the hoops that the admins have applied so far.

Share this post


Link to post
Share on other sites

Well, my idea wasn't to thwart the spammers... (ok, in a way it is 😛)
Instead, it would be meant to keep the forums "readable" after 3 or 4 users have reported the posts.
They'd still be there if one really desires to read them, but they'd be hidden until they get handled by an admin.

personally, they don't bother me (much), but I see the occasional OP who mentions the garbage in the forums (fora, fori, forii, whatever) and /me thinks/ (dangerous thing BTW) that there could be something that could be done besides one or two admins cleaning up garbage left by some 💩jackasses...

Usually we don't get much. It seems that today, though, is a different matter... some "recruiter" must have promised a lot of 💵 to some poor souls...

That's actually my idea behind it. Have as few spamposts as possible visible to users, and I think that could accomplish it (I'm sure there are some of us users that report those spams, and if it's just 3 or 4 per post it would do the trick...)

Just my thought... and then Lking could even enjoy his carb-sugar-caffeine drink in a more leisurely manner ;)

Share this post


Link to post
Share on other sites
6 hours ago, RobiBue said:

Well, my idea wasn't to thwart the spammers... (ok, in a way it is 😛)

I don't like the forum spam because as soon as it is posted, gmail has all forum emails marked with spam reputation.  At this point, I personally would prefer to thwart the spammers similar to bl.spamcop.net if possible.

6 hours ago, RobiBue said:

Instead, it would be meant to keep the forums "readable" after 3 or 4 users have reported the posts.
They'd still be there if one really desires to read them, but they'd be hidden until they get handled by an admin.

Ah, so maybe something automated.  If this were possible, I am all for automating any part of it so to limit human mistakes.

6 hours ago, RobiBue said:

Just my thought... and then Lking could even enjoy his carb-sugar-caffeine drink in a more leisurely manner ;)

Seems like maybe some of the admins might be burning the candle at both ends at times.  I have seen more than one person make mistakes when it comes to cleaning up the spam in the forums.  Anything that might help out would be a plus.

I am tempted to suggest that something similar to the SpamCop BL, where enough bad report and a user cannot post or sign up with a new account for 48 hours.

Share this post


Link to post
Share on other sites
5 hours ago, gnarlymarley said:

I don't like the forum spam because as soon as it is posted, gmail has all forum emails marked with spam reputation.  At this point, I personally would prefer to thwart the spammers similar to bl.spamcop.net if possible.

Well, I don’t know about the forum spams being marked as spam in gmail since I only read them in SC. (Anyway, if you receive them as emails, then you should be able — as I do with other email forums — to mark them as never send to spam, and just delete the ones that are “offensive”, as forum emails come from the forum and not from the person sending them...)

5 hours ago, gnarlymarley said:

Ah, so maybe something automated.  If this were possible, I am all for automating any part of it so to limit human mistakes..

Ah, but automated mistakes are also bad. That’s the reason SC uses human decision to ultimately report the processed spam... 

... of course this would be “semi-automated”, as the automation process would start as soon as 3 or 4 humans decided to mark the post as “spam” (only possible in SC online forums)

6 hours ago, gnarlymarley said:

Seems like maybe some of the admins might be burning the candle at both ends at times.  I have seen more than one person make mistakes when it comes to cleaning up the spam in the forums.  Anything that might help out would be a plus.

The Latin phrase for that is “errare humanum est” (to err is human), and I have informed the admin “in situ” of a few odd misdirected posts (fat fingering and lack of caffeine are usually the reasons 🤫)

6 hours ago, gnarlymarley said:

I am tempted to suggest that something similar to the SpamCop BL, where enough bad report and a user cannot post or sign up with a new account for 48 hours.

Well, as Lking already explained:

On 7/18/2019 at 12:37 PM, Lking said:

Currently:

  • I review each new post to this forum.
  • Hide the spam
  • Restrict the poster from posting - Indefinitely
  • Send a warning email

[…]

Currently there are some 4,450 member accounts ban from posting. Banning vs deleting an account prevents spammers from reusing an email address or user name. 

I figure, since the “spam-poster” needs an email account to sign in, these people have tons of throwaway addresses, since they can only use them once. (I am curious on how many addresses use the same domain, and thus prevent them, depending on the domain they use, to even create a SC account. Of course, if they use throwaway gmail, yahoo, hotmail, et.al. accounts, that wouldn’t be feasible...)

Share this post


Link to post
Share on other sites

It has always been the feeling of the powers-that-be that one of the important audiences for this forum are those struggling with the side effects of having a spammer use their email, IP, infect there system or just be in their neighborhood.  In part this concern is due to the impact of an effective SCBL;  If emailers Alice & Bob temporally share an ISP/IP then Bob's email get blocked because of Alice's spam.

The question then becomes how do "we" help Bob?  How do those impacted contact the forum if any automatic blocking is used?  If there post is delayed (until approved by someone) i'm guessing they just look for help elsewhere. I know I do.

As stated we block reuse of usernames and email.  Blocking IPs would also lock all users of gmail, about 1/5 the users of CenturyLink in Denver, etc and that person who shares an IP with a spammer.  Now I have not done an in depth analysis but a quick look at 4 or 5 pages of 25 banned users (sorted by IP) did not reveal a clusters.  Who ever designed the db screwed the date (mm/dd/yyyy) which makes it hard to look at say yesterdays spammers.  I will work on that while watching the hearings in the morning.

Beefing up the front end to keep out the bots seems to be the only acceptable solution, IMHO.  Holding the first post it seems would discourage first posters that have been "blocked by SC" or are trying to deal with spam incoming to their system, both a primary audience.  Blocking IP's or blocks of IP's has the same affect. (yes there have been lagit posters from Russia and India)

Hiding post after n-number of reports 1) would require adding a feature to an off-the-shelf product (check the bottom of the screen) 2) There is also the reality that by the time I get to spam with my first cup, generally the spam has only been reported by @RobiBue.  Sometimes one other.  After those posted while I sleep, there are seldom any reports before I get to them.  and 3) That type of process would open the forum to another type of attack that needs to be programmed to stop.  (Only reports from certain group(s) of users can block.  What about reports by other users?...)

It is a pain.  I have to work at keeping tract of threads that need attention with all the clutter.  There was a time when @Wazoo had full access to the forum software and db. He tweaked the SW with regularity, which resulted in a system that was generally undocumented and not maintainable after he left the seen.  That resulted in the migration to an ISP maintained package and unfortunately all the bad links in old threads.

There are pros and cons to all changes.  There is an issue but a solution where the pros win out is needed.

 

Share this post


Link to post
Share on other sites
9 hours ago, Lking said:

Now I have not done an in depth analysis but a quick look at 4 or 5 pages of 25 banned users (sorted by IP) did not reveal a clusters. 

Still not an "in depth analysis" but today's sample of spamming IPs does not reveal much of a pattern in the first pass.

Sorting the 23 IPs that posted spam over night today show only 1 duplicate  post in "How To Use" otherwise the IPs are unique.  The most active was 4 post from 146.196.37.0/24  otherwise unique at that level.

Although the system does not provide a tool to search users IPs, I tried to check the 22 different IP that posted 23 times today.  I did not find those IPs in the historical db.

Added

Looking up the 22 IPs using https://www.spamcop.net/w3m?action=map the results at this time are not vary informative.

Of the 22 addresses 8 are listed as "poor", and 1 as "neutral".  I guess I was surprised that only ~1/3 are listed.  The 4 addresses 146.196.37.0/24  identified above are all "poor".  The one IP that posted twice was not listed.

Share this post


Link to post
Share on other sites
10 hours ago, Lking said:

Beefing up the front end to keep out the bots seems to be the only acceptable solution, IMHO.  Holding the first post it seems would discourage first posters that have been "blocked by SC" or are trying to deal with spam incoming to their system, both a primary audience.  Blocking IP's or blocks of IP's has the same affect. (yes there have been lagit posters from Russia and India)

Richard had said he did this with the captcha on May 19, but I don't think I saw any change.  I believe this entirely posted by humans.  If it was a robot, the account creation would be around

1 hour ago, Lking said:

Sorting the 23 IPs that posted spam over night today show only 1 duplicate  post in "How To Use" otherwise the IPs are unique.  The most active was 4 post from 146.196.37.0/24  otherwise unique at that level.

Sounds like they might be jumping around (if one person) the internet to avoid detection like they are with email spam.  Also could be that someone is using a VPN service.  I am fairly certain that it is at least two people posting the junk, but could be more.  (The language style seems to be only two different types.)

The source code of HMTL (from http://forum.spamcop.net/profile/46580-hhhmax85/ on Rob's original example) seems to offer a datetime that appears the spammer is returning back later.

<h4 class='ipsType_minorHeading'>Joined</h4><time datetime='2019-07-18T09:51:20Z' title='07/18/2019 03:51  AM' data-short='Jul 18'>July 18</time>
<h4 class='ipsType_minorHeading'>Last visited</h4><time datetime='2019-07-18T09:55:53Z' title='07/18/2019 03:55  AM' data-short='Jul 18'>July 18</time>

I am not sure if the account has someone returning about four minutes later is robot.  Other users I have looked at can be "returning" as much as 16 minutes later.  They either have a good randomizer, or else this is surely human.

Share this post


Link to post
Share on other sites
16 hours ago, RobiBue said:

I figure, since the “spam-poster” needs an email account to sign in, these people have tons of throwaway addresses, since they can only use them once. (I am curious on how many addresses use the same domain, and thus prevent them, depending on the domain they use, to even create a SC account. Of course, if they use throwaway gmail, yahoo, hotmail, et.al. accounts, that wouldn’t be feasible...)

What an interesting thought.  Though I  wonder if they have a stash of thousands of stolen accounts they have to use or if they might be using their hundred domains (like the ones I see in the URLs) for their signup email.

Share this post


Link to post
Share on other sites
1 hour ago, gnarlymarley said:

What an interesting thought.  Though I  wonder if they have a stash of thousands of stolen accounts they have to use

To create an account the email must be validated stolen emails shouldn't work. Anecdotally, there is a pattern to the emails used to create accounts here. Using the forum tools sorting emails of course groups mailboxes not address domains.

The most of the emails today are gmail and outlook. This looks to be true historically with lots of protonmail.com,  mail.com, and yandex.com  The email(s) used with the one IP use twice to post were mail.com and faithmail.org.

Blocking email domains doesn't seem useful.  A casual review highlights gmail and outlook but also protonmail, yandex and mail.    

Share this post


Link to post
Share on other sites
Posted (edited)
5 hours ago, Lking said:

To create an account the email must be validated stolen emails shouldn't work. Anecdotally, there is a pattern to the emails used to create accounts here. Using the forum tools sorting emails of course groups mailboxes not address domains.

The most of the emails today are gmail and outlook. This looks to be true historically with lots of protonmail.com,  mail.com, and yandex.com  The email(s) used with the one IP use twice to post were mail.com and faithmail.org.

Blocking email domains doesn't seem useful.  A casual review highlights gmail and outlook but also protonmail, yandex and mail.    

Hmmm... now here comes a thought... I know, still dangerous 😉

What if... there is/could be a way to check how old an email account is (when it was created) ... Serious Callers Only (yeah, been reading Iain Banks lately 😉) won't use throwaway (recently created) emails to sign up and post in SC (at least I don't think so) unless they are spammers...

Of course, if I had my own mx/mail server, I would be using emails, new or old, but mostly with @mydomain.tld (historically that used to be done in usenet/newsgroups to ensure that scavenged addresses could be pinpointed to a certain usenet base (at least that's how I remember it from way back when 🙂 )

Aaaanyway, so spammer creates emails galore on gmail/outlook/protonmail/yandex/whatever and tries to sign up in forum. Forum says your email is too new, you need approval from admin to post new posts. I know, you mentioned before about legitimate users that want to post, but their email addresses (on the aforementioned big email houses) are usually long established. So the email address age would prevent this spammer from posting right away, and his address could be placed on the ban list for future attempts...
Now, OTOH, spammer uses own @mydomain.tld addresses. Even if the address was new, he would be allowed to spam as before, but now, the domain could be blocked, and  to buy domain names could turn out to be costly for this kind of spam shop... and then he would drop the domains and someone else, legit picks them up and has them already blocked here, so somewhat a timed block could be set in place, coinciding when the domain name expires ;)

Was busy today and didn't have time to report early ;) but I did read your comments and explanations and agree that IP blocking wouldn't be productive.

Now of course, the whole discussion is more or less moot point, since favicon.icoinvision would have to implement all this and I have no idea how willing they are to make changes at this level... and if (as I mentioned) there could be a way to check big email house creation date of addresses...

also, since SC forum deals with valid spam, a forum spamkiller would unfortunately throw too many false positives...

Edited by RobiBue
wasn't finished -- dumb key was stuck (ctrl or alt)

Share this post


Link to post
Share on other sites
Posted (edited)
19 hours ago, Lking said:

It has always been the feeling of the powers-that-be that one of the important audiences for this forum are those struggling with the side effects of having a spammer use their email, IP, infect there system or just be in their neighborhood.  In part this concern is due to the impact of an effective SCBL;  If emailers Alice & Bob temporally share an ISP/IP then Bob's email get blocked because of Alice's spam.

The question then becomes how do "we" help Bob?  How do those impacted contact the forum if any automatic blocking is used?  If there post is delayed (until approved by someone) i'm guessing they just look for help elsewhere. I know I do.

As stated we block reuse of usernames and email.  Blocking IPs would also lock all users of gmail, about 1/5 the users of CenturyLink in Denver, etc and that person who shares an IP with a spammer.  Now I have not done an in depth analysis but a quick look at 4 or 5 pages of 25 banned users (sorted by IP) did not reveal a clusters.

Completely agree, IP blocking is not an option.

19 hours ago, Lking said:

Beefing up the front end to keep out the bots seems to be the only acceptable solution, IMHO.  Holding the first post it seems would discourage first posters that have been "blocked by SC" or are trying to deal with spam incoming to their system, both a primary audience.  Blocking IP's or blocks of IP's has the same affect. (yes there have been lagit posters from Russia and India)

and don't forget china ;)

19 hours ago, Lking said:

Hiding post after n-number of reports 1) would require adding a feature to an off-the-shelf product (check the bottom of the screen) 2) There is also the reality that by the time I get to spam with my first cup, generally the spam has only been reported by @RobiBue.  Sometimes one other.  After those posted while I sleep, there are seldom any reports before I get to them.  and 3) That type of process would open the forum to another type of attack that needs to be programmed to stop.  (Only reports from certain group(s) of users can block.  What about reports by other users?...)

It is a pain.  I have to work at keeping tract of threads that need attention with all the clutter.  There was a time when @Wazoo had full access to the forum software and db. He tweaked the SW with regularity, which resulted in a system that was generally undocumented and not maintainable after he left the seen.  That resulted in the migration to an ISP maintained package and unfortunately all the bad links in old threads.

There are pros and cons to all changes.  There is an issue but a solution where the pros win out is needed.

  1. true, didn't realize that until you pointed it out
  2. Didn't know there were so few uf us. (if I'm on the tablet I don't report because I have to go into the post to report it. with the pc it's easier using the mouse hover)
  3. yeah, again needed that to be pointed out, but it would require several people to report the post to be hid, and as I mentioned, it wouldn't be unreachable, only marked as hidden, but anybody wanting to read it could still access it.

wrt PITA; I know, that's why the ideas being thrown around. Now an undocumented, unmaintainable/chaotic, up the wazoo system is not exactly what I had in mind... (sorry, pun intended)

hopefully, with input of good ideas and weeding out the bad, a winning system could be proposed for third party implementation :)

 

Edited by RobiBue
doing it again, keyboard is going crazy

Share this post


Link to post
Share on other sites
49 minutes ago, RobiBue said:

What if... there is/could be a way to check how old an email account is (when it was created) ... Serious Callers Only (yeah, been reading Iain Banks lately 😉) won't use throwaway (recently created) emails to sign up and post in SC (at least I don't think so) unless they are spammers...

I am not even sure how the coders would detect how old an email is.  I am not even sure this information is available.

51 minutes ago, RobiBue said:

Aaaanyway, so spammer creates emails galore on gmail/outlook/protonmail/yandex/whatever and tries to sign up in forum. Forum says your email is too new, you need approval from admin to post new posts. I know, you mentioned before about legitimate users that want to post, but their email addresses (on the aforementioned big email houses) are usually long established. So the email address age would prevent this spammer from posting right away, and his address could be placed on the ban list for future attempts...

From what I recall, the forum is double opt-in.  I don't think it lets them post until they verify their email.  That verification could be why it takes 3 to 20 minutes between the post and the sign up.  Spammers are grabbing both domains and abandoned email addresses and have been caught using those in their spams.  What is there to stop them from using what is considered an old email address when they sign up?

5 hours ago, Lking said:

The most of the emails today are gmail and outlook. This looks to be true historically with lots of protonmail.com,  mail.com, and yandex.com  The email(s) used with the one IP use twice to post were mail.com and faithmail.org.

That does not leave any good way to block them.

Share this post


Link to post
Share on other sites
9 hours ago, gnarlymarley said:

I am not sure if the account has someone returning about four minutes later is robot.  Other users I have looked at can be "returning" as much as 16 minutes later.  They either have a good randomizer, or else this is surely human.

I too have noted this variant.  There is also a large number of spams by members that have registered days or more before posting.  For example today (last night) there were 8 new members ~ all spammers.  But there were 23 spam posted.

You can mouse over the member icon and see date/time joined and date of last post.  For a spammer likely their only post.

Share this post


Link to post
Share on other sites
Posted (edited)
13 hours ago, Lking said:

I too have noted this variant.  There is also a large number of spams by members that have registered days or more before posting.  For example today (last night) there were 8 new members ~ all spammers.  But there were 23 spam posted.

You can mouse over the member icon and see date/time joined and date of last post.  For a spammer likely their only post.

today, as of 11AM CDT:

17 new members (listed under All Activity) (well, one from yesterday, but almost midnight)
12 of them posted 1 spam each
2 of them didn't post anything
3 had a post, but it didn't exist (Content Count: 1 post -- but nothing found)

28 new spams
14 of them from listed new members
the other 14 from unlisted members but all created within 1 hour of the post (almost as if they deleted their own user themselves after posting...)

and while I was busy during 1 hour while this post is sitting here, cleanup has started and is just about finished ;) ( I need to rephrase this somehow... my post was sitting idle in the editor while I was busy doing other things. When I got back 1 hour later, I noticed that cleanup was being done.)

Edited by RobiBue

Share this post


Link to post
Share on other sites
6 minutes ago, RobiBue said:

3 had a post, but it didn't exist (Content Count: 1 post -- but nothing found)

Those are post I was in the process of hidding .  The user's post count does not update when I hide their post, but there is nothing for you to see.

Robi we are ships in the night.

Share this post


Link to post
Share on other sites
10 minutes ago, Lking said:

Those are post I was in the process of hiding .  The user's post count does not update when I hide their post, but there is nothing for you to see.

Robi we are ships in the night.

gotcha!

we need a pb_CapeCanaveral.jpg;) 

Share this post


Link to post
Share on other sites
5 hours ago, RobiBue said:

2 of them didn't post anything

interesting, I have wondered if the spammers had a hidden account that was only created to verify that they the emails the forum sends out has their spam.  Though, I would lean more toward an account they created about two years ago for that.

Share this post


Link to post
Share on other sites
4 hours ago, gnarlymarley said:

interesting, I have wondered if the spammers had a hidden account that was only created to verify that they the emails the forum sends out has their spam.  Though, I would lean more toward an account they created about two years ago for that.

well, it is very possible, that those 2 are legit, just found SC, and decided to sign up in the forum.

Share this post


Link to post
Share on other sites
45 minutes ago, RobiBue said:

well, it is very possible, that those 2 are legit, just found SC, and decided to sign up in the forum.

Not to be more cynical than usual, but...  Wandered back and founld one new member "M" who joined ~3UTC (12 hours ago) who's IP is in New Delhi.  We will see

While looking at the list of new users saw three that have not yet validated their emails (2- outlooks & 1- protonmail) all from New Delhi.  But then again RobiBue may be right.

Share this post


Link to post
Share on other sites

Does the forum support keyword filtering redirection for moderation?  (Meaning that if someone posts with something URL that contains supplement or keto, the post will not be seen until a moderator approves it?)

Share this post


Link to post
Share on other sites

Maybe better yet is to have all posts that have a URL, but it is not a spamcop URL (forum or tracking URL) all go to be moderated if it is supported.

Share this post


Link to post
Share on other sites
49 minutes ago, gnarlymarley said:

Does the forum support keyword

 

38 minutes ago, gnarlymarley said:

URL, but it is not a spamcop URL

Let me do some research.  I think I see an opening.

Share this post


Link to post
Share on other sites

Apologies, but I do see a problem with that. I mean, this is a spam fighting forum, and if someone posts a question about a spam and the words include something that would be filtered, then the OP would have to wait until the admin frees it to the forum...

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×