Help - Search - Members - Calendar
Full Version: Complainterator for Linux
SpamCop Discussion > Discussions & Observations > Suggested Tools and Applications
efa
I post also here, a version of Complainterator crossplatform, thinked to work on Linux, but work on every platform with a complete GNU/Posix environment.
As example I also test it with Cygwin on WinXP, but should work on Solaris or MacOS X.

It is written as a Bash scri_pt, so you have no GUI.
The scri_pt is written under the GNU/GPL license so, as everytime other developer are wellcomed.
My thanks to tembow for the idea and many explanations on domain name system
In my experience, it cutted my spam from 20 a day, to 5 a day. It carry my time around spam to near 0.


Invocation:
If run with a parameter, will be interpretated as a URL, and complaint for this.
It recover Registrar, Name Servers, Registrar Name Server, and send all the complaints emails for the spammed domain.

If run without parameter, it download all the spam from an IMAP4 account, from a particular spam folder.
spam should be moved there before by a Bayesian filter like the one included in Mozilla, that after a week of training, indentify 100% of spam with 0% false positive, carrying away from your eyes all the spam.
Then it analyze the spam (decoding as appropriate) to find the URL, and complaint for all the founded URL.
It can follow some simple redirect URL and complaint also for those.
Then it parse again the mail to find scam email, and complaint also for those.
Last, it forward the spam to a configured email like spamcop or knujon.
My suggestion is to use in tandem with Spamcop as are complementary solutions.
This version send complaint emails with domain in obfuscated [dot] format for some Registrars.
Initially are Beijing and Hichina, you can add more in 'RegistrarObfuscated.txt' file.


Take care of configure your user IMAP and SMTP mail account inside the scri_pt.
In particular correctly fill the string 'userMail' with <email>
Example:
userMail="Name <yourEmail[at]domain.tld>"
as some SMTP server require the <>
From this version you can (optional) use an external mail server configuration file.
It must be called 'xComplaintConfig.txt' and should be in this format:

CODE

# mail server user configurations:
export userImap="imap.gmail.com"       # imap4 server
export userUser=""       # user on imap4 server
export userPass=""       # pass on imap4 server
export imapSSL="--ssl"   # insert --ssl when IMAP server require (Gmail)
export spamFold="[Gmail]/spam"   # spam folder on IMAP server
export userSmtp=""       # user smtp server
export userMail=""       # user email "Name <email[at]domain.tld>"
export forwardEmail=""   # a mail to forward all spams



The Registrar list is exacly the same as:
http://www.spamtrackers.eu/downloads/Compl...lainterator.txt
but with unix style new line, sorted alphabetically, and with no line ending space.
Unfortunately the search algoritm is really slow. Need a new version in C language with a GUI in GTK+.


The scri_pt depend on the following external tools executables:

CODE

cat      # text handling
grep     # text handling
awk      # text handling
sed      # text handling
tr       # text handling
cut      # text handling
sort     # text handling
dos2unix # text handling
wc       # text handling
read     # input handling (bash builtin)
base64   # decode base64 encoding
iconv    # international encoding conversion

dig      # NameServer query
host     # IP query
whois    # whois query

wget      # URL html redirect check
telnet    # send emails
fetchmail # receive emails



On Windows you have to install Cygwin with standard package, plus whois, wget, inettools (for telnet), fetchmail, dos2unix, base64, and iconv.
More you have to install the package 'dig' from:
http://members.shaw.ca/nicholas.fong/dig/
as isn't included in Cygwin.


The package can be downloaded from:
http://www.castlecops.com/p1110770-Complai...ux.html#1110770
kamaraju
QUOTE(efa @ Sep 3 2008, 06:25 PM) *

I post also here, a version of Complainterator crossplatform, thinked to work on Linux, but work on every platform with a complete GNU/Posix environment.


I was experimenting with V.0.10.20 2008/09/09 and found out that you are using telnet in the scri_pt as in

if (test 0 == "$debug") then
cat sendoutDoma.txt | telnet $userSmtp 25
cat mailoutDoma.txt >> complaints.txt
echo "====================================================" >> complaints.txt
echo Mail sended. Exit.
else
echo Mail not sended. Exit.
fi


Is there no other alternative available (like ssh)? Telnet is supposed to be insecure and not all the Linux systems I know have telnet installed.

hth
raju
efa
QUOTE(kamaraju @ Sep 25 2008, 06:40 PM) *

Is there no other alternative available (like ssh)? Telnet is supposed to be insecure and not all the Linux systems I know have telnet installed.

the problem is the SMTP server.
Most I know, at ISO/OSI level 5, use simple un-encripted, un-autenticated, telnet connection on TCP port 25.
Exist ESMTP, SMTP/TLS and SMTP-AUTH (ESMTPA) protocol, but which provider use it?
Most provider check only the domain part of the sender email, and the direct connection to block relaing, but no real autentication or encription are used.
Try your SMTP server, it accept ssh connection? If yes, surely you can substitute telnet with ssh in the scri_pt.
Let me know if it work. If yes, please tell me which kind of SMTP server is.

Another solution may be to use 'exim' (or 'ssmtp' or 'msmtp') available everywhere. At simple level they do a telnet on port 25, but can also support all types of encription.
efa
hi all,
it's ready a new bugfix version of xComplaint bash scri_pt.

This include a speedup written in C, stolen from the compiled counterpart that is slowly growing. Now the Registrar contact search is 60 times faster.
The report now include the Domain creation data, usefull to add evidence of just registered domain
You can now pass an argument with a file as raw spam mail, usefull for mail web access.
The complaint now are more similar to Complainterator V22.8
You now can send complaints with a different account from the one you are receiving spam. The xComplaintConfig.txt file has changed the name of some variables to handle this little complex configuration.
The scri_pt do an external command dependancy compliance check before start

Next version will support SSL/TLS SMTP authenticated encripted send (using 'msmtp'), usefull for Gmail account.

Here the changelog:
CODE

# V.0.10.22 2009/05/24
# Add: use a C compiled Registrar searchEmail engine to speedup bash bottleneck
#      really faster now!
# Add: Parse Whois report for Creation and Update Date for Domains
# Add: put domain creation date in email body
# Add: now you can pass a parameter with a text spam file
# Add: often scam email are just registered email, with lot of char before @
# Add: create complaint mails more like Complainterator V22.8 do
# Add: create SMTP query after email confirmation (usefull to add EVIDENCE)
# Add: added evidence in Domain complaints
# Add: check for external dependancy compliance before start
# Fix: better check of external contact/data files
# Fix: reworked mail server account, to let different server for receive&send
#      changed xComplaintConfig.txt file variables requirements
# Fix: handle base64 encoded spam with 4 lines separator, with or without ""
# Fix: skip domain without a dot in the last 5 letter
# Fix: set imap server timeout to 20 seconds
# Fix: remove subdirectories from argument URL before complaint
# Fix: remove ending dot and comma from spamvertized links
# Fix: find Registrar when is marked 'Registrar Name:' (some .ws domain)
# Fix: if traversal on A record fail, try use DNS cache
# Fix: always get Name Server IP address
# Fix: always get Domain IP address
# Fix: skip other domain less often used for scam emails:
#      september11victims.com, right-thoughts.us, jonathanforeman.com,
#      sptimes.com, msn.co.uk, ya.com, ft.com, fundaciongsr.es, fedex.com,
#      guardian.co.uk, ftc.gov, paypal.com
# Fix: updated contact files


The new format of 'xComplaintConfig.txt' configuration file is:
CODE

# insert here the user configurations:
# ------------------------------------
# receive section account 1
export userImap="imap.gmail.com"       # imap4 server
export mailImap="user[at]gmail.com"       # user email for imap4 server
export userPass=""       # pass on imap4 server
export imapSSL="--ssl"   # insert --ssl when required by IMAP server (Gmail)
export spamFold="[Gmail]/spam"   # spam folder on IMAP server

# send section account 1
export userSmtp="smtp.gmail.com"       # user smtp server
export mailSmtp="user[at]gmail.com"        # user email for smtp server
export userName="Name Surname"   # user name and surname

# receive section account 2
export userImap="imap.domain.tld"       # imap4 server
export mailImap="user[at]domain.tld"       # user email for imap4 server
export userPass=""       # pass on imap4 server
export imapSSL=""        # insert --ssl when required by IMAP server (Gmail)
export spamFold="spam"   # spam folder on IMAP server

# send section account 2
export userSmtp="smtp.domain.tld"       # user smtp server
export mailSmtp="user[at]domain.tld"        # user email for smtp server
export userName="Name Surname"   # user name and surname

# list of emails to forward all spam to
export forwardEmail="user[at]wantspam.tld"   # a mail to forward all spams


The scri_pt can be downloaded from:
http://sharebee.com/677aa0b1
efa
hi,
it's time for a major update, as I added lot of new functionalities.
The most interesting one is support for Google/Gmail account SSL/TLS encripted+authenticated complaint emails SMTP send, via 'msmtp'.
(I again little changed the format of xComplaintConfig.txt file to support encription+authentication).
Another interesting new funzionality, add redirected links to email body before forward, this to cheat spammer, and to keep track of spamvertized redirections.
Let manually add a domain for uncatched redirections.
xComplaint now ask one more confirmation, when domain/NS creation date is older than 7 months (configurable)
The links and email estraction is reworked. Is now easy add fake scam email and trusted web site to external config files.
Do not complaint for domain with IP 22.22.22.22 or 44.44.44.44
When found more than one "Registrar:" show a WARN, and get the last (and not the first), this work correct on most domains
General speedup, lot of bug fixes, lot of updated in contact files

Here the complete change log:
CODE

# V.0.11.23 2009/07/10
# Add: support Gmail SMTP SSL/TLS authenticated+encripted send using 'msmtp'
#      changed/added xComplaintConfig.txt file variables names
# Add: parse Whois for Creation date of NS, put it in complaints
# Add: parse Whois for domain/NS Status, show it in the query log
# Add: ask one more confirmation, when creation date is older than 7 months
# Add: add domain creation date in complaint only when younger than 7 months
# Add: add redirected html URL to email body tail before forward
# Add: skip complaint 4 sign.out_me, out_of_the_list, get.me_out_now,
#      get.out_of_the_list, get.out_from_list[at]yahoo.com
# Add: build file xComplaintMyEmails.txt, to skip reporting personal emails
# Add: rework the scam domain engine, using an external text file
# Add: reworked the email parsing engine, now work on real body of spam
# Add: if DNS traversal fail, but domain is OK, complaint for domain only
# Add: skip complainting for domain/NS suspended with 22.22.22.22
# Add: skip complainting for domain/NS suspended with 44.44.44.44
# Add: now put in complaints the Domain IP address
# Add: evidenced redirections link catch or not
# Add: let manually add a domain for uncatched redirections
# Fix: reworked domain extraction, some was wrong: [xxx.]domain.com.cn
# Fix: find domain for some ccTLD name servers: discountwatch.com.cn
# Fix: when main RegistrarNameServer, sometimes complaint 4 previous spam NS
# Fix: get last Registrar: when there is more than one in whois report
#      evidence a WARN to optionally manually check the whois report
# Fix: find Registrar when is marked 'REGISTRAR:' (some .pl domain)
# Fix: find CreationDate when is marked 'registered:' (some .hu domain)
# Fix: cygwin/andLinux only: doesn't found some China/UTF-8 Registrar:
#      abcompany.com.cn,ootraffic.com.cn
# Fix: remove innocent Bcc: Field from list of complaint emails
# Fix: skip try to complaining for URL with IP address only
# Fix: better parse of creation/updated date for domain/NameServer
# Fix: consider domains and redirected links as case insensitive
# Fix: reduce dig timeout to the minimum on IP query
# Fix: reworked some debug and print style part
# Fix: added tail and touch to the dependancies
# Fix: lot of updates in contact files


Here the new format of 'xComplaintConfig.txt':
CODE

## xComplaintConfig file. For V.0.11.23 2009/07/10 or next
# receive section
imapHost="imap.gmail.com"       # user IMAP4 server
imapPort="993"       # default to 143 or 993 with SSL
imapSSL="--ssl"   # insert --ssl when required by IMAP4 server (Gmail)
imapAuth=""       # use when authentication is required
imapMail=""       # user email for IMAP4 server
imapPass=""       # pass on IMAP4 server
imapSpam="[Gmail]/spam"   # spam folder on IMAP4 server
# send section
smtpHost="smtp.gmail.com"       # user SMTP server
smtpPort="587"       # default to 25 or 465 with SSL
smtpSSL="--tls=on --tls-starttls=on"        # insert --tls=on --tls-starttls=on when required
smtpAuth="--auth=on"       # insert --auth=on when required by SMTP server (Gmail)
smtpTrust="/etc/ssl/certs/Thawte_Premium_Server_CA.pem"      # trust path/file issued by CA
smtpMail=""       # user email for SMTP server
smtpPass=""       # pass on SMTP server
userName=""       # user name and surname
replyMail=""      # email for "Reply-To:" field
forwardEmail=""   # a mail to forward all spams


The new version can be downloaded from:
http://sharebee.com/6b6e04a1
Wazoo
QUOTE(efa @ Jul 14 2009, 06:30 AM) *
Another interesting new funzionality, add redirected links to email body before forward, this to cheat spammer, and to keep track of spamvertized redirections.

This sounds an awful lot like a violation of the Reporting Rules .... specifically, "adding data to the spam content that the parser would not discover on its own" Please see Material changes to spam
efa
QUOTE(Wazoo @ Jul 14 2009, 06:53 AM) *
"adding data to the spam content that the parser would not discover on its own"

if you do not want this functionality, simply delete lines from 2193 to 2225 in xComplaint V.0.11.23 2009/07/10,
there are comments explaining this in the code.
The scri_pt is in GNU/GPL opensource license, so you are invited (and encouraged) to modify
the sources for your needs, with two unique requisite: that you pubblish the modified version again in GPL license, and attribute the original author.
efa
hi,
I integrated the hard follow redirection and so a new major update: xComplaint V.0.12.24a 2009/09/23
It download html, parse the code to find redirection to real spammer web site.
Support for no check of SMTP server certificate with "--tls-certcheck=off"
Configurable retry times on SMTP send failure
xComplaint now use OpenDNS when other dig queries fail
finished to write and integrate the manual pages. See it with --help
Officially support for andlinux.org as preferred solution against cygwin on Win32
Obfuscation is now with -dot- as work with @reg.ru
Sources of emailSearch binary are provided in the package
Now domain age calculation work as expected, lot of date format supported!
Synch of contact files with xComplaint 22.15

Here the complete changelog:
CODE
# V.0.12.24a 2009/09/23
# Add: integrated more aggressively follow redirection
# Add: let skip (hard) follow redirection (silent to firewall)
# Add: skip URL ending with .gif, .jpg or .png
# Add: spam domain max age carried to 9 months
# Add: base64 decode application/octet-stream as used for phish .html attach
# Add: support for no check of SMTP server certificates "--tls-certcheck=off"
# Add: use OpenDNS when other dig queries fail
# Add: call xClean before everything when debug is off
# Add: finished to write and integrate man pages
# Add: show version with -V, syntax with -h and man with --help
# Add: tested and work on AndLinux.org a Win32 port of Ubuntu+CoLinux
# Add: obfuscation is now with "-dot-" and not "[dot]" as work with @reg.ru
# Add: let easily skip add redirected/manualURL to spam body
# Add: remove ending /?... part from the links
# Add: package sources of "emailSearch.tgz" together xComplaint
# Add: support for many more date format for domain/NS
# Fix: often error in domain age extraction with many different format
# Fix: sometimes unrecoverable error on age calculation
# Fix: find Registrar when is marked 'organization:' (some .ua domain)
# Fix: find CreationDate when is marked 'Created Date:' (some .th domain)
# Fix: find CreationDate when is marked 'Created:' (some .it domain)
# Fix: find UpdatedDate when is marked 'Last Update:' (some .it domain)
# Fix: find DomainStatus when is marked 'state:' (some .ru domain)
# Fix: remove redirect/HardFollowed fakeUrl, like with direct link
# Fix: when all scam links, show: cat: links91.txt: Nessun file o directory
# Fix: with ctrl-c on HardFollow redirect to past url
# Fix: some rare case of error 'sed -e need a regular expression'
# Fix: sometimes msmtp error: TLS certificate verification failed. Retry
# Fix: remove ending; from URL
# Fix: removed WARN: Trace on Domain/NS IP failed. Using Provider DNS cache ...
# Fix: sometimes obfuscation is not complete: ns5.0m5.ru
# Fix: updates in contact and fakeurl files


You can download this version here:
http://sharebee.com/16c64d2e
Farelf
QUOTE(efa @ Sep 25 2009, 02:42 PM) *
...I integrated the hard follow redirection and so a new major update: xComplaint V.0.12.24 2009/09/22
...
[numerous other features]
...
You can download this version here:
http://sharebee.com/16c64d2e
Thanks efa! You probably don't get the recognition you deserve but I'm sure many Linux users are grateful.
efa
QUOTE(Farelf @ Sep 25 2009, 02:27 AM) *

Thanks efa! You probably don't get the recognition you deserve but I'm sure many Linux users are grateful.

I see that previous version was downloaded 55 times, and this is enough.
Hoping someone else want to collaborate enhancing the scri_pt, maybe with "unique domain sorting" and not unique url sorting as is now. This avoid to ask for reporting for every url when the spamvertized domain is the same. Now I'm developing an anti-phish application
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2009 Invision Power Services, Inc.