Need serious help...

[jeffslife]

Hello!

Obviously I am posting here because my school's network I work for has been blocked by this (and several other) lists.

I don't really know what to do anymore. We've cleaned up virus machines, I've been scanning packets on known trouble ports (eg. 25) for hours, and I've found nothing... I don't know where else to look for the problem.

I've run every open relay test I can find out on the net, and the email server has passed every single one.

There was ONE machine that was sending massive port 25 traffic and was heavily virus infected, but we unplugged it immediately and wiped the drive out, and that machine still isn't even plugged in anywhere.

But several lists (including this one) keep re-blacklisting for some reason, spam I'm guessing, and I just don't know where else to look anymore.

Any help would be greatly appreciated. I'm not here to dispute the block, it sounds like a legitimate block, but I can't seem to find the source... Business is really at a stand still, important e-mails that need to be sent are not getting out, would really appreciate some assistance.

Thank you.

[agsteele]

Any help would be greatly appreciated. I'm not here to dispute the block, it sounds like a legitimate block, but I can't seem to find the source... Business is really at a stand still, important e-mails that need to be sent are not getting out, would really appreciate some assistance.

Hi jeffslife!

The first thing you're likely to face is that you've given us nothing to help you with.

At the very least we need to know te error message you've received that's sent you here and/or the IP address of your mail server.

Take a look at the FAQ which gives some helpful advice on identifying spam sources and how to ask for help and do come back with that extra information or questions.

We'd love to help but can't without a bit of help

Andrew

[jeffslife]

Oops! Forgive me, my brain is fried on the subject.

Return error on outgoing mail:

550-"JunkMail rejected -

mohawk.mtrsd.k12.ma.us (mail.mohawkschools.org)

550-[159.250.29.8]:45506 is

in an RBL, see Blocked - see 550

http://www.spamcop.net/bl.shtml?159.250.29.8 (in reply to RCPT TO

command)

Also, I'd love more than anything to read an FAQ, which one are you referring to?

[Wazoo]

Also, I'd love more than anything to read an FAQ, which one are you referring to?

At the top of this page, both the SpamCop FAQ and the SpamCop WIki are offered via several links. In this Forum section, there was a Pinned entry, also titled "Why am I Blocked?" ....

550-"JunkMail rejected -

mohawk.mtrsd.k12.ma.us (mail.mohawkschools.org)

550-[159.250.29.8]:45506 is

in an RBL, see Blocked - see 550

http://www.spamcop.net/bl.shtml?159.250.29.8

I edited your post to make the link functional .... but you didn't say whether you bothered to follow it yourself or not.

http://www.spamcop.net/w3m?action=blcheck&...ip=159.250.29.8

159.250.29.8 listed in bl.spamcop.net (127.0.0.2)

If there are no reports of ongoing objectionable email from this system it will be delisted automatically in approximately 21 hours.

Causes of listing

* System has sent mail to SpamCop spam traps in the past week (spam traps are secret, no reports or evidence are provided by SpamCop)

* SpamCop users have reported system as a source of spam about 20 times in the past week

Listing History

System has been listed for 7.4 days.

This fits the infected/compromised system scenario, based on both spamtrap hits and user Reports.

http://www.senderbase.org/senderbase_queri...ng=159.250.29.8

Volume Statistics for this IP

Magnitude Vol Change vs. Last Month

Last day ........ 3.7 5%

Last month .... 3.7

No sign of a slow-down in traffic, so it would appear that there is more than just he one system involved.

been scanning packets on known trouble ports (eg. 25) for hours, and I've found nothing.

Per http://forum.spamcop.net/forums/index.php?showtopic=4556 .. you're looking at something around 10,000 e-mails a day. It would seem that if you're not "finding anything" you're not looking in the right spot.

Nothing said about a firewall in use, or log files analyzed.

Nothing said that actually explains anything abut the network, i.e. an actual/separate e-mail server involved (Hostname: mohawk.mtrsd.k12.ma.us doesn't really suggest this) Iis there any wireless networking involved?

dig mohawk.mtrsd.k12.ma.us [at] 208.67.220.220

Dig mohawk.mtrsd.k12.ma.us[at]dns-auth1.crocker.com (204.97.12.58) ...

failed, couldn't connect to nameserver

Dig mohawk.mtrsd.k12.ma.us[at]dns-auth2.crocker.com (204.97.12.57) ...

failed, couldn't connect to nameserver

Dig mohawk.mtrsd.k12.ma.us[at]208.67.220.220 ...

Non-authoritative answer

Recursive queries supported by this server

Query for mohawk.mtrsd.k12.ma.us type=255 class=1

mohawk.mtrsd.k12.ma.us NS (Nameserver) dns-auth2.crocker.com

mohawk.mtrsd.k12.ma.us NS (Nameserver) dns-auth1.crocker.com

telnet 159.250.29.8 25

Trying 159.250.29.8...

telnet: Unable to connect to remote host: Connection timed out

Trace mohawk.mtrsd.k12.ma.us (159.250.29.8) ...

144.232.19.143 RTT: 17ms TTL:170 (sl-crs1-chi-0-12-2-0.sprintlink.net ok)

144.232.18.59 RTT: 39ms TTL:170 (sl-crs2-spr-0-4-5-0.sprintlink.net ok)

144.232.1.9 RTT: 42ms TTL:170 (sl-gw6-spr-15-0-0.sprintlink.net ok)

144.223.76.22 RTT: 37ms TTL:170 (sl-crock5-96615-0.sprintlink.net probable bogus rDNS: No DNS)

159.250.29.8 RTT: 43ms TTL: 50 (mohawk.mtrsd.k12.ma.us ok)

http://www.mxtoolbox.com/index.aspx

ns.amaranth.net did not respond with MX records for 'mohawk.mtrsd.k12.ma.us'

Mail for mohawk.mtrsd.k12.ma.us is handled by mail.mtrsd.k12.ma.us

Trace mail.mtrsd.k12.ma.us (159.250.29.160) ...

144.232.1.9 RTT: 37ms TTL:170 (sl-gw6-spr-15-0-0.sprintlink.net ok)

144.223.76.22 RTT: 46ms TTL:170 (sl-crock5-96615-0.sprintlink.net probable bogus rDNS: No DNS)

159.250.29.8 RTT: 43ms TTL:170 (mohawk.mtrsd.k12.ma.us ok)

* * * failed

* * * failed

* * * failed

* * * failed

telnet 159.250.29.160 25

Trying 159.250.29.160...

telnet: Unable to connect to remote host: Connection timed out

Too much guessing going on at this side of the screen.

[Farelf]

...Also, I'd love more than anything to read an FAQ, which one are you referring to?

See http://www.spamcop.net/fom-serve/cache/75.html - particularly the bottom section "Assistance stopping spam:"

Thanks for the error-rejection messages. Following the link supplied http://www.spamcop.net/bl.shtml?159.250.29.8 you will see some indication of the problem, particularly a link from that link - http://www.spamcop.net/w3m?action=blcheck&...ip=159.250.29.8 - and a link from that one in turn says that member reports would have gone to matthew[at]crocker.com. Are you able to access those? Some members are able to see brief detail of the member reports made and might be kind enough to list some detail. In the meantime, one of those other lists you mention has some information http://cbl.abuseat.org/lookup.cgi?ip=159.250.29.8 - though from the actions you've taken you may have already seen that?

[jeffslife]

.... but you didn't say whether you bother to follow it yourself or not.

Of course I followed it. It does not appear to give me any information I can use other than how long I've been blocked.

This fits the infected/compromised system scenario, based on both spamtrap hits and user Reports.

http://www.senderbase.org/senderbase_queri...ng=159.250.29.8

Volume Statistics for this IP

Magnitude Vol Change vs. Last Month

Last day ........ 3.7 5%

Last month .... 3.7

No sign of a slow-down in traffic, so it would appear that there is more than just he one system involved.

Per http://forum.spamcop.net/forums/index.php?showtopic=4556 .. you're looking at something around 10,000 e-mails a day. It would seem that if you're not "finding anything" you're not looking in the right spot.

Indeed I'm not "finding anything". Hence the request for help.

Nothing said about a firewall in use, or log files analyzed.

Nothing said that actually explains anything abut the network, i.e. an actual/separate e-mail server involved (Hostname: mohawk.mtrsd.k12.ma.us doesn't really suggest this) Iis there any wireless networking involved?

Firewall is in place, it's an Ubuntu machine, firewall is all IPTABLES. I've been logging as much traffic as I can and going over syslog and any other log that I think might help, but as I've said, I'm obviously not as good at this as you guys, hence the help. I DON'T know where to look on top of what I've said (monitoring ports).

We host the mail server on our network, it's also an Ubuntu machine. Postfix/CourierIMAP. All of our employees use the web interface (squirrelmail).

There are wireless points around the building, that are all passworded and only certain machines have access to them. The banned IP in question is our T1 line, and there aren't a lot of people on it, it is where our email server resides.

mail.mtrsd.k12.ma.us is an old email server that is no longer in use. It's still there but all of its mail functions have been stopped.

The server in question is that Ubuntu machine I just mentioned at mail.mohawkschools.org

I've just been reading the stickies in this forum, and in my limited capabilities in this situation, I am unable to use them to any further advantage.

See http://www.spamcop.net/fom-serve/cache/75.html - particularly the bottom section "Assistance stopping spam:"

Thanks for the error-rejection messages. Following the link supplied http://www.spamcop.net/bl.shtml?159.250.29.8 you will see some indication of the problem, particularly a link from that link - http://www.spamcop.net/w3m?action=blcheck&...ip=159.250.29.8 - and a link from that one in turn says that member reports would have gone to matthew[at]crocker.com. Are you able to access those? Some members are able to see brief detail of the member reports made and might be kind enough to list some detail. In the meantime, one of those other lists you mention has some information http://cbl.abuseat.org/lookup.cgi?ip=159.250.29.8 - though from the actions you've taken you may have already seen that?

Yah, I've been testing my efforts mostly on that particular page. We did find that one machine that was sending massive traffic on Port 25 (that page mentions blocking/monitoring 25) and that's how we picked that one up. I do not see any other machines attempting to use port 25 at all (aside from the mail server itself).

We have AVG 8.0 on every machine that is connected to the network where the mail server is, and as of this writing, no massive amounts of trojans/viruses have been found.

None of the machines have any email clients set up (thunderbird, outlook).

[rconner]

My impression is that bots do not send spam around the clock, only sporadically. So, looking at a machine right now and seeing no suspicious traffic does not mean that there won't be any later.

Would you put a dent in users' productivity if you simply blocked outgoing port 25 for everyone (except any bona fide mail hosts you run) using your firewall? My company does this, as do many others I suspect. This would stop outgoing spam, but also outgoing honest mail directed to outside servers (e.g., someone at work sending personal mail via AOL's mail hosts). If properly done, it would not stop people from sending mail through your domain's own mail host.

-- rick

[DavidT]

We have AVG 8.0 on every machine that is connected to the network where the mail server is, and as of this writing, no massive amounts of trojans/viruses have been found. None of the machines have any email clients set up (thunderbird, outlook).

You don't need to have a mail client installed in order for a compromised computer to spew stuff out through your SMTP port. You need to shut down port 25 immediately. Here's what the CBL says about your IP:

IP Address 159.250.29.8 is currently listed in the CBL.

It was detected at 2009-02-10 12:00 GMT (+/- 30 minutes), approximately 15 hours ago.

It has been relisted following a previous removal at 2009-02-08 00:21 GMT

ATTENTION: At the time of detection, this IP was infected with, or NATting for a computer infected with a high volume spam sending trojan - it is participating or facilitating a botnet sending spam or spreading virus/spam trojans.

ATTENTION: if you simply repeatedly remove this IP address from the CBL without correcting the problem, the CBL WILL eventually stop letting you delist it and you will have to contact us directly.

You said that you found a machine spewing stuff out through port 25. You didn't mention if you'd successfully cleaned the machine to the point that it no longer is making those transmissions.

DT

[SpamCopAdmin]

159.250.29.8 sent spam to our spamtrap system as recently as Tuesday, February 10, 2009 06:59:43 -0700

Here is an example header:

Received: from mohawk.mtrsd.k12.ma.us ([159.250.29.8])

by [our trap server] with SMTP; 10 Feb 2009 05:xx:xx -0800

From: Josie <x[at]x>

Subject: Add 3"

Date: Tue, 10 Feb 2009 13:xx:xx GMT

- Don D'Minion - SpamCop Admin -

.

[dbiel]

Try monitoring all traffic coming out of your mail server and look for

Subject: Add 3"

as Don has indicated this is the subject line of a recent spam. It can also look through you logs for the same subject line. Don even gave you a date and time range which should reduce the search greatly.

Good luck

[agsteele]

I see that at the time of writing delisting will take place in 3 hours and the Senderbase data suggests a change in mail volume of -80%

This may indicate that the problem is under control.

Even so, Rick's (rconner) suggestion of limiting port 25 to only your in-house mail server is really the way to go. In the event that another of your users gets infected with a trojan/bot you then stand a chance of identifying the problem without getting listed. And if the bot is now just dormant awaiting an opportunity to strike again, you'll at least stop it.

Andrew

[Wazoo]

http://www.spamcop.net/w3m?action=blcheck&...ip=159.250.29.8

159.250.29.8 not listed in bl.spamcop.net

http://www.senderbase.org/senderbase_queri...ng=159.250.29.8

Volume Statistics for this IP

Magnitude Vol Change vs. Last Month

Last day ...... 0.0 .. N/A

Last month .. 3.7

Not at all sure what to make of those numbers.

Firewall is in place, it's an Ubuntu machine, firewall is all IPTABLES. I've been logging as much traffic as I can and going over syslog and any other log that I think might help

To actually see what's going as far as this goes, one would want to see what's in the list ....

sudo iptables -L will list the various entries placed there .. not sure if you'd want to post all of the contents publically, but those entries dealing with SMTP and Port #25 actions would certainly be of interest. The default logged data would normally be found at /var/log/nessages, but there's a lot of configuration data that could change this location, the quantity and type of data kept track of, etc. .... version of Ubuntu not mentioned, but that could also make a difference.

There are wireless points around the building, that are all passworded and only certain machines have access to them.

Are you aware that WEP is basically 'broken' ..... WPA has some issues with encoding/encryption types ... and MAC addresses can be forged easily????

mail.mtrsd.k12.ma.us is an old email server that is no longer in use. It's still there but all of its mail functions have been stopped.

The server in question is that Ubuntu machine I just mentioned at mail.mohawkschools.org

Based on my trying to dance around and find things, the DNS, MX, etc. records could certainly stand some refreshing to bring them up to date.

[Farelf]

...We have AVG 8.0 on every machine that is connected to the network where the mail server is, and as of this writing, no massive amounts of trojans/viruses have been found. ...

Well, you certainly seemed to have stopped whatever it is/was. Discovery and disinfection. AVG won't pick up everything. Just reminded (by the latest update issuing today) - everyone forgets about Windows Malicious Software Removal Tool. Supposedly it has a better 'rootkit' detection rate than any of the commercials and can remove almost anything it finds (which is also better than the commercial offerings). Any infection is likely to be on a Windows machine - ensure MS updates are run, then do at least the first level check with the tool. See http://forum.spamcop.net/forums/index.php?...ost&p=67453

[jeffslife]

Hey guys,

I can't thank you enough for all the replies and help you're giving me. I've been blazing everywhere trying to solve this.

We thought we were taking care of it finally, but alas, we are back on your ban list (and others I imagine).

The problem with the port 25 thing in IPTABLES is that it's still a little over my head. I've been reading like crazy, but the guy that worked here before I did set this whole scri_pt up, and his method seems very custom, and unlike every other example out there.

I tried to follow the SpamHaus example of limiting port 25 to only the email server, but when added their code, no mail was coming in or out.

Most documentation that I can find talks about the INPUT, FORWARD, and OUTPUT chains, but the guy that wrote this thing has many PREROUTING rules in here that I don't really understand fully yet.

The relative lines are these (10.0.100.4 is the email server's internal ip, 159.250.29.171 is external):

iptables -A PREROUTING -t nat -d 159.250.29.171 -i $EXTDEV -p tcp -m tcp --dport 25 -j DNAT --to-destination 10.0.100.4

iptables -A FORWARD -d 10.0.100.4 -j ACCEPT

I tried adding some LOG, and then DROP entries for port 25 in the FORWARD and OUTPUT sections, but it seemed to kill everything (since I don't really understand the PREROUTING thing as much as I need to).

Thanks again for all the replies, I really appreciate it.

[dbiel]

One thing you could try is to put a limit on the number of email messages each user / ip address is able to send per day. If you have a user with a trojan, he will be blocked real fast and you will be hearing from him as to why he mail is being rejected.

[Wazoo]

I tried to follow the SpamHaus example of limiting port 25 to only the email server, but when added their code, no mail was coming in or out.

Best I could find there was a link to http://cbl.abuseat.org/nat.html ...????

Most documentation that I can find talks about the INPUT, FORWARD, and OUTPUT chains, but the guy that wrote this thing has many PREROUTING rules in here that I don't really understand fully yet.

Basically, PREROUTING are settings for packets that are not going to be filtered, as the first packet received with those 'details' are sent on their way without bothering with the rest of the rule-set.

The relative lines are these (10.0.100.4 is the email server's internal ip, 159.250.29.171 is external):

iptables -A PREROUTING -t nat -d 159.250.29.171 -i $EXTDEV -p tcp -m tcp --dport 25 -j DNAT --to-destination 10.0.100.4

Says that incoming Port#25 traffic gets routed directly to the NAT'd address/device at 10.0.100.4

iptables -A FORWARD -d 10.0.100.4 -j ACCEPT

Hmmm, doesn't quite jive with the above ..... this one seems to suggest that it will play with any traffic, happily throwing it at 10.0.100.4 ... yet in theory, this line shouldn't actually get seen by Port#25 traffic (assumed to be SMTP stuff) .. but why one would throw all other traffic that way ...???? Assumption would have to be that this is what is used to 'allow' web-browsers to make their connections to your squirrelmail interface????

However, the bottom line (as I see it) there's nothing there to "only allow" Port#25 outgoing traffic from the e-mail server.

I tried adding some LOG, and then DROP entries for port 25 in the FORWARD and OUTPUT sections, but it seemed to kill everything (since I don't really understand the PREROUTING thing as much as I need to).

The catch may be in the way those attempts were made???? Simple 'add' commands place new entries at the 'bottom' of the stack, tossing in a 'line' entry should place it somewhere in the middle of things, but ..... was the selection of 'where to put it' correct?

Iptables Tutorial 1.2.2 .... maybe skipping down to Chapter 13 or so might help ...????

[agsteele]

We thought we were taking care of it finally, but alas, we are back on your ban list (and others I imagine).

The problem with the port 25 thing in IPTABLES is that it's still a little over my head.

No disrespect intended but if this is "over your head" perhaps you need to buy in some support. Is the guy who set your idiosyncratic system still available for a few hours consultancy?

Andrew

[kamaraju]

Were you able to figure out how your system got hacked? Also, try checking the logs /var/log/auth.log to see if you have some unautheticated users logging into your system. The suggestion of limiting traffic per user or per computer is a good one IMHO. Implement strong passwords, disable "guest" login accounts. That might help you in the future...

hth

raju

[turetzsr]

...Good news (at least as of right now):

Query bl.spamcop.net - 159.250.29.8

<snip>

159.250.29.8 not listed in bl.spamcop.net

[Wazoo]

...Good news (at least as of right now):

Yeah, but ..... http://www.senderbase.org/senderbase_queri...ng=159.250.29.8

Volume Statistics for this IP

Magnitude Vol Change vs. Last Month

Last day ...... 0.0 .. N/A

Last month .. 3.7

Same as seen n my last data-point post. Therre's a number of possibilities, but only the Topic starter would (should) be able to explain.

[jeffslife]

Hey guys.

No, I can't really explain... heh. Sorry. I re-wrote some of the IPTABLES rules and we've been anti-virusing and anti-malware-tooling like mad.

Seems as though the logging isn't working quite right for some reason. I have a very specific rule in there to log anything that has anything to do with port 25 (and a couple other ports), but when I try to test anything on it, attempts never show up in the logs (the data over 25 successfully gets blocked though). And yah, I checked to make sure IPTABLES logs were being sent to syslog.

I'm hoping all the cleaned machines and the part of the firewall re-written will at least keep us unblocked while we continue to "put out the fires".

Thanks again for all the help. Hopefully you won't hear from me again, eh?

[Wazoo]

No, I can't really explain... heh.

Based on that remark, I'll suggest that moving to a different IP Address wasn't the 'solution' ... So the next best description would probably be that the traffic has dropped down to levels not 'measurable' by the SenderBase monitoring stations. From the perspective of bad spam traffic, this is great news. Congratulations!!

Seems as though the logging isn't working quite right for some reason. I have a very specific rule in there to log anything that has anything to do with port 25 (and a couple other ports), but when I try to test anything on it, attempts never show up in the logs (the data over 25 successfully gets blocked though). And yah, I checked to make sure IPTABLES logs were being sent to syslog.

I don't recall you stating what version of Ubuntu is running ... but of course that would also bring up the question of updating, patches, and such. Doing a "man iptables" here includes notes and things not necessarily stated elsewhere in some tutorials .... like the suggestion that it takes two rules to handle the (varioue levels of) logging ... one rule to log the results, the other rule to actually ACCEPT/REJECT/DROP/whatever the specified traffic.

Hopefully you won't hear from me again, eh?

Quite on the other hand. Once you get it all sorted out, a post intended for a FAQ/Wiki entry on what to look for, how to find it, and how to fix it would be great. Stopping in and seeing if you can help the next person that finds themselves in your same situation and helping them out would be much more apprciated.

[jeffslife]

Alas, I continue to fail. Somehow back on the list...

Ubuntu version is 6.06 LTS

[Wazoo]

Alas, I continue to fail. Somehow back on the list...

http://www.spamcop.net/w3m?action=blcheck&...ip=159.250.29.8

159.250.29.8 listed in bl.spamcop.net (127.0.0.2)

If there are no reports of ongoing objectionable email from this system it will be delisted automatically in approximately 13 hours.

Causes of listing

System has sent mail to SpamCop spam traps in the past week

Only spamtrap hits thus far, though the below suggests that user complaints are probably not long in showing up ...

http://www.senderbase.org/senderbase_queri...ng=159.250.29.8

Volume Statistics for this IP

Magnitude Vol Change vs. Last Month

Last day ...... 3.0 .. -79%

Last month .. 3.7

What are the odds that it turns out to be the 'same' computer that got re-infected/compromised?

Ubuntu version is 6.06 LTS

Running under the same verson ...

iptables -V

iptables v1.3.3

LOG
	   Turn on kernel logging of matching packets.  When this option is set for a rule, the Linux kernel  will  print
	   some information on all matching packets (like most IP header fields) via the kernel log (where it can be read
	   with dmesg or syslogd(8)).  This is a "non-terminating target", i.e. rule  traversal  continues  at  the  next
	   rule.   So  if you want to LOG the packets you refuse, use two separate rules with the same matching criteria,
	   first using target LOG then DROP (or REJECT).

	   --log-level level
			  Level of logging (numeric or see syslog.conf(5)).

pretty much the same language as what I recall under Ubuntu 8.04 used on a system here at the house.