[Resolved] Spam problem, senders use fake headers?

[johnm1]

Hello,

First of all, sorry for my bad english.

My mail server got reported, but the reported spam mail doesn't seems to be send from my mail server.

I checked all the logs and i really cant find the message or any of the strange xxxx[at]destip.nl messages.

All the bounced mail / spam mail have aol servers in the headers.

A copy of the message reported by spamcop message:

> [ Offending message ]

> Return-Path: <www[at]noxa.nl>

> Received: from rly-yc04.mail.aol.com (rly-yc04.mail.aol.com

> [172.18.205.147]) by air-yc01.mail.aol.com (v107.13) with ESMTP id

> MAILINYC14-1d4438bb6b1c3; Mon, 28 Nov 2005 21:02:37 -0500

> Received: from mx1.noxa.nl (ns1.noxa.nl [82.192.89.201]) by rly-

> yc04.mail.aol.com (v107.13) with ESMTP id MAILRELAYINYC48-

> 1d4438bb6b1c3; Mon, 28 Nov 2005 21:02:31 -0500

> Received: by mx1.noxa.nl (Postfix, from userid 80)

> id EB840170D3; Tue, 29 Nov 2005 02:56:24 +0100 (CET)

> To: x

> Subject: astonishment9857[at]destip.nl

> From: UnknownSender[at]UnknownDomain

> X-AOL-ORIG-From: "astonishment9857[at]destip.nl" <him>

> Content-Type: text/plain; charset=\"us-ascii\"

> MIME-Version: 1.0

> Content-Transfer-Encoding: 7bit

> Subject: Companies positioned to move

> Message-Id: <2005_________________70D3[at]mx1.noxa.nl>

> Date: Tue, 29 Nov 2005 02:56:24 +0100 (CET)

> X-AOL-IP: 82.192.89.201

> X-AOL-SDI: PROFILE

>

>

> UNDERVALUED SPECIAL SITUATION -- Huge Appreciation Potential!

> .... etc etc...

The server mx1.noxa.nl (ns1.noxa.nl) is my server, "destip.nl" is a customer of me.. Also lots of this kind of mail got bounced to my account (<catchall>[at]noxa.nl, orig: www[at]noxa.nl)

Even after disabling the "destip.nl" accounts it still goes on.

Is there anybody who knows this kind of problems ?

i use FreeBSD with postfix + clamav + spamassasin

For me it looks like some kind of spammer uses fake headers.

Is there anybody with the same problem ?

Help urgent needed...

[StevenUnderwood]

Submitted: Wednesday, November 30, 2005 8:38:13 AM -0500:

Subject: astonishment9857[at]destip.nl

This is the ONLY report against that IP address listed for us mere mortals.

Received: from mx1.noxa.nl (ns1.noxa.nl [82.192.89.201]) by rly-yc04.mail.aol.com (v107.13) with ESMTP id MAILRELAYINYC48-

1d4438bb6b1c3; Mon, 28 Nov 2005 21:02:31 -0500

If this line is forged, then the server air-yc01.mail.aol.com has become under the control of spammers.

Sorry for dropping this message in the middle....had to get home for the kids. Looks like others have gone in the direction I was heading, so will not complete as of right now.

[Miss Betsy]

Received: from mx1.noxa.nl (ns1.noxa.nl [82.192.89.201]) by rly-

> yc04.mail.aol.com (v107.13) with ESMTP id MAILRELAYINYC48-

> 1d4438bb6b1c3; Mon, 28 Nov 2005 21:02:31 -0500

this is the header line that points to 82.192.89.201

If you cannot find it in your regular logs, try looking at other ports. It may mean that there is an infected computer on your network.

Those who are server administrators may help you more on what to do.

Miss Betsy

[turetzsr]

First of all, sorry for my bad english.

37299[/snapback]

Hi, johnm1!

...First of all, let me apologize for knowing only English which prevents me from being able to reply to you in your first language. Second of all, I found your English to be at least as good as the average Yank! <g>

My mail server got reported, but the reported spam mail doesn't seems to be send from my mail server.

I checked all the logs and i really cant find the message or any of the strange xxxx[at]destip.nl messages.

<snip>

37299[/snapback]

...Please review the "SpamCop FAQ" (link above) entry labeled "I'm receiving spam reports, but my mail server logs don't reflect it. Why?" under the heading "Assistance stopping spam." It may have some information you will find useful.

... Good luck!

[johnm1]

Thanks for your advice.

The server is a colocated server, dedicated for mail only.

There are still coming lots of bounced messages [at]destip.nl

I think i will disable all outbound e-mail, and see if there are still coming (new) messages.

If there are people who want to test my server for relay, let me know. Im willing to pay for it.

[Telarin]

Lets start here:

> Received: from  rly-yc04.mail.aol.com (rly-yc04.mail.aol.com

>    [172.18.205.147]) by air-yc01.mail.aol.com (v107.13) with ESMTP id

>    MAILINYC14-1d4438bb6b1c3; Mon, 28 Nov 2005 21:02:37 -0500

this is normal movement of email from one of AOLs intermediate mail servers to its final destination server. Not much use to us.

> Received: from  mx1.noxa.nl (ns1.noxa.nl [82.192.89.201]) by rly-

>    yc04.mail.aol.com (v107.13) with ESMTP id MAILRELAYINYC48-

>    1d4438bb6b1c3; Mon, 28 Nov 2005 21:02:31 -0500

This is where the email entered AOLs mail system, clearly coming from 82.192.89.201. The only way for this IP to be faked is by the receiving server, so as Steven pointed out, the AOL server would have to be under the control of the spammers. Not very likely.

> Received: by mx1.noxa.nl (Postfix, from userid 80)

>  id EB840170D3; Tue, 29 Nov 2005 02:56:24 +0100 (CET)

Hmm, this is a bit suspicious here. Received line with no from. Possibly an open relay? I would check your mail server settings and verify that you don't have an open relay. I'm not terribly familiar with Postfix, but bet you can find more info on making sure it is not an open relay with a quick google search.

Also, as someone else mentioned, if you are using NAT to have multiple computers share a single IP address, any one of said computers infected with a virus could be zombieing (can I use that as a verb?) spam.

The server mx1.noxa.nl (ns1.noxa.nl) is my server,  "destip.nl" is a customer of me.. Also lots of this kind of mail got bounced to my account (<catchall>[at]noxa.nl, orig: www[at]noxa.nl)

Hmm, interesting

mx1.noxa.nl = 82.192.89.203

ns1.noxa.nl = 82.192.89.201

Two different IPs.

Do you have seperate incoming and outgoing mailservers, or just different IPs on the same server for some reason?

I would say that you can be fairly certain that the mail came from your IP. Further, looking at the headers its a good bet it actually came from your mail server (even if you are using NAT to share public IPs). First thing would be to check and make sure the server is not infected with something, and to make absolutely certain that it is not an open relay.

After that, check your passwords, make sure you don't have someone logging into a weakly passworded admin account at night and sending to their hearts desire. Most hacker/spammers are smart enough to clean up the logs when they are done, so you would be unlikely to find any trace in the logfiles on that computer.

[johnm1]

Hi, turetzsr !

Thanks for your advice, i read the FAQ before.. and also tried disabling sending mail trough our servers for "destip.nl", it didn't work.

We started logging a while ago, outbound e-mail and all rejected / bounced messages.

There wasn't any send message from "destip.nl", There were a lot of incoming spam mail and rejected mail, but all incoming.

Then we inspected the computer of my customer.. all seems to be clean, no spyware / viruses / rootkit viruses, hacksoftware.

In outlook he even never used our server for outgoing mail, he used the one of his internet provider.

It is really freaking me out, cause the only names used for the spam are "destip.nl" names, none of my other customers have this problem.

At this moment my colleague is checking the Squirrelmail environment.... i dont think that spammers found a way to hack into webmail enviroments but you never know.

Thanks for all the hard thinking and response.

[Telarin]

The server is a colocated server, dedicated for mail only.

When you say colocated, do you mean it is shared between multiple users? If this is the case, it may be that another user who has access to the server is using it to send their spam. If so, you would want to refer that to your ISP as quickly as possible.

[johnm1]

Hi, Telarin !

Thanks for your response.

It is indeed strange that it was send by ns1.noxa.nl.. same server, but it should send mail from mx1.

I will check that, but i think it just slipped in by changing so much things to find out where the spam came from.

The server looks to be clean, we also changed the passwords to be sure.

Tomorrow morning i will try just disabling all outbound mail.

It's bedtime for me now (in Netherland it is 11:45 PM so.. bedtime )

I think it must be a verificated user.. but now it is the trick to find out who.

[johnm1]

The server is Collocated..

There are about 10 company's on that server.

We share the costs of the server..

So in fact we are the ISP self.

Me and some friend manage the server.

[Telarin]

Ahh, in that case, I would start with the other companies using the server. Make sure none of them are sending spam without you knowing. You'd be surprised how dishonest some people can be when it comes to business.

[johnm1]

Yes sometimes it could happen..

Last week we ran a check on all computers on all company's.. for spyware, hacktools etc.

All systems are clean.

So:

OR somebody is sending mail and not telling.. (but still strange that all other mail is in the logs and this isnt)

OR somebody has given his account info to somebody else (or somebody knows a username and password)

I with 2 other system managers (best friends) are the only one who can login to the server and change stuff / make new accounts etc.) so that wouldnt be the problem.

[Miss Betsy]

OR somebody is sending mail and not telling.. (but still strange that all other mail is in the logs and this isnt)

If it isn't in the mail logs, it may be in firewall logs.

I with 2 other system managers (best friends) are the only one who can login to the server and change stuff / make new accounts etc.) so that wouldnt be the problem.

There are exploits in Exchange that let someone else get in and change stuff. There is a FAQ on that.

Perhaps if you give more information on what kind of server you are using? I am not a server admin so I can't give more help. IP addresses cannot be 'spoofed' so there is some problem that you have not discovered.

Miss Betsy

[Snowbat]

82.192.89.201 probably HELOs as "mx1.noxa.nl" - the HELO/EHLO string is typically inserted at that point in the header.

I suggest you take a look at /etc/passwd on 82.192.89.201 and find out who is userid 80. Userids below 500 are normally assigned to system accounts. postfix is userid 73 on my system. You may find that userid 80 is the postfix 'user' on 82.192.89.201 but if not, be suspicious.

Check /var/log/mail/info and (or equivalent on your system) for clues.

grep EB840170D3 /var/log/mail/info

If the system runs logrotate, EB840170D3 data may have been rotated so check info.1.gz, info.2.gz etc.:

for i in /var/log/mail/info.*; do gunzip -c $i | grep EB840170D3; done

The headers certainly point to injection by a local user account on 82.192.89.201 (either real user or compromised software). Check for rootkits. Change root password or key, restrict user logins, firewall all non-essential ports and turn off all non-essential services.

Good hunting.

[johnm1]

We found the problem.

The mail is send by the php engine.. so there must be a hacked or abused php scri_pt.

We're searching right now..

[johnm1]

Hi Missbetsy,

You are right it must be somewhere in anykind of log.. We found it in the php log so checking it right now.

Hi Snowbat,

Thanks for the tips, the "for i in var/log..." is very usefull, i was just doing it the basic way.. saves lots of time with your scri_pt.

I let you know when i found out more.

[Miss Betsy]

I am glad that you have probably found the problem. We are looking forward to hearing all is well!

And thanks to Snowbat for giving john some 'real' technical help!

Miss Betsy

[johnm1]

The problem is solved !

A php mail scri_pt seems to be abused on our second server.

The scri_pt had a problem wit <cr><lf> injections.. as all others scripts on that server have.. so fixing the problem on all scripts right now.

The second server has an trusted connection to the mail server and was not logged at all.

I think zombie computers were used to post.

68-112-178-169.dhcp.fdul.wi.charter.com - - [07/Dec/2005:01:33:45 +0100] "POST /contact.php HTTP/1.0" 200 3557 "http://www.destip.nl/"'>http://www.destip.nl/" "-"

68-112-178-169.dhcp.fdul.wi.charter.com - - [07/Dec/2005:01:33:49 +0100] "POST http://www.destip.nl/contact.php HTTP/1.1" 200 1615 "http://www.destip.nl/"'>http://www.destip.nl/" "-"

61.84.16.157 - - [07/Dec/2005:02:27:22 +0100] "POST /contact.php HTTP/1.1" 200 3549 "http://www.destip

and lots more of this in the webserver log.

most of the request done by ppp-82-3-217-212.dialup.iam.net.ma

It seems that it is been going on a very long time...

Hope the fix will work (replacing the characters)

[turetzsr]

...In light of johnm1's latest post, I have marked this article as "Resolved."