OK. I am getting my butt whipped by a machine problem that seems it should be easy to find. Please pipe up with any suggestions on where to look next. I am about ready to image a new machine for this user, but I still want to find the cause...

Problem started last Friday about 1PM EDT. Part of our network started having dropouts of services. I was at another site but the people there tracked the problem down to one machine by visual inspection of the switch activity lights. Removing this machine from the network fixed all the other problems immeditely and stayed fine over the weekend.

Skip forward to Monday... I have the machine and scan for viruses using our Corporate Edition Norton, nothing found. Scan for Spyware using corporate Webroot Spysweeper, nothing found. Search the registry for strange RUn and RunOnce, etc. entries, all check out. Run msconfig and check out the .ini files, etc., nothing. Place the machine on our test network and do not notice anything unusual happening. Put the machine back on the network and let the user start working again, while monitoring the network. Works fine from 1PM Monday until ~3AM Tuesday, when services start dropping again.

At 7:30 AM, on-site person pulls the plug again (because of high activity) and services resume. I arrive at 8AM, again finding nothing unusual running (since the network drop caused the network port to drop everything). Repeating above scans show nothing. Use safety.live.com (MS scanner), Spybot Search & Destroy, and Ad-Aware which all show clean. Checking the firewall logs (for SMTP traffic from the machine, figuring some kind of remote spammer program) reveals that every 10 minutes since the attack started, while it is on the network, the machine makes an http request to a few different IP addresses(207.44.218.102, 209.123.181.7, and 87.118.102.71). The firewall does not list the actual command or directory being attempted. It also shows one connection each night to a server on the conversent.net

207.44.218.102 - Searches on the web for this IP address turns up nothing. It currently resolves to zsexygirls.com, a live web site.

209.123.181.7 - Searches on the web for this IP addresses only turns up one group response from 2004 (According to that post, at that time spamcop Tracking link: http://www.dumpsmarket.net Resolves to 209.123.181.7) (SpamCop version 1.3.4 © SpamCop.net, Inc. 1998-2004 All Rights Reserved, BTW). This site is currently running Apache/1.3.37 @ localhost

87.118.102.71 - Searches on the web for this IP address turns up nothing. This site is currently running Apache/2.0.54 (Fedora) @ secure-server.suroot.com. A lookup on that name does resolve back to the same IP address.

Firewall logs further shows no other attempts to those same 3 sites, so the problem seems localized to this machine. I have setup the firewall to outright drop connection attempts to those IPs. I put the computer back on the network and have not seen the network problem since. I assume this process is looking for a command from the website, then starts some sort of search on the local network, but I can find no indication of this.

If this sounds familiar to anybody, or if you see something I have missed, please let me know. I am trying to block a couple other sites that this machine has connected to once over noght, but lots of our machines connect to those servers (only since 03-sep-2006 if that makes a difference) and I'm not sure it is related or not. Those sites are both converent.net hosts.

Someone's bound to have an idea, if not you might also ask the question in one of the Gibson Research NGs (GRC.com) Steven. Seem to recall your boss doesn't let you "NG" from work though?

[added on edit - as it says somewhere in the www site "Simply aim your newsreader at news.grc.com" on account of the once web-based interface thing is now gone. Need to register to post (some complicated-sounding secure authentication process "The system uses up to the first 29 characters of any user-chosen "passphrase", so yours should probably be at least that long."). Suggesting this group as they seem to have some security/spyware/rootkit uprooting specialists]

http://www.grc.com/newsdown.htm ---- doesn't state that it's down permanently, but .....

NG is alive and well (just looked), but not through the web interface - only through the usual NG "subscription" thing as in my first post (the "added on edit" bit since it occurred to me a few minutes after posting it was a bit difficult to find the NG in a hurry from the website info ... hmmm, where have I heard that before?)

Probable order of things I would try after looking at what you've already done:

• Check the stats out on the switch port. See if there are errors on the interface.
• Switch out the network cable at the original location. As you weren't seeing problems on your test network, the patch cable at the original location could be bad.
• Switch out the NIC.
• Stick another machine on the network at the same physical location. May be a problem in the cable run between the box and your switch.
• Stick a sniffer on it and see WTF it's doing. Either on the switch if it's managed, or stick a hub between the machine and the switch and sniff it from another box on the hub.
• If you're concerned about the web activity. Throw a squid proxy up, set the machine up to send requests through that proxy and look at what it's doing in the squid logs.
• Format and re-install the machine from scratch.

From what you've already looked at it's probably either new and well hidden malware, or (most likely) a hardware problem.

edit to add switch stats. Obvious, but I forgot about it

Hi, Steven,
...I passed this article on to a very knowledgeable colleague of mine and here is his reply:HTH!

Thank you all. I am going to provide this user a new machine and continue to play with this in the background.

GraemeL: I believe the reason I did not have any problems on the test network is that it did not have internet access to get it's commands from. I just rebooted the machine on the live network with the http blocks in place and it acts the same (no network flooding).

You could give Ethereal a try. It's a Windows Winpcap based network sniffer. It might help you figure out what are in the packets or where they are coming from.

Secondly, you might want to give ewido a try. Spywareinfo.com's forum generally recommend running it after posting a copy of a hijackthis log. They have an online scan, and a 30 day trial, both of which are good. After the trial, it simply deactivates active monitoring. I have also heard good things about Prevx which can apparently find more malware than most comparative software.

Finally, you might want to consider that you could have a rootkit.



Hope that helps.

It might have. The AVG scanner found 2 items the other three scans missed.

downloader.small.cyn c:\windows\system32\syst0dr.dll
downloader.tiny.eg c:\windows\system32\syst670.dll

Currently, it is trying to access the internet every hour. I should know by noontime.

I have always found AVG AV very effective, often finding infections the other, more known packages miss. Now I find the same with their spyware package.

Following with interest, really appreciate your efforts in sharing this Steven. Neither of the above "known" to the db from http://www.programchecker.com/default.aspx - the "in situ" check online could be more definitive (not actually familiar with this product), possibly a morphed payload involved.

I am calling this issue resolved as the machine has not attempted to connect to the internet since cleaning it.

Thank you all once again.