Domino 5 - example agent to filter incoming mail

[Mandy Shaw]

Domino 5 (unlike Domino 6) provides no mechanism for specifying blocklists for incoming mail to be checked against. I am hoping to go to Domino 6 soon .... but in the meantime, I got fed up with all the spam and wrote a Domino Java agent that checks each mail item against the SCBL. Following much help from this forum in the past, I thought the least I could do would be to post the agent here. Details, and the code, follow. Feel free to use the code in any way you wish.

******

The agent checks incoming SMTP mails against the SpamCop blocklist (dynamically) and adds either a Spam_True or Spam_False item to each document (the former set to a text string showing the offending source IP address, the latter blank).

Getting at the multiple Received items is a nightmare - see assorted Lotus Knowledge Base articles. I eventually did it by opening the document, reading and deleting the Received items in turn (as recommended by the Knowledge Base), picking up the noteid, closing without saving (so as not to lose the Received items permanently), re-opening using the noteid, adding the Spam_True or Spam_False item and saving.

It goes through all docs in the ToBeChecked view in my mail file, defined as follows:

SELECT [at]IsAvailable(SMTPOriginator) & [at]IsUnavailable(Spam_True) & [at]IsUnavailable(Spam_False)

I have set it up to run on the server when any new mail arrives for me - here is example output from the server log where the document was clean:

20/06/2004 09:41:51 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: Received : from zen.co.uk ([212.23.8.66]) by mailhost.notability.com (Lotus Domino Release 5.0.8) with SMTP id 2004062021282850:260 ; Sun, 20 Jun 2004 21:28:28 +0100

20/06/2004 09:41:57 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: Received : from bay1-dav43.bay1.hotmail.com ([65.54.244.15] helo=hotmail.com) by schroedinger.zen.co.uk with esmtp (Exim 4.30) id 1Bc8wk-0002TA-Lm for mandy.Shaw[at]Notability.com; Sun, 20 Jun 2004 20:

20/06/2004 09:41:58 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: Received : from mail pickup service by hotmail.com with Microsoft SMTPSVC; Sun, 20 Jun 2004 13:29:45 -0700

20/06/2004 09:41:59 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: Received : from 217.155.8.94 by bay1-dav43.bay1.hotmail.com with DAV; Sun, 20 Jun 2004 20:29:45 +0000

20/06/2004 09:42:00 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: This document is not spam

and here is example output where a spam source was detected:

20/06/2004 10:43:25 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: Received : from zen.co.uk ([212.23.8.66]) by mailhost.notability.com (Lotus Domino Release 5.0.8) with SMTP id 2004062022424203:275 ; Sun, 20 Jun 2004 22:42:42 +0100

20/06/2004 10:43:26 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: Received : from c103110.upc-c.chello.nl ([212.187.103.110]) by feynman.zen.co.uk with smtp (Exim 4.30) id 1BcA6c-0003I8-AH for webmaster[at]notability.com; Sun, 20 Jun 2004 21:44:02 +0000

20/06/2004 10:43:27 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: spam source found at 110.103.187.212.bl.spamcop.net

20/06/2004 10:43:27 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: Received : from 176.111.74.225 by en934-t7.y760.precitool.de with DAV; Mon, 21 Jun 2004 00:38:22 +0200

20/06/2004 10:43:32 PM AMgr: Agent ('Check spam' in 'mail/mshaw.nsf') printing: This document is spam

I then have a spam view which picks up all documents with the Spam_True item in them. This spam view can then be processed according to personal preference (a lot of our users, accepting the risk of false positives, simply have it cleared automatically daily via another agent).

Agent follows (no rude comments about my rubbish Java, please). Basically it reads the ToBeChecked view as mentioned above. For each document it pulls the IP addresses out of the Received items and re-formats them as required for a SpamCop lookup (basically 1.2.3.4 is looked up as 4.3.2.1.bl.spamcop.net). If any match is found, the document is marked as spam. Any error in decoding a Received item is ignored (they come in all shapes and sizes and we are only trying to spot as many spams as we can ...)

You could add lookups on other blocklists (e.g. Spamhaus, open relay blocklists) to this if you liked.

It's not ideal - I would much prefer the checking to happen at the SMTP level - but it's a lot better than nothing.

******

import lotus.domino.*;

import java.net.*;

import java.util.*;

public class JavaAgent extends AgentBase

{

public void NotesMain()

{

try

{

System.out.println("Check spam started");

Session session = getSession();

AgentContext agentContext = session.getAgentContext();

Database currentDatabase = agentContext.getCurrentDatabase();

String view = "ToBeChecked";

Document doc = currentDatabase.getView(view).getFirstDocument();

while (doc != null)

{

boolean spamFound = false;

String spamFrom = null;

Item nextItem = doc.getFirstItem("Received");

while (nextItem != null)

{

boolean ipFound = true;

Vector values = nextItem.getValues();

if (values != null) for (int k=0;k<values.size();++k)

{

String received = values.elementAt(k).toString();

System.out.println("Received : " + received);

try

{

String fromAddress = received.substring(received.indexOf("from ") + 5,received.indexOf("by ")).trim();

String ipAddress = null;

if (fromAddress.endsWith(")"))

{

try

{

ipAddress = fromAddress.substring(fromAddress.indexOf("[") + 1, fromAddress.indexOf("]"));

} catch (StringIndexOutOfBoundsException sie) { ipAddress = fromAddress.substring(fromAddress.indexOf("(") + 1, fromAddress.indexOf(")")); }

}

else

{

ipAddress = fromAddress;

}

String ipAddressRest = ipAddress;

String ipAddress1 = ipAddressRest.substring(0,ipAddressRest.indexOf("."));

ipAddressRest = ipAddress.substring(ipAddress1.length()+1);

String ipAddress2 = ipAddressRest.substring(0,ipAddressRest.indexOf("."));

ipAddressRest = ipAddressRest.substring(ipAddress2.length()+1);

String ipAddress3 = ipAddressRest.substring(0,ipAddressRest.indexOf("."));

ipAddressRest = ipAddressRest.substring(ipAddress3.length()+1);

String ipAddress4 = ipAddressRest;

String checkableIpAddress = ipAddress4 + "." + ipAddress3 + "." + ipAddress2 + "." + ipAddress1 + ".bl.spamcop.net";

String checkedIpAddress = null;

try

{

checkedIpAddress = InetAddress.getByName(checkableIpAddress).getHostAddress();

} // try decoding IP address

catch (UnknownHostException he) { ipFound = false; }

if (ipFound)

{

spamFound = true;

spamFrom = checkableIpAddress;

System.out.println("spam source found at " + checkableIpAddress);

}

} // try processing ip address

catch (Exception e)

{

ipFound = false;

System.out.println("Error " + e + " found when parsing " + received);

}

} // if non-null item with values in it

nextItem.remove();

nextItem = doc.getFirstItem("Received");

} // while (items)

/* If spam found for this document, write out new field via new ODP (so we can save it) */

if (spamFound)

{

System.out.println("This document is spam");

}

else

{

System.out.println("This document is not spam");

}

String noteid = doc.getNoteID();

doc.recycle();

Document updateDoc = currentDatabase.getDocumentByID(noteid);

updateDoc.appendItemValue("Spam_" + spamFound, spamFrom);

updateDoc.save(false,false);

/* Get the first document in the view, which will be the next one */

doc = currentDatabase.getView(view).getFirstDocument();

} // while (documents)

} // main try

catch(NotesException e)

{

System.out.println("Domino exception in Check spam");

e.printStackTrace(System.out);

}

catch(Exception e)

{

System.out.println("Non-Domino exception in Check spam : " + e);

}

} // NotesMain

}

[Wazoo]

Moved to Lounge, as this has nothing to do with the Mail-Host configuration .... and we're still limited to these few areas to move things around in.

[Mandy Shaw]

Sorry, finger trouble.

[Wazoo]

no doubt .. that was a fair amount of typing ... and guessing that you went through it more than once <g>