Receiving Emails with the Google App Engine

Last year, in November, I did explore for the first time the Google App Engine for Java (GAE/J). One of the feature my application provides is to accept an email and translate it to an RSS 2.0 item.
The first version of the receiver was done using JavaMail. I did experiment some troubles with the "Quoted-Transfer-Encoding", sometimes the content was broken.
Using the current version of the GAE/J (1.3.0) the emails content received from Gmail or from Thunderbird is correct and the channel seems to be reliable.

Mime4J

An alternative for receiving email is the James Mime4J project.
The library doesn't have dependencies and works on the GAE/J. It is quite simple to use; instead of:

import java.io.InputStream;
import java.util.Properties;
import javax.mail.Session;
import javax.mail.internet.MimeMessage;
...
InputStream input = request.getInputStream(); // ServletRequest
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
MimeMessage message = new MimeMessage(session, input); // MessagingException
...
  

you have to use:

import java.io.InputStream;
import org.apache.james.mime4j.message.Message;
import org.apache.james.mime4j.message.MessageBuilder;
import org.apache.james.mime4j.parser.ContentHandler;
import org.apache.james.mime4j.parser.MimeStreamParser;
import org.apache.james.mime4j.storage.MemoryStorageProvider;
...
InputStream input = request.getInputStream(); // ServletRequest
Message message = new Message();
ContentHandler handler = new MessageBuilder(message, new MemoryStorageProvider());
MimeStreamParser parser = new MimeStreamParser();
parser.setContentHandler(handler);
parser.parse(input); // MimeException
...

The result seems to be better than the one which uses JavaMail. The mails received from the MS Outlook are readable. It remains a small problem with the encoding which I will investigate later this year.

Incoming Mail Syntax

The syntax of the incoming mail was changed:

description
line 1
line 2
...
line n
end

link
http://myserver/mylink
end

categories
category 1
category 2
...
category n
end

Each command block starts with a keyword and ends with end

Informations

The email address of the Mime4J service is mime4j@example-rest.appspotmail.com, the JavaMail based receiver still works with rss2@example-rest.appspotmail.com
The application URL is example-rest.appspot.com.

First Application on the Google App Engine

My first application on the Google App Engine for Java generates a dynamic RSS 2.0 feed.

Motivation

Since I like to investigate the behaviour of some different RSS 2.0 feed consumers; I need a tool which allows the quick publication over the Internet.

Description

Using an E-Mail client I send a message to the application. The message is persisted into the data store and will be used to build the RSS feed as soon as a consumer requests it.

Additional Libraries

Just the Springframework (2.5.6), but core and MVC only.

Components

• A small RSS domain model: the class RSS2Channel and RSS2Item.
• A Mail2RSS2 controller (the infrastructure is provided by Spring).
• A RSS2 controller (the infrastructure is provided by Spring).
• A DAO layer (access to the data store using low level API).

Environment

I did start with Eclipse and the GAE/J plug-in, but I feel comfortable with Maven and batch processing. I work with Maven 2.2.1, SUN JDK 1.6_16 and GAE SDK 1.2.6. I downloaded the GAE SDK for Java and placed the bin directory in my PATH.

• Add the files appengine-api-1.0-sdk-1.2.6.jar and appengine-api-labs-1.2.6.jar to the Maven repository. The files are into the lib/user folder.
• Add the same files as dependency in the POM file. The scope is compile.
• Add the files appengine-api-stubs.jar and appengine-local-runtime.jar to the Maven repository. The files are into the lib/impl folder.
• Add the same files as dependency in the POM file. The scope is test.

With this environment Maven works. The classes can be compiled and the unit tests are successful.

Debugging

The command line environment doesn't provide a debugging version out of the box. I did install Eclipse and I added the m2eclipse pug in. The next step was to import the Maven project.

On Windows I start the engine using:

@%JAVA_HOME%\bin\java -cp "%INST_DIR%\lib\appengine-tools-api.jar" -Dlog4j.configuration="%INST_DIR%\config\user\llog4j.properties" -javaagent:%INST_DIR%\lib\agent\appengine-agent.jar -agentlib:jdwp=transport=dt_socket,server=y,address=5300 com.google.appengine.tools.development.DevAppServerMain %*

On UNIX I start the engine using:

$JAVA_HOME/bin/java -ea -cp "$INST_DIR/lib/appengine-tools-api.jar" -Dlog4j.configuration="$INST_DIR/config/user/llog4j.properties" -javaagent:$INST_DIR/lib/agent/appengine-agent.jar -agentlib:jdwp=transport=dt_socket,server=y,address=5300 com.google.appengine.tools.development.DevAppServerMain $*

The JVM stops an wait on IP port 5300 for the debugger. I switch to Eclipse and open the debug configuration dialog. I define a new remote java application configuration (localhost, port 5300) for the project and start debugging. Now I'm ready for debugging.

Start up

The first attempt to start the application was not successful because I did use the Resource annotation which is not into the white list.

Note: it is impossible to keep in mind which class is allowed and which is not. The developer, sometimes, must have imagination to find a good alternative and avoid the restriction. The good news is that the environment told me about the error using a clear statement.

Looking at the logs I discovered that the application is stopped after a short idle and restarted by the subsequent request. This is an additional requirement for the application: it will be an advantage to have an implementation which doesn't loose time with long initializations.

Receiving Mails

I did define some rules to be able to convert the incoming mail to an RSS item:

• The subject of the mail is the title of the item.
• The description has to be marked with the [Description] TAG.
• The link has to be marked with the [LinkTo] TAG.
• The categories have to be marked using the [Categories] TAG.
• The separator char for the categories is #.
• The mail has to contain a plain text part.

Example:

[Description]Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.[Description]
[LinkTo]http://en.lipsum.com/[LinkTo]
[Categories]lorem ipsum#specimen[Categories]

At the beginning the local environment posted a ByteArrayInputStream as content of the MimeMessage and not a Multipart object. After adding javax.mail (1.4) to my POM the local GAE returns a Multipart object and behaves as expected.

The run time environment works but if I send a mail using the MS Outlook 2007 there are some troubles. If there is a line which is longer than 78 characters, the mail seems to be incomplete. I suspect that is a problem with the content type encoding "quoted-printable". I will investigate, but not right now.

Data Store

Accessing the data store using the low level API is simple and doesn't cause any problem. I did define a simple data access layer (DAO) so I may move a part of the code tested with the GAE to other projects.

Note: due to a bug I inserted a non printable char into the data store (key name), the Data Viewer became unusable (application error) until deleted the entity with the application.

Summary

The environment is very simple and efficient. The GAE console is a good tool to monitor the behaviour of the application. For small sized applications and tools which use anyway Google services may be a good alternative to a conventional application service provider.

My application is hosted at example-rest.appspot.com and the email address is rss2@example-rest.appspotmail.com.