Monday, 5 April 2010

Small J2EE Fedora Development Server On Amazon EC2.

In the last 10 years I used a LINUX server at home to develop my private applications. Twice a year I spend a weekend to update/upgrade the server. Now I should replace the old hardware with a newest one.

Instead of buying a new server I decided to search for alternative solutions. Due to the small amount of time I really use the server, the solution have to be cheap. At the same time, due to the kind of experiments I do, I need root access to the instance.

I signed up for Amazon S3 and EC2 to be able to run a LINUX AMI. The first question is: where may I find a Fedora AMI?

The best solution for me seems to be a self defined AMI built using Elastic Server. My server is based on Fedora 10 and contains the JDK 1.6.0_13, Tomcat 6.0.18, PHP 5.2.6, MySQL 5.0.67 and Apache2. Almost the same configuration I use at home.

Elastic Server build, deploys and starts the AMI automatically. I had to define a S3 bucked, something like daniele-ami where the service will copy the image of the AMI. The builder needs to access my account at Amazon to store the image and start the AMI, so I define a dedicated Access Key. As soon as the new instance works I disable the dedicated key. The key may be re-enabled each time I want to deploy a new AMI.

Elastic Server sends an email to the account owner containing all the informations to access the the running instance.

The login (cftuser) may be performed using PuTTY (Windows) or ssh (UNIX). To gain root access I have to use the command sudo su - . A quick check of the environment confirms that everything works as expected.

Add a Data Volume

Since I don't work with a reserved instance when the instance terminates all changes made after the startup are lost. To reduce the inconvenience I define a 10 GB EBS volume using the AWS console. Note: the volume have to be defined in the same zone as the EC2 instance.

The volume is attached as device sdf (/dev/sdf)

Now the procedure is the same used to add a disk to any LINUX system:

  1. Use fdisk (fdisk /dev/sdf) to delete all existing partitions (select d).
  2. Save the new (empty) partition table (select w).
  3. Add a new partition (select n).
  4. Save the new partition table (select w).
  5. Use mkfs (e.g. mkfs -t ext3 /dev/sdf1) to format the patition.
  6. Use tune2fs (e.g. tune2fs -L /data /dev/sdf1) to put a label on the patition.
  7. Create the mount point (e.g. mkdir /data).
  8. Modify the file /etc/fstab as displayed into the next section.
  9. Mount the partition (e.g. mount /data).
# /etc/fstab: static file system information.
# AMI version
#
# 'file system' 'mount point'   'type'  'options'       'dump'  'pass'
/dev/sda1       /               ext3    defaults,errors=remount-ro 1    1
/dev/sda2       /mnt            ext3    defaults        0       0
/dev/sda3       none            swap    sw              0       0
proc            /proc           proc    defaults        0       0
none            /sys            sysfs   defaults        0       0
none            /dev/shm        tmpfs   defaults        0       0
none            /dev/pts        devpts  gid=5,mode=620  0       0
LABEL=/data     /data           ext3    defaults        1       2

After a new start the last three steps of the previous list have to be done again to remount the disk. I do it using the following script:

#!/bin/bash
mkdir /data
echo "LABEL=/data     /data           ext3    defaults        1       2" >> /etc/fstab
/bin/mount /data

Execute:

  • sudo su -
  • vi setup
  • copy & paste
  • sh setup

The cost for an afternoon of programming and testing is $0.40 which is really cheap.

Wednesday, 27 January 2010

MVC on the Google App Engine with Spring 3.0

On the Google App Engine (GAE) I can deploy and test applications very quickly. What I need now is a programming "toolkit" wich is essential and efficient.
I thought about using Grails but there is a lot of dependencies I don't want.
An implementaion based solely on the Servlet API wolud be essential but time consuming.
I decide to use the last release of the Springframework to see if it is possible to set up a project template for the GAE.

The Application Template

At this time I need two modules:
  • home with the pages "home", "about" and "description".
  • stage with some different controllers service each a page called "list" and a page called "details".
Home
This section is display-only and supports the GET method only. "Welcome" contains the list of the RSS feeds into the HEADER, but if something goes wrong the feed are simply missing. No error message is displayed.
Stage
The stage is a restricted area containing lists and forms.
This section implements the feed registration where "list" contains a short summary of all the available feeds. From this page I may add or delete a feed.
The "details" page is a form containing all the fields. The feed may be modified or deleted.

The Project Layout

The different parts of the application have to be easily identified on the filesystem. By convention the path of the java packages is build on a prefix (application.web) and using the name of the section:
application.web.home 
application.web.stage
Each section has at least one controller. The name is composed by the section name and the word controller.
application.web.home.HomeController 
application.web.stage.StageController
The stage sections contains more controllers one for each editor with restricted access. At this time I have the only the "feed" editor.
application.web.stage.FeedController
I use a similar convention for the views (JSP pages). The views are placed under WEB-INF/views:
WEB-INF/views/home/about.jsp 
WEB-INF/views/home/description.jsp 
WEB-INF/views/home/home.jsp 
 
WEB-INF/views/stage/stage.jsp 
WEB-INF/views/stage/feed.jsp 
WEB-INF/views/stage/feedDetails.jsp
The whole layout was created using Maven ( mvn archetype:generate ). I did merge "maven-archetype-quickstart" and "maven-archetype-webapp".

The Controllers

The home controller is quite simple:
package ch.clx.application.web.home;
...
@Controller
@RequestMapping("/home")
public class HomeController {
    
  @RequestMapping(method = RequestMethod.GET)
  public String get(final ModelMap model) {
    ...
    model.addAttribute("version", Version.version);
    return "home";
  }
    
  @RequestMapping(value="/about", method = RequestMethod.GET)
  public String about(final ModelMap model) {
    model.addAttribute("version", Version.version);
    return "about";
  }
    
  @RequestMapping(value="/description", method = RequestMethod.GET)
  public String description(final ModelMap model) {
    model.addAttribute("version", Version.version);
    return "description";
  }
}
The feed controller is a simple CRUD editor. The GET method returns the list of the feeds (the feed view).
  • Add (C) creates a new entry and displays the form with all the fields (detail view).
  • Delete (D) removes an existing entry.
  • Edit (R) displays an existing entry (detail view).
  • Store (U) updates an existing entry.
package ch.clx.application.web.stage;  
...  
@Controller  
@RequestMapping("/stage/feed")  
public class FeedController {  
      
  @RequestMapping(method = RequestMethod.GET)  
  public String get(final ModelMap model) {  
    ...  
    model.addAttribute("version", Version.version);  
    return "feed";  
  }  
      
  @RequestMapping(value = "/add", method = RequestMethod.POST)  
  public String add(  
    @RequestParam("id") final String id,   
    final ModelMap model) {  
    ...  
    model.addAttribute("version", Version.version);  
    return "feedDetails";  
  }  
      
  @RequestMapping(value = "/delete", method = RequestMethod.POST)  
  public String add(  
    @RequestParam("id") final String id,   
    final ModelMap model) {  
    ...  
    model.addAttribute("version", Version.version);  
    return "feed";  
  }  
      
  @RequestMapping(value="/edit", method = RequestMethod.GET)  
  public String edit(  
    @RequestParam("id") final String id,   
    final ModelMap model) {  
    ... 
    model.addAttribute("version", Version.version);  
    return "feedDetails";  
  }  
      
  @RequestMapping(value="/store", method = RequestMethod.GET)  
  public String store(  
    @RequestParam("id") final String id, 
    ..., 
    final ModelMap model) {  
    ... 
    model.addAttribute("version", Version.version);  
    return "feed";  
  }  
}

The Context File

The context file is very simple because components and URL routing are automatically discovered through annotations.

<?xml version="1.0" encoding="UTF-8"?> 
<beans 
  xmlns="http://www.springframework.org/schema/beans" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xmlns:p="http://www.springframework.org/schema/p" 
  xmlns:context="http://www.springframework.org/schema/context" 
  xsi:schemaLocation=" 
    http://www.springframework.org/schema/beans  
    http://www.springframework.org/schema/beans/spring-beans-3.0.xsd  
    http://www.springframework.org/schema/context 
    http://www.springframework.org/schema/context/spring-context-3.0.xsd" 
> 
<bean 
  class="org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping"/> 
<bean 
  class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter"/> 
 
<context:component-scan base-package="gae.application"/>

<bean 
  id="viewResolver" 
  class="org.springframework.web.servlet.view.ResourceBundleViewResolver"/>

</beans>

The Deployment Descriptor

web.xml
<?xml version="1.0" encoding="UTF-8"?> 
<web-app 
  id=2gae-application" 
  version="2.4" 
  xmlns="http://java.sun.com/xml/ns/j2ee" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee  
      http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"/> 
  <display-name>gae-application</display-name> 
  <servlet> 
    <display-name>Dispatcher Servlet</display-name> 
    <servlet-name>Dispatcher-Servlet</servlet-name> 
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class> 
    <init-param>
      <param-name>contextConfigLocation</param-name>
      <param-value>/WEB-INF/WebCtx.xml</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping
    <servlet-name>Dispatcher-Servlet</servlet-name>
    <url-pattern>/bo/*</url-pattern>
  </servlet-mapping>
  <welcome-file-list>
    <welcome-file>index.jsp</welcome-file>
  </welcome-file-list>
</web-app>

The GAE Deployment Descriptor

appengine-web.xml
<?xml version="1.0" encoding="UTF-8"?> 
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0"> 
<application>gae-application</application> 
<version>beta-1</version> 
<system-properties> 
<property 
  name="java.util.logging.config.file" 
  value="WEB-INF/classes/logging.properties"/>
</system-properties> 
</appengine-web-app>

The Maven Archetype

At the end of my experiment I pack my template application in a Maven archetype. I is enough to add the following pom.xml file to the project:
<?xml version="1.0" encoding="UTF-8"?>
<project
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
    http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>cloud.gae.application</groupId>
  <artifactId>gae-application</artifactId>
  <version>1.0</version>
  <packaging>war</packaging>
  ...
  <build>
    <finalName>${artifactId}</finalName>
    ...
  </build>
</project>
Now type mvn archetype:create-from-project to generate the archetype. Change the directory ( cd target/generated-sources/archetype ) and execute mvn install .
Using mvn archetype:generate now I may generate a GAE application ready to use. It does nothing but as starter kit for GAE applications is not bad.

Saturday, 16 January 2010

URL Shortening Service

A URL shortening service is not essential for the most part of the applications in the WEB and may be dangerous because the surfer doesn't see what lies behind the short URL. But if I would like to publish something on a short messaging system like Twitter I need a very short URL.

The solution would be to register a very short domain like goo.gl and to run my own shortener, but for my application the additional effort is out of scope. So I need to connect to an existing URL shortening service.

The beginner has a first obstacle: where to find an inexpensive reliable URL shortening service. Using a search engine I did found a lot of service which are also free of charge, but no way to proof if the are also reliable.

Service Requirements

Since I want to generate the short URL into the background I need a service with API support. Additionally stating from the short URL I want to obtain the original (long) URL.

The candidates

I didn't want to check all the services, for instance four of them are enough.

bit.ly

The service is free of charge, but  requires a registration and checks each request using an API key.

  • Homepage with a clear description of the service.
  • Simple well documented API.
  • Shorten and expand functions.
  • JSON and XML formatted response.
  • Simple but nice statistics summary.
  • Blog

I like bit.ly and is currently my first choice.

is.gd

The service is free of charge and doesn't require a registration.

  • Homepage with a clear description of the service.
  • Very simple API with documentation.
  • Shorten only function.
  • Plain text (URL only) response.

This service is very spartan; I noticed that after two or three days the short URL was different. If a very short URL is required id.gd may be a good choice. 

snipurl.com

The service is free of charge, but requires a registration and checks each request using an API key.

  • Homepage with a clear description of the service.
  • Simple well documented API (Firefox doesn't display the page and with the MSIE there are some curious side effects, try with is-there-a-snipr-snipurl-api).
  • Shorten and expand functions.
  • XML and plain text (URL only) formatted response.
  • Simple statistics but searchable statistics.
  • Help

SnipURL exists since the year 2000, which is a sign of reliability. I miss the JSON formatted response, but the service is good.

tinyurl.com

The service is free of charge and doesn't require a registration.

  • Homepage with a clear description of the service.
  • Very simple API.
  • Shorten only function.
  • Plain text (URL only) response.

The service is very spartan, but the short URL is the same over the time.

Test the services

I did test this services for a while using a very simple HTTP client.

Client Requirements

The client have to be essential, but should be able to:

  • retry the connection three times before to give up.
  • follow the redirects (max three levels of indirection).
  • support GET & POST.
  • support the secure socket layer (SSL, nice to have).

Implementation

I use a very simple implementation done in Java and based on the HttpURLConnection, the following snippet shows the essential part of the code:

...
URL url = URL("http://myserver.com");
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setRequestMethod("GET");
conn.addRequestProperty("User-Agent","FastShortGood/1.0 Java/6");
...
// Extract the character set from the "Content-Type"
// header using a regular expression. If none available
// assume "UTF-8"
...
StringBuilder buf = new StringBuilder();
InputStream is = conn.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is, charset));
String line;
while (null != (line = br.readLine())) {
  buf.append(line).append("\n");
}
String body = buf.toString();

// Extract the short URL from response body using a
// regular expression.

For the complete implementation I need about 300 lines of code. The most of them handles exceptions and HTTP respose codes. The lightweight client works well with all of the shortener services.

Sunday, 3 January 2010

Receiving Emails with the Google App Engine

Last year, in November, I did explore for the first time the Google App Engine for Java (GAE/J). One of the feature my application provides is to accept an email and translate it to an RSS 2.0 item.
The first version of the receiver was done using JavaMail. I did experiment some troubles with the "Quoted-Transfer-Encoding", sometimes the content was broken.
Using the current version of the GAE/J (1.3.0) the emails content received from Gmail or from Thunderbird is correct and the channel seems to be reliable.

Mime4J

An alternative for receiving email is the James Mime4J project.
The library doesn't have dependencies and works on the GAE/J. It is quite simple to use; instead of:

import java.io.InputStream;
import java.util.Properties;
import javax.mail.Session;
import javax.mail.internet.MimeMessage;
...
InputStream input = request.getInputStream(); // ServletRequest
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
MimeMessage message = new MimeMessage(session, input); // MessagingException
...
  
you have to use:
import java.io.InputStream;
import org.apache.james.mime4j.message.Message;
import org.apache.james.mime4j.message.MessageBuilder;
import org.apache.james.mime4j.parser.ContentHandler;
import org.apache.james.mime4j.parser.MimeStreamParser;
import org.apache.james.mime4j.storage.MemoryStorageProvider;
...
InputStream input = request.getInputStream(); // ServletRequest
Message message = new Message();
ContentHandler handler = new MessageBuilder(message, new MemoryStorageProvider());
MimeStreamParser parser = new MimeStreamParser();
parser.setContentHandler(handler);
parser.parse(input); // MimeException
...
The result seems to be better than the one which uses JavaMail. The mails received from the MS Outlook are readable. It remains a small problem with the encoding which I will investigate later this year.

Incoming Mail Syntax

The syntax of the incoming mail was changed:
description
line 1
line 2
...
line n
end

link
http://myserver/mylink
end

categories
category 1
category 2
...
category n
end
Each command block starts with a keyword and ends with end

Informations

The email address of the Mime4J service is mime4j@example-rest.appspotmail.com, the JavaMail based receiver still works with rss2@example-rest.appspotmail.com
The application URL is example-rest.appspot.com.

Thursday, 19 November 2009

First Application on the Google App Engine

My first application on the Google App Engine for Java generates a dynamic RSS 2.0 feed.

Motivation

Since I like to investigate the behaviour of some different RSS 2.0 feed consumers; I need a tool which allows the quick publication over the Internet.

Description

Using an E-Mail client I send a message to the application. The message is persisted into the data store and will be used to build the RSS feed as soon as a consumer requests it.

Additional Libraries

Just the Springframework (2.5.6), but core and MVC only.

Components

  • A small RSS domain model: the class RSS2Channel and RSS2Item.
  • A Mail2RSS2 controller (the infrastructure is provided by Spring).
  • A RSS2 controller (the infrastructure is provided by Spring).
  • A DAO layer (access to the data store using low level API).

Environment

I did start with Eclipse and the GAE/J plug-in, but I feel comfortable with Maven and batch processing. I work with Maven 2.2.1, SUN JDK 1.6_16 and GAE SDK 1.2.6. I downloaded the GAE SDK for Java and placed the bin directory in my PATH.

  • Add the files appengine-api-1.0-sdk-1.2.6.jar and appengine-api-labs-1.2.6.jar to the Maven repository. The files are into the lib/user folder.
  • Add the same files as dependency in the POM file. The scope is compile.
  • Add the files appengine-api-stubs.jar and appengine-local-runtime.jar to the Maven repository. The files are into the lib/impl folder.
  • Add the same files as dependency in the POM file. The scope is test.

With this environment Maven works. The classes can be compiled and the unit tests are successful.

Debugging

The command line environment doesn't provide a debugging version out of the box. I did install Eclipse and I added the m2eclipse pug in. The next step was to import the Maven project.

On Windows I start the engine using:

@%JAVA_HOME%\bin\java -cp "%INST_DIR%\lib\appengine-tools-api.jar" -Dlog4j.configuration="%INST_DIR%\config\user\llog4j.properties" -javaagent:%INST_DIR%\lib\agent\appengine-agent.jar -agentlib:jdwp=transport=dt_socket,server=y,address=5300 com.google.appengine.tools.development.DevAppServerMain %*
On UNIX I start the engine using:
$JAVA_HOME/bin/java -ea -cp "$INST_DIR/lib/appengine-tools-api.jar" -Dlog4j.configuration="$INST_DIR/config/user/llog4j.properties" -javaagent:$INST_DIR/lib/agent/appengine-agent.jar -agentlib:jdwp=transport=dt_socket,server=y,address=5300 com.google.appengine.tools.development.DevAppServerMain $*

The JVM stops an wait on IP port 5300 for the debugger. I switch to Eclipse and open the debug configuration dialog. I define a new remote java application configuration (localhost, port 5300) for the project and start debugging. Now I'm ready for debugging.

Start up

The first attempt to start the application was not successful because I did use the Resource annotation which is not into the white list.

Note: it is impossible to keep in mind which class is allowed and which is not. The developer, sometimes, must have imagination to find a good alternative and avoid the restriction. The good news is that the environment told me about the error using a clear statement.

Looking at the logs I discovered that the application is stopped after a short idle and restarted by the subsequent request. This is an additional requirement for the application: it will be an advantage to have an implementation which doesn't loose time with long initializations.

Receiving Mails

I did define some rules to be able to convert the incoming mail to an RSS item:

  • The subject of the mail is the title of the item.
  • The description has to be marked with the [Description] TAG.
  • The link has to be marked with the [LinkTo] TAG.
  • The categories have to be marked using the [Categories] TAG.
  • The separator char for the categories is #.
  • The mail has to contain a plain text part.

Example:

[Description]Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.[Description]
[LinkTo]http://en.lipsum.com/[LinkTo]
[Categories]lorem ipsum#specimen[Categories]

At the beginning the local environment posted a ByteArrayInputStream as content of the MimeMessage and not a Multipart object. After adding javax.mail (1.4) to my POM the local GAE returns a Multipart object and behaves as expected.

The run time environment works but if I send a mail using the MS Outlook 2007 there are some troubles. If there is a line which is longer than 78 characters, the mail seems to be incomplete. I suspect that is a problem with the content type encoding "quoted-printable". I will investigate, but not right now.

Data Store

Accessing the data store using the low level API is simple and doesn't cause any problem. I did define a simple data access layer (DAO) so I may move a part of the code tested with the GAE to other projects.

Note: due to a bug I inserted a non printable char into the data store (key name), the Data Viewer became unusable (application error) until deleted the entity with the application.

Summary

The environment is very simple and efficient. The GAE console is a good tool to monitor the behaviour of the application. For small sized applications and tools which use anyway Google services may be a good alternative to a conventional application service provider.

My application is hosted at example-rest.appspot.com and the email address is rss2@example-rest.appspotmail.com.

Saturday, 15 August 2009

Test Title To Setup The Formatting

H1 Title

Test Paragraph

H2 Title

Test Paragraph

H3 Title

Test Paragraph

H4 Title

Test Paragraph


code line 1
code line 2
code line 3

Test Paragraph

Title


  • List item 1
  • List item 2

Blogger