Daniele's Blog

Inexpensive Development E-Mail Server

2012-01-22T16:00:00.000+01:00

A common issue using the company e-mail server for development is SPAM which, sooner or later, will reach the customer or a third party. To avoid the problem I did use a very good dummy e-mail server the Dumbster. In May, last year, I had to send HTML formatted mails from my application and test the result on the most common E-Mail clients.

The Dumbster wasn't enough anymore, but I didn't want to install/configure a true e-mail server and maintain it. My new choice is Apache James because it is quite simple to setup and use. It supports MSOutlook and Thunderbird (and many other which I didn't really check).

I'm hosting the the code I'm describing here @github. To run the script successfully you have to place the files configure.sh, apache-james and conflib into the same folder.

Customization

The core of the implementation is the procedure apache_james_v2 (see conflib). The script check if the SUN JDK is already installed, if not try to install it. To skip this step comment out the line containing sun_java at the beginning of the procedure.

The server will be started using the newly created account of the user james. The user james doesn't have the permission to bind a listener to a port less than 1024 so I have to increase the port numbers as follow:

. . . 
sed -i s/'<port>110<\/port>'/'<port>4110<\/port>'/g SAR-INF/config.xml
sed -i s/'<port>25<\/port>'/'<port>2525<\/port>'/g SAR-INF/config.xml
sed -i s/'<port>119<\/port>'/'<port>4119<\/port>'/g SAR-INF/config.xml
. . .

The next step is to authorize the IP segment of the development workstations:

. . . 
sed -i s/'
  <authorizedAddresses>
  127.0.0.0\/8
  <\/authorizedAddresses>'/'<authorizedAddresses>
  127.0.0.0\/8,192.168.1.0\/24
  <\/authorizedAddresses>
  '/g SAR-INF/config.xml 
. . .

The next step adds the mail domains which should be considered as local

. . .
  fgrep mail.canistracci.oil SAR-INF/config.xml >/dev/null
  if [ "$?" != "0" ]; then
    sed -i '/<servername>localhost<\/servername>/ a\
      <servername>mail.canistracci.oil<\/servername>' SAR-INF/config.xml
  fi
. . .

This code snippet may be repeated for each mail domain which should be handled by James using the syntax mail.<my domain>.

Operations

Using the command sudo service apache-james start James starts-up all configured sections. With stop at the end James terminates the operations and status help you to find out if James is running or not.

Now I have to set up the mail box for my user. First I log in using a telnet client connected on the IP port 4555 as user root with the password root.

Typing adduser petrol petrol, I add the user petrol with the password petrol.

Now I may connect MSOutlook or Thunderbird (new account: POP port 4110, SMTP port 2525) to the POP account petrol/petrol.

As soon as my application sends a mail addressed to petrol@canistracci.oil to the port 2525 my mail client will be able to display it.

Backup of the PostgreSQL Database

2011-04-22T21:38:00.000+02:00

At the end of the project I had to set up a simple but reliable backup for the PostgreSQL database. If you have some interest, I did describe the environment in a previous article "Ubuntu Server as Inexpensive Platform for J2EE". Here I describe the backup on that platform. With less changes the scripts works also on other UNIXes. The principle may be also used on MS Windows, but the implementation will be different.

I'm working on a server called canistracci.oil with the account petrol. I did create a database called drilling with the user offshore and the user backup.

The users

---
--- file name: 01_create_user.sql
---
--- please execute as postgres only, 
--- and be sure to be connected to the postgres database
---
--- The database owner
---
CREATE ROLE offshore LOGIN ENCRYPTED PASSWORD '99!cents'
   VALID UNTIL 'infinity';
COMMENT ON ROLE offshore IS 'drilling read / write user';
---
--- The backup user
---
CREATE ROLE backup LOGIN ENCRYPTED PASSWORD '50!cents'
   VALID UNTIL 'infinity';
COMMENT ON ROLE drilling IS 'drilling backup user';

The Tablespace

The folder /data/postgresql/drilling have to be created in advance using the commands: mkdir -p /data/postgresql/drilling and chown postgres:postgres as root at the UNIX console (or shell).

---
--- file name: 02_create_tablespace.sql
---
--- please execute as postgres only, 
--- and be sure to be connected to the postgres database
---
CREATE TABLESPACE drilling OWNER drilling LOCATION '/data/postgresql/drilling';

COMMENT ON TABLESPACE drilling IS 'drilling database tablespace';

The Database

---
--- file name: 03_create_database.sql
---
--- please execute as postgres only, 
--- and be sure to be connected to the postgres database
---
CREATE DATABASE drilling
  WITH ENCODING='UTF8'
       OWNER=offshore
       TEMPLATE=template1
       LC_COLLATE='en_US.UTF-8'
       LC_CTYPE='en_US.UTF-8'
       CONNECTION LIMIT=-1
       TABLESPACE=drilling;
COMMENT ON DATABASE drilling IS 'drilling database';

The Tables

The tables of the database drilling are: equipment, location, tenant. The details are out of scope.

The Grants
The user offshore is the owner of the database and doesn't need special grantsbut backup is a read-only technical user which needs at least the SELECT grant on every object we wants to backup.

---
--- file name: 05_grants.sql
---
--- please execute as postgres only, 
--- and be sure to be connected to the drilling database
---
GRANT CONNECT ON DATABASE drilling TO backup;
GRANT USAGE ON SCHEMA public TO backup;
---
--- PostgreSQL < 9
---
--- GRANTS ON TABLES generated with
---
--- SELECT 'GRANT SELECT ON ' || tablename || ' TO backup;' FROM pg_tables WHERE schemaname = 'public';
--- psql -qAt -c "select 'GRANT SELECT ON ' || tablename || ' TO backup;' FROM pg_tables WHERE schemaname = 'public'" | psql
---
GRANT SELECT ON equipment TO backup;
GRANT SELECT ON location TO backup;
GRANT SELECT ON tenant TO backup;
---
--- Introduced by PostgreSQL 9
---
--- GRANT SELECT ON ALL TABLES IN SCHEMA public TO backup;
--- ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO backup;
---
--- GRANTS ON SEQUENCES generated with
---
--- SELECT 'GRANT SELECT ON ' || relname || ' TO backup;' FROM pg_statio_user_sequences WHERE schemaname = 'public';
---
GRANT SELECT ON equipment_id_seq TO backup;
GRANT SELECT ON location_id_seq TO backup;
GRANT SELECT ON tenant_id_seq TO backup;

Trusting Backup
I add a local trust to the database configuration for the user backup editing the file /etc/postgresql/8.4/main/pg_hba.conf

petrol@canistracci.oil $ sudo nano /etc/postgresql/8.4/main/pg_hba.conf
. . .
local   all         postgres                          md5
local   drilling    backup                            trust
. . .
petrol@canistracci.oil $ sudo /etc/init.d/postgresql restart

Now, using pg_dump I may dump the content of the database. The command line is the following: sudo pg_dump -E UTF-8 -U backup -w drilling.

The Backup Script
The script exports the database as very long PG-SQL script which may be used torestore the database. After the export the PG-SQL script is compressed to thebinary ZIP format using the best possible compression.

#!/bin/sh
# backup.sh    This script stores all valuable parts of the database.
#
# description : 1) Performs the backup of the drilling database
# process name: backup.sh
#
# Version 1.0 Daniele Denti

LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH

BACKUP_DIR=/data/backup/drilling

NOW=`date +'%Y_%m_%d'`
TIMESTAMP=${1:-$NOW}

BACKUP_LOG=/var/tmp/drilling.backup.$TIMESTAMP.log
DB_DUMP=${BACKUP_DIR}/drilling.$TIMESTAMP.sql
PG_ZIP_FILE=$BACKUP_DIR/drilling_$TIMESTAMP.zip

CURDIR=`pwd`

echo [`date +'%d.%m.%Y %T'` INFO $0 ] started backup.sh on `uname -n`, executed by `id`, process id = $$ >$BACKUP_LOG

pg_dump --column-inserts -E UTF-8 -f ${DB_DUMP} -U backup -v -w drilling >>$BACKUP_LOG 2>&1

cd ${BACKUP_DIR}
zip -9rv $PG_ZIP_FILE ${DB_DUMP} >>$BACKUP_LOG 2>&1

echo [`date +'%d.%m.%Y %T'` INFO $0 ] backup.sh completed >>$BACKUP_LOG

cd $CURDIR

The Clean Up Script
The backup produces new files every day, so it is necessary to regularly delete the older ones.

#!/bin/sh
# cleanup.sh   This is the daily script for cleaning the drilling environment
#
# description : - Performs the removal of old (at least 14 days) SQL and ZIP files.
#               - Performs the removal of old (at leas 7 days) log files.
# process name: cleanup.sh
#
# Version 1.0 Daniele Denti

BACKUP_DIR=/data/backup/drilling

FILE_LIST="*.log *.sql *.zip"
for FILE_TYPE in ${FILE_LIST}; do
  find ${BACKUP_DIR} -name ${FILE_TYPE} -atime +14 -exec rm {} \; >/dev/null 2>&1
done

find /var/tmp -atime +7 -exec rm -f {} \; >/dev/null 2>&1

The Cron Table
On a freshly installed Ubuntu server execute as root echo ALL >>/etc/cron.deny and echo petrol >>/etc/cron.allow.
In order to edit the cron table execute export EDITOR=vi; crontab -l

# m h  dom mon dow   command
0 22 * * * /home/petrol/bin/backup.sh 2>&1
0  5 * * * /home/petrol/bin/cleanup.sh 2>&1

The Remote Script
Now the backup works but the server is into the DMZ and the backup files are confined on the data disk. If something goes wrong the data may be lost. So I copy the data into the Intranet on a second Ubuntu Server named pipeline.oil using the account grease.

The authorization of grease@pipeline.oil on petrol@canistracci.oil is described in the previous article "Key-Based SSH Logins".

#!/bin/sh
# remote.sh    This is the daily script to get the drilling backup on the
#              Intranet central backup server.
#
# description : Gets the last backup from remote.
# process name: remote.sh

# Version 1.0 Daniele Denti

BACKUP_DIR=/data/backup/drilling
BACKUP_REMOTE=/data/backup/drilling

NOW=`date +'%Y_%m_%d'`
TIMESTAMP=${1:-$NOW}

PG_ZIP_FILE=drilling_$TIMESTAMP.zip
REMOTE_LOG=/var/tmp/drilling.remote.$TIMESTAMP.log

OIL_USER=petrol
OIL_SERVER=canistracci.oil

echo [`date +'%d.%m.%Y %T'` INFO $0 ] started remote.sh on `uname -n`, executed by `id`, process id = $$ >$REMOTE_LOG

if [ -f ${BACKUP_REMOTE}/${PG_ZIP_FILE} ]; then
        rm -f ${BACKUP_REMOTE}/${PG_ZIP_FILE}
fi

scp ${OIL_USER}@${OIL_SERVER}:${BACKUP_REMOTE}/${PG_ZIP_FILE} ${BACKUP_DIR}/${PG_ZIP_FILE} >> $REMOTE_LOG 2>&1

echo [`date +'%d.%m.%Y %T'` INFO $0 ] remote.sh completed >>$REMOTE_LOG

Inexpensive Development Platform for J2EE

2011-02-26T18:30:00.105+01:00

Once the server is ready (see my previous article) I need a similar inexpensive development environment. Inexpensive it's OK but uncomfortable would be bad. I have two different choices: the first is to create a small development corner on my office workstation. The second would be to buy a a net-book and use it as dedicated appliance.

The first option has the advantage of a powerful hardware and a large screen. The net-book is small and may be used everywhere. I decided to investigate both environments because the first is suitable for developing a new project or a major release and the second seems to be suitable for support and small changes.

The components I did choose for my environment works and are available for Windows, MacOSX and Linux. I'm really working with this environment on Windows/Linux clients. I don't use a MacOSX but some of my colleagues at work uses all this pieces of software without any problems on different Macs.

Installed Software

Java Development Kit (JDK), the last one, currently 1.6.0_24. (all systems)
PuTTY (ssh connectivity, Windows only).
GIT (all systems).
Tortoise GIT (Windows only).

These software packages are installed on each client for better integration into the environment. To avoid additional documentation I always perform a standard installation. For GIT on Windows I prefer the integration into the windows CMD-Box, with PuTTY and no integration with the Windows explorer (this integration is done by Totoise GIT). I describe my SSH setup here.

Portable Software

WinSCP ( ssh connectivity, Windows only).
Apache Maven
Eclipse STS

I did copy this software on a memory stick (2GB). The performance is acceptable also on the net-book. For project which don't rely on security at all, the stick may be replaced by a service like Dropbox.

Configurations

Windows Environment

SET JAVA_HOME=C:\Program Files\Java\jdk1.6.0_24
SET M2_HOME=E:\java\apache-maven-3.0.2
SET PATH=%PATH%;%JAVA_HOME%\bin;%M2_HOME%\bin

UNIX Environment

JAVA_HOME=/usr/local/java/jdk1.6.0_24
M2_HOME=/usb-mnt/java/apache-maven-3.0.2
PATH=${PATH}:${JAVA_HOME}/bin;${M2_HOME}/bin
export JAVA_HOME M2_HOME PATH

Maven Settings

The file settings.xml have to be placed into the folder .m2 which was created into the home directory of the user on both Windows and UNIX.


  [usb-mnt]/java/m2/repository
  
    
      default
      
        
          org-matheclipse-repository
          http://symja.googlecode.com/svn/maven-repository/
          
            true
          
          
            false
          
        
      
    
  
  
    default

The Source Code
The source code was checked out under [/usb-mnt]/gitroot/[maven-project] using GIT. I did open a CMD-Box/Terminal-Window into this folder and I did execute mvn clean install.

The IDE
I did start the Eclipse STS from the memory stick, after setting my preferences I noticed that the Maven configuration was already OK. After importing the Maven project I could start working as usual.

Conclusion
The environment is quite simple but efficient and enough to program and maintain any J2EE application. The development corner on the strong office workstation doesn't collide in any way with the main purpose of the workstation.

The net-book environment is better than expected. I had to be patient during the installation and the first start of the IDE, but, once started, also the IDE is usable and the small screen doesn't affect my productivity.

I did work a while with the net-book only and without any additional device like an external screen or a mouse. Using the tethering function of my smartphone I was able to connect to my servers from everywhere and travelling by train.

The net-book is light and wakes up fast and did cost 170€ including the USB stick.

The Net-Book

I did work with two net-books the first an ASUS Eee 1001PX and the second a Samsung NP-N150 Plus, both with Intel Atom 450 CPU, 1GB memory, 250 GB HD and Windows 7 Starter.

Key-Based SSH Logins

2011-02-05T20:10:00.010+01:00

I did found a lot of pages with good explanation of the theory and / or how to setup client and server and I don't want to write another one. The motivation of this article is only to have a concise check list to use with "copy & paste" when I have to set up a such login.

The Ecosystem

I have a UNIX server called canistracci.oil holding a bare GIT repository, accessible through the account petrol. Another UNIX server called pipeline.oil (account grease) should access the repository of canistracci.oil for continuous integration. The development workstation (gas-station.oil) is a modern Windows 7 PC and should also access the GIT repository on canistracci.oil.

The Software

On Solaris 10, Linux and MacOSX the SSH package is ready to use. Windows doesn't provide SSH connectivity. I use the excellent PuTTY terminal by Simon Tatham. I did download the Windows installer and I did a standard installation.

Authorize Grease

First connect to pipeline.oil. From MacOSX, Linux or Solaris execute ssh grease@pipeline.oil, on Windows use PuTTY and then execute the following commands.

grease@pipeline.oil $ ssh-keygen -b 2048 -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/grease/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grease/.ssh/id_rsa.
Your public key has been saved in /home/grease/.ssh/id_rsa.pub.
The key fingerprint is:
x6:68:xx:93:98:8x:87:95:7x:2x:4x:x9:81:xx:56:94 grease@pipeline.oil

grease@pipeline.oil $ scp ~/.ssh/id_rsa.pub petrol@canistracci.oil:
grease@pipeline.oil $ ssh petrol@canistracci.oil
petrol@canistracci.oil's password:

petrol@canistracci.oil $ mkdir ~/.ssh
petrol@canistracci.oil $ chmod 700 ~/.ssh
petrol@canistracci.oil $ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
petrol@canistracci.oil $ rm ~/id_rsa.pub
petrol@canistracci.oil $ chmod 600 ~/.ssh/authorized_keys

The next login of grease on canistracci.oil doesn't require a password anymore.

Notice: On Solaris 10 the name of the file authorized_keys is authorized_keys2. The last Ubuntu server I did install (10.10) authorizes a MacOSX using authorized_keys2.

Authorize Gas-Station.oil

Start the program PuTTYgen.exe.
Generate the key-pair.
Save both keys on the hard-disk.
Start the program PuTTY.exe and connect to canistracci.oil.

Execute the following commands:

petrol@canistracci.oil $ vi ~/id_rsa.pub

Copy the public key (in blue on the picture above) to the clipboard, activate the PuTTY window, type i and paste the key. Type the sequence ESC + : + wq!

petrol@canistracci.oil $ chmod 700 ~/.ssh
petrol@canistracci.oil $ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
petrol@canistracci.oil $ rm ~/id_rsa.pub
petrol@canistracci.oil $ mkdir ~/.ssh
petrol@canistracci.oil $ chmod 600 ~/.ssh/authorized_keys

The next login of grease on canistracci.oil doesn't require a password anymore.

Notice: on Solaris 10 the name of the file authorized_keys is authorized_keys2. The last Ubuntu server I did install (10.10) authorizes a MacOSX using authorized_keys2.

Notice: on Windows before the first connection a double click on the PuTTY private key is necessary to activate the program Pageant.exe and to load the key.

Ubuntu Server as Inexpensive Platform for J2EE

2010-11-01T10:41:00.120+01:00

Due to the very small budget of the project I had to find a good but inexpensive platform. The development server is an old single core single CPU Pentium 4 pizza-box with 200 GB hard disk. The productive environment will be much better, but OS and middle-ware should work out-of-the-box and require a very little maintenance.

I decided to try with Ubuntu 10.10 Server because I was impressed by the essential and very good on-line documentation. The productive environment will be (later) installed with Ubuntu 10.04 LTS.

The installation is quite simple: download the ISO image, burn the CD-ROM and boot the server from the CD. There is no special task to do, but be sure the server has a working internet connection and, if you work in a LAN, the DHCP server have to be reachable. I did choose to install OpenSSH and PostgreSQL as additional services.

At the end of the installation I performed an update of the packages (sudo apt-get update/upgrade). After reboot (due to new kernel) Tomcat (6.0.28) and PostgreSQL are up and running. Low memory usage and no unwanted services.

Second Hard-disk

Since the installer takes care of the first disk only I have to prepare and mount the second disk:

Using fdisk (fdisk /dev/sdb) delete all existing partitions (if any).
Save new (empty) partition table (using w).
Add new partition(s) (using n).
Using mkfs (e.g. mkfs -t ext4 /dev/sdb1) format the partition(s).
Using tune2fs (e.g. tune2fs -L /data /dev/sdb1) label the partition(s).
Create the mount point(s) (e.g. mkdir /data).
Modify the file /etc/fstab adding one line per partition (e.g.)
/dev/sdb1 /data ext4 defaults 1 2
Mount the partition(s) (e.g. mount /data).

Tomcat

First add the admin and the manager roles to the file /etc/tomcat6/tomcat-users.xml and add the role to the tomcat user.

<?xml version="1.0"?>
<tomcat-users>
...
  <role rolename="admin"/>
  <role rolename="manager"/>
  <user username="tomcat" password="tomcat" roles="admin,manager"/>
</tomcat-users>

Restart Tomcat (/etc/init.d/tomcat6 restart).

Using this configuration I may access the manager (http://myserver:8080/manager/html) to quickly (re)deploy my application(s).

PostgreSQL

Perform the following setup (copied from the Ubuntu official documentation):

"Now that we can connect to our PostgreSQL server, the next step is to set a password for the postgres user. Run the following command at a terminal prompt to connect to the default PostgreSQL template database:"

sudo -u postgres psql template1

"The above command connects to PostgreSQL database template1 as user postgres. Once you connect to the PostgreSQL server, you will be at a SQL prompt. You can run the following SQL command at the psql prompt to configure the password for the user postgres."

ALTER USER postgres with encrypted password 'your_password';

"After configuring the password, edit the file /etc/postgresql/8.4/main/pg_hba.conf to use MD5 authentication with the postgres user":

local all postgres md5

"Finally, you should restart the PostgreSQL service to initialize the new configuration. From a terminal prompt enter the following to restart PostgreSQL":

sudo /etc/init.d/postgresql-8.4 restart

Apache2

The Apache WEB server wasn't installed by default (maybe I omitted to include it during the installation). Anyway with

sudo apt-get install apache2

the service was installed and started.

In order to use it as reverse-proxy there is some configuration work to do. The first step is to enable the proxy module:

user@myserver: cd /etc/apache2/mods-enabled
user@myserver: sudo ln -s ../mods-available/proxy.conf proxy.conf
user@myserver: sudo ln -s ../mods-available/proxy_http.load proxy_http.load
user@myserver: sudo ln -s ../mods-available/proxy.load proxy.load

The second step is to edit the proxy configuration (proxy.conf) as shown above:

<IfModule mod_proxy.c>

# If you want to use apache2 as a forward proxy, uncomment the
# 'ProxyRequests On' line and the <Proxy *> block below.
# WARNING: Be careful to restrict access inside the <Proxy *> block.
# Open proxy servers are dangerous both to your network and to the
# Internet at large.
#
# If you only want to use apache2 as a reverse proxy/gateway in
# front of some web application server, you DON'T need
# 'ProxyRequests On'.

ProxyRequests Off
<Proxy *>
  AddDefaultCharset off
  Order deny,allow
  Allow from all
</Proxy>

# Enable/disable the handling of HTTP/1.1 "Via:" headers.
# ("Full" adds the server version; "Block" removes all outgoing Via: headers)
# Set to one of: Off | On | Full | Block
#ProxyVia Off

</IfModule>

The third step is to edit the configuration of the default site (/etc/apache2/sites-available/default) as follows:

<VirtualHost *:80>
  ServerAdmin me@work

  ProxyPreserveHost On
  ProxyPass /manager/html !
  ProxyPass host-manager/html !
  ProxyPass / http://localhost:8080/
  ProxyPassReverse / http://localhost:8080/
  ServerName myserver

  ErrorLog ${APACHE_LOG_DIR}/error.log

  # Possible values include: debug, info, notice, warn, error, crit,
  # alert, emerg.
  LogLevel warn

  CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

At the end check the configuration and restart the apache2:

user@myserver: apachectl configtest
user@myserver: sudo /etc/init.d/apache2 restart

Static IP

The last task I had to do was to move the server from DHCP to a static address. Be aware: if you follows this instructions you have to be logged-in at the console of the server.

I did stop the network interface (sudo ifdown eth0) and start to edit the file /etc/resolv.conf as follow:

nameserver 192.168.1.11
domain mydomain
search mydomain

The next file to edit is /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
# auto eth0
# iface eth0 inet dhcp
auto eth0
iface eth0 inet static
address 192.168.1.2
netmask 255.255.255.0
gateway 192.168.1.1
#
# The secondary network interface
#
# auto eth1
# iface eth1 inet dhcp

I did restart the network interface (sudo ifup eth0) and the server was reachable through the static address.

Conclusion

After an hour I was able to start deploying my application on this recycled server. It is essential but enough for many small and mid sized applications. The same work may be done on an Amazon EC2 instance or inside VMWare.

Network Management on Fedora 12

2010-10-08T16:12:00.003+02:00

Recently I had to activate a second network interface on a Fedora 12 installation at work. The server has two interfaces (eth0 and eth1) but only one (eth0) was active and allowing connections from in-house only. The project wishes to provide a preview to the customer so I had to move the server to the DMZ. To avoid collisions between my work as system administrator and the development team I decided to activate the second interface(eth1) for the DMZ.

The last time I did this task was on Fedora 8 or 9; after adding the second interface as usual, nothing did happen. Using the command ifconfig -a I noticed that the adapter was NOT configured at all.

The Network Subsystem

The network subsystem I know is disabled (chkconfig --list | grep network) and down. So every network configuration and tool related to this subsystem doesn't change the network behaviour of Fedora anymore.

The Network Manager

The network manager replaces the network subsystem (chkconfig --list | grep NetworkManager). The network manager is great if you have direct access to the system console (Notebook, Workstation) and you know what you do. I'm accessing the server using an old fashioned TTY connection and I cannot use the Gnome applet.

The ncftool

The ncftool is a part of Netcf a cross platform configuration library. It wasn't installed on the server but as soon as I did type the command Fedora asked me if I wish install the package.

How to add the adapter

First of all I had to put together all information I needed to configure the adapter:

The static IP address of the server
The network mask of the IP segment
The IP address of the gateway
The MAC address of the adapter (optional, but recommended)
The IP adresses of the DNS servers.

The next step was to put all the informations in a XML file which was used by the ncftool to configure the new adapter.

<?xml version="1.0"?>
<interface type="ethernet" name="eth1">
  <start mode="onboot"/>
  <mac address="00:22:19:83:77:1B"/>
  <protocol family="ipv4">
    <ip address="192.168.1.111" prefix="24"/>
    <route gateway="192.168.1.1"/>
  </protocol>
</interface>

The example should be self-explanatory; the prefix is the network mask (24-bit or 255.255.255.0). The MAC address is delivered by the command ifconfig -a (HWaddr). I did store this XML structure into the file /tmp/eth1.xml. Some more interesting samples configurations are available at netcf.git.

At this point I did configure the network interface:

[root@myserver] # ncftool
ncftool > define /tmp/eth1.xml
ncftool > ifup eth1
ncftool > quit

The interface did work but it couldn't resolve the domain names. I didn't find a way to configure the DNS servers using the ncftool so I had to edit the file /etc/sysconfig/network-scripts/ifcfg-eth1:

DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.111
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
TYPE=Ethernet
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
NAME="System eth1"
UUID=9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04
HWADDR=00:22:19:83:77:1B
DNS1=192.168.1.11

The last line was manually added. The syntax for a DNS entry is DNS(n)=xxx.xxx.xxx.xxx. Here it is necessary to restart the network mananger (/etc/init.d/NetworkManager restart). From remote I had to boot the entire server.

The new adapter started to work properly for outgoing connections, but some incoming connections were refused. The reason was the missing IP-tables configuration. I had to edit the file /etc/sysconfig/iptables:

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A INPUT -i eth0 -m state --state NEW -m tcp -p tcp 7001 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

I added the lines 11,12 and 13 which enables connections on the ports 80 and 443 from everywhere to any interface and connections on the port 7001 from everywhere to the interface eth0. Since eth0 is reachable from in-house only, nobody can connect to the port 7001 from the Internet. After changing the rules it is necessary to restart the sub system (/etc/init.d/iptables restart).

Android 2.2 Froyo on the HTC Desire: first experiences

2010-08-08T17:48:00.028+02:00

I did upgrade my HTC Desire from Android 2.1 to 2.2. It's possible to force an upgrade without waiting (settings -> about phone -> system software updates -> check now).

Before you start the upgrade be sure your Desire have at least 25MB free memory or the update will fail. The entire procedure lasts about 10 minutes.

Software Informations

Android version: 2.2
Baseband version: 32.43.00.32U_5.09.00.20
Kernel version: 2.6.32.15-gf9c0527 - htc-kernel@and18-2#1
Build number: 2.09.405.8 CL218634 release-keys
Software number: 2.09.405.8
Browser version: Webkit 3.1

Apps

I did found 5 new applications:

App Sharing : it makes possible to share an application using mail or social network.
Flashlight : uses the flash to convert the phone in a flashlight.
News and Weather: app based on the Google News service.
Voice Search : it searches the words you speak using the Google Search.
Wi-Fi Hotspot : it converts the phone in a Wi-Fi hotspot. One or more notebooks may use, wireless, the 3G connection of the Desire to connect to the Internet.

Features

Support for Flash 10.1: it makes possible to watch television and to play streaming videos embedded into WEB pages. E.g Zattoo works fine with Wi-Fi and with 3G (default browser or Dolphin HD), but be aware: if you don't have a flat rate 3G may become really expensive. My Desire doesn't start to play the videos automatically, at the beginning a green down arrow appears. The video starts after I push the arrow avoiding unwanted costs for embedded videos I don't want to see.
Resources Management: now it is possible to move the applications to the SD chip, but, currently, only a minor part of the applications I use may really be moved to the SD. Anyway I could double the available space on the internal phone storage. The TaskKiller displays also twice so much available memory as before.
Performance: I didn't notice the performance improvement, but I don't use anything which requires a lot of power. Yesterday, during my last getaway, the battery did last remarkably longer as before.

Holidays in Italy with Android

2010-07-23T16:32:00.007+02:00

Last month I did spend some time searching the best rate for using my smartphone in Italy.

It is not only a question of money, the kind of connection is important too.

Using a swiss prepaid card roaming and data roaming are offered at unfair rates (1.10/1.70 Sfr each minute and 15.40 Sfr each MB) without any advice about the availabilty of the signal.

My investigation is based on the homepages of TIM, WIND and Vodafone. I decided to use the prepaid card offered by TIM.

The reasons are:

The offer was easy to find.
The description is clear, easy to understand and up-to-date.
I did easily find a page with the technical informations about my target area.
The tariff (2 Euro for 250MB weekly) is fair.
The additional costs (50 cts per MB) are tolerable.

Buy the Prepaid Card

You need to know the system: not every reseller may sell the card and activate it. You need to find a "centro TIM". Don't expect a huge store, it is mostly a small shop with the required license.

I was lucky and could buy the card on Saturday, late afternoon.

The card has a minimal amount of money preloaded. So you need a "ricarica". The additional credit is sold by small shops called "tabaccheria" and, sometimes, by supermarkets.

You may ask the official reseller, but, normally the answer is: "I'm sorry but we just sold the last one".

Card Setup

I had to wait 4 hours for the activation of the number. The service is good, the only problem was the "ricarica": the number to call did change, but I was informed about the change by TIM.

Eperiences

I was on the hills of the region Toscana, near Volterra. Later I did travel, by car, until Piombino (on the coast).

Sometimes the data channel disappeeared, but, at every turistic spot, I was able to send or receive data at least using the EDGE protocol. Near the most important places the speed rises at 3G.

I did experiment the teetering with a notebook. It works but, even if you disable really every unnecessary connection, 250MB is not enough.

Apps

Due to the connection limitations on lonely, but wonderful places and the limit of 250MB the app has to be suitable.

Browser

I'm using the Dolphin HD. It works everywhere, but I limit myself to WEB pages done for mobile devices: less traffic and better readability.

Google Navigator

Useful for short distances or in a city. It suggests always the shortest way to the target.

For long distances is better to check the route with Google Maps. Once, the navigator told me to leave the highway, go through a small town and go back to the highway simply because it was the shorter way. This was wrong because the route on the highway is longer but much faster.

Social Networks

Facebook, Twitter, Foursquare, Gowalla work fine everywhere, but don't forget that your smartphone may take large pictures and videos. If you load everything you may exceed the weekly limit of the account.

MapDroyd

This is a very useful app, because the maps may be preloaded on the SD-chip. You may locate yourself even if the connection with the carrier is lost.

It is fast, much faster as Google Maps, and with enough details.

Posted via Blogaway

Developing with the Android Emulator: the Beginning

2010-07-08T21:59:00.001+02:00

The documentation on Android Developers is very well done, but there is a lot of informations. Often the informations I need are spread over many different pages.

A quick setup for the development environment is easy to find. There is a lot of good sources about how to setup the virtual device, how to create a new project using the IDE and so on. What I missed is a collection of practical advices.

Don't Restart

The first mistake I did was to stop the emulator after every test of the App. Since the emulator takes very long to restart my productivity was really low. Using Eclipse and the ADT plug-in, it is sufficient to re-run the configuration: the ADT prepares the new version of the App and installs it on the emulated device. This way testing and debugging is easy and efficient.

Play with the Emulator

The my second mistake was to start with the development before I did really understand the environment.

The state of the emulation may be changed using predefined keyboard shortcuts. This is really useful.

The application will support a wide range of devices if you test it with 2 or 3 different profiles.

I defined three profiles: small, medium, and large. Small is a device without touch, without SD card and with less memory. Medium is a device with a 320x480 touch screen, 1GB SD card and 512MB memory. Large is a device with a 480x800 touch screen, 4GB SD card and 576MB memory.

First create the SD card (100 MB should be enough) and than fill the profile form. I use dimensions and not a predefined screen type. Additionally I set up the screen resolution and the device memory. Both are properties to add at the end if the profile form.

Name: Small; Target: Android 1.5 - API Level 3; Resolution: 240x320; Properties: Abstracted LCD Density = 120, Device ram size = 96.

Name: Medium; Target: Android 1.5 - API Level 3; Resolution: 320x480; Properties: Abstracted LCD Density = 160, Device ram size = 512.

Name: Large; Target: Android 2.1 - API Level 7; Resolution: 480x800; Properties: Abstracted LCD Density = 240, Device ram size = 576.

The Note Pad sample may be used to play with the emulator and to discover the development environment. At the beginning don't try to understand the code. Place two or three break points and start to debug the application. This way you will quickly familiarize with Android.

Keep in mind that the emulator does many things but doesn't replace a true device. For example: the emulator cannot vibrate.

Posted via Blogaway

How To: GIT repository on Solaris 10

2010-06-22T13:34:00.002+02:00

Last year I did replace my SVN software repository with GIT.

Last moth I began to use GIT also at the office because I will be able to roll back the experimentations sometimes I do in my projects. The history of the Eclipse IDE isn't enough to archive the goal. GIT creates locally a full featured repository with tags and branches. Any time I may switch to a different branch, so I may work, at the same time, on the mainstream and on the experimental features.

Now I wish to improve the environment because I have to work on different workplaces and I need some help from other employees to deliver the software on time.

Since there is no free LINUX installation which I may use to place the "bare" instance of the repository I have to install the software and the data on a SUN Solaris 10 server. sunfreeware provides the GIT software with instructions and requirements. I was able to install and use GIT on our two year old Solaris 10 instance without any problem.

GIT doesn't require a lot of space: I did place the repository on a 30 GB partition, which holds also our daily backups. I like simple environments, so I did define a single UNIX user (git:git) to access and administrate the repository. GIT keeps anyway track of the user which commits a change, so I don't need to define different access permissions for each developer. Anyway it would be easy to authorize an observer: I would only need to create the user observer:git.

The following section summarizes the steps necessary to create the bare instance. If you don't have root permission on the Solaris server you need to ask your administrator to create the user git:git and the root folder for your repositories (e.g /data/gitroot). Login as git and execute:

mkdir my_project.git
chmod 775 my_project.git
chmod g+s my_project.git
cd my_project.git
git init --shared=group --bare
cd hooks
mv post-update.sample post-update

To initialize the local repository (on UNIX or on Windows) I use the command clone git@myserver:/data/gitroot/my_project.git. In my environment the different GIT instances communicates using a secure shell (SSH) connection with public/private key authorization.

Update (28.10.2010)

Today I tried to clone a GIT repository from a new Solaris 10 server. The GIT client fails and displays the error "bash: git-upload-pack: command not found". The reason is that the default PATH environment variable doesn't contain the path /usr/local/bin. The solution of the problem is to add the missing path changing the file /etc/defaults/login.

Blogging with Android: Blogaway, Wordpress

2010-06-15T23:23:00.009+02:00

Last month I decided to buy a smartphone. I'm a UNIX developer since long time: my choice the HTC Desire.

Display

The display of the smartphone is limited in space. The layout of my blog on Blogger may be squeezed and is readable even with the zoom function.

The layout I use on Wordpress isn't readable on the smartphone, but the mobile site of Wordpress is really good.

Blogging

At the beginning everything seemed unconfortable, but once I did understand how to work, what to select and in which sequence bloggging was easy.

Blog-Away

Blogaway is a Blogspot client for Android. It's everything I need to write down my ideas.

This article was written using the smartphone and Blogaway travelling by train.

Each article may be published as draft and improved later on the PC. The text may be formatted using bold, italic or a color. It is possible to add links, pictures and videos. You may scroll the text very quickly. I like this application because it is simple and efficient.

You may find more informations about Blogaway on blogspot.com or on Twitter.

WordPress

Read my article on Wordpress

Conclusion

Bogging with the smart-phone is possible as additional way to add articles. It is suitable for short and simple text.

If a blog is readable or not depends on the theme.

Posted via Blogaway

Small J2EE Fedora Development Server On Amazon EC2.

2010-04-05T20:09:00.001+02:00

In the last 10 years I used a LINUX server at home to develop my private applications. Twice a year I spend a weekend to update/upgrade the server. Now I should replace the old hardware with a newest one.

Instead of buying a new server I decided to search for alternative solutions. Due to the small amount of time I really use the server, the solution have to be cheap. At the same time, due to the kind of experiments I do, I need root access to the instance.

I signed up for Amazon S3 and EC2 to be able to run a LINUX AMI. The first question is: where may I find a Fedora AMI?

The best solution for me seems to be a self defined AMI built using Elastic Server. My server is based on Fedora 10 and contains the JDK 1.6.0_13, Tomcat 6.0.18, PHP 5.2.6, MySQL 5.0.67 and Apache2. Almost the same configuration I use at home.

Elastic Server build, deploys and starts the AMI automatically. I had to define a S3 bucked, something like daniele-ami where the service will copy the image of the AMI. The builder needs to access my account at Amazon to store the image and start the AMI, so I define a dedicated Access Key. As soon as the new instance works I disable the dedicated key. The key may be re-enabled each time I want to deploy a new AMI.

Elastic Server sends an email to the account owner containing all the informations to access the the running instance.

The login (cftuser) may be performed using PuTTY (Windows) or ssh (UNIX). To gain root access I have to use the command sudo su - . A quick check of the environment confirms that everything works as expected.

Add a Data Volume

Since I don't work with a reserved instance when the instance terminates all changes made after the startup are lost. To reduce the inconvenience I define a 10 GB EBS volume using the AWS console. Note: the volume have to be defined in the same zone as the EC2 instance.

The volume is attached as device sdf (/dev/sdf)

Now the procedure is the same used to add a disk to any LINUX system:

Use fdisk (fdisk /dev/sdf) to delete all existing partitions (select d).
Save the new (empty) partition table (select w).
Add a new partition (select n).
Save the new partition table (select w).
Use mkfs (e.g. mkfs -t ext3 /dev/sdf1) to format the patition.
Use tune2fs (e.g. tune2fs -L /data /dev/sdf1) to put a label on the patition.
Create the mount point (e.g. mkdir /data).
Modify the file /etc/fstab as displayed into the next section.
Mount the partition (e.g. mount /data).

# /etc/fstab: static file system information.
# AMI version
#
# 'file system' 'mount point'   'type'  'options'       'dump'  'pass'
/dev/sda1       /               ext3    defaults,errors=remount-ro 1    1
/dev/sda2       /mnt            ext3    defaults        0       0
/dev/sda3       none            swap    sw              0       0
proc            /proc           proc    defaults        0       0
none            /sys            sysfs   defaults        0       0
none            /dev/shm        tmpfs   defaults        0       0
none            /dev/pts        devpts  gid=5,mode=620  0       0
LABEL=/data     /data           ext3    defaults        1       2

After a new start the last three steps of the previous list have to be done again to remount the disk. I do it using the following script:

#!/bin/bash
mkdir /data
echo "LABEL=/data     /data           ext3    defaults        1       2" >> /etc/fstab
/bin/mount /data

Execute:

sudo su -
vi setup
copy & paste
sh setup

The cost for an afternoon of programming and testing is $0.40 which is really cheap.

MVC on the Google App Engine with Spring 3.0

2010-01-27T13:11:00.004+01:00

On the Google App Engine (GAE) I can deploy and test applications very quickly. What I need now is a programming "toolkit" wich is essential and efficient.

I thought about using Grails but there is a lot of dependencies I don't want.

An implementaion based solely on the Servlet API wolud be essential but time consuming.

I decide to use the last release of the Springframework to see if it is possible to set up a project template for the GAE.

The Application Template

At this time I need two modules:

home with the pages "home", "about" and "description".
stage with some different controllers service each a page called "list" and a page called "details".

Home

This section is display-only and supports the GET method only. "Welcome" contains the list of the RSS feeds into the HEADER, but if something goes wrong the feed are simply missing. No error message is displayed.

Stage

The stage is a restricted area containing lists and forms.

This section implements the feed registration where "list" contains a short summary of all the available feeds. From this page I may add or delete a feed.

The "details" page is a form containing all the fields. The feed may be modified or deleted.

The Project Layout

The different parts of the application have to be easily identified on the filesystem. By convention the path of the java packages is build on a prefix (application.web) and using the name of the section:

application.web.home 
application.web.stage

Each section has at least one controller. The name is composed by the section name and the word controller.

application.web.home.HomeController 
application.web.stage.StageController

The stage sections contains more controllers one for each editor with restricted access. At this time I have the only the "feed" editor.

application.web.stage.FeedController

I use a similar convention for the views (JSP pages). The views are placed under WEB-INF/views:

WEB-INF/views/home/about.jsp 
WEB-INF/views/home/description.jsp 
WEB-INF/views/home/home.jsp 
 
WEB-INF/views/stage/stage.jsp 
WEB-INF/views/stage/feed.jsp 
WEB-INF/views/stage/feedDetails.jsp

The whole layout was created using Maven ( mvn archetype:generate ). I did merge "maven-archetype-quickstart" and "maven-archetype-webapp".

The Controllers

The home controller is quite simple:

package ch.clx.application.web.home;
...
@Controller
@RequestMapping("/home")
public class HomeController {
    
  @RequestMapping(method = RequestMethod.GET)
  public String get(final ModelMap model) {
    ...
    model.addAttribute("version", Version.version);
    return "home";
  }
    
  @RequestMapping(value="/about", method = RequestMethod.GET)
  public String about(final ModelMap model) {
    model.addAttribute("version", Version.version);
    return "about";
  }
    
  @RequestMapping(value="/description", method = RequestMethod.GET)
  public String description(final ModelMap model) {
    model.addAttribute("version", Version.version);
    return "description";
  }
}

The feed controller is a simple CRUD editor. The GET method returns the list of the feeds (the feed view).

Add (C) creates a new entry and displays the form with all the fields (detail view).
Delete (D) removes an existing entry.
Edit (R) displays an existing entry (detail view).
Store (U) updates an existing entry.

package ch.clx.application.web.stage;  
...  
@Controller  
@RequestMapping("/stage/feed")  
public class FeedController {  
      
  @RequestMapping(method = RequestMethod.GET)  
  public String get(final ModelMap model) {  
    ...  
    model.addAttribute("version", Version.version);  
    return "feed";  
  }  
      
  @RequestMapping(value = "/add", method = RequestMethod.POST)  
  public String add(  
    @RequestParam("id") final String id,   
    final ModelMap model) {  
    ...  
    model.addAttribute("version", Version.version);  
    return "feedDetails";  
  }  
      
  @RequestMapping(value = "/delete", method = RequestMethod.POST)  
  public String add(  
    @RequestParam("id") final String id,   
    final ModelMap model) {  
    ...  
    model.addAttribute("version", Version.version);  
    return "feed";  
  }  
      
  @RequestMapping(value="/edit", method = RequestMethod.GET)  
  public String edit(  
    @RequestParam("id") final String id,   
    final ModelMap model) {  
    ... 
    model.addAttribute("version", Version.version);  
    return "feedDetails";  
  }  
      
  @RequestMapping(value="/store", method = RequestMethod.GET)  
  public String store(  
    @RequestParam("id") final String id, 
    ..., 
    final ModelMap model) {  
    ... 
    model.addAttribute("version", Version.version);  
    return "feed";  
  }  
}

The Context File

The context file is very simple because components and URL routing are automatically discovered through annotations.

<?xml version="1.0" encoding="UTF-8"?> 
<beans 
  xmlns="http://www.springframework.org/schema/beans" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xmlns:p="http://www.springframework.org/schema/p" 
  xmlns:context="http://www.springframework.org/schema/context" 
  xsi:schemaLocation=" 
    http://www.springframework.org/schema/beans  
    http://www.springframework.org/schema/beans/spring-beans-3.0.xsd  
    http://www.springframework.org/schema/context 
    http://www.springframework.org/schema/context/spring-context-3.0.xsd" 
> 
<bean 
  class="org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping"/> 
<bean 
  class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter"/> 
 
<context:component-scan base-package="gae.application"/>

<bean 
  id="viewResolver" 
  class="org.springframework.web.servlet.view.ResourceBundleViewResolver"/>

</beans>

The Deployment Descriptor

web.xml

<?xml version="1.0" encoding="UTF-8"?> 
<web-app 
  id=2gae-application" 
  version="2.4" 
  xmlns="http://java.sun.com/xml/ns/j2ee" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee  
      http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"/> 
  <display-name>gae-application</display-name> 
  <servlet> 
    <display-name>Dispatcher Servlet</display-name> 
    <servlet-name>Dispatcher-Servlet</servlet-name> 
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class> 
    <init-param>
      <param-name>contextConfigLocation</param-name>
      <param-value>/WEB-INF/WebCtx.xml</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping
    <servlet-name>Dispatcher-Servlet</servlet-name>
    <url-pattern>/bo/*</url-pattern>
  </servlet-mapping>
  <welcome-file-list>
    <welcome-file>index.jsp</welcome-file>
  </welcome-file-list>
</web-app>

The GAE Deployment Descriptor

appengine-web.xml

<?xml version="1.0" encoding="UTF-8"?> 
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0"> 
<application>gae-application</application> 
<version>beta-1</version> 
<system-properties> 
<property 
  name="java.util.logging.config.file" 
  value="WEB-INF/classes/logging.properties"/>
</system-properties> 
</appengine-web-app>

The Maven Archetype

At the end of my experiment I pack my template application in a Maven archetype. I is enough to add the following pom.xml file to the project:

<?xml version="1.0" encoding="UTF-8"?>
<project
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
    http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>cloud.gae.application</groupId>
  <artifactId>gae-application</artifactId>
  <version>1.0</version>
  <packaging>war</packaging>
  ...
  <build>
    <finalName>${artifactId}</finalName>
    ...
  </build>
</project>

Now type mvn archetype:create-from-project to generate the archetype. Change the directory ( cd target/generated-sources/archetype ) and execute mvn install .

Using mvn archetype:generate now I may generate a GAE application ready to use. It does nothing but as starter kit for GAE applications is not bad.

URL Shortening Service

2010-01-16T21:41:00.006+01:00

A URL shortening service is not essential for the most part of the applications in the WEB and may be dangerous because the surfer doesn't see what lies behind the short URL. But if I would like to publish something on a short messaging system like Twitter I need a very short URL.

The solution would be to register a very short domain like goo.gl and to run my own shortener, but for my application the additional effort is out of scope. So I need to connect to an existing URL shortening service.

The beginner has a first obstacle: where to find an inexpensive reliable URL shortening service. Using a search engine I did found a lot of service which are also free of charge, but no way to proof if the are also reliable.

Service Requirements

Since I want to generate the short URL into the background I need a service with API support. Additionally stating from the short URL I want to obtain the original (long) URL.

The candidates

I didn't want to check all the services, for instance four of them are enough.

bit.ly

The service is free of charge, but requires a registration and checks each request using an API key.

Homepage with a clear description of the service.
Simple well documented API.
Shorten and expand functions.
JSON and XML formatted response.
Simple but nice statistics summary.
Blog

I like bit.ly and is currently my first choice.

is.gd

The service is free of charge and doesn't require a registration.

Homepage with a clear description of the service.
Very simple API with documentation.
Shorten only function.
Plain text (URL only) response.

This service is very spartan; I noticed that after two or three days the short URL was different. If a very short URL is required id.gd may be a good choice.

snipurl.com

The service is free of charge, but requires a registration and checks each request using an API key.

Homepage with a clear description of the service.
Simple well documented API (Firefox doesn't display the page and with the MSIE there are some curious side effects, try with is-there-a-snipr-snipurl-api).
Shorten and expand functions.
XML and plain text (URL only) formatted response.
Simple statistics but searchable statistics.
Help

SnipURL exists since the year 2000, which is a sign of reliability. I miss the JSON formatted response, but the service is good.

tinyurl.com

The service is free of charge and doesn't require a registration.

Homepage with a clear description of the service.
Very simple API.
Shorten only function.
Plain text (URL only) response.

The service is very spartan, but the short URL is the same over the time.

Test the services

I did test this services for a while using a very simple HTTP client.

Client Requirements

The client have to be essential, but should be able to:

retry the connection three times before to give up.
follow the redirects (max three levels of indirection).
support GET & POST.
support the secure socket layer (SSL, nice to have).

Implementation

I use a very simple implementation done in Java and based on the HttpURLConnection, the following snippet shows the essential part of the code:

...
URL url = URL("http://myserver.com");
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setRequestMethod("GET");
conn.addRequestProperty("User-Agent","FastShortGood/1.0 Java/6");
...
// Extract the character set from the "Content-Type"
// header using a regular expression. If none available
// assume "UTF-8"
...
StringBuilder buf = new StringBuilder();
InputStream is = conn.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is, charset));
String line;
while (null != (line = br.readLine())) {
  buf.append(line).append("\n");
}
String body = buf.toString();

// Extract the short URL from response body using a
// regular expression.

For the complete implementation I need about 300 lines of code. The most of them handles exceptions and HTTP respose codes. The lightweight client works well with all of the shortener services.

Receiving Emails with the Google App Engine

2010-01-03T12:45:00.006+01:00

Last year, in November, I did explore for the first time the Google App Engine for Java (GAE/J). One of the feature my application provides is to accept an email and translate it to an RSS 2.0 item.
The first version of the receiver was done using JavaMail. I did experiment some troubles with the "Quoted-Transfer-Encoding", sometimes the content was broken.
Using the current version of the GAE/J (1.3.0) the emails content received from Gmail or from Thunderbird is correct and the channel seems to be reliable.

Mime4J

An alternative for receiving email is the James Mime4J project.
The library doesn't have dependencies and works on the GAE/J. It is quite simple to use; instead of:

import java.io.InputStream;
import java.util.Properties;
import javax.mail.Session;
import javax.mail.internet.MimeMessage;
...
InputStream input = request.getInputStream(); // ServletRequest
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
MimeMessage message = new MimeMessage(session, input); // MessagingException
...

you have to use:

import java.io.InputStream;
import org.apache.james.mime4j.message.Message;
import org.apache.james.mime4j.message.MessageBuilder;
import org.apache.james.mime4j.parser.ContentHandler;
import org.apache.james.mime4j.parser.MimeStreamParser;
import org.apache.james.mime4j.storage.MemoryStorageProvider;
...
InputStream input = request.getInputStream(); // ServletRequest
Message message = new Message();
ContentHandler handler = new MessageBuilder(message, new MemoryStorageProvider());
MimeStreamParser parser = new MimeStreamParser();
parser.setContentHandler(handler);
parser.parse(input); // MimeException
...

The result seems to be better than the one which uses JavaMail. The mails received from the MS Outlook are readable. It remains a small problem with the encoding which I will investigate later this year.

Incoming Mail Syntax

The syntax of the incoming mail was changed:

description
line 1
line 2
...
line n
end

link
http://myserver/mylink
end

categories
category 1
category 2
...
category n
end

Each command block starts with a keyword and ends with end

Informations

The email address of the Mime4J service is mime4j@example-rest.appspotmail.com, the JavaMail based receiver still works with rss2@example-rest.appspotmail.com
The application URL is example-rest.appspot.com.

First Application on the Google App Engine

2009-11-19T13:20:00.017+01:00

My first application on the Google App Engine for Java generates a dynamic RSS 2.0 feed.

Motivation

Since I like to investigate the behaviour of some different RSS 2.0 feed consumers; I need a tool which allows the quick publication over the Internet.

Description

Using an E-Mail client I send a message to the application. The message is persisted into the data store and will be used to build the RSS feed as soon as a consumer requests it.

Additional Libraries

Just the Springframework (2.5.6), but core and MVC only.

Components

A small RSS domain model: the class RSS2Channel and RSS2Item.
A Mail2RSS2 controller (the infrastructure is provided by Spring).
A RSS2 controller (the infrastructure is provided by Spring).
A DAO layer (access to the data store using low level API).

Environment

I did start with Eclipse and the GAE/J plug-in, but I feel comfortable with Maven and batch processing. I work with Maven 2.2.1, SUN JDK 1.6_16 and GAE SDK 1.2.6. I downloaded the GAE SDK for Java and placed the bin directory in my PATH.

Add the files appengine-api-1.0-sdk-1.2.6.jar and appengine-api-labs-1.2.6.jar to the Maven repository. The files are into the lib/user folder.
Add the same files as dependency in the POM file. The scope is compile.
Add the files appengine-api-stubs.jar and appengine-local-runtime.jar to the Maven repository. The files are into the lib/impl folder.
Add the same files as dependency in the POM file. The scope is test.

With this environment Maven works. The classes can be compiled and the unit tests are successful.

Debugging

The command line environment doesn't provide a debugging version out of the box. I did install Eclipse and I added the m2eclipse pug in. The next step was to import the Maven project.

On Windows I start the engine using:

@%JAVA_HOME%\bin\java -cp "%INST_DIR%\lib\appengine-tools-api.jar" -Dlog4j.configuration="%INST_DIR%\config\user\llog4j.properties" -javaagent:%INST_DIR%\lib\agent\appengine-agent.jar -agentlib:jdwp=transport=dt_socket,server=y,address=5300 com.google.appengine.tools.development.DevAppServerMain %*

On UNIX I start the engine using:

$JAVA_HOME/bin/java -ea -cp "$INST_DIR/lib/appengine-tools-api.jar" -Dlog4j.configuration="$INST_DIR/config/user/llog4j.properties" -javaagent:$INST_DIR/lib/agent/appengine-agent.jar -agentlib:jdwp=transport=dt_socket,server=y,address=5300 com.google.appengine.tools.development.DevAppServerMain $*

The JVM stops an wait on IP port 5300 for the debugger. I switch to Eclipse and open the debug configuration dialog. I define a new remote java application configuration (localhost, port 5300) for the project and start debugging. Now I'm ready for debugging.

Start up

The first attempt to start the application was not successful because I did use the Resource annotation which is not into the white list.

Note: it is impossible to keep in mind which class is allowed and which is not.The developer, sometimes, must have imagination to find a good alternative and avoid the restriction. The good news is that the environment told me about the error using a clear statement.

Looking at the logs I discovered that the application is stopped after a short idle and restarted by the subsequent request. This is an additional requirement for the application: it will be an advantage to have an implementation which doesn't loose time with long initializations.

Receiving Mails

I did define some rules to be able to convert the incoming mail to an RSS item:

The subject of the mail is the title of the item.
The description has to be marked with the [Description] TAG.
The link has to be marked with the [LinkTo] TAG.
The categories have to be marked using the [Categories] TAG.
The separator char for the categories is #.
The mail has to contain a plain text part.

Example:

[Description]Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.[Description]
[LinkTo]http://en.lipsum.com/[LinkTo]
[Categories]lorem ipsum#specimen[Categories]

At the beginning the local environment posted a ByteArrayInputStream as content of the MimeMessage and not a Multipart object. After adding javax.mail (1.4) to my POM the local GAE returns a Multipart object and behaves as expected.

The run time environment works but if I send a mail using the MS Outlook 2007 there are some troubles. If there is a line which is longer than 78 characters, the mail seems to be incomplete.I suspect that is a problem with the content type encoding "quoted-printable". I will investigate, but not right now.

Data Store

Accessing the data store using the low level API is simple and doesn't cause any problem. I did define a simple data access layer (DAO) so I may move a part of the code tested with the GAEto other projects.

Note: due to a bug I inserted a non printable char into the data store (key name), the Data Viewer became unusable (application error) until deleted the entity with the application.

Summary

The environment is very simple and efficient. The GAE console is a good tool to monitor the behaviour of the application. For small sized applications and tools which use anyway Google services may be a good alternative to a conventional application service provider.

My application is hosted at example-rest.appspot.com and the email address is rss2@example-rest.appspotmail.com.

Test Title To Setup The Formatting

2009-08-15T12:52:00.015+02:00

Test Paragraph

Title

Test Paragraph

code line 1
code line 2
code line 3

Test Paragraph

Title

List item 1
List item 2

Blogger