Stephen Viles

Producing CSV files from Java for French and German

2011-08-20T12:00:00Z

I’ve been applying some limited internationalization to my work’s US-centric Java-based web app: producing CSV files that give intelligible results when opened in Excel, without having to switch your Windows regional settings from “French (France)” or “German (Germany)” to “English (United States)”, as our long-suffering (and English-speaking) customers have to do.

I’m using Java’s built-in Locale, DateFormat, and NumberFormat classes. Here are a few things I’ve learned along the way.

Excel can expect ; (semi-colon) as the separator in a “comma-separated values” file

The character that Excel expects as the separator in a CSV file is controlled by the “list separator” setting in Windows regional settings. The possible values are , (comma) and ; (semi-colon).

Here are instructions for finding the “list separator” setting in different versions of Windows:

Windows XP: How to change date, time, number, and currency value displays in Windows XP
Windows Vista: Change the display of dates, times, currency, and measurements
Windows 7: Change the display of dates, times, currency, and measurements

Note that even if the Windows regional settings are “French (France)”, the user can change the “list separator” setting between ; (semi-colon) and , (comma). So I added a separate “list separator” user preference in our web app.

Excel can’t handle Java’s French month abbreviations

I’d hoped that Java’s DateFormat for Locale “French (France)” would produce formatted dates that can be parsed by Excel with Windows regional settings of “French (France)”. Sadly, that’s not the case for the MEDIUM date format. Java puts a ‘.’ character at the end of month abbreviations e.g. “1 sept. 2011”, which Excel can’t handle; it only parses the months that have four letters or less, such as “mai”.

The solution was to use the LONG date format. This causes Java to emit “1 septembre 2011”, which Excel parses and renders as “01-sept-11”.

Excel expects UTF16-LE character encoding, not UTF-8

Accented (non-ASCII) characters in month names were garbled in Excel. Great, a character encoding mismatch. It seems that Excel expects UTF16-LE character encoding, which is easy enough to set using an OutputStreamWriter:

Writer writer = new BufferedWriter(new OutputStreamWriter(response.getOutputStream(), "UTF-16LE"));

although you’ll need to catch UnsupportedEncodingException, unless you use Guava’s Charsets.

New York City travel tips

2011-06-21T12:00:00Z

Here’s a few tips on Manhattan street layout, getting around, security, eating and drinking, and swimming.

5th Avenue divides East and West (Streets)

Most of Manhattan has a grid layout of numbered avenues running north-south and numbered streets running east-west (in reality, about 30° clockwise from the compass point). While almost as unimaginative as naming New Zealand’s two main islands North Island and South Island, it shares the same advantage: it’s easy to work out which direction you are heading.

But you need to know that numbered streets are divided into East and West at 5th Avenue, and that building numbers are specific to the East and West parts of the street. So you don’t say “1 43rd St”, you say “1 East 43rd St” or “1 West 43rd St”. To make things more fun, building numbers start at 5th Avenue and increase in the same direction as the street name – to the west for West streets, to the east for East streets. So 1 East 43rd St and 1 West 43rd St are across 5th Ave from each other, but 500 East 43rd St and 500 West 43rd St are on opposite sides of the island. You must know the East or West part of the street name to get to the right building.

Streets are numbered from south to north, and avenues are numbered from east (1st Ave) to west (12th Ave). 4th Avenue has a short existence from East 9th St to East 14th St. In its place, north of 23rd St, are Lexington Avenue, Park Avenue, and Madison Avenue (east to west). There are also avenues to the east of 1st Avenue, south of 14th St, named Avenue A, Avenue B, Avenue C, Avenue D – not Avenue 0 to Avenue -3 as a geek might think. The area is known as Alphabet City (which I suppose is better than Negative City).

It takes about 5 minutes to walk east-west along a street between one avenue and the next (known as a long block). It takes about 1 minute to walk along an avenue between one street and the next (known as a short block).

Most streets and avenues are one-way. Don’t be surprised if a driver heads past the avenue or street of your destination. They will do this to get onto an avenue or street that goes in the right direction. Major cross streets (yellow in Google Maps – 14th, 23rd, 34th, 42nd, 57th) are two-way and have subway stations on them.

Subway orientation: express, local, uptown, downtown

See How to Ride the Subway, and get a MetroCard. Subway maps are free at the subway station and online. The subway map is schematic and not to scale. OnNYTurf provides an overlay of subway lines on a street map.

Subway lines are identified by a letter or digit, and are grouped by color e.g. the green-colored 4, 5, and 6 lines all run on the same set of tracks in Manhattan, but to different endpoints in the Bronx. Express trains stop only at stations marked with white circles on the map. Local trains stop at all stations. On the paper or PDF versions of the map, the lines that stop at each station are listed under the station name.

On the subway map, a line number/letter in a coloured square (rather than a circle) means the last station for the line. This is pretty obvious when the track stops, but the Q and V lines stop in the middle of a track on Manhattan.

Once in a station, look up for signs directing you to platforms. Often the platforms for opposite directions on the same line are at different locations in the station. Uptown means to the north of Manhattan. Downtown means to the south of Manhattan. Once you have the right platform, the express and local tracks may be across the platform from each other.

The “ladder” on the subway map between Lexington Ave/53rd St (F line) and 59th St (4/5/6 lines) does not indicate a tunnel or moving footway. It means only that you can get out to the street and transfer to the other line without paying another fare.

NYC is a big city full of busy people, who are mostly friendly, but who don’t appreciate people standing around and getting in their way. If you have to stop and look up your map, move to one side of the footpath.

Search engines

The USA has been in a heightened state of security awareness since 9/11/01. Some of the impacts are:

In air travel, your luggage may be searched by the Transport Security Administration (TSA). Any locks that cannot be opened by TSA will be cut off and destroyed – so use TSA-openable locks. Lock all parts of your luggage, even empty parts, to deter planting of material in your luggage (this applies worldwide, not just in the US).
On entry to the US, your photograph and index fingerprints will be taken (using a digital scanner, not an ink pad).
In the NYC subway, bags are subject to random search by police. If you do not consent to search, you will not be allowed on the subway.

Good advice from the MTA safety page:

As a general precaution, whether you’re in the subway, the bus, or even in the street, appear confident. Always look as if you know where you’re going [another reason not to pull out your subway map], and you’re better off not displaying money in public. Never keep your wallet or money in a back pocket, and keep all bags, backpacks, and pocketbooks securely closed.

Tips on eating and drinking

Tipping is the US way of compensating for a low minimum wage. Add 15-20% to your restaurant bill. When you pick up your change after being served a drink at a bar, leave one dollar bill on the bar for each drink.

New York tap water is “virtually lead-free”. So you may prefer to drink bottled water, which is available in all supermarkets and corner stores. Large bottles are much cheaper than small bottles.

Supermarkets are at the street level of apartment or office buildings, not in their own separate buildings. Chains include D’Agostino and Gristedes. Whole Foods Market is not cheap but has good food.

Public swimming pools, even outdoors (in the summer)

NYC has several public swimming pools, including outdoor pools open in the summer free of charge. Look for ‘adult swim’ times if you want to be able to swim laps. I used Hamilton Fish Park on the Lower East Side and really enjoyed it. The city even has free swimming lessons.

Introduction to Java web applications

2010-05-31T12:00:00Z

I used the following material to give a one-hour overview of Java and web applications to a group of developers experienced in the Microsoft ecosystem.

The basic Hello World: Java source in .java file compiles to JVM bytecode in .class file

See the following sections of this useful Introduction to Java Programming:

1. Introduction
2. Classpath
3. Your first Java program

Section 13 provides an overview of JAR (Java ARchive) files. More details at the Java Tutorial Lesson: Packaging Programs in JAR Files.

Using Java for Financial Applications

Dates and Times

The standard Java library has two classes for dates and times: java.util.Date and java.util.Calendar. Both are mutable and difficult to use for common date arithmetic. A much better replacement is Joda-Time. In particular, the LocalDate class is an “immutable datetime class representing a date without a time zone”. Most of the time, this is exactly what you need.

Monetary Values

Java does not have a built-in decimal data type. Instead, you need to use the BigDecimal class in the standard library for immutable, arbitrary-precision signed decimal numbers with complete control over rounding behavior.

Do not use either of the binary floating-point types (float or double). It is impossible to represent 0.1 (or any other negative power of ten) as a float or double exactly. The representation errors and rounding errors will bite you. See http://floating-point-gui.de/ for more details on “what every programmer should know about floating-point arithmetic (or why don’t my numbers add up?)”.

I recommend carrying the currency with every monetary value, as documented in the Money pattern (Patterns of Enterprise Application Architecture, Martin Fowler). The Currency class in the standard Java library represents currencies using their ISO 4217 currency codes.

Programming Guidance: Effective Java and Collections

The book Effective Java, Second Edition by Joshua Bloch gives much useful programming guidance in bite-sized articles.

The Java Collections framework in java.util is very useful. See the overview and annotated outline.

Google have written an extension collection framework in the Guava Google Core Libraries for Java.

Structure of Java Web Applications

The most basic component of a Java web application is the servlet: a Java class that responds to HTTP requests.

Start with the overview page Getting Started with Web Applications, then read the following sections of the J2EE 1.4 tutorial:

Chapter 1: Overview
Chapter 3: Getting Started with Web Applications
Chapter 11: Java Servlet Technology

After that, you’ll find the other 32 chapters describe many other technology components that you can add to Java web applications, including Web services.

A Hello World Web Application using Eclipse, Ant and Tomcat

The following pages give detailed instructions on how to build a Hello World web application. Along the way, they introduce the Eclipse IDE, the Ant build tool, and the Tomcat servlet container.

The copyright notice on these pages is rather amusing.

Web Application Security Issues

A public-facing web application has particular security issues. See the Open Web Application Security Project (OWASP) Top Ten for a general overview, and then see the Enterprise Security API for Java EE, which provides a security framework for Java web applications.

Removing line noise with a user style sheet

2010-05-30T12:00:00Z

I find text with underlining or strikethough difficult to read, so I use the following user style sheet to remove it, except when hovering over links:


/*
 * This file can be used to apply a style to all web pages you view
 * Rules without !important are overruled by author rules if the
 * author sets any.  Rules with !important overrule author rules.
 */

:link, :visited {
  text-decoration : none !important ;
}
:link:hover, :visited:hover {
  text-decoration : underline !important ;
}
del {
  text-decoration : none !important ;
}
strike {
  text-decoration : none !important ;
}

In Firefox, the user style sheet file is called chrome/userContent.css in the user profile directory.

A pattern for scripts that update data

2010-04-24T12:00:00Z

The following Bash script is designed to be run from a crontab entry. It performs a database update (using Oracle’s sqlplus), checks the output for errors, and adds a success or failure comment to a JIRA ticket by sending an email to the JIRA instance.

A JIRA instance only accepts email from the addresses of its registered users, so I had to use sendmail with an email constructed from scratch in order to control the From: address. The issue key is specified in the Subject: line.

Note the signature block of the email (after the -- line), which identifies where the script is running, and as which user — crucial information if you want to modify its behaviour.

Caution: You don’t want to use too many of these out-of-band database updates. Like database triggers, they have their uses, but do not scale well. A system with more than a few of these scripts will be hard to understand, and harder to modify. Temporary solutions often become permanent, and permanent solutions should be inside your application, not outside it.


#! /bin/bash
set -u
SCRIPT_NAME=$(basename $BASH_SOURCE)
SCRIPT_DIR=$(dirname $(readlink -f $0))
SQL_OUTPUT=/tmp/$SCRIPT_NAME-$$.out
DB_CREDENTIALS=

# Do the update
sqlplus $DB_CREDENTIALS 2>&1 >> $SQL_OUTPUT << EOF
set echo on
@$SCRIPT_DIR/do-it.sql
EOF

# Look for Oracle errors
ERROR_LIST=/tmp/$SCRIPT_NAME-$$.err
grep 'ORA-' $SQL_OUTPUT >> $ERROR_LIST
grep 'SP2-' $SQL_OUTPUT >> $ERROR_LIST

# Look for success messages
function look_for() {
    grep --silent "$1" $SQL_OUTPUT
    RETURNVAL=$?
    if [[ $RETURNVAL -ne 0 ]] ; then
        echo "Did not find expected output: $1" >> $ERROR_LIST
    fi
}

look_for '1 row updated.'
look_for 'Commit complete.'

# Start the log, which is in email format
SCRIPT_LOG=/tmp/$SCRIPT_NAME-$$.log
cat >> $SCRIPT_LOG << EOF
To: jira@example.com,me@example.com
From: me@example.com
Subject: PRJ-12345

EOF

# Build body of log
if [ -s "$ERROR_LIST" ] ; then
    echo "Errors found while attempting to do it:" >> $SCRIPT_LOG
    cat $ERROR_LIST >> $SCRIPT_LOG
    echo >> $SCRIPT_LOG
    echo "Full command output:" >> $SCRIPT_LOG
    cat $SQL_OUTPUT >> $SCRIPT_LOG
else
    echo "Done it" >> $SCRIPT_LOG
fi

# Add script detail to log and send it to JIRA ticket
cat >> $SCRIPT_LOG << EOF

--
Sent by script $SCRIPT_DIR/$SCRIPT_NAME running on host `hostname`.`dnsdomainname` as user `whoami`
EOF
/usr/lib/sendmail -t < $SCRIPT_LOG >> $ERROR_LIST 2>&1

RETURNVAL=$?
if [[ $RETURNVAL -ne 0 ]] ; then
    echo "sendmail had a non-zero exit code, see above for output" >> $ERROR_LIST
else
    rm $SQL_OUTPUT $ERROR_LIST $SCRIPT_LOG
fi