Don’t use large BLOBs in MySQL from Java

The MySQL database can store large chunks of binary data (up to 1 GB) using the BLOB data types.

However, this does not work well if you access the MySQL database from Java (or any other JVM based language) using the MySQL JDBC driver.

The JDBC API supports an efficient stream based way to handle BLOBs, but the MySQL JDBC driver does not implement this properly. It works, but it will buffer all data in memory in a way which is very inefficient and can make your JVM run out of memory if the BLOBs are large.

A good rule of thumb is to not store any objects larger than 64 KB in MySQL. Store such objects as files outside of the database instead. So never use the MEDIUMBLOB, LONGBLOB, MEDIUMTEXT or LONGTEXT data types.

(I am not sure whether this advise is only valid for JVM, or for usage of MySQL from other environments as well.)

Posted in database, Java | 2 Comments

Don’t use PipedOutputStream on Android

I was using java.io.PipedOutputStream in an Android app. The app performed horribly bad. After doing some profiling, it turned out that the call to PipedOutputStream.write(byte[]) that was really slow. After digging into the Android source code, I discovered that PipedOutputStream.write(byte[]) was not implemented, it just delegated to the default implementation in OutputStream which iterate through the array and call PipedOutputStream.write(byte) for each byte.

Since PipedOutputStream.write(byte) does some synchronization and Object.notifyAll() each time, it is really slow to do this 1000 times when you write a block of 1 KB data.

Just out of curiosity, I had a look on how PipedOutputStream is implemented in Oracle’s standard Java implementation. I haven’t actually done any benchmarks on it, but I can see from the source code that it is a completely different implementation which does implement writing a block properly and probably efficiently.

The bottom line is: Don’t use PipedOutputStream on Android. If you need similar functionality, implement it yourself or find a 3rd party library which does it.

Posted in Android, Java | 1 Comment

Running Adobe Lightroom 4.4 in Linux

I use Adobe Lightroom 4.4 for photo editing. There is one very annoying aspect of this program, it is not available for Linux (only for Windows and Max OS X).

In order to run Lightroom on my computer, I had to use VirtualBox and install Windows 7 in it. This works, but is quite clumsy and annoying. And since Lightroom is the only reason for me to run Windows, I would like to get rid of it.

So I decided to try out Wine. The current version of Wine does not support Lightroom out of the box, but I found some patches which seems make it work. This is how I did it, for Ubuntu 12.04 32-bit.

  1. Uninstall any Wine installation you might already have
  2. Prepare to build Wine from source (for some odd reason, it should be wine1.4 and not wine1.6):
    sudo apt-get build-dep wine1.4
  3. Download Wine 1.6 sources
  4. Unpack Wine 1.6 sources:
    tar -xjf ~/Downloads/wine-1.6.tar.bz2
  5. Download the necessary patches
  6. Unpack and apply the paches:
    tar -xzf ~/Downloads/wine-1.6-lr5-patches.tar.gz
    patch -p0 <0-Menu-wine-1.6.patch
    patch -p0 <1-ConditionVariables-wine-1.6.patch
    patch -p0 <2-InitOnceExecuteOnce-wine-1.6.patch
  7. Build and install the patched Wine 1.6:
    cd wine-1.6
    ./configure
    make
    sudo make install
    
  8. Install winetricks:
    cd /usr/local/bin
    sudo wget http://winetricks.org/winetricks
    sudo chmod +x winetricks
    
  9. Install cabextract:
    sudo apt-get install cabextract
    
  10. Prepare and install Lightroom 4.4.1 32-bit:
    winetricks win7
    wine ~/Downloads/Lightroom_4_LS11_win_4_4_1.exe
    winetricks winxp
    winetricks corefonts gdiplus ie7
    
  11. Download sRGB color profile
  12. Unpack and install sRGB color profile:
    tar -xzf ~/Downloads/lr-wine-1.5.17.tar.gz
    cp "sRGB Color Space Profile.icm" ~/.wine/drive_c/windows/system32/spool/drivers/color
    
  13. Now you can start Lightroom with this command:

    wine "C:\\Program Files\\Adobe\\Adobe Photoshop Lightroom 4.4.1\\lightroom.exe"
    

    I found most of the information here.

Posted in Linux, Ubuntu | 11 Comments

Using an embedded SQL database in Java web application

You have a Java web application needing a relational SQL database. According to JavaEE conventions, you declare the DataSource in web.xml:

<resource-ref>
    <res-ref-name>jdbc/DS</res-ref-name>
    <res-type>javax.sql.DataSource</res-type>
    <res-auth>Container</res-auth>
</resource-ref>

and then fetch it in code with (DataSource)new InitialContext().lookup("java:comp/env/jdbc/DS").

Using Maven, Jetty and HSQLDB, you can very easily run the web application for local testing without having to setup neither a web container, nor a database. Add this to pom.xml:

<build>
    <plugins>
        <plugin>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-maven-plugin</artifactId>
            <version>9.0.5.v20130815</version>
            <configuration>
                <webApp>
                    <contextPath>/${project.name}</contextPath>
                    <jettyEnvXml>src/etc/jetty-ds.xml</jettyEnvXml>
                </webApp>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.hsqldb</groupId>
                    <artifactId>hsqldb</artifactId>
                    <version>2.3.1</version>
                 </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

And add a file src/etc/jetty-ds.xml to your project:

<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">
<Configure id="wac" class="org.eclipse.jetty.webapp.WebAppContext">
  <New id="DS" class="org.eclipse.jetty.plus.jndi.Resource">
    <Arg><Ref id="wac"/></Arg>
    <Arg>jdbc/DS</Arg>
    <Arg>
      <New class="org.hsqldb.jdbc.JDBCDataSource">
        <Set name="DatabaseName">mem:db</Set>
        <Set name="User">SA</Set>
        <Set name="Password">""</Set>
      </New>
    </Arg>
  </New>
</Configure>

Then run it locally with mvn jetty:run. The database will be in-memory and does not presist between runs. It is also possible to configure HSQLDB to persist data on disk, see documentation.

If you want to have access to in-memory database in your tests, add this dependency to pom.xml:

<dependencies>
    <dependency>
        <groupId>org.hsqldb</groupId>
        <artifactId>hsqldb</artifactId>
        <version>2.3.1</version>
        <scope>test</scope>
    </dependency>
</dependencies>

And create the DataSource in your test code like this:

import org.hsqldb.jdbc.JDBCDataSource;

        JDBCDataSource hsqldbDs = new JDBCDataSource();
        hsqldbDs.setDatabaseName("mem:.");
        hsqldbDs.setUser("SA");
        hsqldbDs.setPassword("");

Note: This will not add the DataSource to JNDI.

Posted in Java, JavaEE, web | 1 Comment

GIMP 2.8 in Ubuntu 12.04

This is no longer relevant, since the PPA have updated to GIMP 2.8.10, which has fixed the bug.

If you use Ubuntu 12.04 LTS, you only have access to the old GIMP 2.6 in the standard repositories. This is unfortunate since GIMP 2.8 have a lot of new useful features.

The standard way to get GIMP 2.8 is to use the a PPA, as described here. This used to work fine. However, then the PPA upgraded to GIMP 2.8.6, which contains a horrible UI bug (seems like neither the GIMP team, nor the PPA maintainer has any particularly strong QA process).

I find this bug unacceptable, and the PPA has not yet released any new update. So I chose to use GIMP 2.8.4 instead. This is not available from the PPA, so I decided to build it from source.

I found some instructions for building here. However, I was unable to build GEGL since the latest version uses a newer version of a standard library than what is available in Ubuntu 12.04.

Fortunately, you can still use BABL and GEGL from the PPA, and only build GIMP itself from source.

This is what you need to do:

  1. Install the PPA (if you haven’t done so already), but do not install GIMP.
  2. Uninstall any version of GIMP already installed (from standard repository or PPA):

    sudo apt-get purge gimp gimp-data gimp-gmic gimp-plugin-registry libgimp2.0 gimp-help-common gimp-help-en gimp-lensfun
    
  3. Install various dependencies:

    sudo apt-get -y build-dep gimp
    sudo apt-get purge libgegl-0.0-dev libbabl-0.0-0-dev libgegl-0.0 libbabl-0.0-0
    sudo apt-get -y install gegl libgegl-0.2-0 libgegl-dev libbabl-0.1-0 libbabl-dev
    sudo apt-get -y install libavformat-dev libavcodec-dev valac-0.16 graphviz ruby1.9.1-dev liblua5.1-0-dev libopenexr-dev libopenraw-dev libexiv2-dev libsdl1.2-dev libgs-dev libjasper-dev checkinstall
  4. Build GIMP 2.8.4 from source:

    cd /tmp
    mkdir gimp_build && cd gimp_build
    export PATH=/opt/gimp-2.8/bin:$PATH
    wget ftp://ftp.gimp.org/pub/gimp/v2.8/gimp-2.8.4.tar.bz2
    tar -xjf gimp-2.8.4.tar.bz2
    cd gimp-2.8.4
    ./configure --prefix=/opt/gimp-2.8
    make
    sudo make install
    
  5. Make it possible to launch it from Unity:

    cp /opt/gimp-2.8/share/applications/gimp.desktop ~/.local/share/applications/
    

    and edit~/.local/share/applications/gimp.desktop to include this:

    Exec=/opt/gimp-2.8/bin/gimp-2.8 %U
    Icon=/opt/gimp-2.8/share/icons/hicolor/48x48/apps/gimp.png
    

Unfortunately, I have not managed to get the G’MIC GIMP plugin to work.

Posted in Linux, Ubuntu | Tagged | 3 Comments

Using AAC music files in Android

If you have an file with .aac extension, it is AAC encoded audio in an ADTS container. If you want to play such file on an Android device, you have problems. Android 2.x does not support this file format at all, and not even the latest version of Android supports reading metadata tags from it.

The solution is to repackage the audio in an MPEG-4 container to a file with .m4a extension. Android can both play and read metadata tags from such files.

This can be done with the MP4Box tool. You can also add metadata tags while repackaging:

MP4Box -itags "artist=${ARTIST}:name=${TITLE}:genre=${GENRE}" -add ${FILENAME}.aac#audio ${FILENAME}.m4a

If I understand it correctly, this will not recode the actual audio data, and thus not affect the quality in any way. The file size will also be almost the same.

Posted in Android, Linux | 1 Comment

How to design a RESTful protocol

What is REST?

In most cases, it is sufficient say that REST is a way to design a network protocol based on HTTP. I perfer to call it a RESTful protocol, but it can also be called RESTful API or RESTful Web Service. It is important to keep in mind that it technically is a network protocol, and not an API like an interface with some methods in a regular programming language.

REST is not RPC or RMI. You should not try to map an existing API in a regular programming language to a RESTful protocol directly method by method. Rather, you should map the domain model of your application to the protocol.

REST is not a religion. Be pragmatic and use what works for you. But be sure to understand the rules before you possibly break them, otherwise you may get unpleasant surprises.

How to do REST

The most important rule is to use HTTP properly and leverage it extensively. Read and understand the HTTP 1.1 specification in RFC2616.

Understand the concept of safe and idempotent methods:

  • safe – no observable side-effects
  • idempotent – doing same request several times have the same effect as doing it once

And use the appropriate HTTP methods:

  • GET and HEAD are safe and idempotent
  • PUT and DELETE are not safe but idempotent
  • POST does not have to be either safe or idempotent

Utilize the rich set of response codes defined in HTTP:

  • 200 OK
  • 201 Created
  • 202 Accepted
  • 303 See Other
  • 304 Not Modified
  • 400 Bad Request
  • 401 Unauthorized
  • 403 Forbidden
  • 404 Not Found
  • 405 Method Not Allowed
  • 406 Not Acceptable
  • 409 Conflict
  • 412 Precondition Failed
  • 415 Unsupported Media Type
  • 500 Internal Server Error
  • 503 Service Unavailable

Make use of HTTP headers:

  • Location
  • Allow
  • Authorization, WWW-Authenticate
  • Content-Type, Accept
  • ETag, If-Match, If-None-Match
  • Last-Modified, If-Modified-Since, If-Unmodified-Since
  • Cache-Control
  • etc

The request and response messages should simply contain your domain objects, probably encoded as JSON or XML (do not forget to set proper Content-Type). Do not wrap it in any envelope as SOAP does.

Do this: {"prop1":"foo","prop2":["one","two","three"]}

Do not do this: {"result":{"prop1":"foo","prop2":["one","two","three"]}}

Use HTTP response codes for errors/exceptions.

State and security

Try to avoid storing client session state on the server. You can push small session state to the client in URLs. Use HTTP Authentication, possibly with OAuth, to avoid having to keep track of “logged in users” on the server. Always use HTTPS with proper server certificates if you have any authentication, and consider using client certificates if it makes sense.

HATEOAS

If you follow all advices so far, you will probably be able to design a good and useful protocol. But it will not be RESTful unless you also apply Hypermedia as the Engine of Application State.

This will give you a high degree of decoupling between client and server. It will be easy to evolve the protocol in a backwards compatible way, to allow newer clients make use of new features and still allow old clients to operate. And you can do this without any explicit versioning of the protocol.

The client should only need one entry-point URL, and will get all other URLs in responses from the server.

In HTML, you have semi-standardized link relations:
<link href="http://foo.com/bar/edit" rel="edit" />

Use something similar to allow the server to send back URLs to the client:
{"links":{"edit":"http://foo.com/1/edit","next":"http://foo.com/2"}}

URLs should be opaque to the client. The client should accept both absolute and relative URLs, and resolve relative URLs according to RFC3986 (use a platform or 3rd party library for this). The client may sometimes append query parameters. The client should never inspect or manipulate URLs in any other way. (This is exactly how an ordinary web browser handles links in an HTML web page.)

Posted in web | 1 Comment

How to add password protection to GRUB2

These instructions are tested with Ubuntu desktop 12.04, but will probably be useful in other Linux distros with GRUB2 as well.

The goal is to block everything except booting the default system. In paricular, it should not be possible for anyone to boot into recovery mode, since that will bypass normal login and give root access directly.

  1. Run grub-mkpasswd-pbkdf2 from a terminal and enter the desired password, copy the output.
  2. Edit /etc/grub.d/40_custom and add this to the end:
    set superusers="root"
    password_pbkdf2 root output from grub-mkpasswd-pbkdf2 goes here
    password bogus bogus
    
  3. Make /etc/grub.d/40_custom non-readable for users:
    chmod o-r /etc/grub.d/40_custom
  4. Edit /etc/grub.d/10_linux according to this diff:
    90c90,94
    <   printf "menuentry '${title}' ${CLASS} {\n" "${os}" "${version}"
    ---
    >   if ${recovery} || ${in_submenu}; then
    >     printf "menuentry '${title}' ${CLASS} --users '' {\n" "${os}" "${version}"
    >   else
    >     printf "menuentry '${title}' ${CLASS} {\n" "${os}" "${version}"
    >   fi
    258c262
    <     echo "submenu \"Previous Linux versions\" {"
    ---
    >     echo "submenu \"Previous Linux versions\" --users '' {"
    
  5. Edit /etc/grub.d/20_memtest86+ according to this diff:
    27c27
    < menuentry "Memory test (memtest86+)" {
    ---
    > menuentry "Memory test (memtest86+)" --users "" {
    33c33
    < menuentry "Memory test (memtest86+, serial console 115200)" {
    ---
    > menuentry "Memory test (memtest86+, serial console 115200)" --users "" {
    46c46
    < #menuentry "Memory test (memtest86+, experimental multiboot)" {
    ---
    > #menuentry "Memory test (memtest86+, experimental multiboot)" --users "" {
    52c52
    < #menuentry "Memory test (memtest86+, serial console 115200, experimental multiboot)" {
    ---
    > #menuentry "Memory test (memtest86+, serial console 115200, experimental multiboot)" --users "" {
    
  6. Run update-grub

(I am not sure if it is possible to abuse memtest86+, but better safe than sorry.)

See this page for more information.

Please note that this by itself does not give you a secure system. It should be combined password protection for BIOS setup and for booting from removable media (CD-ROM) and USB devices. And you should not allow login without password in the main Linux system.

None of these measures protect from serious tampering with the hardware, such as removing the internal HDD and connecting it as non-boot device to another computer.

Posted in Linux, security, Ubuntu | 1 Comment

Bachata Linux

Based on what I have desscribed in the two previous posts, I have now made a Linux distribution with an install CD.

It is called Bachata Linux, check it out here.

Posted in Linux | 7 Comments

How to roll your own bootable Linux CD-ROM

When booting a regular Linux system, it just mounts some partition with a nice file system (such as ext4) on your HDD on / read/write and there you go.

When booting from a CD-ROM, it’s not that simple. The CD-ROM file system, ISO 9660, does not support the file names and attributes that a Linux system normally needs. And the CD-ROM is read only.

There are ways to work around these issues. You can use the Rock Ridge extension to ISO 9660 to get the file names and attributes needed. And the read only issue can be handled either as I have described in an earlier post, or by using unionfs.

There is also a different approach which I will describe here. Have an image file of the root file system on the CD which you mount in RAM. This will bypass the restrictions of ISO 9660 since you can use any file system in the image, and it can be read/write after being read into RAM. The obvious disadvantage is that this will consume RAM proportional to the size of the root file system, which can be a problem since you will not have any swap space available. But in several use cases, the size of the root file system can be kept small enough for this to work.

The easiest, and probably best, way to do this is to simply use the initramfs mechanism built into the Linux kernel.

In the initramfs, the Linux kernel will start executing /init as the first process, and this process is not expected to exit. This /init can be (and usually is) a shell script. In a regular root file system, /sbin/init will be executed first. So you can convert a regular root file system into an initramfs by adding an /init like this (don’t forget to make it executable):

#!/bin/sh

exec /sbin/init $@

Then you need to package it as a gzipped cpio archive with these slightly awkward commands (${ROOT_FS} is where you have prepared the root file system, ${ISO_FS} is where you are preparing the file system for the CD-ROM, don’t forget to create the directory ${ISO_FS}/isolinux first):

(cd ${ROOT_FS} && find . -type l -printf '%p %Y\n' | sed -n 's/ [LN]$//p' | xargs -rL1 rm -f)
(cd ${ROOT_FS} && find . | cpio --quiet -o -H newc | gzip -9 >${ISO_FS}/isolinux/initrd.img)

Then put this initramfs image along with the kernel image and pass it to the bootloader. Given that you use isolinux as bootloader, put the kernel image in ${ISO_FS}/isolinux/vmlinuz, copy isolinux.bin to ${ISO_FS}/isolinux/ and create a config file ${ISO_FS}/isolinux/isolinux.cfg:

default linux
label linux
    kernel vmlinuz
    append initrd=initrd.img quiet

Finally build an CD-ROM ISO image with this command:

genisoimage -b isolinux/isolinux.bin -c isolinux/boot.cat \
 -no-emul-boot -boot-load-size 4 -boot-info-table \
 -l -input-charset default -V MyLinuxBoot -A "My Linux Boot" \ 
 -o ${ISO_IMAGE} ${ISO_FS}

Then you can burn the ISO image on a CD-ROM, or use the image directly to test it in a virtual machine.

You can reduce the size of the root file system by placing some large auxiliary files (such as the files to install if you are building an installer) outside, directly on the CD-ROM. Then you need to mount the CD-ROM after boot, which can be a bit tricky. Given that you know that a file install.cgz should be on the CD-ROM, you can do like this to mount the CD-ROM on /media:

mount_cdrom() {
    for CD in /dev/cdrom /dev/cdrom[0-9] /dev/sr[0-9]; do
        if [ -b ${CD} ] ; then 
            if mount -t iso9660 -o ro ${CD} /media 2>/dev/null ; then
                if [ -f ${1}/install.cgz ] ; then
                    return
                else
                    umount /media
                fi
            fi
        fi
    done
    echo "Failed to mount CD!"
}

This is inspired from this article. However, that article uses the older and less efficient initrd instead of initramfs. It also mounts the root file system in a two-step bootstrap process which seems redundant to me. So I have made it simpler and more efficient. The procedure with boot menus and different run levels described in the section Customizing and Adding Scripts can be useful though, I recommend reading that section if you need some kind of menu.

Posted in Linux | 1 Comment