In-memory database for testing

Posted on May 1, 2023 by Mikael Ståldal

I have written about using an embedded database like HSQLDB or H2 for testing Java applications. This can still be useful (although JavaEE is not so popular any longer). But even though H2 claims to emulate PostgreSQL and several other popular production SQL databases, sometimes your application uses very specific features of a particular database and you need to test against the real thing to be really sure your application works correctly.

This can conveniently be done using Testcontainers, which run an instance of the real database in a Docker container for your tests. The downside is that this can be significantly slower than an embedded in-memory database like HSQLDB or H2.

However, you can speed it up by making sure that the database store its data in memory. Some databases, such as PostgreSQL, does not support this directly, but I learned a neat trick from a college of mine to do it anyway: use Docker’s tmpfs mount option.

You can do it like this for PostgreSQL:

docker run --tmpfs=/pgtmpfs -e PGDATA=/pgtmpfs postgres

or like this with Testcontainers: 

new PostgreSQLContainer<>(PostgreSQLTestImages.POSTGRES_TEST_IMAGE).withTmpFs(Map.of("/pgtmpfs", "rw")).addEnv("PGDATA", "/pgtmpfs")
Posted in database, Docker, Linux | Comments Off on In-memory database for testing

Alpine rather than distroless

Posted on April 20, 2023 by Mikael Ståldal

I have been using the distroless Docker base images to package my applications, mainly since I want slim and simple image without unnecessary cruft.

However, they are based on Debian, and Debian is unfortunately not so diligent to fix serious security issues as other distributions like Ubuntu or Alpine. If you scan a distroless image with the grype tool, you get this result:

$ grype gcr.io/distroless/java17-debian11
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
libharfbuzz0b 2.7.4-1 (won't fix) deb CVE-2023-25193 High
libjpeg62-turbo 1:2.0.6-4 (won't fix) deb CVE-2021-46822 Medium
libssl1.1 1.1.1n-0+deb11u4 (won't fix) deb CVE-2023-0464 High
openssl 1.1.1n-0+deb11u4 (won't fix) deb CVE-2023-0464 High

Furthermore, if small image size is important, distroless is quite overrated. They brags about being so small:

Distroless images are very small. The smallest distroless image, gcr.io/distroless/static-debian11, is around 2 MiB. That’s about 50% of the size of alpine (~5 MiB), and less than 2% of the size of debian (124 MiB).

But the gcr.io/distroless/java17-debian11 image is 231MiB, so saving 3 MiB in the base is insignificant if you depend on Java or some other runtime (Node.js or Python) anyway.

So I decided to go with Alpine instead. You can easily build your own Java base image:

FROM alpine:3.17.3

RUN apk update && apk add openjdk17-jre-headless && rm -rf /var/cache/apk/*

and the result is even smaller than distroless-java, only 198 MiB. I bet it’s as easy to do the same with Node.js or Python.

I would say that distroless is only relevent if you can use the smallest gcr.io/distroless/static-debian11 which is indeed very small and does not contain the above vulnerabilities since it does not even contain those packages. All the others are not so small and may contain unpatched vulnerabilities.

Posted in Docker, Linux, security | Comments Off on Alpine rather than distroless

How to enable multi-platform Docker builds on Ubuntu 22.04

Posted on February 10, 2023 by Mikael Ståldal

Docker’s official documentation on how to enable multi-platform Docker builds is a bit intimidating, suggesting you to run the Docker image tonistiigi/binfmt in privileged mode on your machine. I searched for alternatives on Ubuntu, and found this very detailed description. However, with recent versions of Ubuntu and Docker, it is now much easier than that:

  1. Install QEMU’s static user mode emulation binaries:
    sudo apt install qemu-user-static
  2. Install Docker Engine with Buildx support according instructions here (do not install docker-ce or docker.io from Ubuntu’s standard repository).
  3. Create a Docker Buildx builder:
    docker buildx create --name multiplatform --bootstrap --use
  4. Verify it:
    docker buildx inspect

Then you can do multi-platform Docker builds like this:
docker buildx build --platform linux/amd64,linux/arm64 .

Posted in Docker, Linux, Ubuntu | Comments Off on How to enable multi-platform Docker builds on Ubuntu 22.04

How to capture log events in tests with Log4j 2

Posted on November 2, 2022 by Mikael Ståldal

Sometimes you want to verify that your program actually logs what it is supposed to be logging.

When using Apache Log4j 2 as logging framework, it can be done like this:

  1. Include this dependency (for Maven, adjust appropriately for other build systems): 
    <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <type>test-jar</type>
    <scope>test</scope>
</dependency>
  2. Import some stuff: 
    import org.apache.logging.log4j.core.Logger;
import org.apache.logging.log4j.core.LoggerContext;
import org.apache.logging.log4j.test.appender.ListAppender;
  3. Add a ListAppender to the logger you are interested in: 
    var loggerContext = LoggerContext.getContext(false);
var logger = (Logger) loggerContext.getLogger(MyClass.class);
var appender = new ListAppender("List");
appender.start();
loggerContext.getConfiguration().addLoggerAppender(logger, appender);
  4. Exercise your code which should be logging.
  5. Extract what was logged: 
    List<String> loggedStrings = 
        appender.getEvents().stream().map(event -> event.getMessage().toString()).collect(Collectors.toList());

Update:

Starting with Log4j 2.20.0, this has changed to

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core-test</artifactId>
    <scope>test</scope>
</dependency>

and

import org.apache.logging.log4j.core.test.appender.ListAppender;
Posted in Java | Comments Off on How to capture log events in tests with Log4j 2

How to run exec-maven-plugin only if a file is not up-to-date

Posted on July 30, 2019 by Mikael Ståldal

In my Maven build, I use exec-maven-plugin to run an external tool for Java code generation. In this case the flatbuffers compiler (flatc).

By default, flatc always runs, so you always get a new .java file. This will always trigger Java compiler to run, which will in turn always trigger jar packaging to run, etc. I would like flatc to only run if it’s source file is updated, so I can get a properly incremental build and avoid needlessly recompiling.

flatc itself is not Java based, it does not support conditional execution, and there is no specific Maven plugin wrapping this tool. (There are specific Maven plugins for some other similar tools, like protoc.)

I was able to do it by using build-helper-maven-plugin, like this:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <version>3.0.0</version>
  <executions>
    <execution>
      <id>fbs-register-sources</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>add-source</goal>
      </goals>
      <configuration>
        <sources>
          <source>${project.build.directory}/generated-sources/flatc</source>
        </sources>
      </configuration>
    </execution>
    <execution>
      <id>check-fbs-compile</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>uptodate-property</goal>
      </goals>
      <configuration>
        <name>flatc.notRequired</name>
        <fileSet>
          <directory>${project.basedir}/src/main/fbs</directory>
          <outputDirectory>${project.build.directory}/generated-sources/flatc</outputDirectory>
          <includes>
            <include>my.fbs</include>
          </includes>
          <mapper>
            <type>merge</type>
            <to>My.java</to>
          </mapper>
        </fileSet>
      </configuration>
    </execution>
  </executions>
</plugin>
<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>exec-maven-plugin</artifactId>
  <executions>
    <execution>
      <id>fbs-compile</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>exec</goal>
      </goals>
      <configuration>
        <skip>${flatc.notRequired}</skip>
        <!-- ... -->
Posted in Java, maven, programming | Comments Off on How to run exec-maven-plugin only if a file is not up-to-date

Support fake Android TV devices

Posted on June 14, 2016 by Mikael Ståldal

Android TV is included in some modern Smart TVs, and there are also external set-top-boxes to get Android TV for any TV with HDMI input.

However, there are also some set-top-boxes which almost, but not quite, support Android TV.

With not so much extra effort, it is possible to make an Android app supporting those fake Android TV devices as well as the real ones.

In addition to the guidelines from Google for building Android TV apps, do this to support the fake TV Android TV devices as well:

  1. If you build an app for both Android TV and mobile (smartphone/tablet), make sure to build a separate .apk for Android TV, using Gradle flavors. (This is a good idea anyway since it will keep down the .apk size.)
  2. Do not use the television resource qualifier, put TV resources in unqualified resource directories instead. (When building a separate .apk for TV only, this resource qualifier is not necessary.)
  3. Specify both android.intent.category.LAUNCHER and android.intent.category.LEANBACK_LAUNCHER in the intent-filter for the launcher activity in the app manifest.: 
    <intent-filter>
    <action android:name="android.intent.action.MAIN" />
    <category android:name="android.intent.category.LEANBACK_LAUNCHER" />
    <category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
  4. Specify Leanback support with required=false in the app manifest: 
    <uses-feature
    android:name="android.software.leanback"
    android:required="false" />
  5. Specify both android:icon and android:banner in the app manifest: 
    <application
    android:banner="@drawable/my_banner"
    android:icon="@drawable/my_icon">
  6. If you use the Leanback SearchFragment, you need to disable voice search on the fake devices, since they usually does not support speech recognition: 
        static boolean isRealAndroidTvDevice(Context context) {
        UiModeManager uiModeManager = (UiModeManager) context.getSystemService(UI_MODE_SERVICE);
        return (uiModeManager.getCurrentModeType() == Configuration.UI_MODE_TYPE_TELEVISION);
    }        

    if (!isRealAndroidTvDevice(context)) {
        setSpeechRecognitionCallback(new SpeechRecognitionCallback() {
            @Override
            public void recognizeSpeech() {
                // do nothing
            }
        });
    }
  7. The fake Android TV devices usually get the display pixel density wrong. You can programatically fix this in onCreate on the launcher activity: 
        if (!isRealAndroidTvDevice(context)) {
        DisplayMetrics metrics = new DisplayMetrics();
        getWindowManager().getDefaultDisplay().getMetrics(metrics);
        metrics.density = 2.0f;
        metrics.densityDpi = 320;
        metrics.scaledDensity = 2.0f;
        getResources().getDisplayMetrics().setTo(metrics);
    }
Posted in Android | 3 Comments

Data structures and Domain Driven Design

Posted on February 10, 2016 by Mikael Ståldal

After listening to a presentation about Domain Driven Security, I got some additional insights about what a data structure (as opposed to OOP object) should be able to do. Data structures (I call them struct) should have a subset of OOP features which I think is useful and harmless.

A struct should allow constructors. Such constructors can have two purposes, to convert one type of data into another and to validate the input. Such constructors should take parameters and be able to throw an exception if the input is invalid, to make it impossible to create an invalid instance of a struct. Each constructor should be required to set every field of the struct, and if no constructors are defined, a default one should be generated which simply takes all fields as parameters.

It can also be useful to have derived fields, which would be a parameter-less function which returns a value computed from one or more fields in the struct. They should be pure functions from the current state of the struct.

There should be encapsulation so that fields can be marked as public or private. Which one should be default?

structs should be immutable by default, with an option to mark individual fields of them mutable. However, it should be required to mark the whole struct as mutable to be able to mark any field of it mutable. A mutable field cannot be private and it’s value can be set from outside (typically with assignment syntax).

structs should not have any inheritance or polymorphism, and should not have any methods.

structs should be value types without identity, like a struct in C# or a primitive type in Java.

Posted in programming | Comments Off on Data structures and Domain Driven Design

Objects vs. data structures

Posted on February 8, 2016 by Mikael Ståldal

Several popular statically typed programming languages, including C++, Java and C#, have a serious design flaw. They make no useful distinction between objects in the OOP sense and plain data structures.

A plain data structure should only be a dumb container for data. It should not have any behaviour, only simple accessors for data items in it. It should not be possible to override data accessors. It should be clear and transparent what it does.

Good old C has a working construct for plain data structures, the struct.

C++, even though trying to be compatible with C, managed to mess it up. C++ added class for OOP objects, but also extended struct to be more or less equivalent to class. Even though there seems to be a convention in C++ to use struct for plain data structures and class for OOP objects, there is no guarantee that a struct in fact is just a plain data structure without surprising behaviour.

Java made it worse by skipping struct completely and providing no other construct for plain data structures than class which is primarily made for OOP objects. In the Java community, there is no general convention for making any distinction. To make things even worse, the horribly verbose Java Beans convention is quite popular and has support in the standard library.

C# actually have a meaningful difference between struct and class, making struct more suitable for plain data structures. However, it seems like struct have too much OOP like functionality built in and that it is possible to provide surprising behaviour.

Scala has class for OOP objects, but does also provide case classes which are suitable for plain data structures. Case classes also have some bonus features like pattern matching and auto generated methods for equality, hash code and string representation. Together with Scala language features like named parameters and immutability by default, they work reasonably well as plain data structures. However, I wish they could be more restrictive and not allow OOP features like general inheritance (which does not work well for case classes anyway) and overriding accessors.

It is possible to use associative arrays as data structures, and then you get no surprising behaviour. This is popular in Javascript which have nice syntax for this. However, most other languages do not have a convenient syntax for this, and in particular it does not work well in a statically typed language. I like to declare my data structures, and it should not be possible to add arbitrary values to instances of them at runtime. Associative arrays usually also have a runtime performance overhead.

It is also possible to use tuples (which some languages have) as data structures, but I want the items in them to be named, not ordered.

Is my only choice to use C?

Updated: mentioned that it is about statically typed languages

Update: Java finally fixed this by adding record.

Posted in Java, programming | 2 Comments

Typesafe’s Reactive Straw man

Posted on August 3, 2015 by Mikael Ståldal

In their quest to promote Reactive, Typesafe is beating up a straw man by portraying blocking I/O in a particularly stupid way which is rarely (if ever) done in practice.

In a recent webinar, I found this slide which suggests that a blocking I/O operation will waste CPU time while waiting for the I/O to complete.

If I understand it correctly, it does not actually work like this in any reasonable runtime environment / operating system. As an example, consider doing java.io.InputStream.read() on a socket in Java (or any other language on the JVM) on Linux. If there is not data available in the buffer, this call will block until some packet is received by the network interface, which may take several seconds. During that time, the JVM Thread is blocked, but the JVM and/or Linux kernel will reschedule the CPU core to another thread or process and will wait for the network interface to issue an interrupt when some packet is received. You will waste some time on thread scheduling overhead and user/kernel mode switching, but that’s typically much less than the I/O waiting time.

I am quite sure it will work the same way on other runtime environments (such as .Net) and operating systems (such as Microsoft Windows, Solaris, Mac OS X).

There are other reasons why blocking I/O can be problematic, and the Reactive principles are useful in many cases. But please be honest and don’t portray blocking I/O worse than it actually is.

Posted in Java, Linux, reactive | 1 Comment

Leafpad > gedit

Posted on July 21, 2015 by Mikael Ståldal

I want a simple, fast and lightweight text editor for my Linux desktop. I don’t want to learn a lot of new arcane key bindings, so not Vim or Emacs. I want a real GUI desktop application, not a console based one, so not nano (even though nano is nice when you only have a text console available). I don’t want a full-blown IDE for programming (I already have that) so I don’t need syntax highlighting and similar features.

So far I have been using gedit which is bundled as default text editor with Ubuntu. But it is not lightweight and sometimes really slow. It is particularly bad at handling large files and slow at searching, searching for a word in a 5 MB text file was painfully slow on my quite fast computer.

I got fed up with gedit and went out to look for alternatives. I found Leafpad and really like it. It is available in the standard Ubuntu APT repository, so it is trivial to install. It is GTK+ based (just like gedit), so it integrates nicely with the Unity desktop, it even supports the Unity global menu bar.

Leafpad is really lightweight and much faster than gedit. Searching through a 5 MB text file is not a problem at all.

Leafpad lacks some of the features that gedit have. It doesn’t support multiple files in tabs, but since it’s lightweight you can start multiple instances of it in separate windows, and the tab support in gedit is not very good anyway. It does not support syntax highlighting, but for me gedit is not good enough for serious programming anyway, I want a real IDE with more than just syntax highlighting for that.

Now gedit is uninstalled from my machine and I don’t miss it.

Posted in Linux, Ubuntu | 5 Comments