Relentless Coding

A Developer’s Blog

Build Kotlin JMH Benchmarks With Maven

In this post, we will look at how to set up the Maven infrastructure so we can benchmark Kotlin code with JMH (the Java Microbenchmarking Harness).

This post does not look in-depth on how to write JMH benchmarks, but rather how to build and run them.

Create a Sample JMH Project

Helpfully, the JMH project has a Maven archetype for Kotlin:

$ mvn archetype:generate \
    -DinteractiveMode=false \
    -DarchetypeGroupId=org.openjdk.jmh \
    -DarchetypeArtifactId=jmh-kotlin-benchmark-archetype \
    -DgroupId=com.relentlesscoding \
    -DartifactId=benchmarks \
    -Dversion=1.0

This creates a sample project where we can play around with a simple benchmark written in Kotlin.

Things to Keep in Mind When Creating JMH Benchmarks

Benchmarks look like this:

package com.relentlesscoding

import org.openjdk.jmh.annotations.Benchmark

open class MyBenchmark {

    @Benchmark
    fun benchmarkMethod() {
    }
}

See a more elaborate benchmark example that benchmarks signing efficiency of different-size RSA keys at the end of this post. Also see the examples the JMH team provides.

Notice that the class needs to be declared open:

Benchmark classes should not be final.                                                         
   [com.sample.RSASigningMessageBenchmark]

This is because JMH needs to be able to subclass it to add instrumentation code and implement the benchmarking machinery.

If you or your IDE objects to opening classes, you could also use the all-open compiler plugin that is part of the kotlin-maven-plugin:

<plugin>
  <groupId>org.jetbrains.kotlin</groupId>
  <artifactId>kotlin-maven-plugin</artifactId>
  <!-- ... snip ... -->
  <configuration>
    <compilerPlugins combine.children="append">
      <plugin>all-open</plugin>
    </compilerPlugins>
    <pluginOptions>
      <option>
        all-open:annotation=org.openjdk.jmh.annotations.BenchmarkMode
      </option>
    </pluginOptions>
  </configuration>
  <dependencies>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-maven-allopen</artifactId>
      <version>${kotlin.version}</version>
    </dependency>
  </dependencies>
</plugin>

Building Kotlin JMH Benchmarks with Maven

Compiling the Kotlin benchmark requires the following steps:

  1. Compile the Kotlin source to bytecode:

    <plugin>
      <artifactId>kotlin-maven-plugin</artifactId>
      <groupId>org.jetbrains.kotlin</groupId>
      <version>${kotlin.version}</version>
      <executions>
        <execution>
          <id>process-sources</id>
          <phase>generate-sources</phase>
          <goals>
            <goal>compile</goal>
          </goals>
          <configuration>
            <sourceDirs>
             <sourceDir>${project.basedir}/src/main/kotlin</sourceDir>
            </sourceDirs>
          </configuration>
        </execution>
      </executions>
    </plugin>
    
  2. Run the JMH bytecode generator, which:

    • Analyzes the compiled (Kotlin) benchmark classes
    • Generates additional Java source files for benchmark infrastructure
    • Creates metadata for benchmark registration and execution
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>exec-maven-plugin</artifactId>
      <executions>
        <execution>
          <id>generate-benchmark-code</id>
          <phase>process-sources</phase>
          <goals>
            <goal>java</goal>
          </goals>
          <configuration>
            <includePluginDependencies>true</includePluginDependencies>
            <mainClass>
              org.openjdk.jmh.generators.bytecode.JmhBytecodeGenerator
            </mainClass>
            <arguments>
              <!-- compiled-bytecode-dir -->
              <argument>${project.basedir}/target/classes/</argument>
              <!-- output-source-dir -->
              <argument>
                  ${project.basedir}/target/generated-sources/jmh/
              </argument>
              <!-- output-resource-dir -->
              <argument>${project.basedir}/target/classes/</argument>
              <!-- generator-type -->
              <argument>default</argument>
            </arguments>
          </configuration>
        </execution>
      </executions>
      <dependencies>
          <dependency>
              <groupId>org.openjdk.jmh</groupId>
              <artifactId>jmh-generator-bytecode</artifactId>
              <version>${jmh.version}</version>
          </dependency>
      </dependencies>
    </plugin>
    
  3. Add the generated JMH Java files as a source to the Maven reactor (otherwise Maven ignores the source):

    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>build-helper-maven-plugin</artifactId>
      <version>1.8</version>
      <executions>
        <execution>
          <id>add-source</id>
          <phase>process-sources</phase>
          <goals>
            <goal>add-source</goal>
          </goals>
          <configuration>
            <sources>
              <source>
                ${project.basedir}/target/generated-sources/jmh
              </source>
            </sources>
          </configuration>
        </execution>
      </executions>
    </plugin>
    
  4. Have the maven-compiler-plugin compile the whole bunch:

    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>3.8.0</version>
      <configuration>
        <compilerVersion>${javac.target}</compilerVersion>
        <source>${javac.target}</source>
        <target>${javac.target}</target>
        <compilerArgument>-proc:none</compilerArgument>
      </configuration>
    </plugin>
    
  5. Finally, you can use the maven-assembly-plugin or the maven-shade-plugin to create an uberjar (jar with dependencies). I prefer the assembly plugin because it is simpler (the JMH archetype provides an example with the maven-shade-plugin, though, if you are interested):

    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-assembly-plugin</artifactId>
      <executions>
        <execution>
          <id>make-uberjar</id>
          <phase>package</phase>
          <goals>
            <goal>single</goal>
          </goals>
          <configuration>
            <archive>
              <manifest>
                <mainClass>org.openjdk.jmh.Main</mainClass>
              </manifest>
            </archive>
            <descriptorRefs>
              <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
          </configuration>
        </execution>
      </executions>
    </plugin>
    

    Make sure to put org.openjdk.jmh.Main as the Main-Class in the jar manifest, and not your own main class. This will allow you to tweak the JMH parameters. See java -jar benchmarks-jar-with-dependencies -h.

You are now ready to run the jar with java -jar benchmarks-jar-with-dependencies.jar from the target/ directory.

Using a Annotation Processor Instead

This approach above is pretty explicit. Alternatively, we could use the jmh-generator-annprocess annotation processor with deprecated kapt or the Kotlin Symbol Processing (KSP) API (which I have yet to use). That works pretty well for a Kotlin-only project, but is more difficult to get working with mixed sources. Also, it might complicate debugging when things go wrong. The benefit is that, this way, we could skip 2 steps in the Maven build: the explicit generation of the bytecode in the exec-maven-plugin by JmhBytecodeGenerator and adding the sources to the build in the build-helper-maven-plugin.

The following would enable kapt execution as part of the kotlin-maven-plugin:

<plugin>
  <groupId>org.jetbrains.kotlin</groupId>
  <artifactId>kotlin-maven-plugin</artifactId>
  <executions>
    <!-- ... snipped compilation execution ... -->
    <execution>
      <id>kapt</id>
      <goals>
          <goal>kapt</goal>
      </goals>
      <configuration>
        <sourceDirs>
          <sourceDir>src/main/kotlin</sourceDir>
        </sourceDirs>
        <annotationProcessorPaths>
          <annotationProcessorPath>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-generator-annprocess</artifactId>
            <version>${jmh.version}</version>
          </annotationProcessorPath>
        </annotationProcessorPaths>
      </configuration>
    </execution>
  </executions>
  <dependencies>
    <dependency>
      <groupId>org.openjdk.jmh</groupId>
      <artifactId>jmh-generator-annprocess</artifactId>
      <version>${jmh.version}</version>
    </dependency>
  </dependencies>
</plugin>

Example: Benchmark Creating Signatures With Different-Length RSA Keys

package com.relentlesscoding.benchmarks

import org.openjdk.jmh.annotations.*
import org.openjdk.jmh.infra.Blackhole
import java.util.*
import java.util.concurrent.TimeUnit
import java.security.*
import java.security.spec.PKCS8EncodedKeySpec
import java.security.interfaces.RSAPrivateCrtKey

// override with -bm <thrpt | avgt | sample | ss | all>
@BenchmarkMode(Mode.Throughput)
// override with -wi <int>
@Warmup(iterations = 1)  
// override with -i <int>
@Measurement(iterations = 1)  
// override with -tu <s | ms | us | ns>
@OutputTimeUnit(TimeUnit.MILLISECONDS)  
@Fork(1)
open class RSASigningMessageBenchmark {

  @State(Scope.Benchmark)
  open class BenchmarkState {

    private val aPKCS8Encoded512BitRSAPrivKey = "MIIBVgIBADANBgkqhkiG9w0BAQEFAASCAUAwggE8AgEAAkEA0fKfr7qTLwt+UBk8oTVks1MmO6L6Fm/15xG8jLx/y274swbQUXO0RZtZWMr8+W00FgdegRNMNgxdIMfZFyfFzQIDAQABAkEAnjos/1Ot+Za/674ZY6XJ7xyLhAagVKisuyky4R5vcfEjK4GgpQO8oOTVP09w8SOZdSa5Vb+6coFpvWs5zwHaAQIhAOhDNww0vkua0HMw/8+jEN9ZpY5NDGCD2mIkNGcaKg75AiEA52eTqbTcC4O5yLHDIblvw6YSylVmdATQOi8i9PZL3nUCIQC70icAyuIb94ybqkMjoMUzKKZ1pZ7dqaJ+/LIXshPS2QIgGNCusSBICKQTpEYL2u374ktI8JG/7uklO1gas5JGCJECIQDEAyVyFTisbbFHqQCUdduakO2XD1yZ5a4ACSTdJoxkBg=="
    private val aPKCS8Encoded768BitRSAPrivKey = "MIIB5wIBADANBgkqhkiG9w0BAQEFAASCAdEwggHNAgEAAmEAy5XJREg4ki+HZAzgTF7r6wDUy0gi1Njt4svAWfWT3r/vNGcVep3aGEfP5C262257soZMw9wmpbCtFU97FH8nJ3tb5yc6qzvyVnk56M3WMStLtC5dPenExbDLst08co/fAgMBAAECYQCDtXkLiunGcadW7BmkbviUBeqlRRr7twhX5Nehm4Y54tR/g31a4Yq6kKMHjSpJUjTeQ5EmmqxVVErwxjgJYj3FR+aoy8hk39tTW0OytgXGAmYUyRqpQq7/o63H+zvMMPECMQD4K31WQLRfNsyCDsPaHwzL09H6jqtObYXUCDdNeff+9dE8611qGEA94ypOPQpbmrcCMQDSAi2A2E7+9zjTDdkGQTfhTguyuesLMmXZNU6sDeFGdwZpI1P0/pbBw0XfendfLBkCMQCofWZgPBf6GQtqNboVCkW20T5b3adC3SsiVN2vNWMBcEW6FZZbpNFg8y1S5zB0FysCMQCbT+j/JPonLgbkb5VVPt5oziNwpnbh7P/Nx9LLA+jbCCPBldL9mVs9KYF/aT7nL+ECMQDeYptxqot70d4nWsJoZ5QE8RiU6y7ZGjbDh48QQ87HXjj0ezvr/qMg+eKr8U9/mWs="

    private fun parseKey(pkcs8EncodedRSAPrivateKey: String): RSAPrivateCrtKey {
      val decoded = Base64.getDecoder().decode(pkcs8EncodedRSAPrivateKey)
      val keySpec = PKCS8EncodedKeySpec(decoded)
      val keyFactory = KeyFactory.getInstance("RSA")
      return keyFactory.generatePrivate(keySpec) as RSAPrivateCrtKey
    }

    val a512BitRSAPrivateKey = parseKey(aPKCS8Encoded512BitRSAPrivKey)
    val a768BitRSAPrivateKey = parseKey(aPKCS8Encoded768BitRSAPrivKey)

    fun sign(privateKey: PrivateKey, message: ByteArray): ByteArray {
      val signature = Signature.getInstance("SHA256WithRSA")
      signature.initSign(privateKey)
      signature.update(message)
      return Base64.getEncoder().encode(signature.sign())
    }

    val message = "foo".toByteArray()
  }

  @Benchmark
  fun signWith512BitRSAKey(s: BenchmarkState, bh: Blackhole) {
    bh.consume(s.sign(s.a512BitRSAPrivateKey, s.message))
  }

  @Benchmark
  fun signWith768BitRSAKey(s: BenchmarkState, bh: Blackhole) {
    bh.consume(s.sign(s.a768BitRSAPrivateKey, s.message))
  }
}

RSA keys were generated with:

$ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:512 | \
    openssl pkcs8 -topk8 -nocrypt -outform DER | \
    base64

This number of bits is unsafe for production use. I only chose it to get a relatively small key to prevent infinite horizontal scrolling in this blog post. In production, use a minimal of 3072-bit RSA keys, which NIST says is safe to use through 2030.

Build the example:

$ mvn clean package

Run the example:

$ java -jar target/benchmarks.jar
# JMH version: 1.37
# VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7
# VM invoker: /usr/lib/jvm/java-21-openjdk/bin/java
# VM options: <none>
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 1 iterations, 10 s each
# Measurement: 1 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.relentlesscoding.RSASigningMessageBenchmark.signWith512BitRSAKey

# Run progress: 0.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: 19.995 ops/ms
Iteration   1: 20.506 ops/ms

Result "com.relentlesscoding.RSASigningMessageBenchmark.signWith512BitRSAKey":
  20.506 ops/ms

# JMH version: 1.37
# VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7
# VM invoker: /usr/lib/jvm/java-21-openjdk/bin/java
# VM options: <none>
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 1 iterations, 10 s each
# Measurement: 1 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.relentlesscoding.RSASigningMessageBenchmark.signWith768BitRSAKey

# Run progress: 50.00% complete, ETA 00:00:20
# Fork: 1 of 1
# Warmup Iteration   1: 8.953 ops/ms
Iteration   1: 9.193 ops/ms

Result "com.relentlesscoding.RSASigningMessageBenchmark.signWith768BitRSAKey":
  9.193 ops/ms

# Run complete. Total time: 00:00:40

# ... snip ...

Benchmark                                         Mode  Cnt   Score   Error   Units
RSASigningMessageBenchmark.signWith512BitRSAKey  thrpt       20.506          ops/ms
RSASigningMessageBenchmark.signWith768BitRSAKey  thrpt        9.193          ops/ms

Based on this result, it looks like signing with a 512-bit RSA key is about twice as fast as signing with a 768-bit key on my hardware.