Build Kotlin JMH Benchmarks With Maven
In this post, we will look at how to set up the Maven infrastructure so we can benchmark Kotlin code with JMH (the Java Microbenchmarking Harness).
This post does not look in-depth on how to write JMH benchmarks, but rather how to build and run them.
Create a Sample JMH Project
Helpfully, the JMH project has a Maven archetype for Kotlin:
$ mvn archetype:generate \
-DinteractiveMode=false \
-DarchetypeGroupId=org.openjdk.jmh \
-DarchetypeArtifactId=jmh-kotlin-benchmark-archetype \
-DgroupId=com.relentlesscoding \
-DartifactId=benchmarks \
-Dversion=1.0
This creates a sample project where we can play around with a simple benchmark written in Kotlin.
Things to Keep in Mind When Creating JMH Benchmarks
Benchmarks look like this:
package com.relentlesscoding
import org.openjdk.jmh.annotations.Benchmark
open class MyBenchmark {
@Benchmark
fun benchmarkMethod() {
}
}
See a more elaborate benchmark example that benchmarks signing efficiency of different-size RSA keys at the end of this post. Also see the examples the JMH team provides.
Notice that the class needs to be declared open
:
Benchmark classes should not be final.
[com.sample.RSASigningMessageBenchmark]
This is because JMH needs to be able to subclass it to add instrumentation code and implement the benchmarking machinery.
If you or your IDE objects to opening classes, you could also use the all-open
compiler plugin that is part of the kotlin-maven-plugin
:
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<!-- ... snip ... -->
<configuration>
<compilerPlugins combine.children="append">
<plugin>all-open</plugin>
</compilerPlugins>
<pluginOptions>
<option>
all-open:annotation=org.openjdk.jmh.annotations.BenchmarkMode
</option>
</pluginOptions>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-allopen</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
Building Kotlin JMH Benchmarks with Maven
Compiling the Kotlin benchmark requires the following steps:
-
Compile the Kotlin source to bytecode:
<plugin> <artifactId>kotlin-maven-plugin</artifactId> <groupId>org.jetbrains.kotlin</groupId> <version>${kotlin.version}</version> <executions> <execution> <id>process-sources</id> <phase>generate-sources</phase> <goals> <goal>compile</goal> </goals> <configuration> <sourceDirs> <sourceDir>${project.basedir}/src/main/kotlin</sourceDir> </sourceDirs> </configuration> </execution> </executions> </plugin>
-
Run the JMH bytecode generator, which:
- Analyzes the compiled (Kotlin) benchmark classes
- Generates additional Java source files for benchmark infrastructure
- Creates metadata for benchmark registration and execution
<plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <executions> <execution> <id>generate-benchmark-code</id> <phase>process-sources</phase> <goals> <goal>java</goal> </goals> <configuration> <includePluginDependencies>true</includePluginDependencies> <mainClass> org.openjdk.jmh.generators.bytecode.JmhBytecodeGenerator </mainClass> <arguments> <!-- compiled-bytecode-dir --> <argument>${project.basedir}/target/classes/</argument> <!-- output-source-dir --> <argument> ${project.basedir}/target/generated-sources/jmh/ </argument> <!-- output-resource-dir --> <argument>${project.basedir}/target/classes/</argument> <!-- generator-type --> <argument>default</argument> </arguments> </configuration> </execution> </executions> <dependencies> <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-generator-bytecode</artifactId> <version>${jmh.version}</version> </dependency> </dependencies> </plugin>
-
Add the generated JMH Java files as a source to the Maven reactor (otherwise Maven ignores the source):
<plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>1.8</version> <executions> <execution> <id>add-source</id> <phase>process-sources</phase> <goals> <goal>add-source</goal> </goals> <configuration> <sources> <source> ${project.basedir}/target/generated-sources/jmh </source> </sources> </configuration> </execution> </executions> </plugin>
-
Have the
maven-compiler-plugin
compile the whole bunch:<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.8.0</version> <configuration> <compilerVersion>${javac.target}</compilerVersion> <source>${javac.target}</source> <target>${javac.target}</target> <compilerArgument>-proc:none</compilerArgument> </configuration> </plugin>
-
Finally, you can use the
maven-assembly-plugin
or themaven-shade-plugin
to create an uberjar (jar with dependencies). I prefer the assembly plugin because it is simpler (the JMH archetype provides an example with themaven-shade-plugin
, though, if you are interested):<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <executions> <execution> <id>make-uberjar</id> <phase>package</phase> <goals> <goal>single</goal> </goals> <configuration> <archive> <manifest> <mainClass>org.openjdk.jmh.Main</mainClass> </manifest> </archive> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> </execution> </executions> </plugin>
Make sure to put
org.openjdk.jmh.Main
as theMain-Class
in the jar manifest, and not your own main class. This will allow you to tweak the JMH parameters. Seejava -jar benchmarks-jar-with-dependencies -h
.
You are now ready to run the jar with java -jar benchmarks-jar-with-dependencies.jar
from the target/
directory.
Using a Annotation Processor Instead
This approach above is pretty explicit. Alternatively, we could use the
jmh-generator-annprocess
annotation processor with deprecated kapt
or
the Kotlin Symbol Processing (KSP) API (which I have yet to use). That
works pretty well for a Kotlin-only project, but is more difficult to get
working with mixed sources. Also, it might complicate debugging when things go
wrong. The benefit is that, this way, we could skip 2 steps in the Maven build:
the explicit generation of the bytecode in the exec-maven-plugin
by
JmhBytecodeGenerator
and adding the sources to the build in the
build-helper-maven-plugin
.
The following would enable kapt
execution as part of the
kotlin-maven-plugin
:
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<executions>
<!-- ... snipped compilation execution ... -->
<execution>
<id>kapt</id>
<goals>
<goal>kapt</goal>
</goals>
<configuration>
<sourceDirs>
<sourceDir>src/main/kotlin</sourceDir>
</sourceDirs>
<annotationProcessorPaths>
<annotationProcessorPath>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
</annotationProcessorPath>
</annotationProcessorPaths>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
</dependency>
</dependencies>
</plugin>
Example: Benchmark Creating Signatures With Different-Length RSA Keys
package com.relentlesscoding.benchmarks
import org.openjdk.jmh.annotations.*
import org.openjdk.jmh.infra.Blackhole
import java.util.*
import java.util.concurrent.TimeUnit
import java.security.*
import java.security.spec.PKCS8EncodedKeySpec
import java.security.interfaces.RSAPrivateCrtKey
// override with -bm <thrpt | avgt | sample | ss | all>
@BenchmarkMode(Mode.Throughput)
// override with -wi <int>
@Warmup(iterations = 1)
// override with -i <int>
@Measurement(iterations = 1)
// override with -tu <s | ms | us | ns>
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@Fork(1)
open class RSASigningMessageBenchmark {
@State(Scope.Benchmark)
open class BenchmarkState {
private val aPKCS8Encoded512BitRSAPrivKey = "MIIBVgIBADANBgkqhkiG9w0BAQEFAASCAUAwggE8AgEAAkEA0fKfr7qTLwt+UBk8oTVks1MmO6L6Fm/15xG8jLx/y274swbQUXO0RZtZWMr8+W00FgdegRNMNgxdIMfZFyfFzQIDAQABAkEAnjos/1Ot+Za/674ZY6XJ7xyLhAagVKisuyky4R5vcfEjK4GgpQO8oOTVP09w8SOZdSa5Vb+6coFpvWs5zwHaAQIhAOhDNww0vkua0HMw/8+jEN9ZpY5NDGCD2mIkNGcaKg75AiEA52eTqbTcC4O5yLHDIblvw6YSylVmdATQOi8i9PZL3nUCIQC70icAyuIb94ybqkMjoMUzKKZ1pZ7dqaJ+/LIXshPS2QIgGNCusSBICKQTpEYL2u374ktI8JG/7uklO1gas5JGCJECIQDEAyVyFTisbbFHqQCUdduakO2XD1yZ5a4ACSTdJoxkBg=="
private val aPKCS8Encoded768BitRSAPrivKey = "MIIB5wIBADANBgkqhkiG9w0BAQEFAASCAdEwggHNAgEAAmEAy5XJREg4ki+HZAzgTF7r6wDUy0gi1Njt4svAWfWT3r/vNGcVep3aGEfP5C262257soZMw9wmpbCtFU97FH8nJ3tb5yc6qzvyVnk56M3WMStLtC5dPenExbDLst08co/fAgMBAAECYQCDtXkLiunGcadW7BmkbviUBeqlRRr7twhX5Nehm4Y54tR/g31a4Yq6kKMHjSpJUjTeQ5EmmqxVVErwxjgJYj3FR+aoy8hk39tTW0OytgXGAmYUyRqpQq7/o63H+zvMMPECMQD4K31WQLRfNsyCDsPaHwzL09H6jqtObYXUCDdNeff+9dE8611qGEA94ypOPQpbmrcCMQDSAi2A2E7+9zjTDdkGQTfhTguyuesLMmXZNU6sDeFGdwZpI1P0/pbBw0XfendfLBkCMQCofWZgPBf6GQtqNboVCkW20T5b3adC3SsiVN2vNWMBcEW6FZZbpNFg8y1S5zB0FysCMQCbT+j/JPonLgbkb5VVPt5oziNwpnbh7P/Nx9LLA+jbCCPBldL9mVs9KYF/aT7nL+ECMQDeYptxqot70d4nWsJoZ5QE8RiU6y7ZGjbDh48QQ87HXjj0ezvr/qMg+eKr8U9/mWs="
private fun parseKey(pkcs8EncodedRSAPrivateKey: String): RSAPrivateCrtKey {
val decoded = Base64.getDecoder().decode(pkcs8EncodedRSAPrivateKey)
val keySpec = PKCS8EncodedKeySpec(decoded)
val keyFactory = KeyFactory.getInstance("RSA")
return keyFactory.generatePrivate(keySpec) as RSAPrivateCrtKey
}
val a512BitRSAPrivateKey = parseKey(aPKCS8Encoded512BitRSAPrivKey)
val a768BitRSAPrivateKey = parseKey(aPKCS8Encoded768BitRSAPrivKey)
fun sign(privateKey: PrivateKey, message: ByteArray): ByteArray {
val signature = Signature.getInstance("SHA256WithRSA")
signature.initSign(privateKey)
signature.update(message)
return Base64.getEncoder().encode(signature.sign())
}
val message = "foo".toByteArray()
}
@Benchmark
fun signWith512BitRSAKey(s: BenchmarkState, bh: Blackhole) {
bh.consume(s.sign(s.a512BitRSAPrivateKey, s.message))
}
@Benchmark
fun signWith768BitRSAKey(s: BenchmarkState, bh: Blackhole) {
bh.consume(s.sign(s.a768BitRSAPrivateKey, s.message))
}
}
RSA keys were generated with:
$ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:512 | \
openssl pkcs8 -topk8 -nocrypt -outform DER | \
base64
This number of bits is unsafe for production use. I only chose it to get a relatively small key to prevent infinite horizontal scrolling in this blog post. In production, use a minimal of 3072-bit RSA keys, which NIST says is safe to use through 2030.
Build the example:
$ mvn clean package
Run the example:
$ java -jar target/benchmarks.jar
# JMH version: 1.37
# VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7
# VM invoker: /usr/lib/jvm/java-21-openjdk/bin/java
# VM options: <none>
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 1 iterations, 10 s each
# Measurement: 1 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.relentlesscoding.RSASigningMessageBenchmark.signWith512BitRSAKey
# Run progress: 0.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration 1: 19.995 ops/ms
Iteration 1: 20.506 ops/ms
Result "com.relentlesscoding.RSASigningMessageBenchmark.signWith512BitRSAKey":
20.506 ops/ms
# JMH version: 1.37
# VM version: JDK 21.0.6, OpenJDK 64-Bit Server VM, 21.0.6+7
# VM invoker: /usr/lib/jvm/java-21-openjdk/bin/java
# VM options: <none>
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 1 iterations, 10 s each
# Measurement: 1 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.relentlesscoding.RSASigningMessageBenchmark.signWith768BitRSAKey
# Run progress: 50.00% complete, ETA 00:00:20
# Fork: 1 of 1
# Warmup Iteration 1: 8.953 ops/ms
Iteration 1: 9.193 ops/ms
Result "com.relentlesscoding.RSASigningMessageBenchmark.signWith768BitRSAKey":
9.193 ops/ms
# Run complete. Total time: 00:00:40
# ... snip ...
Benchmark Mode Cnt Score Error Units
RSASigningMessageBenchmark.signWith512BitRSAKey thrpt 20.506 ops/ms
RSASigningMessageBenchmark.signWith768BitRSAKey thrpt 9.193 ops/ms
Based on this result, it looks like signing with a 512-bit RSA key is about twice as fast as signing with a 768-bit key on my hardware.