Java HeapDump analysis in Docker container


There are many ways to get a heap dump and many tools to analyse it.

In this pose, I shall discuss memory analysis using
- openJDK8 docker image
- "jhat" (Java heap analysis tool) - available with JDK

Limitation - JDK base image is not used in production because we only need a JRE to run Java. But this post only explains using JDK.

Run a java code

Dockerfile:
    FROM openjdk:8
    COPY target/demo-0.0.1-SNAPSHOT.jar /usr/app/
    WORKDIR /usr/app
    ENTRYPOINT ["java","-jar","demo-0.0.1-SNAPSHOT.jar"]

Note: We are NOT using Alpine image - reason explained towards end of post.

In above, output of my java build is present in /target/deo-0.0.1-SNAPSHOT.jar. We shall be running this JAR in docker image.

Build image
    docker build -t demo-spring-boot-app:0.0.1 .

Run image
    docker run --rm -p 80:8080 --name demo-spring-boot-application demo-spring-boot-app:0.0.1

With this our app is running in docker container.

Take memory dump

Step 1: Find docker container's ID. Mine is 388588d04d56.

Step 2: Create memory profile
    docker exec 388588d04d56 jcmd 1 GC.heap_dump /tmp/heapdump.hprof
Output should look like:
    1:
    Heap dump file created

Step 3: Use java's jhat to analyse memory profile
Output looks like:
    Reading from /tmp/heapdump.hprof...
    Dump file created Fri Nov 08 06:53:52 UTC 2019
    Snapshot read, resolving...
    Resolving 198234 objects...
    Chasing references, expect 39 dots.......................................
    Eliminating duplicate references.......................................
    Snapshot resolved.
    Started HTTP server on port 7000
    Server is ready.

Analyse memory dump

With jhat a server started running on port 7000 in docker container. But it was not exposed when we first created image.

So we have two options.

Option 1: Expose port 7000 in container when creating it
Go back to step where we run docker image, and expose port 7000 also. So command to run changes to
    docker run --rm -p 80:8080 -p 7000:7000 --name demo-spring-boot-application demo-spring-boot-app:0.0.1

OR

Option 2: socat (https://hub.docker.com/r/alpine/socat/)

socat is a lovely docker image that lets us expose a port after an image is created. So in this case, it lets us expose port 7000 of our container after it was created.

It works by acting as a proxy redirecting traffic to target docker image and a port.

So run this command
    docker run --rm --publish 8081:7000 --link demo-spring-boot-application:target alpine/socat tcp-listen:7000,fork,reuseaddr tcp-connect:target:7000

Note: I am exposing port 7000 as 8081 on local just to demonstrate how socat performs port forwarding.

Analyse

Open http://localhost:8081 (if we used second option)

It opens a page that talks about loaded objects as classes. If you go deep, and open a class, it talks about memory consumption, references etc for given object.

On home page on bottom, it has a link saying "Show instance count...". Open this page. It talks about number of instances for each class. This is very useful for identifying objects that are causing memory leak.

On home page on bottom is another link "Show heap histogram". It shows a tabular summary of instances - instance counts and memory consumed.

Problems I faced

Unable to take GC heap dump

Command: docker exec jcmd 1 GC.heap_dump /tmp/heapdump.hprof
Error I got
    1:
    com.sun.tools.attach.AttachNotSupportedException: Unable to get pid of LinuxThreads manager thread
    at sun.tools.attach.LinuxVirtualMachine.(LinuxVirtualMachine.java:86)
    at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
    at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
    at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:147)
    at sun.tools.jcmd.JCmd.main(JCmd.java:131)

This command indicates that there is something wrong with JVM.
Reason: I had used alpine image instead of normal image. Alpine - to have a small footprint - has a different mechanism of running things. So one of the mechanism needed for jcmd is missing.

My Dockerfile (dont use this one for taking dump) was
    FROM java:8-jdk-alpine
    COPY target/demo-0.0.1-SNAPSHOT.jar /usr/app/
    WORKDIR /usr/app
    ENTRYPOINT ["java","-jar","demo-0.0.1-SNAPSHOT.jar"]





Comments