Writing a video player with Java Swing

05.02.2018 | Michael Galetzka

TL;DR: Don’t do it (unless you fancy surprises).

So, suppose you’ve generated a few images which make a nice animation when shown in quick succession. Not like one of those 60fps 4k movies, more like those choppy weather forecast animations on the weather app that came preinstalled with your phone and has approximately 3fps. Now suppose you want to play this (very slow) video/animation with Java Swing.

That doesn’t sound so hard, you hear yourself thinking. You can read the images into a BufferedImage with Java’s ImageIO library and then display that image in a JPanel (or one of the other components that can display an image). And the good news is that this actually kind of works – if you don’t mind the huge latency you experience for every new frame.

The problem is that drawing in Swing is really, really slow. On my machine, drawing a single fullscreen BufferedImage frame in a JPanel takes between 10ms and 150ms. Since our weather forecast is probably made up of different layers, we end up blocking the Event Dispatch Thread (EDT) almost all the time just with our drawing routine. But we can do better than that! What we need is some kind of double buffering, where we can render each frame in a separate thread so it won’t block the EDT. When our frame has finished rendering, we flip the buffer and display it.

When looking at Oracle’s JavaDoc, it seems like the best way to do this is to use the createBufferedStrategy(2) method. After a quick search, we see that JPanel, of course, does not support this. No biggie, we can simply use the AWT Canvas class instead. After we’ve fumbled this into our code, we delightedly see that it works great - that is, until we try to resize the window.

The canvas flickers violently every time the window size of our player changes. A quick debugging session and some guesswork lead to the conclusion that our canvas is cleared by the JVM, or even natively by the OS, when resizing. While our buffer content will eventually be restored, for a few milliseconds the content is blank - hence the flickering. I found some hacks and JVM arguments that should suppress the clearing of the canvas, but nothing that looked very robust.

This means we have to fumble the code back again and use a JPanel with a custom double buffer implementation. We can easily create each half of the buffer with the following code:

Image bufferImage = panel.createImage(width, height);

Now we can happily prerender our different layers into the buffer, and the panel should then be able to render this single image with constant performance - except that it doesn’t. The first few times the image is rendered, it takes about 100ms. After that, it only takes a few microseconds. So it looks like the JVM outsmarted us by copying the image from our main memory into the video card memory. This is usually great to speed up the drawing of frequently used images, but not so much if we want predictable performance: if a frame is drawn 100ms too early or too late, it’s very noticeable.

The fix for this problem is simple, we just have to make sure our buffer is accelerated from the get-go:

Image bufferImage = panel.createVolatileImage(width, height);

VolatileImage is a bit trickier to use than BufferedImage, so have a look at a few examples before using it everywhere in your codebase. It also uses video memory, which is often a lot more scarce than system memory, so keep that in mind.

While searching for a way to speed up the rendering, I noticed that, by default, Java does not use OpenGL. It can be activated with the JVM parameter -Dsun.java2d.opengl=true (Oracle docs). While this did indeed speed up my custom rendering, it made the rest of the application incredibly slow. Resizing a window felt like I was mining bitcoins in the background. So I disabled the OpenGL acceleration again, slightly disappointed. However, you should definitely check it out if you need a lot of drawing performance.

Now that we have our basic drawing problems sorted out, we can start implementing the timer, which advances the displayed frame to the next one in the timeline. My initial concept looked like this:

Timer start
 |
 | 500ms
 v
Timer event -----> Request frame rendering
 |                        |
 | 500ms                  v
 |                       Flip double buffer
 v
Timer event -----> Request frame rendering
 |
 | repeat
 v

Once the timer surpasses the wait time between frames, it signals the render thread to start rendering the next frame. When the rendering is finished, the render thread flips the double buffer so the UI is able to display the new image. This should work great in theory - easy implementation, small memory footprint and reliable framerate. As an additional bonus, the UI actions, like ‘next frame’, can use the same mechanism to display new images.

Implemented. Done. Looks good. Chapter closed - or is it?

When watching closely, I saw that the playback was still sometimes ‘out of tune’, as though it was lagging. The problem was not severe and only appeared every few seconds; but it still bugged me; where was this unpredictable lag coming from? After slapping timers on every part of the video player, the culprit turned out to be the following piece of code:

private void renderFrame(Graphics2D g) {
    g.setColor(Color.white);
    g.clearRect(0, 0, width, height);

    // draw the frame
}

That call to clearRect usually only takes a few microseconds, but every 10th frame or so it takes between 100 and 200 milliseconds. That is a difference of five orders of magnitude. I don’t know what the JVM is doing in all that time, but a 100ms delay sure is a long time for clearing a texture. So, back to the drawing board:

Timer start -----> Request frame rendering of *next* frame
 |                        |
 | 500ms                  v
 |                       Rendering finished
 v
Timer event -----> Flip double buffer, request rendering of next frame
 |                        |
 | 500ms                  v
 |                       Rendering finished
 v
Timer event -----> etc.
 |
 | repeat
 v

As soon as the timer starts, it requests the next frame to be rendered in a separate thread. When the timer event comes around, we can immediately flip the buffer. If the frame is not yet finished by then, we can either wait for it to finish or drop it entirely. Once the buffer is flipped, we again queue the next frame for the renderer. This way, the renderer can take a few extra milliseconds every other frame without impacting the display output. This finally resulted in a reliable framerate!

Here are some quick stats to show the possible framerates I’ve been able to get with this approach:

One would expect that the player gets slower the bigger the image it has to draw. That is indeed the case, except at some point it decides to unleash the power of Grayskull and becomes astoundingly fast. If you haven’t guessed, 3840x2160 is my monitor’s native resolution, so that might have something to do with it; but all things considered it doesn’t make any sense that the smaller players are that much slower.

One downside of the new approach is that the code is now a lot more complex than before, almost doubling the amount of code. There are two different render paths for UI actions such as jumping to the previous frame (which requires the current frame to be rendered) or the timer event (which needs the next frame to be rendered). It’s also more difficult to debug the code, which is already complex enough with the different threads for UI, timer, rendering, and loading the frame data from disk.

I think all of this shows nicely how sometimes a simple idea can gradually grow more complex if you’re aiming at getting that extra bit of performance or quality. Also, and that’s my final note on this topic: Java Swing is a pain to work with.

The title image was published by Jeremy Yap under the Unsplash License.

Share via