Skip to main content
Blog

Efficient Image Handling with Spring

By 3 september 2015juli 4th, 2017No Comments

In a project we are working on, we need to serve a lot of images. These images are user-controlled content and as such we need to dynamically scale and serve them. Throughout the website, we use these images in different sizes. We don’t want to predefine all these sizes since they tend to change quite often with the layout. So how can we make this efficient?

For this, we started out with a controller which looks something like this:

@RequestMapping(value = "image/{width}/{height}/{id}", method = RequestMethod.GET)
public void getImage(final HttpServletResponse response, @PathVariable final int width, @PathVariable final int height, @PathVariable final long id) {
  response.setStatus(HttpServletResponse.SC_OK);
  response.setContentType("image/png");
  InputStream in = imageUtil.openInputStream(id, width, height);
  IOUtils.copy(response.getOutputStream()
  response.getOutputStream().close();
}

We have some imageUtil that does the actual work.  If an image is requested in some size, we simply ask the imageUtil for an InputStream with the content. ImageUtil fetches the image from the database, resizes it and caches it for the next time. It also handles broken images and serves up filler content for that. We’ll not go into this functionality in this blog post. The controller then set the content-type (we assume only PNG images for now) and streams it to the browser.

While this works, it has some problems. Scaling these images takes quite some computing time. If some page is requested with a lot of images that are not cached, a lot of server threads will be busy scaling images. This will quite possibly exhaust our server threads. We’d rather serve a lot of pages quickly and have a bit slower response on new images. So how can we control this?

DeferredResult

The first thing what we can do is not block server threads when we are serving images. For that, Spring has DeferredResult. This allows the servlet container to park the socket connection with the browser, use the server thread for something else and come back to the socket connection when there is an actual result.

@Autowired
ThreadPoolTaskExecutor executor;

@RequestMapping(value = "image/{width}/{height}/{id}", method = RequestMethod.GET)
public DeferredImageResult getImage(@PathVariable final int width, @PathVariable final int height, @PathVariable final long id) {
  final DeferredImageResult deferredResult = new DeferredImageResult(id, width, height, createDefaultResponseEntity());
  executor.execute(deferredResult);
  return deferredResult;
}

private ResponseEntity createDefaultResponseEntity() {
  MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
  headers.set("Retry-After", "60");
  return new ResponseEntity<>(headers,HttpStatus.SERVICE_UNAVAILABLE);
}

Note that we have our own DeferredImageResult (we’ll dive into that next) which is also a Runnable so we can execute at some point. The executor is a thread pool which we can control and will execute all the image requests while limiting the number of resources. We pass along a default response object for if it takes too long for the request to be finalised. We could e.g. send a default filler image but in this case we simply tell the browser that we’re too busy and that it should retry in a minute.

[box type=”info”] Instead of making the DeferredImageResult a Runnable, we could also pass it along to some method creating the image and mark that method as @Async. However, later on, we want to have a bit more control over how and when tasks are executed so we explicitly use an executor here.[/box]

So how does this DeferredImageResult look like?

class DeferredImageResult extends DeferredResult<ResponseEntity> implements Runnable {
  private long id;
  private int width;
  private int height;
  public DeferredImageResult(long id, int width, int height, Object timeoutObject) {
    super(30000L,timeoutObject);
    this.id = id;
    this.width = width;
    this.height = height;
  }
  public void run() {
    if (!isSetOrExpired()) {
      ResponseEntity result = build(id, width, height);
      setResult(result);
    }
  }
  private ResponseEntity build(long id, int width, int height) {
    try {
      InputStream is = imageUtil.openInputStream(id, width, height);
      MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
      headers.set("Content-Type", "image/png");
      ByteArrayResource res = new ByteArrayResource(IOUtils.toByteArray(is));
      return new ResponseEntity<>(res, headers, HttpStatus.OK);
    } catch (IOException e) {
      LOGGER.error("Could not read inputstream to byte[].", e);
      return new ResponseEntity<>(HttpStatus.INTERNAL_SERVER_ERROR);    
    }
  }
}

The run() method first checks if the browser didn’t already give up. If not, it builds a response and sets the result. As soon as the result is set, Spring will return the result to the browser. The build() function does the same thing as before: ask for an inputstream and set the content-type. If you look at the constructor, you can also see that we define a timeout of 30 seconds after which Spring should simply return the timeoutObject. That is the retry-after-a-minute object we defined before.

[box type=”info”] Note that we use a ByteArrayResource here, not an InputStreamResource. This is because Spring would read the input stream twice in our implementation to determine the content length. Since we already control the number of threads, we can also control memory usage and as such we don’t have to stream.[/box] [box type=”warning”] To support this, you will have to mark all your web filters to support async. You’ll have to tell the Spring Dispatcher servlet that as well. (E.g., add some <async-supported>true</async-supported> tags to your filter definitions in the web.xml.) [/box]

With this, we now have asynchronous server side handling, from the browsers perspective nothing has changed.

Browser caching

We can do more, such as making use of browser caching. If the image hasn’t changed we don’t have to serve it again. For that, browsers can send a Last-Modified header along with the request. If the image hasn’t changed since then, we don’t have to serve the image again:

@RequestMapping(value = "image/{width}/{height}/{id}", method = RequestMethod.GET)
public DeferredImageResult getImage(@PathVariable final int width, @PathVariable final int height, @PathVariable final long id) {
  if (request.checkNotModified(imageUtil.getLastModified(id, width, height))) {
    return null;
  }
  final DeferredImageResult deferredResult = new DeferredImageResult(id, width, height, createDefaultResponseEntity());
  executor.execute(deferredResult);
  return deferredResult;
}

Also, we can specify how long a browser can cache the image, again saving us from serving the image. For that, we can set a Cache-Control header in the build() function:

private ResponseEntity build(long id, int width, int height) {
  try {
    InputStream is = imageUtil.openInputStream(id, width, height);
    MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
    headers.set("Content-Type", "image/png");
    headers.set("Cache-Control", "public, max-age=3600");
    ByteArrayResource res = new ByteArrayResource(IOUtils.toByteArray(is));
    return new ResponseEntity<>(res, headers, HttpStatus.OK);
  } catch (IOException e) {
    LOGGER.error("Could not read inputstream to byte[].", e);
    return new ResponseEntity<>(HttpStatus.INTERNAL_SERVER_ERROR); 
  }
}

Priority queue

One last thing we have implemented is prioritising cached images. Requests that ask for a cached image should be served before requests for whom we need to resize an image. This way, we avoid that the queue can be filled up with slow requests and thereby blocking the fast requests. For that, we can make the DeferredImageResult implement Comparable:

class DeferredImageResult extends DeferredResult<ResponseEntity> implements Comparable, Runnable {
  private long id;
  private int width;
  private int height;
  private Long timestamp;
  private boolean isCached;

  public DeferredImageResult(long id, int width, int height, Object timeoutObject) {
    super(30000L,timeoutObject);
    this.timestamp = System.currentTimeMillis();
    this.isCached = imageUtil.isCached(id, width, height);
    this.id = id;
    this.width = width;
    this.height = height;
  }

  @Override
  public int compareTo(DeferredImageResult dir) {
    if (isCached == dir.isCached) {
      return timestamp.compareTo(dir.timestamp);
    } else if (isCached) {
      return 1;
    } else {
      return -1;
    }
  }

  ...
}

As you can see, cached results are given a higher priority. If the cache state is the same, we’ll use the timestamp of the request. We can now change the executor to use a PriorityBlockingQueue so that it uses this priority. We created our own extension to the ThreadPoolTaskExecutor for that, so we can also control the thread pool settings.

Using this, you’ll be able to handle a large amount of traffic even when you are resizing images dynamically. In a worst case scenario the images will load slowly, but the requests will not bring down your system.