In this blog we will show the surprising result that Docker on Ubuntu seems to outperform Docker on CoreOS by ~30%, when tested with a java templating web service.

But first let us discuss the reason on why I have started the tests in the first place: on my blog Docker Web Performance Tests we had found the surprising result that a Rails Web Application on Docker on a CoreOS VM has about 50% better performance, than Rails directly on Windows hardware. Will we find the same result for java?

java on Windows Hardware vs. Ubuntu VirtualBox VM (beware: apples vs. bananas!)

The answer is no: the chosen java application, an Apache Camel templating web service has a factor ~4 better performance on Windows Hardware than on an Ubuntu VirtualBox machine (beware: apples vs. bananas!).

Now comparing apples with apples (all on VirtualBox VMs):

java on Ubuntu vs. Docker/Ubuntu vs. Docker/CoreOS vs. Docker/boot2docker (apples vs. apples)

The java performance on a the chosen Vagrant deploy-able Ubuntu image has almost the same performance on Docker as on native Ubuntu (only 5% performance degradation). With that good result, the Ubuntu Docker alternative has outperformed the other Docker alternatives by more than 30% and has impressed with java startup times that were more than 3 times as fast as on CoreOS.

Note: At the time of writing the blog post, this dockerization vs. virtualization vs. native comparison paper was unknown to me: it follows a more comprehensive scientific approach by testing many different performance aspects separately. I like it a lot: check it out!

This is only a small proof of concept (POC) of a hobby developer and the POC can only give hints about which road to take. Even though now the more scientific paper is available, I am still fine with having invested the time of testing and publishing, since my test results are focused on the applications I develop (Rails and java Apache Camel) and the infrastructure alternatives I have on my laptop: native Windows vs. Ubuntu VirtualBox VM vs. CoreOS VirtualBox VM vs. boot2docker Virtualbox VM. The scientific paper was only testingUbuntu 13.10 and no CoreOS I was interested in.

Document Versions

v1 original post
v2: updated CoreOS Vagrant image from v713.3.0 to v766.4.0 with no performance improvement
v3: giving CoreOS a last chance: updated to alpha v845.0.0 with no performance improvement
v4: changed introduction and re-organized Tldr; and added a link to a more scientific paper.

Tldr;

When comparing the java performance on a native Ubuntu VM with several Docker host VM alternatives (UbuntuCoreOS and boot2docker), we can see that the chosen Vagrant deploy-able Ubuntu image has almost the same performance on Docker as on native Ubuntu. With that, the Ubuntu Docker alternative has outperformed the other Docker alternatives by more than 30% and has impressed with java startup times that were more than 3 times as fast as the one of the second best candidate: CoreOS.

As expected, java performance has shown to be substantially (~4 times) higher on Windows Hardware than on VirtualBox Ubuntu VM with same number of cores. Note that this last comparison is like comparing apples with bananas: Windows on Hardware vs. Ubuntu Linux on VirtualBox virtual machine with less DRAM.

Test Scenarios

In the current blog, we perform performance measurements of a java Apache Camel application for following scenarios:

  1. application run natively on Windows 7 SP1 on the host system (8 GB DRAM, with ~2.2 GB free before start of the application; 2 core CPU i5-2520M@2.5 GHz)
  2. application run within a VirtualBox Ubuntu 14.04 LTS (64 bit) VM with 1.5 GB DRAM and 1 vCPU on the same Windows host system
  3. application run within a docker container on the above Ubuntu VM (vagrant box version 1.8.1)
  4. application run within a docker container on a CoreOS (Vagrant CoreOS stable 717.3.0 and v766.4.0) Virtualbox VM with same resources as the Ubuntu VM (1.5 GB DRAM and 1 vCPU) run on the Windows system
  5. application run within a docker container on a boot2docker Virtualbox VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU), run on the same Windows host.
    Note that boot2docker is deprecated by now, but since we have performed the Rails Docker Web Performance Tests on boot2docker, we keep it as a reference.

Considering the results we have seen with the Rails web application, we expect that 3. and 4. have higher performance than 1.

With 2. and 3. we can easily extract the difference that comes from the additional docker layer. For the option 4., we expect the same or a higher performance than in 3, since CoreOS is optimized for running Docker containers.

As in the other performance test blog, we use Apache Bench as the measurement tool.

Test Results

Java Apache Camel Startup Time

  1. application run natively on Windows on the host system
    1. NEED TO TEST AGAIN WITH STDOUT REDIRECTION INTO FILE!!!
      2 cores: Apache Camel 2.12.2 (CamelContext: camel-1) started in 3.5 +-0.6 seconds
  2. application run within a VirtualBox Ubuntu VM with 1.5 GB DRAM and 1 vCPU
    1. 1 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.0 seconds
    2. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 3.7 seconds
  3. application run within a docker container on the above Ubuntu VM
    1. 1 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.2 seconds
    2. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 4.0 seconds
  4. application run within a docker container on a CoreOS VM with same resources as the Ubuntu VM
    1. 1 vCPU, v713.3.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 15.2 seconds
    2. 2 vCPU
      1. v713.3.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 13.4 seconds
      2. v766.4.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 14.5 +- 1.5 seconds
      3. v845.0.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in [15.0, 13.2, 19.2,13.2] seconds (4 tests)
  5. application run within a docker container on a boot2docker VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU)
    1. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 24.1 +- 0,5 seconds (4 Tests)

Windows: has the quickest Apache Camel startup (95 routes) with ~3.5 sec +- 0.6 sec.

Ubuntu with or without Docker: with 2 vCPUs, the startup is slightly slower. Only with 1 vCPU, the startup takes ~40% more time on Ubuntu (both, native and in Docker).

CoreOS: Apache Camel has a 3.8 and 3,4 times longer (!) startup time on CoreOS Docker than on Windows and on Ubuntu, respectively. CoreOS is more lightweight than the Ubuntu image, and is supposed to be optimized for Docker. Therefore, this is a surprising result.

boot2docker: has the worst Apache Camel bootup time: it takes almost a 7 times longer (!!) than the same process on Windows. boot2docker is deprecated by now. This performance results is another reason not to use it anymore.

Java Apache Camel Throughput

Test script:

# test:
./ab -n 10000 -c 100 "http://<IP>:<Port>/ProvisioningEngine?action=Add%20Customer&CustomerName=ttt&offlineMode=offlineMode"

Shortly before we run the actual test, we train the application by issuing the same command with -n 100 and -c 100.

  1. application run natively on Windows
    1. 2 i5-2520M CPU cores: Requests per second:    46.3 [#/sec] (mean)
  2. application run within a VirtualBox Ubuntu VM with 1.5 GB DRAM and 1 vCPU (on an i5 host)
    1. 1 vCPU: Requests per second:    14.3 [#/sec] (mean)
    2. 2 vCPU: Requests per second:    17.9 [#/sec] (mean) (i.e. ~25% higher than 1 vCPU)
  3. application run within a docker container on the above Ubuntu VM
    1. 1 vCPU: Requests per second:    12.5 [#/sec] (mean)
    2. 2 vCPU: Requests per second:    17.0 [#/sec] (mean) (i.e. ~35% higher than 1 vCPU)
  4. application run within a docker container on a CoreOS VM with same resources as the Ubuntu VM (1.5 GB DRAM and 1 or 2 vCPU)
    1. 1 vCPU, v713.3.0:: Requests per second:    8.82 [#/sec] (mean)
    2. 2 vCPU
      1. v713.3.0: Requests per second:    13.0 [#/sec] (mean) (i.e. ~48% higher than 1 vCPU)
      2. v766.4.0: Requests per second:    11.8 [#/sec] (mean)
      3. v845.0.0: Requests per second:    13.3 [#/sec] (mean)
  5. application run within a docker container on a boot2docker VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU)
    1. 1 vCPU: not tested
    2. 2 vCPU: We get the error „Test aborted after 10 failures“, a memory allocation failure, even though we have 2 GB DRAM instead of 1.5 GB!

Note that each requests is creating tons of output on the terminal (deep tracing). However, in call cases, I have redirected the stream into a log file (using the „>“ operator).

Note also that the comparison between Windows and Linux variants is not fair, since Windows was tested on hardware with 2 real CPU cores, while the Linux variants was tested on Virtualbox VMs with one or two vCPUs (Ubuntu, Docker on Ubuntu and CoreOS) or 2 vCPUs (boot2docker). However, considering you have a Windows notebook and you want to decide, whether to perform your development and tests on native Windows vs. Linux VM vs. on Docker, this is what you get:

  • Windows: on the host system with 2 real CPU cores is ~4 times higher throughput performance than any of the Linux VM variants on 1 vCPU (comparing apples with bananas, but still relevant for my decision)
  • Ubuntu vs Docker on Ubuntu: Native Linux has a ~5 to 15% higher throughput performance than Docker on Linux
  • CoreOS: Interestingly, the Ubuntu Docker VM image has a ~40% higher throughput performance than the optimized CoreOS image. Because of this negative result for CoreOS, I have updated the v713.3.0 image to the latest versions available: stable-v766.4.0 and alpha-v845.0.0. This had no substantial impact on the (still bad) performance.
  • boot2docker: The boot2docker image has memory allocation problems. Since boot2docker is deprecated, I did not investigate this further.

Summary

We have compared the performance of a java Apache Camel application on following platforms: on a Windows 7 laptop, on an Ubuntu Virtualbox image and in a Docker container, where Docker was tested on Ubuntu, CoreOS and the official, but deprecated boot2docker image.

Note that Windows has been tested on hardware, while the Linux variants have been tested on Virtualbox VMs. This is not a fair test between Windows and Linux, but it still helps me to decide, whether I better keep performing my java development directly on my Windows laptop, or whether I can move all java development to a Virtualbox Linux VM on my Windows laptop. For Rails, I had found the surprising result, that the performance was better on the VMs than directly on the hardware. Not for java, though:

2015.10.28-10_22_08-hc_001

We have seen that the performance on Windows 7 without virtualization is ~4 times higher than on any of the Virtualbox VMs, which will account for the expected performance degradation effect of software virtualization. This is no comparison between java on Windows and java on Ubuntu. It is just a comparison of the options I have on a Windows laptop: directly work on Windows or work on the VMs. The CoreOS image has a ~30% lower performance than the Ubuntu image, which is surprising, since it is much more lightweight than the Ubuntu image.

The application’s startup times are quite good for Ubuntu and for Docker on Ubuntu. Surprisingly, both, CoreOS as well as boot2docker fall back substantially with respect to startup times (factor 3 and 6 compared to Ubuntu).

The deprecated boot2docker image has major memory allocation problems and should not be used.

Recommendation

All in all, development is best to be performed directly on the Windows laptop, with Ubuntu or Docker on Ubuntu being a good alternative because of low application startup times. CoreOS does not seem to be a good alternative, since in the development phase, I often need to restart the application, which takes ~13 sec on CoreOS instead of ~3.5 to 4 sec on Windows hardware or on the Ubuntu VM’s Docker.

Performance tests are best to be done on Windows hardware (or on HW-virtualized VMs, which I have not performance-tested yet). However, because of the deployment advantages, I would like to deliver my application as a Docker image, so I need to perform tests on Docker as well. For those Docker tests, the Ubuntu-trusty64-docker Virtualbox image of William Yeh has shown the best performance results.

If I need to test the behavior of the application on clustered Docker hosts, CoreOS and Kubernetes are the only cluster alternatives I have experience with, and I have not done more than a small installation POC in case of Kubernetes.  However, considering the low CoreOS performance, I will need to investigate its alternatives, I guess: e.g. Docker Swarm, Kubernetes, or others (ping me, if you have suggestions).

My Path towards Continuous Integration of my Java Application

I am planning to do the following:

  • I will continue to develop my java application on native Windows and I will continue to push the code to Github often.
  • I have linked Github with TravisCI, so the code is automatically tested in the cloud. This way, I do not need to wait for test results. The test results are sent via email and I can react soon for any newly introduced bugs.
  • I will link TravisCI with Docker Hub, so a Docker image will be created automatically.
  • Locally, I will use the Docker on Ubuntu, if I need to troubleshoot the Docker build process, or I need to manually test, whether the Docker image really works.
  • If it comes to productive deployment including Docker host clustering, I had made good experience with CoreOS, even though clustering behind a HTTP proxy is a challenge; see here. However, the performance is sub-optimal, and I need to evaluate, which alternatives I have (e.g. test new CoreOS versions? Docker Swarm? Google Kubernetes?).

Watch out for my next blog post(s)…

;-))

3 comments

Comments

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.