Wednesday, June 25, 2014

PostgreSQL Performance on Docker

While preparing my presentation on Postgres and Docker to the PDX Postgres User Group for the June meeting, I thought it would be cool to run Postgres on Docker through some rough benchmarks. What I ended up finding was fairly surprising.

The Setup

For this test I used Digital Ocean's smallest droplet 512M RAM/1 CPU/20G SSD Disks.
They have an awesome feature where you can chose the type of application you want to run on the droplet and they'll pre-configure it for you. They even have one for Docker, so that's the one I chose.

Here's a list of all of the versions of the software I used.

Host OS: Ubuntu 14.04 LTS
Kernel: 3.11.0-12-generic
Docker version 1.0.1, build 990021a
PostgreSQL Version: 9.3.4

I used pgtune to generate my configuration, i used these same values across all tests:
maintenance_work_mem = 30MB 
checkpoint_completion_target = 0.7
effective_cache_size = 352MB 
work_mem = 2304kB 
wal_buffers = 4MB 
checkpoint_segments = 8 
shared_buffers = 120MB 
max_connections = 200 

Dockerfiles can be found here

The Tests

I used Greg Smith's pgbench-tools to run my tests with the following configuration
MAX_WORKERS=""
SCRIPT="tpc-b.sql"
SCALES="1 10 100"
SETCLIENTS="1 2 4 8 16"
SETTIMES=3
RUNTIME=60
TOTTRANS=""
SETRATES=""

My plan was to test how much overhead things such as Docker's virtual networking and virtual I/O incurred, and generally how much slower Postgres would be on Docker. So the test that I thought would be representative were:

  • Baseline Postgres on the system w/o Docker
  • Postgres inside Docker with virtual networking and virtual I/O
  • Postgres inside Docker with virtual networking but no virtual I/O
  • Postgres inside Docker with no virtual networking or I/O
After the results of those tests came in I ran another group of tests
  • Baseline Postgres with fsync=on
  • Baseline Postgres with fsync=off
  • Docker Postgres with fsync=on
  • Docker Postgres with fsync=off
After all of those tests were run it occurred to me that the base image I was using for docker was CentOS and my "baseline Postgres" was ubuntu. So i added one final test:
  • Docker Postgres w/Ubuntu base image
I dropped caches, restarted docker and or postgres between each run to ensure that I wasn't dealing with different caches.

EDIT: 2014-07-09

This original post was targeted at DBAs and folks who hadn't heard much of Docker, so I glossed over some of the specifics above. So I'd like to attempt to rectify that.

When I say "With no virtual networking" what I mean is Host networking --net==host
When I say "With no Virtual I/O above" what I mean is that I used a volume 
--volume=/path/to/postgres

I didn't add my volume to my Docker image so that I could use the exact same image for all tests.

With regards to the validity of the test on a VM.

I ran the full suite of tests 20 to 30 times (because I didn't believe the results).
It's certainly possible that runs I managed to hit Digital Ocean at Just.The.Right.Time(tm) each time, but i doubt it.

Regardless, I don't claim that these results are perfect. I've posted my methodology and given access to the source. So please reproduce and dispute. Honestly if you found that I did something significantly wrong in these benchmarks that would make them "fit" into more sensible expectations I'd be very grateful.

The Results

The results for the first round of tests were pretty surprising.

What wasn't surprising is that with the virtual I/O and networking Docker performed predictably slower than the baseline.

However, Docker with host networking and no virtual I/O was by far faster. This was very surprising. And I still don't trust these results, despite repeated runs confirming them.



I presented these results to the PDXPUG and received some good feedback on things to try to help validate the results. 

A prevailing theory was that Docker may be messing with/caching/breaking fsync so I ran the test with fsync on and fsync off in Postgres - here are the results:


So dockerized postgres with fsync ON is faster than vanilla postgres with fsync off...
That seems really wrong, however Digital Ocean gives SSDs to their droplets which may mean that the I/O really isn't much of a factor. That would mean that the difference is elsewhere.

Another theory is that Postgres on CentOS is simply faster than Postgres on Ubuntu despite the fact that they're sharing the same Kernel, it's possible that the CentOS C library has some optimizations that the Ubuntu one does not. I ran a test on Dockerized Postgres on Ubuntu vs Vanilla Postgres on the host machine (ubuntu as well). Here's the output from that run:



Dockerized Ubuntu is pretty much on par with Dockerized CentOS. So that's good at least.

Further thoughts and analysis

First, let me state that I wouldn't be surprised at all if I missed some major fact in these tests or if I was somehow tricked by the results. The results don't feel right at all. I've run the tests over and over again and have gotten consistent results, but that doesn't mean that there isn't some sort of environmental explanation. Perhaps the Digital Ocean droplet itself is biased towards Docker due to something underneath the covers. 

But, let's suppose for a minute that these results are legit and reproducible elsewhere how could we explain it? My theory is that since Docker is a wrapper around LXC that it's possible that LCX has a more efficient execution path within the Kernel and is allowing the Dockerized Postgres to use more resources with less interruptions. 

I have some support for that, during my tests I was also gathering low level system statistics and found that for Dockerized Postgres there are significantly fewer context switches than with normal Postgres.

Dockerized Postgres


Normal Postgres


The Conclusion

It's definitely a surprising result, and one that goes against my expectations. Obviously further research is needed to confirm the result.

The thought that LXC somehow has an optimized path within the kernel is very satisfying, but at the same time there are too many variables to know for sure.

I look forward to suggestions for improving my tests and any conflicting evidence from your own tests. 

11 comments:

  1. Hi,

    very interesting results. Btw. docker starting with 0.9 is not using LXC as default anymore but libcontainer.

    Mike

    ReplyDelete
  2. Interesting discussion of this article on HN at https://news.ycombinator.com/item?id=8010101

    ReplyDelete
  3. Thanks for running this!

    If you're running this on DIgital Ocean, though, you're already using Virtual IO whether you're in Docker or not. For a comparison test, you'd need to run on real hardware. Otherwise you're mostly testing the interaction of Docker with DO's virtual IO system.

    Also, I'm putting together a team to maintain the official Docker packages for Postgres. Interested in being on it?

    ReplyDelete
    Replies
    1. Yeah, I think that the way i phrased that lead to some confusion. What I meant was that I used a Docker volume instead of allowing docker to use it's internal filesystem handling.

      Despite what a few folks have stated. I don't think that benchmarks on VMs are completely worthless. Especially like in my case where 1) that's probably where I'd end up running it. and 2) I ran the benchmark multiple times and averaged out the results.

      The key to benchmarking on VMs: Benchmark early, often, and keep on benchmarking.

      I'd love to be on the Docker Postgres team, sounds awesome!

      Delete
  4. Posgres may have worked better with the system libraries/ glibc in docker ( may be the base image was better suited to the postgress than the host system ) or whatever than in the host system that triggered this performance benefit.

    ReplyDelete
    Replies
    1. That was a hypothesis that I tested. It's about 1/2 way down the post right before I start looking into Context Switches.

      Delete
  5. Are you running the benchmark client in the same container as the postgres-server?
    My colleague suggested me that if they were in the same container, they would be in the same cgroup and the kernel may have done some process scheduling optimization.

    ReplyDelete
    Replies
    1. I ran pgbench from outside of the container.

      Delete
  6. Bit of a mystery. Do you know if it was using loop-lvm (as opposed to say btrfs)?

    The reason I ask is that http://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/ says O_DIRECT doesn't work for loop-lvm. Postgresql does use O_DIRECT for the WAL at least - but putting the postgresql directory on a volume would presumably make that moot; maybe there's something else using O_DIRECT?

    (BTW, your blog software seems to lose the comment if you have to sign in to the auth provider...)

    ReplyDelete
    Replies
    1. I believe it was using devicemapper, although it may have been aufs. I definitely didn't do any tuning around the storage drivers since i was going for more of an out-of-the-box test.

      Weird about the comments, is just blogger, so i guess we have to blame google :/

      Delete

Web Statistics