The PostgreSQL Benchmark on 15GB servers on Amazon RDS and Rackspace Cloud with SSD disks

This is in continuation with the last benchmark we performed in one of our previous posts. When I submitted the last article on the PostgreSQL reddit, a couple of users wanted to know how PostgreSQL performed on the Amazon RDS for PostgreSQL that was launched recently. We were also curious so we did this benchmark. Rackspace also overhauled their cloud infrastructure and launched a new breed of performance servers with SSD disks recently. We wanted to test these too.

The Servers

Amazon RDS

As in previous benchmark, in this benchmark we created 2 servers, one DB server and other benchmark server.

Database Server

This was a db.m1.xlarge instance with 15 GB of RAM. The disk was a 400GB of EBS volume with 4000 provisioned IOPS. (Amazon required the storage to IOPS ratio in the range 1:3 to 1:10) We selected 9.3.1 version of the PostgreSQL, which is the only option as of now. We didn’t use the multi A-Z option that Amazon recommends for production deployments. This was done because all other benchmarks didn’t use replication.

Benchmark Server

This was used to run pgbench. This was same as used in the previous benchmark. An m1.xlarge instance with 15 GB RAM and default 8GB EBS volume without provisioned IOPS. We used the Amazon AMI and installed pgbench using yum.

Rackspace

For Rackspace benchmark we created 2 servers with their Performance 2 Flavor Class each with 15GB of RAM. These servers are equipped with SSD.

Database Server

This ran Ubuntu 12.04. The version of PostgreSQL was 9.1 and was installed using apt-get. Here we could have used PostgreSQL 9.3. But we wanted to keep these numbers relevant to the previous benchmark we did which also used PostgreSQL 9.1 from the standard Ubuntu repository.

Benchmark Server

We used Cent OS 6.4 for this and pgbench was installed using yum.

Monthly Pricing For The Database Servers

Amazon RDS

Around $945 per month

Rackspace

Around $500 per month

The Benchmark

First we ran the pgbench command with a scale of 100, which generates 10 million rows in the accounts table. The size of database thus created was around 1.5GB.

Rackspace

For Rackspace, after setting up the server we changed the postgresql.conf for max connections upto 400. Also we modified /etc/sysctl.conf and set the kernel.shmmax value to around 7GB.

We ran the benchmark 2 times on each of the servers. In the first run (henceforth referred as ‘without optimization’) we used the default values for ‘shared_buffers’ and ‘effective_cache_size’ configuration values. In the second run (henceforth referred as ‘with optimization’) we changed ‘shared_buffers’ to 4GB and ‘effective_cache_size’ to 8GB.

Amazon RDS

For Amazon RDS no changes were made to the configuration. We just used the configuration as provided by Amazon. So for RDS there is only one set of configurations.

The runs

For each run we did 4 sub tests with 96, 128, 256 and 392 clients in read-write (RW), read only (R), Write only (W) modes. Each of these tests were run for 10 minutes (600 seconds.)

The Results

Amazon RDS

Rackspace

Conclusion

As said in the last blog post, these are apples to oranges comparison. Then one would ask why do them? The main reason is about evaluating choices. Many of our clients want to deploy their applications in the cloud. Right now Amazon is the leader in the Cloud based deployments. However many new companies have entered in this market. Each cloud provider has very different architecture and hardware. Right now AWS has definitely one of the most flexible and advanced architectures than most of the cloud providers who are providing solutions based on OpenStack. Given this we need to determine which cloud will give better results for the application in question. There are many parameters for selection of a right cloud for your application. Some of them are cost, ability to scale, stability and uptime.

In a web application, if one uses shared nothing architecture, it is quite easy to scale at application server level. But in this the state gets pushed to the database and database becomes the main resource that needs to scale vertically. Of course one could use sharding but that requires quite a few changes to the data access layer of the application.

This is why I wanted to see how long I could go on with these cloud servers with a reasonably powerful configuration before we start implementing sharding.

We have presented the results above. One thing I would like to note that for RDS now Amazon supports up to 30,000 provisioned IOPS volumes. But that pushes the cost of the instance above $4000 per month.

I hope these tests help you in some way while making the choice for your next cloud deployment. Thanks for reading.