This is in continuation with the last benchmark we performed in one of our previous posts. When I submitted the last article on the PostgreSQL reddit, a couple of users wanted to know how PostgreSQL performed on the Amazon RDS for PostgreSQL that was launched recently. We were also curious so we did this benchmark. Rackspace also overhauled their cloud infrastructure and launched a new breed of performance servers with SSD disks recently. We wanted to test these too.

The Servers

Amazon RDS

As in previous benchmark, in this benchmark we created 2 servers, one DB server and other benchmark server.

Database Server

This was a db.m1.xlarge instance with 15 GB of RAM. The disk was a 400GB of EBS volume with 4000 provisioned IOPS. (Amazon required the storage to IOPS ratio in the range 1:3 to 1:10) We selected 9.3.1 version of the PostgreSQL, which is the only option as of now. We didn’t use the multi A-Z option that Amazon recommends for production deployments. This was done because all other benchmarks didn’t use replication.

Benchmark Server

This was used to run pgbench. This was same as used in the previous benchmark. An m1.xlarge instance with 15 GB RAM and default 8GB EBS volume without provisioned IOPS. We used the Amazon AMI and installed pgbench using yum.

Rackspace

For Rackspace benchmark we created 2 servers with their Performance 2 Flavor Class each with 15GB of RAM. These servers are equipped with SSD.

Database Server

This ran Ubuntu 12.04. The version of PostgreSQL was 9.1 and was installed using apt-get. Here we could have used PostgreSQL 9.3. But we wanted to keep these numbers relevant to the previous benchmark we did which also used PostgreSQL 9.1 from the standard Ubuntu repository.

Benchmark Server

We used Cent OS 6.4 for this and pgbench was installed using yum.

Monthly Pricing For The Database Servers

Amazon RDS

Around $945 per month

Rackspace

Around $500 per month

The Benchmark

First we ran the pgbench command with a scale of 100, which generates 10 million rows in the accounts table. The size of database thus created was around 1.5GB.

Rackspace

For Rackspace, after setting up the server we changed the postgresql.conf for max connections upto 400. Also we modified /etc/sysctl.conf and set the kernel.shmmax value to around 7GB.

We ran the benchmark 2 times on each of the servers. In the first run (henceforth referred as ‘without optimization’) we used the default values for ‘shared_buffers’ and ‘effective_cache_size’ configuration values. In the second run (henceforth referred as ‘with optimization’) we changed ‘shared_buffers’ to 4GB and ‘effective_cache_size’ to 8GB.

Amazon RDS

For Amazon RDS no changes were made to the configuration. We just used the configuration as provided by Amazon. So for RDS there is only one set of configurations.

The runs

For each run we did 4 sub tests with 96, 128, 256 and 392 clients in read-write (RW), read only (R), Write only (W) modes. Each of these tests were run for 10 minutes (600 seconds.)

The Results

Amazon RDS
Clients RW R W
96 1815.773151 13085.82888 2487.41379
128 1847.452656 14497.00823 2451.13935
256 1661.212405 17870.95145 2392.261281
392 1552.277529 18610.69465 2374.507076

amazon-rds

Rackspace
Without Optimization
Clients RW R W
96 2587.959824 19091.42047 3633.294142
128 2544.073905 18773.24138 3500.305015
256 2447.266516 18152.21931 3436.687767
392 2217.713073 17079.95152 3424.557662

rackspace-without-opt

With Optimization
Clients RW R W
96 2579.977816 20218.54149 3543.238222
128 2465.996281 20324.56671 3567.415075
256 2449.03099 19839.5473 3643.524381
392 2326.303463 19164.8051 3685.999467

rackspace-with-opt

Conclusion

As said in the last blog post, these are apples to oranges comparison. Then one would ask why do them? The main reason is about evaluating choices. Many of our clients want to deploy their applications in the cloud. Right now Amazon is the leader in the Cloud based deployments. However many new companies have entered in this market. Each cloud provider has very different architecture and hardware. Right now AWS has definitely one of the most flexible and advanced architectures than most of the cloud providers who are providing solutions based on OpenStack. Given this we need to determine which cloud will give better results for the application in question. There are many parameters for selection of a right cloud for your application. Some of them are cost, ability to scale, stability and uptime.

In a web application, if one uses shared nothing architecture, it is quite easy to scale at application server level. But in this the state gets pushed to the database and database becomes the main resource that needs to scale vertically. Of course one could use sharding but that requires quite a few changes to the data access layer of the application.

This is why I wanted to see how long I could go on with these cloud servers with a reasonably powerful configuration before we start implementing sharding.

We have presented the results above. One thing I would like to note that for RDS now Amazon supports up to 30,000 provisioned IOPS volumes. But that pushes the cost of the instance above $4000 per month.

I hope these tests help you in some way while making the choice for your next cloud deployment. Thanks for reading.

zp8497586rq