Ghost is a professional-grade, open source publishing platform. Volusion uses Ghost to host our blog, where we serve up tasty ecommerce advice. Ghost is a fantastic solution for creating and publishing content but not for serving it.
Figure 1. Naive Ghost setup on GCP.
To test Ghost for our purposes, we set it up in a configuration on Google Cloud Platform (GCP) virtual machine (VM). **Figure 1** illustrates our configuration. Briefly, we created a separate GCP Project for the blog. Inside this project we created a Google Load Balancer (GLB) to terminate SSL traffic. After SSL termination, the GLB then sends the plain HTTP request to an nginx backend which routed the request to Ghost. At this point, we had a working blog; it could securely serve requests on its custom domain. Some observations about this setup:
Traffic entering projects in GCP comes from the public internet. So, if you're routing traffic in the same organization, but across different projects, you're leaving Google's super-fast infrastructure, bouncing off the slow public internet, and then going into your other project. This introduces unwanted latency because you end up doing an extra public DNS lookup, your traffic ends up on the public internet unnecessarily, and you have a redundant SSL-termination.
Installing nginx, Ghost, and MysQL on the same VM for testing is fine. But for production you don't want your web server, CMS and database all fighting for resources on the same machine. This introduces risk of resource contention and increases fragility. This also creates a more complicated single point of failure.
By running MySQL locally we also are responsible for data integrity: we need to backup and test our databases. If we chose Google Cloud SQL the operations (updates, backups, etc.) are handled by Google instead.
We should configure nginx to cache results. This way we never need to hit Ghost or the database.
Our amazing Marketing department loves Ghost and wanted to see it in production. So, how would we design a simple, sufficiently fast and scalable solution to meet their needs? Here's our approach: understand the current system design (described above); benchmark the current system; analyze alternative designs; implement and benchmark our options.
Figure 2. Naive blog implementation alongside other serving infrastructure.
Our blog was installed using the naive configuration and started serving traffic. The end-to-end network request through the blog looked similar to **Figure 2**.
We used Apache Bench (AB) to help determine the load characteristics of our naive solution. Briefly, AB is a tool for benchmarking HTTP servers. It sends a flurry of requests against your infrastructure and calculates timing metrics. Using AB is easy: you specify the URL to test, the number of total requests, and the number of concurrent requests. AB will make the requests and then show you summary of the test. It's very easy and informative. Be careful though, as Max Gutman wrote, AB is unsympathetic to your infrastructure. Here are the AB results for our naive solution:
Under moderate load (400 requests, 100 concurrent), the naive solution is only able to process about 1.87 requests per second; and, half of the requests took over a minute to serve:
This solution clearly has its weaknesses. Now, let's consider how we clean this up and create a more robust solution.
First, we want to eliminate the double-hop to the public internet. So we recommend moving the Ghost VM into the same project as the incoming traffic. Next we recommend reducing the load on the Ghost VM by sending the database responsibilities to Google Cloud SQL. This also frees us up from having to backup the database. Next, our VM will serve nginx and ghost from containers. Containerizing our solution allows us to rapidly move deployment to Kubernetes or other autoscaling infrastructure as our needs grow. Our proposed solutions looks like that in Figure 3:
Figure 3. Proposed implementation.
We implemented this solution and benchmarked it with ab again. Our performance characteristics were much stronger:
As you can see, this new design can serve 108 requests per second, which is about a 5700% improvement over the naive design. As our blog load is currently well under 388,800 requests per hour we decided to stop and move onto other issues.
We designed this system to be easy to extend in case our blog significantly increases in popularity. Since ghost and nginx run from containers we could easily move that to Kubernetes and scale up the nginx caching layer elastically with demand.
At Volusion we try to keep things simple. Right now this solution is simple and meets the needs of the business. So, it's time to move onto solving other problems in the company.