Clojure, Docker, Dokku, Digital Ocean and Shell Script: The New Blog, Part 1

Clojure, Docker, Dokku, Digital Ocean and Shell Script: The New Blog, Part 1

I was in a funk. I had been working on products and consulting for a length of time long enough to escape easy recollection. In that time, I had made little time for one of the primary joys of coding: small, exploratory, harmless projects. Experiments whose only customer is myself, and whose success or failure impacts nothing critical in my daily life. Play, in other words.

I had, months prior, completed an Instrumental project that replaced some Ruby components with Scala. In the course of that effort, I had the chance to use the excellent Netty, and thoroughly enjoyed the abstractions it provided over IO consumption. Wanting an excuse to play with Netty in a different environment, I decided that rewriting my blog in Clojure+Netty would be a simple enough excercise that let me learn Lisp for a trivial project. And, because reasons, I should be able to deploy my blog via git pushes to a Dokku/Docker managed container on an entirely new server. And render Github flavored Markdown. And store blog posts in a MySQL db. And also manage the blog content with existing tools like Sublime Text and Marked. And also automatically upload referenced assets in my posts to S3.

Well, it seemed trivial at the beginning.

The Host

I decided that the first and most effective step I could make was migrate my existing blog (a Wordpress site) from Linode to Digital Ocean and the new Dokku setup; Dokku's simple install of

 wget -qO- https://raw.github.com/progrium/dokku/master/bootstrap.sh | sudo bash

was enough to get the software running, though I found that configuring it beyond its very simple defaults required reading the source regularly. Since I was also porting over a number of other domains that I also wished to host via Dokku, I had to make the following changes:

  • Ensure every domain hosted via dokku was 'bare', eg no "www". This meant creating the Dokku remotes looked like: git remote add deploy git@yeti-factory.org:the-domain-name.com.
  • Change the Dokku Nginx config (/etc/nginx/sites-available/default) to hold a line like the following:

    server {
      # Dokku config above here
      location / {
        if ($host ~* www\.(.*)) {
          set $host_without_www $1;
          rewrite ^(.*)$ http://$host_without_www$1 permanent;
        }
      }
    }
    

    The above would catch any request for a domain with wwww as a prefix and strip it from the requested domain, then send the request back through the Nginx URL resolution phase (and be caught by Dokku's own generated domain configs)

  • Add a custom "static" buildpack to the Dokku progrium/buildstep container, the exact steps of which I omit here because it was late and the process was annoyingly manual. Suffice to say, it involved shelling into the container, updating buildpack indices, committing container changes, and then redeploying an app.
  • Spin up a MySQL DB running outside of Docker that would service all the running containers. I didn't find any super easy ways of handling a container that had specific contents ( MySQL's data dir, let's say ) that should persist across container changes.

Once I had finished these steps, I was in a place to migrate entirely from the expensive Linode ( $40/mo )to the cost efficient Digital Ocean ( $5/mo ). Next up, I just had to learn Lisp and write a bunch of code!

Getting Therious

Already having some experience with Netty, I was able to bootstrap my knowledge of how to build a simple HTTP server by browsing the excellent samples directory. ( Seriously, have I said enough nice things about Netty yet? It's nice! They really do an awesome job w/ that project. )

The main work then was porting the Java basics expressed there to Clojure, something I had further help with thanks to the clojure-netty project I had found. With these two guides in place, I could begin with some simple prototypes and extend my learning of Clojure outward.

Within a few hours, I had my own version of clojure-netty running, but outputting my own and entirely unique "Hello World". During the course of this excercise, a few half formed problems began to occur: how would I handle static assets? Did I plan on supporting conditional GETs? What was the best way to represent a URL-to-app logic structure in Clojure? Was there a template language de riguer that I should use in Clojure?

It's Too Easy To Make "Ring" Puns

It struck me that I wanted something very similar to Ruby's Rack; an easy way to compartmentalize different stages of the request handling app logic. This led me directly to Ring, a project that does exactly that. Unfortunately for me, it's default behavior is to work with Jetty, an entirely nice web server framework that was one letter off from what I needed. Were I able to write my own Ring adapter for Netty ( later of course, I discovered that someone had already done exactly this, but one can only use so many third party libraries ), I'd get my nice abstraction over the request processing pipeline.

One hacked up Jetty adapter later, I had my adapter. At this point, it was simple to wire in some static file serving Ring middleware, and I was ready to start serving actual files! Off the filesystem!

RTFM

Absurdly pleased with myself, I decided to run a few weighttp benchmarks for the inevitable My Clojure Microframework That Is Faster Than Nginx When Serving Static Strings blog post. Oddly enough, I noticed that as I tuned up the concurrency level, I'd start to see timeouts. The same occurred when visiting the page in Chrome; occasionally I'd see odd request timeouts as the browser hung trying to fetch content.

This definitely smelled like a race condition, but I was stymied; I was attempting nothing clever when it came to Netty's parallelism. What on earth could it be?

As it turned out, after a day of googling around in idle time, I accidentally came across the cause: misuse of def. Until this point in my Clojure tutelage, I had labored under the misapprehension that def was merely a poor man's let; to explain for those unacquainted:

(def a_name a_value) # This binds a_name to a_value GLOBALLY
(let [a_name a_value]) # This binds a_name to a_value LOCALLY

The race condition had been inside my brain THE WHOLE TIME.

AGH

Originally There Wasn't A Part Two But I Wrote A Lot Already

A template language's attempt to be "clever", bash as admin UI, and running in "production": what I'll talk about in the follow up to this post.

comments powered by Disqus