For the unfamiliar, Namely is a SaaS company offering an all-in-one HR, payroll and benefits platform. Every month they move more than 1 billion dollars in payroll—which means safe and seamless software delivery is top priority.
For the last few years, Namely has been moving to microservices. The team began adding services to their architecture—Kubernetes, Jenkins, AWS, Spinnaker, to name a few. But soon enough, they began to struggle as they found themselves trying to navigate over three dozen services. To make matters worse, they didn't have the proper tooling or any standards whatsoever for deploying applications.
To pull themselves out of the mud, the team decided to use Jenkins for "continuous deployment." They copied a kube-deploy.sh file to every single project and made the Jenkins file run it at the end of each build.
It didn't take long for them to realize the monstrosity they had created. So, they turned their attention to Spinnaker.
Their first move was to write a design doc to set the rules and requirements for what they wanted to accomplish with the CD platform. But what they wanted didn't match with what Spinnaker had, such as:
- Using Spinnaker Pipelines for all deployments without configuring them via Spinnaker's UI.
- Updating Spinnaker pipelines automatically when changes are pushed to GitHub. (This isn't supported in Spinnaker.)
- Recording releases for all environments for every project. (This isn't supported either.)
Subsequently, the team's next move was focused on building tools around Spinnaker to enable these requirements. This happened in three parts:
1. Pipeline configuration
To avoid error-prone engineers from managing pipelines using the Spinnaker UI, the team used the platform's nifty "Edit as JSON" feature to configure pipelines for projects by hand.
They then used that JSON to create Go structs for what became the k8s-pipeliner tool. This tool allowed them to easily manipulate pipelines, track who made which changes, and specify "how" to deploy (read: imperative delivery) rather than "what."
As Ross proudly explained during his presentation,
"By building k8s-pipeliner and creating concrete types to represent a Spinnaker pipeline, we set ourselves up for even bigger successes with little work."
2. Automatic pipeline updates
As much as Ross loves Spinnaker, he can't deny that the UI and UX for Spinnaker's pipeline management is, well, clunky.
For what the team needed—update an application's pipeline after a GitHub merge—they needed a new tool. So they built Estuary: a "simple HTTP server that received GitHub webhooks, builds a pipeline config, and then updates the Spinnaker Pipeline via the Gate API."
With Estuary, the JSONs generated by their k8s-pipeliner tool get imported automatically, rather than the team pasting them in by hand. Problem solved.
3. Release logs
For the final requirement, the team wanted a structured entry of what was deployed and when, along with some useful metrics and metadata. Fortunately, Spinnaker has a module called Echo that sends webhook requests on task events.
So the team configured Echo to send webhooks to their "Release Management Service." Then, to store the pipeline payload they used the very sophisticated platform: Google Sheets. (Yes, really.)
After the rude discovery that each payload was a tremendous 870kb of JSON, they decided to send the payload to webhook.site instead. As a result, whenever a pipeline was set off, Echo would fill up a new page on the site with all the data they could possibly need.
After retelling his team's adventures of struggling with Spinnaker and juggling JSONs, Ross concluded that open source projects are simply puppets—meaning they're close to useless on their own. Consequently, engineers must be puppeteers and use tooling as a way to bring life to open source projects (like Spinnaker).
As a bottom line and to close his presentation, Ross flashed his last slide reading the main takeaway for his audience:
"Build all the tools. Future you will love you more for it."
If you'd like to watch the good-humored Robert Ross (@bobbytales) giving his presentation on stage, you can find the slide deck and video on the Spinnaker Speakers page.
For those who want something more, you can already make big plans for Spinnaker Summit 2019. Tickets are up and ready to go (with a generous early bird discount, we might add). There will be many of the same great speakers as last year plus new ones you won't want to miss. Save your seat here.