Updated: Our Path to Continuous Delivery

The concept seems so simple at first. You write some code, make sure it passes tests locally and when integrated with the rest of the codebase, then deploy it to staging, make sure the tests pass in staging, and then deploy it to production (also testing the code is working in production too).  Then repeat.  Easy peasy, lemon squeezy, right?

Not so simple

To really do continuous delivery, at its core, you need...

• continuous integration and continuous deployment pipelines
• a full suite of tests that you trust – you trust that if the tests all pass in lower environments, then you are ready to push the code to production
• like minded people that want to focus more time on value delivery and less time on toil

Sounds easy, but anyone that's needed to set this up from scratch at your company knows it is far from easy. But it's often our most challenging accomplishments – the ones where the we had to overcome very difficult obstacles – that fill us with the most pride. I'm excited to say that we accomplish full continuous delivery at STORD, and I was beaming with pride when we did it!

What we've done, and what we have left to do

We started building the second version of our platform in January 2020. We took the learnings from our flag-ship MVP application, that was a monolith, and started building out our next generation application using microservices. I bring this up because testing is a key element toward continuous delivery. Testing gets harder and harder the more responsibilities an application has, so microservices –done right– can help out a lot here because you can keep your services focused and therefore easier to cover their responsibilities with tests.

A complex system is made up of simple working parts, so at the end of the day our microservices need to come together to create our Cloud Supply Chain, which is a complex system. Therefore, we need suites and layers of testing in order to reach the level of confidence in our testing such that we can trust our deployments to go straight to production.

At STORD, we have a good set of unit and integration tests – for both our backend and front-end applications. These quick running tests run within our continuous integration tool (Circle CI).

The next layer of tests we have are Controller tests. Controller tests allow us to test a service in isolation, and mock out any external calls.  The mocking can be done with Elixir, and so we can run these controller tests in a similar way that we do Integration tests.

After that, we need have end-to-end tests, which can cross the boundaries of many services.  These end-to-end tests can be driven through API calls or through the UI. We use Cypress for creating and running our end-to-end tests.

Since STORD uses Docker containers and Kubernetes (GKE on Google Cloud Platform), we also have probe tests ...startup probes to test the configuration is correct, and readiness and liveliness probes to ensure the container is ready to serve and remains ready to serve within its cluster.

With all these test suites working in concert, we can build the same level of confidence, if not more so, as when a person does a manual set of deployments and tests themselves throughout multiple environments.

The next layer of testing that we want to add is Contract tests. A way to know that your consumers of your API are not impacted by any changes to the service backing the API. If so, then maybe you need to create a new version or refactor your changes so that your changes can be backwards compatible.

Figure 1 below is a picture of STORD's vision to achieve full continuous delivery.  You'll notice that the code is going through the same steps it would as if an human is moving the code from their desktop, up through the various environments, and out to production. Just replace what you do today with automation – same environments, same deployments, and same testing. We have the technology, I assure you that this can be done!

Separating a Deployment from a Release

Separating a deployment from a release is a key milestone in achieving Continuous Delivery. We do this using our Feature Flags tool (LaunchDarkly)

Feature flags allow us to practice trunk-based development, and avoid branch merging hell.  But it also allows us to do deployments during the day, when the team is around to support the deployment if something goes wrong, instead of late nights and weekends. And since deployments are separate from releases, just because the deployment had issues, doesn't mean the customers are experiencing any issues on the currently release that they are using.

We did it!

It took us a few quarters to get there, but we finally implemented our full vision for continuous delivery.  

Ready to help us build a Cloud Supply Chain?

We have a great team and we are making great strides. We are also experiencing triple digit, year-over-year growth due the acceleration of ecommerce by COVID. Having a Cloud Supply Chain gives companies  key business capabilities – speed, flexibility, and cost savings. Please checkout our careers page to see how you can help us!