The concept seems so simple at first. You write some code, make sure it passes tests locally and when integrated with the rest of the codebase, then deploy it to staging, make sure the tests pass in staging, and then deploy it to production (also testing the code is working in production too). Then repeat. Easy peasy, lemon squeezy, right?
Not so simple
To really do continuous delivery, at its core, you need...
- continuous integration and continuous deployment pipelines
- a full suite of tests that you trust – you trust that if the tests all pass in lower environments, then you are ready to push the code to production
- like minded people that want to focus more time on value delivery and less time on toil
Sounds easy, but anyone that's needed to set this up from scratch at your company knows it is far from easy. But it's often our most challenging accomplishments – the ones where the we had to overcome very difficult obstacles – that fill us with the most pride. I'm very excited for the day that we accomplish full continuous delivery at STORD, and we will all be beaming with pride on that day.
What we've done, and what we have left to do
We started building the second version of our platform in January 2020. We took the learnings from our flag-ship MVP application, that was a monolith, and started building out our next generation application using microservices. I bring this up because testing is a key element toward continuous delivery. Testing gets harder and harder the more responsibilities an application has, so microservices –done right– can help out a lot here because you can keep your services focused and therefore easier to cover their responsibilities with tests.
A complex system is made up of simple working parts, so at the end of the day our microservices need to come together to create our Cloud Supply Chain, which is a complex system. Therefore, we need suites and layers of testing in order to reach the level of confidence in our testing such that we can trust our deployments to go straight to production.
At STORD, we have a good set of unit and integration tests – for both our backend and front-end applications. These quick running tests run within our continuous integration tool (Circle CI).
The next layer of tests we have are Component tests. Component tests allow us to test a service in isolation, and mock out any external calls at the network layer using WireMock. We are just getting our Component tests build into our continuous deployment pipelines now.
After that, we need have end-to-end tests, which can cross the boundaries of many services. These end-to-end tests can be driven through API calls or through the UI. We still have work to do in order to get our end-to-end tests running and automatically incorporated into our deployment pipelines.
Since STORD uses Docker containers and Kubernetes (GKE on Google Cloud Platform), we also have probe tests ...startup probes to test the configuration is correct, and readiness and liveliness probes to ensure the container is ready to serve and remains ready to serve within its cluster.
With all these test suites working in concert, we can build the same level of confidence, if not more so, as when a person does a manual set of deployments and tests themselves throughout multiple environments.
We also have Harness at STORD to help us manage the various workflows and verification steps required to move containers between environments, validate deployments, and make deployments 'live'.
Figure 1 below is a picture of STORD's vision to achieve full continuous delivery. You'll notice that the code is going through the same steps it would as if an human is moving the code from their desktop, up through the various environments, and out to production. Just replace what you do today with automation – same environments, same deployments, and same testing. We have the technology, I assure you that this can be done!
Separating a Deployment from a Release
Separating a deployment from a release is a key milestone in achieving Continuous Delivery. We do this using our Feature Flags tool (LaunchDarkly)
Feature flags allow us to practice trunk-based development, and avoid branch merging hell. But it also allows us to do deployments during the day, when the team is around to support the deployment if something goes wrong, instead of late nights and weekends. And since deployments are separate from releases, just because the deployment had issues, doesn't mean the customers are experiencing any issues on the currently release that they are using.
Getting very close!
We are getting very close to implementing our full vision for continuous delivery. Some of the key things we have left to do are:
- Separate our monolith repo into services – we tried to run a monorepo with our microservices, and found it too challenging, so we are separating them out now.
- Get our Component tests running within the deployment pipeline
- Create end-to-end tests (both API driven and UI driven), and get them into the pipeline as well.
It won't be easy, as we also have critical features to build that our customer's need, but we will get there! And we will be proud of this major accomplishment!
Ready to help us build a Cloud Supply Chain?
We have a great team and we are making great strides. We are also experiencing triple digit, year-over-year growth due the acceleration of ecommerce by COVID. Having a Cloud Supply Chain gives companies key business capabilities – speed, flexibility, and cost savings. Please checkout our careers page to see how you can help us!