Continuous Delivery Vs. The Human Eye

Continuous delivery, deployment, and integration tools are popping up in more places every day. Developers at Instagram recently wrote a blog post about the internal tool they built to allow the team to automate backend code deployment up to 50 times every day, “whenever engineers commit changes to master…with no human involvement in most cases.”

Developers have a thing for automation. Even if we are at small shops or work independently, we automate common tasks or use automation tools where we can. The goal of stable and automated task management is a common theme that runs through the industry.

If you consider that everyone at Instagram has a single, common goal, which is to improve Instagram and increase its user base, it is clear that near-constant work to the site and service is the heart of each workday at at the company.

Since almost everyone is working on a few common platforms, it makes sense to spend tremendous effort to make the distance between the developer and the user as short as possible.

Automated deployment allows Instagram engineers to identify and fix bad commits immediately, and that means the probability of rolling out faulty code is lower. The engineering team can increase the frequency of deployments through the same continuous process that also ensures the code’s quality.

The Human Eye as Judge

Here at Zivtech, we built Probo.CI for the similar goal of automating builds, but the philosophy is a bit different. Rather than rely on machines to verify our builds, we look at human interaction as the best method of approving changes. Probo makes the assumption that humans need to see the software working to know that it is correct. That’s why every pull request spins up a sharable link in a separate live environment.

Anyone (technical or nontechnical staff like project managers, CEOs, or marketing people) with the link can check out proposed changes with his or her own two eyes. Not only does Probo reduce developer time, but it also reduces overall approval time throughout the deployment process.

Continuous Delivery at the Right Scale

In continuous delivery, it’s not one size fits all. Just as Instagram’s solution is scaled for a large scale operation running a single product, Probo is scaled for smaller shops that may be working on different kinds of client projects at the same time. For companies that do client services, many different projects, and many different kinds of projects, Probo makes more sense.

Probo makes the assumption that each project, while maybe similar, has unique deployment steps and testing requirements. In this case a general purpose automated solution really doesn’t work. Humans need to interact with the system because it is too complex or too fast-changing for pure automation. For small and medium sized shops, Probo is a solution that doesn’t require hyper-specific focus.

How Many Servers?

Instagram’s continuous delivery solution leverages machines to detect problems, but that solution requires a huge development effort not on the product itself, but on the actual testing and automation tools. That works well at companies that have the ability to focus on a single product but is hard, if not impossible, to do in a client sevices company, particularly one that uses many different technologies for each project. But aside from the purity of focus, Instagram also has more liberty to push breaking changes to production because the service is run on a large and very redundant stack (hundreds of servers). If you run 100 servers, you can afford to roll out a breaking change to one to see what happens. Pushing a breaking change to a production server might only cause issues for 1% of users and it will be detected fast and fixed fast.

But most of us don’t have hundreds of servers and most of our applications don’t need more than one or two. For this scale of application, rolling a broken change out might mean breaking things for 50% or 100% of users. That doesn’t fly for smaller organizations like Zivtech, and it probably doesn’t fly for yours.

The goals are the same for both Probo and for Instagram’s continuous delivery tool: speed up deployments, reduce errors, and prevent bottlenecks. For small and medium sized shops, Probo is a tool you can use right away that probably meets your organization’s needs more readily than something purely automated. We all want to make things more efficient and more organized, but we have to acknowledge that sometimes the machine just requires too much time to perfect. In reality, a human brain is the most efficient tool for testing software across a range of projects and systems.