When I used to live further away from the supermarket, I'd drive there less often and come back home with my car full of food. Now that I live next to a supermarket, I usually go there daily and only buy a handful of food for the rest of the day.
As a general rule, the more pain it is to go to the supermarket, the more items you'll bring home, and the less often you'll go there.
In manufacturing, this "pain" to get items from one place to another is called the transaction cost. As in the supermarket example, manufacturers increase batch sizes whenever transaction costs are high. As transaction costs diminish, so do batch sizes.
The reason manufacturers change batch sizes according to transaction costs is to save money. In the same way you wouldn't go to the supermarket to buy a single egg, manufacturers wouldn't rent a truck to transport a single laptop at a time. The more items manufacturers batch up, the smaller the average transaction cost per item.
As software engineers, we intuitively adjust batch sizes the same way. If it's risky and difficult to deploy software to production, we usually allow changes to accumulate before we run deploy.sh. Conversely, if the transaction cost for going from "merged" to "deployed" is low, we'll deploy more often.
In other words, we increase our batch size whenever it's difficult to move work from one stage (merged) to another (deployed).
That's how Git Flow was born a few years ago. At that time, integrating code was painful, and people wanted to feel less pain. However, instead of making integrating code easier, GitFlow incentivized people to integrate more changes less frequently.
In other words, GitFlow attempted to diminish overall costs by reducing the number of transactions instead of lowering the cost of each individual transaction. That approach is problematic, and now I'll explain why.
The problems with Git Flow
The worst problem with Git Flow is that it makes companies move more and more slowly over time until they come to a grinding halt.
That halt happens because Git Flow leads you to integrate large batches of changes into main all at once. Therefore, it's more likely that there will be bugs and that it'll be painful to integrate code when you get to it.
Then, because integrating code is painful, you'll drift into integrating code less and less often, which leads to more and more bugs, which leads to integrating code less and less frequently. Do you know what that causes? More and more bugs.
In other words, large batch sizes lead to pain, which leads to large batch sizes.
One day, batches will be so large, and the pain will be so intense that everything will halt. On that day, the only way to get unstuck is to ask everyone to stop committing new changes so that you can integrate everything into main and start from a clean slate.
Tail latency amplification
Imagine you're trying to hit an endpoint, A, whose response depends on two other services, B and C. In that case, if B takes 10 seconds to respond, A will always take at least 10 seconds to respond, even if C responds in 1 millisecond.
In other words, a single slow service can slow down the entire endpoint that depends on it.
That phenomenon is known as "tail latency amplification" in software engineering.
The same principle applies to large batches of changes. If changes A and B are easy to integrate, but C is full of conflicts, it will delay merging A and B. Had you not batched up changes, you could've already deployed A and B and put a smile on your customers' faces.
To summarise, when trying to merge large batches of changes, the whole batch cannot be quicker than the slowest change within the batch.
Slow feedback
As I explained earlier, large batches of changes are more difficult to integrate. Therefore, it causes companies to ship less often.
When companies ship less frequently, they spend a lot of time building software without having spoken to users first. Consequently, they'll waste time building things no one wants.
And please don't get me wrong. Every company will eventually build something nobody wants. The trick is to ship as quickly as possible so that you realize what to do next.
Diminished efficiency
If you try to fill a jar with large stones, there will be lots of empty space left in the jar, even when you can't fit more stones in. If you could fill it with sand instead, you could fill the jar completely.
Similarly, if you only integrate large batches of changes simultaneously, you'll miss opportunities to merge small improvements into the codebase earlier.
Instead of shipping these small improvements as soon as they're ready, they'll be bundled with large changes that take longer to integrate.
Attrition and low morale
Working at a place that takes weeks or months to ship improvements to customers is demotivating. It's even more demotivating to be the person integrating everyone's changes and cleaning up after them.
No matter the salary, great engineers are motivated by shipping, and Git Flow simply makes you ship less. Then, when engineers get tired of not shipping, they leave.
Why you should always merge to main
Merging straight to main is like forcing yourself to make fewer purchases at a time at the supermarket. If you do that, you'll have to go to the supermarket more frequently. Consequently, you'll be forced to think of better ways to get there or move right next to the supermarket.
In the case of engineering, forcing teams to merge straight to main encourages them to make integration less painful. Consequently, they make smaller changes because those are easier to integrate.
These smaller changes will then lead to faster feedback, which helps teams get to product-market fit earlier.
Additionally, smaller changes (or smaller batches) solve many of the problems I mentioned earlier; they increase efficiency and boost morale.
Downsizing batches also creates a positive reinforcement loop: the smaller changes get, the easier it will be to merge them. Therefore, they get smaller and smaller, making teams faster and faster.
How to merge to main and keep things working
Teams must adopt two primary attitudes when merging straight to main. First, they must automate as much of their processes as possible. Then, they must shift left all of the validation processes they cannot automate, like QA or product and design reviews.
Automating processes
Without automation, validating changes before merging them to main would be impractical.
When engineers are forced not to break main, they're compelled to automate their validation processes. That way, they'll feel more confident that their changes didn't break anything.
Shifting processes left
In addition to automating their processes, teams that always merge to main are also incentivized to "shift processes left."
In the case of security reviews, for example, those will not happen only after the code is merged to an intermediary branch or deployed to "staging." Instead, security concerns will be part of the change's implementation, and security reviews will be part of the pull request review.
Similarly, QA and product or design reviews must be done during the pull request approval process. Otherwise, an engineer might merge code that's undesirable from a business perspective and cause a lot of rework.
For example, imagine an engineer misunderstands how a particular feature should work. In that case, their code and their tests will work, but no one in the QA, product, or design team will see the new feature until it's merged and deployed somewhere.
At that point, main will have already been broken. Thus, it will not be possible to deploy main. Consequently, teams must scramble to revert changes or make hotfixes, which may lead to more bugs and stressful situations.
At Ergomake, we help teams shift QA, product, and design reviews left. We do that by generating a "staging" environment every time an engineer opens a pull request. We then add the link to that staging environment to its corresponding pull request so that QAs, designers, PMs, and other non-technical stakeholders can validate the changes.
Besides enabling engineers to merge code straight into main, these environments also make it easier for engineers to share their work with non-technical stakeholders without having to jump into Zoom calls, record videos, or fight on Slack to see who's going to use the only "staging" environment that's available.
Putting it all together
As transaction costs increase, batch sizes tend to grow as we try to diminish overall costs by making as few transactions as possible. That's true both for manufacturing and also software engineering.
In software engineering, when integrating code is painful, we tend to allow changes to batch up and merge them less frequently.
By merging code less frequently and increasing batch sizes, as Git Flow does, teams become slower. That happens because large changes lead to code being more difficult to integrate, which, in turn, leads to batches becoming even larger. As time passes, batches will be so large and difficult to integrate that companies will come to a grinding halt.
This decrease in speed also leads to slower feedback, low morale, and diminished efficiency.
By incentivizing teams to merge straight into main, they'll make smaller changes so that changes are less risky and easier to validate. Consequently, these teams will move faster.
To enable teams to merge straight into main, they must automate as much of their processes as possible. The ones that are impossible to automate, like security, QA, and product and design reviews, should be done earlier (or, in other words, "shifted left").
By shifting processes left, teams can do multiple smaller rounds of validation instead of validating items in a single big batch. Therefore, they'll move more quickly without compromising quality.
Wanna chat?
If you're interested in setting up an automatic "staging" environment for every pull request, you can set up Ergomake here.
If you've got questions with regards to the setup process, or if you just want to chat about your processes and see if we can help, you can also talk directly with me here.