Git branching strategy to achieve continuous delivery

Continuous delivery is a very common software delivery practice. There are many articles and same terminology used in different meaning. Though it’s a generic software delivery practice, let’s think from a version control and branching strategy perspective.

Photo by Felix Mittermeier from Pexels

It would be awesome, if we could have a git branching strategy and team discipline which helps in achieving following dream goals:

• Project manager, Delivery manager or the business doesn’t push the team to release the feature
• Developer gets blocked if his feature doesn’t reach production sooner.
• The more the delay, it should be painful to take the feature to production
• Developer pushes business to split the requirement so he can take the features incrementally to production
• Conflicting features will be evident at feature branch level rather than merged environment level

Environment branches

Environment branches are quite popular and many teams already follow. The core concept is to have dedicated branches for different environments, for instance:

• development / master branch
• qa
• staging
• uat
• production

It is up to the team to decide the meaning and usage of these environments. Having more environment branches is a pain if we like to achieve true continuous delivery. We will address this in later part of this article.

Branching strategy

There are many possible ways for the team to arrive at what works best for them. Following is one such way which got matured over time and proved useful.

Simple example

production branch — everything starts and ends here

• Any new feature branch is branched out from production
• Once the feature is developed it is merged into QA or equivalent branch for testing.
• Once testing is approved, the feature branch is merged with production.
• Make production branch as the default branch in repository settings. This way, all merge request will be automatically made against this branch, unless we specify a different one
• Make production branch protected, so that no direct commit can be done by anyone against this branch. Only allow merge request or pull request

Prefer less intermediate environments

In a pure continuous delivery team, just one intermediate environment for testing works wonderfully.

• Less intermediate environments forces the team to use shared environment optimally and keep moving features to destination (production).
• Less intermediate environments blocks developers, if some feature blocks other features from moving. Team decides faster on priority to unblock.
• the more environments we have, the more the distance between “What gets cooked and what comes to plate”
• Maintenance of environments
• Dataset (sample data) related issues

Feature branch

Two features developed in parallel

• Feature branch has just production + this feature related commits
• If some other feature got released while this feature is still in development, developer need to rebase with production. Else he won’t be able to merge it to production.
• Developer is forced to handle how his feature behaves with other newly released features. This is handled at feature branch level itself rather than environment branch

Another example with more environments

Handling conflicts while developing overlapping features

This by itself is a signal that the feature priority is not done right. In a short lived feature branch strategy, why are two related features developed in parallel. Can the second feature be started after the first feature is delivered ? If yes, that’s the best to prioritize related features one after another. Remember these are features that are built in small chucks and get released incrementally. This pushes the team to slice and dice the feature so that it can be released in smaller chunks.

Let’s say, if it can’t be prioritized later and need to be developed in parallel, it should be handled by over the table discussions, which caused the breakage in qa etc., In this flow, let’s try to do QA for both the features together and release them together.

No more feature promotion to different environment branches

Many teams follow promoting features to different environment branches and finally reach production.

Typical promotion of features

Disadvantages in this approach:

• feature conflicts arises on environment branches which already has other in development features.
• It becomes cumbersome to identify which feature is conflicting and who introduced the conflict. Let’s say, feature A, B, C is in QA branch. Some of them may not be stable. Some may got de-prioritized for next release. Some are experimental / research flows. It is time consuming and frustrating for the new feature developer to handle merge conflicts introduced by other team members, of which he has little or no idea.

alternate approach

Instead

• merge the feature branch to qa
• complete the testing
• Don’t merge qa branch to production. Merge the feature branch to prod
• Forget the concept of qa being a sacred branch. In this case, qa branch is our convenience branch for testing. What we expect to test goes here. Over a period of time, if it contains code which are out of sync with production, force rebase it from production branch

git checkout production
git pull --rebase
git push origin production:qa --force

• If too many branches are in qa which are not yet in production, it is a signal that team should work towards making quick releases, qa/testing team should unblock them. As we see, team’s focus & thinking should be towards making releases. Either release the feature or stop blocking others. If a feature is unstable for a long time, QA branch can be force rebased from production branch and have only the next feature in queue. Once the feature is stable again, developer merges it with QA.
• If a feature keeps coming to QA and keeps going back, it signals the team to rethink on what’s happening in that feature. Should some one jump in and do something about it. Is the functionality better than what is currently in production ? then why not release it and address the issue in a bug card. This helps the team to move faster.

Signals and practices

• anytime production branch can be compared to qa branch to identify which features are in qa and not in production. If there are too many features or changes., it’s a signal that QA is becoming a bottleneck. It gives team an insight and a chance to take corrective action
• If there are too many pull request against production branch., it’s a signal that team is holding from making a release. As a practice, we should stop holding merge request to production branch. If it’s waiting for a long time, it also gives a signal that our prioritization of features might be an issue — developers are working on features that is out of sync with business rollout plan.
• As a practive keep the changes smaller so that incremental releases are possible. Also it’s easier to do code review and analyse the release impact.
• There are times when developers take the feature branch to `dev` or `qa` environment and forget to take them to destination (prod). As a practice, we could while creating `pull request`, we could create draft mode pull request for other environments., This stays as a reminder in our list of pull request

To create draft pull request in Github

No master branch or development branch

By having a branch dedicated for all the development work defeats the purpose of continuous delivery. That’s why production branch should be the only point of reference. Literally features are developed for production and not for just for the sake of development. By avoiding a branch for development, we force the developers to target production and move feature towards it. Or else, no one is going to use that feature.

Database migration and rollback strategy

If we adopt this kind of rigorous continuous delivery style, then first few steps would be build the safety net, especially about the data.

• These risk are always there irrespective of a rigorous continuous delivery or normal delivery. But CD helps in bringing in the best practices.
• Data backup flows should be available. Capability to roll back the data on extreme cases should be a click away
• Schema migration frameworks, like support down migration. Some times, down migration scripts are not well tested, because it is not used that often.
• In a micro services architecture, these constraints brings in best practices on sensitive areas. It shouldn’t be a question whether to do continuous delivery. It should be what do we need if we need to achieve this.

What should we do when we just start a project

Courtesy: https://medialoot.com/item/free-coming-soon-template/ — Used for educational purpose

• A simple coming soon webpage with a notify email can serve as a start, if team agrees.
• This forces team to come up with ways to achieve a production branch as soon as possible. Let’s say in a week time, if we need to make a release what would that be.
• A Minimum Viable Product (MVP) that would help lay the base set of features that is just enough to start using the application. This is where real users of the product will start consuming the smallest possible slice of the product and give real feedback
• Without a strict branching strategy like this, arriving at a MVP takes lot more time.
• Making the first release now will be pushed harder by the developers than business.
• Team could arrive at quick temporary flows which will be refactored in upcoming releases

How this differs from Trunk based development

IMHO, Trunk based development is too rigid and GitFlow is too flexible. The approach that is talked in this article tries to pick best parts from Trunk based development(TBD) and GitFlow.

If team has 100% automation testing (Unit, Integration and End-to-end) and if the test passes, team is confident to release, then Trunk based development is a good match. Though it’s not impossible to have a 100% automation testing in place, in reality, it is not the case in many projects. Either we don’t have enough test or we can’t guarantee the stability of the application with existing tests in many projects.

In a typical Trunk based development, parts that complicates the flow:

• Need for 100% automation testing to promise us nothing is broken. In contrast, with above branching strategy, it gives us a stop gap environment (qa or staging or uat) to deploy the in-progress version and manually validate the functionality end to end.
• Need for feature toggle capability. In many projects, bringing such a capability adds more effort than the actual business requirement
• Let’s say we have feature toggle, testing with multiple combinations is not practical. Let’s say feature A, B and C exists. Testing different combinations of feature on/off gets harder. As the number of feature grows, these combinations could grow.
• Trunk Based development advocates any push to trunk gets continuously delivered to production. But when test coverage is less, we have the need for promotion of trunk to production and an intervention to decide when to go to production. The same trunk branch goes to qa/staging/uat environments and get’s promoted to production.
• While working on a feature, if there is a need for HotFix, that’s going to have different flows and practices.

Good parts in TBD:

• One branch which will reflect the production state. All feature changes starts and ends here. In the approach that we discussed, we try to bring this practice. Just the name of that branch is ‘production’. In TBD, they refer this as ‘trunk’ or ‘master’

Need for Continuous integration and deployment flows

The approach that is discussed here, depends on following

• Feature branch automatically creates Continuous integration pipelines. This is possible with multi branch strategy pipelines supported by most CI solutions (both hosted and on-premise)
• Environment specific auto deployments are in place. For example, a commit in qa will auto deploy the changes to qa.
• Automation tests are in place. If test coverage is limited, this approach gives scope for manual testing with environment specific auto deployments.
• Code quality checks are in place
• Feature branches need not be deployed. Only merged environment branches gets deployed

Summary

Initially there will be mental blocks for us to adapt this approach. Especially if we are coming from features getting promoted to different environments and finally reaching production. But once we try this out for couple of iterations, it is for sure hard to go back to old ways. Just like it was hard to move from SVN to GIT and once we moved., now going back to SVN kind of version control is very hard.

There will be little more effort and discipline involved from developer side in resolving merge conflicts and keeping feature branches in sync with production. But these brings in good practices to release features quick and stop delaying the release. Over a period of time, making multiple releases each week becomes a norm.

References & additional reads

• Git branching — Rebasing
• Rebase as an alternative to merge