The $20,000 Merge Conflict

Git is the most popular source control system in wide usage today, hands down. It’s also the unfriendliest source control system in use these days, hands down.

TFS and Azure DevOps have adopted Git as their default source control provider, and teams that previously had no experience with Git are now working with it, even though their previous default source control provider (TFVC or SVN) was a completely different animal.

Put those ingredients together, mix in a little frustration and merge conflicts, and you have $20,000 lost in productivity without realizing it, within in the first few months.

A cross-functional enterprise software team recently moved to TFS 2018, and along with it Git as their source control provider. The team was used to having long-lived individual branches for feature work, and they continued this practice with git.

Within a few weeks, however, they started to notice problems: changes that were previously in “master” (the default name for the centralized branch in git) were being merged out. As the team ramped up its feature delivery, this happened more and more. At first, they would reach out to the affected developer and ask them why their feature (that they thought was merged in) wasn’t in the delivered software. This kept happening.

Finally, one of the newer developers that it happened to asked other members of the team: What’s your branching strategy? “We have our own named branches that we use and we use the same branch for every feature” was the reply.

Ah ha! thought the new developer, that’s where the problem is. She proceeded to sit down with other members of the team and show them the problem. Git, you see, makes branches very lightweight — they’re effectively a post-it note at a certain point in the repository. Any new changes are also marked with this same post-it note, and when you’re ready, git will put all those changes right where that post-it note was, and then throw away the note.

TFVC can’t use post-it notes; it requires that you make physical copies of the files and make changes to those copies and then check each copy against its original and merge those changes together.

The problem — and the reason why the team was seeing conflicts merged out; is that it’s very easy and fast to move a post-it note to another point in the repository and tell git to start from there, but that’s not what was happening. The post-it note was staying at that very old place in the repository, and trying to check changes against that old starting point. If people didn’t move the post it note forward, git had a lot more work to do, and the potential for error increased.

The net effect was that every time the team had to stop and figure out how the heck those changes got merged out — if they even realized the problem was in source control in the first place — cost the team money. Delivery downstream impacts, developer time, cycle time as QA waited for new builds, all of that added up to — in some cases, a loss of $20000 per incident in productivity time.

The moral of the story isn’t to go out and buy git training (though it’s well worth it), it’s that the things that sap a team’s productivity and cost delivery teams money could be a mismatch between the process the team has and what the process should be for that team. In this case, the fix was a slight change in how develoeprs branched, an activity that took a sum total of a few seconds.

Where Modern ALMs (like JIRA) fail: The OODA Loop

f you’ve read Col Boyd’s biography, then you already know what I’m about to talk about, and if you haven’t read Col. Boyd’s biography, add it to your list. Col. Boyd was a fighter pilot, the creator of the OODA loop, and also the architect of the Desert Storm Battle Plan. After he’d already led a pretty full life of being one of the best fighter pilots in the Air Force, he started to study a problem that had nagged at strategists: Why are some fighter pilots successful and others aren’t? How can someone in a technologically inferior aircraft beat a pilot in a technologically superior aircraft? Through his efforts, he figured out a theory that drove success and failure in dogfighting, and it’s a lesson that applies to successful software development as well: Some teams win because they’re able to execute the The Observe, Orient, Decide, Act (OODA) Loop faster than their competition. This is the success that Scrum tries to replicate, and one of the reasons why Scrum practitioners are assigned Col. Boyd’s biography. The value in the loop is how quickly you’re able to execute the entire loop for any given problem. For small teams, this is easy, there are very few people to take part in the loop. For larger teams — teams that can be helped through process automation, it’s much harder.

You can read more about the OODA loop here, but for our purposes we’ll tie that loop to how current ALM software works:

Observe – Put in Jira ticket
Orient – Team prioritizes that ticket (manually)
Decide – Team moves that ticket into the “sprint” (manually)
Act – Team member(s) complete that ticket (and hands off to other team members, manually)

The loop itself drives how quickly a problem is manifested and acted upon — something that successful software teams do is that they’re able move through those steps faster than other teams. This is important for all types of teams, not just product teams. If you work in enterprise software, you have necessarily long sales cycles, and it feels like the impact of your work won’t be noticed for 6-9 months — but that’s not the case. Internal stakeholders see your work daily or weekly, and being able to respond quickly to their concerns is important to build and maintain trust in the organization.

So why is JIRA associated with slowness in teams? Why is it maligned by startups and small product teams? Theoretically, it should be the perfect enabler of executing the OODA loop quickly. It knows all the information. If you’ve set up JIRA properly to track that information it knows what the impending ticket affects. It knows who can work on it (who has bandwidth), and it knows based on your past efforts whether that item is likely to get done. All of that data is sitting beneath the surface, waiting to be queried, and yet JIRA hides it.

Our teams need the ability to make reasoned decisions; and JIRA contains all of the data we need to make process decisions, and JIRA can be automated to help your team execute the OODA loop faster.

If you’d like to know how to unlock JIRA to help you make these decisions, hit reply and let’s talk.

Improve the things that affect everyone

Most times the first thing a team will automate is their build process. This is an easy target for productivity improvement, and is table-stakes for most (if not all) software teams. Then, teams will automate their deployment pipeline, and someone will wave a checkered flag and poof the team is now practicing “DevOps”.

Congratulations! There’s nothing left to automate, right?

Sounds absurd when I say it like that, doesn’t it? Like, we all know and feel there’s more to be done, but it’s hard to put our finger on it, isn’t it? The code part has been automated, the part the developers control by themselves is automated. Therefore, there must be nothing left to automate.

Put another way, left to their own devices, developers will automate everything they think can be automated or will improve their day-to-day work. Most developers, absent time and the ability to think deeper on the subject, would never think to automate the handoffs, or the tasks they do repeatedly but aren’t visible problems, or non-tech workflows.

We accept a certain level of manual intervention when we deal with JIRA because that’s what we’re used to. We’re used to developers manually creating their own branches, or manually reaching out to people who know that area of the code well, or manually letting testers know it’s ready for review, or manually de-assigning themselves items when it gets close to the end of the sprint, or letting the build notify everyone when it’s broken instead of a targetted alert to the person that broke it, or manually decide who needs to be in on a feature meeting.

Those things add up, and they add up quickly, but these things I list are automatable — and JIRA already contains all the information we need to automate these processes. What keeps us from automating it (besides not realizing it’s automatable) is that we don’t realize its effect on everyone on the team. How much of your day is spent responding to a code review request only to find out there’s one or two people on the team who have even worked on that feature previously? Or how we create hooks and automations to enforce branch and commit naming conventions but we don’t automate the creation of the branch itself; or automate the commit message itself? Or that we don’t automatically have JIRA send an email to the Project manager de-assigning lower priority items from a developer who is overloaded; and suggesting re-assignment to someone who has worked in that area of the code before and has bandwidth? If you use the ability to label feature areas in JIRA, then you already know who has completed work in which feature areas in the past, and boom, that’s a good starting point for a code review or a feature discussion. No need to spend human time figuring something out that the machine already knows.

In our next segment, we’ll talk about the underlying principles that cause this to be the case, and give a recipe to fix them.

Where does your Organization fall?

I follow John Cutler on Twitter, and if you are into Product Management and developing a “Product Culture”, he’s worth your time to follow.

A “product culture” is one of those new-fangled movements that seeks to replicate the success of good product teams for everyone — even enterprises that may not develop products. Product Culture is the idea that a team, with all of the dependencies, authority, responsibility for outcomes, decision making, and execution contained within a single team, is able to deliver a far superior product or service than a team who does not have all of those components within the team.

You can imagine it this way: If you have to ask someone who is not a part of your team for something, then you don’t yet have a product culture. We’ll leave whether this is possible or desirable for another segment, but to give you an idea of what this visually looks like, John shared an image on his twitter account that captures this idea neatly:

Where does your organization fall? Leave a comment and share your thoughts.

What could success look like today?

It’s easy to get bogged down in the problems. We see problems every day. A never ending stream of problems.

It’s hard to remember that those problems are always going to crop up and it’s better for us to focus on turning them into opportunities.

The Build server is broken again. Tests are failing in the report suite again. Joe submitted bad code, again.

We have an opportunity to improve the build server. Let’s take a look at what makes the report suite so hard to test, and see if we can fix that. Joe, let’s take as look at what you’re working on. It may be you have a module with a lot of inter-dependencies. It may be that you don’t understand what the module does as well as we thought. It may be the module isn’t well documented. Let’s fix that.

There are no magic bullets to improving productivity wholesale; but there are indicators. A good first step is to see where the problems are and turn those indicators into opportunities.