Author: geostock

The Problem With Sprinting

Sprint, Such a cruel name for the practice of delivering software features at a fixed interval, especially because of the carelessness in which the name was created:
​

And so my team embarked on what we called “Sprints”. We called them that because the name evoked a quality of intensity. We were going to work all out for a short period of time and then stop to see where we were. — Jeff Sutherland, Scrum: The Art of Doing Twice the Work in Half the Time

Some teams make their time-box two weeks, some make it a month, some make it 6 weeks, nearly all call it a sprint, and perhaps disastrously, Scrum dictates that you should be able to ‘sprint’ continuously:

​”Working in Sprints at a sustainable pace improves the Scrum Team’s focus and consistency.” – ​The 2020 Scrum Guide​, “Scrum Team”

“Sprints are the heartbeat of Scrum, where ideas are turned into value.

They are fixed length events of one month or less to create consistency. A new Sprint starts immediately after the conclusion of the previous Sprint.” — The 2020 Scrum Guide, “Scrum Events”

So “Sprint” at a sustainable pace, indefinitely? That makes a ton of sense.

But, the unfortunate name that stuck is not the only reason Sprints are inherently a problem, even if it’s one of the lowest hanging fruit. The Sprint introduces regularity into your development cycle, no doubt; but it also artificially constrains your team’s thinking. How much time is spent by your team trying to fill the ‘capacity’ for a sprint, or deal with the fact that a piece of work might have to go into the next sprint, or whether the schedule should be adjusted because the sprint ends up falling into a holiday, or event trying to match up your sprints with a fixed deadline like a trade conference or a quarterly budgetary review by the C-Suite?

Besides the management to the sprint issues, there’s the constant pressure to deliver a chunk of work every two weeks, constantly, and without fail.

Do you know what the prime ingredient of a death-march is? It’s the relentless nature of the work. And nothing sounds more relentless than the definition of a sprint, especially combined with the idiotic “agile capacity planning” that happens.

The idea of software development as having the regularity of manufacturing is appealing, and far too appealing to folks who have nothing else to latch on to. But the myopic nature of the sprint causes teams to artificially stuff their work into that timebox, without giving them the benefits of a forcing function like a timebox.

Does the overhead of a sprint really help your development team deliver value? Or does it give them extra work to do without any extra benefit? Put another way, is the juice really worth the squeeze in the year of our Lord 2024 when continuous delivery has never been easier?

Moving on, another issue is size. Once work is ‘too big’ to fit into a sprint, someone has the bright idea, “Well let’s try to see what we can deliver in that two weeks that has value but may not be the whole enchilada feature?” And so the team spends valuable time trying to figure out how to deliver something in that two weeks, while the business folks are watching with equal parts impatience and amusement at the strange rituals of these software developers.

If you’ve asked for a car and someone hands you a tricycle because that’s what they could deliver in two weeks, your first thought probably isn’t to thank them for the tricycle.

Much like the majority of Scrum, the sprint and its events manifest as a dog-and-pony show meant to soothe the egos of the folks asking for them, and not at all empirically useful to the folks doing the work or the folks who will benefit from the work being done.

If your team is product-focused and it feels like the wild-wild-west, Scrum can give you structure and regularity. But once your team has that structure and regularity, you’re far better off jettisoning scrum for something custom suited to your company and its culture. Some folks say “scrum, but” as if it’s a insult, when in reality scrum, but is the only sane way to practice scrum.

What I mean when I use the term ‘agile’.

I always wondered what authors meant when they said they sat down to write one thing and ended up writing something else completely different; or that they sat down with one motivation for their protagonist, and halfway through had to change the entire premise because they realized their protagonist’s motivation was different than they thought it was.
​
‘Wondered’ here is in past tense because as of yesterday, I no longer wonder. It happened to me. I sat down to write an email titled ‘what I mean when I use the term agile’, and got halfway through it and realized I was writing a completely different email entirely. So here’s the email you were originally supposed to get when I wrote the “agile is not a process” email. (Authors note: For the folks viewing this on my blog, this paragraph may seem out of place; Every so often I post things from my email list here, and this just happened to make it. To be sure you get these sorts of things when they’re published and not potentially weeks after, if at all, subscribe to my email list in the box below).

I want to start out by saying that despite my experience and belief showing that we’re entering a post-agile world, that unlike some folks, I do have a concrete set of things I mean when I use the term ‘agile’. This may not agree with your definition (if that’s the case hit the reply button and let me know what your definition is), but it’s a definition, and if we’re going to have any sort of coherent conversation on the merits of agile for software development, we’ve got to start somewhere. This is that somewhere.

Agile is a contrast to waterfall-based development; in that waterfall treated software development like construction or other hard engineering applications with the following discrete and sequential phases: Requirements, Planning, Design, Development, Testing, Maintenance. Once you went to a new phase, you didn’t go back to the previous phase. Waterfall projects are typically defined with lots of ‘gates’ and documentation, and do not handle any change in requirements well at all. They’re expensive when they work, and even more expensive when they go off the rails, and since Software Development is still in its infancy as a discipline, they go off the rails quite a bit.
​
Agile instead defines software development as an iterative process, where those phases may happen dozens of times during a project’s planning and launch; and even more so, jettisons the idea of extensive documentation with embedding the customer into the team, and working with the business people in a collaborative fashion, instead of through these well-specified hierarchical channels. If you can produce working software quickly, you have less need of extensive documentation — since the documentation was there to describe how it worked.

Another principle of agility is the idea that since requirements change quite unpredictably, the software team should expect that and operate with flexibility towards that outcome. To that end requirements (according to some agile processes) can change every planning cycle.
​
So to distill the meaning of agile down to something that can fit on a postcard:
​
Agile is a philosophy of software development where software is released frequently and iteratively, based on the requirements needed in that moment, with the business and development team collaborating and involving input from the customer during the development process.

Who Benefits from Agile processes?

Not everyone would benefit from the various processes that implement the agile philosophy.
​
Besides the structural deficiency in the agile principles themselves; the various processes that tout their alignment to agile principles ignore one of the most foundational elements of any effective process: they pay no attention to introducing scientific process rigor into the experiments that are “Iterations”.

Leaving that gaping process hole aside, there’s also that even if your favorite flavor of ‘agile’ was perfect for the scientific rigor needed for your context, not all processes are suited to all companies in all contexts.

We fully expect this sort of nuance in other industries, even if we completely ignore it in software development. What works for McDonald’s wouldn’t and shouldn’t work for your local mom-and-pop burger joint, because their aims are different.
​
So too with Software Development. The aims, structure, and goals of an organization dictate what processes should be used to achieve those aims. Slapping some agile on a team because it’s the most popular methodology is the height of folly the bounds of which are only surpassed by the insistence that it’s the best we can do.

Those issues aside, even assuming for our purposes that agile really is the best we can do, here are three situations where it’s ill-suited for success:

  1. A known scope, known timeline project wouldn’t benefit from an ‘agile’ approach because the customer both knows what they want, and they have expectations on how to get there. A team ‘inspecting and adapting’ is a likely to be irritating to them as it is useless to the overall goal.
    ​
    A corporate IT project falls in this category quite a bit, especially for a system rewrite. Now rewrites are terrible for other reasons, but if you’re trying to fit an ‘agile’ process into that situation, at best it will feel awkward, and at worst becomes a sort of Kabuki theater.
    ​
    2. A variable scope, known timeline project fits better, to be sure, but it suffers from the “adapt” part of agile. If you have a known timeline all work has to be finished by (whatever that work is), then what matters is getting the most done in that amount of time. If agile is the art of “maximizing the amount of work not done”, then a situation where you want as much done as possible is antithetical to the agile way of thinking.

​
3. A true fixed-budget startup cannot afford the ‘trial and error’ nature of agile. You only get so many iterations before you fail, and you need something deeper than ‘inspect and adapt’ to survive.

If you find yourself in any of these situations, adapting an ‘agile’ process, especially one like Scrum, is going to feel kafka-esque at times, and any success you may achieve may be in spite of your choice to adopt an agile philosophy, and not because of it.
​

The nice bits of agile

I don’t dislike agile. I think Agile is a welcome change if your team’s current method of developing software is “we release every 6 weeks and conduct a UAT, and come up with a detailed plan before we start working, complete with timeline and commitments from the team”, and in those situations I’d shove the nearest copy of The Phoenix project or The Unicorn Project into your hands and say “Let’s give it a go, where you’re going is better than where you’ve been, but it’s going to be a bumpy ride so buckle up.”

And it would be better than the thing you’re doing, and maybe you even decide to keep it.
​
That’s all fine. No issues there, and in fact there are lots of bits of agile I happen to think are good practices for a team to consider.

But, and this is the important part: Not all teams should ‘do’ agile and all teams that are doing agile but not having the success they are looking for are probably running into a foundational problem with the philosophy of agile itself.

But, to keep this positive, I want to talk for a minute about some of the practices I like that the various agile methodologies tout.
​
Talking to the folks using your software daily? Seems basic now, but can you imagine 20 years ago? It was revolutionary.
​
Getting to the point where you can release whenever you want? Besides the immediate benefits, it also means that all of those silly little things you were doing manually are automated, and from a software developer’s perspective automation is akin to godliness.
​
Ditching the plan at the first sight of trouble? I’m a Big fan, especially since software as a discipline is still in its infancy. We aren’t building bridges where we’re dealing with all known quantities and calculations and domains (not to mention for our purposes the rather iron-clad laws of physics). Software development is still at least one-part magic, and until that’s not the case the ability to ditch the plan at a moment’s notice is a good thing.

What about you? What do you like about agile?​

This post originally appeared on my daily email list. I’m reproducing it here in vain hopes you’ll join said email list. The box to join should be below. Someday I’ll have this process automated so my daily emails pop up here too, but the cobbler’s kids and all that.

Agile is not a process.

I bandy about the term ‘agile’ on my email list a lot. While it may seem like I use it to mean “practices we adopted but didn’t solve our problems”, it also has both a specific set of things it means, and at the same time is a too broad categorical list of things executives want it to mean. To say it’s a process or a methodology is being too generous, as it’s a process in only the most hand wavy sense of the word.

It’s a set of values and principles, and we’ve somehow created from that a central thesis that it is useful as a definition of a development methodology, and is somehow prescriptive in what it means.

“We’re agile” has come to mean “We’re moving fast and breaking things to get new, not-thought-out untested features delivered this quarter to make our executives look good at the industry conference”. Or, “No we don’t have a plan, we react to whatever is happening now”, or “Our development team runs at 100% utilization because idle developers cost us money, and developers are expensive.”
​
But, those archetypes of modern agile aside, even if we were to take the agile values and principles at face value, you’d be unable to articulate an actual process you’re supposed to follow if you adhere to these principles and values.

If you hired a consultant, and they said to you, “Value People and interactions over Processes and tools”, I hope you’d slap them silly (after asking them what they possibly mean by that in a practical sense). Does that mean runbooks are out? Style Guides? Checklists? Of course not, they’d exclaim! Then what does it mean? And at this point they’d probably just repeat what they said, and both of you could share an awkward silence. What does it mean? Seriously, I’m asking. At best it means “don’t let following the book keep you from throwing the book out when it needs to be thrown out”, and that both vague and, to be fair, not a bad fortune cookie fortune. When would you throw out the book? Under what circumstances? When it’s inconvenient? When it actively detracts from what a particular user wants? When it gets in the way of the quarterly profit? When? The problem with this sort of ‘value’ is that it sounds good but doesn’t actually define its parameters, making it terrible as part of an actual process definition.

Likewise, taking at (not really) random an agile principle: “The best architectures, requirements, and designs emerge from self-organizing teams.”
​
I have to ask what sort of hate the inventors of agile have for the rest of us that they imposed their context and view of the world on us and we just said, “Yea, that sounds right”.

The best architectures, requirements, and designs demonstrably do not emerge from self-organizing teams, because those self-organizing teams have constraints they don’t control. It doesn’t matter if the four of us on a team get to pick what we work on, or what tech-stack we use, or whether we make it a relational database or a no-sql database. We’re not the Avengers. We don’t have superpowers that obviate the need for resources and budgets. (As an aside, in the MCU movies, shield doesn’t seem to do anything other than get in the Superheroes way; whereas for the non-superheroes among us, we’re always dependent on some sort of bureaucracy to support our work).

Normal teams don’t control their budget, the market, the board of directors, or the C-suite that is hunting for a quarterly profit. It is both defeating to claim there is such a thing as a self-organizing team (and have it be true) and downright dangerous to think that with any sort of seriousness. There will always be external constraints, and putting the idea that the team, by virtue of being ‘self-organizing’ can somehow navigate an entire corporate culture, and know what is right for the customer, product, and business, is foolhardy — at best.
​
It is also stupidly dangerous because it isolates the ‘development team’ from the business in which it operates with a blue flag that says “Just let us organize ourselves!”. The flag may as well have a snake and say “don’t tread on me” (It’s election season here in the USA).

Now, I understand the desire behind that principle: That if you let a team figure out what they need to do and how they’ll be more bought in and do a better job than if you had some VP or director standing over their shoulder indicating what they can and cannot do and how. And if you say, “well it doesn’t actually mean any of that, it only means that they get to choose how they get the job done”, I’d like to sell you a bridge. Agile pretended that somehow multi-million dollar budgets were in the control of the team, or head-count, or technology choices, or what sort of features get worked on. As a principle it’s unrealistic, and even if it were completely realistic and attainable, would be about the worst way to run a software team as can be imagined. Human Hierarchies and interdependent groups exist because that’s the ‘best’ way we’ve figured out how to work for at least since the middle-ages. The builders are not also the dreamers, and the dreamers are not also the folks with the money, and you need all three to get anything done.
​
Agile is a set of values and principles; it’s not a well-defined (or even an ill-defined) process, and treating it as such will at best lead folks to misunderstand each other and lead to bad outcomes. It’s a set of values and principles you can use to create a process, but it’s not a process in of itself. As much as I dislike the results of Scrum, it’s at least a process.

This post originally appeared on my daily email list. I’m reproducing it here in vain hopes you’ll join said email list. The box to join should be below. Someday I’ll have this process automated so my daily emails pop up here too, but the cobbler’s kids and all that.

“Agile” isn’t a system. It’s a tool.

Does your software team:

  • fight a lot of fires?
  • release a lot of ‘hot fixes’
  • constantly push back releases?
  • release half-baked features only to never return to them again?

​

Do your senior leaders blame any of the above on “the market” or “market forces” or “business dynamics” or even “the team”?
​
The answer in reality is a bit closer to home (than even blaming the development team!)
​
The problem is you. Specifically, That thing that surrounds your entire company that would be called a ‘system’ in systems thinking. It surrounds your company, every action you take, how your customers respond to that action, other ‘market forces’, and just about anything you can (and can’t!) see, smell, touch, and experience as an executive.
​
It’s like the Force, except it’s real.
​
The book definition is that a system is ” system is any group of interacting, interrelated, or interdependent parts that form a complex and unified whole that has a specific purpose. ” (Kim, Daniel H. (1999) Introduction to Systems Thinking. Pegasus Communications, (Pg. 21))
​
Yea, you have a system, even if you don’t know it. — and I don’t mean agile — that’s not a system for developing software. It can be a part of a piece of a system, but it’s not the whole system itself. In this case, a system is both the parts, their dependencies, and inputs and outputs that result in software being delivered and loved (or hated) by the folks that use it.
​
Things like:
​

  • Culture
    – Management Culture
    – HR Culture
    – Company Culture
    – Developer team culture
  • Past Events and how your senior leadership handled them
  • Policy
    – PTO policy
    – reimbursement policy
    – training policy
    – BYOC (Bring your own Computer) policy
    – Requiring specific brand/model computers purchased by IT
    – Financial policies meant to save the company money
    – Travel policy
    – WFH Policy
    – RTO Policy
  • Training
    – Management Training
    – Developer Training
    – Financial Training
  • Layers of Management
  • What gets rewarded
  • What gets punished
  • How the team receives feedback
  • How the team receives information
  • Your budgeting process

​
​And right at the very bottom, after all those things that actually matter, are the systems we think of when we talk about developing software:

  • The methodology the team uses to develop software

​
But, that’s just one tiny bit of the picture, and on its own its almost meaningless to how your software actually gets developed and used and whether your business is successful or not. Don’t believe me? Here’s one simple way to find out: If you’re an agile team, look around at your agile process and ask yourself, is the business’s success with software totally in our hands? Or are we dependent on someone external to the development team for success?
​
A final thought before we go: If the agile methodology de jour isn’tthe whole of the system your business uses to develop software, then why do you give it the importance of being responsible for the whole of the system? It’s a tool, and a tool for a tiny piece of your business’s overall success or failure.

This post originally appeared on my mailing list. From time to time I put posts from my mailing list onto here, but not always. If you want to read what I’m writing, your best bet is to sign up for my email list, below.

After Agile.

We are uncovering better ways of developing
software by doing it and helping others do it.

The First lines of the agile manifesto

Since the adoption of the agile manifesto, we’ve spawned dozens of ‘agile’ methodologies[0], in pursuit of better ways of developing software. Given the staleness of software today, it’s evident we haven’t uncovered better ways of working.

Computers microprocessors (such a quaint word, given where we are today) are thousands of times faster than what we had available when the agile manifesto was written, and yet I write this on WordPress’s online block editor, on a MBA 2019 model, in Firefox, and the cursor is very slow. It takes ten seconds to paint 12 characters. I’ve force-quit everything I could, and only one tab is open.

23 years ago, the major problem facing software teams was the planning process. Software Development methodologies looked a lot like manufacturing methodologies: capability research, capacity planning, design, development, testing, and maintenance. Each step was discrete and carried into the next. The folks that met at Snowbird saw this process as the enemy of good software; and for a time[1] they were right. For a time the ability to take an idea and put it into production in weeks instead of months and years was the greatest improvement in the delivery of software. Instead of conquering that mountain and moving on to the next, we’ve stayed there, built a city, and let it decay.

Software delivery times have decreased[citation needed], and software has gotten worse to use. We’ve spent so long optimizing for delivery that we’ve completely forgotten the purpose of software: To make our lives better.

Cory Doctorow calls the decay of software enshittification, and he explains that it’s due to platform lock-in[2]. Even without platform lock-in, we’d be in this mess. The incentives are the same in all cases — a desire for faster, cheaper software — but the problem is culture, not market position. It’s endemic to how we create software. We’ve drilled it into ourselves that we have to do things cheaply and faster than fast. Since most software organizations have adopted agile in its various forms (sigh: Scrum), then I’m going to blame our current predicament on our over-reliance of agile as the answer to our problems in developing software[3].

It’s 22 years old, it can take it.

I accidentally touch my trackpad (also a quaint word) and the block editor dialog menu pops up. I move my cursor across the page and the editor menu pops up over the words I’m reading. Frustrated, I have to move the page and click further down to get the editor to stop popping up over the words I’m trying to proofread.

The company that makes wordpress dot com, and who makes this block editor is Automattic. Automattic is valued at over $7.5 billion. $7.5 billion! I hope they would spend a mere fraction of a fraction of that on fixing the problems with this editor. The wordpress dot com block editor editor is more downtrodden than a Dickensian protagonist in the first act.

There is one common theme that unites the software process of teams at tech unicorns, mom and pop shops, and startups, Agile. Popularized by the agile manifesto (yes, yes, I know, the Scrum folks will claim they were first, but the wheel by itself wasn’t popular until someone put it on a wheelbarrow), agile has come to mean any iterative software development process that ideally pays lip service to the principles of the manifesto[4].

Agile was an iterative software process codified in the year 2001 by mostly programmers and consultants, at a ski lodge and focused on the areas of agreement, along with principles for software teams and businesses to follow in the development of software.

What emerged was the Agile ‘Software Development’ Manifesto. Representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes convened.

From “About the Manifesto“

Far better than the waterfall methods that preceded it, the agile manifesto focused on producing working software in weeks instead of months.[5]

Agile’s tenets are simple[6]:

1. Get working software in front of a customer on regular, short intervals.
2. Use the feedback from customers to iterate on it.
3. Inspect how you build software and adapt to change brought on by the your customers, the market, stakeholders, and the changing skill-sets in your own team.
4. Make adjustments to your process, and rinse, lather, and repeat — indefinitely.

The process itself relies on a focus on the team building the software, with a bit of ignorance towards the processes that businesses need to survive. It all but ignores the cost of producing features in favor of just doing it. It ignores the cost of long term maintenance, of saying yes, in favor of a process that says yes and delivers in short intervals. It does not elevate the creation of software to the realities of the business that relies on that software, and instead posits that somehow if you inspect and adapt enough, you’ll be a better software feature widget maker and that is what your business needs.

Agile delivery pairs perfectly with a sales-driven feature factory, where new features drive new sales, until all of it collapses on the weight of itself and the business is forced to sell itself or start over from scratch. IT Projects are either too simple to be useful, or too complex to be delivered (or replaced) successfully, all while the team inspects and adapts every two weeks, like clockwork.[7]

Of course, it’s not supposed to be this way. One of those cute little things we say but rarely practice is, “Agile is the art of maximizing the work not done”, where you can be agile by saying “no”, and where saying “yes” creates commitments, which wears down the team’s ability to move… well.. agily.

Is this agile’s fault? Yes. If the rules of the game are vague enough to drive a truck through, the rules are at fault. If the rules of the game regularly produce outcomes that are sub-optimal, then it’s up to the rules of the game to be fixed.[8]

Agile has several truck-sized vagaries that we ignore at our own peril:

  1. There’s a cost to being agile. There’s a cost to delivering that widget that the CEO or the sales team says the customer wants. To delivering what someone asks for, and not looking deeper into their expensive problems and seeing if it should even be solved how they’re asking. This has to be the cornerstone of any software process: uncovering the root of the symptoms the customer talks about, and digging deep into that, and its first order causes and second order effects. If you thought you’ve dug deep enough, you probably haven’t. I like Jonathan Stark’s Why Conversation for this; because it tries to talk the person out of that thing they say they want and uses three questions: Why This?, Why Now?, and Why Are We the ones that need to solve it?

    A process that optimizes for quick-turnaround, for learning through iteration, encourages us to stop at the level of “produce what is asked for, not what is needed”.

2. Inspecting and Adapting is not nearly enough. You can’t solve the right problem without being in the room where the problems are discussed. By the time the problem has gotten to the ‘agile team’, it’s become a directive with a solution. The idea of a cross-functional team doesn’t take into account budgetary and business considerations. Those become inputs that the product manager or “Product Owner” uses to decide what to build that fits those parameters, without the team being fully involved in the business problem definition process and the budget process. If the team isn’t talking budget numbers, they aren’t close enough to the business processes that define success and failure.

How many times have we seen a matrix of decision making? The Product Manager decides what to build, the developers decide how to build it, and the business decides how to use that to solve its problems. Those divisions have caused more harm to our industry than any other singular mindset.

Businesses don’t make line teams part of the budget or business goal planning process. By the time the realities of the business get down to the teams, it’s purely an exercise in specific features or at best KPIs/OKRs that need moving. This is a fundamentally broken process that agile encourages in part of its origins of being development team focused.

3. Features are promises realized. Likewise, broken features break trust. The idea of deliver, then iterate presupposes two conditions that are not universally true:

  1. That the broken feature will be improved until it’s good (instead of “good enough”), and trust regained as the feature improves.
  2. That you can make up the loss of trust brought on by broken features with more features.

While this problem has existed long before agile, agile normalized delivering broken features and iterating on it until it wasn’t broken any more. It did this for good reason — that it’s better to deliver something half-baked early than it is to deliver the perfect thing months too late. And the sentiment holds truth: if you want to know whether a feature will be used, ship the feature. But the realities of software development have made this mantra a punchline.

As I keep the browser window open that I’m typing this into, WordPress’s cursor and text being painted into the editor is taking longer and longer, with me waiting 10s of seconds for it to catch up to me, and futilely moving my cursor to correct a spelling mistake because WordPress was 10 seconds behind me by the time I stopped typing.

Move fast and break things

Facebook internal motto until 2014

The agile world has brought us a world where broken is normalized, where Perfect is the enemy of the good has become a rallying cry for agile software delivery. This is where I align with Cory Doctorow. Customer lock-in is an easy path to repeatable profits, and VC money means there’s a new market to chase, and no need to keep the existing market segment happy, and that the fundamental practice of making good software is shunned in favor of finding new ways to extract attention from folks. Our whole process for creating software is optimized for this world[9].

4. The gulf between the software team and the team making the business decisions grows wider each year.

Ben Affleck: “Wouldn’t it be easier for NASA to train astronauts how to drill rather than training drillers to be astronauts?”
Michael Bay: “Shut up.”

DVD Commentary, Ben Affleck on the premise of the movie Armageddon

There’s one key ingredient that’s missing from software teams, and that’s the budgetary authority and responsibility for the software that’s produced.

Teams don’t control their own Profit and Loss, they don’t control their own headcount, and they are often the last to know about any strategic changes in the software they’re charged with delivering. While this practice makes (some) sense in developed industries like Construction and manufacturing; it falls flat when you try to apply that to a software team. Software teams don’t have a known cost for a feature. The budget for a block editor (like wordpress’s block editor) could range from a few hundred dollars a year and forty hours of developer time to several hundred thousands — or even millions of dollars. There’s a whole ton of context around the budget that the developers have that most other folks won’t have. In some companies, getting that budgetary authority requires the CTO to step in, and often it becomes a negotiation (at best) with the CEO and CFO to allocate a pot of money. By the time it gets down to the team, there’s no discretion or authority to change from an obviously bad approach.

It’s sort of like 20 privates trying to tell congress how to run the Army. Yea, they’re in deep, and yea, they understand the ‘on the ground’ problems, but the Army’s on the ground problems and their strategic problems are miles apart in theory, even if they aren’t in practice.

The second biggest issue — besides the wrong framing for trying to fix the problem — is the lack of institutional buy-in from the business side. Agile requires business folks to buy in to the idea of incremental delivery, frequent introspection, developers being as close to the customer as possible, and maximizing the amount of work not done. And I mean really buy in. Owners of businesses will often fashion themselves after Steve Jobs and believe they’re a good proxy for the customer. They aren’t.

CEOs will often claim incremental and iterative delivery is in their best interest, and then want a delivery plan for the next quarter, or 6 months, or a year.

It’s impossible to have an accurate, or even 50% accurate, delivery plan for a year in an industry where priorities change weekly or even monthly, and yet software folks are expected to stick to their estimates as gospel quotes.

Even if agile is a perfect software delivery process (it doesn’t claim to be), the problem lies not in agile, but in our reliance on agile as the be all and end all to creating software.

We’re doing things the same way we did them 23 years ago.

“Agile” is 20+ years old, SAFe is taking the enterprise world by storm, Scrum Masters are being laid off while the Scrum industry deals with fallout from the broken promises of Scrum. Experienced software folks have moved on from Scrum to kanban or “scrum, but” (we do scrum, but we don’t do X, Y, or Z from the Scrum guide — or my personal favorite — we do the parts of scrum we like but we haven’t actually read the scrum guide recently), folks are still calling “scrum events” ‘ceremonies’ or ‘rituals’, and we live in this bastardized world where the rules are made up and the story points don’t matter — except to the executives that take them as gospel truth.

This is not a great place to be.

We’ve known story points were bad for over a decade now; and the last four years it’s become mainstream enough you’d think we’d have dropped them, but we haven’t.

We’ve known developers can’t estimate unless you break down ‘stories’ so small that it becomes trivial to estimate each part, and we also know there’s very little value in spending enough time on breaking down stories that small you can estimate them, but we still have business folks pressuring software folks for estimates and then holding them to those estimates.

We’ve known that focusing on feature delivery severely undermines both the user experience and quality of the software being delivered; and yet the idea of ‘slack’ in capacity is seen as a cardinal sin. If reading this doesn’t make your skin crawl, you and I probably won’t agree on much. I wish you well.

And yet, in 2023, these very serious flaws in development are alive and well and present in just about every “agile” team out there, and even if they aren’t, it’s a fight against entropy and business trying to control the situation by bringing up these very flawed ideas.

We thought Scrum would save us, it did not. We thought Kanban would save us, it did not. The conventional wisdom now is that ShapeUp will save us. Spoiler alert: it will not.

The problem is none of these methodologies take into account the realities of how decisions get made. By the time you’ve gotten to improving software delivery, the decisions have been made and are (for all intents and purposes) cast in stone. These methodologies are pretty good when it comes to behaviors of the team making the software itself; but somehow ignores the entire business process before the act of creating and delivering software. They miss that there’s this pot of money that has to be allocated, and they treat that money allocation as a given, and as a limitless resource. These methodologies also miss that business priorities do not align neatly to features, they align neatly to problems, but at the moment the software team becomes involved, we’ve skipped past the problem and are solutioning.

It just so happen that that solution looks a lot like things the management team wanted anyway.

Another common problem is that teams don’t know how to talk about money with the business; or communicate the carrying cost and opportunity cost associated with certain courses of action. And when we do talk about these problems, we phrase them as ‘technical debt’, which not only obfuscates their impact, but uses a term from the financial world that misses the actual problems faced by teams when dealing with technical debt.

‘debt’ is a two-dimensional term, financially speaking. I’m either paying to alleviate that debt through reduction of the principal, or I’m paying just interest on the debt. But at no point do I find out that I can’t even start my car because I’ve taken out this debt to buy the car. Debt doesn’t affect the current operation of the thing I bought with it. It affects what I buy, but it doesn’t affect my ability to go about my day-to-day business of what I’ve already bought. 

That’s not true for technical debt. How many times have you had to deal with the problems arising from technical debt keeping you from doing a simple bug fix? Or adding a new feature? Or not even being able to adopt a new style of working because of the ‘technical debt’ the team took on? Or existing features become impossible to maintain because of the debt you took on? Sometimes it feels like building software is negotiating with a sentient and malevolent being hell-bent on your mental destruction, manifested as no more than text files on the ethereal plane.

So when we try to talk technical foundations in business terms as software developers, we get it wrong. When we focus on optimizing software delivery and ignore the alignment of business, finances, and economic realities before us, we get it wrong.

The point is this:

Agile is not (and never was) sufficient to produce software that folks love to use and that meets the goals for which it was created. We spend more time creating churn and problems for ourselves, our businesses, and our customers through our over-reliance on agile than the benefits we get from optimizing software delivery with agile methodologies.

It’s time to look past agile. It’s time to focus on a holistic software creation process that aligns with budget realities and the realities of business. A software creation process that aligns what software creation is with what we need it to be: A way to improve people’s lives and enrich our culture. A source of joy, not frustration.

We’ve optimized software delivery. Now let’s focus on the realities surrounding software creation. Let’s optimize the next 20 years optimizing for the purpose of software: To make people’s lives better. We can’t do that until we fully align software development with the underlying business realities funding it.

Endnotes:

[0]: Scrum, Kanban, XP, Lean, LESS, SAFe. KPIs. OKRs. Knolway. You may ask why KPIs and OKRs are in this list. They have all the same characteristics as other ‘agile’ methodologies. The chief distinction between a waterfall/pre-planned approach to developing software is the ability to point to a goal outside of building the software itself, and iterate towards that goal, knowing the conditions that will consider that goal met (inspect and adapt). KPIs and OKRs provide an iterative framework to achieve a goal, and to inspect and adapt to reach that goal. At the most basic level, all agile methodologies seek to do that exact same thing, for that exact same reason. Just because some guys didn’t come up with it at a ski resort doesn’t make it any less agile.

[1]: So there’s data here; and I haven’t dug into it deeply (yet), but a cursory glance suggests that in 1995, 83.7% of software projects can be considered ‘failures’ in 1995 (“the project is completed on-time and on-budget, with all features and functions as initially specified.”), and the Project Management Institute survey suggests (edit: 8/20/2024, link updated) that around 69% are failures today (“Do not meet their goals”). Now, data-wise this is not a big jump, and not enough to tell businesses to, “adopt this approach because you’re 14% less likely to fail”.

This ungodly long opinion piece is not about whether agile was ever right, just that it’s past time to look at “What’s next”, and treat agile the same way Waterfall was treated 30 years ago; as standard, but insufficient in creating good software that people love to use and brings joy to their life.

[2]: Cory Doctorow is a luminary in tech. But, I see a particular blind-spot with his way of thinking, and that is it reduces our problems to a specific economic model, whereas if this problem were only about the economics of VCs and startups and “Get big fast and corner the market”, then it wouldn’t be endemic to all software. But as I write this, it’s hard to find software that hasn’t allowed this entropy to set in (even as I hesitate to call it ‘entropy’ because that pre-supposes that it’s inevitable, and I don’t think it is, as it’s entirely in our power to maintain quality and joy of use. Nothing is stopping us, except us).

[3]: The problem was never the mechanics of software delivery. When that’s the easy thing to blame (and the easiest to fix politically!) we focus on that. I haven’t gathered the data to back this up (this is a wake up call, not a research paper), but I’m willing to bet we’ve been looking in the wrong places for answers all these years. Optimizing software delivery has turned into the goal, the game, where the game all along should have been alignment of capital, direction, and purpose, and delighting the folks using our software.

[4]: Even more ‘fun’ (if you’re into that sort of thing) is that folks will tell you how the slightest deviation from those principles means you aren’t agile, and that somehow all these processes that derive from agile aren’t actually agile (Except theirs, of course). I will concede the folks that hate on SAFe have a bit of a point, however.

[5]: There’s that word. working. Not great. Not amazing. Not “Software you’ll be proud of”. Working software. That’s a low bar, but when you compare it to what came before — software that was produced for months and years but never saw the light of day because it was obsolete before being shipped — it was a godsend.

[6]: Feels weird to reword the tenets of agile for the purposes of this post, but I’m doing so because if I use the same words they use, it would defeat the purpose of trying to improve our understanding of the problems at play.

[7]: we have traded one level of software purgatory for another. Either we spend months building a thing that ultimately is obsolete when it’s shipped, or we spend weeks, ship it, realize it’s broken, fix it just enough to not be apparently broken, and move on once the opportunity cost is greater than the value to us to fix it. We’re still left with a process that disconnects us from the desires of the person using our software, and substituting our own process as a barometer instead of that desire. We’ve disconnected software creation from the mechanisms that determine how much runway we have, and wonder why the plane runs out of gas right after we take off.

[8]: Or find a different game to play. At some point you can’t trim around the edges any more and you have to fundamentally change the type of game you’re playing. If you just keep adding rules on top of rules to address the vagaries, you end up with American football, where you have to have 25-30 $100,000 cameras fixed at ungodly angles to determine whether someone caught a football.

[9]: I can’t tell if that’s a good thing or not that we’ve optimized how we build software for the realities of the industry around us, but I find it depressing that we’ve normalized building crap, but it’s comically rare that we build good software that brings joy to people’s life.

Author’s note: This post is the culmination of a writing process that took place over 5 months in 2023 and up until the time of publishing. I’ve spent so long writing and re-writing and editing it that it may come across as rushed. Ironically, it’s better to get the ideas out there and let them ruminate in the world than to keep working on this one post until it’s polished enough to be what I hoped for it to be when I started writing it. This is the beginning of the conversation, not the end. As an aside, once I upgraded to a 2023 MBP M2, the block editor started behaving again. That’s one way to socialize the losses from ineffective software development: Make your customer pay for better hardware to get a good experience.

Moq adds nagware; Where’s the .NET Foundation in all this?

Moq is a mocking library for .NET Unit Testing (cue the TDD folks reminding us mocks are unnecessary), and it is by far the most widely used mocking library in .NET (475 million downloads vs 87 million for the next largest, NSubstitute). Yesterday, its author released version 4.20.1; which added nagware and a backdoor to Moq, in a bid to drive up paid usages of Moq through ‘Sponsorships’.

This had the predictable result of enraging folks that use Moq, and the change was subsequently removed in version 4.20.2 (released a few hours ago as of this writing), ostensibly because it broke MacOS restore. The author of Moq wants discussion on the topic of sponsership under Github issue #1374.

I’ll get the tactical analysis out of the way first by way of these three points:

  1. This was a backdoor (unintentionally) to supply-chain attacks. The project linked to an obfuscated DLL that provided two functions: a) retrieving its settings from a third-party server, and b) spawning a git process, and sending hashed git email addresses to that same server.
  2. This is also nagware. A message popped up in the build window if the person wasn’t a supporter (by checking hashed email addresses found locally vs. what’s on the server), to tell them to sponsor the project.
  3. This action by the maintainer is a cry for help. They have 475 Million downloads, and their work on Moq can’t support them financially. This is a cruel and unjust world where giving away software that supports Fortune 50 companies would result in poverty if they didn’t keep their full time job. They get to choose between maintaining Moq, or seeing their family.

For the security minded among us, there are any number of ways for this to be exploited, from DNS based attacks to any code vulnerabilities to simply waiting for that domain to expire and taking it over, to any number of attacks that an enterprising malware author would find. This is a textbook opening for a supply-chain attack.

There’s also a betrayal of the social contract involved with OSS software: You don’t rely on closed-source software as a necessary part of ensuring your software does what it says it does, and you don’t surprise the folks that depend on your work.

This is also one of those character-building opportunities for the author and an object lesson to everyone else in the .NET OSS community on what not to do. But more so an object lesson for us that .NET open source is not sustainable. Popular projects become force multipliers for Fortune 50 companies that would otherwise have to spend hundreds of thousands of dollars in salaries recreating the functionality, or spending tens of thousands of dollars on licenses to buy software that does what OSS provides for free.

I won’t spend too much more time on the tactical part because there’s a much more interesting strategic problem sitting here waiting to be dined on: The .NET Foundation and .NET Open Source Software.

I wrote a while back about the .NET Foundation’s fundamental problem, and if the Discussions tab is any indication, the .NET Foundation as a community is dead. 4 discussions in 2022, and 4 in 2023 is not a healthy community. Also, all but one committee stopped meeting (wg-projects, wg-maintainers, wg-marketing, wg-membership, wg-corporate-relations, wg-education have all stopped meeting/posting their minutes/updating agendas) — the wg-outreach team is the only committee to have met recently.

Update: Glenn Watson, .NET Foundation Board Member, had the following response to the above:


Committees are meeting regularly. The thing is I went hunting for meeting details. if I wasn’t getting invites by the foundation I wouldn’t know the times and where and how to join. Something I need to fix. Also we hire a company at the moment to take notes and minutes and I need to discover where that is going. I will get back to you on this one too.

With that update down, back to the original post:

Ok, great, George, so an OSS project is feeling the pressure of producing software for free, and the .NET Foundation has about as much activity as your blog, how does this all connect? Allow me to quote from the .NET Foundation’s own front page:

The .NET Foundation is an independent, non-profit organization established to support an innovative, commercially friendly, open-source ecosystem around the .NET platform.

It appears the problem of supporting an innovative, commercially friendly, open-source ecosystem around the .NET platform is right up their alley. Literally. Now, I have my concerns with the way they wrote commercially friendly, and there’s been no clarification since that blog post of their intentions, so at this point that part is an open question: Who does the .NET Foundation want software to be made commercially friendly for? Is it for maintainers? Is it for Microsoft? Is it for other large companies? Who is the beneficiary of the .NET Foundation’s work? Who does it serve?

The .NET Foundation should be all over the open-source commercial viability problem. We should see leadership from them not only on this Moq issue, but on the state of open-source in .NET. Maintainers aren’t business folks, and they don’t know how to turn something that has 475 million downloads into a viable business, and yet every enterprise that uses .NET more than likely uses Moq. Heck, even the streaming music industry pays better than .NET Open source. If the foundation is foundering, and looking for problems to solve, the viability of open-source is right up there, waiting to be tackled.

The .NET Foundation should be that interest group for open source, enacting policies and frameworks to help OSS maintainers get paid, to build the foundation for what it means to be sustainable for open source.There’s no question, Open source software has to change, and money has to come into the equation. In one possible future, projects join the .NET Foundation, and their maintainers are paid by the .NET Foundation, while corporate members pay thousands to 10s of thousands in dues every year in return for support and commercial licenses to use .NET Foundation projects. Dual-licenses, and an interest group sitting in between that handles the payment backbone of OSS is a possible path forward, but it’s going to take someone like Microsoft leading the way. Do they have that leadership potential in them? Only time will tell. For our part, we should continue to remind the .NET Foundation as to why they exist, and use our soap-boxes to get them to take action on this pressing issue. We won’t have open-source viability until we fix the sustainability issue.

Think ‘experiment’ instead of ‘project’

I ranted a little bit yesterday on killing the word ‘project’, and and I was asked, “well, what should we use instead?”
​

Great question, I would love to tell you.

Projects give off an aura of “beginning, middle, and end”. At some point, this thing will be done. And that’s not true in software. There may be a point where we don’t use it any more (we ‘kill’ it), or it may fail at its purpose, but as long as we’re using it, it’s not done. It’s going to need security updates, framework updates, and at the very least bug fixes.

And so calling software a “project” is a bit like having a kid and calling it an accessory to your lifestyle.

Instead, software that isn’t meant to be a ‘product’ (that is, have the understanding going into it that there will be long term support and a budget) should be called an experiment. Because it is.

We’re not naive enough to believe this IT Modernization project (see, I just did it!) will succeed because we spent millions of dollars and tens of thousands of people hours on it. It’s an experiment. It’s an experiment because Sue in accounting who has been here for the last 25 years basically runs your entire payroll infrastructure even though her title is “Sue in accounting”. Your modernization project will fail if Sue in Accounting isn’t happy, regardless of how you dress up the project. So it’s not a project; it does not have a beginning, middle, or end. It has a beginning, Oh-god-I-hope-we-win-Sue-over, and then support. And if you don’t win Sue in accounting over, you’re going to have a bad day.

So call it an experiment, and don’t let anyone lie to you about the real nature of software: it only succeeds if we successfully sell it to the people who are going to use it.

This post originally appeared in my daily email list on January 13, 2023. I publish emails daily, and put them here when I remember to. If you want to stay up to date, join my mailing list. Details below.