The Root Cause of Microservices Disasters

There’s a blog post out about Disasters I’ve Seen in the Microservices World, and the OP is right, those are disasters. It seems easy to avoid the disasters, right? You can look at all those situations and think “whew, that won’t happen to me”, and they may not — if you have a plan.

How many times in tech have you been in a ready-fire-aim situation?

Let me help your recollection.

How many times in tech have you heard the phrase, “We thought it would be easy?”

That phrase is basically its own punchline.

The majority of programmers have never dealt with a distributed system, and the majority of those that have have treated it like it’s RPC. Couple that with a lack of planning and you have a recipe for…well… disaster.

The root cause of microservices disasters is not understanding what problems microservices solve and not planning for your move to microservices.

Questions like:

  1. What will Microservices mean for my team structure? What will my team structure mean for microservices?
  2. Do we even need microservices?
  3. How should our microservices communicate?
  4. How should our services be broken up?
  5. What organizational changes do I need to make to support moving to microservices?

There’s a lot you need to know before deciding to move to Microservices, so I came up with a Free email course to help you answer these questions. Just enter your email and subscribe below.

Size is relative: Microservices Edition

​One of the go-to wars around microservices is how small they should be (I have beef with this beef, but that’s another topic for another day).

Some people say they should be no bigger than the code that fits on your screen.

Some people say they should be no bigger than what’s necessary to encompass their reason for existing.

Some people say they should encompass a ‘bounded context’

Some people say you should be able to rewrite a microservice as quickly as you could find and fix a bug in it.

They’re all right, and they’re all wrong.

How big or how small ​a service ​should be​ is relative to your comfort level.

If your monolith is 1MM lines of code, anything 10,000 lines or less is going to be small.

If your monolith is small (50,00 lines of code), then 10,000 lines of code isn’t very ‘micro’.

If you have five people on your team, having 10,000 lines of code per service (with 5 services) seems reasonable.

If you have fifty-five people on your team, then 10,000 lines of code services (5 services) gives you a lot of collaboration points and a higher potential for merge conflicts.

Micro is relative.  How you size your services, depends on a lot of context that ‘some people’ don’t have.  Don’t worry about how others size their services, do what makes sense for your context, your business, and your team.

And because I can’t resist, here’s an image of all the sci-fi ships sized, relative to one another

[Last Week in .NET #45] – A deal with the censorship devil

It’s a light week this week; everyone is coming down from Build. If you missed that, check out last last week’s newsletter. Now on to what happened Last week in .NET.

🔧 Jared Parsons, member of the Roslyn core team, talks about string vs. String. That is, for those of you listening to this instead of reading it, the keyword string vs. the class String. As it turns out, they’re not the same thing. There is also a special circle of hell for people who override String. @ me on Twitter @gortok if you think I’m wrong about this.

🤗 Not about .NET but relevant to our interests, Michele Hansen’s preorder for “Deploy Empathy” is open. Michele is the founder of https://geocod.io, which is, as the name says, a geocoding API. She does a lot of customer interviews for Geocodio, and previously she was a product manager for The Motley Fool, where she — you guessed it — did a lot of customer interviews. Anyway, she’s written a book (and she has a newsletter!) about customer interviews that will give you the feedback that you need for your product or service. I don’t do sponsored content here, and if you work on a product or are a consultant trying to sell a service, you need to read this book. Periodt. Benefits of the preorder is you get rough drafts of the book. Seriously, buy it.

This is one of the best produced virtual keynotes I’ve seen ever Scott Hanselman “and friends” bring you a Build keynote unlike any other. I mentioned this last week, but it’s worth noting again. Watch it. It’s that good.

🙋‍♂️ Raymond Chen talks about Arm32 If this is your introduction to Raymond Chen, you’re one of today’s lucky 10,000. Feel free to peruse his back catalog and be amazed and entertained for thousands of hours. Today he talks about Windows and Arm32.

🤑😈 Microsoft is partnering with Morgan Stanley to provide reference cloud architectures for highly regulated industries (like the financial industry). This is akin to Las Vegas partnering with Satan, but I get it. This is corporate synergy.

C# 9’s blazor ‘colorization’ and appropriate C# 9 syntax highlight and documentation is live If this sentence is confusing to you I’d like to point out I present the links; I do not vet them for sanity.

🎨 Paint.NET Is smackdab in the middle of its migration to .NET Core and some parts are already live. If you aren’t aware of Paint.NET. It’s… Paint. In .NET. That’s it, that’s the hook. All joking aside, it’s a rather wonderful paint program and it just happens to be written in .NET — now .NET Core.

🌉 Microsoft.IO.RecyclableMemoryStream 2.1.0 is released Could someone explain to me how a .NET 4.6.2 targeted application can now use Span<T>? email me at george+whatthefuckisthisshit@georgestocker.com.

Microsoft wants to be twitter’s main character for a day by censoring the “Tienanmen Square Tank Man” image on the anniversary of the Tienanmen Square massacre. rubs head with hands… Do you see how this is bad, Microsoft? Do you? I can’t rub a company’s nose in their own mess, but I’d sure like to.

And that’s it for what happened last week in .NET. Tip your service staff, and tune in next week.

Questions to ask before pursuing Kubernetes

Today’s post brought to you by an innocent sounding question in the Rands Leadership Slack (paraphrased):

I’m joining a company as their first Devops person and their infra needs a serious upgrade. The codebase is a monolith, but they want to pursue microservices, and if growth keeps on its path, that seems likely. I think I’m going to be allowed to dictate a better setup, but I’m currently having an existential crisis between going with ECS and K8s. I realize this is a huge question, but all things being equal: if you had the chance to start fresh, would you go towards K8s or ECS?​

#devops channel, Rands Leadership Slack

I felt like that “step on the brakes” TikTok meme when I read this (you can google that, there’s no way I’m linking to it). This person is well meaning but it’s a question without necessary context — and a question I’ve seen asked by people who ​haven’t figured out the outcome and strategy they want in the first place!

As an aside, Kubernetes seems to be all the rage these days; and that’s OK. Beanie Babies were once cool too.  But, much like Beanie Baby mania, some of the hype precedes the value, and let’s be clear: Kubernetes is not an outcome or strategy. ​It’s a tactic​.

Let me explain.

When someone says, “Hey, you should adopt Kubernetes”, that’s skipping a few crucial steps.  LIke, “What would buying all these beanie babies get me?”

Or 

“Why do I even feel the need to buy beanie babies?”


So let’s start there.


What pain-points are you having with your current architecture and topology?  Is it legitimate scaling issues? Is the database server experiencing timeouts due to high load? Are you writing the same data to multiple places and finding that you’re clogging up the pipes, as it were? Are users experiencing latency or are you having operational issues?  Can your system handle the scale you need it to?

The answers to those questions will give you the idea of what outcomes you want to fix:

  • Ensuring the database can handle OLAP and OLTP transactions without requiring a complete system rewrite
  • Ensuring the database doesn’t timeout
  • Improving the ability to increase resiliency and not lose data during high load times
  • Ensuring the system can scale to 10x transactions per hour due to expected future load

Keep in mind, none of those outcomes specifies beanie babies or Kubernetes.  Those are desired ​outcomes​.

Next is strategy. What strategy will we use to achieve those outcomes?

Well, each outcome can be solved different ways; only one of those requires buying beanie babies (using Kubernetes).

For the Database; we could implement CQRS; we could add ETL process/data warehousing operations; we could improve the performance of our OLTP transactions so they aren’t blocking OLAP; we could implement a queuing mechanism, etc.

Break the system out into services and have those services communicate with each other through an event-driven architecture.

Those are strategies: they’re ways to solve the problem in an overarching fashion.  ​Strategies don’t include implementations​.

Next are tactics:

What tactics can we use to fulfill the chosen strategy?

That’s where Kubernetes comes in. It’s a tactic to a chosen strategy where your chosen strategy is: ​I want to implement micro-services but I don’t want to use a vendor-specific solution​to orchestrate them​. 

Kinda feels like a self-licking ice cream cone when I put it like that, but alas.

So the next time you hear someone say, “We should adopt Kubernetes”, have them put on the brakes and talk you through the pain-points first, then the outcomes they want, ​then​ the strategy, and then if all of that works out, you can have the Kubernetes discussion.

Before deciding that you need Kubernetes, let’s talk about where you’re going, and what you want.  What ​outcome​ do you want? What strategies are on the table? How will they achieve that outcome? How can they be measured?

Once you’ve done that work, then we can talk about which beanie babies you should buy.

Why “Technical Debt” should be narrowly focused

My post What Technical Debt is and Isn’t got a lot of feedback, and today we’re going to talk about why my definition for Technical Debt is so narrow, and why yours should be too.

The feedback encompassed a few categories, “Future change”, and “Bad Decisions”. First, on future Change:

Why having to change in the future is Technical Debt:

I’ve linked to the tweet above and from there you can see the whole thread; but in sum: New technology causes you have to rearchitect and rethink your system in order ot catch up with a competitor.

That’s not technical debt. That’s the normal course of business.

As I’ve said before, Technical Debt is a poor metaphor; but the idea is the same: an action you take that causes you to go faster now at the expense of later.

Technical debt is not a lack of omniscience. By making technical debt about something that happened that you couldn’t have foreseen, you’re encountering (at least) two problems:

  1. The ability to shrug off actual business competition as technical debt. “Oh, that’s technical debt.” as if you and your team made a purposeful decision. You didn’t, the world changed! Think about it this way, when Cars came along, the people that sold horses didn’t do anything wrong that caused them to go out of business, The world changed! In this case, there is no elusive ‘debt’ to blame for this; it’s just a function of change. How you and your team respond to it is important; but it’s critically important not to lump it in with shortcuts you’ve taken that caused this to occur.
  2. You end up lumping in things you can’t control (the world changing) with things you can control (how you build your tech stack). By labeling things outside of your control as Techdebt, you’re just putting a target on your back, as if to say “Well if we hadn’t taken shortcut x this wouldn’t have happened”, when that’s not the case. No, companies probably shouldn’t operate according to blame, but by mischaracterizing what technical debt is, you’re giving ammunition for someone to blame you, wash their hands of it, and call it a day.

The second category of feedback is that “Technical Debt” == “Bad choice”:

And once again, I’ll go to the well. Technical Debt is not the same thing as making bad choices. “Bad choice” is an outcome. It’s not something you can know at the time (presumably if you did know you wouldn’t have made it). Making bad choices is making bad choices. That’s trainable and that’s something that can be managed by experience. The solution for preventing bad technical choices is different than it is for preventing Technical Debt.

The cause of bad choices can be: politics, bad hiring practices, not understanding the technology in use, and internal culture.

Those have their own solutions, like focusing not on individual ‘getting ahead’ but on providing team success towards a specific outcome, improving hiring practices through training, having a mandate that technology choices are vetted or known before they’re used, and taking steps to identify how the internal culture results in bad decisions and working to improve that.

With Technical debt, the team made a decision at a certain point in time to purposefully take shortcut x to produce outcome y faster. It was a conscious decision, made with intent.

This is important to note because on the one hand, the company or team is owning the responsibility on the choice (technical debt), and on the other hand, the company itself is the problem and may not own the responsibility for causing the problem. As the saying goes, we have seen the enemy, and he is us.

Or, think of it this way: “Yes, we purposefully chose to do X and that got us to our short term goal faster but now we need to re-architect X to meet goal y”.

vs.

“We didn’t know what we were doing so we chose X because it seemed popular”.

How do each of those conversations speak to the competency of the team? Does the latter speak to this amorphous ‘technical debt’ that exists that we seemingly can’t do anything about? No.

Bottom Line

For your company and for your technology organization, it’s important to separate out the decisions made because you didn’t know better from the decisions where you did know better, but did it anyway. It’s not just for accountability, it’s to improve your organization and your team. By throwing all decisions that you don’t like into a pile called “Technical Debt”, you lose the ability to separate out the instances where you can do better, vs instances where you couldn’t have done anything differently, or more importantly, anyone else in your shoes wouldn’t have done anything differently. That’s important, especially if you want to keep wearing your shoes at your company.

Clear is Kind

I’ve been reading Dare to Lead by Brené Brown​ and one of the chapters has this as its title: ​Clear is Kind​. 

The full quote is Clear is Kind, Unclear is Unkind.

There’s so much here but I’m going to focus on one aspect of this mantra.

What do your team’s ​Architectural Decision Records​ look like?  It doesn’t ​have​ to be your ADRs, it can be any record of a decision, whether it’s minutes, emails, design decisions, etc, but specifically: Are they clear as to why the decision is being made? Is there a hidden subtext to them? A context that only some people will have? A ‘public’ reason and a ‘private’ reason?

It might, and when I first started using ADRs to help our team understand why certain architectural decisions were being made, there was a bit of pushback from above when I included these hidden contexts in the document itself.

But, the contexts helped us as a team. They helped everyone understand the non-technical forces behind the decision, and more importantly, they built trust. By clearly stating the parts that may have been uncomfortable but important to communicate, it helped the team have the full context behind decisions. “We’re doing this because executives want it done, and while we view it as a distraction from <larger goal everyone agreed to>, the people who sign the checks don’t see it as a distraction, so in order to make it happen we need to make changes x, y, and z.” There are other benefits to writing down the reasons that may not seem apparent, but it really all comes down to the quote from Dare to Lead:

Clear is kind. Unclear is Unkind.

What Technical Debt is and isn’t

​Technical debt. In tech, I’m not sure there are two words that are more confusing than technical debt. It means something different to everyone.

It’s important to understand what technical debt is and isn’t, because some day, some day soon, you’re going to encounter misalignment on what it means in your own teams, and there, as they say, be dragons.

Tech Debt is, at its heart, the natural consequence of shortcuts taken ​purposefully​.

Tech debt is not:

  • bugs
  • design flaws
  • design choice tradeoffs
  • unknown requirements being realized
  • a way to go further (it’ll let you go faster, but not further)
  • revolving debt

All of the above exist, of course, but they aren’t ​technical debt​.  The difference is that technical debt is a ​purposeful design choice designed to get to <goal> faster by taking a shortcut​. ​There are lots of ways of going faster without taking shortcuts, but teams rely on technical debt because it’s easy to accrue (this may be the only sense in which it is like credit card debt). ​

Why does this matter?

For all the pedantry developers have, we sometimes are loose with our words. Technical debt is one such phrase that’s been used loosely to mean all of the above, and it’s caused quite a bit of consternation across the industry because of that.   CEOs hear ‘debt’ and think something they can just pay the interest on forever on without paying down the principal; and technical debt doesn’t work that way (that’s primarily why ‘debt’ is a bad metaphor).  Technical Debt is not credit card debt.  At some point in your company’s life, you will pay off the debt.  That time may be never, or it may be tomorrow.  But the most insidious thing about technical debt is once you realize you have to pay it off, it’s often politically easier to rewrite the application instead (even though that’s generally far more expensive than fixing the debt).  


How does your team use the term “Technical Debt”? Reply and let me know.

[Last Week in .NET #42] – Barn Door Security

🔧 Dave a Brock writes on how to use Configuration with C# 9 Top Level Programs One of the nicer features of C# 9 was pulling out the ceremony of the Main method. Dave uses this blog post to show how you can use configuration in this new world of no Main method. Now if only there weren’t years of documentation showing varying ways to use configuration for varying versions of .NET Core.

📔 There’s an Adobe Reader 0day vulnerability that’s been exploited in the wild this is part of CVE-2021-28550, and as usual patch when you can, as soon as you can.

💣 Fragattacks.com documents “Fragmentation and aggregation attacks” against Wifi. It’s shockingly clear from the website name that no one involved ever read about the Vietnam war.

🌼 CVE-2021-31204 is an Elevation of Privilege Vulnerability that affects Single File Deployment applications on Mac and Linux and the latest patch fixes this vulnerability; so again, patch your systems if you’re on .NET Core 5.05 or lower, or .NET Core 3.1.14 or lower.

Speaking of that latest patch,

📢 .NET 5.0.6 has been released and includes the CVE fix I mentioned before as well as a smattering of bug fixes.

📢 .NET Core 3.1.15 has been released and has that CVE fix and some SignalR fixes, among others.

📢 Visual Studio 2019 Version 16.9.5 has been released and this update includes a lot of estoric sounding stuff that you’d proably not even realize was an issue. What you would probably notice is that this version now includes Xcode 12.5 support. This version also fixes the aforementioned CVE as well as CVE-2021-27068 which is a Remote Code Execution vulnerability that could affect you if you use Python.exe in a scripts subfolder. If you do use Python with Visual Studio I’d like to point out that you’re rarer than an honest politician.

🏴‍☠️ [Microsoft] Office based malware is “one of the biggest threats to companies” and yet it seemingly gets very little attention from Microsoft on how to mitigate it. Instead of making a better zipper, Microsoft chooses to tattoo “Remember to close the barn door” on people’s hands.

📈 Build is May 25-27, 2021 register now to hear three days of Azure Marketing KPIs being realized.

📢 Speaking of Microsoft’s Marketing KPIs which, Azure Static Web Apps is now GA. If you have a static website, and you aren’t enamored by the plethora of other possibilities for static site generation, to include Hugo, Ghost, Netily, Github pages, you now have… Azure. The least cool (and probably most corporate) option.

FileStream operations are getting faster in .NET 6 to the tune of 2.5 times faster reading a 1MB file, and writing is 5.5 times faster. If you’re an allocation junkie, they drop in .NET 6 from 39Kb to 192 bytes. For all you corporate behemoths out there that have corned your market, it appears that blowing everything up and starting over does have some perks.

🛅 Microsoft is shutting down its Azure Blockchain Service which was abbreviated “BaaS”, which I maintain stands for “Bullshit as a Service”.

📢 TypeScript 4.3 RC is out and this is your periodic reminder that TypeScript — even though it transpiles down to JavaScript and uses NPM — does not, I say again, does not support SemVer, so every release is potentially a breaking release. This release is no different, so plan accordingly. Another note, they did not drop this release with a version suffix on “Release Candidate”, reminding us that Programmers are nothing if not optimistic.

📢 Visual Studio 16.10 Preview 3 has been released and the big note here is that the compiler is now “C++20 feature-complete”. I’ve never actually seen a masochist in the wild; but I have to believe someone that still uses C++ qualifies. For the rest of us, There are improvements in MSBuild based code-bases. I have no idea what that means but if it affects you, you probably do.

🎁 try-convert v0.7.226301 has been released If you want to port .NET Framework projects to .NET Core, try-convert is your huckleberry. Also, holy cow does Microsoft’s versioning vary among teams.

🏫 Let’s Learn .NET: Accessibility is happening on May 21, 2021. This big note here is that not only will you learn more about accessilbility in general and using ASP.NET Core, you’ll also learn how to improve Accessibility in Xamarin.

That’s it for what happened Last Week in .NET, Thank you, and I’ll see you next week.

[Last Week in .NET #40] – Pour one out for .NET Framework 4.6.1

.NET Framework 4.5.2, 4.6, 4.6.1 will reach End of Support on April 26, 2022 At least, that’s the word right now. Governments around the world are still using Windows XP, so it’s not like this is a firm ‘end of support’.

🤡 Basecamp lost a third of its employees after a controversional series of blog posts last week A CEO couldn’t destroy their company’s reputation any faster if they tried. This is truly impressive in the depth and breadth of DHH and Jason Fried’s stupidity here.

🎁 The Developers @ Redhat blog has a blog post titled “Some more C# 9” Lots of goodies in here that people who write unsafe code will want to know about. And if someone shared this link with you, they’re saying your code is unsafe. Something to think about.

🔎 .NET Interactive video up for watching Cecil Philip and crew take you through what .NET Interactive is, and how to use it.

📅 .NET Days (produced by Jetbrains) is May 11-12, 2021 No need to attend if you can’t — the videos will be available online after.

📢 Akka.NET 1.4.19 has been released This maintenance release, includes (quote)

“Akka.NET v1.4.19 is a substantial release that includes a number of critical Akka.Cluster fixes, baseline Akka.NET performance improvements, and entirely new dispatcher that has shown to improve performance when used across all of the major actor groups that run both inside the /user hierarchy and the /system actor hierarchy as well.”

🛑 If You haven’t updated Windows 10 in the last week, maybe wait a bit longer? Looks like last tuesday’s update can cause your system to … misbehave. Looks like the problem centers around WSL2.

👨‍💻 The “State of .NET” on Survey is done, and the results are… not great 92% of respondents were dudes. That’s embarassing.

🌨 Easily build real-time apps with WebSockets and Azure Web PubSub—now in preview Being a cloud provider means having terrible naming

📅 Microsoft’s Build (virtual) conference is May 25th-May27th, 2021. I don’t see conferences like this going back to in person. The networking is the real value (and the free stuff Microsoft gives away).

If you enjoyed this, please subscribe to the newsletter. If you hated this, please subscribe an enemy (or two) to the newsletter. Oh, and there’s also a podcast version, if emojis aren’t your jam.

[Last Week in .NET #39] – Microsoft’s MVP Program has a new requirement: Shilling

Not much in the way of releases last week, but lots of shenanigans. I spelled that word on the first try which matters not a whit to anyone else but I’m proud of myself. The shenanigans themselves are an age old story: Big Corporation finds feeble consumers, and exploits them.

🤑Microsoft pushes MVP Influencers to spruik Azure in Lead up to AWS: Reinvent. That’s the headline, here’s the story: Microsoft wants the community MVPs to shill for SQL Server on Azure and claim it’s cheaper and better, when the fact that SQL Server is more expensive on AWS is due to Microsoft’s own licensing model. When an MVP called this out as the bad behavior it is, Microsoft removed them from the MVP Program (for violating an NDA that conveniently hides terrible choices on Microsoft’s part)

Microsoft seems to want MVPs to shill for Microsoft, forgetting that any of the trust these MVPs have is because they don’t shill for Microsoft.

🚚Meet the .NET Upgrade Assistant, Your .NET 5 Moving Company In this blog post, Dave Brock from Telerik shows you what it takes to move from Framework to .NET 5 (Core), with the help of this Upgrade Assistant. Coffee not included.

🌊‘Agile’ F-35 fighter software dev techniques failed to speed up supersonic jet deliveries In other news, water is wet, and Government agency buys into marketing hype and complains when they were hoodwinked.

🕹There’s a series on building Tetris in Blazor A fun way to learn blazor, no doubt.

📢Visual Studio 2022 is announced and it includes making Visual Studio 64-bit. A lot can change in 10 years, especially when the world changes around you.

Update existing projects to the latest release of Project Reunion. The latest stable release of Project Reunion is 0.5.5, and as it says on the tin this blog post shows you how to upgrade.

💰Octopus Deploy raises 172.5 Million in venture capital. The 500K on the end seems like an odd number, doesn’t it? I’m not saying the founder needs a little “I’ve been doing this for way too long without a bonus” money, but…

🏬Microsoft is building a new app store for Windows 10 in major revitalization effort. Microsoft is, at its heart, an enterprise software company. It’s the only way to explain why so many of their consumer efforts have failed. To have a successful app store, you have to give developers a compelling reason to write for it and for consumers to adopt it. As of yet, neither has happened with desktop App stores.

💸NetEscapades.AspNetCore.SecurityHeaders 0.14.0 has been released and it supports opting out of FLoC. FloC is a Google’s replacement for Third party cookies. You know Google, right? The Ad giant and owner of the web browser with 90% market share? Neither of those facts say that Google “has the best interests of its users at heart”.

🐧Windows now supports Linux GUI Apps. Somewhere the founder of Lindows is crying.

🔚David Fowler shows off how small ASP.NET Endpoints will be in the future A svelte 7 lines to get an endpoint. Of course, there’s no Authentication, Authorization, or any of the database connection code, but still. 7. lines.

You can run and debug AWS Lambda from Jetbrain’s Rider. I have nothing snarky to say here. This is just cool.

🐣Intro to Uno Platform, Intelligent Virtual Agents and Cloudflare These three topics are brought to you by scattergories and the Cape Town MS Developer User Group.

💨EF Core is now at 93.5% speed of Dapper. Dapper is well known enough for Microsoft to compare to, but not backed by enough money for Microsoft Legal to care enough to change the name of “Dapr” to something that doesn’t conflict.