How many Browser tabs do you have open?

Right now I have only one browser window open, but I have 21 tabs open (including the one where I’m writing this).

I couldn’t tell you what’s in those 20 other tabs right now if I tried; and they’re all recent tabs.

If we can have 21 tabs open but can’t remember what’s open in each, why do we expect our codebase to be any different?

That’s one of those little annoyances that Microservices can solve. You’re able to work on a problem and only have to worry about one tab; not twenty-one.

If you’re just spreading the information in one tab across 21, it’s not going to solve anything. The goal is to make each microservice self-contained with what you need, so you never have to switch contexts to get work done.


Isn’t that worth it?

Support Layers in Microservices Topologies

​One thing I mention frequently in the daily emails is the fact that microservices require a lot more operational support and development support than a monolith does.

In a monolith, once you’ve got your CI/CD set up; it’s set up. Production-wise, you only have to worry about your application server and your database server (and any reverse proxy), and the most complex your architecture gets is when it includes a load balancer and a web farm.

All in all, not relatively complex at all.

Now, Microservices are not synonymous with containerization; but generally microservices end up being containerized for ease-of-deployment.  You can also containerize your monolith (which I generally recommend as a forcing function to make it seamless for new developers to get started in your system), and so containers aren’t just a microservices fad, though they work nicely with microservices.

If you think of microservices as self-contained applications that each need deployment, operational support, and scaffolding to make it easy to develop a new microservice, then you start to realize there are lots of repeated problems you have to solve when developing Microservices:

  1. How do we add new microservices in a uniform way so that we don’t have twelve different ways to do logging, or healthchecks, or monitoring, or authentication or authorization?
  2. How do we make scaffolding that makes it easy to have a generated container image for a new microservice that has all the business and industry specific stuff we need? For instance, if you work in the US government space, you’re going to hear two phrases a lot: “STIGged” and “FIPS-140 compliant”. Your industry may have its own terms, but it’s the non-functional part thatevery application needs to have that you’d rather bake in than worry about doing it each time.
  3. How do we make tooling that makes it easy to generate contracts when new microservices are made?
  4. How do we provision new hardware (when in a private data-center) or provision new instances (when in a public cloud setting)?

Susan Fowler talks about these four types of problems in her book Production Ready Microservices​​, which talks about the ways that you get a microservice from development to production in a sustainable and scalable manner.

Susan mentions four ‘support’ layers you have with Microservices that you don’t have in a monolith (or if you have them, they were already solved long ago and you don’t need to worry about it now).

Layer 4: Microservices
Layer 3: Application Platform
Layer 2: Communication
Layer 1: Hardware/Host*

*I’ve modified layer 1 to be Hardware/Host (because like it or not, a docker image is a host, and has its own patch cycle).

For a development organization, here’s the sort of things you typically need to worry about in those layers:

Layer 1: OS updates; OS library updates; the actual hardware (if in a private datacenter); Virtual Instances staying up to date; local Docker registry
Layer 2: Message contracts; event queue infrastructure; scaffolding for generated types; OpenAPI tooling; Thrift/gRPC tooling
Layer 3: (if in .NET): private nuget (package) registry maintenance and tooling; Keeping .NET up to date;  CI/CD for each microservice, development tooling; internal tooling to make development easier; (above scaffolding for generated types can also be in this layer); logging and monitoring for microservices;making the application systemd compatible
Layer 4: tools to generate microservice-specific configurations; SSL certificates for each service (if needed); environment files; and any tooling that we’d need to apply to a specific type of microservice.

If you have these four support layers in place; then a developer simply has to create a new microservice and go; this tooling takes care of the rest.  This can look like all of the scaffolding being generated for them by internal CLIs.

Sounds like a lot, doesn’t it?  It is. But whether you automate it or make your development team do it manually, it still all has to be done, and is a cost to adopting microservices.

EDA+ToC=J

By now you’ve heard of event-driven architecture (EDA): A scheme by which all of the remote calls in the system are replaced by one of four types of asynchronous messages; and the parts of the system only communicate asynchronously.

It’s as if someone told you from now on you must communicate through email instead of walking over to bug someone in the middle of their work.

With that sort of description, I can’t tell if that makes me love EDA more or less.

Anyway, back to our scheme.

Blindly sending emails all the time has its own problem: No one will read thousands of emails in any appreciable time. At that point you may as well just go over and interrupt the person because that would the fastest way to get your answer. As long as they aren’t passed out at their desk from exhaustion. Or maybe they’re in a napping pod. That was — at least in the before times, the most recent addition to startup culture.

But this is not a post about startup culture, it’s about event-driven architecture.

So blindly sending thousands of emails to someone to deal with, and going over and interrupting them every two seconds isn’t going to work either.

So what do we do?

That’s where the theory of constraints (ToC) come in.

Your software system, just like a production manufacturing line, has a natural limit to what each sub-system can do. To put this in concrete terms: Your database can only store items so fast. Your indexer can only index so fast. Your order system can only report so fast.

The theory of constraints provides a method of reasoning and resolving bottlenecks in your system. Unchecked concurrency won’t make your system any faster, and we can use the theory of constraints to help the system run optimally.Identify the system’s constraint(s).Decide how to exploit (make maximum use) of the system’s constraint(s).Subordinate everything else to the above decision(s).Alleviate the system’s constraint(s).If this constraint is no longer a constraint, go back to step 1, but do not allow inertia (“how we’ve always done it”) to cause a system’s constraint

Even though you have an asynchronous and naturally concurrent system; you still have bottlenecks. Your bottlenecks will determine how fast you can actually process data/get things done. Find those bottlenecks and put your resources to resolving that bottleneck. Rate limit/turn off whatever you need to to keep that from being the bottleneck. Fix the bottleneck. Iterate.

Story time: In an Event Driven system I had designed; we elected to use node.js and TypeScript as the technology stack for quasi-custom parsing DSL (It was regex based parsing plus an XML based DSL so customers could add their own — this ability being a core requirement; and we wanted to use existing standards; so regex and XML were it). Node.js is famously single-threaded; and we thought this was a limiting factor when we noticed that in our system, the parsing service had a lot of events waiting on it to finish processing messages. So we focused on speeding up the parsing service; and turned everything else down — it wouldn’t have mattered how fast the rest of the system went, the parsing service was the bottleneck. Ultimately, it turned out that some of the regexes we used in combination with a particular line in our parsing DSL were quadratic. Oops.

Resolving this bottleneck sped up parses considerably, and as a result the whole system sped up.

Blind event-driven architecture failed us; but combining EDA and ToC helped us find joy1.


1: Yes, the title was actually a Game of Thrones reference.

Types of messages in an Event Driven Architecture

An event-driven architecture is based on the idea of message passing as the primary (if not only) means of communication between services and systems. Last time I talked about the anatomy of an event message in an event-driven architecture, and today I want to (briefly) go over the other types of messages in an event-driven system.

Events are temporal verbs: They convey what has happened or is happening.
Commands are imperative verbs: Do this thing. Start this Process.
Queries
are requests for information. “What are the orders for customer ID 123?”
Replies
follow Queries: They give the answer to the query. “Customer 123 is a cheapskate. They haven’t ordered anything.”

A few tips:

  • You can make Queries and replies part of your internal system communication mechanism that happens over the message bus. I generally don’t recommend this — as you’re essentially reproducing HTTP semantics over AMQP, and that’s not necessarily a business you want to be in, but it’s up to your context. Both RabbitMQ and Kafka support this set of semantics.
  • If the resilience of a message bus is not required for your use-case, then I’d suggest using synchronous HTTP requests against your HTTP/REST APIs for queries and replies instead of trying to make everything fit around your message bus.

[Last Week in .NET #54] – Disorderly Sunset

Microsoft sunsets OneNote, only to expand OneNote, and the .NET Compiler has a bit of chaos inside of it. Let’s get to it.

⛔✅ David Fowler, member of the .NET team, writes that “null checking in C# has gotten out of hand”. David’s right, of course, and a follow up tweet in that thread narrows it down to merely three methods to checking for null. Another day, another chance to tap the sign: Just because you can doesn’t mean you should. It’s felt like that ever since C# was de-coupled from the .NET Framework, the language has exploded with new syntax; and yes, while newly divorced people sometimes do go through a sowing phase, you reap what you sow.

👨‍🦯 Adam Lein breaks down the user experience and human centered design problems with Windows 11.

🚫🍆 Announcing Code of Conduct Enforcement Services for member projects!. The .NET Foundation now provides CoC enforcement across all .NET Member projects. If you’re a dick in one place, you’re going to get banned from all the places. Don’t be a dick.

💳 Techbash Tickets are now onsale Appropos of the current delta variant issues, Techbash has also kindly responded to my request for information about cancellation:

If we cannot hold the event due to safety concerns, we’ll work with the Kalahari to handle the event cancellation and refunding as we did in 2020. However, our current plan is to continue to have a safe and fun event for all in October.

📹 Humans of Microsoft S02E01: Abel Wang You may know that ABel Wang passed away recently; but we are lucky enough to live in an age where we can hear his words even now. In this video Abel talks about life, health, and his favorite software project ever.

1️⃣1️⃣Top 11 things you can do to make your app great on Windows 11 This is a good list and it dovetails nicely with the design issues in windows 11 we spoke about previously. We never successfully got Winforms applications to be updated to WPF, and now suddenly we’re expecting three generations of old Windows applications to get updated to Windows 11. So long as software backwards compatibility remains paramount to Microsoft’s business arm, design will suffer.

🏃‍♂️ Install WSL with a single command now avialable in Windows 10 version 2004 and higher Now dropping Windows is just a command away. This reminds me of using Internet Explorer to install Chrome back in the day.

🐦 One thing I missed last week is that Random.Shared is available in .NET 6. Yes, a threadsafe Random API, as opposed to a threadsafe random API.

📃 There’s a List of Features available for all the C# versions; including what’s coming in C# 10 and C# Next and with no hint of irony at all towards the ample ways to check for null in C#, there’s a parameter null checking proposal.

⚡ There’s word that LINQ statements will be twice as fast in .NET 6 than they are in .NET 5. David focuses on performance so I have no reason to doubt his word, and apparently the benchmarks will be coming soon.

🚗 Rider 2021.2 has been released and it now includes Blazor WebAssembly debugging, support for removing redundant suppressions, support for refactorings in source generators, and lots more.

📹 The monthly .NET MAUI Community Toolkit Standup was last week.

🙃 A helpful safety tip: Stick to the Beta channel if you’re on Windows 11 Preview, with the Dev channel you can’t go back.

🧵 Infoworld’s SImmon Bisson talks about project Coyote: a way to unit-test multithreaded asynchronous C# Code You can learn more about Coyote on its project site.

🔓 Some Infosec folks looked into the ‘base’ level security in Windows 365 and were… not impressed. From cleartext password dumps, and making everyone admin, it’s a little embarassing what the out of the box settings are.

🌇 Microsoft to Sunset OneNote for Windows 10, OneNote is the Future That sentence is not a typo. Apparently there were two different applications called OneNote, and now in the future there will be one. Also, Microsoft clarified that they are not building a third application called OneNote. Just ‘evolving’ the current applications.

💥 There seem to be no end to the ways pistachios can kill, suffocation, explosion, and fire, to spoil the lede.

💁‍♂️ If you’re running ASP.NET Core on IIS, make sure you’ve enabled the UriCacheModule. It’s recommended for ASP.NET Core deployments but is not enabled by default. Let’s pour one out for everyone still running ISAPI.

💁‍♂️ Marc Gravell reminds us that not even the compiler in .NET can reliably tell he declaration order of types or members. If your product depends on that being knowable, you’re in for a world of pain. It’s also worth noting that this knowledge is about as inside baseball as it gets; and yet at least one of you has written a hack to deal with it.

🥴 Semver doesn’t mean MAJOR.MINOR.PATCH, it means FAILS.FEATURES.BUGS. No, this has nothing to do with .NET, but it is insightful and funny.

And that’s it for what happened Last Week in .NET. I’m giving a free webinar on August 18, 2021 about Event Driven architecture: Bringing Order to Chaos. If you’re thinking about breaking up your monolith or moving to microservices, this talk is for you.

Anatomy of an Event in Event Driven Architecture

In event-driven architecture, services collaborate and share events: Things that have happened or need to happen in the system1. These events should have a common format, both for ease of creating and maintaining contracts, and for debugging and tracing events in the system.

Here are the parts of an event, and what they’re for:

Timestamp: UTC time of when the event occurred. UTC is important because not all parts of your system can be guaranteed to be in the same timezone, and an ability to order events by absolute time helps to maintain an understanding of what happens and when in the system.

EventId: An identifier that uniquely identifies an event. This should be a globally unique ID (typically a GUID), and generated by the application code, not a centralized database.

CorrelationId: An identifier that correlates all event messages that are sent due to the same event or action. One event in the system can result in 1-n messages being sent.

EventName: The actual name of the event: employee_updated, employee_created, audit_completed.

Sender: The name of the service that sent the message.
SenderIPAddress: The IP address of the service that sent the message

Payload: The actual contents of the event; the structure of object depends on the EventName.

This is where things start to diverge based on your needs.

Authentication and Authorization

If events in your system are ‘trusted’; that is, an event is only raised if the permissions of the system allow it to raise an event, then you don’t have a need to authenticate events being raised by services in your system. This also means that your events are ‘back-end’, that is that no user or non-system actor can ever raise an event.

If you can push security down to the network level, do so. If a non-trusted actor can’t get access to your message bus, all the better. Defense in depth is a good thing as well; so I say all this to say that what you need depends on your context.

If events in your system are not trusted; then you need to include an authentication and possibly authorization properties in your events; including an authentication tokens, claims, and a signature.

Personally I’d recommend slicing up services in such a way that you don’t *need* cross-service authorization; and keeping services aligned along their bounded contexts can help in that endeavor.

Event Sourcing

If you want to pursue event sourcing, then the following properties are useful:

Version: The version of the event raised. This allows for different formats for the Payload property; and for the system to evolve.

ETag: A hash of the payload. Can be used to verify changes or no changes to state for a particular event and payload.

Warning: Don’t version until you have to. Versioning means being able to process multiple formats of messages and keeping multiple same-named classes around at different ‘versions’.

Other Considerations

Being able to generate contracts programmatically can help ensure clients stay up to date with contracts.

If your team decides to use a nuget package of classes as contracts, this can couple all consumers of that nuget package together — if there’s a bug in that package, it will affect all consumers.

All messages having the same structure is critical; both to event sourcing and to saving all events in an event store for auditing purposes.

Designing your messages and consumers to be idempotent (or re-entrant) is important. A consumer should be able to receive the same message multiple times without mutating data or causing side effects.

In another post, I’ll go into authentication and authorization, and when you’d want to pursue that in your messaging architecture and when you wouldn’t.

Footnotes

1: Events aren’t the only type of message in an event-driven system; there are queries, commands, and replies as well. I tend to architect EDAs in such a way that queries and replies happen through synchronous means like HTTP; and Commands and Events happen through an asynchronous message bus; but you should use what’s best for your context.

Why Contexts Matter

​One of the words you’ll see *a lot* in these posts is ​context​.

Context is probably the most important word when you’re running a team, or you’re an architect, or in general you just want to enact change on your team or your organization.

There’s a lot of advice out there about ‘best practices’, and generally that advice lacks ​context.

​In your business, your team, and even in this moment in time, you exist in a particular set of contexts:

Social Context: What is the social structure of your team? Who gets along with whom? Who has lunch together? Who does everyone tolerate but not really ​talk to​?  Who if they left today would make the biggest hole socially?

Financial Context: Has your business given you a raise in the past year?  Did it track inflation?  Was it a 1-2% raise?  Did your company make its sales goals last quarter? Last year?  When’s the last time you asked about the financial health of the company? Do people even do that in your company? Are there all-hands meetings where the financials are brought up? Do they talk about profit or just revenue?  What’s your run rate?  What does it take to get a book in your company? Any book related to your job? What does it take to get a conference approved? A new piece of software?  Is pay a big secret? Do you know what your coworkers make?​ How long can projects last before they’re considered a failure?

Technical Context: How does your team and business feel about the word ​legacy​? Is your software considered ​legacy ​or ​modern​? How do team members feel about making changes to your system? What did the last big project feel like? Did it succeed? fail? What caused it to fail? What helped it succeed?

Cultural Context (can also be called “Political Context”): Who gets ahead and how? Who is rewarded?  Why was the last person fired? Do you know? Does anyone? What’s the turnover rate, voluntary and involuntary? Does the messenger routinely get shot?  How long does it take you to hire? To fire?  Do you have yearly reviews? How are they conducted?  

​Business Context​: What’s your team’s business model? Your company’s?  How does the code you write get translated into value? Is your team considered a profit center or a cost center? (Do you make the business money or cost the business money with each new thing you build?)  If the former, is your value creation axis centered around saving money or making money? How do new initiatives get approved? Who has the easiest time getting new initiatives approved? Who has the hardest?  Quite simply: Who holds the power in your chain of command? Is it the person who is in charge by title? or is there a shadow chain of command? People who have the influence but not the title? Is your boss or your boss’s boss a key decision maker or do they report to the key decision maker?  Does your CTO sit on your company’s board? If not, who does besides your CEO?


Why do all these questions matter?

They matter because whether we realize it or not, all architectural decisions your team makes has to take all of these contexts into account.  Let’s say for the sake of this discussion that you want to ​modernize​ your .NET application. Maybe move it to .NET Core, maybe containerize it, maybe even split it up into microservices.  All of these questions matter, because they help you decide what you ​can​feasibly do​. If a project that last longer than three months is culturally or fiscally shunned, then you’re going to need to find a way to make a splash in 3 months, something that will help the business respond ​positively​ to the changes you want and need to make.  Technical decisions aren’t simply technical decisions. They’re decisions that are made with all these questions in mind, because change — technological change — is a tactic to fulfill a goal or improve one or more of these contexts. If it’s not made with these contexts in mind, it’s busy work at best, and damaging to your organization and reputation at worst.

And most importantly, if you don’t have the pulse of your organization — if you can’t readily answer these questions, then you’re ignoring the part of the organization that ultimately dictates success and failure: the human part.

Organic Design or Planned Design?

Do you want all data Up To Date or The Best We Know right now?
Do you want to Wait or Work?
Do you want to Ask or Know?
​Do you want action Soon or Now?
Do you want action to be Global or Local?
When building new features, do you care more about Velocity or Repeatability?

Microservices design is about tradeoffs, and answering those questions will help you decide whether you want synchronous communication or move to an event-driven architecture.

[Last Week in .NET #53] – Requiem for a use case

🕵️‍♀️ Using Secrets in .NET Core Console Applications Console applications remain one of the least documented parts of the .NET Core experience (compared to ASP.NET), and I’m always happy to share content on that topic. Why are console applications important? If you’re in an event-driven microservices world in .NET, using a Console application to connect to your message queue and receive messages and put them into a database of some sort is an integral part of the work; as are services that respond to events but don’t necessarily expose HTTP APIs.

🔨 Erik J markets his EF Core Power Tools Visual Studio Extension I did not know this existed. I mean, I vaguely had heard of it, but had no idea what EF Core Power Tools would even do. Luckily Erik shared a link to his extension, which according to the download page, lets you Reverse engineer a context and classes from an existing SQL Server Database, has diagramming support, right-click migration support in Visual Studio, and more.

🔌 The .NET Download Site had an outage last week and there is not, and I quote, “There is no workaround using Azure DevOps.”
Can you imagine the protocols Microsoft put in place to push Azure at all times?

👨‍💼 “Mention Azure.”
👨‍💻”But sir, this is an outage on our public website.”
👨‍💼”MENTION AZURE.”

🦜 David Fowler tweets about some new additions to ASP.NET in C# 10 and .NET Core 6: Default Global Usings, File Scoped Namespaces, and a “minimal” Hosting API.

🐈 Nicole Express blogs about the cause and fix for a long standing ALF Bug. Yes. That adorable animatronic 80s TV star that had its own movie and video game, and liked to eat cats.

🧟‍♂️ What if Github Copilot worked like a real programmer Not listed: Copilot engaging in a flame war over whether The Last Jedi was the worst Star Wars movie ever made, and reminding other programmers that programming is a meritocracy, while failing to see how self-serving that statement is. This is satire, of course. The Last Jedi is arguably the best Star Wars movie ever made.

🍪 If You use Chocolately, a fresh install of Visual Studio can inadvertantly nuke your nuget package source configuration oops.

🐢 How To: Use Azure AD Powershell to Work With Extension Properties (User Attributes) This blog post does what it says on the tin, but for you the use case here is you need to use Powershell to retrieve and set extension properties from Azure AD. If you know what that sentence means, please reach out and let me know. Thanks.

🐿 On July 27th, 1993, Windows NT 3.1 was released. I know it wasn’t NT, but Windows 3.1 was glorious, but only for me because that’s the first OS I played Chip’s Challenge on. Also Chip’s Challenge and its sequel is available on Steam. You’re welcome.

❤️ Brent Ozar has released an update to his First Responder and Consultant’s Toolkit. Ok, naming aside. If you use SQL Server, and you’re a DBA or even a C# developer that needs to interact with Sql Server,you will want to download, install, and run these scripts. They’re very useful in understanding performance issues in SQL Server, in understanding if your table structure and indexes are optimal, and helping you resolve emergent issues with SQL Server. These scripts should be in every team’s toolkit that uses SQL Server.

🍭David Lee Roth retells the famous “brown M&Ms” story that Van Halen used in its Rider. The reason they used it is not what you think. It’s well worth your time to listen to. Thanks to @textfiles on Twitter for sharing a link to this.

🆙 Visual Studio 2022 will not be able to build .NET Applications that target anything in .NET 4 before .NET 4.5.2. The writing is on the wall: Upgrade your framework, folks.

📢 Dapr v1.3 has been released and this minor update includes several minor updates but still no explanation of why drop-in-replacement architecture is such a ‘win’. Developing to the Lowest Common Denominator gets you… The most boring and undifferentiated features of all of your options.

✈️ I asked the CEO of Jetbrains for an update to the Solarwinds/Team city mess and he obliged. If you’re new to this, the NYTimes ran an article that claimed — anonymously, of course — that TeamCity was why the Solarwinds attack happened. Because of that, some companies and organizations have dropped using Jetbrains products. We hadn’t heard from the CEO of Jetbrains since their ‘update’ several months ago, and I asked them to let us know if anything ahd changed. They obliged by reinforcing that the NYTimes article probably shouldn’t have been published in the first place.

🔈 .NET Conf Call for Speakers is Open I have submitted a session that will undoubtedly be turned down because I don’t mention Azure in the abstract at all.

🤯 C# 10 will also support var as a lambda expression initializer, and I’ve hit the point where I’m now souring on var. I have no idea what that type is or well be, and I can’t see how that’s a good thing. @ me @gortok on twitter if you think I’m wrong, and why.

💸 Marten, the Generic Host Builder in .Net Core, and why this could be the golden age for OSS in .Net. Jeremy Miller spells out why the addition of Generic Host Builder has made his life better as an OSS Maintainer. Personally I think the problems with OSS in .NET are mostly commercial interference by Microsoft; and I’m not so sure we can fix that.

😥 Abel Wang passed away last week He was a Principal Program Manager, and Technical Assistant to the CTO of Azure. Take a moment and read the accompanying link to learn more about Abel and his life.

The virtues of Autonomy

Pretty often when teams start out with microservices; they split up a monolith and make the intra-monolith calls into remote RPC calls. Or, in every day terms: They put an HTTP API over those old function calls and now call that instead.

Why would anyone do that? What do you gain?

You’re still coupled to that service being available.

You don’t have request autonomy.

You’ve added extra network latency and another point of failure.

Sure you can scale that new service; but the same problems remain.


That’s one of the reasons why if you’re going to pursue microservices; you should pursue an architecture that allows each service to stay autonomous.


Event Driven Architecture does just that.

I’m hosting a webinar on August 18, 2021 at 12pm EDT titled “Event Driven Systems: Bringing Order to Chaos” that helps you understand what Event-Driven means, and how it’s a good choice for microservices.