I think this is something incredibly useful in life, to absorb information that is useful and discard what is not. While simple in theory, it’s learning to determine the difference between what is useful and what is not that is the difficult part.

Take for instance some examples:

Source Control

I have worked with cvs, svn (subversion), tfs, hg (mercurial) and git.

With subversion, we would often have large repositories using nested folders to separate projects, this can work fine with the older versions subversions indexing at a folder level. However with mercurial or git. We would create smaller more isolated repositories. I guess the svn was often the monolith, and git the micro version.

We would have sourceforge and codeplex hosting open source codebases, now it’s all github.

Often this is decided by the architects, software developers knowledge of the languages, the platforms used within the business etc, all fairly common sense stuff.

I have worked on a lot of different platforms over the years, and find that they are often suited to different purposes, despite the fact you can still acheive results with them. I often see really bizaare decisions made that come back to bite later on.

Here I will go over 2 very popular and widely used general server side platforms. Some of the pros and cons for each, and what factors truly matter for scalable robust software applications.

Components of platforms

• The language, syntax, design.
• The platform, runtimes, os support
• Runtime Performance
• Tooling such as IDE’s, Servers, auxillary tools
• Longevity of design
• Integration with third parties
• Package Management
• Database tooling and support
• Data Manipulation, analysis, reporting.

C# .NET 6

• Static strongly typed compiled language
• Open source cross platform
• Lightweight, highly performant, multithreaded 32/64 bit application runtimes.

Tooling

• Best in class tooling, Visual Studio 2022 is a complete IDE for building web, desktop, mobile, blazor web assembly applications
• LINQPad

Package Manage

But sometimes there are ones which are well…… damn…. I have seen quite a few interesting pieces of the years. Though ones that stick out.

Easy login

Many years ago when I was brought on to do some PHP maintenance for a high profile telecommunications accessories companies e-commerce website, I found a couple of gotcha’s from the previous developers. One of them was in performing logins, now on this particular website it was using the equivalent of :

SELECT * FROM users WHERE 'username'='[username-from-form]' and password='[password-from-form]'

Notice the [password-from-form] . That was literally the value entered directly in the form, no sanitization, or validation was in place at the time.

Now this was around 2004 so things like SQL injection and the like were not so prevalent. Anyway all it would take to login as any user (including the admin user with admin priviledges) was to enter a nice single asterisk * in the login form to produce the following.

SELECT * FROM users WHERE 'username'='[username-from-form]' and password='*'

Now ‘*’ would mean match anything for that field!.

And voila now you could place nice orders, change orders, view and edit user details, and a variety of other cool things.

I was very suprised to see a high profile company to have severe security issues such as this….. and yet there it was.

Novel Database record update process.

WARNING: Example of how NOT to perform updates

Working on a .NET Core / Blazor Server project and found a novel (by whoever created it) way to update records in a database using EF Core.

Now the client Blazor app was sending up a collection of items to update from a grid edit table. Not just a single row, rather the entire grid of rows. Odd, ok now the database update code went like this.

1. Delete all rows in the table. (yep all rows)
2. And add every single row back in from the collection sent from client, as the client had the entire list of rows from the table.

Voila!!! Database records updated…… lol.

12/02/2023

A few days ago, was looking into how to convert a large .NET solution to .NET 7.

So

1. define the problem
2. come up with ideas to solve the problem
3. solve the problem

Important :

• Framework for the most part refers to TargetFramework spec in .csproj files. not the actual .NET Framework from years past before .NET Core except in the case where a specific .NET Framework vx.x.x is mentioned.

Warning

This article describes an approach to solve a programming problem, it may seem overly verbose, and is not focused so much around a particular code or language, rather how to approach the problem-solving aspect of it.

In my day to day work for most programming related concerns, I don’t draw up flow-charts and pseudo-code (as it usually fits in the head) unless it’s quite a complex problem that I believe others may also have difficulty with. Or when doing architectural work related to processes.

Update: 18/02/2023

As Of Feb 15th 2023, Microsoft Released a .NET upgrade assistant that can be used inside of Visual Studio to update projects to the latest version of .NET. I haven’t tried this yet, however one of the issues of doing this inside VS is when dealing with large solutions VS can be quite slow, especially with resharper.

Visual Studio Gets a .NET Upgrade Assistant

The Problem

The goal is to convert all projects within a large solution to .NET 7.

By large, I mean 350 projects in a single solution

• 32 Azure Functions - 618 source files
• 9 Web applications, a couple with react, some with blazor - 667 source files
• 28 Console applications - 227 src files
• 89 Test projects - 1217 source files

With varying versions of .NET and pacakages

• .NET Standard2.0/2.1
• .NETCore 2.1/2.2/3.1/5
• Lots of different versions of nuget packages across projects

There was previously

• .NET Framework 471

Some Considerations:

• When updating any project, Visual Studio will ask to reload/refresh etc
• When updating framework versions, it’s quite likely that nuget package versions will need to be updated. This is especially true with Microsoft specific packages that follow the framework version. And also third-party libs that need updating to support newer framework versions.

Now, I have refactored many solutions before, across multiple solutions, upgraded from .NET framework to .net core when it became useful (ie .NET Core 2.1).

The Candidates for solutions

1. Use visual studio to change targetframeworks
2. Use resharper to try and change targetframeworks
3. Directly modify .csproj files to change target frameworks with an editor (ie vs code)

Issues recognized with these approaches

1. Visual studio targetframework
   1. having to go into every single project at a time
   2. with every change made to a project file, visual studio refreshes itself, this can be slow across 10’s or in this case hundreds of projects
   3. Quickly determined this would not work after 30 seconds.
2. Use resharper
   1. not sure if resharper has this capability, but even if so, is still affected by VS refresh after update, and resource intensive, slow across the entire solution.
   2. Deemed same issues as (1)
3. Directly modify .csproj files, I’ve done this before multiple times, it seems like the best solution (directly modifying .csproj). Now there are multiple ways to do this.
   1. Simple find/replace across all .csproj files. Ie find: TargetFramework>netstandard2.0</TargetFramework> replace with : TargetFramework>net7.0</TargetFramework> Repeat with each version….. this however is error-prone and tedious
4. Use the new Visual Studio .NET Upgrade Assistant from .NET Upgrade Assistant

(1) and (2) were already out due to being painfully tedious or slow, (4) runs inside VS so will suffer from any VS slow-down. (3) was the closest, so a variation of (3) to reduce human-error (mine or anyone else)

1. Write scripts, or a console app or similar to automate the process.

Ideally (5) is the conclusion to naturally come to, its a desired state for a software dev who likes to create things. However it must make viable sense to apply it. If there is a simpler solution that is much quicker, than that maybe the better approach.

Some Pre-Requisites - Understanding the .csproj project file sdk and targetframework

Let’s Quickly Analyze what’s in a .NET project file.

Now for this context we are just going to look into .NET Core style projects (Sdk style), .NET framework project files contain a lot more information.

CSharp ClassLib

Ok lets say we create a new classlib project using dotnet new classlib with .NET7.0, here is what we get.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <RootNamespace>dotnet_proj</RootNamespace>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
</Project>

Now the key points to look at are the Sdk and TargetFramework inside the PropertyGroup element. Our classlib shows

<Project Sdk="Microsoft.NET.Sdk">

and framework

<TargetFramework>net7.0</TargetFramework>

With web projects (think WebApis, MVC, etc) the Sdk will be

<Project Sdk="Microsoft.NET.Sdk.Web">

Existing TargetFrameworks

In the Problem statement above, I mentioned that there were multiple versions of .NET Core to upgrade, they will typically look like the following

<!-- class libraries -->
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFramework>netstandard2.1</TargetFramework>

<!-- apps of some sort, unit tests etc -->
<TargetFramework>netcoreapp2.1</TargetFramework>
<TargetFramework>netcoreapp2.2</TargetFramework>
<TargetFramework>netcoreapp3.0</TargetFramework>
<TargetFramework>netcoreapp3.1</TargetFramework>

<!-- from net5.0 onwards, can use for any project type,
replaces need for netstandard -->
<TargetFramework>net5.0</TargetFramework>

Now that we know that the TargetFramework is essentially what’s needed to change .NET versions, it becomes much simpler.

Except for perhaps Azure functions, which also have an additional element, the AzureFunctionsVersion

<Project Sdk="Microsoft.NET.Sdk">
	<PropertyGroup>
		<TargetFramework>netcoreapp2.1</TargetFramework>
		<AzureFunctionsVersion>v2</AzureFunctionsVersion>
	</PropertyGroup>
    ...
</Project>

PackageReferences

Now the next thing is to analyze nuget package references. Remember we are using .NET Core style projects which use PackageReference elements directly in the .csproj, rather than the older Packages.config file that contained nuget packages and binary dll references in the .csproj (ie .NET Framework style project file)

<ItemGroup>
	<PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.20.0" />
	<PackageReference Include="Microsoft.AspNetCore.SpaServices.Extensions" Version="3.1.25" />
	<PackageReference Include="Microsoft.AspNetCore.Authentication.AzureADB2C.UI" Version="2.2.0" />
</ItemGroup>

NOTE: PackageRerences can also include version in an element tag, though is not the default

<PackageReference Include="Microsoft.ApplicationInsights.AspNetCore">
	<Version>2.20.0</Version>
</PackageReference>

The Solution

Creating a high level design

Ok given the information gathered, we can have a simple process to represent what we need to do at a high level. Eventually we will be able to use A.I to translate high level requirements to code. ChatGPT3 and Github Co-pilot have demonstrated some capability to assist with this, but there is still a way to go for them.

Pseudocode

Given

• Project files have an Sdk element
• TargetFramework is the .NET version
• PackageReferences contain nuget packages and versions
• Azure Functions have an AzureFunctionsVersion element as version number in format v2 etc

PARSE solutionfile for projects
LOOP through projects
	Open ProjectFile
		Determine ProjectType
			Is it a web project from Sdk
			Does it have AzureFunctionsVersion element
		Update TargetFramework to new version (ie 7.0)
		IF ProjectType IS AzureFunction THEN
			Update AzureFunctionsVersion to latest
		PARSE PackageReferences
		LOOP through PackageReferences
			Get Current Version of Package
			Find Latest Version of Package - from remote nuget repositories
				Cache Latest Version of Package as it may be used in other projects
			Update PackageVersion
		ENDLOOP
	Save and Close ProjectFile
ENDLOOP

We can visualize this approach as a series of flowcharts

Process Project Files

Process Package References

Now, onto the actual approaches

Now one of the ways to automate something like this is write a console application, with parameters and so on. We used to do stuff like this decades ago.

Though my first go-to for these types of things is LinqPad. I write a little about it on my .net tools page . The reason for this, is I want to be able to analyze things as I’m going through, dump output results etc without stepping through the debugger…… and without writing unit tests to fulfil each case.

Side Note: I’m fully aware of various approaches to testing, TDD and others.

So in order to update projects en-masse.

1. Close visual studio solution, otherwise any changes get picked up and refresh vs as it’s happening.
2. Open up Visual Studio solution and parse to get list of project files (or get project files from directory path)

Now inside LinqPad we could just directly update, but I prefer to analyze what needs changing first, this is something I could present to other(s) aswell to show what will change if needed. There could be oddities found, unexpected results.

Step 1. Analyze Solution File to build a collection of project files to update

1. define a collection to hold analysis results
2. loop through each project file and open
   1. check if TargetFramework is in an array of ones to update
   2. if so, than add to a collection with an object containing filename, currentstate (current framework version), and targetstate (new framework version) - similar to how an event-sourcing operation might look.

Build Collection of Projects To Update Framework

Now we have a collection of items showing what project files need changing to update. Very simple (also do you like Mermaid ).

Step 2. Now lets execute it. Update the project files

So we have a collection of project files spread across many directories that are due for an update from Step1.

Ok so now it’s simply a matter of going through the collection of project files and updating the TargetFramework property of the .csproj.

Now a .csproj is essentially an Xml document. this means we can use XDocument from Linq To Xml with relative ease, or the older XmlDocument DOM to process and update the project file.

For Reference: a .NET Project file is described at Microsofts Understanding the project file

So in order to load and update a .NET .csproj file we can:

1. Load the .csproj with libaries to manipulate it as XML such as XDocument
2. Load some Microsoft Libraries that deal with project files
3. Consider simple string replace

So, what is the “correct” way to do this?.

Generally we like to find if something exists, and if it does what is required to apply it, how much learning is needed, how simple or complex is it.

We are dealing with a simple text file that happens to be an xml format, however we are just aiming to update a few strings in the file. Nothing more. So with text we can simply use Find/Replace.

If we were doing more manipulation on the project file than using XDocument may be more suitable or even search for .NET specific libs for managing project files. But in this case, it’s not needed. Find/Replace aligns with the simple Visual Studio Code approach.

flow to update targetframeworks

So the flow is

1. Loop through project files
   1. Open project file into memory (project files are not large, this wont be issue)
   2. Replace TargetFramework>{CurrentFrameworkVersion}</TargetFramework with TargetFramework>{NewFrameworkVersion}</TargetFramework. I add the TargetFramework to be more explicit.
   3. Check if AzureFunctionsVersion exists in projectfile and replace the version. Ie replace AzureFunctionsVersion>v2</AzureFunctionsVersion with AzureFunctionsVersion>v4</AzureFunctionsVersion (which requires .NET6+)
   4. Save file

Very simple, now all projects will have net7.0 (or whatever is desired).

Wait a minute, what about nuget package references?

The next thing is to implement the nuget package reference updates. Now this happens on processing each project file. What we want to do is open the project file, collect the package references, and look for the latest stable version of each package, than update the package reference.

Now analyzing package reference updates could have been done in step (1) at the cost of more analysis time (especially fetching remote version info). Though it may have simply been too much information (ie which projects to update). Remember we are dealing with 350 projects.

General flow is like so:

So the new project file update would look like:

Note: this is the general success path, obviously have to apply error checking conditions and the like

Wouldnt fetching latest package version be costly api calls?

As many projects may share similar package references, we want to cache the latest package versions to reduce need to call a remote resource more than once.

Or alternative caching strategy where we attempt to get item directly from cache without checking if cacheitem exists. Generally simpler, though depends on how cache is implemented.

so, why the flowcharts, where’s the code?

This article was about solving a problem, solving a rather simple problem. refer to the Warning.

As programmers, we want to jump into code, get things going as soon as possible. But how do we know if the program is correct if we don’t know what the end result should be. At some point we need to map out the core steps required.

Once flows and logic have been determined, its easy to apply in pretty much any language, be it C#, Java, JavaScript/TS (with nodejs), rust, basic whatever really.

In order to solve any problem, you first need to understand it, otherwise you end up with spaghetti, or complex pathways as your constantly adjusting/hacking code to get it to fit/work as expected. The better you understand what is required, the easier and simpler it will be to both do and explain.

In my early development days, whenever I came across a slightly tricky problem, I would sketch it out on pen/pencil and paper, the flow and logic, eventually it got to a point where I just now do that in my head before starting on something, it makes the development process much smoother. Though diagramming/documenting is important for future reference.

Again for a simple problem such as this, it’s not typically required, this is just to demonstrate some of it.

Infact with this particular solution, I did the code before writing this article.

If there is enough interest in the code itself, I’ll be happy to share that, just let me know.

Ok, so what about LinqPad?

As mentioned at the beginning of this article, I prefer LinqPad for these types of things, its just fantastic for getting results fast for experimenting and the like. So I initially wrote a collection of linqpad scripts to apply these changes, analyze projects to update, apply changes and the like and dump out results to the linqpad output window which is fantastic.

Ok, here’s what happened.

So after writing some linqpad “scripts” (I call them that because they are not the typical .cs project and execute from LinqPad ). Here is what happened.

1. Upgraded all .NET projects within a solution with 350 projects to .NET7, including updating all nuget package references to latest. The execution time was about 2-3 minutes first run, I think mainly due to fetching the latest nuget package versions from the remote nuget repo. Still much faster than it would have taken in Visual Studio alone.
2. Opened in VS 2022 - Now this is where the real work begins.
3. Tried build, expected errors, got thousands, though 90% were due to some libraries could not be built, causing cascading effect of not being able to find dlls.
4. major issues encountered
   • upgrading .net core 2.1 –> 7 . Now I have done this in previous projects, the biggest changes are actually from 2.x to 3.x. So looked up the migration/breaking changes (hint: there’s quite a few)
   • found issues with Automapper
   • some issues with Entity Framework

This follows on from Part 1 of jekyll to astro.build migration.

So now I have a basic astro.build blog site setup with dated blog posts. The next steps were to:

• Copy over portfolio pages
• Apply blog by years blog page

Portfolio

Now this was rather simple, once I learnt about the astro specifics.

1. Copy my portfolio folder over which included .md files to a /pages/portfolio folder in astro
2. Create a /pages/portfolio.astro view page

In the front matter, I put TypeScript to gather the portfolio items sorted by date with

const portfolioItems = (await Astro.glob("~/pages/portfolio/**/*.{md,mdx}")).sort(
	(a, b) => new Date(b.frontmatter.pubDate).valueOf() - new Date(a.frontmatter.pubDate).valueOf()
);

Than applied html containers to house the content.

And next up was adding a link to the navigation component.

This all worked rather well, and was quite simple in essence.

Add Blog By Years

In my legacy (Jeykyll) site, I had a page that displayed blog posts grouped by years. With a multi-column grid for the years.

I wanted to re-create this in astro with some js (typescript). So here I go.

1. copy the general html layout, and some css styles
2. write some javascript to gather blog posts, than group them into years in descending date order
3. render out for the arrays of years, with arrays of posts in each.

Well this was simpler than I first figured, the main gotcha seems to be the same with any js app, typing, property naming etc. It’s so nice to be able to use simple Javascript/Typescript for this, gives complete flexibility. I will prob clean everything to use typescript a bit later but for now just want to test things out.

Writing some Javascript to group blog posts by year

Now just wrote up some simple js functions to load and group the posts that than get imported into the astro page. The end result is actually a typescript typed definition.

Snippet

export default function groupBy(arr, prop) {
    const map = new Map(Array.from(arr, obj => [obj[prop], []]));
    arr.forEach(obj => map.get(obj[prop]).push(obj));
    return Array.from(map.values());
}

const posts = fetchPosts();   // returns sorted, filtered posts

const postYear = posts.map((p)=>{
      return { year: new Date(p.pubDate).getFullYear(), post: p};
    })

// variable to cache results
_postByYearGrp = groupBy(postYear, 'year').map(yrPost => {
    return { 
      year: yrPost[0].year, 
      posts: yrPost.map(p => p.post)
    };
});

Now need taxonomy for list of years

Now to create the taxonomy for list of years

export const getPostByYearTaxonomy = () => {
    return _postByYearGrp.map(byYear => {
          return {
            year: byYear.year, 
            postCount: byYear.posts.length,
            url: `#${byYear.year}`
          }
      });
}

First off, my original website was generated from Jekyll.

Jekyll and Astro are what is known as SSG - Static Site Generators. Essentially they take a variety of file sources such as markdown, and other text sources (potentially scripts) and convert that to a fully fleshed out static HTML/CSS website. Generally with less javascript and bloat so pages remain responsive and fast, with the added benefit of being able to host anywhere… github pages or netlify being a popular choice.

In some ways it is similar to the old web developments we used do 20 years ago with plain HTML/CSS….. with a big difference, these new static sites can call api’s and perform other dynamic actions as they can still have javascript in them, infact with the newer SSG’s you can plug in SPA’s with them, react, vue, svelte etc.

Jekyll was prob one of the first SSG to arrive around 2010 from the github co-founder, and is currently the most popular. However many others have cropped up since than including Hugo, Gatsby, and others.

The thing with Jekyll is it’s built with ruby, and requires ruby libraries etc installed, it’s not too fast at generation, and was designed in an era before JavaScript SPA’s. So I have been looking for a new SSG (Gatsby, Hugo) and so on, however they didn’t seem to fit for one reason or another (build pipelines, platform choice, logic semantics, etc).

Enter Astro.build, simple, works with modern build pipelines for Javascript and it’s platforms, preact, react, svelte etc with Typescript. I believe this will provide flexibilty and better build performance.

So my goal is to convert my current website using Jekyll to Astro.build….at this point I have only started with it for a few days, so will see how it goes…. here is some lessons learned.

Converting Jekyll to Astro

1. Initial Setup

I head on over to https://docs.astro.build/en/getting-started/ to create a new project.

So simple, just uses standard npm (from Node)

npm create astro@latest

I create the blog template option that comes up, which provides a pre-configured blog structure. I copy my original blog markdown files across into the pages/blog folder and it all works, a list of blog pages are shown, and can click each one…… this is fantastic, so quick.

2. Blog Layouts

Ok, now in Jekyll there was config that specified layouts for certain types of content, ie blogs, general pages and the like.

With Astro, the documentation demonstrates how to apply a layout for markdown within the markdowns frontmatter, this works fine with a few files, but with a lot of pages it could get tedious, also changing layouts means changing in each file… not ideal.

So what I want to do is define a default layout for content within the pages/blog folders, so they are all consistent.

Fortunately, Astro provides Markdown plugins that allow you to add remark based plugins. Remark appears to be a markdown transformer, and looks like the way to accomplish what I’m after.

Creating the set blog layout functionality

After a bit of trial and error. I came up with the following.

So, I create a file set-blob-layout.mjs and place in the following (also logged out console output of file to debug):

As I only want to apply specific layout for blogs, need the current file path it’s processing, this is in the file.history array.

export default function remarkSetBlogLayout(options) { 
    return function (tree, file) {
        //console.log(file);
        if (file.history[0].includes('pages/blog')){
            file.data.astro.frontmatter.layout = '/src/layouts/BlogPost.astro';
        }
    }
}

Now the astro.config.mjs, I add the remarkSetBlogLayout function import to the markdown property to process .md files and in integrations mdx options to process .mdx files.

NOTE: It’s important to apply extends for mdx, and extendDefaultPlugins true for markdown to ensure Astro’s default plugins are enabled.

import { defineConfig } from 'astro/config';
import mdx from '@astrojs/mdx';
import remarkSetBlogLayout from './set-blog-layout.mjs';
import sitemap from '@astrojs/sitemap';

// https://astro.build/config
export default {
	site: 'https://example.com',
	integrations: [mdx({
		remarkPlugins : {
			extends: [remarkSetBlogLayout]
		}	
	}), sitemap()],
	markdown: {
		remarkPlugins: [remarkSetBlogLayout],
		extendDefaultPlugins: true
	}
};

SUCCESS !!

And now all the blog posts in markdown files have the default layout I wanted without affecting other pages. This is very nice, it did not take long to figure out once I got the remark config setup correctly with support from the awesome guys at the Astro Support Thread on discord.

3. Set publish date for blogs from the filename

So in my Jekyll setup, all my posts are in a single folder and the publish date is inferred from the filename, in the format:

posts/YYYY-MM-DD-post-title.md

I prefer to have the explicit date in the file name as it makes organization easier at a glance. And reduces need to apply inside each file.

So there were 2 things to apply:

1. set blog post date from filename
2. being able to organize blog posts, by year into folders

So for (1.) I created a remark plugin to apply the pubDate publishDate.

export default function setPubDateFromFileRemarkPlugin() {
    return function (tree, file) {
        if (file.history 
            && file.history.length 
            && file.history[0].includes('pages/blog')){
            const histFile = file.history[0];
            const fileNameOnly = getFileName(histFile);

            try {
                const dateStr = getDateStrFromFile(histFile);
                //console.log(`Date-Found-In-FileName: file:${fileNameOnly} Date:${dateStr}`);
                file.data.astro.frontmatter.pubDate = dateStr;
            } catch(e) {
                //console.log(`Date-Not-Found: ${fileNameOnly} Could be in Meta - Error: ${e.message}`);
            };
        }

    }
  }

Once adding this plugin to the config arrays similar to the set-blog-layout, all the dates were correctly applied to all the markdown posts.

And for (2.) I simply moved the files into suitably organized folders.

Conclusion

Day 1 concluded with an excellent result, and Astro is looking like a winner so far. I am keen to get more moved over. Copying the the markdown files over were literally just a copy, and just a few adjustments to apply the layout and publishing date.

Here are some key infrastructural components I have found that apply to microservices (which are essentially mini web backends) with some references to .NET specific tooling, though the concepts can apply to most platforms.

More to be added later.

• Apis
  • Api Gateways
    • proxy
  • Api Visibility
    • publicly accessible
    • private
  • Api routing and route naming, verbs
  • Api Style
    • RPC Remote Procedure Call style
    • REST style
  • Api Conventions and Standards
    • OpenAPI/Swagger
    • Serialization (Binary, JSON, MessagePack)
      • Json
      • MessagePack
  • Api Controllers
    • CRUD Verbs, POST, PUT, GET, DELETE, PATCH
    • Http Calls
      • POST to create often returns 201, 202 with location header set
      • GET often returns 200
      • DELETE to delete resource with id, often returning 200, 204
    • Handle errors
      • Middleware to handle unhandled exceptions to not duplicate exception handling
      • Return client validation errors with 400 with standard data structure providing key error details
      • Standardised ProblemDetails to return 500 and 400 errors
• Microservice event pipeline
  • on-demand like call api get response once completed, ideal for simple CRUD
  • long-running, post call to api, with 202 accepted return immediately, poll a status uri until completed, get results
  • listen to message bus events, queues etc and process message, publish messages (or respond) for event notifications, status updates, etc.
  • apply job tracking to track new jobs coming in, processing, completed etc in a database.
• HealthChecks
  • azure blob storage
  • database connections
  • service process
• Authentication
  • Auth Server
  • Resource Servers
  • JWT tokens
  • Auth flows
• Authorization
  • Roles
  • Claims
  • Resource Specific Permissions
  • Authorization handlers
• Messaging
  • Message/Event bus
    • RabbitMQ
    • AzureServiceBus
    • EventGrid
    • Kafka
  • Message Abstractions
    • MassTransit
    • NServiceBus
  • Message Envelopes
  • Message protocols, conventions
    • Cloudevents.io
• DAL (Data Access Layers)
  • Sql Server
  • Postgressql
  • Mysql
  • MongoDb
  • Cosmos
  • Repositories
  • Entities
  • Mapping Entities with DTO’s/Models
  • Migrations (an Example is Entity Framework)
    • Adding migrations
    • Updating databases
      • Sql scripts
      • EF Core Bundles
• Services
  • Business logic
• Configuration
  • Json
  • Azure Key Vault
  • Environment Variables
  • Command line
  • 3rd Party Config providers
• Dependency Injection
  • Singleton (Application Scoped)
  • Scoped (Per Request, or other Scope on demand)
  • Transient (per resolution)
• Logging and Telemetry
  • Logging providers
  • Logging Abstractions
  • Logging ingestors
• Localization
  • Language
  • Currency
  • Culture
  • Timezones
• Templates
  • server side to generate microservice components, base layers
  • client side to generate openapi clients, ui components
• Docker
  • docker files
  • docker compose
  • docker image builds
  • common docker services
    • azurite
    • sql server
    • rabbitmq
    • redis
• DevOps
  • Build Versioning
    • CI generated versions - Build numbering
    • gitversion
    • version endpoints
      • assembly version
      • informational version
  • Build Pipelines
    • build
    • unit tests
    • code coverage
    • code syntax formatting / linting
  • deployment pipelines
    • environment setups
      • app services / web servers / cloud
      • automated testing with tools like newman (Postman collection runner)
    • hosted
    • docker/kubernetes
    • per environment configurations
      • database connections
      • azure/aws storage connections
      • message bus connections
      • config options
• Hosting
  • Cloud
  • Scaling
    • scale up (vertically)
    • scale out (horizontally)
    • Autoscale conditions

Ever since we came into the Web Api era around 2009, there was a big shift to using the REST pattern and terminology with json, sure we have things like Graphql now, though for the most part REST is a fairly simple pattern to use. However it’s often misunderstood even by experienced developers.

REST is really just designed to work the way the stateless HTTP protocol was designed for. Contrary to popular belief, It doesn’t force certain approaches, rather just uses http the way it was intended. Using verbs, resources, http status codes and the like.

Though some popular patterns have emerged via REST that make it simple and consistent to apply, irrespective of platform used. Let’s look at some now.

Rest deals with resources, now a resource could represent an entity record in a database, it could represent a process, a job, task, a collection of items. typically a resource refers to a single element. Though a resource, could have sub-resources and the like.

Let’s look at HTTP verbs in the REST context:

• POST used for creating resources.
• GET used to query resources, get a collection, get a single item (with an id), with filters and so on.
• PUT for updating a specific resource
• DELETE to delete/remove a specific resource
• PATCH for making amendments, changes to specific parts of a resource

Now lets look at the most popular HTTP Status Codes :

Anything in 2xx range is a Success status

• 200 OK - everything’s a-ok, request and process is success
• 201 Created - ok, we have created a new resource that’s ready, and a link to that resources metadata/status will be in the responses location header.
• 202 Accepted - ok, we acknowledge the request though may not have fulfilled it yet, it could be a long running process, however we have a link in the location header to obtain completion status, and possible retry headers. A client app can potentially poll this location to see when it’s completed or reached another status ( note: this is http semantics, not talking about event messaging and the like at this point )
• 204 No Content - ok success, though we have no content to return..

Now 4xx codes represent an issue with the client request

• 400 Bad Request - something in the clients request is malformed, invalid, or simply incorrect, often used for field validation. This means a client should not retry a request without changing something in it.
• 401 Unauthorized - the client has not sent up adequate authorization, be it an Authorization header, or a cookie, or it’s expired, etc. Usually an indication for a client to login.

And 5xx indicate a server error of some sort. They may be transient errors, and a retry could potentially work.

• 500 Internal server error - unhandled or unknown server errors, exceptions and the like. (dont use 400’s for these).
• 502 Bad Gateway - could be a range of issues, from the web server, to proxies, dns etc.

Ok got it, so now look at how they typically work.

• POST is used to create a new resource with data in body, the server returns http 201 Created for a resource that is created and ready, or 202 Accepted to indicate acknoledgement of the request, though resource may be long running. In both instances the url to retrieve the resources metadata/status via a GET is supplied in the location header.
• GET is used to query a collection of resource items, or a single item with an id… it typically returns a 200 OK. Now when using GET with an explicit id to fetch a single item, this will often (not always) represent the same url route as what the POST returns in the location header.
• PUT is used to update an explicit resource. It could return a 200 OK or 204 No Content.
• PATCH is not used as often, however it is similar to a PUT except where a put is about updating a resource, PATCH is designed for updating specific parts of a resource, lets say explicit fields, perhaps a status etc.
• DELETE is to delete/remove/cancel an explicit resource. It could return the deleted resource with a 200 OK or just confirmation with a 204 No Content.

Notice the key point is when using POST, the webapi should provide the location header with the url to the resource created/acknoledged

Ahhh so what about REST routes

Now routing is where things get interesting… often devs will call their API’s REST based because they use verbs and response codes, however their routing conventions may be more aligned with RPC (Remote Procedure Call) approaches.

As mentioned before REST does not enforce particular conventions, however recommends certain concepts. Let’s look at a typical REST convention example.

Let’s say we have a resource called products, in rest we typically use the pluralized form “products” to represent a collection of product in a similar way to how database tables are often named (NOTE: remember REST uses resources, does not neccassarily have to tie back to database records.). This way we can apply consistent routing conventions regardless of the resource for most CRUD like operations.

NOTE: before you consider that CRUD won’t suit your particular services, and therefore REST wont work. CRUD here is just used as an example for a base set of operations, as it’s familiar for database operations.

Example REST-like routing style:

POST /api/products
GET /api/products
GET /api/products/?optionalFilter1=filterValue&limit=10
GET /api/products/{id}
PUT /api/products/{id}
PATCH /api/products/{id}
DELETE /api/products/{id}

With this approach in this example there are effectively 2 routes that use different HTTP VERBS to determine the actual operation on the resource. One route defines the resource collection, the other defines an explicit item within that resource collection

/api/products       - resource collection
/api/products/{id}  - explicit item within resource collection

REST style API routes define resources, and HTTP Verbs define operations. REST came about as a way to provide consistency and standardization for a web based http environment.

Now contrast this with an RPC style api routing

POST /api/addproduct
GET /api/getproductslist
GET /api/getproduct/{id}
POST /api/updateproduct
DELETE /api/deleteproduct

Now with the RPC routing, we have 5 routes, indicating the operation, and using potentially any verb, ie POST is often used instead of PUT, and even sometimes DELETE…. the core point being that the route itself defines the operation…. its akin to calling a function, or procedure by name as in Remote Procedure Call.

Now RPC is essentially the style many traditional developers have used to call API’s for decades as it translates to older traditional API’s.

RPC style API routes define operations, HTTP Verbs are typically POST/GET, though can be any combination.

Sometimes there is a mix of REST/RPC styles in routing.

Recommendation:

Aim for consistent, and standardized web practices, in essence REST is very helpful in this regard, especially as API’s grow in scale and functionality. The larger the API the more you need consistency.

References:

• For more details on Http status codes, look at the Mozilla HTTP status page.
• http://apihandyman.io/do-you-really-know-why-you-prefer-rest-over-rpc/
• https://restcookbook.com/
• https://cloud.google.com/blog/products/application-development/rest-vs-rpc-what-problems-are-you-trying-to-solve-with-your-apis
• https://blog.jscrambler.com/rpc-style-vs-rest-web-apis/

REST Api Examples:

• https://docs.github.com/en/rest
• https://developer.okta.com/docs/reference/core-okta-api/

For this post, I will use .NET as an example.

AVOID: Sending Bad Request (400) Status codes to a client as a catch-all

Let’s look at a webapi action that does some stuff, and catches exceptions, and returns any exception to the client.

[HttpPost("/route/blah/blah")]
public IActionResult SomeAwesomeApi()
{
     try {
       // do some stuff
       ...
     } catch(Exception ex){
        Return new BadRequestObjectResult(ex.Message);    // returns a 400 bad request regardless of server/client issue
     }
}

Now, there are a couple of issues with this:

1. [.NET Specific] The entire try/catch block is unneccassary, as a .NET Core middleware could be written to handle the exception across all actions, rather than repeating it.
2. Red Flag: Returning a BadRequest to the client for exceptions, which is a HTTP 400 status code.

With (1) that’s just a good practice for any platform to reduce code duplication.

(2) is the main issue. The reason being is that an HTTP 400 status means that the client request is invalid in some way. see more about about HTTP 400. This means that the client should modify it’s request before repeating it. Now in the example above where the BadRequest is returned on any exception, the exception could be caused from anything, server or client request issue. We don’t know if it’s due to the request, or due to something in the server code path. Perhaps network connectivity is down, there could just be something wrong with the server and nothing to do with the request. In which case we should never just send a 400 badrequest to the client. a 400 effectively means a client should not retry without changing the request

The only time we should send a BadRequest (HTTP 400) status to the client is when we know it’s request related. For example properties don’t validate, or a request header is missing or invalid. An unknown exception should just return a 500 (Internal server Error)

Same example as an Azure Function HTTP Trigger Api Endpoint

[FunctionName("SomeFunction")]
[OpenApiOperation(operationId: "SomeFunction", tags: new[] { "name" })]
[OpenApiParameter(name: "id", In = ParameterLocation.Path, Required = true, Type = typeof(Int32), Description = "an id parameter")]
[OpenApiResponseWithBody(statusCode: HttpStatusCode.OK, contentType: "application/json", bodyType: typeof(SomeResponse), Description = "Result")]
public async Task<IActionResult> SomeFunction([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "route/blah/{id}")] HttpRequest req, ILogger log, int id)
{
    try
    {
        // do some stuff
        ...
    }
    catch(Exception ex)
    {
        return new BadRequestObjectResult(ex.Message);
    }

}

All too often I see devs, and architects that are not completely familiar with .NET Core implement solutions that are essentially already baked into .NET Core itself. They may have used .NET Framework in the past years ago, and use the same principles applied to .NET Core. This is not a good approach to take, for one it’s reinventing the wheel, usually with code that is less battle tested, and often not as extensible. It also requires extra maintenance and custom documentation, with less support.

Well here I aim to explain the core concepts in .NET Core’s base layers to help you get up to speed quickly. I am assuming you already know C# to some degree, and how it plugs into .NET in general. You also understand design patterns and understand how to write clean maintainable code.

Quick overview with .NET Framework to .NET Core

First off .NET Core != .NET Framework …. that’s right, while they share syntactical similarities, nuget, and tooling and so on, they are essentially completely different platforms.

• .NET Framework from .NET Framework 1 - .NET Framework 4.8
• .NET Core 1.0 - .NET Core 3.1 and now just .NET 5 / .NET 6 ….. etc

.NET Framework
Designed specifically for Windows OS and released around 2001. Requires Windows tooling such as Visual Studio for editing, and I.I.S for ASP.NET Web Servers, was the .NET ecosystem up until .NET Core come around.

• Closed source Proprietary framework
• Windows only development and runtimes (there were spinoffs such as Mono for other platforms, and various other runtimes)
• ASP.NET Minmum memory for a request was approx 26kb
• ASP.NET Extensibility was somewhat complex.
• Suited SOAP, WCF, XML techs

.NET Core
Re-architected from the ground up for new open cross platform developments. Released 2014

• Cross-platform development and runtimes
• Open source
• Open standards
• Highly configurable and extensible
• Command line tooling infrastructure
• Lighter, significant performance improvements
• Dependency Injection is first class principle
• Web Api minimum request memory ~2kb
• designed for modern development approaches such as Web Api’s, JSON, REST.
• Provided a command line tool dnx which was later changed to dotnet in later versions.

In essence, .NET 5/6/7+ is the current and future of .NET…… there is no point going back to the older .NET Framework unless legacy projects have no alternative

Core concepts, patterns, considerations for .NET Core/5+ apps

Explicit Abstractions