Dogs Chasing Squirrels

A software development blog

Unknown's avatar

About Mike Bennett

Parsing streaming JSON in Typescript

0

In our servers, we will write large volumes of data out to the response stream directly in order to avoid the memory pressure of having all the data in memory at once.

In JavaScript/TypeScript, all though the fetch API supports streaming responses, there is no such thing, in the browser today, as a streaming JSON parser. As an exercise, I wrote a function that will wrap a request in an AsyncGenerator to yield objects of some type T where the response body consists of a JSON array of that type.
Note that the code yields parsed objects in batches. If every parsed object was yielded individually, the “for async” loop would create so many promises that the performance would degrade terribly.

  // Given a response that returns a JSON array of objects of type T, this
  // function returns an async generator that yields batches of objects of type T of a given batch size.
  // Note: If the batch size is too small, the many async calls reduce the performance
  async function* parseStreaming3(response: Response, batchSize: number = 100): AsyncGenerator {
    // If the response has no response body, stop.  This will only happen if something went wrong with the request.
    if (null === response.body) {
      console.warn(`Response has no body.`)
    } else {
      // The JSON object start character, '{'
      const START_OBJECT = 123;
      // The JSON object end character, '}'
      const END_OBJECT = 125;
      // Create a decoder
      const decoder = new TextDecoder('utf-8');
      // Get a streaming reader for the response body
      const reader = response.body.getReader();
      // Keep track of the object depth
      let depth = 0
      // If an object spans two chunks, the previous bytes that represent the end of the previous buffer
      let previousBytes: Uint8Array | undefined = undefined;
      // The start index of the current object
      let startIndex: number | undefined = undefined;
      // The current batch of items
      let batch = new Array()
      // eslint-disable-next-line no-constant-condition
      while (true) {
        // Get the bytes and whether the body is done
        const { done, value: bytes } = await reader.read();
        // If there's no value, stop.
        // If we have values...
        if (undefined !== bytes) {
          // noinspection JSIncompatibleTypesComparison
          // For each byte in the value...
          for (let i = 0; i < bytes.length; i++) {
            // Get the byte
            const byte = bytes[i];
            // If the byte is the start of a JSON object...
            if (START_OBJECT === byte) {
              // Increment the depth
              depth += 1;
              // If the depth is 1, meaning that this is a top-level object, set the start index
              if (1 === depth) {
                startIndex = i;
              }
              // If the byte is the end of an object...
            } else if (END_OBJECT === byte) {
              // If this is a top-level object...
              if (1 === depth) {
                // If there's a previous start index and previous bytes...
                if (undefined !== previousBytes) {
                  try {
                    // Combine the previous bytes with the current bytes
                    const json = decoder.decode(previousBytes)
                      + decoder.decode(bytes.subarray(0, i + 1));
                    // Parse the JSON into an object of the given type
                    const obj: T = JSON.parse(json);
                    // Add the parsed object to the batch
                    batch.push(obj);
                  } catch(e) {
                    console.warn(e)
                    console.log(` - previous json = `, decoder.decode(previousBytes))
                    console.log(` - json = `, decoder.decode(bytes.subarray(0, i + 1)))
                    // Stop
                    return
                  }
                  // Reset the previous bytes
                  previousBytes = undefined;
                  // If there's a start index...
                } else if (undefined !== startIndex) {
                  try {
                    // Get the JSON from the start index to the current index (inclusive)
                    const json = decoder.decode(bytes.subarray(startIndex, i + 1));
                    // Parse the JSON into an object of the given type
                    const obj: T = JSON.parse(json);
                    // Add the parsed object to the batch
                    batch.push(obj);
                    // Un-set the start index
                    startIndex = undefined;
                  } catch(e) {
                    console.warn(e)
                  }
                }
                // If the batch is at the batch size...
                if (batch.length === batchSize) {
                  // Yield the batch
                  yield batch;
                  // Reset the batch
                  batch = new Array()
                }
              }
              // Decrement the depth
              depth -= 1;
            }
          }
          // Because the start index is cleared at the end of each object,
          // if we're ending the loop with a start index, we must not have
          // encountered the end of the object, meaning that the object
          // spans (at least) two reads.
          if (undefined !== startIndex) {
            // If we have no previous bytes...
            if (undefined === previousBytes) {
              // Save the bytes from the start of the object to end of the buffer.
              // We'll combine this json with the next when we encounter the end of the
              // object in the next read.
              previousBytes = bytes.subarray(startIndex);
            } else {
              // There must not have been an end of the object in the previous read,
              // meaning that the read contains some middle section of an object
              // It happens sometimes, if we happen to get a particularly short read.
              // Combine the previous bytes with the current bytes, extending the data.
              const combinedBytes: Uint8Array = new Uint8Array(previousBytes.length + bytes.length);
              combinedBytes.set(previousBytes);
              combinedBytes.set(bytes, previousBytes.length);
              previousBytes = combinedBytes
            }
          }
        }
        // If we're at the end of the response body, stop.  There's no more data to read.
        if (done) {
          break;
        }
      }
      // If items remain in the batch, yield them
      if (batch.length > 0) {
        yield batch;
      }
    }
  }

Hosting a compiled SPA on .net core

0

Let’s imagine we have a single page application (SPA) to handle the client-side logic and a .NET WebAPI application to handle the server side. It’s common to run a compiled single page application (SPA) on something like NodeJS or Vite, and Microsoft has examples of how to create a proxy application that passes SPA-related calls through to it.

Let’s imagine, though, that we want to compile our single page application and have .NET serve up the compiled files statically, without our having to run another server. Here, I’ll demonstrate how to do that. The source code for the demo is on GitHub.

First, let’s create the folders we need. I’m going to call the project “SpaDemo”. Also note that I’m running the following on Linux, so adapt the shell commands as you see fit.

Getting Started

Make the main folder.

$ mkdir SpaDemo

Server Project

In our main folder, create a folder for our .NET server project.

$ cd SpaDemo
$ mkdir server
$ cd server

Create a .NET solution:

$ dotnet new sln -n DemoServer

In it, create a new webapi project and add the project to the solution:

$ dotnet new webapi -n DemoServer
$ dotnet sln DemoServer.sln add DemoServer/DemoServer.csproj

This will create .NET’s default “WeatherForecast” app. Let’s get rid of that stuff and add a new controller that listens to the “/api/demo” route:

using Microsoft.AspNetCore.Mvc;
namespace DemoServer.Controllers; 
[ApiController]
[Route( "/api/[controller]")]
public class DemoController  : ControllerBase {
}

Imagine we want to offer up some .NET configuration data to our SPA client. One way to do this would be to add some values to appsettings.json, e.g.:

{"someName":"SomeValue"}

Then host them with a custom endpoint in our controller:

	private readonly IConfiguration _configuration;

	public DemoController(IConfiguration configuration) {
		this._configuration = configuration;
	}

	[HttpGet]
	[Route("config")]
	public IActionResult GetConfig() {
		// Create a dynamic configuration object
		var config = new {
			SomeName = this._configuration.GetValue<string>( "SomeKey" )
		};
		// Return it as JSON
		return new JsonResult( config );
	}

If we run the application now, we can go to /api/demo/config and see the following:

{"someName":"SomeValue"}

Client Project

I’m going to create the client project in Svelte.

First, go back to our “SpaDemo” folder and create the client application:

$ cd ..
$ npm create vite@latest
✔ Project name: … client
✔ Select a framework: › Svelte
✔ Select a variant: › TypeScript

Scaffolding project in /home/mike/Repos/SpaDemo/client...

Done. Now run:

  cd client
  npm install
  npm run dev

If you do as instructed, you’ll get the “Vite + Svelte” demo page with is counter application.

I want to test routing, so install the svelte-routing library:

$ npm install svelte-routing

added 1 package, and audited 97 packages in 654ms

10 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Then we’ll modify the application to a very simple 3-page application.

App.svelte:

<script lang="ts">
  import { Router, Route } from 'svelte-routing'
  import Home from "./lib/Home.svelte";
  import Page1 from "./lib/Page1.svelte";
  import Page2 from "./lib/Page2.svelte";
</script>
<main>
    <Router>
        <Route path="/">
            <Home/>
        </Route>
        <Route path="/page1">
            <Page1/>
        </Route>
        <Route path="/page2">
            <Page2/>
        </Route>
    </Router>
</main>

Home.svelte:

<script lang="ts">
    import Links from "./Links.svelte";
</script>
<h1>Home</h1>
<Links/>

Page1.svelte:

<script lang="ts">
    import Links from "./Links.svelte";
</script>
<h1>Page 1</h1>
<Links/>

Page2.svelte:

<script lang="ts">
    import Links from "./Links.svelte";
</script>
<h1>Page 2</h1>
<Links/>

Links.svelte:

<script lang="ts">
    import { Link } from 'svelte-routing'
</script>
<div>
    <Link to="/">Go Home</Link>
    <Link to="/page1">Go to Page 1</Link>
    <Link to="/page2">Go to Page 2</Link>
</div>
<style lang="css">
    div {
        display: flex;
        flex-direction: column;
    }
</style>

Run the application with npm run dev. It will show a Home page with links to Page1 and Page2, which you can then navigate around.

Compile the Client

We want to compile the client to static files and we want to put the result in the server’s wwwroot directory.

First, modify the client’s vite.config.ts file to build to our desired folder:

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [svelte()],
  build: {
    outDir: '../server/DemoServer/wwwroot',
    emptyOutDir: true,
    rollupOptions: {
      output: {
      }
    }
  }
})

Run npm run build to compile:

$ npm run build

> client@0.0.0 build
> vite build

vite v4.4.9 building for production...
✓ 42 modules transformed.
../server/DemoServer/wwwroot/index.html                  0.46 kB │ gzip: 0.29 kB
../server/DemoServer/wwwroot/assets/index-acd3aff5.css   1.09 kB │ gzip: 0.58 kB
../server/DemoServer/wwwroot/assets/index-1625dc16.js   24.76 kB │ gzip: 9.57 kB
✓ built in 560ms

If you navigate to the folder, you’ll see that we have an index.html file in the root and then some CSS and JS files in the assets folder.

Configure the Server

If you run the .NET application now and go to / you’ll get a nice 404. To tell it that it should handle static files, we’ll add “UseStaticFiles”, and to let it know that / should go to /index.html we’ll add “UseDefaultFiles”:

app.UseDefaultFiles();
app.UseStaticFiles();

Restart your .NET application, navigate to / and, voilà, you’ll have a working Svelte application. You should be able to navigate around to /page1 and /page2 and it will all work correctly.

Now try loading /page1 directly in the browser. 404! When we started at /, .NET routed that to index.html and thereafter our SPA was sneakily rewriting the URL to fake /page1 and /page2 endpoints. If we go directly to those URLs, .NET won’t know how to route it. To fix that, we finally add:

app.MapFallbackToFile( "index.html" );

Now when we go to /page1, .NET will happily pass it back to our SPA which will route it correctly.

Conclusion

We now have a compiled Svelte single-page application running as static files behind .NET with routing working as expected. Again, the source code for the demo is on GitHub.

Increasing shared memory on mac OS

2

Today I installed pgpool-ii, the Postgresql connection pool application, on a mac. By default, it tries to use allocate a local memory cache, but the shared memory pool on a mac is lower than the amount it tries to allocate, resulting in an error like “Could not create shared memory segment: Invalid argument.”.

You can see the available memory by running:

sysctl -a|grep "\.shm"

To increase the available shared memory, run this:

sudo sysctl -w kern.sysv.shmmax=268435456

Svelte HTTPS on localhost – vite edition

0

In a previous post, I showed how to enable HTTP on Svelte, back before Svelte had switched over to Vite. Now, on Vite:

First, generate a certificate and key. Save these as certificate.crt and certificate.key in the project’s folder.

Next, edit vite.config.ts. Add the following to the top:

import fs from 'fs';

Then, in the server configuration:

    server: {
        https: {
            key: fs.readFileSync('certificate.key'),
            cert: fs.readFileSync('certificate.crt'),
        }
    },

The Last Guy’s Code is Terrible and Needs to be Rewritten

2

It’s a cliché in software development that every developer taking over a project will declare that the last guy’s code is terrible and needs to be rewritten.  Surely the last guy thought his code was perfectly fine.  Who is right?  Both are. 

Maintainability is a quality of code.  An important one, in fact.  The last guy understood his code and what it was doing so it was perfectly maintainable and so was of high quality.  The new guy coming in can’t understand what’s going on and so to him it’s of low quality.  The real question is: How can we make code more maintainable?

The most maintainable code is code that you yourself wrote in the last day or so.  You can remember what you were doing and why.

The next most maintainable code is code that you wrote in the past.  You may not remember what you were doing, but the code is familiar and so you can usually figure it out.

After that, the next most maintainable code is code that somebody else wrote and that person is still around.  You may not understand what it does, but at least you can ask the author.

The least maintainable code is code that somebody else wrote and that person is long gone.  You don’t understand what it does and there’s no way to find out other than to trace through it.

The way to make code more maintainable, then, is to get the least maintainable code – code written by another person long ago – to be as understandable as code that you yourself wrote 10 minutes ago.  We can accomplish this with to things:

  1. Rigorous coding standards
  2. Extensive code commenting

Rigorous coding standards ensures that everyone’s code looks the same.  This includes alignment, code ordering, and code naming strategies.  For example, I ensure all my classes’ properties and methods are declared alphabetically.  My team does the same.  Any one of us can easily find any method or property in anybody else’s code.

Extensive code commenting means commenting nearly every line of code, even if it seems redundant.  If you know what your code is doing, you should be able to write that down in good English.  If you don’t know what your code is doing, you shouldn’t be coding.  This makes reading code like reading a book – rather than trying to decipher a line, you can just read, in English, what it does.  It adds a further validation step, which is that if what the code does and what the comment says the code does are different, this indicates a potential bug. 

Getting developers to write extensive code comments is hard.  Humans are lazy by nature and developers are as bad as any.  Many mediocre developers got into the industry because they saw software development as a way to make a good income while remaining seated and not working too hard.   A good developer, though, will understand the use of this technique and once he or she has experienced how much it helps the maintainability of software, will come to embrace it.

If you’ve achieved your goal, you should not be able to tell whose code is whose even in a large organization and any developer should be able to look at any other developer’s code as if it’s their own.

Svelte HTTPS on localhost

2

There are a few things you need to do to get the Svelte default app to run over HTTPS on localhost.

First, generate a certificate and key. Save these as certificate.crt and certificate.key in the project’s folder.

Next, edit package.json. Change the “start” command to:

"start": "sirv public --no-clear --single --http2 --port 5000 --cert certificate.crt --key certificate.key"

Note that port 5000 is the default, so technically --port 5000 is redundant, but if you were to change it, this is where you would change it. When you run npm run dev, the application will now run on https://localhost:5000. Note, though, that livereload.js will still be running as http and will fail. Here’s how we fix that.

Edit rollup.config.js. Import the node fs command we need (Note: this is a node library so you don’t need any new imports in package.json):

import { readFileSync } from 'fs'

Replace the line:

 !production && livereload('public'),

With:

 !production && livereload({
    watch: 'public',
    port: 5001,
    https: {
        key:  readFileSync( 'certificate.key'),
        cert: readFileSync('certificate.crt'),
    },
}),

Here I’ve set the port to 5001, but if omitted it will default to some other port.

Adaptive firewalls will be the death of me

0

We had an issue today where requests to our Azure App Service were extraordinarily slow. According to our app service metrics, requests were being handled in around 15 milliseconds, however clients were seeing requests take half a minute. Clearly this was something related to the network. Our service is behind an Azure Application Gateway, though nothing in Azure that I could find would show me the end-to-end request time and where the bottleneck was. After doing some testing on my own, I found that my initial requests were instant but then subsequently slowed. This was the tip-off. When you see a request slow over time, it’s an indication that some adaptive firewall is sitting in the middle and, after some initial traffic, has seen something it doesn’t like and has decided to start interfering with the traffic. Hunting around, I found the firewall rule enabling the firewall’s inspection of the body of requests. After disabling that, it’s been smooth sailing.

When I was initially trying to find the source of the problem, I went through Microsoft and Azure’s own troubleshooting guide where it ran checks on my software and made suggestions. Its “documents from the web that might help you” were no help at all.

Shockingly, “Hollywood: Where are they now?” didn’t help me fix my Azure App Service problems.

Rider – The specified task executable “fsc.exe” could not be run.

0

One of my F# projects started throwing this error on build after I uninstalled and reinstalled some other, unrelated .net core versions.

    C:\Program Files\dotnet\sdk\5.0.103\FSharp\Microsoft.FSharp.Targets(281,9): error MSB6003: The specified task executable "fsc.exe" could not be run. System.ComponentModel.Win32Exception (193): The specified executable is not a valid application for this OS platform. [C:\FSharp.fsproj]
    C:\Program Files\dotnet\sdk\5.0.103\FSharp\Microsoft.FSharp.Targets(281,9): error MSB6003:    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo) [C:\FSharp.fsproj]
    C:\Program Files\dotnet\sdk\5.0.103\FSharp\Microsoft.FSharp.Targets(281,9): error MSB6003:    at System.Diagnostics.Process.Start() [C:\FSharp.fsproj]
    C:\Program Files\dotnet\sdk\5.0.103\FSharp\Microsoft.FSharp.Targets(281,9): error MSB6003:    at Microsoft.Build.Utilities.ToolTask.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [C:\FSharp.fsproj]
    C:\Program Files\dotnet\sdk\5.0.103\FSharp\Microsoft.FSharp.Targets(281,9): error MSB6003:    at Microsoft.Build.Utilities.ToolTask.Execute() [C:\FSharp.fsproj]

Googling the error didn’t turn up anything useful. I started a new F# project which compiled without errors, which led me to the solution, which was to delete my failing projects .idea folder. After reopening Rider, problem solved.

Streaming a response in .NET Core WebApi

1

We, as web developers, should try to avoid loading files into memory before returning them via our APIs. Servers are a shared resource and so we’d like to use as little memory as we can. We do this by writing large responses out as a stream.

In the ASP.NET MVC days, I would use PushStreamContent to stream data out in a Web API. That doesn’t seem to exist in .NET core and, even if it did, we don’t need it anyway. There’s an easy way to get direct access to the output stream and that’s just with the controller’s this.Response.Body, which is a Stream.

In this sample, I just grab a file out of my downloads folder and stream it back out:

[HttpGet]
[Route( "streaming" )]
public async Task GetStreaming() {
    const string filePath = @"C:\Users\mike\Downloads\dotnet-sdk-3.1.201-win-x64.exe";
    this.Response.StatusCode = 200;
    this.Response.Headers.Add( HeaderNames.ContentDisposition, $"attachment; filename=\"{Path.GetFileName( filePath )}\"" );
    this.Response.Headers.Add( HeaderNames.ContentType, "application/octet-stream"  );
    var inputStream = new FileStream( filePath, FileMode.Open, FileAccess.Read );
    var outputStream = this.Response.Body;
    const int bufferSize = 1 << 10;
    var buffer = new byte[bufferSize];
    while ( true ) {
        var bytesRead = await inputStream.ReadAsync( buffer, 0, bufferSize );
        if ( bytesRead == 0 ) break;
        await outputStream.WriteAsync( buffer, 0, bytesRead );
    }
    await outputStream.FlushAsync();
}

This does the same thing in F#:

[<HttpGet>]
[<Route("streaming")>]
member __.GetStreaming() = async {
    let filePath = @"C:\Users\mike\Downloads\dotnet-sdk-3.1.201-win-x64.exe"
    __.Response.StatusCode <- 200
    __.Response.Headers.Add( HeaderNames.ContentDisposition, StringValues( sprintf "attachment; filename=\"%s\"" ( System.IO.Path.GetFileName( filePath ) ) ) )
    __.Response.Headers.Add( HeaderNames.ContentType, StringValues( "application/octet-stream" ) )
    let inputStream = new FileStream( filePath, FileMode.Open, FileAccess.Read )
    let outputStream = __.Response.Body
    let bufferSize = 1 <<< 10
    let buffer = Array.zeroCreate<byte> bufferSize
    let mutable loop = true
    while loop do
        let! bytesRead = inputStream.ReadAsync( buffer, 0, bufferSize ) |> Async.AwaitTask
        match bytesRead with
        | 0 -> loop <- false
        | _ -> do! outputStream.WriteAsync( buffer, 0, bytesRead ) |> Async.AwaitTask
    do! outputStream.FlushAsync() |> Async.AwaitTask
    return EmptyResult()
}

A couple of important notes:
1. By default, you have to write to the stream using the Async methods. If you try to write with non-Async methods, you’ll get the error “Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.” and, as the error says, you’ll have to enable the AllowSynchronousIO setting.
2. On C# you can have your streaming controller method return nothing at all. If you try the same on F#, you’ll get the error, midway through the response, “StatusCode cannot be set because the response has already started”. The solution to this is to have the method return an EmptyResult().

Starting and Stopping Azure Web Apps with a DevOps pipeline

0

We’re using Azure web services for our dev and test environments. Since we only use these environments during the day, I wanted to write some automated functions to turn them off at night and turn them back on in the morning. It can be done with Azure Functions, but I thought it would be easier to build into an Azure Pipeline where developers are all doing their work anyway, so if someone needed to work after hours they would know how to get at it.

First is the PowerShell script to start or stop a service, based on this post.

# This controls an app service.
Param(
    # The tenant ID
    [Parameter(Mandatory=$true)][string] $TenantId,
    # The subscription ID
    [Parameter(Mandatory=$true)] [string] $SubscriptionId,
    # The App Registration client ID
    [Parameter(Mandatory=$true)][string] $ClientId,
    # The App Registration client secret
    [Parameter(Mandatory=$true)][string] $ClientSecret,
    # The resource group of the service to control
    [Parameter(Mandatory=$true)][string] $ResourceGroup,
    # The name of the service to control
    [Parameter(Mandatory=$true)][string] $AppService,
    # The switch if we want to start
    [switch] $Start = $false,
    # The switch if we want to start
    [switch] $Stop = $false
)

$StartOrStop = ""
if ( $Start ) {
    $StartOrStop = "start"
}
if ( $Stop ) {
    $StartOrStop = "stop"
}

# Get the authentication token
$Auth = Invoke-RestMethod `
 -Uri "https://login.microsoftonline.com/$TenantId/oauth2/token?api-version=1.0" `
 -Method Post `
 -Body @{"grant_type" = "client_credentials"; `
 "resource" = "https://management.core.windows.net/"; `
 "client_id" = "$ClientId"; `
 "client_secret" = "$ClientSecret"}

$HeaderValue = "Bearer " + $Auth.access_token

# Control the service
Write-Output "Executing $AppService $StartOrStop..."
Invoke-RestMethod `
-Uri "https://management.azure.com/subscriptions/$SubscriptionId/resourceGroups/$ResourceGroup/providers/Microsoft.Web/sites/$AppService/$StartOrStop`?api-version=2018-02-01" `
-Method Post `
-Headers @{Authorization = $HeaderValue}   
Write-Output "Done $Command $AppService"

Then there's the pipeline yaml to run the script. This is the "Start" version. I'll leave "Stop" as an exercise for the reader.

name: $(Date:yyyyMMdd)-$(Rev:r)
jobs:
  - job: "Build"
    pool:
      vmImage: 'windows-2019'
    variables:
      tenantId: 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'
      subscriptionId: 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'
      clientId: 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'
      # Note: clientSecret set in Azure Pipeline
      resourceGroup: 'MY_RESOURCE_GROUP'
      appService: 'MY_APP_SERVICE'
    steps:
      - task: PowerShell@2
        displayName: 'Start $(appService)'
        inputs:
          errorActionPreference: Stop
          targetType: filePath
          filePath: '$(Build.SourcesDirectory)\ControlAppService.ps1'
          arguments: '-Start -AppService $(appService) -TenantId $(tenantId) -SubscriptionId $(subscriptionId) -ClientId $(clientId) -ClientSecret $(clientSecret) -ResourceGroup $(resourceGroup)'

I have the clientSecret set up in the Azure Pipeline’s variables as a secret value.

That’s pretty much it. The pipelines are set to run on a timer: Stop at 7pm; Start at 7am.