Using the browser for serial data in JavaScript

Yesteday I ended the post with a cliff hanger ending. Who know that programming could be so exciting? Who knew? Anyhoo, on with the story. For those who are new to what I'm trying to do, the aim is to be able to plug a brand new embedded devices (ideally an ESP32 or ESP8266 based device) into a computer and then, by using a browser (perhaps Chrome or Edge) visit a web page that can load the device with a program. The web page actually exists so I did have some success. In this set of posts I'm going through how I did it.

https://www.connectedlittleboxes.com/gettingstarted.html

I'm replicating in the browser the behaviour of the esptool program supplied by Espressif. The esptool is "baked in" to the process of deploying a program when you use Arduino or Platform IO to put a program into a device. Esptool is written in Python so I'm essentially translating a big program from Python to JavaScript. I've got some very small parts of my program working. I now know how to reset a device and prepare it to receive a program. I also know how to encode the messages that I want to send into the device.

The first message that our program must send is called "sync" and it just allows the ESP device and the host computer to agree on the rate that they are going to send serial data and that they are both connected and awake.

To do the sync I have to make the browser send a message and then wait for a response from the device. I'm not actually guaranteed to get a response however, because the target device might need several messages before it can work out the speed it should reply at.

To bring things right up to date, yesterday I discovered that when a JavaScript program asks for data from a serial device via the browser the request only returns when some data arrives. So, if my program shouts "Hi" and the device doesn't respond my program will get stuck waiting for a response.

In real life, if you shout "Hi" to someone and they don't respond you'll wait a decent interval and then shout "Hi" again. We need this behaviour in our code. We need a way of "timing out" a read request so that we can send another message again. It turns out that this can be done. But it involves some knowledge of how JavaScript asychronous code works, and what a JavaScript Promise does. So stand buy to learn something.

Asynchronous code in JavaScript

Don't you hate it when a web site gets stuck? When you click a button, nothing happens, you click it again, and then you notice that everything on the site is now locked. The people at JavaScript know your pain. They hate it when the code behind your button click does something that takes a while (or fails) and you are left waiting. So they've invented a way of trying to stop web pages getting stuck. The principle they are using is not unique to JavaScript. Other languages (for example the wonderful C#) also provide what they call "asynchronous" operation.

Asynchronous is a way of saying "I'll get round to doing this, but in my own time". It is how I used to tidy my bedroom when I was a teenager. Mum would ask me to tidy up the room and I'd say "Later..". Sometimes mum would refuse to leave my room until it was tidy. Making mum wait like this is a good example of "synchronous" operation. Mum can't leave until I've finished tidying up.

As I got older and more mature (oh yes) mum started to trust me a bit more. Now she would ask me to tidy up and then she could go back to cooking lunch, cleaning the house, ironing my clothes or any one of a number of active tasks. When I'd finished tidying up I'd call out to mum, she would come back and praise my handiwork. The tidying processs happened asynchronously as far as mum was concerned.

Nice family story, but what does this have to do with JavaScript? Well, some operations that a program needs to do are a bit like me tidying my room. They will take a while to complete and there is a chance that they might never finish. So JavaScript uses a mechanism called a "promise" to manage operations like these. Rather than calling a function and then being stuck waiting for it to complete (which is synchronous operation) we can call a function that returns a promise to do something (which is hopefully more solid than my promises to clean my room). The promise provides a means by which the asynchronous operation can say the digitial equivalent of "I've tidied my bedroom" by accepting two functions. One is called when the promise is fullfilled. The other, more depressing function is called when the promise could not be fulfilled.

if (this.reader == null) {
  this.reader = this.port.readable.getReader();
}

if (this.readPromise == null) {
  this.readPromise = this.reader.read();
}

This is how I start reading from a port. The action looks very like the write action we saw yesterday. For writing you get a writer object and then ask it to write. For reading you get a reader object and then ask it to read. However the read you get from a reader is not data from the port. It is a Promise to fetch some data. The above code sets the value of the variable this.readPromise to this promise. When the serial data arrives the promise will be fulfilled, but in the meantime my code can keep running.

The sharper eyed ones amongst you will have spotted that I only make a new reader and readPromise if I don't already have one. This is important. The first time round, and after a successful read, my program will need to make new reader and readPromise items. But if this code is running on the back of a timeout (no data arrived down the serial port after a time interval) then the program must re-use the existing reader and promise. So now we have the beginnings of a way of dealing with timeouts, but next we actually have to make the timeout itself. We are going to use another Promise to do this.

Making Promises

Next our program performs the following statements to create a timeout Promise:

const timeoutPromise = new Promise((resolve, reject) => {
  this.timeoutID = setTimeout(resolve, 500);
});

So, how does a Promise object work? Well, I've no idea really. I'm fairly sure it doesn't involve pixies, but beyond that I'm drawing a blank here. But I do know how to use a Promise, which is fine by me. I'm going to give the Promise a lump of code to run as a promise. The lump of code will take the form of a function. The promise will run that code for me and manage the outcome. My code might take a while but at some point later the outcome will either be resolved (hurrah) or rejected (boo). The code above is making a timeout, so this particular Promise will resolve when the time out period has expired. The read call that we made above to fetch serial data will resolve when a packet of data has been received from the serial port.

The code being managed by the Promise needs a way of telling the Promise either it worked or it failed. It does this by calling one of the two functions that are passed into it. If you look at the code above you will see that my timeout promise code accepts "resolve" and "reject" as references to functions to call when my promise is resolved or rejected. It turns out that the timeout always works, it can never reject, so my function doesn't use the rejected parameter. But it does use the resolve parameter. It puts it straight into a call of the setTimeout function. So (deep breath) what does setTimeout do?

A timeout is a way that a JavaScript program can ask for an event to be fired at some point in the future. The first parameter to the setTimeout call is a function to be called when the timeout expires. I can pass it a reference to the function to that will be called when the promise is fulfilled (so that it will be called when the promise is fulfilled). The second parameter to setTimeout is the time in milliseconds that JavaScript will wait before calling the timeout function. I want the program to wait half a second before giving up on a serial read request, so I've set this to 500.

The setTimeout function returns an id (actually a number) that the program can use to turn a timeout off before it occurs. We will use this later. So, a recap of what is going to happen when I set up my timout is in order. Here's the code again:

const timeoutPromise = new Promise((resolve, reject) => {
  this.timeoutID = setTimeout(resolve, 500);
});

The program hits the statement and is asked to make a new Promise. The Promise accepts the function that is supplied to it. In this case the function looks like this:

(resolve, reject) => {
  this.timeoutID = setTimeout(resolve, 500);
};

The function controled by a Promise takes two parameters, does something and then calls the resolve function if it ends well and the reject function if it ends badly. In our case the code in the function takes the resolve parameter (the function we want to call when the promise has ended well) and drops it into a call to setTimeout. setTimeout will call the resolve function at some point (500 milliseconds) in the future, ending the Promise.

A Promise race

Right. Now I've got two promises. One of them will be fulfilled when it gets some data from the serial port. The other will be fulfilled in half a second. What I can do now is creates a promise race:

const { value, done } = await Promise.race([this.readPromise, timeoutPromise]);

The JavaScript Promise class provides a method called race. The race method accepts a list of promises. It completes when one of the promises in the list is fulfilled and it returns the result from the fastest promise. And no, I don't really know how it works either.

However, it is very useful in making our timeout behaviour work. If the read promise is fulfilled first (i.e. we get some data) the race will return an object containing a value and a done property (of which more in a minute). If timeoutPromise is fulfilled first the race returns nothing. So I can test to see which promise "won" the race as follows:

if (value === undefined) {
  // The timout has won the race - leave the read hanging and 
  // abandon this loop.
  console.log("Timeout");
  timeout = true;
  break;
}

If there is nothing in the value return (which contains the data returned by read) then the timeout promise has been fulfilled. If this is the case we print a message, set a flag and then break out of the loop which is controlling the read process. The read promise is still "out there doing its thing". My code can handle the timeout (perhaps by re-sending the message to the device) and then come back and run another "promise race" against a new timeout.

If the read promise is fulfilled first the program has got some data to deal with. But before it handles the data it has some tidying up to do.

// Clear the timeout timer
clearTimeout(this.timeoutID);

The first thing that the program must do is turn off the timeout promise. Remember that timeoutID that we grabbed earlier? We feed that into the clearTimeout function to clear it because we no longer need the timeout.

The next piece of tidying up is where we make ready for the next read operation. It took me a while to get my head around this bit, and I'm still not completely sure, but the code I've written works, so I'm standing by my reading of what is happening....

As far as I can tell a reader is good for one read. Once you've read from it you must give it away and then get a new one for next time. If you re-read the contents you will get back exactly the same content. This can lead to a lot of confusion as your program keeps getting back the same data. In my experience the next thing that happens is that the fans on your computer start to run at full speed, the web page becomes unresponsive and you have to close the browser tab to regain control again. So, once the reader has worked we have to throw it away. The same goes for the promise:

// Clear down the reader that we have just successfully used
// so that the loop will make another one. 
// Note that if we timeout these statements are not performed
// and the existing reader and promise is used next time 
// getSLIPpacket is called

this.reader.releaseLock();
this.reader = null;
this.readPromise = null;

The way the code is sequenced means that if we get a timeout it will just reuse the reader and the promise that have not completed yet. So I can use this cleverness to try to read data from the serial port and get control when nothing arrives.

How I use this, and what form the data takes will have to wait for another day..