Serial conversations in JavaScript

Well, I've put it off as long as I could. I've delved into SLIP and DTR lines and allsorts, but today I have to get my head around the way that the JavaScript usb serial library sends and receives data. Way back on day 2 we discovered how a JavaScript application can get a reference to an object that represents a connection to a serial port, now I actually have to use it to move data.

try {
  this.port = await navigator.serial.requestPort();
  await this.port.open({ baudRate: 115200, bufferSize: 10000 });
}
catch (error) {
  return { worked: false, message: `Serial port open failed:${error.message}` };
}

For a refresher, this is what you do to open a port. At the end of this, if the user selects a port properly, you end up with a port object (in a member variable called this.port). We can ask this object to give us reader and writer objects (if it is able to) which we can then use to read and write serial data. The data is always sent and recieved in UInt8Array objects.

Sending serial data

Sending serial data is easy.

async sendBytes(bytes) {
  let buffer = "";
  let limit = bytes.length<10?bytes.length:10;
  for(let i=0;i<limit;i++){
    buffer = buffer + bytes[i].toString(10)+":"+bytes[i].toString(16)+"  ";
  }
  console.log(`Sending:${buffer}...`);
  const writer = this.port.writable.getWriter();
  await writer.write(bytes);
  writer.releaseLock();
}

This is my sendBytes method. You give it a UInt8Array and it sends it out. It also prints out the first 10 bytes of the array on the console so you can take a quick peek at what is being sent. Note that it uses the writable property of the port which provides a method called getWriter() that delivers a write object that I called writer. I use this to write the bytes and then release the lock on this writer, effectivly discarding it and making it possible for other code to grab a write and write something. Note also that all of this is awaited so that the writing of the data can take place while other parts of JavaScript keep going.

Receiving serial data

Receiving serial data in JavaScript is hard. The main reason for this is the way that on the whole JavaScript takes a very dim view of programs that hang around and wait for stuff. Python doesn't care so much. In pytool, when the program wants to read something from the serial port it just calls the read method on the serial port and waits for a response. Either something will arrive (good) or nothing will arrive (bad). If nothing arrives the serial port will time out at some point in the future and tell the program that nothing was received. The program can note this and give the remote system another prod to see if it is awake or display a message or whatnot. In the case of JavaScript it looks quite easy to read data:

this.reader = this.port.readable.getReader();
const { value, done } = await this.reader.read();

The tricky bit is not the way that reader returns a result. It returns two items:

  • value - the buffer of data you were waiting for
  • done - a flag that is set to true if the reader is finished (perhaps it has been closed or the user has unplugged their device)

The syntax of the call above means that when the reader has finished it will return those two values which my program can then test. If done is true I shut everything down and go home. Otherwise I take a look at what is in value, which will be a UInt8Array full of bytes that have been received from the device. The tricky bit is that the read operation will not return untill it has received something. There is also added trickiness, in that if anything bad happens to the data stream the read operation will throw an exception that I need to handle. Bad things that might happen include remote devices sending badly formed serial data, just the sort of thing that happens when you reset something.

So, my code must handle the fact that read may never return (the remote device might not have detected the right baud rate after reset) and the fact that read may explode with an exception, at which point I need to tidy everything up. Tricky indeed.

I quite like a cliff hanger ending. And I've got other things to do today as well as code. So I'll leave it at that for now.

Given the SLIP

Serial Line Internet Protocol goes back a long, long way. Back to the early days of the internet. The original internet connections used packet based transport protocols. What does that mean? Well, it means that data was sent in lumps. A lump had a header, the data and probably a checksum at the end. EtherNet connections (which were shiny new and used to underpin the early internet) shouted packets into the ether (actually a wire) and then waited for responses that made sense.

This was all fine and dandy for connecting together machines in the computer room, but what if you were using a serial connection? On a serial connection the unit of transmission is a single 8 bit byte, not a lump of stuff. If all you can receive is individual bytes, how do you know when you've got a packet's worth?

Enter SLIP. This is a protocol that lets you send blocks of data down a channel which can only send individual bytes. It works by nominating a character called "Frame End" (0xc0) that marks the end of a packet. A receiver can know that it has received a complete packet when it receives a frame end. Of course, the next question is "How do we send the character 0xc0 in a stream of data that could contain any 8 bit value". We do this by nominating another character as "Frame Escape" (0xdb). This character marks the start of an "escape sequence". There are actually only two escape sequences. One sequence (0xdb 0xdc) means "this is actually the character 0xC0" and the other sequence (0xdb 0xdd) means "this is actually the character 0xdb". (I'm giving my numbers in hex to show that I'm a real computer person. For the record 0xc0 is the value 192)

In olden, olden, times you would buy a modem so you could connect your home computer to a distant server and then you would have to write a SLIP driver for your computer that could assemble the internet packets recieved froom the phone line and then pass them into the TCP/IP stack on your home computer. Happy days.

But what has this got to do with the price of fish, or even sending data to an ESP device? Well, it turns out that ESP devices use SLIP as the protocol for their link to the PC. That packet that I carefully assembled yesterday now has to be made into a SLIP packet so that I can send it to the device. This is actually quite a simple piece of code. It has to do three things:

  1. Look for the character 0xC0 in the input data and convert it into the sequence 0xDB 0xDC.
  2. Look for the character 0xDB in the input data and convert it into the sequence 0xDB 0xDD.
  3. Put the character 0xC0 on the end of the data packet.

There's a function in pytool (at line 394) that does this conversion:

def write(self, packet):
    buf = b'\xc0' \
            + (packet.replace(b'\xdb', b'\xdb\xdd').replace(b'\xc0', b'\xdb\xdc')) \
            + b'\xc0'
    self.trace("Write %d bytes: %s", len(buf), HexFormatter(buf))
    self._port.write(buf)

It uses some neat Python features that allow replacment of elements in arrays. You can see how it swaps the escape characters.

packSlip(message) {
    // find out how many extra spaces we need for escape
    // count the 0xdb and 0xc0 values in the message
    let extra = 0;
    message.map((i) => { if ((i == 0xdb) || (i == 0xC0)) extra++ });

    let inputLength = message.length;

    // add in extra plus space for 0xc0 at start and end
    let totalLength = inputLength + extra + 2;

    let out = new Uint8Array(totalLength);

    let wPos = 0;

    out[wPos++] = 0xc0;

    for (let b of message) {
        switch (b) {
            case 0xdb:
                out[wPos++] = 0xdb;
                out[wPos++] = 0xdd;
                break;

            case 0xc0:
                out[wPos++] = 0xdb;
                out[wPos++] = 0xdc;
                break;

            default:
                out[wPos++] = b;
        }
    }

    out[wPos++] = 0xc0;

    return out;
}

This is my JavaScript slip packing code. You give it an array of byte values and it gives you back a slip encoded string with 0xc0 on the end (and at the beginning too - which clears out any nastiness with old characters lying around in the input buffer of the receiver).

When you start using UInt8Arrays you discover that all the nice dynamic array stuff in JavaScript (which will make an array automatically grow when you put things it it) has disappeared. You have to make the array the exact length you need it to be.

I use the map function to count the number of characters that need to be encoded and use this to work out how long the result array needs to be. Then I spin through the array and do the encoding.

There are almost certainly much more efficient and cunning ways of doing this, but the above code does work. So now I have a nice block of data that I want to send to the ESP device. And then I'll want to see what comes back...

Talking to the ESP

Before I try to send any messages, perhaps I'd better try to discover what the messages are. This seems to be a good place to find out what is going on:

https://github.com/espressif/esptool/wiki/Serial-Protocol

Each command is a SLIP packet (ah - that brings back memories) which starts with a 0 which is followed by an opcode, size field, checksum and then a bunch of data. If I dig into the esptool.py file and at line 425 I find the command method, which is used in the program to send a command to the ESP device. After a bit of setup stuff I find this statement:

pkt = struct.pack(b'<BBHI', 0x00, op, len(data), chk) + data

Aha. this looks like the statement that builds a command. What does struct.pack do?

https://docs.python.org/3/library/struct.html

It's all explained here. It is how Python manages the movement of data from variables in the program into bytes of data. The '<BBHI' data-preserve-html-node="true" string tells pack the format of the data it is to pack, in this case it says:

  • < - use "little endian" (in other words when assembling values over more than one byte put the lower bytes first)
  • B - pack a single 8 bit value - a byte
  • H - this is an unsigned "short" int that is stored in 16 bits, that is two bytes of data
  • I - this is an unsigned int that is stored in 32 bits, that is two bytes of data.

From this I can work out that the 0 at the start and op code are single byte values, the length of the data is sent over two bytes and the checksum is sent over four bytes. The statement above that assembles the packet then puts a thing called "data" on the end, which must be the block of data to be sent.

Assembling a command

JavaScript handles 8 bit data using a special type called a Uint8Array. This is my first attempt at packing a command into such an array.

packCommand(opCode, dataBlock, check) {
    let dataLength = dataBlock.length;
    let totalLength = dataLength + 8;
    const message = new Uint8Array(totalLength);
    let pos = 0;
    message[pos++] = 0;
    message[pos++] = opCode;
    message[pos++] = dataLength & 0xff;
    message[pos++] = (dataLength >> 8) & 0xff;
    message[pos++] = check;
    pos += 3;
    for (let i = 0; i < dataLength; i++) {
        let value = dataBlock[i];
        message[pos++] = value & 0xFF;
    }
    return message;
}

The packCommand method takes in an opcode, a block of data and a checksum value. It returns a Uint8Array with the command at the top and the data. It splits out the parts of each value by masking off the required bits and shifting them into position. It's probably not an optimal way of doing this, but it should work.

Sending a sync

The first command that is sent is a sync message that allows the host computer and the ESP device to talk to each other. Serial data (which is what we are using to exchange messages) is sent at a particular speed, called the baud rate. We set the baud rate when we created our JavaScript serial connection. Now we have to send a message to the ESP device so that it can work out what baud rate we are using and then start receiving at that rate. The communications protocol has a sync message which contains data values which have been chosen to produce a nice sequence of pulses for the ESP device to look at and work out the data rate. I've had a look at the packet and carefully assembled the data that we need. This is my sync command in JavaScript:

async sync() {
    console.log("Syncing the connection");
    const params = {
        op: ESPToolJS.ESP_SYNC,
        dataBlock: [0x07, 0x07, 0x12, 0x20,
            0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55,
            0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55,
            0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55,
            0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55],
        check: 0,
        waitResponse: true,
        timeout: ESPToolJS.DEFAULT_TIMEOUT
    };
    let { val, data } = await this.command(params);
    return data;
}

I've copied all the command numbers into the program as static values, the value of ESpToolJS.ESP_SYNC is 0x08, which is the sync command opcode. The datablock is not delivering any data as such, but it does contain a lot of 0x55 values. If you look at the binary for the value 0x55 you find that it is 0101010101, which means that the serial data signal will bounce up and down in a square wave which is perfect for the ESP to use to work out the serial data speed.

This sync configuration means that the first sync command from the computer probably won't get a response, as the ESP device will use this to work out what speed it should listen. The pytool code sends multiple sync commands, so I'll have to as well.

Serial Reset

Task of the day is get a JavaScript program running in a browser to send messages to a serial port. And maybe even reset an ESP device using the hardware handshake lines. Then I can start having a conversation with a device. This website tells me how to make a serial connection from a browser:

https://web.dev/serial/

I’m going to make a little SerialManager class to manage the connection to the serial port.

Connecting to a serial port

Let’s have a look at the function that makes the connection to the device:

async connectToSerialPort() {
  // Prompt user to select any serial port.

  if (!"serial" in navigator) {
    this.port = null;
    return { worked: false, message: "This browser doesn't support serial connection. Try Edge or Chrome." };
  }

  try {
    this.port = await navigator.serial.requestPort();
    await this.port.open({ baudRate: 115200, bufferSize: 10000 });
  }
  catch (error) {
    return { worked: false, message: `Serial port open failed:${error.message}` };
  }

  return { worked: true, message: "Connected OK" };
}

When the function runs it pops up a dialog asking the user to select a serial device. Then it creates a connection. Note that I'm using a funky error return mechanism where the function returns whether or not it worked along with a message. I can use this to get a serial connection and it seems to work.

Resetting the ESP device

Next we need to reset the ESP device into upload mode. This seems to involve some fancy footwork with the hardware handshake lines. You can find the code in the esptoo.py file at line 564:

self._setDTR(False)  # IO0=HIGH
self._setRTS(True)   # EN=LOW, chip in reset
time.sleep(0.1)
if esp32r0_delay:
    # Some chips are more likely to trigger the esp32r0
    # watchdog reset silicon bug if they're held with EN=LOW
    # for a longer period
    time.sleep(1.2)
self._setDTR(True)   # IO0=LOW
self._setRTS(False)  # EN=HIGH, chip out of reset
if esp32r0_delay:
    # Sleep longer after reset.
    # This workaround only works on revision 0 ESP32 chips,
    # it exploits a silicon bug spurious watchdog reset.
    time.sleep(0.4)  # allow watchdog reset to occur
time.sleep(0.05)
self._setDTR(False)  # IO0=HIGH, done

It uses the DTR and RTS lines (which are normally used for handshaking the serial connection) to reset the chip. Not all ESP devices have these connected, but the Wemos devices that I want to specifically target with my program do. You can kind of see what they do:

  1. drop DTR
  2. raise RTS
  3. wait a while
  4. raise DTR
  5. drop RTS
  6. drop DTR

This combination of changes means "please start the chip in bootloader mode". Here's my JavaScript interpretation:

async resetIntoBootloader() {
  console.log("Resetting into the Bootloader");
  await this.port.setSignals({ dataTerminalReady: false });
  await this.port.setSignals({ requestToSend: true });
  await this.delay(100);
  await this.port.setSignals({ dataTerminalReady: true });
  await this.port.setSignals({ requestToSend: false });
  await this.delay(50);
  await this.port.setSignals({ dataTerminalReady: false });
  console.log("Reset into bootloader");
}

This does exactly the same sequence. It uses a little function called delay that I've made that will pause the program for a while.

async delay(timeInMs) {
  return new Promise(async (kept, broken) => {
    setTimeout(async () => {
      return kept("tick");
    }, timeInMs);
  });
}

Note that the JavaScript does a lot of awaiting. I haven't really got time to go into this in detail, suffice it to say that the allows my code to stop for a while without the webpage that it is part of getting stuck. Anyhoo, it seems to work, in that the ESP device stops when I hit it with the reset sequence. Tomorrow I'm going to try sending some commands.

Playing with esptool

So, where to start? The pytool program is a large chunk of Python that I know nothing about. I know what it does: it takes files off my PC and squirts them into ESP32 and ESP8266 devices. However, I’ve no idea how it works.

Fetching esptool

The good news is that the program source is a free download from here:

https://github.com/espressif/esptool

I used the GitHub desktop tool to clone this and then opened it with Visual Studio code. The first part of my plan is to get it running and then use the Python debugging tools to find out what it does.

Sending a command

The first thing I have to do is get pytool to do something. I settled for just reading the MAC address from a device. I’m using version 3.2 (which is interesting because it came out after the date of this blog post - I’ll leave you to figure out how that works). With this version you can “hard-wire” commands into the program so that it performs them when it starts. You have to find the main function in the Python program and add some arguments. In the version I’m using I found the main function at line 3789 in the esptool.py program file:

def main(argv=[ 'read_mac'], esp=None):

I've added the argument “read_mac”. When I run the program I now get this:

esptool.py v3.2-dev
Found 2 serial ports
Serial port COM12
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: ec:fa:bc:28:56:27
Uploading stub...
Running stub...
Stub running...
MAC: ec:fa:bc:28:56:27
Hard resetting via RTS pin...

The worrying part of this conversation is the message “Uploading stub”, but we’ll get to that later. For now, lets figure out how we can get a JavaScript program to reset a remote device and talk to it over the serial port.