The below speeds were observed using a SanDisk Ultra 16Gb A1 MicroSD Card on a hand-wired prototyping board with 2Mb files.

The very first iteration of the readFile function was an naive implementation:

1. Open file pointer to the requested file
2. Seek to the requested offset
3. Read requested number of bytes
4. Close file pointer
5. Return data

This was slow: it took 11953726 to transfer 1048579 bytes (85.6kbyte/s).

The obvious optimisation that can be made here is to persist the file pointer between calls to readFile:

1. Check if file pointer is open, if the current file pointer is for the correct file
   1. If not, close the current file pointer and open the requested one
2. Seek to the requested offset
3. Read requested number of bytes
4. Return data

This improved speeds slightly to 124.2kbyte/s.

We can make use of a byte buffer to buffer data from the SD card, allowing us to read in large chunks:

1. Attempt to read requested number(usually 247) of bytes from the buffer
   1. If successful, return data
2. Else, read (up to) 8192 bytes from the SD card and write it into the buffer
3. Read up to requested number of bytes from the buffer again
4. Return data

This improves speeds to 229.8kbyte/s.

To prevent buffering too many ESP-NOW packets (or Wi-Fi memory allocations start to fail), code was written to track the number of buffered packets. This code issued a 10ms delay if there were too many buffered packets. It was observed that most of the time after the delay, there were 0 buffered packets left. This means that the 10ms delay might be too much, resulting in dead time where no more data was being sent. Removing this delay improved speeds to 338.8kbyte/s.

While writing this post, we came across this comment which suggested using setvbuf to buffer reads. Using this over our home-brewed ring buffer solution improves speeds substantially to 440.7kbyte/s. This isn’t too far from the 549250bytes/s we observed when testing transfers speeds without doing any processing, though we need to test this out in the field to confirm its performance.