As mentioned in prior posts, we implemented a protocol based off TFTP, which we call Modified Trivial File Transfer Protocol (mtftp). The diagram above illustrates how it works:

1. The Client (collector on the UAV) sends a Read Request (RRQ), which specifies what file (file index) and where in the file (file offset) to transmit. A window size is also transmitted which indicates how many data packets the Server should send before waiting for a response.
2. The Server then sends up to window size DATA packets to the Client (the end of file (EOF) is indicated by sending a partial DATA packet, or zero bytes if the file size is a multiple of block length).
3. The Client then sends an Acknowledge (ACK) packet with the block number of the largest block it successfully received.
   1. This allows for correction of missing blocks: for example, if the Client receives block number 0, 1, 2, 4, 5, 6, 7 with a window size of 8, the Client can detect that block 3 is missing, sending an ACK for block 2.
   2. If the Client receives a block of less than 247 bytes, this indicates the end of file. For example, if the client receives block 0, 1, 2, 3, where block 3 contains 200 bytes, it will not expect any further blocks and sends an ACK for block 3.
   3. If the final block in a window (ie block 7 when window size = 8) is missing, the client will never know that the window has ended. In this case, both the client and server will timeout after a short delay, terminating the entire transfer.
4. If EOF has not been reached, the Server then advances the file offset by the number of bytes acknowledged, then starts a new window with block number 0.

We use the following terms:

1. Block – A single chunk of data in a DATA packet. The maximum length of a block is fixed to 247 bytes in our application since ESPNOW has a maximum payload length of 250 bytes, and we use 3 bytes (1 for opcode, 2 for block number) as a packet header.
2. Window – Transmission of up to window size DATA packets before a response is expected from the client.
3. Transfer – The transfer of multiple Windows to transfer a file until EOF.

Initial tests demonstrates that the above works in close proximity, but when the Client and Server are far from each other in an urban environment (SPMS Atrium), packet loss is non trivial. As mentioned above, if the Client receives block 0, 1, 2, 4, 5, 6, 7, it sends an ACK for block 2 and the Server starts transmitting the data at block 3 again. This is wasteful – the Client has already received data for block 4, 5, 6, 7.

The flowchart on the left illustrates the initial implementation of the mtftp client. Solid lines depict flows involving actual packet transmission. Blocks received in-order are written to disk (block numbers 0 and 1 in the example at the bottom). However, when a block is missing (block number 2), all future blocks are discarded (block number 3, 4, 5, 6, 7, 8) because we cannot save it to file – there is no guarantee we’ll receive the missing block, and even so, we would need a way to mark that block as missing. Rather, the client sends an ACK for block 1 (the last in-order block received) and the server resumes transmission at block 2. This is obviously inefficient especially when the window size is large, say, 32 blocks. If block 1 is lost, 30 other blocks that may be received successfully are discarded.

Our solution to this is to introduce re-transmission – missing block numbers are stored and subsequent blocks are buffered. This is illustrated by the yellow blocks in the flowchart on the right. At the end of the window, the client transmits a RTX packet containing the missing block numbers. The server then re-transmits these blocks, and the client writes the initially missing blocks along with the buffered blocks to disk. This eliminates the issue described above.

However, there is still another issue here – if the RTX or ACK packets (depicted by the thick black lines in the flowchart on the right) are lost, the communication times out. If it was a RTX requesting for one of the earlier blocks in the window, most of the window will be discarded again. A possible fix for this is for the client to re-transmit a RTX or ACK should communication time out in at this portion. However, a quick implementation proved to be buggy and was reverted. We will revisit this issue later as we we would like to implement other functionality first.

One of the objectives we have been working towards is ensuring samples from different nodes are time-synchronised (ie sampled at the same point in time). As mentioned here, GPS receivers are one method to obtain accurate timestamps.

The commonly available U-blox Neo-6M GPS receiver outputs a 1 Hz timepulse, which is configurable between 0.25 Hz to 1 kHz. It would be entirely possible to configure the timepulse frequency to match our target sampling frequency, then use an edge of the timepulse to start sampling. However, keeping a GPS receiver constantly powered on has severe implications on battery life.

As presented in Multi-Channel Data Acquisition System with Absolute Time Synchronization:

Measurement of the last (1000th) sample within a given second disables the timer/counter reset (it still increases every clock cycle) and introduces the microcontroller into the awaiting mode (waiting for the next synchronization pulse). When the pulse arrives, the timer/counter value is first read and then it is immediately reset. If the local clock has its nominal frequency (48 MHz), the read value should be equal to C, which corresponds to 1 ms. If it is not, the difference between C and the number of measured cycles is used to adjust C. This technique compensates for possible drifts of the local-oscillator frequency, i.e., it ensures that recorded samples are uniformly distributed over time with 1 ms spacing between them.

We can tweak this slightly by assuming that the drift of our local oscillator is relatively fixed and will not change that rapidly. This allows us to compute C periodically, say, every hour by powering on the GPS receiver, getting a few timepulse edges, then powering off the GPS receiver. The sampling rate until the next synchronisation to GPS time will then be based off the computed value of C.

The use of an ESP32 here complicates things:

1. Hardware timers are available. Although these timers support periodic mode (ie fire a ISR at a certain interval), they are clocked by the APB clock which may not be stable over time.
2. A RTC timer is available, which supports the use of an external 32.768 kHz crystal, but there is no support for using this timer to fire interrupts on the main SoC

With these constraints in mind, we propose the following method of synchronising the time at which sensors are sampled:

1. Start a hardware timer
2. Wait for at least 2 timepulse edges, store the difference of hardware timer value at each edge as C
   1. Set system (RTC) time based on the timepulse and NMEA sentences (use this to mark time when storing samples)
3. C represents the number of timer counts (based off APB clock) in a 1 second interval, start a periodic hardware timer with period = C / target sampling frequency
   1. Configure the periodic ISR to start sampling of sensors

A quick experiment suggests that for the particular ESP32-DevKitC that was used, C is approximately 999990:

I (81745) sampling_task: pulse: 81362899
I (82745) sampling_task: pulse: 82362890
I (83745) sampling_task: pulse: 83362880
I (84745) sampling_task: pulse: 84362870
I (85745) sampling_task: pulse: 85362860
I (86745) sampling_task: pulse: 86362850
I (87745) sampling_task: pulse: 87362840
I (88745) sampling_task: pulse: 88362830
I (89745) sampling_task: pulse: 89362820
I (90745) sampling_task: pulse: 90362809
I (91745) sampling_task: pulse: 91362799
I (92745) sampling_task: pulse: 92362789
I (93745) sampling_task: pulse: 93362779
I (94745) sampling_task: pulse: 94362769
I (95745) sampling_task: pulse: 95362759
I (96745) sampling_task: pulse: 96362749
I (97745) sampling_task: pulse: 97362739
I (98745) sampling_task: pulse: 98362729
I (99745) sampling_task: pulse: 99362719

The observed drift of approximately 10 us every second would result in 36 ms of drift over 1 hour, and 864 ms over 24 hours. Given that our target sampling frequency of 80 Hz has a period of 12.5 ms, if uncorrected, this drift would be an issue.

Further testing for a longer period of time is necessary to establish whether the amount of drift is constant (and probably temperature dependent). Exploring the drift of the RTC clock (once our PCBs with a RTC crystal arrive and are assembled) may also provide alternative possibilities for synchronisation.

The below speeds were observed using a SanDisk Ultra 16Gb A1 MicroSD Card on a hand-wired prototyping board with 2Mb files.

The very first iteration of the readFile function was an naive implementation:

1. Open file pointer to the requested file
2. Seek to the requested offset
3. Read requested number of bytes
4. Close file pointer
5. Return data

This was slow: it took 11953726 to transfer 1048579 bytes (85.6kbyte/s).

The obvious optimisation that can be made here is to persist the file pointer between calls to readFile:

1. Check if file pointer is open, if the current file pointer is for the correct file
   1. If not, close the current file pointer and open the requested one
2. Seek to the requested offset
3. Read requested number of bytes
4. Return data

This improved speeds slightly to 124.2kbyte/s.

We can make use of a byte buffer to buffer data from the SD card, allowing us to read in large chunks:

1. Attempt to read requested number(usually 247) of bytes from the buffer
   1. If successful, return data
2. Else, read (up to) 8192 bytes from the SD card and write it into the buffer
3. Read up to requested number of bytes from the buffer again
4. Return data

This improves speeds to 229.8kbyte/s.

To prevent buffering too many ESP-NOW packets (or Wi-Fi memory allocations start to fail), code was written to track the number of buffered packets. This code issued a 10ms delay if there were too many buffered packets. It was observed that most of the time after the delay, there were 0 buffered packets left. This means that the 10ms delay might be too much, resulting in dead time where no more data was being sent. Removing this delay improved speeds to 338.8kbyte/s.

While writing this post, we came across this comment which suggested using setvbuf to buffer reads. Using this over our home-brewed ring buffer solution improves speeds substantially to 440.7kbyte/s. This isn’t too far from the 549250bytes/s we observed when testing transfers speeds without doing any processing, though we need to test this out in the field to confirm its performance.

Now that we’ve started on the protocol for file transfer, all we have to do is to attach it to ESP-NOW and the SDMMC controller for data transmission and retrieval respectively as illustrated in the diagram below.

We soldered a quick prototype of a SD Card socket connected to a ESP32-DevKitC:

For the software, we add another layer over mtftp to manage sessions: the collector broadcasts a synchronisation packet. The node(s), upon receiving this packet, echos this packet back to the collector and adds the collector as an ESP-NOW peer (necessary for unicast communication between two ESP-NOW devices). Upon receiving the echoed synchronisation packet, the collector also adds the node as an ESP-NOW peer and hands over control to the mtftp handlers.

Output from the node (top) and collector (bottom)

The software discussed above can be found here.