We’ve set the following objectives to guide us in the design of the actual data logging stage:

1. Samples should be timestamped with absolute time
2. Different nodes should start/take each sample at the same time (within a certain degree of error)

As such, to simplify setup and maintain accuracy of sample times over a long period of time, we decided to attach a GPS module to the board.

Additionally, we also identified the following methods we can use to save power:

1. Power off the GPS/SD Card when not in use
   1. Power on the SD Card when necessary:
      1. Client attempts to read file
      2. Samples have to be written to a file
   2. Power on the GPS at a fixed interval, say, every hour, synchronise time, then power off again
2. Power on the transceiver on the ESP32 at a low duty cycle eg for 1s every minute
3. Put the ESP32 into light-sleep mode between samples

With these in mind, our firmware does the following for sampling:

1. Wait until a rising edge is seen on the PPS signal from the GPS module
2. Configure the Ultra-Low Power Processor on the ESP32 to wake then halt immediately at a fixed period. This is for two reasons:
   1. A wake fires a ISR on the main cores which will then take a sample from the sensor
   2. This wake-halt sequence can be clocked by the 32kHz crystal for the RTC (the main crystal is disabled in light-sleep mode)
3. Samples are buffered to reduce the number of write operations needed, making use of the 1MB of PSRAM on the WROVER module

Samples are buffered into two alternating buffers (ie samples are written to one until full. While the first buffer is written to file, samples are written to the second buffer). Buffers are prefixed with the following header:

1. HDR as a header marker (3 bytes)
2. Number of microseconds since Unix Epoch (uint64_t, 8 bytes)
3. Size of one sample in bytes (1 byte)
4. Number of samples following this header (uint32_t, 4 bytes)

The header and buffer are then written to the SD card. A crude version of the workflow described above has been implemented. However, multiple issues still remain:

1. The RTC is still clocked by the internal 150kHz RC oscillator – when we configure the RTC to use the 32kHz crystal, ISRs are fired much faster than expected (eg the ISR fires at 50Hz when clocked by the internal oscillator, but fires at 250Hz when clocked by the 32kHz crystal). This appears to be a software issue – we’ve verified that the crystal is presenting a 32.768kHz signal to the ESp32 by probing with an oscilloscope.
2. An ISR was configured for rising edges on the PPS signal. The ISR logs the current absolute time (in us) before updating the system time (based on the PPS info). The output is as follows:

I (497416) logging_task: pulse: 1600768700999984
I (498416) logging_task: pulse: 1600768701999956
I (499416) logging_task: pulse: 1600768702999980
I (500416) logging_task: pulse: 1600768703999980
I (501416) logging_task: pulse: 1600768704999987
I (502416) logging_task: pulse: 1600768705999972
I (503416) logging_task: pulse: 1600768706999987
I (504416) logging_task: pulse: 1600768707999974
I (505416) logging_task: pulse: 1600768708999980
I (506416) logging_task: pulse: 1600768709999972
I (507416) logging_task: pulse: 1600768710999980
I (508416) logging_task: pulse: 1600768711999975
I (509416) logging_task: pulse: 1600768712999972
I (510416) logging_task: pulse: 1600768713999980
I (511416) logging_task: pulse: 1600768714999980
I (512416) logging_task: pulse: 1600768716000006
I (513416) logging_task: pulse: 1600768716999954
I (514416) logging_task: pulse: 1600768717999980
I (515416) logging_task: pulse: 1600768718999988
I (516416) logging_task: pulse: 1600768719999972

The time logged varies substantially from line to line (even though the system time is slower, we still expected a consistent difference). A quick cursory examination here suggests that the system time is about 20us slower than the time reported by the GPS module for every second that passes. Over an hour, this results in a difference of 72ms (this presents issues if we only power on the GPS module for time syncing once in a while). We’ve came up with a few possible reasons for this:

1. Inaccuracies in the main crystal/internal RC oscillator (due to the issue discussed above)
2. Interrupt latency (can it really vary that much?)
   1. This was discussed here, however, switching to a FPGA or another microcontroller is not possible at this point in the project
3. The PPS signal that our GPS module outputs is not accurate enough. The datasheet specifies a deviation of less than 1us, if the module performs to spec, this cannot be the cause.

More work is needed to resolve these issues. However, even with these limitations in mind, it should be possible to present a full proof-of-concept of what we set out to do.

Since we now have multiple nodes which can read/write to SD cards (only one of the hand soldered prototypes worked well, the other had signal integrity issues), we can finally write received data from the client to the SD card.

Current Software Architecture

For the sake of simplicity, handling of received packet was done entirely in the ESP-NOW ISR. This worked fine when MtftpClient did not actually write contents to a file (but rather just logged metadata).

However, once MtftpClient started writing to the SD card, the write operations took a substantial amount of time, leading to severe packet loss (as the ESP-NOW ISR took a long time to exit). The obvious solution is to push received data to a ring buffer, then MtftpClient::loop() reads from this ring buffer and actually writes the data to the SD card. While this works fine, we run into yet another issue here – since the server is transmitting data packets as fast as it can read from the SD card, the client cannot actually write data to the SD card fast enough. This results in the ring buffer overflowing.

Rather than bolting on flow control, we can refactor the client to handle incoming packets differently – all received packets are pushed to a ring buffer, then MtftpClient::loop() parses incoming packets and writes them to the SD card as seen below.

Refactored Architecture

By refactoring the packet handling on the client in this way, we solve two problems – firstly, we no longer perform long operations in the ISR. Secondly, if we make the ring buffer larger than the maximum size of the data transferred in a window (plus a bit of overhead for the ring buffer itself to manage the items), we can now buffer an entire window worth of packets. MtftpClient::loop() will only send an ACK once an entire window has been processed successfully, which will only happen once all the writes are done.

This also doubles up as flow control:

1. The server will send an entire window at once
2. The client buffers (and requests retransmission of missing blocks if necessary) the entire window in memory
3. The client slowly reads from the ring buffer and writes data to the SD card
4. The client acknowledges the window once all data has been processed
5. The server continues transmission (ie go back to Step 1)

With this implemented, we can fully test the communication between the client and server. A test with 2 nodes next to each other transferring a 1Mb file was successful, the transferred file has been confirmed to be the same by hashing it and comparing that to the original file.

However, in a realistic environment, there would be interference from the environment (whether from Wi-Fi networks, or even from our UAV remote control equipment), which will cause packets to be lost (but not corrupted, the frame check sequence ensures that data is valid). This presents problems especially when key signaling packets (eg ACK) are lost, and one side is left waiting forever for a packet that was already lost. Although the current time-out functionality will handle these cases, relying entirely on the time-out to handle missing packets affects throughput severely. Certain lost packets (eg the server’s SYNC response) can be detected by the other party and handled immediately without having to wait for the timeout.

Identifying these cases and handling them separately should improve throughput.

Separately, multiple bugs were fixed:

1. Race condition between onTimeout call and loop
2. The counter used to track buffered ESP-NOW packets was observed to underflow then lock-up the entire system since it never fell below MAX_BUFFERED_TX.  This was fixed by using a counting Semaphore instead of just a variable.

On our most recent test flight last Sunday, we strapped the collector to a UAV and monitored the ground node. Although none of the nodes locked up this time, the data written to the SD card by the collector was corrupted. More debugging is needed to identify and fix this issue.

Edit: The data corruption observed in data written by the collector has been fixed. file_offset is incremented after every write of an in-order, non-buffered block, but writing the buffer to file after handling missing blocks did not increment file_offset.