One of the main highlights of the Hoverfish is the lightweight but versatile Raspberry Pi Zero W, which allows for rapid prototyping and a large range of features to be provided by the Hoverfish. This post will cover the development process involving the Raspberry Pi.
How we ended up with the Raspberry Pi
Originally, the Hoverfish had intended to use an Arduino to control the motors and servos to be mounted on the Hoverfish. A rudimentary code had been tested on an Arduino Uno together with components available in the Arduino Starter Kit simply to show control of the servos and motors were possible. From here we had intended to switch over to the Arduino Nano, acquiring a Wi-Fi chip for wireless communications. However, further research into how we were intending to stream video from the Hoverfish revealed that the Arduino Nano simply wasn’t going to make it. The primary concern was that it did not have the processing power to encode images at a rate fast enough to provide a working video feed.
Henceforth, we opened our search to consider a wider range of microprocessors outside the Arduino brand. In consideration were the Orange Pi Zero, an open-source single board computer and the Raspberry Pi series due to their significantly higher processing power and range of features. Between the two, we decided to go with the Raspberry Pi due to its greater support in terms of hardware and development as we were new to such technologies. In particular, we had chosen the Raspberry Pi Zero W for its small size and weight. The benefits of the Raspberry Pi Zero W were the inbuilt Wi-Fi modules present on newer versions as well as easy to install camera modules. As a result, we were able to cut down on the interfacing required, on top of the number of pieces of hardware to include onto the Hoverfish. By choosing the Raspberry Pi, we were able to do away with both the Wi-Fi shield and USB webcam, borrowed from the Making and Tinkering (MnT) lab, as well as installing and incorporating the webcam into our code. Additionally, as the USB webcam required an open USB port and was mounted on a rigid PCB, it was bulkier and unsuited for our needs as compared to the more Raspberry Pi camera module, mounted with a ribbon cable.
Setting up the Raspberry Pi
Having decided on the hardware we would be using for our project, we borrowed a Raspberry Pi 3 from the MnT lab, together with a touchscreen to begin the development process. The touchscreen not only enabled us to observe what was going on with the Pi 3 directly without an extra monitor and HDMI cable, it also allowed us to interact with the Pi 3 without connecting a USB mouse. This made the Pi 3 a convenient platform for testing our software even in later stages of the project, when we had begun implementing it on the Pi Zero W. Our first task was to install an Operating System (OS) onto the Pi 3, and we chose to go with Raspbian, the standard OS shipped with the NOOBS starter package provided by the Raspberry Pi foundation on their website. Installing Raspbian was relatively simple, and was done by extracting the NOOBS package onto a 32GB microSD card provided by Ryan, inserting it into the Pi 3, powering it up and selecting the Raspbian OS. After a short wait, we were greeted by a rather friendly UI, albeit a slightly laggy one. Given that we had selected the Raspberry Pi due to its wireless capabilities, our next task was to enable the Pi to be remotely enabled.
To do so, we first had to connect to a wireless network. It was at this juncture that we learnt that the Pi was only able to connect to WPA-Personal networks such as our personal Wi-Fi networks or the NTUWL network due to the encryption configuration of WPA-Enterprise networks such as NTUSECURE or eduroam. This turned out to be an issue as we wanted a prevalent Wi-Fi network such that we could test and run the Hoverfish anywhere, yet the only option, NTUWL required one to log in at regular intervals, which would be a problem when the Pi was to be controlled remotely in headless mode. An additional benefit to connecting the Pi to such networks would be that it could be connected to from any distance, so long as both the Pi and the device controlling it were connected to the same network. Therefore, we searched for a solution and found that by manually adding the Wi-Fi details in the config file containing the various network details, /etc/wpa_supplicant/wpa_supplicant.conf, the Pi would now automatically connect to the desired network. For reference, the network details are provided below:
network={
ssid="YOUR_NETWORK_SSID_HERE"
proto=RSN
key_mgmt=WPA-EAP
pairwise=CCMP
auth_alg=OPEN
eap=PEAP
identity="LOGIN_USER_NAME"
password="PASSWORD" # A hashed password can also be provided
phase1="peaplabel=0"
phase2="auth=MSCHAPV2"
priority=1 # To ensure that it would be connected to first
}
Later in our testing, we discovered that high network latency resulted in extremely poor performance of the camera stream as well as unresponsive controls. This was especially a problem for our project as NTUSECURE turned out to have an extremely high latency in the MnT lab despite working fine almost everywhere else. To address this issue, we added the option to connect directly to the Pi by having it host its own wireless access point while still connecting to the wifi.
To do so was a long and not entirely successful journey through various configuration files with the terminal. The main issue was that a large number of guides simply describe how to convert the wlan0 interface on the Pi to an interface to generate the wireless access point, which requires the Pi to be connected via an ethernet cable to remain connected to the internet. Additionally, from the small pool of guides which did touch on maintaining 2 wireless interfaces to achieve what we wanted, many of them were depreciated as they were meant for a previous version of Raspian (Jessie) while we were running (Stretch). After a good number of clean installs, we finally stumbled on this blog post which we followed. The procedure used was to install dnsmasq and hostapd, used to provide the network infrastructure and turn the Pi’s wifi card to be to act as an access point. Following that, we created a virtual ap0 interface tied to the wlan0 interface. dnsmasq was then used to set up the DHCP range and hostapd to set up the settings for the wireless access point. A number of other instructions were followed to solve some issues that arose and to forward packets through the Pi. The Pi was then able to connect to the internet, as well as connect to other devices and connect them to the internet as well. The significance of this is that it allowed the Pi to be accessed both by its wireless access point’s IP address and its IP address on the network it was connected to, offering multiple options based on the user’s needs. On a side note, we also attempted to make the virtual ap0 interface grab the wlan0’s macaddress on bootup, although this was unsuccessful. This meant that each SD was configured to their respective hardware.
After acquiring a stable connection to the Pi, we then enabled the VNC and SSH interfaces, to connect wirelessly to the Pi and access its filesystems respectively. By using a VNC client, we were able to remotely control the Pi, and with FileZilla, a File Transfer Protocol (FTP) solution, we were able to push files to the Pi. Additional interfaces to be enabled were the camera, for the camera, and remote GPIO which would be elaborated on later.
Code development
Finally, we were now ready to develop code for the Pi. The final code is modular, reflected here by 3 different sections: the server in charge of handling user input and sending the video as well as feedback back to the user, the web app in charge of providing the user interface and sending user input to the Pi, and finally the script to control the servos. This allowed us to work on the different modules concurrently, and make changes as and when they were needed.
We chose to develop the scripts for the Pi in Python as it came preinstalled in the Pi, with a number of libraries conveniently available. The web app was developed with HTML, CSS, and javascript.
Server Code
The code used to launch the server was a script adapted from this tutorial. The script would start a server and the Pi camera, pushing the feed from the camera to an image element on the page hosted by the Pi, whose source was .mjpeg files being pushed by the camera.
Key to the server is the HTTP request handler which allows the server to respond to requests made by the user. Requests include requests made to load the site, load the video feed and the user inputs. These requests fall into ‘GET’ and ‘POST’ requests depending on what they are for. Loading of the page and elements use the ‘GET’ method, to the home path,’/’ and ‘/mjpeg’ routes. The server responds by sending the required data back to the user. User inputs contain the angle to sweep the tail around, the angle at which to sweep it between and the angle to tilt the CG mechanism as key-value pairs, sent in a POST request. The server takes these values and passes it to the servo moving script to actuate the servos. As it does so, it sends a 200 reply to inform the user’s device that it has received the instructions.
Servo actuating code
The methods we used to actuate the servos underwent a number of large changes, to improve on their performance. However, behind the logic, the role of this script was to send pulses of different lengths to the signal pin on the servo, with the different lengths corresponding to different angles.
The first version of the code made use of the RPi.GPIO library. However, we decided to change the library we used to interact with the GPIO pins because of the issues arising as a result of how the library generated the signal. Firstly, the RPi.GPIO library would set a pin to send signals by pulse wave modulation (PWM) at a rate of 50Hz, meaning that it would send alternating high and low signals at a frequency of 50 high-low cycles a second. Angles in this library when then represented different duty cycles or the percentage of each high-low cycle were high, therefore controlling the length of the high signal. The issue that arose from those however, was that the RPi.GPIO used software timed PWM which were unstable, resulting in irregularly timed signals. As a result, the servo jittered around the angle provided. This was unfortunate as the RPi.GPIO was a standard library with built in support for various actuators which seemed promising.
Consequently, we went looking for another solution for communicating with the GPIO pins, and settled with pigpio. In terms of complexity, pigpio utilized the remote GPIO interface, had to be installed and required to be initialized before running the server script, which complicated matters. Fortunately, we were able to get it set up and it did resolve the jittering issue.
On the logic side of things, we had originally intended for the tail servo to slowly move to its instructed position in small steps in set intervals, changing its direction when it exited an envelope around the angle given by the user. However due to the jittering, it was difficult to achieve the smooth motion that we desired. Hence we settled to have the tail sweep between the ends of the envelope, adjusting the size of the envelope to adjust the speed. This method of controlling the servos remained in place while we were testing the Hoverfish prototype and displayed favorable results. As such, we chose to stick with it rather than going back to the original idea. The buoyancy control servo on the other hand had always used the second method of control. However, as its movements were often from one extreme to the other, this resulted in the balloon being jerked around as the movable mass was swung from front to back. Here we decided to implement the original method of control, achieving a less jerky response.
Despite having solved a number of issues to get to a decent level of performance, we identified another issue which could possibly be improved to achieve a smoother, more favorable actuation. At this point, the commands were being actuated as they were received by the Pi, resulting in inconsistent intervals between each movement of the servos due to differences in round trip timing. To address this issue, we introduced a command queue on board the Pi. Commands would be stored in the queue instead of being executed immediately, and actuated at set intervals. To do this alongside the rest of the code running on board the Pi, we had to study up on threading in order to run this code in a separate thread. The work paid off, allowing the servo code to be actuated on a separate thread from the rest of the code. Performing a bit of extra work to time the two threads in order to ensure the servos remained responsive resulted a smooth satisfying conclusion to this part of the code.
The final issue was simply to address the particularities of each different servo we tried out, in order to ensure that they worked as intended.
User interface
In the beginning, we made use of keyboard inputs as they were the only inputs we were able to read from the terminal. Consequently, the initial script for the user to input commands had to be run from the terminal. The script utilized the python curses module to grab the last keyboard input in the buffer. However, this brought with it the constraint that only one input could be read at a time. Regardless, this allowed us to provide some commands to the servos to test some rudimentary inputs and actuation.
After testing the server script, we realized that we could utilize the browser displaying the video feed to capture user input as well. Following on the keyboard inputs used for the original servo script, we tested on_key_down and up events to control some elements on the site. At this point we considered touch as an alternate input method, to make the controls more intuitive as well as usable on mobile devices. Touch inputs proved much more complicated than keyboard inputs, however, and it was a long road to getting them to work as intended. Furthermore, at this time we had yet to work out how to send the commands from the user interface to the server, meaning that the touch interface was being developed without a guarantee that it would work. Nevertheless, we had confidence that it was possible and continued to work on the touch interface.
To begin with, the event listeners for the touch events had to be attached to an element, however, I was unsure if I should have attached them to the window or a canvas element sized to the viewport. When I was testing it initially, I utilized the window element due to difficulties scaling the canvas element. (It should be noted that our group was not familiar with HTML, CSS or javascript.) The window element proved to be unsuitable as the preventDefault() property did not work with it, forcing us back to working with the canvas element. The preventDefault() property was vital as it stopped the touch events from acting as emulated mouse events, allowing them to work as intended as opposed to performing the usual browser actions of scrolling, resizing, etc. Due to the sizing issues, while we were able to register the touch events, the position of the touches were off. Convinced by this article, we strived to make a touch interface that would generate a joystick on the first touch location, and a different control interface on the second touch. By registering the touch.identifier on touchstart, we are able to identify the first and second touches and their initial positions. With the touchmove event, we were able to find how each touch object moved relative to the initial position. Finally, the touchend and touchcancel events allowed us to recycle the joysticks back when the user releases them.
First iteration of the touchscreen controller showing the position of the touch relative to the initial touch, demonstrating touchstart and touchcancel events working
To test the control system, we incorporated values which would be later sent to the servos, and displayed them on the screen. Satisfied that this was working as intended, I then attempted to introduce visual feedback to the user as to the locations of the controls and their fingers. This was done with a draw() function ran on a timer to refresh the screen with the appropriate shapes. Implementing this revealed that the sizing issue remained a problem. Embarrassingly, the issue lay in our inexperience and syntax errors. Some tweaking later and we were able to get a working display.
Getting the interactive touchscreen was a significant success for our team, however, now came the time to tackle the issue we had been avoiding, how to send the requests from the user’s device to the server. To do so, we once again dabbled in beginner’s guides on Hyper text transfer protocol (HTTP) to get our clueless selves on the right track. There we learnt of GET and POST requests, routes and phrasers. With our newfound knowledge, we took another look at the python HTTP server code, and fixed up a route, handler and phraser for our POST requests as explained above. On the client’s side we set up a function to prepare a xhttp request with the relavant details and send an asynchronous POST request to allow the rest of the code to run while waiting for a response. To add on, we also included a function to handle the server response, informing the user that their request was being processed. After troubleshooting on both sides, we managed to get our first working prototype of the user interface.
Working prototype of the touch interface controlling the tail, showing left-handed support.
As latency was found to be an issue, we also modified the client to keep track of each request, so as to be able to tell how long it each request took to receive a response.
User interface updated to show the round trip timings. Camera mounted on the Hoverfish
A major benefit to creating the user interface by ourselves was that the flexibility allowed us to address control issues specific to the Hoverfish as well as solve power issues related to excessive servo movement through the user interface.
New UI prototype, better reflective of the control available to the user. Additionally reduces excessive movement of the buoyancy servo and allows the user to go straight more easily.