Planning, finalising project ideas, and self-learning (30th Apr 2020 – 28th June 2020)
Discussed with Tony on project ideas and decided to do an AI microscope and came up with a project proposal for approval from Dr Ho. Meetings after meetings, Dr Ho suggested us to consult with Prof. Török on how to approach the project. We also discussed on some journals regarding fluorescence microscopy and the possibilities of 3D-printing the microscope.
Week 1 (29th June 2020 – 3rd July 2020)
Ending the discussions with Prof. Török, we started to learn the basics of Fusion360 and Arduino programming and wiring to prepare for our initial project hardware building and 3D-printing parts. At the same time, after looking through several microscope ideas about 3D-printing from the papers we have read, we feel that it is quite time-consuming for us to design the entirety of a microscope as our team only consists of two persons. As such, we proposed to focus on controlling the microscope without direct/physical contact. That’s the summary of how we came up with the Remote Microscope.
Week 2 (6th July 2020 – 10th July 2020)
Introduced to the RepRap Smart Full LCD Controller, RAMPS v1.4, and Arduino Mega hardware to control our motors. We have also learnt the wiring and coding the correct pins and also recognising the different motor types and motor drivers to attach to the RAMPS v1.4. Furthermore, we explored the AccelStepper library to control the stepper motors for x- and y-axis, but was not able to make the z-axis motor moved.
Figure 1. Reprap 3D Print.
Figure 2. Snippet of the AccelStepper Library Code used.
We also tried to design and 3D-print for the first time after gathering ideas from papers that we have read to see the precision and how well-fit 3D printed things can be. It turned out that the microscope head and the filter holder were difficult to match each other well. If assembled together, it may be difficult to have a stable structure of the microscope to get a good precision while in operation.
Figure 3. 3D design of the microscope head and filter cube with Fusion360 drawing tools.
Figure 4. 3D-printed microscope head and filter holder.
Week 3 (13th July 2020 – 17th July 2020)
Sadly, after able to control the motors with AccelStepper library, we found out that there were better methods to gain better control of the stepper motors 🙁
However, we have learnt more than what we expected, that is C++ which is used in coding in Arduino. We were then introduced to the Marlin firmware. We learnt how to customise the firmware according to our needs, e.g. by disabling the temperature sensors, inverting motor movements, identifying the correct stepper motor drivers models, etc., and most importantly, the wiring changes everything and affecting what we do.
Figure 5. Marlin Firmware.
We have also made a purchase for our very own biological microscope, the Geefuntech Biological Microscope, which will be used for our project. Fortunately, it would be arriving in one week time.
Week 4 (20th July 2020 – 24th July 2020)
Pronterface was also introduced and we learnt how to use Gcode to move the motor movements simultaneously, but no control yet. So, we tried to explore the different parts of the Marlin firmware and see how we can gain control of each stepper motors.
Here is the link to the video on how the Pronterface looks like and the demo of how we control the motor simultaneously using Pronterface.
While learning to control the motors, our long-awaited microscope has finally arrived! Firstly, we try to understand how the microscope works and it turned out there were 4 knobs to control. So, we decided to use 4 stepper motors for this case.
Figure 6. Trying out the brand new microscope.
We were able to try out the stepper motors to turn the knobs by printing the appropriate motor-to-knob adapter. We also required tubing as the intermediary for flexibility and friction to allow the adapter to turn the knobs. Additionally, we started to build a square aluminium profile frame for our microscope foundation and also for the placement of the stepper motors, aligned to the appropriate knobs.
We tried to explore the camera module/imager in the MnT Lab and attached it to our microscope. It was difficult to install to the laptop because we thought that the camera requires an external driver for it to run. It turned out that we only need to disable the laptop’s internal camera for the camera module to be activated. The image quality was acceptable with decent details but requires adequate brightness from the light source. At low brightness, the image quality was rather bad. We have taken photos of the image quality mentioned in figure 7 and 8 below.
Figure 7. First Image captured of the pre-prepared sample.
Figure 8. Second Image captured of the pre-preprepared sample.
Week 5 (27th July 2020 – 30th July 2020)
We managed to build a structurally stable aluminium frame to host the microscope and the stepper motors as shown in figure 3 below. In this picture, we were trying to position the microscope and adjusting the motor shaft and its adapter position such that it can rotate the knobs when running with Pronterface.
Figure 9. Positioning Microscope and Motor Adjustment.
After correct positioning of all the hardware, we moved on to calibrate the movement of the stepper motors. This is mostly a trial and error method with ratio calculation, where the input distance movement had to be calibrated with the actual displacement of the XY-caliper and the microscope tube (1:1 ratio). In other words, if the input distance is 1 mm, the expected displacement of the XY calliper or the microscope tube would also be 1 mm.
Figure 10. Rough design for aluminium frame of the prototype.
Concurrently, we tried to 3D-print different designs of the stepper motor adapters and it took so many trials until we get to the current suitable ones. We might also need to design tube clamps to hold the tubing fixed to the motor shafts and the knobs.
Figure 11. Several trials and broken 3D-printed stepper motor adapters.
Well, you see those aluminium frames? They are just a temporary structure. We will be rebuilding the frame when our purchase order for aluminium brackets and nuts arrive.
Week 6 (3rd August 2020 – 7th August 2020)
We tried to “automate” the movement of the microscope stage, i.e. centring it to the stage opening where the light source transmits through and homing to the origin set (X = 0 mm, Y = 15 mm). This is to prevent the objective lens from crashing to the XY-caliper metal parts during operation. This was done by inputting the Gcode coding in the shortcut button in the Pronterface application as highlighted in the red shade below.
Figure 12. Home (X, Y) and Center Stage Slide Shortcut Button.
We have also tried to attach the camera module to the microscope installed in the aluminium frame and tried with the automation created.
Here is a video of the first trial of our Remote Microscope.
We can see that the camera caught up with the movement of the stage. It worked pretty well but there were still some drawbacks, such as doing the remote control and observing the sample simultaneously in the camera app was difficult when using a tablet. After some trial, we found out that there were offsets, meaning the knob will not rotate until it reached certain distance during the first step in a particular direction for coarse and fine knobs, as shown below. The offset for coarse knob was 3 mm and the fine knob was 0.6 mm.
The possible reasons that we concluded were (1) twisting motion of the tubing against the interior of the adapter and (2) the weight of the camera module on the eyepiece was too heavy for the motor to reverse motion. As such, we changed the stepper motor to a gear motor with 5 times higher torque and 3D-printed the suitable adapter. We managed to program the movement of the motor but had not installed to the microscope, as shown here.
Week 7 (10th August 2020 – 14th August 2020)
We rebuilt the whole frame (timelapse video 1 and video 2) into a better and simpler structure when the new 30X30 Aluminium L-Bracket Connector and the M6 T-slot nuts have been delivered. We reuse the aluminium profile and cut it to the appropriate length to match and strengthen the frame structure.
Figure 13. Cutting Aluminium Profile process.
We started to strengthen the outer frame with 3 brackets at the corners, reposition the motors and the bracket support of the profile housing the motors. After all the motors position was set, we begin calibrating the motors’ steps to accommodate the shifted position of the microscope.
As the new motor has a different shaft shape, we designed a new adapter with two adapter head, mediated by tubing. The new stepper motor was able to rotate the coarse knob effortlessly, with a precision of 0.25 mm. This was a great improvement from the previous motor, which was only 3 mm. The twisting motion of the tube was minimised by reducing the length of the tube. However, the rotation of the coarse knob sometimes skipped steps before starting to rotate, which could be caused by the lack of friction.
Figure 14. New adapter for the gear motor head.
The fine knob first step offset was reduced by 17% to 0.1 mm from the previous 0.6 mm. This was enabled by a more secure connection by a cable tie on the adapter and repositioning of the knob. If the rotation was uni-direction after passing the offset, the motor can rotate down to 0.02 mm.
Week 8 (17th August 2020 – 21st August 2020)
The skipping steps problem causing offsets has to be resolved. We noticed that the new motor has a different shaft shape which requires a special design to fit the head. This shape provides an advantage where the adapter fitting into the head can prevent skipping of steps.
After understanding the shape of the motorhead shaft, we designed and 3D-printed the new adapter. Going through several trials and errors of the sizing, we managed to print the best fit size for the adapter. Below is the figure for the new design of the adapter without a tubing system.
Figure 15. Adapter for the gear motor without the tubing system (sketch video).
We tested the adapter with our firmware, and it performed much better than expected. The precision of the coarse knob was lowered to 0.1 mm, compared to the previous 0.25 mm. We also felt that the fine knob required more refinement in the precision. As such, we changed the stepper motor to the gear motor and redesigned the adapter to fit the fine knob. After several trials, we managed to print the correct size for the knobs and after being put to test, the precision was about 0.1 mm.
We discussed among ourselves and we came to an agreement that Pronterface has its limitations for our remote microscope application. One critical component that is missing was the unavailability of a live camera monitoring interface while the motors were adjusting the sample slides. Thus, we decided to look for another software that has the features that we wanted.
We have found several alternatives along the way, such as Repetier, Cura, and OctoPrint. Out of the three alternatives, we found that OctoPrint was the most suitable for our needs as it supports live webcam monitoring and custom controls of the motors. So, we decided to research more on OctoPrint and will be doing the configuration on the following week.
Week 9 (24th August 2020 – 28th August 2020)
After thorough searching, we found out that OctoPrint also uses Gcode as the language to program the commands, so there were no big issues with the programming technicalities. However, it is usually coupled with a Raspberry Pi (RPI). This mini-computer is usually used for its lightweight, compact size, and adequate processing power for 3D-printing. Unfortunately, since our camera module might not be able to connect to RPI due to software limitation, we decided to try with our laptop in place of RPI. We tried to install a Virtual Machine (VM) and load in the RPI Desktop booted from our Windows-based laptop. However, the installation of OctoPrint was not smooth as we thought it should be due to limited knowledge on how to use Debian – Linux operating system. After several attempts to install the OctoPrint in the VM and failed, we chose to drop the idea of using a VM and use Windows laptop instead.
The setup was also not easy in Windows because we went through multiple errors during the manual installation, but we managed to successfully install. After successfully installed the OctoPrint, we needed to establish localhost to access the control interface. The next thing that we find challenging was activating the camera module. We plugged in the camera module to our laptop but the camera was not detected by the OctoPrint. We tried to troubleshoot the problem for days and we managed to find a 3rd party app, called Yawcam, that supports the connection of the camera module to the OctoPrint interface.
However, Yawcam itself posed another problem. We had to manually configure the IP address and ports to allow access in Yawcam and connection of the camera module to OctoPrint. After spending a few hours searching on how to configure, we were not successful in activating the imager. We decided to configure it further on the following day and we managed to activate the imager by inputting the correct stream URL and port for the HTTP and Stream settings in OctoPrint. We explored the Yawcam and OctoPrint app for a good while and we found out that it also supports time-lapse video capture, which is not supported in Pronterface.
After setting up the OctoPrint and switching the motor adapter for the new stepper motors, we calibrated the stepper motors again. After calibration, we conducted several trials in the OctoPrint while observing the sample, and we felt that the movement was still too large for higher magnification. This time around, we tried to increase the precision of the coarse knob down to as small as 0.01 mm. We successfully adjusted the precision to 0.01 mm (video below) but the offset problem persisted with lesser occurrences of offsetting during the rotation for coarse and knob ≤ 10 steps. For XY stage, it was a different case. It required more steps to escape from the offset and dead zone (the hardware limitation of the XY stage that prevented the movement of the slides despite the movement of the gear motors and knobs), which was around 30-40 steps.
Week 10 (31st August 2020 – 4th September 2020)
We also tested the precision of the stepper motor movements with the apparatus as shown in the video below, using a dial indicator. The measurement showed that the movement caused by the motors were accurate, aligned with the input distance inserted in the OctoPrint.
We realised that after the second trial (video below), we feel that there was still something missing. There was no possible way to track the movement and location of the slides externally. After discussing, we thought of adding an additional camera and tried to see if OctoPrint supports multi camera module.
We found a new plug-in within the OctoPrint software called the Multicam. This enables users to use multiple webcams simultaneously during operation. We find that this is beneficial for the remote microscope, especially in tracking the position of the microscope slide if it has moved to the specified distance. However, the installation was not easy because Yawcam can only gain access to one camera module only. So, we thought of another idea by using a second desktop to activate a second Yawcam to activate the laptop webcam. We were also required to figure out the port and IP address for the laptop webcam. After several trial and error, we finally managed to activate the second camera for external tracking.
Figure 16. Multicam plugin support for OctoPrint
Week 11 (7th September 2020 – 11th September 2020)
As the final presentation day was drawing near and we felt that our project was almost complete, we brainstormed how to demonstrate the application of our final prototype. We thought of using live samples because they are mostly used in live biological works. Thus, we obtained an E. coli sample to visualise under the microscope. Here, we took the chance to try out our interface system with a live camera to verify the usability and convenience of how a remote microscope can be operated in real-life application. We have recorded a short clip below to demonstrate how we tried the OctoPrint to track the movement of E. coli. Our microscope was able to catch up with the E. coli to a certain extent by controlling the motors via OctoPrint.
In the end, our team came to the same conclusion that it requires a lot of practice to track the movements of live samples. The user needs to know how sensitive the motors movements are when, let us say, 100 is clicked, and how the tracking can affect his/her observation.
Week 12 (14th September 2020 – 18th September 2020)
We tried to demonstrate other functionality of the remote microscope by utilising a screen pixel detector to enhance our monitoring system. We wanted to monitor a live sample similar to the one shown in this video, but we were not able to obtain one. So, as a demonstration, we used this iodine dye to test the reliability of our monitoring system shown in the video below. An SOS message will be displayed on the screen when a movement occurs.
Since our laptop was used as the second camera to track the slides movement, it required a static platform for the laptop to rest on. So, we used the aluminium frame to build the surface as shown in the figure below. On top of the profile, we have attached sponge fabric serving as soft cushion for the laptop to prevent scratches if contacted with the rough aluminium surface. We have also applied finishing touches to our prototype by attaching the remote microscope letters and hung a “blue duck” attribute.
The video below shows the combination of the features we have made previously and used the remote desktop to control and operate the microscope remotely.
Week 13 (21st September 2020 – 25th September 2020)
We prepared for our final presentation and tidy up the blog. We also did a final check on our final prototype to make sure everything was working well and smoothly for demonstration during the final presentation.
©2020 BluDaç.