Schematic for Ideation process:
Let’s break this down…
[PROTOTYPE ONE]
Thoughts about the Design: When we first embarked on this project, the objective was to create a commute-assistive device that was easy to use and can be placed on chest level, so as to enable the device to recognise the traffic light clearly. We thought of an “iron-man” style because it is a simplistic design that will not look too ostentatious on the user. Furthermore, after conducting a literature review for devices used by the visually-impaired, we realise that many of the available designs were very bulky or had to be used with a laptop. This made us wonder, what if we can create something portable yet can fulfil a task that no one has thought before? Hence, we embarked on the project aiming to allow visually-impaired users to have greater independence when crossing roads in Singapore.
Thoughts about the Coding: In order to allow the device to recognise traffic lights, we wanted to train the Raspberry Pi to recognise and detect both the shapes and colours of the pedestrian lights. For this, we chose to utilise the image processing capabilities of the Raspberry Pi, and we purchased a PiCamera along with other accessories. We also purchased vibrator motors so that we can programme the Pi to vibrate accordingly to the signal recognised. When the green light is detected, we wanted the Pi to vibrate constantly, telling the user to move and cross the road. Using Python and OpenCV, we started doing some research and familiarising ourselves with utilising OpenCV as well as the Raspberry Pi.
[PROTOTYPE TWO]
Thoughts about the Design: After consulting different supervisors (Dr Koh and Professor Domenico) regarding our first prototype and designs, they gave us feedback allowing us to tweak and change our designs. Some of the feedback were as follows:
- Although plenty in the market, we can also think of doing obstacle detection, allowing the visually-impaired user to identify obstacles a few metres away
- Placing on the chest may be too bulky, perhaps can think of something light-weight and able to pan easily, rather than have to turn the entire body
As such, we thought of changing our design to a cane instead, and place ultrasonic sensors to the top, middle and ends so as we can perform image detection.
Thoughts about the Coding: As for the coding, all the algorithm stays the same, however, we do need to also do some programming for the ultrasonic sensors so that the vibrator motors can work effectively as well. We would need to change the intensity of the vibration so as to allow the visually-impaired user to distinguish between “green light” and “obstacle ahead”. For machine learning, we had to create a specific and powerful dataset ranging between 300-1500 photos of traffic lights. These photos will then be written in binary code before being processed. The written codes will then analyse each pixel and try to identify the similarities in the test images. Some codes range from a single layer to multiple layers. The codes that we were working with were a single layer programme and a 5-layer programme. However, we were only able to achieve a maximum of 55% training accuracy and 35% test accuracy in our first few rounds which was a good start, but can be better!
Here are some of our thought processes when we were creating our prototype:
- The group will make the assumption that the visually-impaired user has independence and is able to navigate around his/her surroundings relatively well.
- DESIGN CRITERIA
- Has to be a device easily incorporated into user’s life
- The design has to be ergonomic
- Cane has to be adjustable to user’s height
- Has to be large enough to accommodate all parts (i.e. Raspberry Pi, PiCam, Power source)
- Light to be used easily by the visually-impaired users
- Able to pan and identify surroundings easily
While we were sourcing for the ultrasonic sensors, the sensors that would best value-add the project was out of the budget, hence we have decided to focus on and exploit the uses of machine learning for our project. However, the assumptions and design criteria remain the same throughout our project!
[PROTOTYPE THREE]
Thoughts about the Design: It was after the 2nd project update that we received feedback that our ideas may not be relevant to the market. A simple app can be programmed to perform the tasks we set out to do. A phone has in-built, powerful and complex vibration motors as well as a beautiful, high-definition camera, one that the PiCamera cannot compete. By learning and creating an app which can make use of these components, we are technically also able to create a commute-assistive application that any visually-impaired user will be able to use. Furthermore, we also found out during the updated that the Google Cloud Platform is a suite of cloud computing services that run on the same infrastructure that Google uses internally for its products such as Google Search and YouTube. This platform can be used by everyday users in order to utilise its complicated and advanced machine learning capabilities. Shocked and defeated by the comments we received from the ground, we set back to work. The group spent a whole week on Google Cloud Platform in order to learn how to make use of the platform to feed our dataset and to use its Machine Learning capabilities for our project. Half of the group also started learning how to programme apps via the android development just in case we were unable to convince our supervisors that the cane idea was still one we truly believed in.
Google Cloud Platform
Although it sounded like a brilliant idea from the start and could have saved us a lot of work, turns out the platform is not as user-friendly as we hoped it could be. Like any other package, it also requires a significant amount of programming. Should we have enough time, we believe that using this platform can be successful, however as it takes a long time for us to learn a new language, it would be faster should the group stick to Python and continued our programming from there.
App Development
Developing an application sounded very interesting initially, however, as our group thought about it further, we realised that it did not satisfy our design criteria. We wanted something that can be easily incorporated into the user’s life. A cane is already something that the user would already have carrying around him/her all the time. By simply altering the design of the cane, we can enhance the life of a visually-impaired person. By holding the phone all the time, it may create trouble for the user as he/she would have to take out his/her phone every other minute to cross the road. Furthermore, the application would also have to travel via cloud which will consume data and battery, issues which would also become an inconvenience to the user.
Hence, after much-heated discussion and debates over 5 hours, the group decided to stick to the cane design and we managed to convince Dr Ho to allow us to keep the direction we were heading towards (i.e. Prototype 2)! As for the Google cloud platform, after trying the platform for a whole week to no avail, the group also decided to continue utilising the raspberry and to make use of our strengths and knowledge of Python to continue programming.
[PROTOTYPE FOUR / FINAL PROTOTYPE]
Thoughts about Design: As we continued our work with the second prototype, we realised a few problems that may occur with the use of a cane. Here are some issues the group realised:
- Attaching the camera on the cane may cause the camera to be unstable
- Resulting cane will be very heavy as we will need to attach the Pi, power source, motors and potentially a GPIO with a breadboard
Furthermore, after consulting with Prof Domenico and his fellow PhD students, they also encouraged us to revert back to a device to be placed on the chest/waist area as well. This is because it will reduce shaking and optimal flow as the cane can cause shaking due to lots of moving. At the same time, after interviewing and speaking to a few blind employees from Ngee Ann Polytechnic’s Dialogue in the Dark (DiD) social enterprise, they have given the same suggestion as well, to have a device on the chest or one that can be handheld as the cane is very flimsy. They also do not want to add extra weight to the cane.
Hence, the group reverted back to a chest design and we tried to make it light, minimalistic and not too ostentatious for the user.
Thoughts about Coding: After consulting Prof Domenico’s PhD students and gotten some guidance from them, we have decided to utilise template matching in order to get the Pi to recognise pedestrian lights. This method is far faster than the machine learning programmes we have been trying to develop. It also does not consume as much power and processing strength as compared to machine learning. After initial testings, we realise that at a close proximity, the algorithm is able to recognise the traffic lights, however, at a small 50m junction, the camera is too far to detect the pedestrian light across it. What we are doing now is to feed the algorithm with a better dataset so that it can work better.
We will also utilise colour detection and limit the range of green and red colours the PiCamera can detect. This allows the PiCamera to only recognise the specific green and red colours emitted from pedestrian lights. We have also successfully tested the colour detection codes and it can successfully distinguish the red and green colours from pedestrian lights, and it does not detect other shades of red and green from surrounding environments (i.e. trees, buses, cars, motorbikes, etc.) We have also developed “Day” and “Night” scripts for the PiCamera so that it is able to identify the colours in different settings.
We are also coding a script via the hough line transformation method, in order to detect only the pedestrian lines one the road. So far, the script is not optimal and still in progress. With this method, we are able to detect pedestrian lines, allowing the Pi to crop images taken to only analyse pedestrian lights only. This will ensure that the template matching is more efficient. Furthermore, with this method, we can also detect whether the vision-impaired person is walking in a straight line. Feedback was given during the interview with blind employees in DiD, saying that crossing the roads at especially large junctions were difficult as the roads were wide and long, and they do not have guidelines to help them walk straight. Some may even veer off the straight road as it was difficult to navigate without assistance. Hence, by detecting only the pedestrian lines, we may be able to potentially provide assistance to enable the blind to cross large junctions safely.
