Designing the Front Page using XML Coding

As we waited for the machine learning models to be ready, the Android Studio team decided to work on the User Interface first. Although android studio provides a drag and drop tool to assist in designing pages (strictly speaking, activities), Wei Bin had ran into some problems whilst using them. The alignment became a huge mess as everything is aligned to the top right. After quite a bit of googling and cycling between different layouts (linear layout, relative layout…), we managed to find a solution by attaching each object to the bottom of each other and inputting blank spaces to widen elements if necessary. Coding for image buttons in the activities also came up with unexpected issues. As the boxes are much smaller than the original images, we had to play with some of the values such as padding to make sure that the image is not too zoomed in.

Getting Acquainted with Machine Learning

The next step forward is to understand the basics of machine learning, the different kinds of machine algorithms used and the type of algorithms suitable for object detection. The popular models used are Single Shot Detection (SSD), You Only Look Once (YOLO) and Faster RCNNs (Regions with Convoluted Neural Networks). While Jian Xian attempted to understand the basics of the models, it proved to be very difficult as these models are highly mathematical and it is not practical to understand each model. Hence, he opted to understand the overarching tensorflow process from data labelling to pre-training and fine tuning. In addition, he had to read up on important configurable parameters such as learning rate, quantisation, decay factors etc.

By reading up on the topic, he managed to understand how machine learning works on the surface, just enough for him to configure the parameters needed.

First Steps into Augmented Reality

The first step is usually the toughest but most important one. After we have decided on building an Augmented Reality Instructional Manual, we set out to build a basic AR app. This basic app would only be able to place an object on the screen and would serve as a template for us to add our own features to.

As Wei Bin was in charge of the development of the app, he went through CodeAcademy’s guide to Java so he would be able to better understand the language that will be used to code our app. Then, as he had no prior knowledge in app development and AR, he looked up online tutorials on how to build a basic android AR app using ARCore and Android studio. However, as those guides seem to be outdated, he faced multiple bugs along the way and the app would not even compile properly. After a long 2 days of googling and looking at online tutorial for bug fixes, he finally manage to get a solution and got the AR app to work properly.

Initial Designs

Choy Boy created some designs on how the app might look like:

Introduction to Machine Learning

The learning curve for machine learning is inherently very steep as it requires a good background in Python, of which Tensorflow is written on. Tensorflow is a Software Development Kit (SDK) for machine learning offering hundreds of configurations where developers can tweak based on their project. Machine learning is used from audio translation to natural language processing to object detection.

For the purpose of this project, as Jian Xian is in charge of the machine learning, he focused on object detection where he needed the app to know that the object is in the field of view. To begin, he had to setup my development environment in Ubuntu as it is much more convenient to program in a Unix system. After a few days, he had set up the environment and had begun reading on the process of machine learning.