Author Archives: HOW WEI BIN

Week 8 16th July- 22nd July

Posted on by
Reply

After attaining our minimum viable product (completion of seamlessly integrating step 1 of the procedure), we decided to prioritize finishing the entire application. Firstly, We created a mathematical counter to loop through the numbers 0 to 4. Then, after some trial and error, we managed to get the code to remove any existing 3d objects. After which, we pieced these 2 together. When the trigger for the next step is activated, the counter will increase by 1, which will represent the progress of the user throughout the 5 steps. When the counter changes, an update listener will proceed to remove the current 3d objects and overlay the 3d object corresponding to the counter. Thus, allowing us to transition from step to step using a trigger.

 

Currently the trigger is designated as a button press but we decided to implement an automatic step tracker so that the transition is much smoother and the experience could be more hands-free. To achieve that, we would have to improve the machine learning model to make it much more reliable and also develop an algorithm to track the steps based on the visual input. As of now, the algorithm is ready and the machine learning model is still in the midst of training.

Week 7 8th July- 15th July

Posted on by
Reply

A Glimmer of Hope for the app development

After hitting brick wall after brick wall when trying to integrate our machine learning model with ARCore. We finally managed to get a glimpse on how the integration would work out. We currently managed to get tensorflow to work alongside ARCore and get the models to be displayed on the position that tensorflow has defined. Although we have currently hit our Minimum Viable Product (MVP), much more can still be done to integrate the different features we want to integrate to make the app experience as seamless as possible.

Week 5: 24th June – 30th June

Posted on by
Reply

Encoding the Animations

Before embarking on incorporating both apps, Wei Bin decided to try to get animations to work on an Android AR app to ensure that everyone can continue on with their work without having to wait if the animation file is compatible. Following the process from incorporating the machine learning model, he tried to search for similar apps and managed to find one done by the google ARCore development team. However, the code is extremely confusing and he has still not yet found the portion where the code pulls out the animation file and thus, has not yet been able to load our animation to see whether it can work. However, we are slowly figuring out different parts of the code through online tutorials and we should be able to get it up by next week.

Improving the Machine Learning Model

In previous attempt to produce a model, it did not perform as well as we have expected. It was unable to pick up intermediate steps when the card slot has been ejected partially. As we envisioned the app to be seamless, it was critical for the app to identify intermediate steps and guide the user along. Hence, Jian Xian saw the need to retrain the model to account for these transitions. In the image below, he had to explicitly label the partially ejected cardslot to improve the accuracy of the model.

To test if the model works, he managed to integrate the model into an Android App and it was exciting when the app managed to identify the intermediate step. This is illustrated in the screenshot below:           

Week 2: 3rd June – 9th June

Posted on by
Reply

Designing the Front Page using XML Coding

As we waited for the machine learning models to be ready, the Android Studio team decided to work on the User Interface first. Although android studio provides a drag and drop tool to assist in designing pages (strictly speaking, activities), Wei Bin had ran into some problems whilst using them. The alignment became a huge mess as everything is aligned to the top right. After quite a bit of googling and cycling between different layouts (linear layout, relative layout…), we managed to find a solution by attaching each object to the bottom of each other and inputting blank spaces to widen elements if necessary. Coding for image buttons in the activities also came up with unexpected issues. As the boxes are much smaller than the original images, we had to play with some of the values such as padding to make sure that the image is not too zoomed in.

Getting Acquainted with Machine Learning

The next step forward is to understand the basics of machine learning, the different kinds of machine algorithms used and the type of algorithms suitable for object detection. The popular models used are Single Shot Detection (SSD), You Only Look Once (YOLO) and Faster RCNNs (Regions with Convoluted Neural Networks). While Jian Xian attempted to understand the basics of the models, it proved to be very difficult as these models are highly mathematical and it is not practical to understand each model. Hence, he opted to understand the overarching tensorflow process from data labelling to pre-training and fine tuning. In addition, he had to read up on important configurable parameters such as learning rate, quantisation, decay factors etc.

By reading up on the topic, he managed to understand how machine learning works on the surface, just enough for him to configure the parameters needed.

 

Week 1: 27th May – 2nd June

Posted on by
Reply

First Steps into Augmented Reality

The first step is usually the toughest but most important one. After we have decided on building an Augmented Reality Instructional Manual, we set out to build a basic AR app. This basic app would only be able to place an object on the screen and would serve as a template for us to add our own features to.

As Wei Bin was in charge of the development of the app, he went through CodeAcademy’s guide to Java so he would be able to better understand the language that will be used to code our app. Then, as he had no prior knowledge in app development and AR, he looked up online tutorials on how to build a basic android AR app using ARCore and Android studio. However, as those guides seem to be outdated, he faced multiple bugs along the way and the app would not even compile properly. After a long 2 days of googling and looking at online tutorial for bug fixes, he finally manage to get a solution and got the AR app to work properly.

Initial Designs

Choy Boy created some designs on how the app might look like:

Introduction to Machine Learning

The learning curve for machine learning is inherently very steep as it requires a good background in Python, of which Tensorflow is written on. Tensorflow is a Software Development Kit (SDK) for machine learning offering hundreds of configurations where developers can tweak based on their project. Machine learning is used from audio translation to natural language processing to object detection.

For the purpose of this project, as Jian Xian is in charge of the machine learning, he focused on object detection where he needed the app to know that the object is in the field of view. To begin, he had to setup my development environment in Ubuntu as it is much more convenient to program in a Unix system. After a few days, he had set up the environment and had begun reading on the process of machine learning.