Software Development Log

Phase 1

During this phase, we had 3 main aims:

  1. Have our camera module scan and read a barcode
  2. Search the barcode on an online database and retrieve the product information
  3. Store the information in a CSV file for inventory tracking.

Code breakdown and explanation:

Libraries used:

import pandas as pd
from urllib.request import urlopen
import json
import cv2 #read image from camera
from pyzbar.pyzbar import decode
import time
import pandas as pd
from datetime import date,datetime

We then created a function that starts the camera and scans for barcodes.[1] Once a barcode found, it is searched on an online database for the product name. If no such product is found, user is required to input the product name. This input is stored in a CSV to reference for future scans of the same product. The barcode and product name is returned by the function.

def get_product_data():
    camera = input("Use camera to scan barcode?(y/n) ").lower()
    if camera == 'y':
        camera = True
        cap = cv2.VideoCapture(-1)
        cap.set(3, 640) #3 – Width
        cap.set(4, 480) #4 – Height
        while camera:
            #to get barcode
            success, frame = cap.read()
            for code in decode(frame): #one frame can have multiple Bar'codes'
                print(code.type)
                print('Is the Barcode read: ', code.data.decode('utf-8'), '?(y/n) ', sep='', end='')
                ans = input().lower()
                if ans == 'y':
                    barcode = str(code.data.decode('utf-8'))
                    camera = False
                else:
                    barcode = input('Please key in barcode: ')
                    print()
                    camera = False
            cv2.imshow('Test', frame)
            cv2.waitKey(1)
            
    else:
        barcode = input('Please key in barcode: ')
        print()
    try:
        cv2.destroyWindow("Test")
        cap.release()
    except:
        pass
    
    url = f"https://world.openfoodfacts.org/api/v0/product/{barcode}.json"
    response = urlopen(url)
    data_json = json.loads(response.read())
    if data_json['status'] == 1:
        print(f"Is this the product: {data_json['product']['product_name']}?(y/n)",end = '')
        ans = input().lower()
        if ans == 'y':
            product_detail = data_json['product']['product_name']
        else:
            product_detail = input('Key in product name: ')
            
    else:
        new_df = pd.read_csv('new_prod.csv')
        for i in range(len(new_df)):
            #check new_prod.csv for previous entries
            if int(new_df.loc[i,'barcode']) == int(barcode):
                product_detail = new_df.loc[i,'name']
                print(f"Is this the product: {product_detail}? (y/n)")
                cont = input()
                if cont.lower() == 'y':
                    break
                
        else:
            print('nError: Item not foundn')
            product_detail = input('Key in product name: ')        
            #new data is saved to the new_prod.csv
            new_df_temp = {'barcode':barcode,'name':product_detail}
            new_df = new_df.append(new_df_temp,ignore_index = True)
            new_df.to_csv('new_prod.csv',index = False)
    
    return barcode, product_detail

Function for inputting items into the inventory:

It calls the previous get_product_data function to get the product barcode and name. The expiry date is keyed in by the user.

def input_item():
    global df
    barcode, product_detail = get_product_data()
    expiry_date_str = input("Expiry date? (DD/MM/YY)")
    expiry_date = datetime.date(datetime.strptime(expiry_date_str, '%d/%m/%y'))
    scan_date = date.today()

    df_temp = {'barcode':barcode,'product':product_detail, 'scan_date':scan_date,'expiry_date':expiry_date}
    df = df.append(df_temp,ignore_index = True)

Function for removing items from the inventory:

It calls on the get_product_data to get the barcode of the item. The appropriate item is then found and removed from the CSV file.

def remove_item():
    barcode, product_detail = get_product_data()
    for i in range(len(df)):
        #print(df.loc[i,'barcode'])
        if int(df.loc[i,'barcode']) == int(barcode):
            
            df.drop(labels=i, axis=0, inplace =True)
            break
    else:
        print('Item not in fridge. Nothing removed from DF.')

The code is then run in loop, until an invalid input is given.

while True:
    df = pd.read_csv('df.csv')
    cont = input('input/remove?:')
    if cont.lower()[0] =='r':
        remove_item()
        print(df)
        df.to_csv('df.csv',index = False)
    elif cont.lower()[0] =='i':
        input_item()
        print(df)
        df.to_csv('df.csv', index = False)
    else:
        break

Short clip on how items can be scanned and inputted into the inventory.

 

Moving forward into Phase 2:

While implementing Phase 1, we realized there were shortfalls with our approach. Although the system is able to keep track of items inside the fridge, there is a lack of convenience, especially when it comes to the inputting of expiry dates. The user has to manually input the expiry dates, especially for new items. Even though the expiry date of the same item can be estimated from previous entries, it still difficult to get the precise date without user input.

This lack of convenience when using our system convinced us to find a way to automatically read the expiry date of the product. Hence, we decided to try implementing Text Recognition using Optical Character Recognition (OCR) technology for our Phase 2.

 

 

 

Phase 2

In exploring the integration of Text Recognition using OCR, these were the areas of focus:

  1. Using a suitable OCR and Text Recognition Model
  2. Obtaining optimal pictures through picture processing
  3. Recognising the dates in the pictures

Part 1

We first tried out the Tesseract [2] (an open source OCR) and used the EAST Model [3], a deep learning text detection model from OpenCV.

Code breakdown and explanation:

Libraries used.

import numpy as np
import cv2
from imutils.object_detection import non_max_suppression
import pytesseract
from matplotlib import pyplot as plt

Setting of Tesseract environment, variables and file paths.

pytesseract.pytesseract.tesseract_cmd = 'C:\\Program Files\\Tesseract-OCR\\tesseract.exe'
args = {"C:\\Users\\Andre\\OneDrive\\Desktop\\Text Recog\\Milk\\Milk Cropped\\Test.jpg", "east":"C:\\Users\\Andre\\OneDrive\\Desktop\\Text Recog\\frozen_east_text_detection.pb", "min_confidence":0.5, "width":320, "height":320}

Reading and resizing of image. rW and rH are scale factors which will be used later.

image = cv2.imread(args['image'])
orig = image.copy()
(origH, origW) = image.shape[:2]
(newW, newH) = (args["width"], args["height"])
image = cv2.resize(image, (newW, newH))
(H, W) = image.shape[:2]
rW = origW / float(newW)
rH = origH / float(newH)

We then convert image to a blob, which is a processed image with altered RGB values. Following that, we read the EAST Model to begin out text detection. We do this to ensure the model we are feeding it to has the optimum picture. layerNames refers to the outputs we want to get from the EAST Model – Probability score for whether the region contains text and coordinates of text in the picture. We also define a function to return the bounding box of texts as well as the probability score if it exceeds the initial threshold stated above. Non-maxima suppression helps to select one bounding box out of the many overlapping ones.

blob = cv2.dnn.blobFromImage(image, 1.0, (W, H),
	(123.68, 116.78, 103.94), swapRB=True, crop=False)
net = cv2.dnn.readNet(args["east"])
layerNames = [
	"feature_fusion/Conv_7/Sigmoid",
	"feature_fusion/concat_3"]
net.setInput(blob)
(scores, geometry) = net.forward(layerNames)
def predictions(prob_score, geo):
	(numR, numC) = prob_score.shape[2:4]
	boxes = []
	confidence_val = []

	# loop over rows
	for y in range(0, numR):
		scoresData = prob_score[0, 0, y]
		x0 = geo[0, 0, y]
		x1 = geo[0, 1, y]
		x2 = geo[0, 2, y]
		x3 = geo[0, 3, y]
		anglesData = geo[0, 4, y]

		# loop over the number of columns
		for i in range(0, numC):
			if scoresData[i] < args["min_confidence"]:
				continue

			(offX, offY) = (i * 4.0, y * 4.0)

			# extracting the rotation angle for the prediction and computing the sine and cosine
			angle = anglesData[i]
			cos = np.cos(angle)
			sin = np.sin(angle)

			# using the geo volume to get the dimensions of the bounding box
			h = x0[i] + x2[i]
			w = x1[i] + x3[i]

			# compute start and end for the text pred bbox
			endX = int(offX + (cos * x1[i]) + (sin * x2[i]))
			endY = int(offY - (sin * x1[i]) + (cos * x2[i]))
			startX = int(endX - w)
			startY = int(endY - h)

			boxes.append((startX, startY, endX, endY))
			confidence_val.append(scoresData[i])

	# return bounding boxes and associated confidence_val
	return (boxes, confidence_val)



(boxes, confidence_val) = predictions(scores, geometry)
boxes = non_max_suppression(np.array(boxes), probs=confidence_val)

Lastly, we display the results of the model in the picture using the values returned from the above function.

results = []
for (startX, startY, endX, endY) in boxes:
	# scale the coordinates based on the respective ratios in order to reflect bounding box on the original image
	startX = int(startX * rW)
	startY = int(startY * rH)
	endX = int(endX * rW)
	endY = int(endY * rH)

	#extract the region of interest
	r = orig[startY:endY, startX:endX]

	#configuration setting to convert image to string.  
	configuration = ("-l eng --oem 1 --psm 8")
    ##This will recognize the text from the image of bounding box
	text = pytesseract.image_to_string(r, config=configuration)

	# append bbox coordinate and associated text to the list of results 
	results.append(((startX, startY, endX, endY), text))

	#Display the image with bounding box and recognized text
orig_image = orig.copy()

# Moving over the results and display on the image
for ((start_X, start_Y, end_X, end_Y), text) in results:
	# display the text detected by Tesseract
	print("{}\n".format(text))

	# Displaying text
	text = "".join([x if ord(x) < 128 else "" for x in text]).strip()
	cv2.rectangle(orig_image, (start_X, start_Y), (end_X, end_Y),
		(0, 0, 255), 2)
	cv2.putText(orig_image, text, (start_X, start_Y - 30),
		cv2.FONT_HERSHEY_SIMPLEX, 0.7,(0,0, 255), 2)


plt.imshow(orig_image)
plt.title('Output')
plt.show()

Sample output images:

As you can see, the text detection was not exactly… the best. So, we decided to further venture in picture processing. Some of the solution was simple. For example, the left image could be somewhat solved simply by rotating the image as follows:

The image on the right on the other hand, needed another fix. The issue the model was having was due to the words being dotted, making it hard for it to read. Thus, we looked for a way to connect the dots using morphological operations [4], merging them. The code is as follows. Similarly, the code uses the cv2 library.  We also initialized some constants that will be used later.

import cv2
i = 3
KERNEL = 7
kernelSize = (KERNEL, KERNEL)

Firstly, images are converted to grayscale

def picProcess(img_name='Test.jpg'):
	# load the image, convert it to grayscale, and display it to our
	# screen
	image = cv2.imread(img_name)
	gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Next, the image undergoes ‘opening’. This process ‘erodes’ then ‘dilates’ the image.

kernel = cv2.getStructuringElement(cv2.MORPH_RECT, kernelSize)
opening = cv2.morphologyEx(gray, cv2.MORPH_OPEN, kernel)

Lastly, the image undergoes one last erosion process and is saved.

eroded = cv2.erode(opening.copy(), None, iterations=i)
cv2.imwrite(f'edited_{img_name}', eroded)

The following is the output of the image and the text detection.

We thought these solutions would be enough. However, we encountered other issues…

Part 2

Despite our picture processing, the EAST Model was not going to cut it. In order for it to work, the product had to be placed properly, image taken in just the right angle, fonts had to be a certain style etc. All these variables had to align in order for the program to work. Therefore, we decided to find alternative OCR tools – Google Cloud Vision.[5]

Essential libraries used:

import io
import os
import pandas as pd
from google.cloud import vision

Setting environment:

def saveResponse(image_name='cam0.png'):
    client = vision.ImageAnnotatorClient() #extract image properties

Reading image file, passing it into the Vision API:

with io.open(image_name, 'rb') as image_file:
	content = image_file.read()

image = vision.Image(content=content)
response = client.text_detection(image=image)

Extracting resulting text read and saving it:

texts = response.text_annotations
df = pd.DataFrame(columns=['Description'])
for text in texts:
	df = df.append(dict(Description = text.description), ignore_index=True)

	df.to_excel(f"{image_name[:-4]}.xlsx")
	return df['Description'][0]

With this, we were now able to extract text data from a much wider range of product images! The function returns a string containing all the text read. Eg:

  • Product
  • Returned String

Therefore, what was left to do was to extract the date.

Essential libraries:

import re
from datetime import date

Setting Constants:

MONTH_DICT = {'Jan':1,'Feb':2,'Mar':3,'Apr':4,'May':5,'Jun':6,
'Jul':7,'Aug':8,'Sep':9,'Oct':10,'Nov':1,
'Dec':12,
'JAN':1,'FEB':2,'MAR':3,'APR':4,'MAY':5,'JUN':6,
'JUL':7,'AUG':8,'SEP':9,'OCT':10,'NOV':11,
'DEC':12,
'JANUARY':1,'FEBRUARY':2,'MARCH':3,'APRIL':4,
'MAY':5,'JUNE':6,'JULY':7,'AUGUST':8,
'SEPTEMBER':9,'OCTOBER':10,'NOVEMBER':11,
'DECEMBER':12,
'January':1,'February':2,'March':3,'April':4,
'May':5,'June':6,'July':7,'August':8,
'September':9,'October':10,'November':11,
'December':12}

DIVIDERS = '[\s/.,-]'
DIGIT_MTH = '0[1-9]|1[012]'
DIGIT_DAY = "[012][0-9]|3[01]"
DIGIT_YEAR_4 = "202[1-9]"
DIGIT_YEAR_2 = "2[1-9]"
MONTHS = '''(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|
   Nov|Dec|
	JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|
   NOV|DEC|
	JANUARY|FEBRUARY|MARCH|APRIL|MAY|JUNE|JULY|
   AUGUST|SEPTEMBER|OCTOBER|NOVEMBER|DECEMBER|
	January|February|March|April|May|June|July|
   August|September|October|November|December)'''

Function to extract date from string:

def getDate(strDate):
	dates = []

	#dd/mon/yy
	pattern = re.compile(f'({DIGIT_DAY})({DIVIDERS})({MONTHS})({DIVIDERS})({DIGIT_YEAR_2})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[2],a[0]))
	

	#dd/mon/yyyy
	pattern = re.compile(f'({DIGIT_DAY})({DIVIDERS})({MONTHS})({DIVIDERS})({DIGIT_YEAR_4})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[2],a[0]))

	#dd/mm/yy
	pattern = re.compile(f'({DIGIT_DAY})({DIVIDERS})({DIGIT_MTH})({DIVIDERS})({DIGIT_YEAR_2})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[2],a[0]))
		

	#dd/mm/yyyy
	pattern = re.compile(f'({DIGIT_DAY})({DIVIDERS})({DIGIT_MTH})({DIVIDERS})({DIGIT_YEAR_4})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[2],a[0]))
		

	#mm/dd/yyyy
	pattern = re.compile(f'({DIGIT_MTH})({DIVIDERS})({DIGIT_DAY})({DIVIDERS})({DIGIT_YEAR_4})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[0],a[2]))

	try:
		return(max(dates))
	except:
		return None

Function to convert string to date format:

def convert_date(year,mth,day):
	day_int = int(day)
	#get mth_int
	try:
		mth_int = int(mth)
	except:
		mth_int = MONTH_DICT[mth]
	#get year_int
	if len(year) == 4:
		year_int = int(year)
	else:
		year_int = 2000 + int(year[:2])

	return date(year_int,mth_int,day_int)

def convert_date(year,mth,day):
	day_int = int(day)
	#get mth_int
	try:
		mth_int = int(mth)
	except:
		mth_int = MONTH_DICT[mth]
	#get year_int
	if len(year) == 4:
		year_int = int(year)
	else:
		year_int = 2000 + int(year[:2])

	return date(year_int,mth_int,day_int)

Moving forward into Phase 3

Towards the end of phase 2, we realised that due to the somewhat limited number of API calls per month (1000), it would be best to optimise the number of calls we make. We also aim to bring the different operations together to make it as convenient to use as possible.

 

 

 

Phase 3

In the final phase, we aimed to further improve text recognition accuracy, minimise API calls and finally tie all the codes together with the GUI on the RPi.

Part 1

Merging Images

In a bid to optimise and reduce our Google Vision API calls, we decided to merging the images with our Merge.py module.

from PIL import Image, ImageOps

def merge_images(file1, file2, file3, file4, file5, file6):
    files = [file1, file2, file3, file4, file5, file6]
    images = []
    widths = []
    heights = []
    for file in files:
        img = ImageOps.exif_transpose(Image.open(file))
        (width, height) = img.size
        images.append(img)
        widths.append(width)
        heights.append(height)

    result_width = int(sum(widths)/2)
    result_height = max(heights) * 2
    result = Image.new('RGB', (result_width, result_height))

    count = 0
    half_of_img = len(images)/2
    for image in images:
        if count < half_of_img:
            result.paste(im=image, box=(count*widths[0], 0))
        else:
            result.paste(im=image, box=((int((count-half_of_img)*widths[0]), heights[0])))  
        count += 1

    result.save('result.png')
    

if __name__ == '__main__':
    result = merge_images('Images_b.jpg', 'Images_c.jpg','Images_d.jpg', 'Images_e.jpg', 'Images_f.jpg', 'Images_g.jpg')
  • Merged images from all 6 cameras
Pattern Recognition

Using the Python Regular Expressions library, we ran our date recognition pattern to extract expiry dates from the texts scanned off our product. (DatePat.py)

import re
from datetime import date
import pandas as pd

MONTH_DICT = {'Jan':1,'Feb':2,'Mar':3,'Apr':4,'May':5,'Jun':6,'Jul':7,'Aug':8,'Sep':9,'Oct':10,'Nov':1,'Dec':12,
'JAN':1,'FEB':2,'MAR':3,'APR':4,'MAY':5,'JUN':6,'JUL':7,'AUG':8,'SEP':9,'OCT':10,'NOV':11,'DEC':12,
'JANUARY':1,'FEBRUARY':2,'MARCH':3,'APRIL':4,'MAY':5,'JUNE':6,'JULY':7,'AUGUST':8,'SEPTEMBER':9,'OCTOBER':10,'NOVEMBER':11,'DECEMBER':12,
'January':1,'February':2,'March':3,'April':4,'May':5,'June':6,'July':7,'August':8,'September':9,'October':10,'November':11,'December':12}

DIVIDERS = '[\s/.,-]{0,2}'
DIGIT_MTH = '0[1-9]|1[012]'
DIGIT_DAY = "[012][0-9]|3[01]"
DIGIT_YEAR_4 = "202[1-9]"
DIGIT_YEAR_2 = "2[1-9]"
MONTHS = '''(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec|
	JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC|
	JANUARY|FEBRUARY|MARCH|APRIL|MAY|JUNE|JULY|AUGUST|SEPTEMBER|OCTOBER|NOVEMBER|DECEMBER|
	January|February|March|April|May|June|July|August|September|October|November|December)'''


def convert_date(year,mth,day):
	day_int = int(day)
	#get mth_int
	try:
		mth_int = int(mth)
	except:
		mth_int = MONTH_DICT[mth]

	#get year_int
	if len(year) == 4:
		year_int = int(year)
	else:
		year_int = 2000 + int(year[:2])


	return date(year_int,mth_int,day_int)

def getDate(strDate):
	dates = []

	#dd/mon/yy
	pattern = re.compile(f'({DIGIT_DAY})({DIVIDERS})({MONTHS})({DIVIDERS})({DIGIT_YEAR_2})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[2],a[0]))
		
	#dd/mon/yyyy
	pattern = re.compile(f'({DIGIT_DAY})({DIVIDERS})({MONTHS})({DIVIDERS})({DIGIT_YEAR_4})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[2],a[0]))

	#dd/mm/yy
	pattern = re.compile(f'({DIGIT_DAY})({DIVIDERS})({DIGIT_MTH})({DIVIDERS})({DIGIT_YEAR_2})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[2],a[0]))

	#dd/mm/yyyy
	pattern = re.compile(f'({DIGIT_DAY})({DIVIDERS})({DIGIT_MTH})({DIVIDERS})({DIGIT_YEAR_4})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[2],a[0]))

	#mm/dd/yyyy
	pattern = re.compile(f'({DIGIT_MTH})({DIVIDERS})({DIGIT_DAY})({DIVIDERS})({DIGIT_YEAR_4})')
	for a in re.findall(pattern, strDate):
		dates.append(convert_date(a[4],a[0],a[2]))
	
	try:
		return(max(dates))
	except:
		return None

if __name__ == '__main__':
	print(getDate('01,AUG21'))
Pre-Processing of Images

When our initial images fail to return a valid expiry date, we pre-process these images (PicProcess.py) before running them through the Google Vision API again. This is done to achieve even more accurate text recognition.

import cv2

i = 3
KERNEL = 7
kernelSize = (KERNEL, KERNEL)

def picProcess(img_name='cam0.jpg'):
	image = cv2.imread(img_name)
	gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
	kernel = cv2.getStructuringElement(cv2.MORPH_RECT, kernelSize)
	opening = cv2.morphologyEx(gray, cv2.MORPH_OPEN, kernel)
	eroded = cv2.erode(opening.copy(), None, iterations=i)

	cv2.imwrite(f'edited3_{img_name}', eroded)

if __name__ == '__main__':
	picProcess()
  • Merged Image after Pre-Processing

Part 2

Running all 6 USB webcams on Raspberry Pi and taking pictures with them with our CameraModule.py.

import cv2
import _thread
def take_all_pic():
    for i in [0,2,4,6,8,17]:
        takePic(i)

def takePic(camID=0):    
    os.system(f'fswebcam --no-banner -d /dev/video{camID} -r 2560x1440 -S 1 Images_{camID}.png')

if __name__ == '__main__':
    takePic(1)

Part 3

Telegram Notifications Bot @Fridget Spinners

Another essential aspect of our project is to ensure that the user is updated when items are about to expire. Additionally, the user should be able to check the fridge inventory with ease. In order to achieve this, we have decided to utilise a telegram bot [6], @Fridget Spinners, to push notifications to the user.

With the /track command, the user can start the daily reminder service. At 8am daily, the user will be reminded of items which will expire in 3 days or less.

The /get_inventory command allows the user to be updated instantly on the products in the fridge, as well as their individual expiry dates.

Additionally, the /check_exp_from_days command allows users to check on products in the fridge based on the number of days left till expiry.

  • /track Command
  • /check_exp_from_days Command
  • /get_inventory Command

The full code for our telegram bot can be seen in our TestTelegram.py and Tracking.py modules.

# TestTelegram.py
# importing all required libraries
import telebot
import Tracking

API_KEY = '1723623670:AAFrznP82RF2GAf9w2l9blmsg4JCL5XOgsw'
bot = telebot.TeleBot(API_KEY)
CHATIDS = []

def send_update(chatID, msg):
    bot.send_message(chatID, msg)

@bot.message_handler(commands=['track'])
def start(message):
    # global CHATIDS
    Tracking.track_exp(message.chat.id, CHATIDS) 

# Return inventory list
@bot.message_handler(commands=['get_inventory'])
def get_inventory(message):
	Tracking.get_inventory(message.chat.id)

# TEST FUNCTIONS
@bot.message_handler(commands=['start', 'hello'])
def greet(message):
    bot.send_message(message.chat.id, f"Hello {message.chat.first_name}!")

@bot.message_handler(commands=['message_contents'])
def message_contents(message):
    bot.send_message(message.chat.id, message)

@bot.message_handler(func=lambda m: m.text == 'Hello')
def echo_all(message):
	bot.reply_to(message, message.text)

@bot.message_handler(commands=['check_exp_from_days'])
def check_exp_from_days(message):
    bot.send_message(message.chat.id, 'How many days left till expiry?\n/0, /1, /3, /5, /7 or /14?')
    
    #reminds for exp date with 1 day left
    @bot.message_handler(commands=['0'])
    def check_exp_from_days_0(message):
        Tracking.track_exp(message.chat.id, CHATIDS,0)
    #reminds for exp date with 1 day left
    @bot.message_handler(commands=['1'])
    def check_exp_from_days_1(message):
        Tracking.track_exp(message.chat.id, CHATIDS,1)
    #reminds for exp date with 3 days left
    @bot.message_handler(commands=['3'])
    def check_exp_from_days_3(message):
        Tracking.track_exp(message.chat.id, CHATIDS,3)
    #reminds for exp date with 5 days left
    @bot.message_handler(commands=['5'])
    def check_exp_from_days_5(message):
        Tracking.track_exp(message.chat.id, CHATIDS,5)
    #reminds for exp date with 7 days left
    @bot.message_handler(commands=['7'])
    def check_exp_from_days_7(message):
        Tracking.track_exp(message.chat.id, CHATIDS,7)
    #reminds for exp date with 14 days left
    @bot.message_handler(commands=['14'])
    def check_exp_from_days_14(message):
        Tracking.track_exp(message.chat.id, CHATIDS,14)

def start_tele():
    bot.polling()

if __name__ == '__main__':
    bot.polling()

# Tracking.py
import _thread
import time
from datetime import datetime, timedelta
import pandas as pd
import TestTelegram

HOUR = 8
MINUTE = 0
SECOND = 0
MICROSECOND = 0
FILE = 'Inventory.xlsx'
REMINDER_DAYS = 3
EXP_DATE = 'Exp Date'
PRODUCT_NAME = 'Name'

def get_inventory(chatID):
    TestTelegram.send_update(chatID, "--------------Current Inventory--------------")
    df = pd.read_excel(FILE)
    TestTelegram.send_update(chatID, f"Name:\nExpiration:")
    for index, row in df.iterrows():
        TestTelegram.send_update(chatID, f"{row[PRODUCT_NAME][:35]}\n{row[EXP_DATE].date():%d-%m-%Y}")
    TestTelegram.send_update(chatID, f"{'-'*50}")

def track_exp(chatID, CHATIDS,rem_days = REMINDER_DAYS):    
    df = pd.read_excel(FILE)
    # message = ''
    for index, row in df.iterrows():
        days_left = (row[EXP_DATE].date() - datetime.today().date()).days
        if days_left < 0:
            # print(f"{row[PRODUCT_NAME]} has expired!")
            TestTelegram.send_update(chatID, f"{row[PRODUCT_NAME]} has expired!")

        
        elif days_left == 0:
            # print(f"{row[PRODUCT_NAME]} expires today.")
            TestTelegram.send_update(chatID, f"{row[PRODUCT_NAME]} expires today.")

        elif days_left <= rem_days:
            # print(f"{row[PRODUCT_NAME]} expires in {days_left} day(s).")
            TestTelegram.send_update(chatID, f"{row[PRODUCT_NAME]} expires in {days_left} day(s).")

    if chatID not in CHATIDS:
        CHATIDS.append(chatID)
        try:
            _thread.start_new_thread(start_track, (chatID,))
        except:
            print("Unable to start new track!")   
    else:
        return


def start_track(chatID):
    while True:            
            initial_date = datetime.today()
            time_wait = initial_date.replace(day=initial_date.day, hour=HOUR, minute=MINUTE+1, second=SECOND, microsecond=MICROSECOND) + timedelta(days=0) - initial_date        
            
            print(time_wait.seconds)
            
            time.sleep(time_wait.seconds)
            # Check inventory
            df = pd.read_excel(FILE)
            for index, row in df.iterrows():
                days_left = (row[EXP_DATE].date() - datetime.today().date()).days
                if days_left < 0:
                    # print(f"{row[PRODUCT_NAME]} has expired!")
                    TestTelegram.send_update(chatID, f"{row[PRODUCT_NAME]} has expired!")
            
                elif days_left == 0:
                    # print(f"{row[PRODUCT_NAME]} expires today.")
                    TestTelegram.send_update(chatID, f"{row[PRODUCT_NAME]} expires today.")

                elif days_left <= REMINDER_DAYS:
                    # print(f"{row[PRODUCT_NAME]} expires in {days_left} day(s).")
                    TestTelegram.send_update(chatID, f"{row[PRODUCT_NAME]} expires in {days_left} day(s).")
            #to prevent multiple printing
            time.sleep(1) 

if __name__ == '__main__':
    print(datetime.today().date())

Part 4

Integration of different modules that we’ve created to run the full scan of a product via our Top.py module. 

from numpy import exp
import CameraModule
import BarcodeModule
import CloudVision
import DatePat
from datetime import date
import PicProcessing
import Merge
import UpdateData
import get_user_input

def scan_fn():
    CameraModule.take_all_pic()
    Merge.merge_images('Images_0.png', 'Images_2.png','Images_4.png','Images_6.png','Images_8.png','Images_17.png')
    barcode = BarcodeModule.scanBarcode('result.png')
    productName = BarcodeModule.getProductInfo(barcode)
    expDate = DatePat.getDate(CloudVision.saveResponse('result.png'))

    # if pic has no exp date
    if expDate == None:
        PicProcessing.picProcess('result.png')
        expDate = DatePat.getDate(CloudVision.saveResponse('edited3_result.png'))

    while expDate == None:
        expDate = get_user_input.get_date("Please Select Expiry Date:")
    
    UpdateData.updateData(barcode, productName, expDate)

if __name__ == '__main__':
    scan_fn()

Part 5

Graphic User Interface (GUI) [7]

List of functions that the buttons in the interface uses, like switching from frame to frame, scanning the item and pop-up keyboard.

#List of Functions
def updateAll(description='Completed!', text='Confirm'):
	updateVar()
	updateListbox()
	Popup(description, text)

def updateVar():
	global inventory_df, list_of_entries
	inventory_df = pd.read_excel(filepath)
	list_of_entries = inventory_df['Name'].tolist()

def Item_list(event):
	index = listbox1.curselection()[0]
	Barcode2.config(text = inventory_df.iloc[index, 0])
	Product2.config(text = inventory_df.iloc[index,1])
	Input2.config(text = inventory_df.iloc[index,2].date())
	Expiry2.config(text = inventory_df.iloc[index,3].date())

def forget_all():
    hidekeyboard()
    home_frame.forget()
    item_frame.forget()
	# nutrition_frame.forget()
	# calories_frame.forget()
    manual_frame.forget()
    pname.delete(0,END)
    

def chng_to_home():
	forget_all()
	home_frame.pack(fill = 'both', expand =1)


def chng_to_item():
    forget_all()
    item_frame.pack(fill = 'both', expand =1)
	
def chng_to_manual():
    forget_all()
    manual_frame.pack(fill = 'both', expand =1)
	

def Popup(description, text):
	popup = Toplevel()
	popup.title(description)
	popup.geometry("150x100+300+200")
	popup.config(bg = 'white')
	labelBonus = Label(popup, text=description)
	labelBonus.config(bg = 'white')
	labelBonus.place(x = 75, y = 25,anchor='center')
	B1 = Button(popup, text = text, command = popup.destroy)
	B1.place(x = 35, y = 50)
 
class ImageLabel(Label):
    """
    A Label that displays images, and plays them if they are gifs
    :im: A PIL Image instance or a string filename
    """
    def load(self, im):
        if isinstance(im, str):
            im = Image.open(im)
        frames = []
 
        try:
            for i in count(1):
                frames.append(ImageTk.PhotoImage
                (im.copy()))
                im.seek(i)
        except EOFError:
            pass
        self.frames = cycle(frames)
 
        try:
            self.delay = im.info['duration']
        except:
            self.delay = 100
 
        if len(frames) == 1:
            self.config(image=next(self.frames))
        else:
            self.next_frame()
 
    def unload(self):
        self.config(image=None)
        self.frames = None
 
    def next_frame(self):
        if self.frames:
            self.config(image=next(self.frames))
            self.after(self.delay, self.next_frame)

def scan():
	loading_win = Toplevel()
	loading_win.geometry("200x200+200+200")
	loading_win.config(bg = 'white')
	lbl = ImageLabel(loading_win)
	lbl.pack()
	lbl.load('loading.gif')
	Top.scan_fn()
	loading_win.destroy()
	updateAll()

def back_manual():
 	home_frame.pack(fill = 'both', expand =1)
 	manual_frame.forget()
 	keyboard_frame1.forget()
 	pname.delete(0,END)
 	pname.bind('<Button-1>', showkeyboard)

def updateListbox():
	list_of_entries = inventory_df['Name'].tolist()
	listbox1.delete(0, END)
	for entry in list_of_entries:
		listbox1.insert(END, entry)

def delete():
	index = listbox1.curselection()[0]	
	if get_user_input.get_yn("Are you sure?"):
		global inventory_df
		inventory_df.drop(labels=index, axis=0, inplace=True)
		inventory_df.to_excel(filepath, index=False)
		updateAll("Items Deleted!")
			
def manual_input():
	global inventory_df
	
	inputDate = datetime.today().date()

	if clicked2.get() == 'Custom':
		exp = get_user_input.get_date('Please Select Expiry Date')
	
	else:
		exp = timedelta(days = int(clicked2.get()[:-5])) + inputDate

	food_name = equation.get()
	pname.delete(0, END)
	UpdateData.updateData('-', food_name, exp)
	updateAll() 

def drawKeyboard_man(parent):
	global keyboardFrame
	keyboardFrame = Frame(parent)
	keyboardFrame.pack()

	keys = [
        [ ("Alpha Keys"),
          [ ('q', 'w', 'e', 'r', 't', 'y', 'u', 'i', 'o', 'p'),
            (' ', 'a', 's', 'd', 'f', 'g', 'h', 'j', 'k', 'l'),
            ('capslock', 'z', 'x', 'c', 'v', 'b', 'n', 'm'),
            ('Clear', 'backspace', 'Space', 'Hide')
          ]
        ],
        [ ("Numeric Keys"),
          [ ('7', '8', '9'),
            ('4', '5', '6'),
            ('1', '2', '3'),
            ('0')
          ]
        ]
    ]

	for key_section in keys:
		sect_vals = key_section[1]
		sect_frame = Frame(keyboardFrame)
		sect_frame.pack(side = 'left', expand = 'yes', fill = 'both', padx = 1, 
                pady = 10, ipadx = 5, ipady = 10)
		for key_group in sect_vals:
			group_frame = Frame(sect_frame)
			group_frame.pack(side = 'top', expand = 'yes', fill = 'both')
			for key in key_group:
				key = key.capitalize()
				if len(key) <= 1:
					key_button = Button(group_frame, text = key, width 
                                = 3)
				else:
					key_button = Button(group_frame, text = 
                                key.center(4, ' '))
				if ' ' in key:
					key_button['state'] = 'disable'
				key_button['command'] = lambda q=key.lower(): 
                                key_command(q)
				key_button.pack(side = 'left', fill = 'both', expand = 
                                'yes')

def key_command(event):
	if event == 'clear':
		pyautogui.press('end')
		pyautogui.hotkey('shift', 'home', 'delete')

	elif event == 'hide':
		hidekeyboard()

	else:
		pyautogui.press(event)
		return

def showkeyboard(*args):
	keyboard_frame1.pack(side=BOTTOM)
	drawKeyboard_man(keyboard_frame1)
	pname.unbind('<Button-1>', showkeyboard)

def hidekeyboard():
	keyboardFrame.forget()
	keyboard_frame1.forget()
	pname.bind('<Button-1>', showkeyboard)

def update_drop1(*args):
	clicked2.set(Product_Details
        [ProductVar1.get()])

Creating the main window of the interface, as well as creating the different frames that allow users to switch frame to frame.

root = Tk()
global word
word = StringVar()

# Potential Images of buttons
# addImg = ImageTk.PhotoImage(Image.open
('AddIcon.jpg').resize((100,100), Image.ANTIALIAS))


# Declare variables
var = StringVar(value = list_of_entries)


#Root Frame
root.title("Fridget Spinners")
root.configure(bg = 'white')
width_value=root.winfo_screenwidth()
height_value=root.winfo_screenheight()
#root.geometry("%dx%d+0+0"%(width_value,height_value))    
root.geometry("800x450+0+0")

Label1 =  Label(root, text = "Welcome Home, User!", bg = 'white')
Label1.config(font = ("Monotype Corsiva", "30", "bold"))
Label1.place(x = 235, y = 200)

home_frame = Frame(root, bg = 'white')
item_frame = Frame(root, bg = 'white')
manual_frame = Frame(root, bg = 'white')

Creating the different widgets on the different frames of the interface.

#Keyboard frames
keyboard_frame1 = Frame(manual_frame)
keyboardFrame = Frame()

#Home frame
Label1 = Label(home_frame, text = 'What would you like to do, User?', bg = 'white')
Label1.config(font = ("Times", "20", "bold"))
Label1.place(x = 215, y = 45)

Button1 = Button(home_frame, command = scan, text = 'Scan Item', width = 16, height = 6, bg = '#0ABAB5')
Button1.config(font =("Piboto","12"))
Button1.place(x = 225, y = 110)

Button2 = Button(home_frame, text = 'Manual Input', bg = '#0ABAB5', width = 16, height = 6, command = chng_to_manual)
Button2.config(font =("Piboto","12"))
Button2.place(x = 435, y = 110)

Product_Details = {"Red Meat":'3 days',
	"Processed Meat" : '14 days',
	"Seafood" : '2 days',
	"Poultry": '3 days',
	"Vegetable": '12 days',
	"Fruit": '10 days'}


#Manual Input Frame with keyboard
Label1 = Label(manual_frame, text = 'Manual Input', bg = 'white')
Label1.config(font = ("Times", "18", "bold"))
Label1.place(x = 215, y = 45)

Label2 = Label(manual_frame, text = 'Product Name: ', bg = 'white')
Label2.place(x = 215, y = 85)

global equation
equation = StringVar()
pname = Entry(manual_frame, bg = '#C0C0C0', width = 20, textvariable = equation)
pname.place(x = 320, y = 85)
pname.bind('<Button-1>', showkeyboard)

Label2 = Label(manual_frame, text = 'Product Type: ', bg = 'white')
Label2.place(x = 215, y = 115)

guideline_c = Canvas(manual_frame, width = 250, height = 260, highlightthickness = 0)
guideline_c.config(bg = 'white')
guideline_c.place(x = 500, y = 5)

guideline_frame = Frame(bg = 'light grey')
guideline_frame.pack()
guideline_c.create_window(0, 0, anchor = NW, window = guideline_frame, width = 250, height = 260)

LabelGG = Label(guideline_frame, text = 'General Guideline', bg = 'light grey')
LabelGG.config(font = ('Times 15 underline')) 
LabelGG.place(x = 49, y = 15)

LabelPT = Label(guideline_frame, text = 'Product Type:', bg = 'light grey' )
LabelPT.config(font = ('Times','12','underline'))
LabelPT.place(x = 25, y = 40)

LabelGE = Label(guideline_frame, text = 'Expires In:', bg = 'light grey')
LabelGE.config(font = ('Times','12','underline'))
LabelGE.place(x = 150, y = 40)

redmeat = Label(guideline_frame, text = 'Red Meat', bg = 'light grey', anchor = 'center')
redmeat.place(x = 40, y = 70)
redmeate = Label(guideline_frame, text = '3 days', bg = 'light grey', anchor = 'center')
redmeate.place(x = 165, y = 70)

pmeat = Label(guideline_frame, text = 'Processed Meat', bg = 'light grey', anchor = 'center')
pmeat.place(x = 30, y = 100)
pmeate = Label(guideline_frame, text = '14 days', bg = 'light grey', anchor = 'center')
pmeate.place(x = 165, y = 100)

seafood = Label(guideline_frame, text = 'Seafood', bg = 'light grey', anchor = 'center')
seafood.place(x = 45, y = 130)
seafoode = Label(guideline_frame, text = '2 days', bg = 'light grey', anchor = 'center')
seafoode.place(x = 165, y = 130)

poultry = Label(guideline_frame, text = 'Poultry', bg = 'light grey', anchor = 'center')
poultry.place(x = 45, y = 160)
poultrye = Label(guideline_frame, text = '3 days', bg = 'light grey', anchor = 'center')
poultrye.place(x = 165, y = 160)

veg = Label(guideline_frame, text = 'Vegetable', bg = 'light grey', anchor = 'center')
veg.place(x = 35, y = 190)
vege = Label(guideline_frame, text = '12 days', bg = 'light grey', anchor = 'center')
vege.place(x = 165, y = 190)

fruit = Label(guideline_frame, text = 'Fruit', bg = 'light grey', anchor = 'center')
fruit.place(x = 45, y = 220)
fruite = Label(guideline_frame, text = '10 days', bg = 'light grey', anchor = 'center')
fruite.place(x = 165, y = 220)

ProductVar1 = StringVar()
ProductVar1.trace('w', update_drop1)

ProductMenu = OptionMenu(manual_frame, ProductVar1, *list(Product_Details))
ProductVar1.set(list(Product_Details)[0])
ProductMenu.place(x = 320, y = 113)

Label3 = Label(manual_frame, text = 'Expiry Date: ', bg = 'white')
Label3.place(x = 215, y = 155)

clicked2 = StringVar()

DaysOption = list(set(list(Product_Details.values())))
DaysOption = sorted(DaysOption, key = lambda x: int(x[:2]))
DaysOption.append("Custom")

DaysMenu = OptionMenu(manual_frame, clicked2, *DaysOption)
clicked2.set(Product_Details[ProductVar1.get()])
DaysMenu.place(x = 320, y = 151)

Input = Button(manual_frame, text = "Input", command = manual_input)
Input.place(x = 355, y = 225)

Back = Button(manual_frame, text = "Back", command = back_manual)
Back.place(x = 290, y = 225)


#Logo
canvas = Canvas(root, width = 148, height = 42, bg = 'white', highlightthickness = 0)      
canvas.place(x = 0, y = 0)     
img = ImageTk.PhotoImage(Image.open(LOGO))  
canvas.create_image(0,0, anchor = NW, image = img) 


#Listbox
listbox_frame1 = Frame(item_frame)
listbox_frame1.place(x=215,y=80)

scrollbar1=Scrollbar(listbox_frame1, orient='vertical',width=20)
listbox1 = Listbox(listbox_frame1, selectmode=SINGLE, listvariable = var, width = 48, height = 12,yscrollcommand=scrollbar1.set)
scrollbar1.config(command=listbox1.yview)
scrollbar1.pack(side=RIGHT,fill=BOTH)
listbox1.pack(side=LEFT)
listbox1.bind('<<ListboxSelect>>', Item_list)


#Item Details Label
delete_item = Button(item_frame, text = 'Delete Item', font = 'Times 9 bold', bg = '#cf1d0c', command = delete)
delete_item.place(x = 355, y = 310 )

Label1 = Label(item_frame, text = 'Item List', bg = 'white')
Label1.config(font = ("Times", "18", "bold"))
Label1.place(x = 215, y = 45)

Barcode = Label(item_frame, text = "Barcode", bg = 'white')
Product = Label(item_frame, text = "Product", bg = 'white')
Input = Label(item_frame, text = "Input Date", bg = 'white')
Expiry = Label(item_frame, text = "Expiry Date", bg = 'white')

Barcode.place(x = 215, y = 350)
Product.place(x = 215, y = 370)
Input.place(x = 215, y = 390)
Expiry.place(x = 215, y = 410)


#Item Detail Display
Barcode2 = Label(item_frame, bg = 'white')
Product2 = Label(item_frame, bg = 'white')
Input2 = Label(item_frame, bg = 'white')
Expiry2 = Label(item_frame, bg = 'white')

Barcode2.place(x = 315, y = 350)
Product2.place(x = 315, y = 370)
Input2.place(x = 315, y = 390)
Expiry2.place(x = 315, y = 410)


#Sidebar Buttons
home = Button(root, text = "Home", width = 18, height = 2, bg = '#0ABAB5', fg ='white', command = chng_to_home)
itemlist = Button(root, text = "Item List", width = 18, height = 2, bg ='#0ABAB5', fg ='white', command = chng_to_item)

home.config(font=("Piboto","12","bold"), borderwidth = 0)
itemlist.config(font=("Piboto","12","bold"), borderwidth = 0)

home.place(x= 0, y = 45)
itemlist.place(x= 0, y = 94)


_thread.start_new_thread(TestTelegram.start_tele,())
root.mainloop()

And here are some screen captures of our finalised GUI!

  • Home Screen
  • Starting Screen
  • Inventory List
  • Manual Input of Products

 

 

 

 

References:

[1] Scan QR codes and barcodes with Python. YouTube, 2020. [Online]. Available: https://www.youtube.com/watch?v=IOhZqmSrjlE. [Accessed: 06-Aug-2021].

[2] A. Rosebrock et al., “OpenCV OCR and text recognition with tesseract,” PyImageSearch, 17-Sep-2018. [Online]. Available: https://www.pyimagesearch.com/2018/09/17/opencv-ocr-and-text-recognition-with-tesseract/. [Accessed: 06-Aug-2021].

[3] A. Rosebrock et al., “OpenCV Text Detection (EAST text detector),” PyImageSearch, 20-Aug-2018. [Online]. Available: https://www.pyimagesearch.com/2018/08/20/opencv-text-detection-east-text-detector/. [Accessed: 06-Aug-2021].

[4] A. Rosebrock et al., “OpenCV Morphological Operations,” PyImageSearch, 28-Apr-2021. [Online]. Available: https://www.pyimagesearch.com/2021/04/28/opencv-morphological-operations/. [Accessed: 06-Aug-2021].

[5] “Detect text in images | cloud vision api | google cloud,” Google. [Online]. Available: https://cloud.google.com/vision/docs/ocr. [Accessed: 06-Aug-2021]. 

[6] Badiboy, (2021) PyTelegramBotAPI (Version 3.8.2) [Github]. https://github.com/eternnoir/pyTelegramBotAPI.

[7] Tkinter Course – Create Graphic User Interfaces in Python Tutorial. YouTube, 2019. [Online]. Available: https://www.youtube.com/watch?v=YXPyB4XeYLA. [Accessed: 06-Aug-2021].