imgReg.py

The imgReg.py contains the image processing part of the Image Recognition Subsystem, as well as passing the images to the CNN.

Variables Defined
  • THRESHOLD – Threshold value of 0.999. Only probability values greater than this threshold value will be considered.
  • img_bound – Value used of 240 to crop the raw image. This corresponds to half the height of the image, to crop out most of the irrelevant part of the image.
  • left_bound and right_bound – Values 150 and 420 respectively. Bounding rectangles X-coordinates lesser or greater than these values respectively means that the target image is 1 grid behind or ahead of our robot’s center point.
  • class_mapping – Python dictionary which contains the mapping of class labels to their respective image number
Functions Defined
  • getBoundingBoxes(contours) – Contours passed into this function are used to generate all the possible bounding rectangles. Based on some facts that we know (rectangles cannot be too small or unproportional), it further filters the possibilities and returns the filtered list of possible bounding rectangles.
  • run(frame, graph, model, count) – Function called by when the HTTP server receives an image. Parameter frame is the raw image array, graph is the tensorflow graph object, model is the CNN object and count is the image count for naming purposes. Raw images are first processed by cropping the irrelevant part of the image using the variable defined above, then converted to grayscale and Gaussian Blur is applied to remove background noise. Next, image thresholding is applied to partition the image into its background and foreground, which enables us to generate the contours of the image. The contours are then passed into the above function to obtain the possible bounding rectangles. Each of the bounding boxes are then cropped out, resized and normalized before passing them into the CNN. If there were no predictions, the function returned a no prediction value (-1). Else, each of this prediction will be compared the threshold value, and we keep the one that is the highest after the comparison. The processed image is then saved into the disk. The coordinates of the bounding rectangles are then analyzed to obtain its relative position to our robot center point. The function then returns the image number of the class prediction, along with its filename (for final display of the captured images), bounding rectangle area, and the relative position of the bounding rectangle.