Low Latency Image Processing

Welcome to the world of multithreading where we make life extremely difficult. Combine that with image processing, just how much scarier can it get? But here I’ll show you how easy they can also be. We will be working with Python 3.5.2 and OpenCV 3.2.0. Let’s just get started, shall we?

Multithreading in Python

If not done right this will happen

In short terms, multithreading is a technique that allows the code to be run on multiple processors. Multithreading is used primarily for tasks that would block the code such as I/O operations. With the code blocked, the rest of the code will have to wait before continuing. Multithreading allows us to do other tasks while still processing the I/O operation.

So how does this apply to computer vision? To lower our latency we can simply create a thread that does nothing but polls the camera for new frames. This way we don’t have to wait for the camera to give us a new image but rather have a new image to process the second we are done the current image.

The concept behind it is easy but is rarely seen in examples since it can add quite a few lines of code. Also, multithreading can make debugging the code harder, but if it is done correctly you should be seeing lower latency overall. Now on to the code.

Decreasing Latency with Python and OpenCV

The first step is to create a class that’ll do the threading.

import cv2
from threading import Thread

class WebcamThread:
    def __init__(self, source=0):
        self.cam = cv2.VideoCapture(source)
        self.grabbed, self.frame = self.cam.read()

        self.stopped = False

First, we import all the necessary modules including threading. Now we create the constructor for our WebcamThread class that will take a source. If the source is a number, then we open the webcam with that number. But if it is a string, then we will open the corresponding video file.

From there we will get a pointer to our camera or video file. With that pointer, we can now read the frames from the camera. We also create a boolean that signals if our thread should be stopped.

    def start(self):
        <span class="crayon-k ">Thread</span><span class="crayon-sy">(</span><span class="crayon-v">target</span><span class="crayon-o">=</span><span class="crayon-r">self</span><span class="crayon-sy">.</span><span class="crayon-v">update</span><span class="crayon-sy">,</span> <span class="crayon-v">args</span><span class="crayon-o">=</span><span class="crayon-sy">(</span><span class="crayon-sy">)</span><span class="crayon-sy">)</span><span class="crayon-sy">.</span><span class="crayon-e">start</span><span class="crayon-sy">(</span><span class="crayon-sy">)</span>
        return self

    def update(self):
        while True:
            if self.stopped:
                return
            self.grabbed, self.frame = self.cam.read()

    def read(self):
        return self.frame

    def release(self):
        self.cam.release()

    def isOpened(self):
        return self.cam.isOpened()

Next, we create multiple helper methods. When the start method is called, it will create a new thread that will, in turn, keep calling the update method on a separate thread. The update method starts an infinite loop that checks to see if it should be stopped and if it shouldn’t then we keep on grabbing frames.

The read method will return the frame that is grabbed by the update method. The release method will end the thread and release the camera resource so it can be used with other programs. Finally, we have an isOpened method that just checks to see if our webcam is opened. And that is the end of our webcam thread class. That wasn’t so bad, was it?

Now we can create a demo class just to show this off:

import cv2
from WebcamThread import WebcamThread

camera = WebcamThread(src=0).start()

Now in our new file, we import cv2 and also our WebcamThread class. Further down we initialize the WebcamThread and start it directly. Note that we have src=0. This means that we are using the first webcam that is plugged into our computer. You can also pass in a string which means we will play a video file or if you are feeling geeky you can also pass in a mjpeg-stream.

while True:
    frame = camera.read()
    cv2.imshow("Frame", frame)

    key = cv2.waitkey(1) & 0xFF
    if key == ord("q")
    break

camera.release()
cv2.destroyAllWindows()

Now we set a while loop.  In it we continuously read the frame from the WebcamThread class then it shows it on the screen. The code then checks to see if the user has pressed the letter q but this can be easily changed for any keyboard key.  If the user was to press the key specified then it would exit the loop and stop the thread. After doing so will it destroy the windows. One thing to note, however, using imshow is also an I/O blocking operation which means your code will still run a little slower than if you did not have imshow.

That wasn’t so bad, was it? Today we learned how to use threading to import our latency. In almost all situations, you can employ threading to ensure you always have a frame ready so you can lower your latency.