In view of these constraints, we selected a sensor that performs embedded counting, and only communicates the telemetry data to the cloud. This way, we don’t need to upload the entire video stream, which allows us to save bandwidth and guarantee system privacy by design. The sensor is a Kendryte K210, a low-cost micro-controller that includes a camera interface, a neural network accelerator (kpu) and a dual core cpu.

Working with a device with limited power is a performance bottleneck. Unlike classic applications where we can always add more GPUs, here we needed to rely on the capabilities of the available hardware. Based on Kapernikov’s long experience with optimizing industrial computer vision applications, for throughput and latency, we were able to find an appropriate solution for Telraam.

Improving concurrency

A quick profiling session of the first version of the code resulted in something that looks like this:

As the “take picture” phase is showing,  the process starts by waiting for the camera to finish taking its picture, followed by a bit of image processing performed on the CPU. After this, the YOLO inference is done in part by the KPU, although the CPU also needs to do some (non-accelerated) parts. This code is fully sequential: the cpu, camera and kpu all spend a significant amount of time waiting and doing nothing. There must be a more efficient, concurrent way to do this.

The first and easiest way to improve the frame rate is to eliminate the waiting time. Instead, we work with the previous frame while the camera is still doing its job. This way, cpu and camera tasks can overlap. Of course, this requires extra (already scarce) memory: we need a buffer of 147Ko to store the last image while the new one is coming in. However, we already win 41 milliseconds.

The astute reader will notice that there is a gap between the YOLO block and the “picture” block. This is for illustration purposes. In reality, the camera and the “picture” block could happen anytime, since they are asynchronous (the order could be different in reality). YOLO will simply pick up the latest picture taken.

Now that we have the camera out of the way, let’s have a look at our other resources.  Apparently, we are almost using 90% of the cpu. Conveniently, that is without considering the second core that is at our disposal. If we move the task of running the YOLO model to the second core, it costs us another 147 ko of buffering, but this time, we only make up with 10 milliseconds.

Before we continue, it seems almost unfair to have such a huge workload imbalance between the two cores. The first  block is a trivial upload of the picture to a kpu dedicated memory. The only problem with putting it back in the first cpu is that “Uploading is prohibited when the kpu is busy”. Apparently, we are at the mercy of the chip’s specification. But there is a hidden way: direct memory access. This is a feature which provides access to the main system memory, independently of the cpu.

Although this hack does not conform with the unit’s specifications, it has no apparent drawbacks. It does not consume memory, and it enables us to reach the glorious 13.33 fps mark. 

At this point, you may expect that the next step is to make the work done by the kpu and by cpu 2 overlap. But did you notice that gray line at the 70th millisecond ? That is right, there is a late kpu job at the end of YOLO. What’s more, the actual workflow looks a bit different, as the neutral network accelerator is constantly interrupted by the download of the outputs.

Diving into the lion’s den

A finer benchmark reveals some interesting facts:

• The slowest operation is image upscaling. This precisely doubles the size of the channels.
• Channel concatenation takes ages, for an operation that is basically a vector concatenation.
• Some (de)quantize operations are chained on the same data. Maybe we can aggregate those.

The first thing we can do is add a specialized path for upscaling by a factor of 2. The standard implementation provides a generic method that performs a (bilinear) interpolation, but this involves costly floating point operations. Without much effort, this path reduces the resizing to 1 millisecond.

Next, we can remove the channel concatenation operations. In the memory, the data is arranged channel by channel. This means that, provided that the previous operations store their outputs at the right address, there is nothing to do for a concatenation.

Finally, we can look at the operations that convert float values to an 8 bit float (quantize) and vice versa (dequantize). Those operations apply a trivial linear mapping, with some coefficient. In our model, there is an occurrence where the two mappings cancel each other. We can thus remove both operations.

A nice side effect of getting rid of all of those redundant operations, is that it reduces the memory footprint to run the model by 1.1Mo.