Research · engineering case

How we taught cameras to see a car park

This part of TPS Gate did not fit on the project page. We collected our own datasets, trained a detector for Belarusian number plates, built a recognition device on a Raspberry Pi and a service that counts free spaces from ordinary cameras.

Period: 2014 to 2020
Plate at the gate: PR1: a Raspberry Pi and an IP camera
Free spaces: PR 2.0: cameras and a neural net on GPU
First dataset: 982 photos, autumn 2014
Plate detector: Our own, for Belarusian plates
Where it ran: Minsk, a pilot in Vilnius

A large car park shot from above: dozens of cars in rows — What the camera sees and what our neural network saw: every car it found is painted its own colour. The car park by the EPAM office, August 2019.

The same frame after our neural network: every car painted its own colour — What the camera sees and what our neural network saw: every car it found is painted its own colour. The car park by the EPAM office, August 2019.

In short

Recognition was a large line of work of its own, not an add-on to barriers. We started around 2013. I collected the first dataset in autumn 2014: I drove round car parks with a camera in the car and photographed number plates.

By 2019 we had two products. PR1 stands at the barrier: a Raspberry Pi with a camera finds the plate and passes it to the controller. PR 2.0 counts free spaces from the ordinary cameras of a car park and shows them on a map in the app.

There was no ready detector for Belarusian plates, so we trained our own: 3,430 examples of plates and 8,404 images without them. The training command survived inside the device. The cars in the frames were found by a neural network we trained on our own GPU servers.

Not verified: accuracy in real operation. The requirements set a target of 90 to 95% for the number of spaces; a December 2019 presentation said at least 95%. I found no measurements in the archive.

982plate photos in 16 days, 2014
3,430plates used to train the detector
200cameras per server: the PR 2.0 target
10car parks in Minsk, December 2019

Where the facts come fromMy archive: photos with camera metadata, the system image of the PR1 device, requirements and presentations from 2018 to 2021, the platform’s server code.

The whole projectThe controller, the history and what comes next: TPS Gate Europe.

A dataset I collected myself

Autumn 2014, Minsk.

There were no ready sets of Belarusian plates. So I fixed a camera in the car and drove round car parks.

The shoot took 16 days, from 21 October to 6 November 2014: 982 frames on a Nikon D5100, almost 6 GB. Shopping centre car parks, courtyards, office car parks. Many frames show the windscreen pillar: I shot straight from the car.

A training set is built from photos like these: you mark where the plate is and where the background is, and the detector learns from that.

24 small frames from the 2014 dataset: cars in car parks, some shot through the windscreen — Fragments of the 2014 dataset. The photos are coarsened on purpose so that the plates cannot be read.

A frame from the dataset: the back of a car in a car park, shot from inside through the windscreen — A frame from the dataset, shot from the car on 6 November 2014. Plates and a passer-by are blurred.

How it worked

Two products on one platform.

The diagram scrolls sideways.

The bright blue frame marks what we trained ourselves. The rest is ready-made components and our code around them.

PR1: the plate at the gate

A device at the barrier, 2019.

PR1 was installed at the barrier together with an IP camera. It read the plate of an arriving car and passed it to the PMCN controller. The controller opened the way if the plate was on its list.

Hardware. A Raspberry Pi running Raspbian. The system image of 11 May 2019 survived in the archive; our software version was 01.03.01.
Recognition. ffmpeg takes frames from the camera and OpenALPR, an open library for plate recognition, looks for the plate. It had no region for Belarus, so we added a “by” region: our own detector and settings for the size and format of Belarusian plates.
Speed. The plate was read on the device itself, without sending frames to a server, in about 0.3 seconds. We also managed to run our neural network on the Raspberry Pi.
Our code. In Python: parsing the results, a log on the memory card, sending the plate and photo to the server, a process watchdog. If something crashes, the service restarts itself.
Link to the controller. Over UART, on our own protocol with link, network and application layers. The checksum is the same as in the rest of the system: CRC16.
Server. Compares the plate with a tolerance for recognition errors, drops repeats within a set time, and can let guests in by plate.

How we trained the detector

The command we used to train the Belarusian plate detector survived inside the PR1 image:

opencv_traincascade -data …/PlateDetector/OpenAlpr/train-detector-master/out
  -vec …/positive/vecfile.vec  -bg …/negative/negative.txt
  -w 68 -h 15  -numPos 3430  -numNeg 8404
  -featureType LBP  -numStages 12  -minHitRate 0.9999  -maxFalseAlarmRate 0.3

The conf.txt file in the PR1 image. Paths shortened.

3,430 examples of plates, 8,404 images without plates, 12 stages, LBP features. A detector like this runs fast on a weak processor, so it could run right on the Raspberry Pi.

plate_width_mm = 480
plate_height_mm = 110
char_height_mm = 80
postprocess_min_characters = 5
postprocess_max_characters = 8
detector_file = by.xml
ocr_language = leu

Settings of the “by” region: the size of the plate and characters, the length of the plate, our own detector. The characters were read by the standard European OCR in OpenALPR.

A camera frame at the barrier: the lowered boom, the drive cabinet, a courtyard and a taxi behind — A frame from the recognition log at the barrier, 5 June 2019. Contact details on the boom and the cabinet are blurred.

PR 2.0: free spaces

Video analytics on cameras, 2019.

The second task was to know how many spaces a car park has free without putting a sensor on every space. We connected to the cameras already hanging on site and counted spaces from the frames.

It was needed for two things: to show drivers where there is room, and to set the price by occupancy, that is, dynamic pricing. We wrote the requirements down in April 2019:

data refreshes every 15 to 60 seconds;
the number of free spaces is found with 90 to 95% accuracy;
up to 200 cameras on one server;
the algorithm can move to a Raspberry Pi or a similar controller;
several cameras on one car park with overlapping frames;
markup on each camera: road, closed areas, parking spaces;
an archive of frames for up to 6 months at one frame a minute, a heat map and a hint for the driver where to look for a space.

The service had two parts. The manager takes frames from the cameras, stores them and keeps the markup. The recognition module gets a frame with the camera’s settings and returns where the cars stand. The result goes to the main server and from there to the app and the signs.

A courtyard camera frame with coloured markup: green parking zones, blue road lanes — Markup on a camera: parking spaces in green, the road in blue. May 2019.

A real-time recognition screen: the model’s output on a blue background above, frames from two courtyard cameras below — Real-time recognition: the model’s output above, frames from two cameras below. May 2019.

A top view of the car park at Biruzova 4/5 in Minsk with free spaces marked by green dots — Free spaces on the test car park, Biruzova 4/5, Minsk. June 2019.

The ParkMyCarNow app with a map of free spaces at Biruzova 4/5 — The same map in the ParkMyCarNow app.

Neural networks and servers

Our own training on our own hardware.

The cars in the frames were found by a neural network. We trained such networks ourselves on our own GPU servers: we worked out how to build them, how to train them and how to collect data for them.

Training needed large car parks shot from above. In August 2019 we photographed the car park by the EPAM office from a height: more than a hundred frames on the same Nikon D5100. Two of those frames open this page.

Another angle of the same car park after the neural network: rows of cars painted in different colours — The same car park from another point.

A street with parked cars, each outlined by a coloured mask labelled car with a number from 0.97 to 0.99 — A street: each car is labelled car with the network’s confidence, from 0.97 to 0.99.

The ParkMyCarNow architecture had a separate block for this: GPU video analytics servers next to the platform’s main servers.

I cannot recall now exactly how the network was built, and the models and training code did not survive in the archive: they stayed on the team’s servers. I remember the main thing: the network worked very well, and we managed to run it right on the Raspberry Pi.

What came of it

From documents of 2019 to 2021 and my memory.

October 2019: the first car park in Vilnius.
December 2019: 10 car parks in Minsk, a pilot in Lithuania, a project with the Vilnius city administration.
A separate product: video analytics on any street cameras: finding free spaces and recording parking violations.
Data for the city: how long cars stay parked and where there are free spaces. The Vilnius and Kaunas administrations were ready to buy such data to manage traffic and tariffs.
Bike garage: a solar-powered pilot on the ParkMyCarNow engine. Our controller, our camera and the full control kit. Payments and dynamic pricing worked.
2021: PR1 with a camera was part of the kit we offered to barrier makers.

The archive holds failures too: separate folders for “not recognised” and “camera problems”. From them one can reconstruct where the system went wrong.

A model car park with a barrier, marked spaces and a building, labelled ParkMyCarNow.by — A demo model of a car park with a barrier for shows, April 2019.

A camera frame inside a round bike garage with wooden walls, bikes standing in a circle — Our camera in the solar-powered pilot bike garage. A frame from the archive.

All of this really worked. But we never assembled a finished product to sell: we went wide, trying to solve every client’s problem at once. My conclusion: real problems are solved only with focus. More in the story of TPS Gate Europe.

What it means now

For TPS Gate Europe.

Plates and GDPR. In Europe a number plate is usually treated as personal data. Reading plates at the entrance needs a legal basis and a GDPR risk assessment.
Spaces without plates. Free spaces can be counted without reading plates at all: the network only needs to see cars. For the directions on space allocation and bike parking this is the main groundwork.
The code does not carry over, the experience does. Today such tasks are solved by modern open models, so there is no point in carrying over the 2019 code. The value of the archive is in the data, the requirements and the experience of running it on real car parks.

To the TPS Gate Europe page

Sources in the archive

What stands behind each fact.

The plate dataset, October and November 2014: 982 frames with camera metadata.
The PR1 system image of 11 May 2019: software version 01.03.01, the “by” region, the detector’s training command.
PR 2.0 requirements, April 2019.
PR 2.0 camera frames and screenshots, April to June 2019; the car park shot from above, August 2019.
Materials for EPAM with the architecture and the network’s results, October 2019.
ParkMyCarNow presentations from 2019 to 2021 and a draft article for dev.by, December 2019.
Server code of the ParkMyCar platform from 2016 to 2020: receiving recognised plates and counting spaces.