Why you need Video analytics and video content analysis powered by artificial intelligence

Video analytics has been around for quite some time. Think of CcTV systems, IP cameras , even ring or nest for security and surveillance.

Most of the use cases have been for loss prevention, intrusion detection, catching casino card sharks or some private investigators catching cheating spouses .

It’s always been a manual process involving pre recorded videos or some person or persons actually watching the live footage.

Video analytics is also known as video content analysis or video intelligence or intelligent video analytics. But now thanks to the enormous advances made in AI, machine learning , deep learning and emerging Video technology hardware, video analytics has introduced the comfort of intelligent automation where AI can process more cameras , more objects and less human errors.

Many new uses cases not really thought of before have emerged due to intelligent video analytics and AI. Call these a game-changers if you will, but use cases and applications like traffic cam analytics to counting people at events to automatic license plate recognition to automatic facial recognition. The world is changing and video analytics with AI is a big part of that.

Would a CCTV surveillance camera detecting and searching for people or vehicles, in real-time, to recognize specific events or activities , such as detecting certain risks or accidents and trigger notifications and alerts

So what exactly is video analytics powered by AI and how is it used in the real world to automate processes and gain valuable insights, and what you should consider when implementing an intelligent video analytics solutions in your organization?

video analytics and AI

The main goal of video analytics is to automatically recognize temporal and spatial events in videos. A boat, ship or barge that moves into a harbor and port; a person or boat that moves suspiciously, traffic signs that are not obeyed, boats that leak oil or sewage or containers with to much wear and tear and rust; the sudden appearance of big waves and clouds and smoke; these are just a few examples of what a video analytics solutions with AI can detect.

Real-time video analytics and video search mining

Perform real-time monitoring in which objects, object attributes, movement patterns, or behavior related to the monitored environment are detected and tracked. These types of video analytics systems can also be used to analyze historical data to discover and predict trends as well as make better informed decisions.

It’s a typical forensic analysis video intelligence task to detect and predict trends and patterns that answer business questions such as :

  • How often does this boat come in and out of harbor , port or marina ?
  • How many people or packages are loaded onto a boat or ship ?
  • How many times is a red light run, and what are the specific license plates of the vehicles doing it?
  • How often are employees wearing safety helmets or masks ?
  • Pattern, trends and timing for shipping containers moved around a harbor , port or yard how fast is the process and how many people involved ?
  • corrosion detection patterns for raw materials and other larger equipment object detection
  • When is customer presence at its peak in my store and what is their age distribution?

Video surveillance use cases

Video surveillance is a well known use case and application for Video analytics. It’s a common task that has existed for over 50 years. An individual or company will install cameras strategically to allow human operators to monitor what happens in a room, area, or space.

In theory it seems simple , just have someone watch the camera monitors and feeds. Yet this task is time consuming , inefficient and filled with human error. A human operator is usually responsible for multiple cameras, and in some instances hundreds of cameras and, as several studies have shown, upping the number of cameras to be monitored adversely affects the operator’s performance. Humans just can’t sit and watch 50 different cameras in real time 24/7 and actually be efficient and accurate.

The reality is even if a large amount of hardware is available and generating videos and audio signals, a bottleneck is formed when it is time to process those signals due to human limitations.

Quality Video analytics software with AI can contribute in a major way by providing a means of accurately dealing with the big data volumes of Video and audio information.

Video analytics with Artificial intelligence and object recognition

Computer vision , image and object recognition ,Machine learning and deep learning revolutionized video analytics.

The use of Deep Neural Networks (DNNs) , YOLO and more has made it possible to train video analytics systems that mimic human behavior, resulting in a paradigm shift. Early AI powered video analytics systems were based on classic computer vision techniques (e.g. triggering an alert if the camera image gets too dark or changes drastically) and moved to systems capable of identifying specific objects in an image and tracking their path.

Companies have used OCR (Optical Character Recognition) for years to extract text from images. Think scanning barcodes or letters on a box or document. So expanding that a little , you could utilize OCR algorithms to interpret an image of a license plate. And with that you’d get the license plate number and metadata and that vehicle and registrants, if integrated with some law enforcement like database. But this has limitations. You have to place the camera in a particular position and angle so it could utilize the OCR algorithm and be sure it was filming a license plate.

A real-world application of this would be the license plate recognition at commercial
or residential parking garages where the camera is located near the gates and could film the license plate when the car stops. However, running OCR constantly on images from a traffic camera is not reliable: if the OCR returns a result, how can we be sure that it really corresponds to a license plate?

OCR was the old model, now with better hardware and AI algorithms, deep learning and better video object recognition models are able to identify the exact area of an image in which license plates appear. And then these systems can utilize multiple technologies for better efficiency and accuracy.

So while the deep learning object recognition and HD cameras are step one , OCR algorithms can still be applied to the exact region, call it bounding boxes, leading to more accurate results.

Industry applications

Maritime industry

Counting Boats , ships, barges , containers, railcars, etc and differentiating between large, medium, small as well as law enforcement, military, commercial or pleasure crafts and so on, generates high-value statistics used to obtain insights about traffic patterns, usage patterns and potential risks and violations.

Installing cameras or integrating existing CCtV and IP cameras into a video analytics platform with AI allows for precise control of notifications, metrics , trends , sailors , pilots , drivers, etc.

Automatic license plate or boat or containers identity recognition identifies boats or ships that commit an infraction or, thanks to real-time searching, spots a boat that has been stolen or used in a potential crime.

Smart cities

Video analytics has proven to be a game Changing technology when it comes to transport, aiding in the development of smart cities and smart transportation hubs.

Traffic is getting worse, not better, in many growing cities. This result in an increase in accidents and traffic jams and thus a need for more adequate traffic management measures need to be taken. Video analytics solutions powered by AI can play a key role in these smart city scenarios.

Traffic analysis can be used to dynamically adjust traffic light control systems and to monitor traffic jams. It can also be useful in detecting dangerous situations in real time, such as a vehicle stopped in an unauthorized space on the highway, someone driving in the wrong direction, a vehicle moving erratically, or vehicles that have been in an accident. In the case of an accident, these systems are helpful in collecting evidence in case of litigation.

Vehicle counting, or differentiating between cars, trucks, buses, taxis, and so on, generates high-value statistics used to obtain insights about traffic. Installing speed cameras allows for precise control of drivers. Automatic license plate recognition identifies cars that commit an infraction or, thanks to real-time searching, spots a vehicle that has been stolen or used in a crime.

Real-time parking spot detection

Instead of using sensors in each parking space, a smart parking system based on video analytics helps drivers find a vacant spot by analyzing images from security cameras.

Smart Security

Facial and license plate recognition (LPR) techniques can be used to identify people and vehicles in real-time and make appropriate decisions. It is possible to search for a suspect both in real-time and in stored video footage, or to recognize authorized personnel and grant access to a secured facility.

Crowd management is another key function of security systems. Cutting edge video analytics systems can make a big difference in places such as ports , harbors , marinas , construction sites , stadiums, retail plazas, shopping malls, hospitals, bus and train stations and airports.

These video analytics with AI systems can provide an estimated crowd count in real time and trigger notifications and alerts when a threshold is reached or surpassed. They can also analyze crowd flow to detect movement in unwanted or prohibited directions.

A great example of video analytics used to solve real-world problems is a larger metropolitan city and their smart city pilot platform . They wanted to better understand major traffic events, so they implemented a video analytics and machine learning system to detect traffic jams, analyze weather patterns and weather trend analysis , track parking violations and more. The cctv and IP cameras capture the events and activities, process them and send real-time alerts to city officials.

Real-time people detection.

One great use case for video analytics with AI is a daily counting of the number of people passing. Integration with historical data, and predicting trends may determine the “normal” flow of people according to the day of the week and time of day, and generate notifications and alerts in case of unusual traffic.

If the monitored area is pedestrian-only, the system could be trained to detect unauthorized objects such as motorcycles or cars and, again, trigger some kind of notification and alert.

Video content analysis systems can be trained to detect specific events, specific activities and objects and sometimes with a higher degree of sophistication. One such example is to detects safety risks and violations as soon as possible. Or, in the case of airports, to raise an alert when someone enters a forbidden area or walks against the direction intended for passengers. Another use case is real-time detection of unattended baggage in a public space.

Intrusion detection is still a viable use case , only made better with AI. Filter out motion caused by wind, rain, snow, or animals. Eliminate unwanted or indeed noise interference.

The functionality offered by intelligent video analysis grows day by day in the security domain, and this is a trend that will continue in the future.

A ice hockey rink in North America introduced facial recognition technology in 2019 to improve safety on game days and tournaments at its rink. The system identifies banned people from attending games and enables staff to prevent them from entering the rink.

How does video analytics with AI work exactly ?

Video content analysis can be done in two different ways: in real time, by configuring the system to trigger alerts for specific events and incidents that unfold in the moment, or in post processing, by performing advanced searches to facilitate forensic analysis tasks.

Feeding the video analytics system

The data being analyzed can come from various streaming video sources. The most common are CCTV or IP cameras, traffic cameras and online video feeds. But any video source that uses the appropriate protocol (e.g. RTSP: real-time streaming protocol or HTTPS) can generally be integrated into the solution.

A key goal is coverage: having a clear view of the entire area, and from various angles, where the events being monitored might occur. Big data , aka more data is better, if it can be processed.

Central processing , edge processing decentralized processing

Video analytics software can be run centrally on servers that are generally located in the monitoring station, which is known as central processing. It can be processed back on the cloud (aws, azure ) where the live analytics might be slowed by latency or internet connection. Ring and Nest do most of the real processing back on amazon or google cloud
Servers. Another option is embedded software in the cameras themselves, a strategy known as edge processing.

There are other new emerging tech solutions like blockchain and decentralized peer to peer processing. But nobody has a handle yet on solving the decentralized latency issues regarding real time video streams and real time analytics. Right now things like iPFS can be utilized as a for or decentralized video data storage.

The choice of cameras should be carefully considered when designing a solution. A lot of legacy software was developed with central processing capabilities only. In recent years, though, it is not uncommon to come across hybrid solutions. A good practice is to concentrate, whenever possible, real-time processing on cameras and forensic analysis functionalities on the central server.

With a hybrid approach, the processing performed by the cameras reduces the data being processed by the central servers, which otherwise could require extensive processing capabilities and bandwidth as the number of cameras increases.

In addition, it is possible to configure the software to only send data about suspicious events to the server over the network, reducing network traffic and the need for storage.

Meanwhile, centralizing the data for forensic analysis allows for multiple search and analysis tools to be used, from general algorithms to ad-hoc implementations, all utilizing different sets of parameters that help to balance the noise and silence in the results obtained.

Essentially, you can enter in your own algorithms to get the desired results, which is a particularly flexible and attractive scheme.

Defining scenarios and training models

After the actual physical camera and infrastructure architecture is designed and planned and implemented and configured you need to define the scenarios on which you want to focus and then train the AI models that are going to detect the specific target events and activities.

Boat or container crashes? Crowd flow? Facial recognition , corrosion detection, safety risk violations or at a yacht club recognizing known thrives ? Or a construction site or port harbor warehouse detecting safety violations or potential risks ?

Each scenario leads to a series of basic tasks that the system must know how to perform and act upon.

An example: detect certain boats, ships or containers and eventually recognize their type (e.g. cruise ship, barge , Large container , law enforcement or military boat, train, car, truck), track their trajectory frame by frame, and then study the evolution of those paths to detect a possible risk , problem or crash.

The most frequent, basic tasks in video analytics are:

Image classification:
select the category of an image from among a set of predetermined categories (e.g. boat, container, rust, car, train , person, physically object , tree, forklift, statue).

locate an object in an image (drawing a bounding box around the object).

Object detection:
locate and categorize an object in an image.

Object identification:
given a target object, identify all of its instances in an image (e.g. find all 24 foot sailboats in the image).

Object tracking:
track an object that moves over time in a video or multi video cameras

To know more about the basic tasks performed and why you can benefit from video analytics with AI software, reach out to me Peter.jamack@pagarba.io