Blog

How can Artificial Intelligence and Computer Vision be used by ports?

The key questions facing ports, harbors and the broader maritime industry are the challenge of handling an ever growing quantity of cargo in a healthy, effective and environmentally friendly way.

Not only are more ships and vessels calling in at ports, but these ships are also ramping up in size and capacity, along with the amount of data, image and video information being produced and processed. Internet of things , automation and artificial intelligence are key to the digital transformation of intelligent and smart ports and harbors.

The knowledge from big data images and videos are itself a challenge, as is finding the best ways to use them for actionable insights, alerts and predictions.

Ports look to artificial intelligence (AI) and video analytics as a way forward more often than not, but what is AI and how can AI video analytics help ports profit from it?

For ports, what are the main benefits of AI ?

At shore, in the harbors, on land and in the air, Artificial intelligence and computer vision are changing transportation. It has the ability to minimize human error, make tasks more efficient and quicker as well as reduce pollution and carbon footprints.

AI for a more sustainable maritime industry and humanity is not just story. Artificial intelligence itself, however is one aspect of a wider framework for digitizing, improving and optimizing port and harbor operations. Smart intelligent ports and harbors are changing the maritime industry.

The same AI solutions used by some of the world’s largest ports, especially Hamburg, Rotterdamn, Singapore, and Los Angeles, are improving business operations, security and safety challenges now.

One way this enhances activities is to develop a framework of support for proactive decision-making based on a predictive behavioral model. It uses methods of machine learning, data science and deep learning to process data and image recognition while video knowledge analytics is processes in a much more efficient manner.

Computers and AI are helping seafarers and harbor masters and security guards and many others with their jobs and challenges. Computer vision video analytics solutions train and test models based on the massive amount of image, video, facial and audio recognition techniques , and much more, as well as historical details for data science and advanced analytics.

Predicting and forecasting the future

Data, images, and videos are collected, organized, transformed and processed; AI is then used to define trends in the logistics supply chain and provide detailed forecast times for the arrival of boats, lorries and containers at terminals, thereby facilitating greater preparation and a more efficient safer process flow.

An AI platform can predict that a container leaving a port six weeks from now is 67.5 percent more likely than average to go missing. It is possible to find an alternate route and suggest it within minutes.

Computer Vision solutions for image and object recognition can be used to forecast potential equipment requirements, safety violations, threat detections , heavy equipment safety issues and risks , even Covid-19 close proximity alerts. AI also can predict and alert based on long-term and short term yard use, identify containers and other assets , identify rust and other under water ship and dock issues with drones, monitor employees, alert proper channels on damage to containers, number of visits to gates, and much more.

Efficiency and stability in ports and terminals are only two major advantages of AI; the other, perhaps greater, is that it can connect various stakeholders in the maritime and supply chain channels.

By aligning their individual digital transformation roadmaps, an AI-enabled smart port and harbor ecosystem driven by Port Community Systems and Port authorities may improve cooperation between various parties, such as the port authorities, cargo owners, third-party logistics providers, container ship companies, and more.

This in turn, might provide mutually beneficial opportunities for efficiency improvement and waste reduction because integrated AI and Video analytics solutions allows data to be exchanged by parties and provides a common interface for observations, forecasts and constraints allowed by AI.

It is evident that AI has great potential for use in ports and terminals, but there are concerns about how to use it. What data elements, such as image recognition or robotics, would be most important to terminals in the future is key to this. As the industry has seen with automation, there is no process or technique that fits the needs of every port or terminal. Therefore, ports and terminals should be hyper-aware of just what they need and where the technology can best be used when applying AI tools to operations.

Pagarba solutions specializes in AI and computer vision based video analytics for the maritime industry and boating and marine industries. Contact us today for a free strategy session.

Schedule your session now

Read More
Cargo ships and the Maritime Industry run the world, but how ?

The maritime industry is filled with various types of sea freight cargo, and with that, there are a ‘boatload’ of different kinds of cargo ships and modes of shipping based on the cargoes available


Container Vessels
currently the most common and popular mode of transport used for carrying 20′, 40′ and 45′ containers. These come in various capacities ranging from about 85 teus (twenty equivalent units) to 15,000+ teus

Bulk Vessels
Used for the carriage of bulk commodities like wheat, Sulphur, iron ore, coal

Breakbulk Vessels
Used for the carriage of various kinds of cargoes :
– bagged cargo (cement, sugar)
– palletized cargo (paint, chemicals)
– timber

Ro-Ro (Roll On / Roll Off) Vessels 
Used for the carriage of wheeled cargo like cars, buses, trucks, excavators.

These vessels can also carry some project cargoes as long as these are loaded on mafi trailers or any other wheeled modes
Can further be classified as PCC (Pure Car Carriers) & PCTC (Pure Car & Truck Carriers)

Multi-purpose Vessels
Used for the carriage of a combination of above cargoes.

Very versatile, popular and useful vessels specially along certain routes which require self-geared vessels and do not have shore handling facilities

Tanker Vessels 
Used for the carriage of various liquid cargoes like oil, chemicals

Crude Carriers 
Used for the carriage of crude oil
– further classified as  VLCC (Very large Crude Carriers) and ULCC (Ultra large Crude Carriers)

LNG Carriers
Used for the carriage of Liquified Natural Gas

Reefer Vessel
Used for the carriage of frozen cargoes or temperature controlled cargoes like fruits, meat, fish


Cargo ships are also classified under different categories based on their size, dimension and weight

The most common classifications (at the time of this post) are :

  • Handy size
    • ships weighing between 28,000-40,000 DWT
  • Handymax
    • ships weighing between 40,000-50,000 DWT
  • Panamax
    • the largest size of ship which can pass through the Panama Canal – DWT of between 60,000 to 80,000 tons
  • Aframax
    • generally tankers weighing between 75,000 and 115,000 DWT
  • Suezmax
    • the largest size of ship which can pass through the Suez Canal – DWT of around 150,000 tons
  • Malaccamax
    • the largest size of ship which can navigate through the Malacca Straits – would have a DWT of ideally between 280000 to 300,000 tons in terms of container ships
  • Capesize
    • vessels larger than Panamax and Suezmax, which cannot pass through either the Panama Canal or Suez Canal and has to pass through the Cape of Good Hope and Cape Horn – above 150,000 long tons in DWT
  • VLCC (Very Large Crude Carrier)
    • supertankers between 150,000 and 320,000 DWT
  • ULCC (Ultra Large Crude Carrier)
    • supertankers between 320,000 and 550,000 DWT
  • Seawaymax
    • the largest size of ship that can fit through the canal locks of the St. Lawrence Seaway – has a DWT of between 10,000 to 60,000 tons
Read More
Why you need Video analytics and video content analysis powered by artificial intelligence

Video analytics has been around for quite some time. Think of CcTV systems, IP cameras , even ring or nest for security and surveillance.

Most of the use cases have been for loss prevention, intrusion detection, catching casino card sharks or some private investigators catching cheating spouses .

It’s always been a manual process involving pre recorded videos or some person or persons actually watching the live footage.

Video analytics is also known as video content analysis or video intelligence or intelligent video analytics. But now thanks to the enormous advances made in AI, machine learning , deep learning and emerging Video technology hardware, video analytics has introduced the comfort of intelligent automation where AI can process more cameras , more objects and less human errors.

Many new uses cases not really thought of before have emerged due to intelligent video analytics and AI. Call these a game-changers if you will, but use cases and applications like traffic cam analytics to counting people at events to automatic license plate recognition to automatic facial recognition. The world is changing and video analytics with AI is a big part of that.

Would a CCTV surveillance camera detecting and searching for people or vehicles, in real-time, to recognize specific events or activities , such as detecting certain risks or accidents and trigger notifications and alerts

So what exactly is video analytics powered by AI and how is it used in the real world to automate processes and gain valuable insights, and what you should consider when implementing an intelligent video analytics solutions in your organization?

video analytics and AI

The main goal of video analytics is to automatically recognize temporal and spatial events in videos. A boat, ship or barge that moves into a harbor and port; a person or boat that moves suspiciously, traffic signs that are not obeyed, boats that leak oil or sewage or containers with to much wear and tear and rust; the sudden appearance of big waves and clouds and smoke; these are just a few examples of what a video analytics solutions with AI can detect.

Real-time video analytics and video search mining

Perform real-time monitoring in which objects, object attributes, movement patterns, or behavior related to the monitored environment are detected and tracked. These types of video analytics systems can also be used to analyze historical data to discover and predict trends as well as make better informed decisions.

It’s a typical forensic analysis video intelligence task to detect and predict trends and patterns that answer business questions such as :

  • How often does this boat come in and out of harbor , port or marina ?
  • How many people or packages are loaded onto a boat or ship ?
  • How many times is a red light run, and what are the specific license plates of the vehicles doing it?
  • How often are employees wearing safety helmets or masks ?
  • Pattern, trends and timing for shipping containers moved around a harbor , port or yard how fast is the process and how many people involved ?
  • corrosion detection patterns for raw materials and other larger equipment object detection
  • When is customer presence at its peak in my store and what is their age distribution?

Video surveillance use cases

Video surveillance is a well known use case and application for Video analytics. It’s a common task that has existed for over 50 years. An individual or company will install cameras strategically to allow human operators to monitor what happens in a room, area, or space.

In theory it seems simple , just have someone watch the camera monitors and feeds. Yet this task is time consuming , inefficient and filled with human error. A human operator is usually responsible for multiple cameras, and in some instances hundreds of cameras and, as several studies have shown, upping the number of cameras to be monitored adversely affects the operator’s performance. Humans just can’t sit and watch 50 different cameras in real time 24/7 and actually be efficient and accurate.

The reality is even if a large amount of hardware is available and generating videos and audio signals, a bottleneck is formed when it is time to process those signals due to human limitations.

Quality Video analytics software with AI can contribute in a major way by providing a means of accurately dealing with the big data volumes of Video and audio information.

Video analytics with Artificial intelligence and object recognition

Computer vision , image and object recognition ,Machine learning and deep learning revolutionized video analytics.

The use of Deep Neural Networks (DNNs) , YOLO and more has made it possible to train video analytics systems that mimic human behavior, resulting in a paradigm shift. Early AI powered video analytics systems were based on classic computer vision techniques (e.g. triggering an alert if the camera image gets too dark or changes drastically) and moved to systems capable of identifying specific objects in an image and tracking their path.

Companies have used OCR (Optical Character Recognition) for years to extract text from images. Think scanning barcodes or letters on a box or document. So expanding that a little , you could utilize OCR algorithms to interpret an image of a license plate. And with that you’d get the license plate number and metadata and that vehicle and registrants, if integrated with some law enforcement like database. But this has limitations. You have to place the camera in a particular position and angle so it could utilize the OCR algorithm and be sure it was filming a license plate.

A real-world application of this would be the license plate recognition at commercial
or residential parking garages where the camera is located near the gates and could film the license plate when the car stops. However, running OCR constantly on images from a traffic camera is not reliable: if the OCR returns a result, how can we be sure that it really corresponds to a license plate?

OCR was the old model, now with better hardware and AI algorithms, deep learning and better video object recognition models are able to identify the exact area of an image in which license plates appear. And then these systems can utilize multiple technologies for better efficiency and accuracy.

So while the deep learning object recognition and HD cameras are step one , OCR algorithms can still be applied to the exact region, call it bounding boxes, leading to more accurate results.

Industry applications

Maritime industry

Counting Boats , ships, barges , containers, railcars, etc and differentiating between large, medium, small as well as law enforcement, military, commercial or pleasure crafts and so on, generates high-value statistics used to obtain insights about traffic patterns, usage patterns and potential risks and violations.

Installing cameras or integrating existing CCtV and IP cameras into a video analytics platform with AI allows for precise control of notifications, metrics , trends , sailors , pilots , drivers, etc.

Automatic license plate or boat or containers identity recognition identifies boats or ships that commit an infraction or, thanks to real-time searching, spots a boat that has been stolen or used in a potential crime.

Smart cities

Video analytics has proven to be a game Changing technology when it comes to transport, aiding in the development of smart cities and smart transportation hubs.

Traffic is getting worse, not better, in many growing cities. This result in an increase in accidents and traffic jams and thus a need for more adequate traffic management measures need to be taken. Video analytics solutions powered by AI can play a key role in these smart city scenarios.

Traffic analysis can be used to dynamically adjust traffic light control systems and to monitor traffic jams. It can also be useful in detecting dangerous situations in real time, such as a vehicle stopped in an unauthorized space on the highway, someone driving in the wrong direction, a vehicle moving erratically, or vehicles that have been in an accident. In the case of an accident, these systems are helpful in collecting evidence in case of litigation.

Vehicle counting, or differentiating between cars, trucks, buses, taxis, and so on, generates high-value statistics used to obtain insights about traffic. Installing speed cameras allows for precise control of drivers. Automatic license plate recognition identifies cars that commit an infraction or, thanks to real-time searching, spots a vehicle that has been stolen or used in a crime.

Real-time parking spot detection

Instead of using sensors in each parking space, a smart parking system based on video analytics helps drivers find a vacant spot by analyzing images from security cameras.

Smart Security

Facial and license plate recognition (LPR) techniques can be used to identify people and vehicles in real-time and make appropriate decisions. It is possible to search for a suspect both in real-time and in stored video footage, or to recognize authorized personnel and grant access to a secured facility.

Crowd management is another key function of security systems. Cutting edge video analytics systems can make a big difference in places such as ports , harbors , marinas , construction sites , stadiums, retail plazas, shopping malls, hospitals, bus and train stations and airports.

These video analytics with AI systems can provide an estimated crowd count in real time and trigger notifications and alerts when a threshold is reached or surpassed. They can also analyze crowd flow to detect movement in unwanted or prohibited directions.

A great example of video analytics used to solve real-world problems is a larger metropolitan city and their smart city pilot platform . They wanted to better understand major traffic events, so they implemented a video analytics and machine learning system to detect traffic jams, analyze weather patterns and weather trend analysis , track parking violations and more. The cctv and IP cameras capture the events and activities, process them and send real-time alerts to city officials.

Real-time people detection.

One great use case for video analytics with AI is a daily counting of the number of people passing. Integration with historical data, and predicting trends may determine the “normal” flow of people according to the day of the week and time of day, and generate notifications and alerts in case of unusual traffic.

If the monitored area is pedestrian-only, the system could be trained to detect unauthorized objects such as motorcycles or cars and, again, trigger some kind of notification and alert.

Video content analysis systems can be trained to detect specific events, specific activities and objects and sometimes with a higher degree of sophistication. One such example is to detects safety risks and violations as soon as possible. Or, in the case of airports, to raise an alert when someone enters a forbidden area or walks against the direction intended for passengers. Another use case is real-time detection of unattended baggage in a public space.

Intrusion detection is still a viable use case , only made better with AI. Filter out motion caused by wind, rain, snow, or animals. Eliminate unwanted or indeed noise interference.

The functionality offered by intelligent video analysis grows day by day in the security domain, and this is a trend that will continue in the future.

A ice hockey rink in North America introduced facial recognition technology in 2019 to improve safety on game days and tournaments at its rink. The system identifies banned people from attending games and enables staff to prevent them from entering the rink.

How does video analytics with AI work exactly ?

Video content analysis can be done in two different ways: in real time, by configuring the system to trigger alerts for specific events and incidents that unfold in the moment, or in post processing, by performing advanced searches to facilitate forensic analysis tasks.

Feeding the video analytics system

The data being analyzed can come from various streaming video sources. The most common are CCTV or IP cameras, traffic cameras and online video feeds. But any video source that uses the appropriate protocol (e.g. RTSP: real-time streaming protocol or HTTPS) can generally be integrated into the solution.

A key goal is coverage: having a clear view of the entire area, and from various angles, where the events being monitored might occur. Big data , aka more data is better, if it can be processed.

Central processing , edge processing decentralized processing

Video analytics software can be run centrally on servers that are generally located in the monitoring station, which is known as central processing. It can be processed back on the cloud (aws, azure ) where the live analytics might be slowed by latency or internet connection. Ring and Nest do most of the real processing back on amazon or google cloud
Servers. Another option is embedded software in the cameras themselves, a strategy known as edge processing.

There are other new emerging tech solutions like blockchain and decentralized peer to peer processing. But nobody has a handle yet on solving the decentralized latency issues regarding real time video streams and real time analytics. Right now things like iPFS can be utilized as a for or decentralized video data storage.

The choice of cameras should be carefully considered when designing a solution. A lot of legacy software was developed with central processing capabilities only. In recent years, though, it is not uncommon to come across hybrid solutions. A good practice is to concentrate, whenever possible, real-time processing on cameras and forensic analysis functionalities on the central server.

With a hybrid approach, the processing performed by the cameras reduces the data being processed by the central servers, which otherwise could require extensive processing capabilities and bandwidth as the number of cameras increases.

In addition, it is possible to configure the software to only send data about suspicious events to the server over the network, reducing network traffic and the need for storage.

Meanwhile, centralizing the data for forensic analysis allows for multiple search and analysis tools to be used, from general algorithms to ad-hoc implementations, all utilizing different sets of parameters that help to balance the noise and silence in the results obtained.

Essentially, you can enter in your own algorithms to get the desired results, which is a particularly flexible and attractive scheme.

Defining scenarios and training models

After the actual physical camera and infrastructure architecture is designed and planned and implemented and configured you need to define the scenarios on which you want to focus and then train the AI models that are going to detect the specific target events and activities.

Boat or container crashes? Crowd flow? Facial recognition , corrosion detection, safety risk violations or at a yacht club recognizing known thrives ? Or a construction site or port harbor warehouse detecting safety violations or potential risks ?

Each scenario leads to a series of basic tasks that the system must know how to perform and act upon.

An example: detect certain boats, ships or containers and eventually recognize their type (e.g. cruise ship, barge , Large container , law enforcement or military boat, train, car, truck), track their trajectory frame by frame, and then study the evolution of those paths to detect a possible risk , problem or crash.

The most frequent, basic tasks in video analytics are:

Image classification:
select the category of an image from among a set of predetermined categories (e.g. boat, container, rust, car, train , person, physically object , tree, forklift, statue).

Localization:
locate an object in an image (drawing a bounding box around the object).

Object detection:
locate and categorize an object in an image.

Object identification:
given a target object, identify all of its instances in an image (e.g. find all 24 foot sailboats in the image).

Object tracking:
track an object that moves over time in a video or multi video cameras

To know more about the basic tasks performed and why you can benefit from video analytics with AI software, reach out to me Peter.jamack@pagarba.io

Read More
Seattle kraken deep diving into analytics

The NHL has a new franchise called the Seattle Kraken. They don’t start playing till next season and there are no players drafted or signed yet.

Their team will be built or selected via The Expansion Draft, which going by prior expansion drafts , will wind up filling the majority of roster positions. Then comes the NHL Entry Draft, which is more for young prospects. And finally free agency where they can sign various players.

Not sure how many free agents get signed by expansion teams. Would a Taylor Hall have signed with the Seattle Kraken and not the Buffalo Sabres this year ? Who knows. And too bad they missed out on all the goaltenders changing teams.

Analytics and data science are here to stay in the NHL

NHL teams are at the edge of artificial intelligence and can utilize a number of technological and analytical tools to narrow the search for prospects and the right player fit or to break a tie between a few players whose potential may appear even at first glance.

There is the traditional video and scouting reports, but now advanced analytics and data science departments on some NHL teams are changing the way they find and sign and build teams. Ron Francis and the Seattle Kraken are at the forefront of building an analytics driven organization.

So over the next year the Seattle Kraken will be analyzing data and building models to help them build a team.

Read More
Artificial intelligence bias

Machine learning models are known to amplify the biases present in the data. These data biases frequently do not become apparent until after the models are long deployed. And sometimes not at all until something goes viral on social media and a company or model gets heat.

To tackle this issue and to enable the preemptive analysis of large-scale dataset, REVISE (REvealing VIsual biaSEs) is a tool that assists in the investigation of a visual dataset, surfacing potential biases currently along three dimensions:

1 – Object-based :

Object-based biases relate to size, context, or diversity of object representation

2 – Gender-based :

Gender-based metrics aim to reveal the stereotypical portrayal of people of different genders

3 – Geography-based

Geography-based analyses consider the representation of different geographic locations

REVISE is a open source tool that automatically detects possible forms of bias in a visual dataset along the axes of object-based, gender-based, and geography-based patterns, and from which next steps for mitigation are suggested.

https://github.com/princetonvisualai/revise-tool

.

Read More
From an idea to Artificial intelligent product gold
artificial intelligence company
artificial intelligence company

Once you know how to analyze everything, your information can be gold. The value and importance of leveraging machine learning to create a true competitive advantage is known to most successful startups & businesses. Artificial intelligence provides actionable insights,  which lead to a stronger business strategy and plan of action. It is essential to have usable data and actionable predictive analytics to make the most of your analytics and data science pipelines, and for a business person to understand the typical product life-cycle flow.

Artificial Intelligence Drives Innovation

Retail Case Study

For several years, inventory and sales information has been obtained by a  network of integrated but silo department store systems. Mergers, Acquisitions, various consultancies and data warehouses turned this business intelligence and ‘big data’ platform into a data swamp, not a data lake. Nobody wanted to use it because it was like a scary swamp monster. 

This system acted as the analytic and AI center of excellence used to train a machine learning model by the Pagarba Solutions data science team. The team helped the founders, directors and C-Level executives identify strategic actionable priorities and act on them in a timely manner.  The current system was the method used to create a demand forecasting  model to make future predictable trends and forecasts.

By analyzing data from the past five years of retail sales, an adaptive  selective model approximated demand for the coming month. Vector auto-regression models coupled with external data sources are often used in machine learning. That made it possible at any time to forecast demand for any particular product. The company was able to maximize inventory and warehouse management by using this solution to provide alternatives available.

BioTech Smart Medicine Inventory Management Hardware Case Study

The Pagarba Solutions client was advised to use multi-modal bio-metric authentication object recognition system that required a user to pass both a facial and voice recognition to gain appropriate authorized access.

Pagarba Solutions Artificial Intelligence experts  suggested introducing enhanced security features to  the system, such as image and object recognition technology like optical character recognition (OCR). This was  for driver’s license or passport ID authentication. Another strategy and plan was to reduce outliers and overfitting by using deep learning algorithms to prevent fake or biased facial recognition. And finally using Natural Language Processing technology( NLP) for security based written quiz like challenges. These changes also culminated in the development of an event-as-a-Service (EVaaS) platform.

Artificial Intelligence Process Flow & Pipeline

Executives, Directors, & Business owners want to know how to use AI and emerging technology to achieve actionable and realistic business goals. What’s the Return on Investment? They are less interested in the details of how the technology works.

Steps of a Artificial Intelligence Strategy & Development project
1> User Experience (UX) is Key

How will your customers and end users use this Artificial Intelligence (AI) product ? Think of the “old school” recommendation engines we see on Amazon, Wal-mart, any ecommerce site or even youtube and netflix or Hulu. A recommendation engine is really a subclass of information filtering systems that predict the “rating” or “preference” a end user would give to an item.

Recommendation systems work best with unbiased explicit user feedback like Amazon and shopping cart purchases or watching a video on Youtube or Netflix or listening to a song on Pandora or Spotify or Apple Music. But in many use cases the application and platform offers mostly implicit feedback scenarios. Think click through rates, page views, google search queires, like or dislikes, up or down voting and so on. This data is heavily biased and we’ve seen people cheat these systems or bully younger users. Click-through rate is heavily dependent on the position of content on a page. Poor UX/UI and web design and it’s like putting Milk at the front of a grocery store. Not ideal if you think with a data-heavy analytics scope. Implicit feedback also tends to perform worse. Google search results are heavily sponsored and ad based on page one now, so a lot of the content may show sponsored results and/or clickbait headlines. And then the site is all spam. This results in high bounce rates after initially high click through rates. This may not matter to a spam site, but a legit business might not be getting the results they desired.

The user experience (UX) can be broken down into three parts:
  1. Initial Impression
    1. What is the user trying to achieve?
    2. How does the user arrive at this experience?
    3. Where do they go?
    4. What should they expect?
  2. Interacting Stage
    1. What should they see ?
    2. Is it clear what to do next?
    3. How are they guided through errors and processes ?
  3. Feedback Loop
    1. Did the end user achieve their goal?
    2. Is there a clear “end” to the experience? Call to Action ?
    3. What are the follow-up steps (if any)?

Knowing what an end user should see during the start, into the interacting stage and feedback after of your app or software will ensure the data science and AI team pick the right features and variables for the models and the data science team trains and continuously improves and updates the AI models on accurate proper data from the start. And it helps the engineering and devops teams to automate the MLOps process flow. And they can build and design valuable automated feedback loops for analytics that offers the best solution for end users.

Read More