A Computer Vision-Based System for Metal Sheet Pick Counting

1  Introduction

Companies in the manufacturing sector deal with inventories daily. Inventory in the companies directly affects the business’s cost and ability to meet demand [1]. Failing to monitor the companies’ inventory can lead to excessive or insufficient inventory and pilferage. Excessive inventory can lead to a high inventory holding cost and insufficient inventory risks product shortage costs and can damage customers’ satisfaction. Uncertain inventory levels affect production and procurement planning, resulting in inefficient production and excessive inventory as the company keeps spare resources in case of shortages. To combat the risk of uncertain inventory levels, an accurate material counting process is required. Accurate material counting can reduce inventory costs, material loss, and pilferage. Companies with a large number of laborers are susceptive to pilferage. Workers may exploit the lack of inventory monitoring of the company to steal material from the warehouses or production facilities. This can incur incessant costs to the company and the incurred cost can be significant if the pilfered items are high in value. Effects from pilferage can be reduced drastically by having an accurate material counting system that deters workers from stealing. In addition, a reliable inventory monitoring system ensures that the amounts of materials shown in the financial statements are accurate. This is crucial for companies that require audits to ensure compliance with accounting standards and government agencies.

Material counting is one of the main components of the inventory management system, therefore it requires accurate recording to keep track of the materials inventory. Traditional material counting systems of typical factories rely on employees to count raw materials, Work-in-process, and finished goods periodically, e.g., at least once a year (end of a calendar year), every six months, three months, or every month. This is usually done manually by an operator with a pen and paper and then inputting the records into a computer. The counting and monitoring of materials are repetitive activities, which naturally create stress and boredom for employees and can lead to counting errors resulting from human errors. The activities are ineffective use of human resources that do not add value to the production. The accuracy of the counting process relies on the scrupulousness of the workers. When materials are lost for unidentified reasons, the production cost is negatively affected. Modern technologies used to aid in material counting include Radio Frequency Identification (RFID), barcode systems, and sensors. However, these technologies are intrusive to the workers and machines. On the other hand, cameras do not interfere with the daily operations of the workers and machines. More data in the form of visual images can be recorded by the cameras that can also be used by the analysts to check for any irregular activities in the storage areas.

This study proposes an automatic material counting system using computer vision (CV) to decrease the chance of human errors and thus reduce inventory holding and improve production planning accuracy. The system focuses on computer vision technology that can analyze images from video cameras, which is currently a relatively cheap technology. The cameras are installed at the workstation in the production process. The system reduces repetitive work for high-quality workers and allows the organization’s human resources to focus more on value-added activities for the organization. The significant contributions of this study are as follows:

1)   The proposed CV system can detect metal sheet picks and automatically records the pick information into a data file.

2)   The proposed CV system is able to distinguish the storage area that is being picked from.

3)   The proposed CV system demonstrates the benefits of using CV in industrial applications and a framework in applying CV in the industries.

2  Background

With the advancement of communications and computation technology, electronic devices are better at interacting with other devices, thus leading to more use of modern smart technology, artificial intelligence, and internet of things (IoT) devices in the manufacturing sector [2]. The implementations of these modern technologies in manufacturing companies are the core components of the Industry 4.0 concept. Data is collected from various sources such as sensors, cameras, industrial machines, smart devices, social media, and business transactions. A large number of available data (big data) accelerated artificial intelligence development such as computer vision algorithms that allow computers to detect objects with higher speed and accuracy. But due to a large amount of data and a variety of data formats, it is difficult for humans to manually analyze them. This has resulted in the acceleration of the development of Artificial Intelligence (AI) technology to support such analytics that can analyze big data quickly and extract knowledge that is otherwise difficult for humans to do so. Many studies demonstrate the implementation of AI in the manufacturing industry. For example, Shao et al. [3] developed a time series classification model based on a convolutional neural network for fault detection and product identification from sensors in the process of building a smart factory.

Computer vision technology is a branch of artificial intelligence that trains computers to understand and respond to information in a visual form such as digital image data from cameras. The strength of computer vision technology is its flexibility over sensors because camera images or videos can be analyzed for a variety of purposes such as material counting, material quality checking, people counting, and face detection. Most sensors are designed for specific applications, therefore require complicated processes of integration with other devices to work in a specific application system. Cameras are mostly already installed in the work area and require very little setup compared to sensors. A camera is installed independently from other devices in the production line, unlike a sensor which is a part of the production system and is more complicated to work with and can affect other devices in the same production system. Most sensors used for material counting provide binary data (i.e., count or no count). But if the recorded data from the sensors do not match the amount of material in the facility, the analyst is unable to find the root cause of the discrepancy. A camera can capture the videos for the analyst to check for any abnormalities that may have caused the discrepancy. There are various computer vision models used for specific tasks, the most popular task being object detection. Liu et al. [4] propose a Holistically-Nested edge detection and adaptive Canny fusion model to detect objects. As the computation power of modern computers increases, computer vision models implementing deep learning are becoming more popular. Balarammurthy et al. [5] developed a deep learning-based object detection model to detect pedestrians.

Many industries have leveraged computer vision to improve operations, improve the consumer experience, reduce costs, and increase safety. Kakani et al. [6] explore use cases of computer vision in helping farmers in the agricultural industry. Taheri-Garavand et al. [7] survey computer vision methods that facilitate food grading and quality inspection in the food processing industry. Galata et al. [8] discuss the effective application of computer vision in the pharmaceutical industry. Martinez et al. [9] review the latest developments in computer vision applications in the construction industry. Computer vision technology can also be used to detect defective goods or products in real time. As each product is produced off the production line, a computer processes the images and detects abnormal patterns in each product. This capability can be achieved even with very small materials [10]. In addition, Li et al. [11] use computer vision to detect anomalies such as stains and scratches on mobile phone screens. Ji et al. [12] proposed a method of using computer vision technology to time study by testing it in a simulated manufacturing environment. A common process in most industries that can benefit greatly from applying computer vision is the counting process. Many researchers have applied computer vision in counting objects in various industries. Lins et al. [13] develop a software to automate the counting and classification of aphids to evaluate and predict crop damage from the insects. Franca Albuquerque et al. [14] and Duan et al. [15] utilize computer vision to count fishes and fish eggs, respectively. Yang et al. [16] develop a computer vision-based counting model for steel pipes in the construction industry using YOLOv3 object detection model. Massarelli et al. [17] develop a computer vision system to count and classify microplastics. Using computer vision for the counting process in the industries has been gaining popularity in recent years and our study aims to contribute to the research field by demonstrating a computer vision-based counting system for a metal sheet picking process.

3  Methodology

The proposed computer vision system in this study monitors certain conditions so that the computer vision system can perform detection. This section describes the framework of the computer vision system and the detail of the proposed system.

3.1 Framework of the Proposed Counting System

A computer vision model is developed to automatically count the number of workpieces from a video. The framework is based on a Manufacturing-oriented CV system by Zhou et al. [18] as illustrated in Fig. 1. A camera captures a series of images (video) of the storage areas and sends them to a computer to process the images with the proposed CV algorithms. The CV algorithms used in this study are object detection, glare detection, and optical character recognition algorithms. Each of the CV algorithms is explained in the following sections. Outputs from the CV algorithms are then used by the proposed counting system to determine if a workpiece movement has been detected and the counting system automatically updates the records of the number of workpieces. These two steps are collectively referred to as the CV system and are the extension of the aforementioned Manufacturing-oriented CV system framework. The recorded number of workpieces is then sent to the production planning and inventory control department for further planning and analysis. With accurate information on the inventory levels, planners can perform more accurate production planning and optimization.

Figure 1: Computer vision system framework of the proposed CV system for counting

Equipment used in this study includes an optical device, i.e., a surveillance camera, and a computer. In general, a camera with higher specifications can capture more pixels in a frame, and therefore, provides more information to the CV system. However, the camera cost is higher and high-resolution videos require more computational power. The camera must be set up in a position that can capture the target storage areas and their surroundings. It is recommended that the operation is recorded from the top view so that the camera can capture most of the relevant motions and objects. If one camera cannot capture all the motions and objects related to the operation, additional camera(s) may be needed. This study tests the proposed CV system with only one surveillance camera as it can adequately capture all relevant information about the target storage areas. Specification of the computer required depends on the type of machine learning tools that are implemented. For example, an object detection model generally requires high computational power. If the CV system uses an object detection model, a computer with a graphics processing unit (GPU) may be needed to process the video [19]. A computer with higher specifications can process videos faster, and potentially in real time. The CV system takes digital images captured through the sensing module as inputs and gives detected features and descriptions of the images as outputs. CV algorithms that can be applied in the proposed CV system are feature detection, recognition, segmentation, and 3-D modeling. Detection results of these CV algorithms are then processed by the counting system that automatically updates the records of the number of materials based on pre-defined conditions. The flowchart of the proposed CV system is shown in Fig. 7 at the end of the methodology section.

3.2 Detail of the Case Study

The storage area used to demonstrate the proposed CV system is a metal sheet picking area at a commercial refrigerator manufacturer. The video recording process follows the Personal Data Protection Act B.E. 2019 of Thailand. There is no sensitive personal data from individuals included in the analysis. The steps of the metal sheet picking process are shown in Fig. 2. The metal sheet picking process consists of the following steps:

1.    Two operators walk to the metal sheet storage area with a cart.

2.    The operators pick up the sheet from the storage area.

3.    The operators place the sheet onto the cart.

4.    The operators leave the storage area with the cart.

Figure 2: The steps in the metal sheet picking operation

The metal sheets are lifted and placed on a cart one sheet at a time from the storage area. That is, steps 2 and 3 are repeated as many times as the number of metal sheet picks. The condition to detect metal sheet picking must be clearly defined in the developed computer vision model. Therefore, the condition for when the metal sheet pick is detected and counted is based on the condition in step 3, when the operators place the metal sheet onto the cart. Once a pick is detected, the model detects that a metal sheet has been moved from the storage area.

3.3 Storage Area Assignment

The first step is to define the storage areas. Polygons are drawn by the OpenCV library [20] so that the location where the metal sheet is picked can be known. OpenCV is a popular open-source computer vision software library that is easy to use and has a huge community of developers. Many research papers have used OpenCV to perform image processing in their works [21–24] and, therefore, also used in this study. This step is manually performed by the CV system developer only one time. A periodic check might be required in the case that the camera might have been slightly moved. There are four storage areas defined in this study, as shown in Fig. 3.

Figure 3: The metal sheet storage areas defined by the CV system developer

3.4 Cart Detection

A pick can occur only if a cart has been detected and parked beside one of the storage areas. Therefore, a cart detection model is needed. We collected video feeds from the surveillance camera and randomly selected 100 frames for the cart detection model. All the images are annotated via the labelImg graphical image annotation tool [25]. 80 images were set aside for training and the remaining 20 for validation.

The developed cart detection model uses the TensorFlow framework [26] that features the transfer learning approach [27]. We fine-tuned a Single Shot Detector (SSD) model with MobileNetV2 as the backbone [28]. We trained the model using TensorFlow Object Detection API. We used 320 × 320 size input for our model and trained the model for 5,000 iterations. The optimizer used in the model training is a Gradient Descent with Momentum optimizer. The parameters were configured to learning rate lr = 0.08 with decay factor ρ = 0.997. These parameters are recommended by the TensorFlow Object Detection API and, therefore, used during the training on the customized dataset. After training, the final results achieved were mean Average Precision (