In developing AI, annotation is the work of labeling the learning data created to teach the AI judgment criteria—in other words, training data.
When developing AI for the first time or outsourcing it for the first time, it is surprisingly difficult to realize that annotation work is extremely important as it determines the performance of the AI system.
While annotation work currently requires human effort, why is complete automation difficult?
In this article, we will introduce the reasons why it is difficult to automate annotation work, the weaknesses of tools, typical approach methods for achieving automation, and how manual handling demonstrates its power in annotation work.
In this article, we explain specifically what is done in annotation work and why it is important.
| 【Table of Contents】 |
It can be said that complete automation of annotation is difficult. To perform annotation, there are two major methods: human-led and machine-led automation, but currently, manual annotation is required in many areas.
Why is it difficult to automate annotation? Is it impossible to develop an AI that performs annotation work automatically? We will explain from the following two perspectives.
| 1. Full automation of annotation using tools is difficult 2. Adaptability to specific products or industries and quality control |
At this point, it is difficult to completely automate annotation work with tools.
For example, with image data annotation, it is necessary to label which area of the image contains what named object. Also, in performing sentiment analysis of text, information such as negative/positive or "happy"/"sad" is attached to the text data.
However, the automatic detection accuracy of tools is not certain, and incorrect judgments may be made. Furthermore, there are constraints, such as difficulty in handling exceptional cases or difficulty in performing accurate labeling according to requirements.
For these reasons, it is difficult to perform annotation work completely with tools, and human intervention is currently required.
Data annotation needs to use data that reflects the characteristics of specific products or industries. General datasets or datasets from other industries cannot cover everything.
Because information not included in general data is handled, it is necessary to update annotation guidelines or set new annotation rules.
Also, when replacing data with in-house data, that data may affect the performance of the AI system. Human checking and monitoring are necessary to ensure the quality of the data.
These processes require retraining annotators, and these factors make the automation of annotation difficult.
The following various approaches are being considered to streamline annotation work efficiency.
| 1. Active Learning 2. Annotation work efficiency methods 3. Semi-supervised learning |
Each approach is explained below.
In the method called active learning, after performing a certain amount of annotation, an AI uses those as input to suggest tagging for the remaining data. In this process, for items where accuracy is sufficiently secured, the tags provided by the AI are adopted, while for items where accuracy is insufficient, accuracy is improved by performing manual annotation.
This leads to streamlining the annotation work.
There are also approaches to enable annotation work with a small amount of work. For example, in annotation for image data, when performing object detection using a Bounding Box, technology is also being considered that automatically predicts the area simply by clicking on the target object*1.
Also, to support Segmentation, which extracts the area of a specific object, research is being conducted on technology where the machine performs the task in cases that can be sufficiently judged by a machine*2.
Although not an approach for the annotation work itself, there is also a method of using semi-supervised learning as an AI development technique. Semi-supervised learning is a method that attempts to perform classification with high precision even with a small amount of training data by assigning labels to the remaining data based on that small amount of training data.
*1 Training object class detectors with click supervision
*2 Predicting Sufficient Annotation Strength for Interactive Foreground Segmentation
Although various methods for annotation efficiency are being studied, the current situation is that we have not reached a point where all annotation work can be completely automated. In particular, complex annotation work requires a lot of human effort. The situation where annotation work must be performed manually by skilled annotators will continue.
Below, we take up some specific cases where manual annotation is necessary and introduce what kind of effects can be demonstrated by manual work.
In cases where specialized knowledge is required in annotation work, manual work by skilled annotators is important.
For example, when constructing an AI that supports diagnosis based on the content of interviews with patients in medicine, it is difficult to automate with annotation tools. A tool that does not have specialized judgment ability cannot determine what kind of disease is suspected under what conditions. In performing such annotation work, implementation is difficult without a certain amount of knowledge regarding medicine.
Even in streamlining annotation work through active learning, training data that serves as its base is still necessary. This training data must be performed manually.
Creating the training data that serves as the source for active learning with high quality affects the quality of the training data created from the original data. It is necessary to perform the work while possessing knowledge such as what quantity and with what variation training data should be created to achieve efficient annotation work through active learning.
To reduce the cost and man-hours of annotation, many companies consider two options: automation or outsourcing to a specialized annotation proxy service.
Annotation requires a massive amount of time and human resources. To develop high-precision AI, a large amount of data is required; reducing this data improves work efficiency, but learning volume becomes insufficient and AI accuracy declines.
Securing sufficient data and annotation personnel itself becomes a new challenge, which may affect a company's core business.
As a solution, there are annotation automation tools, but there are limits to completely customizing these for in-house use. Especially when adapting them to in-house product data, a problem of new personnel placement may arise.
Furthermore, it will be realistically difficult to cover all annotation processes in-house. This is because roles such as training, checking, and project management are required in addition to the work of the annotators.
To solve these challenges and reduce the cost and man-hours of annotation, many companies choose to request a specialized annotation proxy service rather than pouring man-hours into annotation automation whose effectiveness is hard to predict.
Annotation proxy services are involved from the initial stages of a project and propose the most effective progression methods. As a result, it becomes possible to reduce the cost and man-hours required for annotation and concentrate on AI development.
In this article, we introduced the difficulty of automating annotation work and the effectiveness of manual annotation work. For the streamlining of high-load annotation work, various research and development projects are progressing, but the situation where human effort is important for the work will continue for the time being. High-quality training data creation can be achieved through annotation work by annotators with the expertise and skills corresponding to the implementation content.