Annotation is a crucial task in building AI. Annotation refers to the process of tagging data such as text, audio, and images to give them meaning.
| Related Article: What is annotation? Why is it necessary for AI use? Explaining the process and work involved |
The quality of annotation affects the accuracy of the model being built. The larger the amount of data used, the greater the volume of annotation work, necessitating handling by many members. In cases such as outsourcing annotation, to perform the work efficiently and with high quality, the implementation content must be organized, and annotation quality control must be thorough. This article explains the points of annotation quality control.
| 【Table of Contents】 |
Since annotated data is used for AI learning and evaluation, the accuracy of tagging greatly affects the accuracy of the AI.
By tagging data through annotation, the AI is made ready for learning. Since annotated data is used for AI learning and evaluation, the accuracy of tagging is extremely important. In many cases, annotation is performed by human hands, so depending on the data volume, it will require a large number of personnel and man-hours.
Research by the Association for Natural Language Processing reports that there was a difference of more than double in AI accuracy depending on whether the person performing the annotation was an expert or a beginner. Therefore, "annotation quality control" is important to improve the accuracy of annotation performed in-house.
To perform annotation quality control, it is important to determine clear specifications and tools, and to ensure that annotators and evaluators can be identified. Specifically, implementing the following items will lead to improved annotation quality.
| ① Creation of specifications (rules) for quality control ② Determining the amount of annotation data ③ Selection of tools for performing annotation |
To perform annotation correctly, specifications must be clearly determined. The most important factor in determining specifications is the "annotation standard."
Annotation evaluators set standards based on the purpose of the AI being built. For example, in an object detection AI that detects humans in images, workers tag rectangles for the human area, but it must be decided whether the height direction should be from the top of the head to the tips of the toes or just the upper body, according to the objective. If standards are not clear, differences in accuracy will arise, much like the difference between expert and beginner annotation. In other words, how clearly specifications can be defined directly links to the accuracy of the AI. If specifications are clear, differences in accuracy are less likely to occur even if there are multiple workers. These specifications must be recorded in a specification document so that everyone involved in the annotation can check them at any time.
At the start of annotation, workers are not used to the specifications and tagging accuracy tends to decrease, so initially, the evaluator checking the data in detail and pointing out mistakes leads to quality improvement.
As annotation progresses, there may be data where workers cannot judge how to apply tags. For example, when tagging object positions in object detection, there may be cases where objects overlap. For things that cannot be judged, evaluators should establish new standards and update the specification document.
After tagging is complete, a double check of the annotation is performed by the worker and evaluator, and if there are no problems, the process proceeds to AI construction and evaluation. Evaluators should check for any corrections to the annotation standards based on the AI evaluation results.
The data created in annotation has many variations, and the higher the tagging accuracy, the higher the accuracy of the target AI. Therefore, annotation should be performed on as much data as possible, but as the data volume increases, the burden on workers increases, which can lead to tagging mistakes. Therefore, evaluators control quality by adjusting the amount of data and considering the burden on workers. Annotation quality can be improved by adjusting the amount of data tagged per day based on the worker's tagging speed and the deadline for when training data is needed, or by adjusting the number of workers.
Tools used for annotation must be prepared according to the target AI. Tool selection is very important in quality control because the built-in functions and ease of use affect the speed and accuracy of annotation. Methods for preparing tools include selecting from open source or creating them in-house. If there is an open-source tool that fits the target AI, it can be prepared without spending time or money, but if an optimal tool cannot be found, it may be created in-house. Benefits of creating tools in-house include being able to add requests from workers as features after the work starts and being able to repurpose them for other tagging by making improvements.
There are two main types of annotation methods:
| ① Prepare annotation tools and implement in-house ② Outsourcing to other companies |
When performing annotation, dedicated tools are prepared to improve work efficiency. Dedicated tools allow for intuitive operation, which increases work speed and also leads to the prevention of human error. When implementing in-house using dedicated tools, note that while securing work personnel is natural, personnel with knowledge of annotation and AI are also needed to perform tasks such as tool selection.
There are many companies that offer annotation agency services. When outsourcing, everything from the formulation of specifications and rules to tool preparation can be left to them, so training data can be prepared as long as the purpose of the AI being built is clear. If the financial burden becomes too great by outsourcing all tasks, it is also possible to outsource only a portion. By utilizing agency services while considering what can be done in-house, such as requesting only the execution of annotation, high-quality annotation can be implemented while lowering costs.
| Related Article: Should I outsource to an annotation company or do it in-house? How to choose a company? A comprehensive guide to the benefits of outsourcing! |
We introduced an overview of annotation and methods for quality control. The points of annotation quality control are the following three:
① Creation of specifications (rules) for quality control
② Determining the amount of annotation data
③ Selection of tools for performing annotation
By keeping these points in mind, you can aim to improve the quality of annotation. Annotation quality control directly links to the accuracy of the AI being built, so be sure to execute it. If it is difficult to organize these three points in-house, consider outsourcing annotation to other companies.