Annotation is a crucial task in building AI. Annotation refers to the process of tagging data such as text, audio, and images to provide "meaning." To make data usable as training data for AI, it is necessary to perform tagging manually. This process is annotation. The quality of annotation is an important factor that affects the accuracy of the model being built.
The larger the volume of data used, the more the volume of annotation work increases, requiring many members to handle it. If it is difficult to handle with internal resources, outsourcing the annotation work becomes an option. To ensure that the outsourcing partner performs the work efficiently and with high quality, it is necessary to organize the implementation details and compile them into a specification document. But how should a specification document for ordering annotation work be created?
In this article, we explain how to create an annotation specification document.
| Related Article: Should I outsource to an annotation company or do it in-house? How to choose a company? A comprehensive guide to the benefits of outsourcing! |
|
【Table of Contents】 |
Since the automation of annotation work is often difficult, multiple workers (annotators) must perform the task to implement annotation for large amounts of data. It is ideal if you can secure enough internal annotators, but if that is difficult, you will outsource the work. In this case, the document that defines the outsourcing conditions is the annotation specification document.
The more data that is available, the higher the possibility of improving AI performance. Therefore, a large amount of data must be prepared in the development of AI.
The quality of annotation directly relates to the performance of the AI being developed. No matter how excellent an AI model you build, if the quality of the data used for training is poor, the AI will not be able to perform processing with sufficient accuracy. By using data correctly tagged through proper annotation work, you will be able to draw out the maximum performance of the AI.
When outsourcing work, it is necessary to clarify the specifications. If the work is not performed according to your intentions, you will not be able to create high-quality training data, even if you spend cost and time on the annotation work.
Therefore, it is necessary to organize the implementation method, implementation details, volume and type of target data, etc., into a document and clarify them as contract conditions.
So, how should an annotation specification document be created? Below, we introduce the steps for creating an annotation specification document.
The required annotation data differs depending on the machine learning model to be built. Annotation is implemented for the purpose of model construction, and naturally, the details of the machine learning model construction must be finalized in order to place an order for annotation work.
For example, if you are developing a machine learning model that implements visual inspection in a factory, one example of required data would be image data of "good products" and "defective products." To classify these accurately, in addition to tagging each image as "good" or "defective," it is necessary to tag scratches, dents, chips, etc.
In this way, the annotation specifications are considered based on the premise of what kind of training data is required for the AI model to be built.
Next, you concretely organize what kind of annotation work will be performed.
First, define the sense of volume for the annotation work. Using the construction of the machine learning model for visual inspection mentioned above as an example, target the amount considered necessary for training, such as 1,000 images of "good products" and 1,000 images of "defective products." Since the volume of work is important information when obtaining quotes from candidates for the annotation work, ensure it is defined correctly.
Also, clarify what kind of annotation work you are requesting. If you are not well-versed in AI development and specific descriptions are difficult, work out the details while consulting with the company developing the AI or candidates for the annotation work. For example, in the visual inspection of products, the task details would include the distinction between scratches and dents effective for classifying good/defective products, classification by color, etc.
In addition, to ensure quality when implementing annotation work, the establishment of rules is also important. If it is difficult to establish annotation rules in-house, it is a good idea to outsource this task as well. When creating the annotation specification document, clearly state whether you are also requesting the creation of rules.
If the annotation rules are not clear and easy to understand, each annotator will perform annotation using their own rules. This can cause a decrease in annotation quality, so caution is required.
To improve the quality of work, elements related to the task, besides the content of the work itself, should also be defined as specifications. Specifically, we recommend defining the following details.
If possible, the procedure for proceeding with annotation work should also be organized as a specification document. To improve the accuracy of annotation, double-checks by reviewers are also effective. As a specification, you can consider asking for a role to check the annotation results as a reviewer, in addition to the annotators who actually perform the annotation. Also, to improve annotation rules, it may be effective to request as a specification that rules be reviewed after annotation work has been implemented for a certain period.
Consideration of security is also necessary, especially when requesting the annotation of highly confidential data. While ensuring this on the contract level by setting confidentiality clauses in the agreement, security measures should also be required at the actual work level.
For example, if information management standards exist in your company, you should share them and require compliance. If such standards do not exist, you should consider necessary measures while consulting with potential outsourcing partners.
Furthermore, it is also necessary in practice to determine the storage method for annotation results, file formats, etc. If the company developing the AI and the company outsourced for annotation work are different, there is a possibility that data cannot be obtained in the form desired by the AI development partner due to differences in perception. While checking the requirements of the AI development partner, define the format in which the annotation work results should be delivered.
In this article, we explained how to create the specification document required when outsourcing annotation work. If you are not well-versed in AI development, creating a specification document for outsourcing annotation work may be a difficult task. One option is to consult companies that undertake annotation work, including ours, about creating specifications. If you are having trouble ordering annotation work, please feel free to contact us.