In machine learning, data collection is a crucial step that forms the foundation of a model. However, AI system development generally requires a vast amount of data. Furthermore, because a balance of data types and other factors must be maintained, many people may find it difficult to collect data on their own.
In this article, we explain the procedures, methods, and points to consider regarding data collection. We also explain whether collecting data in-house or outsourcing to a specialist with high data collection technology is more advantageous in terms of cost. If you prioritize cost-performance, please use this as a reference.
| 【Table of Contents】 |
Source: Ministry of Internal Affairs and Communications"Data Science Process Using Machine Learning"
Data collection for machine learning refers to the process of gathering information (data) for an AI to learn from. Data collection is a vital process that forms the foundation of a model.
Because a system is built based on the information obtained through data collection, the quality of the data significantly impacts the precision of the model. Specifically, factors such as consistency, quality, diversity, and balance of data collection are emphasized. Collecting data that meets these elements is the foundation for a model to make more accurate predictions.
Methods for data collection in machine learning can be broadly divided into three categories.
| ① Collecting data in-house ② Using open datasets ③ Outsourcing to specialists |
Here, we explain the difficulty of each method and how much effort is required.
If your company has accumulated sufficient data or has enough resources for data collection, you can complete the data collection internally.
However, data collection is not simply about gathering a large amount of data. Without personnel such as AI engineers or data analysts who can accurately judge data bias and quality, there is a high probability that the system will have low precision. Since such personnel are rarely available in-house, it can be said that performing data collection independently is difficult.
It is also possible to develop systems using open data collected by governments or universities. Data from various media is published, such as the following:
| Data Type | Representative Examples of Datasets |
| Image | MNIST Open Image |
| Video | YouTube-8M Dataset UCF101- Action Recognition Data Set |
| Text | Google Books Aozora Bunko |
| Audio | AudioSet Voice Actor Statistics Corpus |
Open data is basically available for anyone to use. However, please check the terms of each dataset regarding whether commercial use is permitted.
By utilizing open data, data collection can be completed without any effort or cost. However, because appropriate open data for all industries and fields does not necessarily exist, there are many cases where open data cannot be utilized.
Additionally, it would be difficult to build a model optimized for your company using only open data. Therefore, to construct a high-precision model, ingenuity is required, such as utilizing it in combination with data collected independently.
If a unique, high-precision model is required, outsourcing to specialists is the most realistic choice. Among specialists, there are companies that handle everything from data collection to modeling and revisions based on evaluation.
If there are steps within your system construction that you are outsourcing, it is a good idea to check if you can also request data collection.
There are also companies that handle only data collection. Data collection is an important process that serves as the foundation of a model. By requesting data collection from a company with excellent quality, quantity, and know-how, you can build a foundation for a high-precision system.
Data collection is performed through the following steps:
| ① Problem definition and goal setting ② Identifying required data ③ Gathering data according to the objective ④ Adding required information to data (Annotation) |
In the first step, specific problems and goals are clearly set. For example, suppose you are a manufacturing company and want to create a system that automatically detects defective products on a production line. In that case, the goal might be "automatically detecting defective products on the production line with AI."
Once the goal is set, the data necessary to achieve it is identified. For example, if you are building a defective product detection system, you will need image data of good products and defective products. At this time, depending on the type of defect, you must consider that data corresponding to each type will also be required.
Next, data is actually collected. If you are developing an image recognition system, collect image data; if you are developing a text analysis system, collect text data.
If sufficient data exists within your company, you can utilize it. However, collecting anomaly data is often difficult.
For example, in building a defective product detection system, there will often be a shortage of image data for defective products. In that case, methods such as intentionally creating defective products on the production line and collecting that image data can be considered.
Additionally, if you want to suppress effort and cost, using open data is an option. However, because it is difficult to develop a system that matches your company using only open data, gathering data from external sources can be said to be essential for developing a high-quality model.
In this process, maintaining data quality is important. Gathering inappropriate data can cause the model's learning results to be distorted. Therefore, data collection should be performed while coordinating closely with engineers or data analysts who possess specialized knowledge.
Furthermore, this process is not a one-time event; it is required to be performed repeatedly to increase the precision of the system.
An AI cannot start learning simply by importing data. To train the system, information must be added to the data. This is annotation work.
For example, when making a system to detect suspicious persons, you label suspicious movements as "suspicious" and have it learn normal movements as "normal." Data where the correct answer corresponding to each example is properly labeled is called training data. By learning from training data, an AI becomes able to distinguish between suspicious and normal persons. Then, it acquires the ability to accurately identify objects in new images as well.
Because this annotation work requires expertise and a large amount of time, there are many cases where it is outsourced to a specialized annotation company. Accurate annotation by humans with specialized knowledge is indispensable for obtaining high-quality learning data.
Therefore, when considering outsourcing related to data collection, it is recommended to also take the annotation work into consideration.
When constructing a dataset, it is necessary to pay attention to the following three points:
| ① Data quality must be maintained ② A large amount of data is required ③ Attention must be paid to personal information and privacy |
We will explain the reasons for each.
| 注意点 | 内容 |
| Is there any bias in the data? | If data is biased toward certain classes or features, the AI can only learn the characteristics of that biased data. Therefore, ensure that data for each class is gathered in a balanced manner during data collection. |
| Are necessary patterns covered? | It is important to collect data that covers all cases the AI must handle. In other words, you should predict all situations the AI might encounter and collect data that encompasses them. |
| Is there any noise? | Noise refers to unnecessary information contained in the data, which can impact the AI's learning precision if there is a lot of it. To reduce noise, methods include strictly performing quality control during the data collection process or removing noise through post-processing. |
If a high-quality dataset cannot be gathered, the system will have low precision. To gather high-quality data, it is necessary to pay attention to points such as the following:
If even one of these is missing, there is a possibility that it will become an AI system with low precision. For example, if you want to make an image recognition AI for horses and donkeys but gather many images of horses and extremely few images of donkeys—meaning the data is biased—a system may result that can only judge all subjects as horses.
Additionally, if necessary patterns are missing, the system will process unlearned data, causing inference precision to drop significantly. For example, when making a facial recognition AI, you must collect face image data under various angles, expressions, and lighting conditions.
Data quality is important, but if the quantity cannot be secured in the first place, it is highly likely to result in a model with low precision. Especially in image recognition tasks, it is normal for thousands to tens of thousands of data points to be required per class.
Also, to create a high-precision model, it is necessary to gather negative data (dummy data) that does not contain the target objects. Furthermore, as a result of data selection, data that will not be used inevitably occurs. Therefore, a large amount of data is required at the data collection stage.
As a result of collecting necessary data, information related to personal information or privacy may be included. Failure to manage this information appropriately will lead to issues such as personal information leaks or privacy violations.
Especially when performing annotation work through crowdsourcing or offshore, the risk of information leakage increases. Therefore, when requesting data-related operations externally, it is important to choose a company with an established security structure.
Include appropriate data processing and security measures in the contract. Additionally, it is a good idea to consider methods for protecting privacy by performing data anonymization or pseudo-anonymization as necessary.
In conclusion, for the following reasons, outsourcing is often more advantageous in terms of efficiency and cost.
| ① Focus on in-house core business ② Potential for cost advantages in the long run ③ Access to high-quality datasets |
Here, we explain each reason.
By outsourcing data collection, employees have the advantage of being able to concentrate on core business that only internal staff can perform. If internal staff were to handle everything from data collection to core business, they might face a situation where they must manage unfamiliar data collection work alongside their core tasks, potentially leading to a shortage of internal resources.
In that case, it will take extra time to build a machine learning model, and the speed of the PDCA cycle will drop. As a result, it leads to a situation where technological innovation is delayed, falling behind competitors.
From the perspective of effectively utilizing limited resources, it can be seen that outsourcing provides a temporal advantage.
By outsourcing data collection, education and labor costs can be reduced, potentially leading to cost advantages in the long run.
Of course, outsourcing data collection incurs a corresponding cost, but if in-house personnel unfamiliar with the task were to handle it, it is highly likely that even more man-hours would be required. Furthermore, it is necessary to spend time and money training employees who are not familiar with data collection.
Therefore, even if the funds paid externally increase, it is often more advantageous in terms of cost in the long run.
Companies specializing in data collection and annotation possess numerous success stories and examples of failure. Because specialized companies perform data collection by utilizing those insights, high-quality datasets can be obtained.
Gathered data serves as the foundation of a system, so high-quality data is indispensable for constructing high-precision models. If you want to increase precision as much as possible, try requesting services from a proven specialized company.
In this article, we explained the methods and points to consider regarding data collection.
In machine learning, training data serves as the foundation of a system, so it significantly impacts model quality. While some data can be obtained easily for free, like open data, data collection tailored to your company is essential for obtaining high-quality data.
However, gathered data may go to waste if data collection is not performed through appropriate procedures. To avoid such situations, in addition to understanding the quantity and quality that should be collected, ensure you collect data while also paying attention to personal information and privacy.