.svg){kind=logo}
PoC (Proof of Concept) is an essential process when introducing new AI (Artificial Intelligence) models such as generative AI. However, many people may have questions such as "How specifically does it proceed?" or "What are the challenges and risks associated with implementation?"
In this article, we explain basic information such as the core concept and importance of AI PoC. In the latter half of the article, we focus on specific procedures and challenges.
This content provides practical insights for companies planning to implement AI PoCs.
|
【Table of Contents】 |
1. What is PoC (Proof of Concept) in AI Development?
PoC (Proof of Concept) refers to an experimental initiative to verify whether new technologies or ideas actually demonstrate their effectiveness before introducing them into specific operations or projects. It is translated as "Gainen Jissho" in Japanese.
An AI PoC involves verifying whether a specific AI technology or algorithm actually functions and whether expected effects can be obtained on a small and limited scale before full-scale introduction.
Specifically, AI introduction PoCs are implemented with the following objectives:
- Verification of Technical Feasibility: Whether the AI can execute the assumed tasks using available data.
- Confirmation of Business Improvement Potential: Verifying whether AI technology actually contributes to operational efficiency, automation, and cost reduction.
- Risk Assessment and Identification of Challenges: Identifying anticipated problems and risks before full introduction.
Implementing an AI PoC allows for experimental introduction at low risk, confirming the effectiveness of AI technology in a short period, especially before making large-scale investments.
The Position of PoC in AI Development Steps
AI introduction projects involve multiple processes from the planning stage to development, implementation, and operation, with PoC occupying an important position. The following is the position of PoC in an AI introduction project:
- Planning and Conception: Clarify business challenges and objectives to be solved with AI.
- PoC (Proof of Concept): Build a simplified model using a small amount of data and conduct feasibility tests for specific functions.
- Development: Data collection, model design, and training based on PoC results and requirement definitions.
- Introduction and Operation: Integrating the AI into actual business systems or applications.
Through PoC, it is possible to formulate a full-scale introduction plan and demonstrate a specific direction for integrating AI into the overall system.
The Important Role of PoC
PoC serves the following roles not only in the development process but also in the overall flow of utilizing AI in business.
Gaining Internal Understanding
When AI is introduced for the first time or as a replacement for legacy systems, many employees may feel skeptical about AI technology. By demonstrating how AI functions and contributes to business improvement at the PoC stage, consensus building for technology adoption can be facilitated.
Promoting Data Utilization
Data accumulated within a company transforms into a resource with high value through AI. In the PoC stage, you can verify how existing data can be utilized by AI.
2. How to Proceed with PoC in AI Development
When proceeding with a PoC for AI introduction, it progresses in stages from problem definition to AI model selection and development. Here, we introduce how to proceed with an AI development PoC.
Problem Definition and Requirement Organization
First, it is necessary to define the business processes and challenges to be solved. For example, in a manufacturing site, "poor product yield" might be identified as a challenge.
It is important for the team to identify which processes will yield the greatest effect if automated and where the current bottleneck lies, thereby specifying the target area for the PoC.
Technical requirements necessary for the challenge are extracted. Specifically, the types of data to be used, such as time-series sensor information, are listed, and requirements such as scalability and security are organized as part of the system requirements.
Setting Goals and Evaluation Metrics
Numerical goals are set to measure the success of the PoC. By establishing specific goals like those below, project progress can be tracked quantitatively.
- Reduce the defective rate from the current 5% to 2% or less.
- Increase inspection speed to 100 cases per minute or more.
Metrics for evaluating AI results are also determined. For example, for a classification task in visual inspection, Precision, Recall, and F1 score are used; for regression tasks, MSE (Mean Squared Error) is common.
At the PoC stage, it is often difficult to directly measure or set evaluation metrics for "business process improvement" or "ROI" itself. Therefore, it is important to appropriately set technical evaluation metrics (Precision, Recall, MSE, etc.) that can serve as proxy metrics for achieving final business goals.
The entire team must reach a consensus on why those technical metrics and target values lead to business value.
Data Collection and Preprocessing
First, identify where the data necessary for the PoC objective is located and collect it from relevant data sources (internal DBs, logs, external data, etc.). In the PoC phase, it is common to target a more limited range and volume of data than at the time of full-scale introduction.
Then, the collected data is prepared in a format usable by the AI model. Data cleaning is performed using Python or other tools, followed by missing value imputation or deletion, outlier detection, normalization, and standardization to align scales between data points.
Next, in the case of supervised learning, annotation (labeling) is performed on the data according to requirements and evaluation metrics. In PoC, this is often done within the minimum range required for verification, but the quality of this annotation directly affects model performance.
Finally, feature engineering—such as statistical methods, time-series conversion, or text tokenization—is carried out to extract and generate useful features from raw data to maximize AI model performance.
Selection and Development of AI Models
Candidate AI technologies and algorithms are selected based on the challenges and data characteristics (images, text, numerical values, etc.) to be verified in the PoC. After selection, trial system or prototype design of the model structure is carried out.
First, a simplified baseline model is constructed to grasp baseline performance for comparison. For example, for a classification task, a simple algorithm such as logistic regression or decision trees is adopted as the baseline first, and more advanced methods are considered as needed.
Then, based on the selected method, a simplified model (prototype) is developed to verify the PoC objective. In PoC, the goal is the minimum functionality and precision required for verification.
Each model is trained on small-scale data to narrow down the optimal architecture while checking feature extraction capabilities and inference speed, with parameter adjustments performed if necessary.
Prototype Experiments
In the prototype experiments, which are the core of the PoC, data experiments and simulations are conducted using the built prototype under conditions as close to the production environment as possible.
Since AI model precision fluctuates significantly based on data characteristics and noise, experiments should use production data or equivalent sensor/logs as much as possible to confirm model stability.
During experiments, the trained model is evaluated against a validation dataset to calculate pre-defined metrics. Signs of overfitting or underfitting are checked, and adjustments to model configuration or data volume are made as necessary.
Evaluation of Usefulness
In the final phase of the PoC, verification of technical feasibility is required by checking experimental results against quantitative evaluation metrics.
Next, an assessment is made on whether the obtained improvement effects align with the business value expected in production. For example, whether concrete results like reduced defect rates or increased inspection throughput contribute to business goals is confirmed.
Areas where expectations were not met or new issues that surfaced during the evaluation process are identified, specifying improvement points for data quality, model configuration, system requirements, etc.
Experimental conditions are adjusted based on feedback, and if necessary, ETL processing or hyperparameter re-optimization is performed iteratively. This allows for increasing the reliability of the PoC.
Finally, a comprehensive decision is made regarding business value, technical feasibility, and operational risk to determine whether to proceed to full-scale development.
3. Case Studies of AI System PoC Implementation
There are many cases where PoCs were implemented when introducing AI systems, successfully demonstrating the feasibility of AI. Here, we introduce case studies of AI system PoC implementation.
Evaluating and Verifying AI Chatbot Functions (Showcase)
At "Omotenashi Suite," a platform connecting companies and customers developed by Showcase Inc., a PoC of new AI chatbot functions utilizing generative AI was implemented. This PoC was part of an initiative aimed at further improving business efficiency within JCB Co., Ltd.
The goal of the PoC was to utilize the vast amounts of data and documents accumulated within JCB and automatically generate QA (questions and answers) using generative AI technology, thereby providing internal knowledge to users in a rapid and easy-to-understand format.
As a result of the PoC, it is expected to eliminate the personalization of internal inquiry operations and realize man-hour reduction. Feedback is also planned for service development that simplifies information organization.
Conducting PoC for Voice Order AI Analysis Solution (Advanced Media)
Advanced Media, Inc. and Data Applications Co., Ltd. jointly implemented a PoC for a "Voice Order AI Analysis Solution (tentative name)." It aims to streamline order processing by digitizing voice orders made between sellers and buyers over the telephone and linking them with sales management systems.
In this PoC, it was demonstrated that conversation content can be digitized with high precision through speech recognition rates using the "AmiVoice API" and prompt creation technology using generative AI that leverages insights from DAL (data architecture language) data linkage technology.
The PoC showed the possibility of building new data platforms starting from voice.
4. Four Common Failures in AI Development PoCs
There are points to watch out for in an AI development PoC. Here, through four common failures in AI development PoCs, we introduce precautions and countermeasures.
Insufficient Data Quality and Quantity
If sufficient training data is not gathered in the PoC phase, or if data quality is low, evaluating model performance becomes difficult. Since AI models use large amounts of data for training, high-precision results cannot be obtained with limited data.
Therefore, data needs to be enriched by considering the following means:
- Reviewing the data collection process
- Combining with public datasets
- Utilizing simulation data
It is also important to improve data quality by outsourcing data annotation to specialized companies.
Using Data That Differs Too Much from Reality
If the data used in a PoC differs significantly from production data, PoC results may not reflect actual operating status. For example, even if high precision is achieved in the PoC, performance might fail to meet expectations (e.g., precision drops significantly) once introduced in production.
Therefore, it is important to use data as close as possible to the production environment data. Results should be reflected more realistically using actual data or data based on real scenarios.
Adopting Overly Complex Models from the Start
In an AI PoC, attempting to use unnecessarily complex and high-level models (e.g., latest deep learning models) from the start should be avoided.
This is because large investments may be required during the trial stage, necessitating careful evaluation of cost-effectiveness.
A PoC is meant to verify the effectiveness of AI technology; seeking high precision too early results in excessive investment.
To address this problem, it is best to first perform basic verification with a simple model (baseline model) or a small-scale prototype, then plan for resource expansion for full introduction once results are confirmed. Proceeding in stages minimizes the risk of requiring excessive investment.
Gap Between Excessive Expectations and Results
A PoC is strictly a proof of concept and does not mean a practical-level system will be built immediately. Therefore, having excessive expectations may lead to unfair evaluations of PoC results.
Before starting a PoC, it is extremely important to clearly define and agree among stakeholders on what is aimed for (objective), how much will be done (scope), and what constitutes success (success criteria). Communication to share progress and intermediate results appropriately during the PoC period to correct gaps in perception is also indispensable.
5. Summary
A PoC in AI is an experimental initiative to clarify technical and business risks and determine whether to introduce new AI technology into a company. Through PoC, you confirm how much the AI technology can be applied to operations and decide whether to proceed to the next step.
It is also important for facilitating an understanding of AI within the company. In particular, it serves as a first step for resolving doubts and concerns about AI introduction and forming consensus among stakeholders.
To succeed in a PoC, the quality of data annotation and the selection of evaluation metrics are vital. If data quality is low or appropriate evaluation metrics are not set, the PoC will end in failure, making it difficult to proceed to the final full-scale introduction.
Author
Latest Articles
