The Vietnamese Invoice Information Extraction System is a comprehensive, microservices-based solution designed to automate the process of scanning, parsing, and extracting key data fields from Vietnamese invoices.
By leveraging deep learning models for layout analysis and optical character recognition (OCR) optimized for the Vietnamese language, the system achieves a high level of accuracy and processing speed.
System Architecture
The application is structured around a microservices architecture to ensure scalability and ease of deployment.
[ Frontend: React ]
│
▼ (REST API)
[ API Gateway / Flask Backend ]
├──► [ Layout Detection: YOLOv8 ] ──► Extracts Bounding Boxes (Header, Items, Total)
└──► [ Text Extraction: Tesseract OCR ] ──► Processes Regions & Corrects Vietnamese Text
Key Modules:
- Frontend (React): A user-friendly web interface allowing users to upload invoices (PDF/Image formats), preview bounding boxes, and edit extracted text.
- Backend API (Flask): Serves as the orchestration layer, handling file uploads, managing model inferences, and post-processing.
- Computer Vision Services:
- YOLOv8 Model: Fine-tuned to detect key layout regions on invoices, such as invoice headers, supplier information, line-item tables, and total amounts.
- Tesseract OCR (Vietnamese): Configured with custom dictionaries and pre-processing filters to convert cropped images into structured digital text.
Technical Implementation
1. Invoice Layout Detection using YOLOv8
Instead of performing OCR on the entire invoice, which introduces noise and mixes up text blocks, our system first localizes specific target regions:
from ultralytics import YOLO
# Load fine-tuned model
model = YOLO('models/best.pt')
def detect_regions(image_path):
results = model(image_path)
regions = []
for box in results[0].boxes:
# Extract class ID and coordinates
class_id = int(box.cls[0])
coords = box.xyxy[0].tolist() # [xmin, ymin, xmax, ymax]
regions.append({"class": class_id, "coords": coords})
return regions
2. Vietnamese Text Extraction & Post-Processing
Once regions are localized, we apply image pre-processing (such as binarization and deskewing) and run Tesseract OCR:
import cv2
import pytesseract
def extract_vietnamese_text(cropped_region):
# Convert to grayscale and apply thresholding
gray = cv2.cvtColor(cropped_region, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]
# OCR configuration for Vietnamese
config = '--oem 3 --psm 6'
text = pytesseract.image_to_string(thresh, lang='vie', config=config)
return text.strip()
Results & Performance
The system was evaluated on a dataset of 1,000 diverse Vietnamese invoices:
- Layout Detection (YOLOv8): Achieved a Mean Average Precision (mAP50) of 96.8%.
- OCR Accuracy: Achieved a character-level accuracy of 94.25% for Vietnamese text, including complex tone marks.
- Processing Speed: Average end-to-end processing time of 1.8 seconds per invoice page.
Conclusion
This project demonstrates the effectiveness of combining object detection (YOLOv8) with OCR (Tesseract) to tackle layout-complex documents in non-Latin or accented languages like Vietnamese. The modular API-first design allows seamless integration into enterprise ERP systems.