This commit is contained in:
254
docs/version2.x/ppocr/model_train/PPOCRv3_det_train.en.md
Normal file
254
docs/version2.x/ppocr/model_train/PPOCRv3_det_train.en.md
Normal file
@@ -0,0 +1,254 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-OCRv3 text detection model training
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
PP-OCRv3 is a further upgrade of PP-OCRv2. This section introduces the training steps of the PP-OCRv3 detection model. For an introduction to the PP-OCRv3 strategy, refer to [document](../blog/PP-OCRv3_introduction.md).
|
||||
|
||||
## 2. Detection training
|
||||
|
||||
The PP-OCRv3 detection model is an upgrade of the [CML](https://arxiv.org/pdf/2109.03144.pdf) (Collaborative Mutual Learning) collaborative mutual learning text detection distillation strategy in PP-OCRv2. PP-OCRv3 further optimizes the detection teacher model and student model. Among them, when optimizing the teacher model, the PAN structure LK-PAN with a large receptive field and the DML (Deep Mutual Learning) distillation strategy are proposed; when optimizing the student model, the FPN structure RSE-FPN with a residual attention mechanism is proposed.
|
||||
|
||||
PP-OCRv3 detection training includes two steps:
|
||||
|
||||
- Step 1: Use DML distillation method to train detection teacher model
|
||||
|
||||
- Step 2: Use the teacher model obtained in step 1 to train a lightweight student model using CML method
|
||||
|
||||
### 2.1 Prepare data and operating environment
|
||||
|
||||
The training data uses icdar2015 data. For the steps of preparing the training set, refer to [ocr_dataset](./dataset/ocr_datasets.md).
|
||||
|
||||
For the preparation of the operating environment, refer to [document](./installation.md).
|
||||
|
||||
### 2.2 Train the teacher model
|
||||
|
||||
The configuration file for teacher model training is [PP-OCRv3_det_dml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml). The Backbone, Neck, and Head of the teacher model structure are Resnet50, LKPAN, and DBHead respectively, and are trained using the DML distillation method. For a detailed introduction to the configuration file, refer to [Document](./knowledge_distillation.md).
|
||||
|
||||
Download ImageNet pre-trained model:
|
||||
|
||||
```bash linenums="1"
|
||||
# Download ResNet50_vd pre-trained model
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams
|
||||
```
|
||||
|
||||
**Start training**
|
||||
|
||||
```bash linenums="1"
|
||||
# Single card training
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \
|
||||
Architecture.Models.Student2.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \
|
||||
Global.save_model_dir=./output/
|
||||
# If you want to use multi-GPU distributed training, please use the following command:
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \
|
||||
Architecture.Models.Student2.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
|
||||
The model saved during training is in the output directory, which contains the following files:
|
||||
|
||||
```bash linenums="1"
|
||||
best_accuracy.states
|
||||
best_accuracy.pdparams # The model parameters with the best accuracy are saved by default
|
||||
best_accuracy.pdopt # The optimizer-related parameters with the best accuracy are saved by default
|
||||
latest.states
|
||||
latest.pdparams # The latest model parameters saved by default
|
||||
latest.pdopt # The optimizer-related parameters of the latest model saved by default
|
||||
```
|
||||
|
||||
Among them, best_accuracy is the model parameter with the highest accuracy saved, and the model can be directly used for evaluation.
|
||||
|
||||
The model evaluation command is as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/eval.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml -o Global.checkpoints=./output/best_accuracy
|
||||
```
|
||||
|
||||
The trained teacher model has a larger structure and higher accuracy, which is used to improve the accuracy of the student model.
|
||||
|
||||
**Extract teacher model parameters**
|
||||
best_accuracy contains the parameters of two models, corresponding to Student and Student2 in the configuration file. The method to extract the parameters of Student is as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
import paddle
|
||||
# Load pre-trained model
|
||||
all_params = paddle.load("output/best_accuracy.pdparams")
|
||||
# View the keys of weight parameters
|
||||
print(all_params.keys())
|
||||
# Model weight extraction
|
||||
s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key}
|
||||
# View the keys of model weight parameters
|
||||
print(s_params.keys())
|
||||
# Save
|
||||
paddle.save(s_params, "./pretrain_models/dml_teacher.pdparams")
|
||||
```
|
||||
|
||||
The extracted model parameters can be used for further fine-tuning or distillation training of the model.
|
||||
|
||||
### 2.3 Training the student model
|
||||
|
||||
The configuration file for training the student model is [PP-OCRv3_det_cml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml)
|
||||
The teacher model trained in the previous section is used as supervision, and the CML method is used to train a lightweight student model.
|
||||
|
||||
Download the ImageNet pre-trained model of the student model:
|
||||
|
||||
```bash linenums="1"
|
||||
# Download the pre-trained model of MobileNetV3
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams
|
||||
```
|
||||
|
||||
**Start training**
|
||||
|
||||
```bash linenums="1"
|
||||
# Single card training
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Architecture.Models.Student2.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Architecture.Models.Teacher.pretrained=./pretrain_models/dml_teacher \
|
||||
Global.save_model_dir=./output/
|
||||
# If you want to use multi-GPU distributed training, please use the following command:
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Architecture.Models.Student2.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Architecture.Models.Teacher.pretrained=./pretrain_models/dml_teacher \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
|
||||
The model saved during the training process is in the output directory.
|
||||
The model evaluation command is as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/eval.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml -o Global.checkpoints=./output/best_accuracy
|
||||
```
|
||||
|
||||
best_accuracy contains the parameters of three models, corresponding to Student, Student2, and Teacher in the configuration file. The method to extract Student parameters is as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
import paddle
|
||||
# Load pre-trained model
|
||||
all_params = paddle.load("output/best_accuracy.pdparams")
|
||||
# View the keys of weight parameters
|
||||
print(all_params.keys())
|
||||
# Model weight extraction
|
||||
s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key}
|
||||
# View the keys of model weight parameters
|
||||
print(s_params.keys())
|
||||
# Save
|
||||
paddle.save(s_params, "./pretrain_models/cml_student.pdparams")
|
||||
```
|
||||
|
||||
The extracted Student parameters can be used for model deployment or further fine-tuning training.
|
||||
|
||||
## 3. Fine-tune training based on PP-OCRv3 detection
|
||||
|
||||
This section describes how to use the PP-OCRv3 detection model for fine-tune training in other scenarios.
|
||||
|
||||
Fine-tune training is applicable to three scenarios:
|
||||
|
||||
- Fine-tune training based on the CML distillation method is applicable to scenarios where the teacher model has higher accuracy than the PP-OCRv3 detection model in the usage scenario and a lightweight detection model is desired.
|
||||
|
||||
- Fine-tune training based on the PP-OCRv3 lightweight detection model does not require the training of the teacher model and is intended to improve the accuracy of the usage scenario based on the PP-OCRv3 detection model.
|
||||
|
||||
- Fine-tune training based on the DML distillation method is applicable to scenarios where the DML method is used to further improve accuracy.
|
||||
|
||||
**Finetune training based on CML distillation method**
|
||||
|
||||
Download PP-OCRv3 training model:
|
||||
|
||||
```bash linenums="1"
|
||||
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar
|
||||
tar xf ch_PP-OCRv3_det_distill_train.tar
|
||||
```
|
||||
|
||||
ch_PP-OCRv3_det_distill_train/best_accuracy.pdparams contains the parameters of Student, Student2, and Teacher models in the CML configuration file.
|
||||
|
||||
Start training:
|
||||
|
||||
```bash linenums="1"
|
||||
# Single card training
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml \
|
||||
-o Global.pretrained_model=./ch_PP-OCRv3_det_distill_train/best_accuracy \
|
||||
Global.save_model_dir=./output/
|
||||
# If you want to use multi-GPU distributed training, please use the following command:
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml \
|
||||
-o Global.pretrained_model=./ch_PP-OCRv3_det_distill_train/best_accuracy \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
|
||||
**Finetune training based on PP-OCRv3 lightweight detection model**
|
||||
|
||||
Download PP-OCRv3 training model and extract model parameters of Student structure:
|
||||
|
||||
```
|
||||
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar
|
||||
tar xf ch_PP-OCRv3_det_distill_train.tar
|
||||
```
|
||||
|
||||
The method to extract Student parameters is as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
import paddle
|
||||
# Load pre-trained model
|
||||
all_params = paddle.load("output/best_accuracy.pdparams")
|
||||
# View the keys of weight parameters
|
||||
print(all_params.keys())
|
||||
# Model weight extraction
|
||||
s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key}
|
||||
# View the keys of the model weight parameters
|
||||
print(s_params.keys())
|
||||
# Save
|
||||
paddle.save(s_params, "./student.pdparams")
|
||||
```
|
||||
|
||||
Train using the configuration file [PP-OCRv3_mobile_det.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/PP-OCRv3/PP-OCRv3_mobile_det.yml).
|
||||
|
||||
**Start training**
|
||||
|
||||
```bash linenums="1"
|
||||
# Single card training
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_mobile_det.yml \
|
||||
-o Global.pretrained_model=./student \
|
||||
Global.save_model_dir=./output/
|
||||
# If you want to use multi-GPU distributed training, please use the following command:
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_mobile_det.yml \
|
||||
-o Global.pretrained_model=./student \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
|
||||
**Finetune training based on DML distillation method**
|
||||
|
||||
Take the Teacher model in ch_PP-OCRv3_det_distill_train as an example. First, extract the parameters of the Teacher structure. The method is as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
import paddle
|
||||
# Load pre-trained model
|
||||
all_params = paddle.load("ch_PP-OCRv3_det_distill_train/best_accuracy.pdparams")
|
||||
# View the keys of weight parameters
|
||||
print(all_params.keys())
|
||||
# Model weight extraction
|
||||
s_params = {key[len("Teacher."):]: all_params[key] for key in all_params if "Teacher." in key}
|
||||
# View the keys of model weight parameters
|
||||
print(s_params.keys())
|
||||
# Save
|
||||
paddle.save(s_params, "./teacher.pdparams")
|
||||
```
|
||||
|
||||
**Start training**
|
||||
|
||||
```bash linenums="1"
|
||||
# Single card training
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./teacher \
|
||||
Architecture.Models.Student2.pretrained=./teacher \
|
||||
Global.save_model_dir=./output/
|
||||
# If you want to use multi-GPU distributed training, please use the following command:
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./teacher \
|
||||
Architecture.Models.Student2.pretrained=./teacher \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
251
docs/version2.x/ppocr/model_train/PPOCRv3_det_train.md
Normal file
251
docs/version2.x/ppocr/model_train/PPOCRv3_det_train.md
Normal file
@@ -0,0 +1,251 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-OCRv3 文本检测模型训练
|
||||
|
||||
## 1. 简介
|
||||
|
||||
PP-OCRv3在PP-OCRv2的基础上进一步升级。本节介绍PP-OCRv3检测模型的训练步骤。有关PP-OCRv3策略介绍参考[文档](../blog/PP-OCRv3_introduction.md)。
|
||||
|
||||
## 2. 检测训练
|
||||
|
||||
PP-OCRv3检测模型是对PP-OCRv2中的[CML](https://arxiv.org/pdf/2109.03144.pdf)（Collaborative Mutual Learning) 协同互学习文本检测蒸馏策略进行了升级。PP-OCRv3分别针对检测教师模型和学生模型进行进一步效果优化。其中，在对教师模型优化时，提出了大感受野的PAN结构LK-PAN和引入了DML（Deep Mutual Learning）蒸馏策略；在对学生模型优化时，提出了残差注意力机制的FPN结构RSE-FPN。
|
||||
|
||||
PP-OCRv3检测训练包括两个步骤：
|
||||
|
||||
- 步骤1：采用DML蒸馏方法训练检测教师模型
|
||||
- 步骤2：使用步骤1得到的教师模型采用CML方法训练出轻量学生模型
|
||||
|
||||
### 2.1 准备数据和运行环境
|
||||
|
||||
训练数据采用icdar2015数据，准备训练集步骤参考[ocr_dataset](./dataset/ocr_datasets.md).
|
||||
|
||||
运行环境准备参考[文档](./installation.md)。
|
||||
|
||||
### 2.2 训练教师模型
|
||||
|
||||
教师模型训练的配置文件是[PP-OCRv3_det_dml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml)。教师模型模型结构的Backbone、Neck、Head分别为Resnet50, LKPAN, DBHead，采用DML的蒸馏方法训练。有关配置文件的详细介绍参考[文档](./knowledge_distillation.md)。
|
||||
|
||||
下载ImageNet预训练模型：
|
||||
|
||||
```bash linenums="1"
|
||||
# 下载ResNet50_vd的预训练模型
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams
|
||||
```
|
||||
|
||||
**启动训练**
|
||||
|
||||
```bash linenums="1"
|
||||
# 单卡训练
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \
|
||||
Architecture.Models.Student2.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \
|
||||
Global.save_model_dir=./output/
|
||||
# 如果要使用多GPU分布式训练，请使用如下命令：
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \
|
||||
Architecture.Models.Student2.pretrained=./pretrain_models/ResNet50_vd_ssld_pretrained \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
|
||||
训练过程中保存的模型在output目录下，包含以下文件：
|
||||
|
||||
```bash linenums="1"
|
||||
best_accuracy.states
|
||||
best_accuracy.pdparams # 默认保存最优精度的模型参数
|
||||
best_accuracy.pdopt # 默认保存最优精度的优化器相关参数
|
||||
latest.states
|
||||
latest.pdparams # 默认保存的最新模型参数
|
||||
latest.pdopt # 默认保存的最新模型的优化器相关参数
|
||||
```
|
||||
|
||||
其中，best_accuracy是保存的精度最高的模型参数，可以直接使用该模型评估。
|
||||
|
||||
模型评估命令如下：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/eval.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml -o Global.checkpoints=./output/best_accuracy
|
||||
```
|
||||
|
||||
训练的教师模型结构更大，精度更高，用于提升学生模型的精度。
|
||||
|
||||
**提取教师模型参数**
|
||||
best_accuracy包含两个模型的参数，分别对应配置文件中的Student，Student2。提取Student的参数方法如下：
|
||||
|
||||
```bash linenums="1"
|
||||
import paddle
|
||||
# 加载预训练模型
|
||||
all_params = paddle.load("output/best_accuracy.pdparams")
|
||||
# 查看权重参数的keys
|
||||
print(all_params.keys())
|
||||
# 模型的权重提取
|
||||
s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key}
|
||||
# 查看模型权重参数的keys
|
||||
print(s_params.keys())
|
||||
# 保存
|
||||
paddle.save(s_params, "./pretrain_models/dml_teacher.pdparams")
|
||||
```
|
||||
|
||||
提取出来的模型参数可以用于模型进一步的finetune训练或者蒸馏训练。
|
||||
|
||||
### 2.3 训练学生模型
|
||||
|
||||
训练学生模型的配置文件是[PP-OCRv3_det_cml.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml)
|
||||
上一节训练得到的教师模型作为监督，采用CML方式训练得到轻量的学生模型。
|
||||
|
||||
下载学生模型的ImageNet预训练模型：
|
||||
|
||||
```bash linenums="1"
|
||||
# 下载MobileNetV3的预训练模型
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams
|
||||
```
|
||||
|
||||
**启动训练**
|
||||
|
||||
```bash linenums="1"
|
||||
# 单卡训练
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Architecture.Models.Student2.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Architecture.Models.Teacher.pretrained=./pretrain_models/dml_teacher \
|
||||
Global.save_model_dir=./output/
|
||||
# 如果要使用多GPU分布式训练，请使用如下命令：
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Architecture.Models.Student2.pretrained=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Architecture.Models.Teacher.pretrained=./pretrain_models/dml_teacher \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
|
||||
训练过程中保存的模型在output目录下，
|
||||
模型评估命令如下：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/eval.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml -o Global.checkpoints=./output/best_accuracy
|
||||
```
|
||||
|
||||
best_accuracy包含三个模型的参数，分别对应配置文件中的Student，Student2，Teacher。提取Student参数的方法如下：
|
||||
|
||||
```bash linenums="1"
|
||||
import paddle
|
||||
# 加载预训练模型
|
||||
all_params = paddle.load("output/best_accuracy.pdparams")
|
||||
# 查看权重参数的keys
|
||||
print(all_params.keys())
|
||||
# 模型的权重提取
|
||||
s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key}
|
||||
# 查看模型权重参数的keys
|
||||
print(s_params.keys())
|
||||
# 保存
|
||||
paddle.save(s_params, "./pretrain_models/cml_student.pdparams")
|
||||
```
|
||||
|
||||
提取出来的Student的参数可用于模型部署或者做进一步的finetune训练。
|
||||
|
||||
## 3. 基于PP-OCRv3检测finetune训练
|
||||
|
||||
本节介绍如何使用PP-OCRv3检测模型在其他场景上的finetune训练。
|
||||
|
||||
finetune训练适用于三种场景：
|
||||
|
||||
- 基于CML蒸馏方法的finetune训练，适用于教师模型在使用场景上精度高于PP-OCRv3检测模型，且希望得到一个轻量检测模型。
|
||||
- 基于PP-OCRv3轻量检测模型的finetune训练，无需训练教师模型，希望在PP-OCRv3检测模型基础上提升使用场景上的精度。
|
||||
- 基于DML蒸馏方法的finetune训练，适用于采用DML方法进一步提升精度的场景。
|
||||
|
||||
**基于CML蒸馏方法的finetune训练**
|
||||
|
||||
下载PP-OCRv3训练模型：
|
||||
|
||||
```bash linenums="1"
|
||||
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar
|
||||
tar xf ch_PP-OCRv3_det_distill_train.tar
|
||||
```
|
||||
|
||||
ch_PP-OCRv3_det_distill_train/best_accuracy.pdparams包含CML配置文件中Student、Student2、Teacher模型的参数。
|
||||
|
||||
启动训练：
|
||||
|
||||
```bash linenums="1"
|
||||
# 单卡训练
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml \
|
||||
-o Global.pretrained_model=./ch_PP-OCRv3_det_distill_train/best_accuracy \
|
||||
Global.save_model_dir=./output/
|
||||
# 如果要使用多GPU分布式训练，请使用如下命令：
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_cml.yml \
|
||||
-o Global.pretrained_model=./ch_PP-OCRv3_det_distill_train/best_accuracy \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
|
||||
**基于PP-OCRv3轻量检测模型的finetune训练**
|
||||
|
||||
下载PP-OCRv3训练模型，并提取Student结构的模型参数：
|
||||
|
||||
```
|
||||
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar
|
||||
tar xf ch_PP-OCRv3_det_distill_train.tar
|
||||
```
|
||||
|
||||
提取Student参数的方法如下：
|
||||
|
||||
```bash linenums="1"
|
||||
import paddle
|
||||
# 加载预训练模型
|
||||
all_params = paddle.load("output/best_accuracy.pdparams")
|
||||
# 查看权重参数的keys
|
||||
print(all_params.keys())
|
||||
# 模型的权重提取
|
||||
s_params = {key[len("Student."):]: all_params[key] for key in all_params if "Student." in key}
|
||||
# 查看模型权重参数的keys
|
||||
print(s_params.keys())
|
||||
# 保存
|
||||
paddle.save(s_params, "./student.pdparams")
|
||||
```
|
||||
|
||||
使用配置文件[PP-OCRv3_mobile_det.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release%2F2.5/configs/det/PP-OCRv3/PP-OCRv3_mobile_det.yml)训练。
|
||||
|
||||
**启动训练**
|
||||
|
||||
```bash linenums="1"
|
||||
# 单卡训练
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_mobile_det.yml \
|
||||
-o Global.pretrained_model=./student \
|
||||
Global.save_model_dir=./output/
|
||||
# 如果要使用多GPU分布式训练，请使用如下命令：
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_mobile_det.yml \
|
||||
-o Global.pretrained_model=./student \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
|
||||
**基于DML蒸馏方法的finetune训练**
|
||||
|
||||
以ch_PP-OCRv3_det_distill_train中的Teacher模型为例，首先提取Teacher结构的参数，方法如下：
|
||||
|
||||
```bash linenums="1"
|
||||
import paddle
|
||||
# 加载预训练模型
|
||||
all_params = paddle.load("ch_PP-OCRv3_det_distill_train/best_accuracy.pdparams")
|
||||
# 查看权重参数的keys
|
||||
print(all_params.keys())
|
||||
# 模型的权重提取
|
||||
s_params = {key[len("Teacher."):]: all_params[key] for key in all_params if "Teacher." in key}
|
||||
# 查看模型权重参数的keys
|
||||
print(s_params.keys())
|
||||
# 保存
|
||||
paddle.save(s_params, "./teacher.pdparams")
|
||||
```
|
||||
|
||||
**启动训练**
|
||||
|
||||
```bash linenums="1"
|
||||
# 单卡训练
|
||||
python3 tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./teacher \
|
||||
Architecture.Models.Student2.pretrained=./teacher \
|
||||
Global.save_model_dir=./output/
|
||||
# 如果要使用多GPU分布式训练，请使用如下命令：
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/PP-OCRv3/PP-OCRv3_det_dml.yml \
|
||||
-o Architecture.Models.Student.pretrained=./teacher \
|
||||
Architecture.Models.Student2.pretrained=./teacher \
|
||||
Global.save_model_dir=./output/
|
||||
```
|
||||
149
docs/version2.x/ppocr/model_train/angle_class.en.md
Normal file
149
docs/version2.x/ppocr/model_train/angle_class.en.md
Normal file
@@ -0,0 +1,149 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Text Direction Classification
|
||||
|
||||
## 1. Method Introduction
|
||||
|
||||
The angle classification is used in the scene where the image is not 0 degrees. In this scene, it is necessary to perform a correction operation on the text line detected in the picture. In the PaddleOCR system,
|
||||
The text line image obtained after text detection is sent to the recognition model after affine transformation. At this time, only a 0 and 180 degree angle classification of the text is required, so the built-in PaddleOCR text angle classifier **only supports 0 and 180 degree classification**. If you want to support more angles, you can modify the algorithm yourself to support.
|
||||
|
||||
Example of 0 and 180 degree data samples：
|
||||
|
||||
|
||||
|
||||
## 2. Data Preparation
|
||||
|
||||
Please organize the dataset as follows:
|
||||
|
||||
The default storage path for training data is `PaddleOCR/train_data/cls`, if you already have a dataset on your disk, just create a soft link to the dataset directory:
|
||||
|
||||
```bash linenums="1"
|
||||
ln -sf <path/to/dataset> <path/to/paddle_ocr>/train_data/cls/dataset
|
||||
```
|
||||
|
||||
please refer to the following to organize your data.
|
||||
|
||||
### Training set
|
||||
|
||||
First put the training images in the same folder (train_images), and use a txt file (cls_gt_train.txt) to store the image path and label.
|
||||
|
||||
- Note: by default, the image path and image label are split with `\t`, if you use other methods to split, it will cause training error
|
||||
|
||||
0 and 180 indicate that the angle of the image is 0 degrees and 180 degrees, respectively.
|
||||
|
||||
```text linenums="1"
|
||||
" Image file name Image annotation "
|
||||
|
||||
train/word_001.jpg 0
|
||||
train/word_002.jpg 180
|
||||
```
|
||||
|
||||
The final training set should have the following file structure:
|
||||
|
||||
```text linenums="1"
|
||||
|-train_data
|
||||
|-cls
|
||||
|- cls_gt_train.txt
|
||||
|- train
|
||||
|- word_001.png
|
||||
|- word_002.jpg
|
||||
|- word_003.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
### Test set
|
||||
|
||||
Similar to the training set, the test set also needs to be provided a folder
|
||||
containing all images (test) and a cls_gt_test.txt. The structure of the test set is as follows:
|
||||
|
||||
```text linenums="1"
|
||||
|-train_data
|
||||
|-cls
|
||||
|- cls_gt_test.txt
|
||||
|- test
|
||||
|- word_001.jpg
|
||||
|- word_002.jpg
|
||||
|- word_003.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
## 3. Training
|
||||
|
||||
Write the prepared txt file and image folder path into the configuration file under the `Train/Eval.dataset.label_file_list` and `Train/Eval.dataset.data_dir` fields, the absolute path of the image consists of the `Train/Eval.dataset.data_dir` field and the image name recorded in the txt file.
|
||||
|
||||
PaddleOCR provides training scripts, evaluation scripts, and prediction scripts.
|
||||
|
||||
### Start training
|
||||
|
||||
```bash linenums="1"
|
||||
# Set PYTHONPATH path
|
||||
export PYTHONPATH=$PYTHONPATH:.
|
||||
# GPU training Support single card and multi-card training, specify the card number through --gpus.
|
||||
# Start training, the following command has been written into the train.sh file, just modify the configuration file path in the file
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3,4,5,6,7' tools/train.py -c configs/cls/cls_mv3.yml
|
||||
```
|
||||
|
||||
### Data Augmentation
|
||||
|
||||
PaddleOCR provides a variety of data augmentation methods. If you want to add disturbance during training, Please uncomment the `RecAug` and `RandAugment` fields under `Train.dataset.transforms` in the configuration file.
|
||||
|
||||
The default perturbation methods are: cvtColor, blur, jitter, Gauss noise, random crop, perspective, color reverse, RandAugment.
|
||||
|
||||
Except for RandAugment, each disturbance method is selected with a 50% probability during the training process. For specific code implementation, please refer to:
|
||||
[rec_img_aug.py](../../ppocr/data/imaug/rec_img_aug.py)
|
||||
[randaugment.py](../../ppocr/data/imaug/randaugment.py)
|
||||
|
||||
### Training
|
||||
|
||||
PaddleOCR supports alternating training and evaluation. You can modify `eval_batch_step` in `configs/cls/cls_mv3.yml` to set the evaluation frequency. By default, it is evaluated every 1000 iter. The following content will be saved during training:
|
||||
|
||||
```bash linenums="1"
|
||||
├── best_accuracy.pdopt # Optimizer parameters for the best model
|
||||
├── best_accuracy.pdparams # Parameters of the best model
|
||||
├── best_accuracy.states # Metric info and epochs of the best model
|
||||
├── config.yml # Configuration file for this experiment
|
||||
├── latest.pdopt # Optimizer parameters for the latest model
|
||||
├── latest.pdparams # Parameters of the latest model
|
||||
├── latest.states # Metric info and epochs of the latest model
|
||||
└── train.log # Training log
|
||||
```
|
||||
|
||||
If the evaluation set is large, the test will be time-consuming. It is recommended to reduce the number of evaluations, or evaluate after training.
|
||||
|
||||
**Note that the configuration file for prediction/evaluation must be consistent with the training.**
|
||||
|
||||
## 4. Evaluation
|
||||
|
||||
The evaluation dataset can be set by modifying the `Eval.dataset.label_file_list` field in the `configs/cls/cls_mv3.yml` file.
|
||||
|
||||
```bash linenums="1"
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
# GPU evaluation, Global.checkpoints is the weight to be tested
|
||||
python3 tools/eval.py -c configs/cls/cls_mv3.yml -o Global.checkpoints={path/to/weights}/best_accuracy
|
||||
```
|
||||
|
||||
## 5. Prediction
|
||||
|
||||
### Training engine prediction
|
||||
|
||||
Using the model trained by paddleocr, you can quickly get prediction through the following script.
|
||||
|
||||
Use `Global.infer_img` to specify the path of the predicted picture or folder, and use `Global.checkpoints` to specify the weight:
|
||||
|
||||
```bash linenums="1"
|
||||
# Predict English results
|
||||
python3 tools/infer_cls.py -c configs/cls/cls_mv3.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.load_static_weights=false Global.infer_img=doc/imgs_words_en/word_10.png
|
||||
```
|
||||
|
||||
Input image:
|
||||
|
||||
|
||||
|
||||
Get the prediction result of the input image:
|
||||
|
||||
```bash linenums="1"
|
||||
infer_img: doc/imgs_words_en/word_10.png
|
||||
result: ('0', 0.9999995)
|
||||
```
|
||||
151
docs/version2.x/ppocr/model_train/angle_class.md
Normal file
151
docs/version2.x/ppocr/model_train/angle_class.md
Normal file
@@ -0,0 +1,151 @@
|
||||
---
|
||||
comments: true
|
||||
typora-copy-images-to: images
|
||||
---
|
||||
|
||||
# 文本方向分类器
|
||||
|
||||
## 1. 方法介绍
|
||||
|
||||
文本方向分类器主要用于图片非0度的场景下，在这种场景下需要对图片里检测到的文本行进行一个转正的操作。在PaddleOCR系统内，
|
||||
文字检测之后得到的文本行图片经过仿射变换之后送入识别模型，此时只需要对文字进行一个0和180度的角度分类，因此PaddleOCR内置的
|
||||
文本方向分类器**只支持了0和180度的分类**。如果想支持更多角度，可以自己修改算法进行支持。
|
||||
|
||||
0和180度数据样本例子：
|
||||
|
||||
|
||||
|
||||
## 2. 数据准备
|
||||
|
||||
请按如下步骤设置数据集：
|
||||
|
||||
训练数据的默认存储路径是 `PaddleOCR/train_data/cls`,如果您的磁盘上已有数据集，只需创建软链接至数据集目录：
|
||||
|
||||
```bash linenums="1"
|
||||
ln -sf <path/to/dataset> <path/to/paddle_ocr>/train_data/cls/dataset
|
||||
```
|
||||
|
||||
请参考下文组织您的数据。
|
||||
|
||||
### 训练集
|
||||
|
||||
首先建议将训练图片放入同一个文件夹，并用一个txt文件（cls_gt_train.txt）记录图片路径和标签。
|
||||
|
||||
**注意：** 默认请将图片路径和图片标签用 `\t` 分割，如用其他方式分割将造成训练报错
|
||||
|
||||
0和180分别表示图片的角度为0度和180度
|
||||
|
||||
```text linenums="1"
|
||||
" 图像文件名 图像标注信息 "
|
||||
train/cls/train/word_001.jpg 0
|
||||
train/cls/train/word_002.jpg 180
|
||||
```
|
||||
|
||||
最终训练集应有如下文件结构：
|
||||
|
||||
```text linenums="1"
|
||||
|-train_data
|
||||
|-cls
|
||||
|- cls_gt_train.txt
|
||||
|- train
|
||||
|- word_001.png
|
||||
|- word_002.jpg
|
||||
|- word_003.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
### 测试集
|
||||
|
||||
同训练集类似，测试集也需要提供一个包含所有图片的文件夹（test）和一个cls_gt_test.txt，测试集的结构如下所示：
|
||||
|
||||
```text linenums="1"
|
||||
|-train_data
|
||||
|-cls
|
||||
|- cls_gt_test.txt
|
||||
|- test
|
||||
|- word_001.jpg
|
||||
|- word_002.jpg
|
||||
|- word_003.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
## 3. 启动训练
|
||||
|
||||
将准备好的txt文件和图片文件夹路径分别写入配置文件的 `Train/Eval.dataset.label_file_list` 和 `Train/Eval.dataset.data_dir` 字段下，`Train/Eval.dataset.data_dir`字段下的路径和文件里记载的图片名构成了图片的绝对路径。
|
||||
|
||||
PaddleOCR提供了训练脚本、评估脚本和预测脚本。
|
||||
|
||||
### 开始训练
|
||||
|
||||
*如果您安装的是cpu版本，请将配置文件中的 `use_gpu` 字段修改为false*
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU训练 支持单卡，多卡训练，通过 '--gpus' 指定卡号。
|
||||
# 启动训练，下面的命令已经写入train.sh文件中，只需修改文件里的配置文件路径即可
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3,4,5,6,7' tools/train.py -c configs/cls/cls_mv3.yml
|
||||
```
|
||||
|
||||
### 数据增强
|
||||
|
||||
PaddleOCR提供了多种数据增强方式，如果您希望在训练时加入扰动，请在配置文件中取消`Train.dataset.transforms`下的`RecAug`和`RandAugment`字段的注释。
|
||||
|
||||
默认的扰动方式有：颜色空间转换(cvtColor)、模糊(blur)、抖动(jitter)、噪声(Gasuss noise)、随机切割(random crop)、透视(perspective)、颜色反转(reverse),随机数据增强(RandAugment)。
|
||||
|
||||
训练过程中除随机数据增强外每种扰动方式以50%的概率被选择，具体代码实现请参考：
|
||||
[rec_img_aug.py](../../ppocr/data/imaug/rec_img_aug.py)
|
||||
[randaugment.py](../../ppocr/data/imaug/randaugment.py)
|
||||
|
||||
*由于OpenCV的兼容性问题，扰动操作暂时只支持linux*
|
||||
|
||||
## 4. 训练
|
||||
|
||||
PaddleOCR支持训练和评估交替进行, 可以在 `configs/cls/cls_mv3.yml` 中修改 `eval_batch_step` 设置评估频率，默认每1000个iter评估一次。训练过程中将会保存如下内容：
|
||||
|
||||
```bash linenums="1"
|
||||
├── best_accuracy.pdopt # 最佳模型的优化器参数
|
||||
├── best_accuracy.pdparams # 最佳模型的参数
|
||||
├── best_accuracy.states # 最佳模型的指标和epoch等信息
|
||||
├── config.yml # 本次实验的配置文件
|
||||
├── latest.pdopt # 最新模型的优化器参数
|
||||
├── latest.pdparams # 最新模型的参数
|
||||
├── latest.states # 最新模型的指标和epoch等信息
|
||||
└── train.log # 训练日志
|
||||
```
|
||||
|
||||
如果验证集很大，测试将会比较耗时，建议减少评估次数，或训练完再进行评估。
|
||||
|
||||
**注意，预测/评估时的配置文件请务必与训练一致。**
|
||||
|
||||
## 5. 评估
|
||||
|
||||
评估数据集可以通过修改`configs/cls/cls_mv3.yml`文件里的`Eval.dataset.label_file_list` 字段设置。
|
||||
|
||||
```bash linenums="1"
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
# GPU 评估， Global.checkpoints 为待测权重
|
||||
python3 tools/eval.py -c configs/cls/cls_mv3.yml -o Global.checkpoints={path/to/weights}/best_accuracy
|
||||
```
|
||||
|
||||
## 6. 预测
|
||||
|
||||
### 训练引擎的预测
|
||||
|
||||
使用 PaddleOCR 训练好的模型，可以通过以下脚本进行快速预测。
|
||||
|
||||
通过 `Global.infer_img` 指定预测图片或文件夹路径，通过 `Global.checkpoints` 指定权重：
|
||||
|
||||
```bash linenums="1"
|
||||
# 预测分类结果
|
||||
python3 tools/infer_cls.py -c configs/cls/cls_mv3.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.load_static_weights=false Global.infer_img=doc/imgs_words/ch/word_1.jpg
|
||||
```
|
||||
|
||||
预测图片：
|
||||
|
||||
|
||||
|
||||
得到输入图像的预测结果：
|
||||
|
||||
```bash linenums="1"
|
||||
infer_img: doc/imgs_words/ch/word_1.jpg
|
||||
result: ('0', 0.9998784)
|
||||
```
|
||||
248
docs/version2.x/ppocr/model_train/detection.en.md
Normal file
248
docs/version2.x/ppocr/model_train/detection.en.md
Normal file
@@ -0,0 +1,248 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Text Detection
|
||||
|
||||
This section uses the icdar2015 dataset as an example to introduce the training, evaluation, and testing of the detection model in PaddleOCR.
|
||||
|
||||
## 1. Data and Weights Preparation
|
||||
|
||||
### 1.1 Data Preparation
|
||||
|
||||
To prepare datasets, refer to [ocr_datasets](../../datasets/ocr_datasets.en.md).
|
||||
|
||||
### 1.2 Download Pre-trained Model
|
||||
|
||||
First download the pre-trained model. The detection model of PaddleOCR currently supports 3 backbones, namely MobileNetV3, ResNet18_vd and ResNet50_vd. You can use the model in [PaddleClas](https://github.com/PaddlePaddle/PaddleClas/tree/release/2.0/ppcls/modeling/architectures) to replace backbone according to your needs.
|
||||
And the responding download link of backbone pre-trained weights can be found in (<https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.0/README_cn.md#resnet%E5%8F%8A%E5%85%B6vd%E7%B3%BB%E5%88%97>).
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/
|
||||
# Download the pre-trained model of MobileNetV3
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams
|
||||
# or, download the pre-trained model of ResNet18_vd
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet18_vd_pretrained.pdparams
|
||||
# or, download the pre-trained model of ResNet50_vd
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams
|
||||
|
||||
```
|
||||
|
||||
## 2. Training
|
||||
|
||||
### 2.1 Start Training
|
||||
|
||||
*If CPU version installed, please set the parameter `use_gpu` to `false` in the configuration.*
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained
|
||||
```
|
||||
|
||||
In the above instruction, use `-c` to select the training to use the `configs/det/det_mv3_db.yml` configuration file.
|
||||
For a detailed explanation of the configuration file, please refer to [config](../blog/config.en.md).
|
||||
|
||||
You can also use `-o` to change the training parameters without modifying the yml file. For example, adjust the training learning rate to 0.0001.
|
||||
|
||||
```bash linenums="1"
|
||||
# single GPU training
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml -o \
|
||||
Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Optimizer.base_lr=0.0001
|
||||
|
||||
# multi-GPU training
|
||||
# Set the GPU ID used by the '--gpus' parameter.
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml -o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained
|
||||
|
||||
# multi-Node, multi-GPU training
|
||||
# Set the IPs of your nodes used by the '--ips' parameter. Set the GPU ID used by the '--gpus' parameter.
|
||||
python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained
|
||||
```
|
||||
|
||||
**Note:** For multi-Node multi-GPU training, you need to replace the `ips` value in the preceding command with the address of your machine, and the machines must be able to ping each other. In addition, it requires activating commands separately on multiple machines when we start the training. The command for viewing the IP address of the machine is `ifconfig`.
|
||||
|
||||
If you want to further speed up the training, you can use [automatic mixed precision training](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_en.html). for single card training, the command is as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True
|
||||
```
|
||||
|
||||
### 2.2 Load Trained Model and Continue Training
|
||||
|
||||
If you expect to load trained model and continue the training again, you can specify the parameter `Global.checkpoints` as the model path to be loaded.
|
||||
|
||||
For example:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml -o Global.checkpoints=./your/trained/model
|
||||
```
|
||||
|
||||
**Note**: The priority of `Global.checkpoints` is higher than that of `Global.pretrained_model`, that is, when two parameters are specified at the same time, the model specified by `Global.checkpoints` will be loaded first. If the model path specified by `Global.checkpoints` is wrong, the one specified by `Global.pretrained_model` will be loaded.
|
||||
|
||||
### 2.3 Training with New Backbone
|
||||
|
||||
The network part completes the construction of the network, and PaddleOCR divides the network into four parts, which are under [ppocr/modeling](../../ppocr/modeling). The data entering the network will pass through these four parts in sequence(transforms->backbones->
|
||||
necks->heads).
|
||||
|
||||
```bash linenums="1"
|
||||
├── architectures # Code for building network
|
||||
├── transforms # Image Transformation Module
|
||||
├── backbones # Feature extraction module
|
||||
├── necks # Feature enhancement module
|
||||
└── heads # Output module
|
||||
```
|
||||
|
||||
If the Backbone to be replaced has a corresponding implementation in PaddleOCR, you can directly modify the parameters in the `Backbone` part of the configuration yml file.
|
||||
|
||||
However, if you want to use a new Backbone, an example of replacing the backbones is as follows:
|
||||
|
||||
1. Create a new file under the [ppocr/modeling/backbones](../../ppocr/modeling/backbones) folder, such as my_backbone.py.
|
||||
2. Add code in the my_backbone.py file, the sample code is as follows:
|
||||
|
||||
```python linenums="1"
|
||||
import paddle
|
||||
import paddle.nn as nn
|
||||
import paddle.nn.functional as F
|
||||
|
||||
|
||||
class MyBackbone(nn.Layer):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(MyBackbone, self).__init__()
|
||||
# your init code
|
||||
self.conv = nn.xxxx
|
||||
|
||||
def forward(self, inputs):
|
||||
# your network forward
|
||||
y = self.conv(inputs)
|
||||
return y
|
||||
```
|
||||
|
||||
3. Import the added module in the [ppocr/modeling/backbones/\_*init\_*.py](../../ppocr/modeling/backbones/__init__.py) file.
|
||||
|
||||
After adding the four-part modules of the network, you only need to configure them in the configuration file to use, such as:
|
||||
|
||||
```yaml linenums="1"
|
||||
Backbone:
|
||||
name: MyBackbone
|
||||
args1: args1
|
||||
```
|
||||
|
||||
**NOTE**: More details about replace Backbone and other module can be found in [doc](../../algorithm/add_new_algorithm.en.md).
|
||||
|
||||
### 2.4 Mixed Precision Training
|
||||
|
||||
If you want to speed up your training further, you can use [Auto Mixed Precision Training](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_cn.html), taking a single machine and a single gpu as an example, the commands are as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True
|
||||
```
|
||||
|
||||
### 2.5 Distributed Training
|
||||
|
||||
During multi-machine multi-gpu training, use the `--ips` parameter to set the used machine IP address, and the `--gpus` parameter to set the used GPU ID:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained
|
||||
```
|
||||
|
||||
**Note:** (1) When using multi-machine and multi-gpu training, you need to replace the ips value in the above command with the address of your machine, and the machines need to be able to ping each other. (2) Training needs to be launched separately on multiple machines. The command to view the ip address of the machine is `ifconfig`. (3) For more details about the distributed training speedup ratio, please refer to [Distributed Training Tutorial](../blog/distributed_training.en.md).
|
||||
|
||||
### 2.6 Training with knowledge distillation
|
||||
|
||||
Knowledge distillation is supported in PaddleOCR for text detection training process. For more details, please refer to [doc](./../model_compress/knowledge_distillation.en.md).
|
||||
|
||||
### 2.7 Training on other platform(Windows/macOS/Linux DCU)
|
||||
|
||||
- Windows GPU/CPU
|
||||
The Windows platform is slightly different from the Linux platform:
|
||||
Windows platform only supports `single gpu` training and inference, specify GPU for training `set CUDA_VISIBLE_DEVICES=0`
|
||||
On the Windows platform, DataLoader only supports single-process mode, so you need to set `num_workers` to 0;
|
||||
|
||||
- macOS
|
||||
GPU mode is not supported, you need to set `use_gpu` to False in the configuration file, and the rest of the training evaluation prediction commands are exactly the same as Linux GPU.
|
||||
|
||||
- Linux DCU
|
||||
Running on a DCU device requires setting the environment variable `export HIP_VISIBLE_DEVICES=0,1,2,3`, and the rest of the training and evaluation prediction commands are exactly the same as the Linux GPU.
|
||||
|
||||
### 2.8 Fine-tuning
|
||||
|
||||
In actual use, it is recommended to load the official pre-trained model and fine-tune it in your own data set. For the fine-tuning method of the detection model, please refer to: [Model Fine-tuning Tutorial](./finetune_en.md).
|
||||
|
||||
## 3. Evaluation and Test
|
||||
|
||||
### 3.1 Evaluation
|
||||
|
||||
PaddleOCR calculates three indicators for evaluating performance of OCR detection task: Precision, Recall, and Hmean(F-Score).
|
||||
|
||||
Run the following code to calculate the evaluation indicators. The result will be saved in the test result file specified by `save_res_path` in the configuration file `det_db_mv3.yml`
|
||||
|
||||
When evaluating, set post-processing parameters `box_thresh=0.6`, `unclip_ratio=1.5`. If you use different datasets, different models for training, these two parameters should be adjusted for better result.
|
||||
|
||||
The model parameters during training are saved in the `Global.save_model_dir` directory by default. When evaluating indicators, you need to set `Global.checkpoints` to point to the saved parameter file.
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/eval.py -c configs/det/det_mv3_db.yml -o Global.checkpoints="{path/to/weights}/best_accuracy" PostProcess.box_thresh=0.6 PostProcess.unclip_ratio=1.5
|
||||
```
|
||||
|
||||
- Note: `box_thresh` and `unclip_ratio` are parameters required for DB post-processing, and not need to be set when evaluating the EAST and SAST model.
|
||||
|
||||
### 3.2 Test
|
||||
|
||||
Test the detection result on a single image:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/det_db/best_accuracy"
|
||||
```
|
||||
|
||||
When testing the DB model, adjust the post-processing threshold:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/det_db/best_accuracy" PostProcess.box_thresh=0.6 PostProcess.unclip_ratio=2.0
|
||||
```
|
||||
|
||||
Test the detection result on all images in the folder:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/" Global.pretrained_model="./output/det_db/best_accuracy"
|
||||
```
|
||||
|
||||
## 4. Inference
|
||||
|
||||
The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment.
|
||||
|
||||
The model saved during the training process is the checkpoints model, which saves the parameters of the model and is mostly used to resume training.
|
||||
|
||||
Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems.
|
||||
|
||||
Firstly, we can convert DB trained model to inference model:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/export_model.py -c configs/det/det_mv3_db.yml -o Global.pretrained_model="./output/det_db/best_accuracy" Global.save_inference_dir="./output/det_db_inference/"
|
||||
```
|
||||
|
||||
The detection inference model prediction：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer/predict_det.py --det_algorithm="DB" --det_model_dir="./output/det_db_inference/" --image_dir="./doc/imgs/" --use_gpu=True
|
||||
```
|
||||
|
||||
If it is other detection algorithms, such as the EAST, the det_algorithm parameter needs to be modified to EAST, and the default is the DB algorithm:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer/predict_det.py --det_algorithm="EAST" --det_model_dir="./output/det_db_inference/" --image_dir="./doc/imgs/" --use_gpu=True
|
||||
```
|
||||
|
||||
## 5. FAQ
|
||||
|
||||
Q1: The prediction results of trained model and inference model are inconsistent?
|
||||
|
||||
**A**: Most of the problems are caused by the inconsistency of the pre-processing and post-processing parameters during the prediction of the trained model and the pre-processing and post-processing parameters during the prediction of the inference model. Taking the model trained by the det_mv3_db.yml configuration file as an example, the solution to the problem of inconsistent prediction results between the training model and the inference model is as follows:
|
||||
|
||||
- Check whether the [trained model preprocessing](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/configs/det/det_mv3_db.yml#L116) is consistent with the prediction [preprocessing function of the inference model](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/tools/infer/predict_det.py#L42). When the algorithm is evaluated, the input image size will affect the accuracy. In order to be consistent with the paper, the image is resized to [736, 1280] in the training icdar15 configuration file, but there is only a set of default parameters when the inference model predicts, which will be considered. To predict the speed problem, the longest side of the image is limited to 960 for resize by default. The preprocessing function of the training model preprocessing and the inference model is located in [ppocr/data/imaug/operators.py](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/ppocr/data/imaug/operators.py#L147).
|
||||
- Check whether the [post-processing of the trained model](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/configs/det/det_mv3_db.yml#L51) is consistent with the [post-processing parameters of the inference](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/tools/infer/utility.py#L50).
|
||||
236
docs/version2.x/ppocr/model_train/detection.md
Normal file
236
docs/version2.x/ppocr/model_train/detection.md
Normal file
@@ -0,0 +1,236 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
|
||||
# 文字检测
|
||||
|
||||
本节以icdar2015数据集为例，介绍PaddleOCR中检测模型训练、评估、测试的使用方式。
|
||||
|
||||
## 1. 准备数据和模型
|
||||
|
||||
### 1.1 准备数据集
|
||||
|
||||
准备数据集可参考 [ocr_datasets](../../datasets/ocr_datasets.md) 。
|
||||
|
||||
### 1.2 下载预训练模型
|
||||
|
||||
首先下载模型backbone的pretrain model，PaddleOCR的检测模型目前支持两种backbone，分别是MobileNetV3、ResNet_vd系列，
|
||||
您可以根据需求使用[PaddleClas](https://github.com/PaddlePaddle/PaddleClas/tree/release/2.0/ppcls/modeling/architectures)中的模型更换backbone，
|
||||
对应的backbone预训练模型可以从[PaddleClas repo 主页中找到下载链接](https://github.com/PaddlePaddle/PaddleClas/blob/release%2F2.0/README_cn.md#resnet%E5%8F%8A%E5%85%B6vd%E7%B3%BB%E5%88%97)。
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/
|
||||
# 根据backbone的不同选择下载对应的预训练模型
|
||||
# 下载MobileNetV3的预训练模型
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/MobileNetV3_large_x0_5_pretrained.pdparams
|
||||
# 或，下载ResNet18_vd的预训练模型
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet18_vd_pretrained.pdparams
|
||||
# 或，下载ResNet50_vd的预训练模型
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/pretrained/ResNet50_vd_ssld_pretrained.pdparams
|
||||
```
|
||||
|
||||
## 2. 开始训练
|
||||
|
||||
### 2.1 启动训练
|
||||
|
||||
*如果您安装的是cpu版本，请将配置文件中的 `use_gpu` 字段修改为false*
|
||||
|
||||
```bash linenums="1"
|
||||
# 单机单卡训练 mv3_db 模型
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained
|
||||
|
||||
# 单机多卡训练，通过 --gpus 参数设置使用的GPU ID
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained
|
||||
|
||||
```
|
||||
|
||||
上述指令中，通过-c 选择训练使用configs/det/det_mv3_db.yml配置文件。
|
||||
有关配置文件的详细解释，请参考[链接](../blog/config.md)。
|
||||
|
||||
您也可以通过-o参数在不需要修改yml文件的情况下，改变训练的参数，比如，调整训练的学习率为0.0001
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml -o Optimizer.base_lr=0.0001
|
||||
```
|
||||
|
||||
### 2.2 断点训练
|
||||
|
||||
如果训练程序中断，如果希望加载训练中断的模型从而恢复训练，可以通过指定Global.checkpoints指定要加载的模型路径：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml -o Global.checkpoints=./your/trained/model
|
||||
```
|
||||
|
||||
**注意**：`Global.checkpoints`的优先级高于`Global.pretrained_model`的优先级，即同时指定两个参数时，优先加载`Global.checkpoints`指定的模型，如果`Global.checkpoints`指定的模型路径有误，会加载`Global.pretrained_model`指定的模型。
|
||||
|
||||
### 2.3 更换Backbone 训练
|
||||
|
||||
PaddleOCR将网络划分为四部分，分别在[ppocr/modeling](../../ppocr/modeling)下。 进入网络的数据将按照顺序(transforms->backbones->
|
||||
necks->heads)依次通过这四个部分。
|
||||
|
||||
```bash linenums="1"
|
||||
├── architectures # 网络的组网代码
|
||||
├── transforms # 网络的图像变换模块
|
||||
├── backbones # 网络的特征提取模块
|
||||
├── necks # 网络的特征增强模块
|
||||
└── heads # 网络的输出模块
|
||||
```
|
||||
|
||||
如果要更换的Backbone 在PaddleOCR中有对应实现，直接修改配置yml文件中`Backbone`部分的参数即可。
|
||||
|
||||
如果要使用新的Backbone，更换backbones的例子如下:
|
||||
|
||||
1. 在 [ppocr/modeling/backbones](../../ppocr/modeling/backbones) 文件夹下新建文件，如my_backbone.py。
|
||||
2. 在 my_backbone.py 文件内添加相关代码，示例代码如下:
|
||||
|
||||
```python linenums="1"
|
||||
import paddle
|
||||
import paddle.nn as nn
|
||||
import paddle.nn.functional as F
|
||||
|
||||
|
||||
class MyBackbone(nn.Layer):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(MyBackbone, self).__init__()
|
||||
# your init code
|
||||
self.conv = nn.xxxx
|
||||
|
||||
def forward(self, inputs):
|
||||
# your network forward
|
||||
y = self.conv(inputs)
|
||||
return y
|
||||
```
|
||||
|
||||
3. 在 [ppocr/modeling/backbones/\_*init\_*.py](../../ppocr/modeling/backbones/__init__.py)文件内导入添加的`MyBackbone`模块，然后修改配置文件中Backbone进行配置即可使用，格式如下:
|
||||
|
||||
```yaml linenums="1"
|
||||
Backbone:
|
||||
name: MyBackbone
|
||||
args1: args1
|
||||
```
|
||||
|
||||
**注意**：如果要更换网络的其他模块，可以参考[文档](../../algorithm/add_new_algorithm.md)。
|
||||
|
||||
### 2.4 混合精度训练
|
||||
|
||||
如果您想进一步加快训练速度，可以使用[自动混合精度训练](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/01_paddle2.0_introduction/basic_concept/amp_cn.html)， 以单机单卡为例，命令如下：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained \
|
||||
Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True
|
||||
```
|
||||
|
||||
**注意:** 文本检测模型使用AMP时可能遇到训练不收敛问题，可以参考[discussions](https://github.com/PaddlePaddle/PaddleOCR/discussions/12445)中的临时解决方案进行使用。
|
||||
|
||||
### 2.5 分布式训练
|
||||
|
||||
多机多卡训练时，通过 `--ips` 参数设置使用的机器IP地址，通过 `--gpus` 参数设置使用的GPU ID：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/det/det_mv3_db.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/MobileNetV3_large_x0_5_pretrained
|
||||
```
|
||||
|
||||
**注意:** （1）采用多机多卡训练时，需要替换上面命令中的ips值为您机器的地址，机器之间需要能够相互ping通；（2）训练时需要在多个机器上分别启动命令。查看机器ip地址的命令为`ifconfig`；（3）更多关于分布式训练的性能优势等信息，请参考：[分布式训练教程](../blog/distributed_training.md)。
|
||||
|
||||
### 2.6 知识蒸馏训练
|
||||
|
||||
PaddleOCR支持了基于知识蒸馏的检测模型训练过程，更多内容可以参考[知识蒸馏说明文档](../model_compress/knowledge_distillation.md)。
|
||||
|
||||
**注意：** 知识蒸馏训练目前只支持PP-OCR使用的`DB`和`CRNN`算法。
|
||||
|
||||
### 2.7 其他训练环境
|
||||
|
||||
- Windows GPU/CPU
|
||||
在Windows平台上与Linux平台略有不同:
|
||||
Windows平台只支持`单卡`的训练与预测，指定GPU进行训练`set CUDA_VISIBLE_DEVICES=0`
|
||||
在Windows平台，DataLoader只支持单进程模式，因此需要设置 `num_workers` 为0;
|
||||
|
||||
- macOS
|
||||
不支持GPU模式，需要在配置文件中设置`use_gpu`为False，其余训练评估预测命令与Linux GPU完全相同。
|
||||
|
||||
- Linux DCU
|
||||
DCU设备上运行需要设置环境变量 `export HIP_VISIBLE_DEVICES=0,1,2,3`，其余训练评估预测命令与Linux GPU完全相同。
|
||||
|
||||
### 2.8 模型微调
|
||||
|
||||
实际使用过程中，建议加载官方提供的预训练模型，在自己的数据集中进行微调，关于检测模型的微调方法，请参考：[模型微调教程](./finetune.md)。
|
||||
|
||||
## 3. 模型评估与预测
|
||||
|
||||
### 3.1 指标评估
|
||||
|
||||
PaddleOCR计算三个OCR检测相关的指标，分别是：Precision、Recall、Hmean（F-Score）。
|
||||
|
||||
训练中模型参数默认保存在`Global.save_model_dir`目录下。在评估指标时，需要设置`Global.checkpoints`指向保存的参数文件。
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/eval.py -c configs/det/det_mv3_db.yml -o Global.checkpoints="{path/to/weights}/best_accuracy"
|
||||
```
|
||||
|
||||
### 3.2 测试检测效果
|
||||
|
||||
测试单张图像的检测效果：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/det_db/best_accuracy"
|
||||
```
|
||||
|
||||
测试DB模型时，调整后处理阈值：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/img_10.jpg" Global.pretrained_model="./output/det_db/best_accuracy" PostProcess.box_thresh=0.6 PostProcess.unclip_ratio=2.0
|
||||
```
|
||||
|
||||
- 注：`box_thresh`、`unclip_ratio`是DB后处理参数，其他检测模型不支持。
|
||||
|
||||
测试文件夹下所有图像的检测效果：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_det.py -c configs/det/det_mv3_db.yml -o Global.infer_img="./doc/imgs_en/" Global.pretrained_model="./output/det_db/best_accuracy"
|
||||
```
|
||||
|
||||
## 4. 模型导出与预测
|
||||
|
||||
inference 模型（`paddle.jit.save`保存的模型）
|
||||
一般是模型训练，把模型结构和模型参数保存在文件中的固化模型，多用于预测部署场景。
|
||||
训练过程中保存的模型是checkpoints模型，保存的只有模型的参数，多用于恢复训练等。
|
||||
与checkpoints模型相比，inference 模型会额外保存模型的结构信息，在预测部署、加速推理上性能优越，灵活方便，适合于实际系统集成。
|
||||
|
||||
检测模型转inference 模型方式：
|
||||
|
||||
```bash linenums="1"
|
||||
# 加载配置文件`det_mv3_db.yml`，从`output/det_db`目录下加载`best_accuracy`模型，inference模型保存在`./output/det_db_inference`目录下
|
||||
python3 tools/export_model.py -c configs/det/det_mv3_db.yml -o Global.pretrained_model="./output/det_db/best_accuracy" Global.save_inference_dir="./output/det_db_inference/"
|
||||
```
|
||||
|
||||
DB检测模型inference 模型预测：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer/predict_det.py --det_algorithm="DB" --det_model_dir="./output/det_db_inference/" --image_dir="./doc/imgs/" --use_gpu=True
|
||||
```
|
||||
|
||||
如果是其他检测，比如EAST模型，det_algorithm参数需要修改为EAST，默认为DB算法：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer/predict_det.py --det_algorithm="EAST" --det_model_dir="./output/det_db_inference/" --image_dir="./doc/imgs/" --use_gpu=True
|
||||
```
|
||||
|
||||
更多关于推理超参数的配置与解释，请参考：[模型推理超参数解释教程](../blog/inference_args.md)。
|
||||
|
||||
## 5. FAQ
|
||||
|
||||
Q1: 训练模型转inference 模型之后预测效果不一致？
|
||||
|
||||
**A**：此类问题出现较多，问题多是trained model预测时候的预处理、后处理参数和inference model预测的时候的预处理、后处理参数不一致导致的。以det_mv3_db.yml配置文件训练的模型为例，训练模型、inference模型预测结果不一致问题解决方式如下：
|
||||
|
||||
- 检查[trained model预处理](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/configs/det/det_mv3_db.yml#L116)，和[inference model的预测预处理](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/tools/infer/predict_det.py#L42)函数是否一致。算法在评估的时候，输入图像大小会影响精度，为了和论文保持一致，训练icdar15配置文件中将图像resize到[736, 1280]，但是在inference model预测的时候只有一套默认参数，会考虑到预测速度问题，默认限制图像最长边为960做resize的。训练模型预处理和inference模型的预处理函数位于[ppocr/data/imaug/operators.py](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/ppocr/data/imaug/operators.py#L147)
|
||||
- 检查[trained model后处理](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/configs/det/det_mv3_db.yml#L51)，和[inference 后处理参数](https://github.com/PaddlePaddle/PaddleOCR/blob/c1ed243fb68d5d466258243092e56cbae32e2c14/tools/infer/utility.py#L50)是否一致。
|
||||
|
||||
Q1: 训练EAST模型提示找不到lanms库？
|
||||
|
||||
**A**：执行pip3 install lanms-nova 即可。
|
||||
227
docs/version2.x/ppocr/model_train/finetune.en.md
Normal file
227
docs/version2.x/ppocr/model_train/finetune.en.md
Normal file
@@ -0,0 +1,227 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Fine-tune
|
||||
|
||||
## 1. Background and meaning
|
||||
|
||||
The PP-OCR series models provided by PaddleOCR have excellent performance in general scenarios and can solve detection and recognition problems in most cases. In vertical scenarios, if you want to obtain better model, you can further improve the accuracy of the PP-OCR series detection and recognition models through fine-tune.
|
||||
|
||||
This article mainly introduces some precautions when fine-tuning the text detection and recognition model. Finally, you can obtain a text detection and recognition model with higher accuracy through model fine-tuning in your own scenarios.
|
||||
|
||||
The core points of this article are as follows:
|
||||
|
||||
1. The pre-trained model provided by PP-OCR has better generalization ability
|
||||
2. Adding a small amount of real data (detection:>=500, recognition:>=5000) will greatly improve the detection and recognition effect of vertical scenes
|
||||
3. When fine-tuning the model, adding real general scene data can further improve the model accuracy and generalization performance
|
||||
4. In the text detection task, increasing the prediction shape of the image can further improve the detection effect of the smaller text area
|
||||
5. When fine-tuning the model, it is necessary to properly adjust the hyperparameters (learning rate and batch size are the most important) to obtain a better fine-tuning effect.
|
||||
|
||||
For more details, please refer to Chapter 2 and Chapter 3.
|
||||
|
||||
## 2. Text detection model fine-tuning
|
||||
|
||||
### 2.1 Dataset
|
||||
|
||||
* Dataset: It is recommended to prepare at least 500 text detection datasets for model fine-tuning.
|
||||
|
||||
* Dataset annotation: single-line text annotation format, it is recommended that the labeled detection frame be consistent with the actual semantic content. For example, in the train ticket scene, the surname and first name may be far apart, but they belong to the same detection field semantically. Here, the entire name also needs to be marked as a detection frame.
|
||||
|
||||
### 2.2 Model
|
||||
|
||||
It is recommended to choose the PP-OCRv3 model (configuration file: [PP-OCRv3_mobile_det.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/det/PP-OCRv3/PP-OCRv3_mobile_det.yml)，pre-trained model: [ch_PP-OCRv3_det_distill_train.tar](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar), its accuracy and generalization performance is the best pre-training model currently available.
|
||||
|
||||
For more PP-OCR series models, please refer to [PP-OCR Series Model Library](../models_list.en.md).
|
||||
|
||||
Note: When using the above pre-trained model, you need to use the `student.pdparams` file in the folder as the pre-trained model, that is, only use the student model.
|
||||
|
||||
### 2.3 Training hyperparameter
|
||||
|
||||
When fine-tuning the model, the most important hyperparameter is the pre-training model path `pretrained_model`, `learning_rate` and `batch_size`，some hyperparameters are as follows:
|
||||
|
||||
```yaml linenums="1"
|
||||
Global:
|
||||
pretrained_model: ./ch_PP-OCRv3_det_distill_train/student.pdparams # pre-training model path
|
||||
Optimizer:
|
||||
lr:
|
||||
name: Cosine
|
||||
learning_rate: 0.001 # learning_rate
|
||||
warmup_epoch: 2
|
||||
regularizer:
|
||||
name: 'L2'
|
||||
factor: 0
|
||||
|
||||
Train:
|
||||
loader:
|
||||
shuffle: True
|
||||
drop_last: False
|
||||
batch_size_per_card: 8 # single gpu batch size
|
||||
num_workers: 4
|
||||
```
|
||||
|
||||
In the above configuration file, you need to specify the `pretrained_model` field as the `student.pdparams` file path.
|
||||
|
||||
The configuration file provided by PaddleOCR is for 8-gpu training (equivalent to a total batch size of `8*8=64`) and no pre-trained model is loaded. Therefore, in your scenario, the learning rate is the same as the total The batch size needs to be adjusted linearly, for example
|
||||
|
||||
* If your scenario is single-gpu training, single gpu batch_size=8, then the total batch_size=8, it is recommended to adjust the learning rate to about `1e-4`.
|
||||
* If your scenario is for single-gpu training, due to memory limitations, you can only set batch_size=4 for a single gpu, and the total batch_size=4. It is recommended to adjust the learning rate to about `5e-5`.
|
||||
|
||||
### 2.4 Prediction hyperparameter
|
||||
|
||||
When exporting and inferring the trained model, you can further adjust the predicted image scale to improve the detection effect of small-area text. The following are some hyperparameters during DBNet inference, which can be adjusted appropriately to improve the effect.
|
||||
|
||||
| hyperparameter | type | default | meaning |
|
||||
| :--: | :--: | :--: | :--: |
|
||||
| det_db_thresh | float | 0.3 | In the probability map output by DB, pixels with a score greater than the threshold will be considered as text pixels |
|
||||
| det_db_box_thresh | float | 0.6 | When the average score of all pixels within the frame of the detection result is greater than the threshold, the result will be considered as a text area |
|
||||
| det_db_unclip_ratio | float | 1.5 | The expansion coefficient of `Vatti clipping`, using this method to expand the text area |
|
||||
| max_batch_size | int | 10 | batch size |
|
||||
| use_dilation | bool | False | Whether to expand the segmentation results to obtain better detection results |
|
||||
| det_db_score_mode | str | "fast" | DB's detection result score calculation method supports `fast` and `slow`. `fast` calculates the average score based on all pixels in the polygon’s circumscribed rectangle border, and `slow` calculates the average score based on all pixels in the original polygon. The calculation speed is relatively slower, but more accurate. |
|
||||
|
||||
For more information on inference methods, please refer to[Paddle Inference doc](../infer_deploy/python_infer.en.md).
|
||||
|
||||
## 3. Text recognition model fine-tuning
|
||||
|
||||
### 3.1 Dataset
|
||||
|
||||
* Dataset：If the dictionary is not changed, it is recommended to prepare at least 5,000 text recognition datasets for model fine-tuning; if the dictionary is changed (not recommended), more quantities are required.
|
||||
|
||||
* Data distribution: It is recommended that the distribution be as consistent as possible with the actual measurement scenario. If the actual scene contains a lot of short text, it is recommended to include more short text in the training data. If the actual scene has high requirements for the recognition effect of spaces, it is recommended to include more text content with spaces in the training data.
|
||||
|
||||
* Data synthesis: In the case of some character recognition errors, it is recommended to obtain a batch of specific character dataset, add it to the original dataset and use a small learning rate for fine-tuning. The ratio of original dataset to new dataset can be 10:1 to 5:1 to avoid overfitting of the model caused by too much data in a single scene. At the same time, try to balance the word frequency of the corpus to ensure that the frequency of common words will not be too low.
|
||||
|
||||
Specific characters can be generated using the TextRenderer tool, for synthesis examples, please refer to [data synthesis](../../applications/光功率计数码管字符识别.md)
|
||||
. The synthetic data corpus should come from real usage scenarios as much as possible, and keep the richness of fonts and backgrounds on the basis of being close to the real scene, which will help improve the model effect.
|
||||
|
||||
* Common Chinese and English data: During training, common real data can be added to the training set (for example, in the fine-tuning scenario without changing the dictionary, it is recommended to add real data such as LSVT, RCTW, MTWI) to further improve the generalization performance of the model.
|
||||
|
||||
### 3.2 Model
|
||||
|
||||
It is recommended to choose the PP-OCRv3 model (configuration file: [PP-OCRv3_mobile_rec_distillation.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/rec/PP-OCRv3/PP-OCRv3_mobile_rec_distillation.yml)，pre-trained model: [PP-OCRv3_mobile_rec_train.tar](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar)，its accuracy and generalization performance is the best pre-training model currently available.
|
||||
|
||||
For more PP-OCR series models, please refer to [PP-OCR Series Model Library](../model_list.en.md).
|
||||
|
||||
The PP-OCRv3 model uses the GTC strategy. The SAR branch has a large number of parameters. When the training data is a simple scene, the model is easy to overfit, resulting in poor fine-tuning effect. It is recommended to remove the GTC strategy. The configuration file of the model structure is modified as follows:
|
||||
|
||||
```yaml linenums="1"
|
||||
Architecture:
|
||||
model_type: rec
|
||||
algorithm: SVTR
|
||||
Transform:
|
||||
Backbone:
|
||||
name: MobileNetV1Enhance
|
||||
scale: 0.5
|
||||
last_conv_stride: [1, 2]
|
||||
last_pool_type: avg
|
||||
Neck:
|
||||
name: SequenceEncoder
|
||||
encoder_type: svtr
|
||||
dims: 64
|
||||
depth: 2
|
||||
hidden_dims: 120
|
||||
use_guide: False
|
||||
Head:
|
||||
name: CTCHead
|
||||
fc_decay: 0.00001
|
||||
Loss:
|
||||
name: CTCLoss
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
......
|
||||
transforms:
|
||||
# remove RecConAug
|
||||
# - RecConAug:
|
||||
# prob: 0.5
|
||||
# ext_data_num: 2
|
||||
# image_shape: [48, 320, 3]
|
||||
# max_text_length: *max_text_length
|
||||
- RecAug:
|
||||
# modify Encode
|
||||
- CTCLabelEncode:
|
||||
- KeepKeys:
|
||||
keep_keys:
|
||||
- image
|
||||
- label
|
||||
- length
|
||||
...
|
||||
|
||||
Eval:
|
||||
dataset:
|
||||
...
|
||||
transforms:
|
||||
...
|
||||
- CTCLabelEncode:
|
||||
- KeepKeys:
|
||||
keep_keys:
|
||||
- image
|
||||
- label
|
||||
- length
|
||||
...
|
||||
|
||||
|
||||
```
|
||||
|
||||
### 3.3 Training hyperparameter
|
||||
|
||||
Similar to text detection task fine-tuning, when fine-tuning the recognition model, the most important hyperparameters are the pre-trained model path `pretrained_model`, `learning_rate` and `batch_size`, some default configuration files are shown below.
|
||||
|
||||
```yaml linenums="1"
|
||||
Global:
|
||||
pretrained_model: # pre-training model path
|
||||
Optimizer:
|
||||
lr:
|
||||
name: Piecewise
|
||||
decay_epochs : [700, 800]
|
||||
values : [0.001, 0.0001] # learning_rate
|
||||
warmup_epoch: 5
|
||||
regularizer:
|
||||
name: 'L2'
|
||||
factor: 0
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
data_dir: ./train_data/
|
||||
label_file_list:
|
||||
- ./train_data/train_list.txt
|
||||
ratio_list: [1.0] # Sampling ratio, the default value is [1.0]
|
||||
loader:
|
||||
shuffle: True
|
||||
drop_last: False
|
||||
batch_size_per_card: 128 # single gpu batch size
|
||||
num_workers: 8
|
||||
|
||||
```
|
||||
|
||||
In the above configuration file, you first need to specify the `pretrained_model` field as the `ch_PP-OCRv3_rec_train/best_accuracy.pdparams` file path decompressed in Chapter 3.2.
|
||||
|
||||
The configuration file provided by PaddleOCR is for 8-gpu training (equivalent to a total batch size of `8*128=1024`) and no pre-trained model is loaded. Therefore, in your scenario, the learning rate is the same as the total The batch size needs to be adjusted linearly, for example:
|
||||
|
||||
* If your scenario is single-gpu training, single gpu batch_size=128, then the total batch_size=128, in the case of loading the pre-trained model, it is recommended to adjust the learning rate to about `[1e-4, 2e-5]` (For the piecewise learning rate strategy, two values need to be set, the same below).
|
||||
* If your scenario is for single-gpu training, due to memory limitations, you can only set batch_size=64 for a single gpu, and the total batch_size=64. When loading the pre-trained model, it is recommended to adjust the learning rate to `[5e-5 , 1e-5]`about.
|
||||
|
||||
If there is general real scene data added, it is recommended that in each epoch, the amount of vertical scene data and real scene data should be kept at about 1:1.
|
||||
|
||||
For example: your own vertical scene recognition data volume is 1W, the data label file is `vertical.txt`, the collected general scene recognition data volume is 10W, and the data label file is `general.txt`.
|
||||
|
||||
Then, the `label_file_list` and `ratio_list` parameters can be set as shown below. In each epoch, `vertical.txt` will be fully sampled (sampling ratio is 1.0), including 1W pieces of data; `general.txt` will be sampled according to a sampling ratio of 0.1, including `10W*0.1=1W` pieces of data, the final ratio of the two is `1:1`.
|
||||
|
||||
```yaml linenums="1"
|
||||
Train:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
data_dir: ./train_data/
|
||||
label_file_list:
|
||||
- vertical.txt
|
||||
- general.txt
|
||||
ratio_list: [1.0, 0.1]
|
||||
```
|
||||
|
||||
### 3.4 Training optimization
|
||||
|
||||
The training process does not happen overnight. After completing a stage of training evaluation, it is recommended to collect and analyze the badcase of the current model in the real scene, adjust the proportion of training data in a targeted manner, or further add synthetic data. Through multiple iterations of training, the model effect is continuously optimized.
|
||||
|
||||
If you modify the custom dictionary during training, since the parameters of the last layer of FC cannot be loaded, it is normal for acc=0 at the beginning of the iteration. Don't worry, loading the pre-trained model can still speed up the model convergence.
|
||||
227
docs/version2.x/ppocr/model_train/finetune.md
Normal file
227
docs/version2.x/ppocr/model_train/finetune.md
Normal file
@@ -0,0 +1,227 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 模型微调
|
||||
|
||||
## 1. 模型微调背景与意义
|
||||
|
||||
PaddleOCR提供的PP-OCR系列模型在通用场景中性能优异，能够解决绝大多数情况下的检测与识别问题。在垂类场景中，如果希望获取更优的模型效果，可以通过模型微调的方法，进一步提升PP-OCR系列检测与识别模型的精度。
|
||||
|
||||
本文主要介绍文本检测与识别模型在模型微调时的一些注意事项，最终希望您在自己的场景中，通过模型微调，可以获取精度更高的文本检测与识别模型。
|
||||
|
||||
本文核心要点如下所示。
|
||||
|
||||
1. PP-OCR提供的预训练模型有较好的泛化能力
|
||||
2. 加入少量真实数据（检测任务>=500张, 识别任务>=5000张），会大幅提升垂类场景的检测与识别效果
|
||||
3. 在模型微调时，加入真实通用场景数据，可以进一步提升模型精度与泛化性能
|
||||
4. 在图像检测任务中，增大图像的预测尺度，能够进一步提升较小文字区域的检测效果
|
||||
5. 在模型微调时，需要适当调整超参数（学习率，batch size最为重要），以获得更优的微调效果。
|
||||
|
||||
更多详细内容，请参考第2章与第3章。
|
||||
|
||||
## 2. 文本检测模型微调
|
||||
|
||||
### 2.1 数据选择
|
||||
|
||||
* 数据量：建议至少准备500张的文本检测数据集用于模型微调。
|
||||
|
||||
* 数据标注：单行文本标注格式，建议标注的检测框与实际语义内容一致。如在火车票场景中，姓氏与名字可能离得较远，但是它们在语义上属于同一个检测字段，这里也需要将整个姓名标注为1个检测框。
|
||||
|
||||
### 2.2 模型选择
|
||||
|
||||
建议选择PP-OCRv3模型（配置文件：[PP-OCRv3_mobile_det.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/det/PP-OCRv3/PP-OCRv3_mobile_det.yml)，预训练模型：[ch_PP-OCRv3_det_distill_train.tar](https://paddle-model-ecology.bj.bcebos.com/paddlex/official_pretrained_model/PP-OCRv3_mobile_det_pretrained.pdparams)）进行微调，其精度与泛化性能是目前提供的最优预训练模型。
|
||||
|
||||
更多PP-OCR系列模型，请参考[PP-OCR 系列模型库](../model_list.md)。
|
||||
|
||||
注意：在使用上述预训练模型的时候，需要使用文件夹中的`student.pdparams`文件作为预训练模型，即，仅使用学生模型。
|
||||
|
||||
### 2.3 训练超参选择
|
||||
|
||||
在模型微调的时候，最重要的超参就是预训练模型路径`pretrained_model`, 学习率`learning_rate`与`batch_size`，部分配置文件如下所示。
|
||||
|
||||
```yaml linenums="1"
|
||||
Global:
|
||||
pretrained_model: ./ch_PP-OCRv3_det_distill_train/student.pdparams # 预训练模型路径
|
||||
Optimizer:
|
||||
lr:
|
||||
name: Cosine
|
||||
learning_rate: 0.001 # 学习率
|
||||
warmup_epoch: 2
|
||||
regularizer:
|
||||
name: 'L2'
|
||||
factor: 0
|
||||
|
||||
Train:
|
||||
loader:
|
||||
shuffle: True
|
||||
drop_last: False
|
||||
batch_size_per_card: 8 # 单卡batch size
|
||||
num_workers: 4
|
||||
```
|
||||
|
||||
上述配置文件中，首先需要将`pretrained_model`字段指定为`student.pdparams`文件路径。
|
||||
|
||||
PaddleOCR提供的配置文件是在8卡训练（相当于总的batch size是`8*8=64`）、且没有加载预训练模型情况下的配置文件，因此您的场景中，学习率与总的batch size需要对应线性调整，例如
|
||||
|
||||
* 如果您的场景中是单卡训练，单卡batch_size=8，则总的batch_size=8，建议将学习率调整为`1e-4`左右。
|
||||
* 如果您的场景中是单卡训练，由于显存限制，只能设置单卡batch_size=4，则总的batch_size=4，建议将学习率调整为`5e-5`左右。
|
||||
|
||||
### 2.4 预测超参选择
|
||||
|
||||
对训练好的模型导出并进行推理时，可以通过进一步调整预测的图像尺度，来提升小面积文本的检测效果，下面是DBNet推理时的一些超参数，可以通过适当调整，提升效果。
|
||||
|
||||
| 参数名称 | 类型 | 默认值 | 含义 |
|
||||
| :--: | :--: | :--: | :--: |
|
||||
| det_db_thresh | float | 0.3 | DB输出的概率图中，得分大于该阈值的像素点才会被认为是文字像素点 |
|
||||
| det_db_box_thresh | float | 0.6 | 检测结果边框内，所有像素点的平均得分大于该阈值时，该结果会被认为是文字区域 |
|
||||
| det_db_unclip_ratio | float | 1.5 | `Vatti clipping`算法的扩张系数，使用该方法对文字区域进行扩张 |
|
||||
| max_batch_size | int | 10 | 预测的batch size |
|
||||
| use_dilation | bool | False | 是否对分割结果进行膨胀以获取更优检测效果 |
|
||||
| det_db_score_mode | str | "fast" | DB的检测结果得分计算方法，支持`fast`和`slow`，`fast`是根据polygon的外接矩形边框内的所有像素计算平均得分，`slow`是根据原始polygon内的所有像素计算平均得分，计算速度相对较慢一些，但是更加准确一些。 |
|
||||
|
||||
更多关于推理方法的介绍可以参考[Paddle Inference推理教程](../infer_deploy/python_infer.md)。
|
||||
|
||||
## 3. 文本识别模型微调
|
||||
|
||||
### 3.1 数据选择
|
||||
|
||||
* 数据量：不更换字典的情况下，建议至少准备5000张的文本识别数据集用于模型微调；如果更换了字典（不建议），需要的数量更多。
|
||||
|
||||
* 数据分布：建议分布与实测场景尽量一致。如果实测场景包含大量短文本，则训练数据中建议也包含较多短文本，如果实测场景对于空格识别效果要求较高，则训练数据中建议也包含较多带空格的文本内容。
|
||||
|
||||
* 数据合成：针对部分字符识别有误的情况，建议获取一批特定字符数据，加入到原数据中使用小学习率微调。其中原始数据与新增数据比例可尝试 10:1 ～ 5：1， 避免单一场景数据过多导致模型过拟合，同时尽量平衡语料词频，确保常用字的出现频率不会过低。
|
||||
|
||||
特定字符生成可以使用 TextRenderer 工具，合成例子可参考 [数码管数据合成](../../applications/光功率计数码管字符识别.md)
|
||||
，合成数据语料尽量来自真实使用场景，在贴近真实场景的基础上保持字体、背景的丰富性，有助于提升模型效果。
|
||||
|
||||
* 通用中英文数据：在训练的时候，可以在训练集中添加通用真实数据（如在不更换字典的微调场景中，建议添加LSVT、RCTW、MTWI等真实数据），进一步提升模型的泛化性能。
|
||||
|
||||
### 3.2 模型选择
|
||||
|
||||
建议选择PP-OCRv3模型（配置文件：[PP-OCRv3_mobile_rec_distillation.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/rec/PP-OCRv3/PP-OCRv3_mobile_rec_distillation.yml)，预训练模型：[ch_PP-OCRv3_rec_train.tar](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar)）进行微调，其精度与泛化性能是目前提供的最优预训练模型。
|
||||
|
||||
更多PP-OCR系列模型，请参考[PP-OCR 系列模型库](../model_list.md)。
|
||||
|
||||
PP-OCRv3 模型使用了GTC策略，其中SAR分支参数量大，当训练数据为简单场景时模型容易过拟合，导致微调效果不佳，建议去除GTC策略，模型结构部分配置文件修改如下：
|
||||
|
||||
```yaml linenums="1"
|
||||
Architecture:
|
||||
model_type: rec
|
||||
algorithm: SVTR
|
||||
Transform:
|
||||
Backbone:
|
||||
name: MobileNetV1Enhance
|
||||
scale: 0.5
|
||||
last_conv_stride: [1, 2]
|
||||
last_pool_type: avg
|
||||
Neck:
|
||||
name: SequenceEncoder
|
||||
encoder_type: svtr
|
||||
dims: 64
|
||||
depth: 2
|
||||
hidden_dims: 120
|
||||
use_guide: False
|
||||
Head:
|
||||
name: CTCHead
|
||||
fc_decay: 0.00001
|
||||
Loss:
|
||||
name: CTCLoss
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
......
|
||||
transforms:
|
||||
# 去除 RecConAug 增广
|
||||
# - RecConAug:
|
||||
# prob: 0.5
|
||||
# ext_data_num: 2
|
||||
# image_shape: [48, 320, 3]
|
||||
# max_text_length: *max_text_length
|
||||
- RecAug:
|
||||
# 修改 Encode 方式
|
||||
- CTCLabelEncode:
|
||||
- KeepKeys:
|
||||
keep_keys:
|
||||
- image
|
||||
- label
|
||||
- length
|
||||
...
|
||||
|
||||
Eval:
|
||||
dataset:
|
||||
...
|
||||
transforms:
|
||||
...
|
||||
- CTCLabelEncode:
|
||||
- KeepKeys:
|
||||
keep_keys:
|
||||
- image
|
||||
- label
|
||||
- length
|
||||
...
|
||||
|
||||
|
||||
```
|
||||
|
||||
### 3.3 训练超参选择
|
||||
|
||||
与文本检测任务微调相同，在识别模型微调的时候，最重要的超参就是预训练模型路径`pretrained_model`, 学习率`learning_rate`与`batch_size`，部分默认配置文件如下所示。
|
||||
|
||||
```yaml linenums="1"
|
||||
Global:
|
||||
pretrained_model: # 预训练模型路径
|
||||
Optimizer:
|
||||
lr:
|
||||
name: Piecewise
|
||||
decay_epochs : [700, 800]
|
||||
values : [0.001, 0.0001] # 学习率
|
||||
warmup_epoch: 5
|
||||
regularizer:
|
||||
name: 'L2'
|
||||
factor: 0
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
data_dir: ./train_data/
|
||||
label_file_list:
|
||||
- ./train_data/train_list.txt
|
||||
ratio_list: [1.0] # 采样比例，默认值是[1.0]
|
||||
loader:
|
||||
shuffle: True
|
||||
drop_last: False
|
||||
batch_size_per_card: 128 # 单卡batch size
|
||||
num_workers: 8
|
||||
|
||||
```
|
||||
|
||||
上述配置文件中，首先需要将`pretrained_model`字段指定为3.2章节中解压得到的`ch_PP-OCRv3_rec_train/best_accuracy.pdparams`文件路径。
|
||||
|
||||
PaddleOCR提供的配置文件是在8卡训练（相当于总的batch size是`8*128=1024`）、且没有加载预训练模型情况下的配置文件，因此您的场景中，学习率与总的batch size需要对应线性调整，例如：
|
||||
|
||||
* 如果您的场景中是单卡训练，单卡batch_size=128，则总的batch_size=128，在加载预训练模型的情况下，建议将学习率调整为`[1e-4, 2e-5]`左右（piecewise学习率策略，需设置2个值，下同）。
|
||||
* 如果您的场景中是单卡训练，因为显存限制，只能设置单卡batch_size=64，则总的batch_size=64，在加载预训练模型的情况下，建议将学习率调整为`[5e-5, 1e-5]`左右。
|
||||
|
||||
如果有通用真实场景数据加进来，建议每个epoch中，垂类场景数据与真实场景的数据量保持在1:1左右。
|
||||
|
||||
比如：您自己的垂类场景识别数据量为1W，数据标签文件为`vertical.txt`，收集到的通用场景识别数据量为10W，数据标签文件为`general.txt`，
|
||||
|
||||
那么，可以设置`label_file_list`和`ratio_list`参数如下所示。每个epoch中，`vertical.txt`中会进行全采样（采样比例为1.0），包含1W条数据；`general.txt`中会按照0.1的采样比例进行采样，包含`10W*0.1=1W`条数据，最终二者的比例为`1:1`。
|
||||
|
||||
```yaml linenums="1"
|
||||
Train:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
data_dir: ./train_data/
|
||||
label_file_list:
|
||||
- vertical.txt
|
||||
- general.txt
|
||||
ratio_list: [1.0, 0.1]
|
||||
```
|
||||
|
||||
### 3.4 训练调优
|
||||
|
||||
训练过程并非一蹴而就的，完成一个阶段的训练评估后，建议收集分析当前模型在真实场景中的 badcase，有针对性的调整训练数据比例，或者进一步新增合成数据。通过多次迭代训练，不断优化模型效果。
|
||||
|
||||
如果在训练时修改了自定义字典，由于无法加载最后一层FC的参数，在迭代初期acc=0是正常的情况，不必担心，加载预训练模型依然可以加快模型收敛。
|
||||
BIN
docs/version2.x/ppocr/model_train/images/angle_class_example.jpg
Normal file
BIN
docs/version2.x/ppocr/model_train/images/angle_class_example.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
BIN
docs/version2.x/ppocr/model_train/images/en_paper.jpg
Normal file
BIN
docs/version2.x/ppocr/model_train/images/en_paper.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 154 KiB |
BIN
docs/version2.x/ppocr/model_train/images/icdar_rec.png
Normal file
BIN
docs/version2.x/ppocr/model_train/images/icdar_rec.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 921 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.5 MiB |
BIN
docs/version2.x/ppocr/model_train/images/long_text_examples.jpg
Normal file
BIN
docs/version2.x/ppocr/model_train/images/long_text_examples.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
476
docs/version2.x/ppocr/model_train/kie.en.md
Normal file
476
docs/version2.x/ppocr/model_train/kie.en.md
Normal file
@@ -0,0 +1,476 @@
|
||||
---
|
||||
comments: true
|
||||
typora-copy-images-to: images
|
||||
---
|
||||
|
||||
# Key Information Extraction
|
||||
|
||||
This tutorial provides a guide to the whole process of key information extraction using PaddleOCR, including data preparation, model training, optimization, evaluation, prediction of semantic entity recognition (SER) and relationship extraction (RE) tasks.
|
||||
|
||||
## 1. Data Preparation
|
||||
|
||||
### 1.1. Prepare for dataset
|
||||
|
||||
PaddleOCR supports the following data format when training KIE models.
|
||||
|
||||
- `general data` is used to train a dataset whose annotation is stored in a text file (SimpleDataset).
|
||||
|
||||
The default storage path of training data is `PaddleOCR/train_data`. If you already have datasets on your disk, you only need to create a soft link to the dataset directory.
|
||||
|
||||
```bash linenums="1"
|
||||
# linux and mac os
|
||||
ln -sf <path/to/dataset> <path/to/paddle_ocr>/train_data/dataset
|
||||
# windows
|
||||
mklink /d <path/to/paddle_ocr>/train_data/dataset <path/to/dataset>
|
||||
```
|
||||
|
||||
### 1.2. Custom Dataset
|
||||
|
||||
The training process generally includes the training set and the evaluation set. The data formats of the two sets are same.
|
||||
|
||||
#### (1) Training set
|
||||
|
||||
It is recommended to put the training images into the same folder, record the path and annotation of images in a text file. The contents of the text file are as follows:
|
||||
|
||||
```text linenums="1"
|
||||
" image path annotation information "
|
||||
zh_train_0.jpg [{"transcription": "汇丰晋信", "label": "other", "points": [[104, 114], [530, 114], [530, 175], [104, 175]], "id": 1, "linking": []}, {"transcription": "受理时间:", "label": "question", "points": [[126, 267], [266, 267], [266, 305], [126, 305]], "id": 7, "linking": [[7, 13]]}, {"transcription": "2020.6.15", "label": "answer", "points": [[321, 239], [537, 239], [537, 285], [321, 285]], "id": 13, "linking": [[7, 13]]}]
|
||||
zh_train_1.jpg [{"transcription": "中国人体器官捐献", "label": "other", "points": [[544, 459], [954, 459], [954, 517], [544, 517]], "id": 1, "linking": []}, {"transcription": ">编号:MC545715483585", "label": "other", "points": [[1462, 470], [2054, 470], [2054, 543], [1462, 543]], "id": 10, "linking": []}, {"transcription": "CHINAORGANDONATION", "label": "other", "points": [[543, 516], [958, 516], [958, 551], [543, 551]], "id": 14, "linking": []}, {"transcription": "中国人体器官捐献志愿登记表", "label": "header", "points": [[635, 793], [1892, 793], [1892, 904], [635, 904]], "id": 18, "linking": []}]
|
||||
...
|
||||
```
|
||||
|
||||
**Note:** In the text file, please split the image path and annotation with `\t`. Otherwise, error will happen when training.
|
||||
|
||||
The annotation can be parsed by `json` into a list of sub-annotations. Each element in the list is a dict, which stores the required information of each text line. The required fields are as follows.
|
||||
|
||||
- transcription: stores the text content of the text line
|
||||
- label: the category of the text line content
|
||||
- points: stores the four point position information of the text line
|
||||
- id: stores the ID information of the text line for RE model training
|
||||
- linking: stores the connection information between text lines for RE model training
|
||||
|
||||
#### (2) Evaluation set
|
||||
|
||||
The evaluation set is constructed in the same way as the training set.
|
||||
|
||||
#### (3) Dictionary file
|
||||
|
||||
The textlines in the training set and the evaluation set contain label information. The list of all labels is stored in the dictionary file (such as `class_list.txt`). Each line in the dictionary file is represented as a label name.
|
||||
|
||||
For example, FUND_zh data contains four categories. The contents of the dictionary file are as follows.
|
||||
|
||||
```text linenums="1"
|
||||
OTHER
|
||||
QUESTION
|
||||
ANSWER
|
||||
HEADER
|
||||
```
|
||||
|
||||
In the annotation file, the annotation information of the `label` field of the text line content of each annotation needs to belong to the dictionary content.
|
||||
|
||||
The final dataset shall have the following file structure.
|
||||
|
||||
```text linenums="1"
|
||||
|-train_data
|
||||
|-data_name
|
||||
|- train.json
|
||||
|- train
|
||||
|- zh_train_0.png
|
||||
|- zh_train_1.jpg
|
||||
| ...
|
||||
|- val.json
|
||||
|- val
|
||||
|- zh_val_0.png
|
||||
|- zh_val_1.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
-The category information in the annotation file is not case sensitive. For example, 'HEADER' and 'header' will be seen as the same category ID.
|
||||
|
||||
- In the dictionary file, it is recommended to put the `other` category (other textlines that need not be paid attention to can be labeled as `other`) on the first line. When parsing, the category ID of the 'other' category will be resolved to 0, and the textlines predicted as `other` will not be visualized later.
|
||||
|
||||
### 1.3. Download data
|
||||
|
||||
If you do not have local dataset, you can download the source files of [XFUND](https://github.com/doc-analysis/XFUND) or [FUNSD](https://guillaumejaume.github.io/FUNSD) and use the scripts of [XFUND](../../ppstructure/kie/tools/trans_xfun_data.py) or [FUNSD](../../ppstructure/kie/tools/trans_funsd_label.py) for transform them into PaddleOCR format. Then you can use the public dataset to quick experience KIE.
|
||||
|
||||
For more information about public KIE datasets, please refer to [KIE dataset tutorial](../../datasets/kie_datasets.en.md).
|
||||
|
||||
PaddleOCR also supports the annotation of KIE models. Please refer to [PPOCRLabel tutorial](https://github.com/PFCCLab/PPOCRLabel/blob/main/README.md).
|
||||
|
||||
## 2. Training
|
||||
|
||||
PaddleOCR provides training scripts, evaluation scripts and inference scripts. We will introduce based on VI-LayoutXLM model in this section.
|
||||
This section will take the VI layoutxlm multimodal pre training model as an example to explain.
|
||||
|
||||
> If you want to use the SDMGR based KIE algorithm, please refer to: [SDMGR tutorial](../../algorithm/kie/algorithm_kie_sdmgr.en.md).
|
||||
|
||||
### 2.1. Start Training
|
||||
|
||||
If you do not use a custom dataset, you can use XFUND_zh that has been processed in PaddleOCR dataset for quick experience.
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir train_data
|
||||
cd train_data
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar && tar -xf XFUND.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
If you don't want to train, and want to directly experience the process of model evaluation, prediction, and inference, you can download the training model provided in PaddleOCR and skip section 2.1.
|
||||
|
||||
Use the following command to download the trained model.
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir pretrained_model
|
||||
cd pretrained_model
|
||||
# download and uncompress SER model
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar & tar -xf ser_vi_layoutxlm_xfund_pretrained.tar
|
||||
|
||||
# download and uncompress RE model
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar & tar -xf re_vi_layoutxlm_xfund_pretrained.tar
|
||||
```
|
||||
|
||||
Start training:
|
||||
|
||||
- If your paddlepaddle version is `CPU`, you need to set `Global.use_gpu=False` in your config file.
|
||||
- During training, PaddleOCR will download the VI-LayoutXLM pretraining model by default. There is no need to download it in advance.
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU training, support single card and multi-cards
|
||||
# The training log will be save in "{Global.save_model_dir}/train.log"
|
||||
|
||||
# train SER model using single card
|
||||
python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml
|
||||
|
||||
# train SER model using multi-cards, you can use --gpus to assign the GPU ids.
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml
|
||||
|
||||
# train RE model using single card
|
||||
python3 tools/train.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml
|
||||
```
|
||||
|
||||
Take the SER model training as an example. After the training is started, you will see the following log output.
|
||||
|
||||
```bash linenums="1"
|
||||
[2022/08/08 16:28:28] ppocr INFO: epoch: [1/200], global_step: 10, lr: 0.000006, loss: 1.871535, avg_reader_cost: 0.28200 s, avg_batch_cost: 0.82318 s, avg_samples: 8.0, ips: 9.71838 samples/s, eta: 0:51:59
|
||||
[2022/08/08 16:28:33] ppocr INFO: epoch: [1/200], global_step: 19, lr: 0.000018, loss: 1.461939, avg_reader_cost: 0.00042 s, avg_batch_cost: 0.32037 s, avg_samples: 6.9, ips: 21.53773 samples/s, eta: 0:37:55
|
||||
[2022/08/08 16:28:39] ppocr INFO: cur metric, precision: 0.11526348939743859, recall: 0.19776657060518732, hmean: 0.14564265817747712, fps: 34.008392345050055
|
||||
[2022/08/08 16:28:45] ppocr INFO: save best model is to ./output/ser_vi_layoutxlm_xfund_zh/best_accuracy
|
||||
[2022/08/08 16:28:45] ppocr INFO: best metric, hmean: 0.14564265817747712, precision: 0.11526348939743859, recall: 0.19776657060518732, fps: 34.008392345050055, best_epoch: 1
|
||||
[2022/08/08 16:28:51] ppocr INFO: save model in ./output/ser_vi_layoutxlm_xfund_zh/latest
|
||||
```
|
||||
|
||||
The following information will be automatically printed.
|
||||
|
||||
|Field | meaning|
|
||||
| :----: | :------: |
|
||||
|epoch | current iteration round|
|
||||
|iter | current iteration times|
|
||||
|lr | current learning rate|
|
||||
|loss | current loss function|
|
||||
| reader_cost | current batch data processing time|
|
||||
| batch_ Cost | total current batch time|
|
||||
|samples | number of samples in the current batch|
|
||||
|ips | number of samples processed per second|
|
||||
|
||||
PaddleOCR supports evaluation during training. you can modify `eval_batch_step` in the config file `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` (default as 19 iters). Trained model with best hmean will be saved as `output/ser_vi_layoutxlm_xfund_zh/best_accuracy/`.
|
||||
|
||||
If the evaluation dataset is very large, it's recommended to enlarge the eval interval or evaluate the model after training.
|
||||
|
||||
**Note:** for more KIE models training and configuration files, you can go into `configs/kie/` or refer to [Frontier KIE algorithms](./algorithm_overview_en.md).
|
||||
|
||||
If you want to train model on your own dataset, you need to modify the data path, dictionary file and category number in the configuration file.
|
||||
|
||||
Take `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` as an example, contents we need to fix is as follows.
|
||||
|
||||
```yaml linenums="1"
|
||||
Architecture:
|
||||
# ...
|
||||
Backbone:
|
||||
name: LayoutXLMForSer
|
||||
pretrained: True
|
||||
mode: vi
|
||||
# Assuming that n categories are included in the dictionary file (other is included), the the num_classes is set as 2n-1
|
||||
num_classes: &num_classes 7
|
||||
|
||||
PostProcess:
|
||||
name: kieSerTokenLayoutLMPostProcess
|
||||
# Modify the dictionary file path for your custom dataset
|
||||
class_path: &class_path train_data/XFUND/class_list_xfun.txt
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
# Modify the data path for your training dataset
|
||||
data_dir: train_data/XFUND/zh_train/image
|
||||
# Modify the data annotation path for your training dataset
|
||||
label_file_list:
|
||||
- train_data/XFUND/zh_train/train.json
|
||||
...
|
||||
loader:
|
||||
# batch size for single card when training
|
||||
batch_size_per_card: 8
|
||||
...
|
||||
|
||||
Eval:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
# Modify the data path for your evaluation dataset
|
||||
data_dir: train_data/XFUND/zh_val/image
|
||||
# Modify the data annotation path for your evaluation dataset
|
||||
label_file_list:
|
||||
- train_data/XFUND/zh_val/val.json
|
||||
...
|
||||
loader:
|
||||
# batch size for single card when evaluation
|
||||
batch_size_per_card: 8
|
||||
```
|
||||
|
||||
**Note that the configuration file for prediction/evaluation must be consistent with the training file.**
|
||||
|
||||
### 2.2. Resume Training
|
||||
|
||||
If the training process is interrupted and you want to load the saved model to resume training, you can specify the path of the model to be loaded by specifying `Architecture.Backbone.checkpoints`.
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- Priority of `Architecture.Backbone.checkpoints` is higher than`Architecture.Backbone.pretrained`. You need to set `Architecture.Backbone.checkpoints` for model finetuning, resume and evaluation. If you want to train with the NLP pretrained model, you need to set `Architecture.Backbone.pretrained` as `True` and set `Architecture.Backbone.checkpoints` as null (`null`).
|
||||
- PaddleNLP pretrained models are used here for LayoutXLM series models, the model loading and saving logic is same as those in PaddleNLP. Therefore we do not need to set `Global.pretrained_model` or `Global.checkpoints` here.
|
||||
- If you use knowledge distillation to train the LayoutXLM series models, resuming training is not supported now.
|
||||
|
||||
### 2.3. Mixed Precision Training
|
||||
|
||||
coming soon!
|
||||
|
||||
### 2.4. Distributed Training
|
||||
|
||||
During multi-machine multi-gpu training, use the `--ips` parameter to set the used machine IP address, and the `--gpus` parameter to set the used GPU ID:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml
|
||||
```
|
||||
|
||||
**Note:** (1) When using multi-machine and multi-gpu training, you need to replace the ips value in the above command with the address of your machine, and the machines need to be able to ping each other. (2) Training needs to be launched separately on multiple machines. The command to view the ip address of the machine is `ifconfig`. (3) For more details about the distributed training speedup ratio, please refer to [Distributed Training Tutorial](../blog/distributed_training.en.md).
|
||||
|
||||
### 2.5. Train with Knowledge Distillation
|
||||
|
||||
Knowledge distillation is supported in PaddleOCR for KIE model training process. The configuration file is [ser_vi_layoutxlm_xfund_zh_udml.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml). For more information, please refer to [doc](../model_compress/knowledge_distillation.en.md).
|
||||
|
||||
**Note:** The saving and loading logic of the LayoutXLM series KIE models in PaddleOCR is consistent with PaddleNLP, so only the parameters of the student model are saved in the distillation process. If you want to use the saved model for evaluation, you need to use the configuration of the student model (the student model corresponding to the distillation file above is [ser_vi_layoutxlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml).
|
||||
|
||||
### 2.6. Training on other platform
|
||||
|
||||
- Windows GPU/CPU
|
||||
The Windows platform is slightly different from the Linux platform:
|
||||
Windows platform only supports `single gpu` training and inference, specify GPU for training `set CUDA_VISIBLE_DEVICES=0`
|
||||
On the Windows platform, DataLoader only supports single-process mode, so you need to set `num_workers` to 0;
|
||||
|
||||
- macOS
|
||||
GPU mode is not supported, you need to set `use_gpu` to False in the configuration file, and the rest of the training evaluation prediction commands are exactly the same as Linux GPU.
|
||||
|
||||
- Linux DCU
|
||||
Running on a DCU device requires setting the environment variable `export HIP_VISIBLE_DEVICES=0,1,2,3`, and the rest of the training and evaluation prediction commands are exactly the same as the Linux GPU.
|
||||
|
||||
## 3. Evaluation and Test
|
||||
|
||||
### 3.1. Evaluation
|
||||
|
||||
The trained model will be saved in `Global.save_model_dir`. When evaluation, you need to set `Architecture.Backbone.checkpoints` as your model directory. The evaluation dataset can be set by modifying the `Eval.dataset.label_file_list` field in the `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` file.
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU evaluation, Global.checkpoints is the weight to be tested
|
||||
python3 tools/eval.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy
|
||||
```
|
||||
|
||||
The following information will be printed such as precision, recall, hmean and so on.
|
||||
|
||||
```bash linenums="1"
|
||||
[2022/08/09 07:59:28] ppocr INFO: metric eval ***************
|
||||
[2022/08/09 07:59:28] ppocr INFO: precision:0.697476609016161
|
||||
[2022/08/09 07:59:28] ppocr INFO: recall:0.8861671469740634
|
||||
[2022/08/09 07:59:28] ppocr INFO: hmean:0.7805806758686339
|
||||
[2022/08/09 07:59:28] ppocr INFO: fps:17.367364606899105
|
||||
```
|
||||
|
||||
### 3.2. Test
|
||||
|
||||
Using the model trained by PaddleOCR, we can quickly get prediction through the following script.
|
||||
|
||||
The default prediction image is stored in `Global.infer_img`, and the trained model weight is specified via `-o Global.checkpoints`.
|
||||
|
||||
According to the `Global.save_model_dir` and `save_epoch_step` fields set in the configuration file, the following parameters will be saved.
|
||||
|
||||
```text linenums="1"
|
||||
output/ser_vi_layoutxlm_xfund_zh/
|
||||
├── best_accuracy
|
||||
├── metric.states
|
||||
├── model_config.json
|
||||
├── model_state.pdparams
|
||||
├── best_accuracy.pdopt
|
||||
├── config.yml
|
||||
├── train.log
|
||||
├── latest
|
||||
├── metric.states
|
||||
├── model_config.json
|
||||
├── model_state.pdparams
|
||||
├── latest.pdopt
|
||||
```
|
||||
|
||||
Among them, best_accuracy.*is the best model on the evaluation set; latest.* is the model of the last epoch.
|
||||
|
||||
The configuration file for prediction must be consistent with the training file. If you finish the training process using `python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml`. You can use the following command for prediction.
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg
|
||||
```
|
||||
|
||||
The output image is as follows, which is also saved in `Global.save_res_path`.
|
||||
|
||||
|
||||
|
||||
During the prediction process, the detection and recognition model of PP-OCRv3 will be loaded by default for information extraction of OCR. If you want to load the OCR results obtained in advance, you can use the following method to predict, and specify `Global.infer_img` as the annotation file, which contains the image path and OCR information, and specifies `Global.infer_mode` as False, indicating that the OCR inference engine is not used at this time.
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False
|
||||
```
|
||||
|
||||
For the above image, if information extraction is performed using the labeled OCR results, the prediction results are as follows.
|
||||
|
||||
|
||||
|
||||
It can be seen that part of the detection information is more accurate, but the overall information extraction results are basically the same.
|
||||
|
||||
In RE model prediction, the SER model result needs to be given first, so the configuration file and model weight of SER need to be loaded at the same time, as shown in the following example.
|
||||
|
||||
```bash linenums="1"
|
||||
python3 ./tools/infer_kie_token_ser_re.py \
|
||||
-c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/best_accuracy/ \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/image/ \
|
||||
-c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o_ser Architecture.Backbone.checkpoints=pretrain_models/ \
|
||||
ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/
|
||||
```
|
||||
|
||||
The result is as follows.
|
||||
|
||||
|
||||
|
||||
If you want to load the OCR results obtained in advance, you can use the following method to predict, and specify `Global.infer_img` as the annotation file, which contains the image path and OCR information, and specifies `Global.infer_mode` as False, indicating that the OCR inference engine is not used at this time.
|
||||
|
||||
```bash linenums="1"
|
||||
python3 ./tools/infer_kie_token_ser_re.py \
|
||||
-c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/best_accuracy/ \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/val.json \
|
||||
Global.infer_mode=False \
|
||||
-c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o_ser Architecture.Backbone.checkpoints=pretrain_models/ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/
|
||||
```
|
||||
|
||||
`c_ser` denotes SER configurations file, `o_ser` denotes the SER model configurations that will override corresponding content in the file.
|
||||
|
||||
The result is as follows.
|
||||
|
||||
|
||||
|
||||
It can be seen that the re prediction results directly using the annotated OCR results are more accurate.
|
||||
|
||||
## 4. Model inference
|
||||
|
||||
### 4.1 Export the model
|
||||
|
||||
The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment.
|
||||
|
||||
The model saved during the training process is the checkpoints model, which saves the parameters of the model and is mostly used to resume training.
|
||||
|
||||
Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems.
|
||||
|
||||
The SER model can be converted to the inference model using the following command.
|
||||
|
||||
```bash linenums="1"
|
||||
# -c Set the training algorithm yml configuration file.
|
||||
# -o Set optional parameters.
|
||||
# Architecture.Backbone.checkpoints Set the training model address.
|
||||
# Global.save_inference_dir Set the address where the converted model will be saved.
|
||||
python3 tools/export_model.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm
|
||||
```
|
||||
|
||||
After the conversion is successful, there are three files in the model save directory:
|
||||
|
||||
```text linenums="1"
|
||||
inference/ser_vi_layoutxlm/
|
||||
├── inference.pdiparams # The parameter file of recognition inference model
|
||||
├── inference.pdiparams.info # The parameter information of recognition inference model, which can be ignored
|
||||
└── inference.pdmodel # The program file of recognition
|
||||
```
|
||||
|
||||
The RE model can be converted to the inference model using the following command.
|
||||
|
||||
```bash linenums="1"
|
||||
# -c Set the training algorithm yml configuration file.
|
||||
# -o Set optional parameters.
|
||||
# Architecture.Backbone.checkpoints Set the training model address.
|
||||
# Global.save_inference_dir Set the address where the converted model will be saved.
|
||||
python3 tools/export_model.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/re_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/re_vi_layoutxlm
|
||||
```
|
||||
|
||||
After the conversion is successful, there are three files in the model save directory:
|
||||
|
||||
```text linenums="1"
|
||||
inference/re_vi_layoutxlm/
|
||||
├── inference.pdiparams # The parameter file of recognition inference model
|
||||
├── inference.pdiparams.info # The parameter information of recognition inference model, which can be ignored
|
||||
└── inference.pdmodel # The program file of recognition
|
||||
```
|
||||
|
||||
### 4.2 Model inference
|
||||
|
||||
The VI layoutxlm model performs reasoning based on the ser task, and can execute the following commands:
|
||||
|
||||
Using the following command to infer the VI-LayoutXLM SER model.
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
python3 kie/predict_kie_token_ser.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--ser_model_dir=../inference/ser_vi_layoutxlm \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx"
|
||||
```
|
||||
|
||||
The visualized result will be saved in `./output`, which is shown as follows.
|
||||
|
||||
|
||||
|
||||
Using the following command to infer the VI-LayoutXLM RE model.
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
python3 kie/predict_kie_token_ser_re.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--re_model_dir=../inference/re_vi_layoutxlm \
|
||||
--ser_model_dir=../inference/ser_vi_layoutxlm \
|
||||
--use_visual_backbone=False \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx"
|
||||
```
|
||||
|
||||
The visualized result will be saved in `./output`, which is shown as follows.
|
||||
|
||||
|
||||
|
||||
## 5. FAQ
|
||||
|
||||
Q1: After the training model is transferred to the inference model, the prediction effect is inconsistent?
|
||||
|
||||
**A**：The problems are mostly caused by inconsistent preprocessing and postprocessing parameters when the trained model predicts and the preprocessing and postprocessing parameters when the inference model predicts. You can compare whether there are differences in preprocessing, postprocessing, and prediction in the configuration files used for training.
|
||||
469
docs/version2.x/ppocr/model_train/kie.md
Normal file
469
docs/version2.x/ppocr/model_train/kie.md
Normal file
@@ -0,0 +1,469 @@
|
||||
---
|
||||
comments: true
|
||||
typora-copy-images-to: images
|
||||
---
|
||||
|
||||
# 关键信息抽取
|
||||
|
||||
本文提供了PaddleOCR关键信息抽取的全流程指南，包括语义实体识别 (Semantic Entity Recognition) 以及关系抽取 (Relation Extraction, RE) 任务的数据准备、模型训练、调优、评估、预测，各个阶段的详细说明。
|
||||
|
||||
## 1. 数据准备
|
||||
|
||||
### 1.1. 准备数据集
|
||||
|
||||
在训练信息抽取相关模型时，PaddleOCR支持以下数据格式。
|
||||
|
||||
- `通用数据` 用于训练以文本文件存储的数据集(SimpleDataSet);
|
||||
|
||||
训练数据的默认存储路径是 `PaddleOCR/train_data`,如果您的磁盘上已有数据集，只需创建软链接至数据集目录：
|
||||
|
||||
```bash linenums="1"
|
||||
# linux and mac os
|
||||
ln -sf <path/to/dataset> <path/to/paddle_ocr>/train_data/dataset
|
||||
# windows
|
||||
mklink /d <path/to/paddle_ocr>/train_data/dataset <path/to/dataset>
|
||||
```
|
||||
|
||||
### 1.2. 自定义数据集
|
||||
|
||||
训练过程中一般包含训练集与验证集，二者数据格式相同，下面介绍如何自定义数据集。
|
||||
|
||||
#### （1）训练集
|
||||
|
||||
建议将训练图片放入同一个文件夹，并用一个文本文件记录图片路径和标签，文本文件里的内容如下:
|
||||
|
||||
```python linenums="1"
|
||||
" 图像文件名 图像标注信息 "
|
||||
zh_train_0.jpg [{"transcription": "汇丰晋信", "label": "other", "points": [[104, 114], [530, 114], [530, 175], [104, 175]], "id": 1, "linking": []}, {"transcription": "受理时间:", "label": "question", "points": [[126, 267], [266, 267], [266, 305], [126, 305]], "id": 7, "linking": [[7, 13]]}, {"transcription": "2020.6.15", "label": "answer", "points": [[321, 239], [537, 239], [537, 285], [321, 285]], "id": 13, "linking": [[7, 13]]}]
|
||||
zh_train_1.jpg [{"transcription": "中国人体器官捐献", "label": "other", "points": [[544, 459], [954, 459], [954, 517], [544, 517]], "id": 1, "linking": []}, {"transcription": ">编号:MC545715483585", "label": "other", "points": [[1462, 470], [2054, 470], [2054, 543], [1462, 543]], "id": 10, "linking": []}, {"transcription": "CHINAORGANDONATION", "label": "other", "points": [[543, 516], [958, 516], [958, 551], [543, 551]], "id": 14, "linking": []}, {"transcription": "中国人体器官捐献志愿登记表", "label": "header", "points": [[635, 793], [1892, 793], [1892, 904], [635, 904]], "id": 18, "linking": []}]
|
||||
...
|
||||
```
|
||||
|
||||
**注意：** 文本文件中默认请将图片路径和图片标签用 `\t` 分割，如用其他方式分割将造成训练报错。
|
||||
|
||||
其中图像标注信息字符串经过json解析之后可以得到一个列表信息，列表中每个元素是一个字典，存储了每个文本行的需要信息，各个字段的含义如下。
|
||||
|
||||
- transcription: 存储了文本行的文字内容
|
||||
- label: 该文本行内容所属的类别
|
||||
- points: 存储文本行的四点位置信息
|
||||
- id: 存储文本行的id信息，用于RE任务的训练
|
||||
- linking: 存储文本行的之间的连接信息，用于RE任务的训练
|
||||
|
||||
#### （2）验证集
|
||||
|
||||
验证集构建方式与训练集相同。
|
||||
|
||||
#### （3）字典文件
|
||||
|
||||
训练集与验证集中的文本行包含标签信息，所有标签的列表存在字典文件中（如`class_list.txt`），字典文件中的每一行表示为一个类别名称。
|
||||
|
||||
以XFUND_zh数据为例，共包含4个类别，字典文件内容如下所示。
|
||||
|
||||
```text linenums="1"
|
||||
OTHER
|
||||
QUESTION
|
||||
ANSWER
|
||||
HEADER
|
||||
```
|
||||
|
||||
在标注文件中，每个标注的文本行内容的`label`字段标注信息需要属于字典内容。
|
||||
|
||||
最终数据集应有如下文件结构：
|
||||
|
||||
```text linenums="1"
|
||||
|-train_data
|
||||
|-data_name
|
||||
|- train.json
|
||||
|- train
|
||||
|- zh_train_0.png
|
||||
|- zh_train_1.jpg
|
||||
| ...
|
||||
|- val.json
|
||||
|- val
|
||||
|- zh_val_0.png
|
||||
|- zh_val_1.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
**注：**
|
||||
|
||||
- 标注文件中的类别信息不区分大小写，如`HEADER`与`header`会被解析为相同的类别id，因此在标注的时候，不能使用小写处理后相同的字符串表示不同的类别。
|
||||
- 在整理标注文件的时候，建议将other这个类别（其他，无需关注的文本行可以标注为other）放在第一行，在解析的时候，会将`other`类别的类别id解析为0，后续不会对该类进行可视化。
|
||||
|
||||
### 1.3. 数据下载
|
||||
|
||||
如果你没有本地数据集，可以从[XFUND](https://github.com/doc-analysis/XFUND)或者[FUNSD](https://guillaumejaume.github.io/FUNSD/)官网下载数据，然后使用XFUND与FUNSD的处理脚本([XFUND](../../ppstructure/kie/tools/trans_xfun_data.py), [FUNSD](../../ppstructure/kie/tools/trans_funsd_label.py))，生成用于PaddleOCR训练的数据格式，并使用公开数据集快速体验关键信息抽取的流程。
|
||||
|
||||
更多关于公开数据集的介绍，请参考[关键信息抽取数据集说明文档](../../datasets/kie_datasets.md)。
|
||||
|
||||
PaddleOCR也支持了关键信息抽取模型的标注，具体使用方法请参考：[PPOCRLabel使用文档](https://github.com/PFCCLab/PPOCRLabel/blob/main/README_ch.md)。
|
||||
|
||||
## 2. 开始训练
|
||||
|
||||
PaddleOCR提供了训练脚本、评估脚本和预测脚本，本节将以 VI-LayoutXLM 多模态预训练模型为例进行讲解。
|
||||
|
||||
> 如果希望使用基于SDMGR的关键信息抽取算法，请参考：[SDMGR使用](../../algorithm/kie/algorithm_kie_sdmgr.md)。
|
||||
|
||||
### 2.1. 启动训练
|
||||
|
||||
如果你没有使用自定义数据集，可以使用PaddleOCR中已经处理好的XFUND_zh数据集进行快速体验。
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir train_data
|
||||
cd train_data
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar && tar -xf XFUND.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
如果不希望训练，直接体验后面的模型评估、预测、动转静、推理的流程，可以下载PaddleOCR中提供的预训练模型，并跳过2.1部分。
|
||||
|
||||
使用下面的方法，下载基于XFUND数据的SER与RE任务预训练模型。
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir pretrained_model
|
||||
cd pretrained_model
|
||||
# 下载并解压SER预训练模型
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar & tar -xf ser_vi_layoutxlm_xfund_pretrained.tar
|
||||
|
||||
# 下载并解压RE预训练模型
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar & tar -xf re_vi_layoutxlm_xfund_pretrained.tar
|
||||
```
|
||||
|
||||
开始训练:
|
||||
|
||||
- 如果您安装的是cpu版本，请将配置文件中的 `use_gpu` 字段修改为false
|
||||
- PaddleOCR在训练时，会默认下载VI-LayoutXLM预训练模型，这里无需预先下载。
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU训练 支持单卡，多卡训练
|
||||
# 训练日志会自动保存到 配置文件中"{Global.save_model_dir}" 下的train.log文件中
|
||||
|
||||
# SER单卡训练
|
||||
python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml
|
||||
|
||||
# SER多卡训练，通过--gpus参数指定卡号
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml
|
||||
|
||||
# RE任务单卡训练
|
||||
python3 tools/train.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml
|
||||
```
|
||||
|
||||
以SER任务为例，正常启动训练后，会看到以下log输出：
|
||||
|
||||
```bash linenums="1"
|
||||
[2022/08/08 16:28:28] ppocr INFO: epoch: [1/200], global_step: 10, lr: 0.000006, loss: 1.871535, avg_reader_cost: 0.28200 s, avg_batch_cost: 0.82318 s, avg_samples: 8.0, ips: 9.71838 samples/s, eta: 0:51:59
|
||||
[2022/08/08 16:28:33] ppocr INFO: epoch: [1/200], global_step: 19, lr: 0.000018, loss: 1.461939, avg_reader_cost: 0.00042 s, avg_batch_cost: 0.32037 s, avg_samples: 6.9, ips: 21.53773 samples/s, eta: 0:37:55
|
||||
[2022/08/08 16:28:39] ppocr INFO: cur metric, precision: 0.11526348939743859, recall: 0.19776657060518732, hmean: 0.14564265817747712, fps: 34.008392345050055
|
||||
[2022/08/08 16:28:45] ppocr INFO: save best model is to ./output/ser_vi_layoutxlm_xfund_zh/best_accuracy
|
||||
[2022/08/08 16:28:45] ppocr INFO: best metric, hmean: 0.14564265817747712, precision: 0.11526348939743859, recall: 0.19776657060518732, fps: 34.008392345050055, best_epoch: 1
|
||||
[2022/08/08 16:28:51] ppocr INFO: save model in ./output/ser_vi_layoutxlm_xfund_zh/latest
|
||||
```
|
||||
|
||||
log 中自动打印如下信息：
|
||||
|
||||
| 字段 | 含义 |
|
||||
| :----: | :------: |
|
||||
| epoch | 当前迭代轮次 |
|
||||
| iter | 当前迭代次数 |
|
||||
| lr | 当前学习率 |
|
||||
| loss | 当前损失函数 |
|
||||
| reader_cost | 当前 batch 数据处理耗时 |
|
||||
| batch_cost | 当前 batch 总耗时 |
|
||||
| samples | 当前 batch 内的样本数 |
|
||||
| ips | 每秒处理图片的数量 |
|
||||
|
||||
PaddleOCR支持训练和评估交替进行, 可以在 `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` 中修改 `eval_batch_step` 设置评估频率，默认每19个iter评估一次。评估过程中默认将最佳hmean模型，保存为 `output/ser_vi_layoutxlm_xfund_zh/best_accuracy/` 。
|
||||
|
||||
如果验证集很大，测试将会比较耗时，建议减少评估次数，或训练完再进行评估。
|
||||
|
||||
**提示：** 可通过 -c 参数选择 `configs/kie/` 路径下的多种模型配置进行训练，PaddleOCR支持的信息抽取算法可以参考[前沿算法列表](../../algorithm/overview.md)。
|
||||
|
||||
如果你希望训练自己的数据集，需要修改配置文件中的数据配置、字典文件以及类别数。
|
||||
|
||||
以 `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` 为例，修改的内容如下所示。
|
||||
|
||||
```yaml linenums="1"
|
||||
|
||||
Architecture:
|
||||
# ...
|
||||
Backbone:
|
||||
name: LayoutXLMForSer
|
||||
pretrained: True
|
||||
mode: vi
|
||||
# 由于采用BIO标注，假设字典中包含n个字段（包含other）时，则类别数为2n-1; 假设字典中包含n个字段（不含other）时，则类别数为2n+1。否则在train过程会报：IndexError: (OutOfRange) label value should less than the shape of axis dimension 。
|
||||
num_classes: &num_classes 7
|
||||
|
||||
PostProcess:
|
||||
name: kieSerTokenLayoutLMPostProcess
|
||||
# 修改字典文件的路径为你自定义的数据集的字典路径
|
||||
class_path: &class_path train_data/XFUND/class_list_xfun.txt
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
# 修改为你自己的训练数据目录
|
||||
data_dir: train_data/XFUND/zh_train/image
|
||||
# 修改为你自己的训练数据标签文件
|
||||
label_file_list:
|
||||
- train_data/XFUND/zh_train/train.json
|
||||
...
|
||||
loader:
|
||||
# 训练时的单卡batch_size
|
||||
batch_size_per_card: 8
|
||||
...
|
||||
|
||||
Eval:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
# 修改为你自己的验证数据目录
|
||||
data_dir: train_data/XFUND/zh_val/image
|
||||
# 修改为你自己的验证数据标签文件
|
||||
label_file_list:
|
||||
- train_data/XFUND/zh_val/val.json
|
||||
...
|
||||
loader:
|
||||
# 验证时的单卡batch_size
|
||||
batch_size_per_card: 8
|
||||
```
|
||||
|
||||
**注意，预测/评估时的配置文件请务必与训练一致。**
|
||||
|
||||
### 2.2. 断点训练
|
||||
|
||||
如果训练程序中断，如果希望加载训练中断的模型从而恢复训练，可以通过指定`Architecture.Backbone.checkpoints`指定要加载的模型路径：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy
|
||||
```
|
||||
|
||||
**注意**：
|
||||
|
||||
- `Architecture.Backbone.checkpoints`的优先级高于`Architecture.Backbone.pretrained`，需要加载之前训练好的训练模型进行模型微调、恢复训练、模型评估时，需要使用`Architecture.Backbone.checkpoints`指定模型参数路径；如果需要使用默认提供的通用预训练模型进行训练，则需要指定`Architecture.Backbone.pretrained`为`True`，同时指定`Architecture.Backbone.checkpoints`为空（`null`）。
|
||||
- LayoutXLM系列模型均是调用了PaddleNLP中的预训练模型，模型加载与保存的逻辑与PaddleNLP基本一致，因此在这里不需要指定`Global.pretrained_model`或者`Global.checkpoints`参数；此外，LayoutXLM系列模型的蒸馏训练目前不支持断点训练。
|
||||
|
||||
### 2.3. 混合精度训练
|
||||
|
||||
coming soon!
|
||||
|
||||
### 2.4. 分布式训练
|
||||
|
||||
多机多卡训练时，通过 `--ips` 参数设置使用的机器IP地址，通过 `--gpus` 参数设置使用的GPU ID：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml
|
||||
```
|
||||
|
||||
**注意:** （1）采用多机多卡训练时，需要替换上面命令中的ips值为您机器的地址，机器之间需要能够相互ping通；（2）训练时需要在多个机器上分别启动命令。查看机器ip地址的命令为`ifconfig`；（3）更多关于分布式训练的性能优势等信息，请参考：[分布式训练教程](../blog/distributed_training.md)。
|
||||
|
||||
### 2.5. 知识蒸馏训练
|
||||
|
||||
PaddleOCR支持了基于U-DML知识蒸馏的关键信息抽取模型训练过程，配置文件请参考：[ser_vi_layoutxlm_xfund_zh_udml.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)，更多关于知识蒸馏的说明文档请参考：[知识蒸馏说明文档](../model_compress/knowledge_distillation.md)。
|
||||
|
||||
**注意**： PaddleOCR中LayoutXLM系列关键信息抽取模型的保存与加载逻辑与PaddleNLP保持一致，因此在蒸馏的过程中仅保存了学生模型的参数，如果希望使用保存的模型进行评估，需要使用学生模型的配置（上面的蒸馏文件对应的学生模型为[ser_vi_layoutxlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml)）
|
||||
|
||||
### 2.6. 其他训练环境
|
||||
|
||||
- Windows GPU/CPU
|
||||
在Windows平台上与Linux平台略有不同:
|
||||
Windows平台只支持`单卡`的训练与预测，指定GPU进行训练`set CUDA_VISIBLE_DEVICES=0`
|
||||
在Windows平台，DataLoader只支持单进程模式，因此需要设置 `num_workers` 为0;
|
||||
|
||||
- macOS
|
||||
不支持GPU模式，需要在配置文件中设置`use_gpu`为False，其余训练评估预测命令与Linux GPU完全相同。
|
||||
|
||||
- Linux DCU
|
||||
DCU设备上运行需要设置环境变量 `export HIP_VISIBLE_DEVICES=0,1,2,3`，其余训练评估预测命令与Linux GPU完全相同。
|
||||
|
||||
## 3. 模型评估与预测
|
||||
|
||||
### 3.1. 指标评估
|
||||
|
||||
训练中模型参数默认保存在`Global.save_model_dir`目录下。在评估指标时，需要设置`Architecture.Backbone.checkpoints`指向保存的参数文件。评估数据集可以通过 `configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` 修改Eval中的 `label_file_path` 设置。
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU 评估， Global.checkpoints 为待测权重
|
||||
python3 tools/eval.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy
|
||||
```
|
||||
|
||||
会输出以下信息，打印出precision、recall、hmean等信息。
|
||||
|
||||
```bash linenums="1"
|
||||
[2022/08/09 07:59:28] ppocr INFO: metric eval ***************
|
||||
[2022/08/09 07:59:28] ppocr INFO: precision:0.697476609016161
|
||||
[2022/08/09 07:59:28] ppocr INFO: recall:0.8861671469740634
|
||||
[2022/08/09 07:59:28] ppocr INFO: hmean:0.7805806758686339
|
||||
[2022/08/09 07:59:28] ppocr INFO: fps:17.367364606899105
|
||||
```
|
||||
|
||||
### 3.2. 测试信息抽取结果
|
||||
|
||||
使用 PaddleOCR 训练好的模型，可以通过以下脚本进行快速预测。
|
||||
|
||||
默认预测的图片存储在 `infer_img` 里，通过 `-o Architecture.Backbone.checkpoints` 加载训练好的参数文件：
|
||||
|
||||
根据配置文件中设置的 `save_model_dir` 和 `save_epoch_step` 字段，会有以下几种参数被保存下来：
|
||||
|
||||
```text linenums="1"
|
||||
output/ser_vi_layoutxlm_xfund_zh/
|
||||
├── best_accuracy
|
||||
├── metric.states
|
||||
├── model_config.json
|
||||
├── model_state.pdparams
|
||||
├── best_accuracy.pdopt
|
||||
├── config.yml
|
||||
├── train.log
|
||||
├── latest
|
||||
├── metric.states
|
||||
├── model_config.json
|
||||
├── model_state.pdparams
|
||||
├── latest.pdopt
|
||||
```
|
||||
|
||||
其中 best_accuracy.*是评估集上的最优模型；latest.* 是最新保存的一个模型。
|
||||
|
||||
预测使用的配置文件必须与训练一致，如您通过 `python3 tools/train.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml` 完成了模型的训练过程。
|
||||
|
||||
您可以使用如下命令进行中文模型预测。
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg
|
||||
```
|
||||
|
||||
预测图片如下所示，图片会存储在`Global.save_res_path`路径中。
|
||||
|
||||
|
||||
|
||||
预测过程中，默认会加载PP-OCRv3的检测识别模型，用于OCR的信息抽取，如果希望加载预先获取的OCR结果，可以使用下面的方式进行预测，指定`Global.infer_img`为标注文件，其中包含图片路径以及OCR信息，同时指定`Global.infer_mode`为False，表示此时不使用OCR预测引擎。
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer_kie_token_ser.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False
|
||||
```
|
||||
|
||||
对于上述图片，如果使用标注的OCR结果进行信息抽取，预测结果如下。
|
||||
|
||||
|
||||
|
||||
可以看出，部分检测框信息更加准确，但是整体信息抽取识别结果基本一致。
|
||||
|
||||
在RE任务模型预测时，需要先给出模型SER结果，因此需要同时加载SER的配置文件与模型权重，示例如下。
|
||||
|
||||
```bash linenums="1"
|
||||
python3 ./tools/infer_kie_token_ser_re.py \
|
||||
-c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/best_accuracy/ \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/image/ \
|
||||
-c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o_ser Architecture.Backbone.checkpoints=pretrain_models/ \
|
||||
ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/
|
||||
```
|
||||
|
||||
预测结果如下所示。
|
||||
|
||||
|
||||
|
||||
如果希望使用标注或者预先获取的OCR信息进行关键信息抽取，同上，可以指定`Global.infer_mode`为False，指定`Global.infer_img`为标注文件。
|
||||
|
||||
```bash linenums="1"
|
||||
python3 ./tools/infer_kie_token_ser_re.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./pretrain_models/re_vi_layoutxlm_udml_xfund_zh/re_layoutxlm_xfund_zh_v4_udml/best_accuracy/ Global.infer_img=./train_data/XFUND/zh_val/val.json Global.infer_mode=False -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o_ser Architecture.Backbone.checkpoints=pretrain_models/ser_vi_layoutxlm_udml_xfund_zh/best_accuracy/
|
||||
```
|
||||
|
||||
其中`c_ser`表示SER的配置文件，`o_ser` 后面需要加上待修改的SER模型与配置文件，如预训练权重等。
|
||||
|
||||
预测结果如下所示。
|
||||
|
||||
|
||||
|
||||
可以看出，直接使用标注的OCR结果的RE预测结果要更加准确一些。
|
||||
|
||||
## 4. 模型导出与预测
|
||||
|
||||
### 4.1 模型导出
|
||||
|
||||
inference 模型（`paddle.jit.save`保存的模型）
|
||||
一般是模型训练，把模型结构和模型参数保存在文件中的固化模型，多用于预测部署场景。
|
||||
训练过程中保存的模型是checkpoints模型，保存的只有模型的参数，多用于恢复训练等。
|
||||
与checkpoints模型相比，inference 模型会额外保存模型的结构信息，在预测部署、加速推理上性能优越，灵活方便，适合于实际系统集成。
|
||||
|
||||
信息抽取模型中的SER任务转inference模型步骤如下：
|
||||
|
||||
```bash linenums="1"
|
||||
# -c 后面设置训练算法的yml配置文件
|
||||
# -o 配置可选参数
|
||||
# Architecture.Backbone.checkpoints 参数设置待转换的训练模型地址
|
||||
# Global.save_inference_dir 参数设置转换的模型将保存的地址
|
||||
|
||||
python3 tools/export_model.py -c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/ser_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/ser_vi_layoutxlm
|
||||
```
|
||||
|
||||
转换成功后，在目录下有三个文件：
|
||||
|
||||
```text linenums="1"
|
||||
inference/ser_vi_layoutxlm/
|
||||
├── inference.pdiparams # inference模型的参数文件
|
||||
├── inference.pdiparams.info # inference模型的参数信息，可忽略
|
||||
└── inference.pdmodel # inference模型的模型结构文件
|
||||
```
|
||||
|
||||
信息抽取模型中的RE任务转inference模型步骤如下：
|
||||
|
||||
```bash linenums="1"
|
||||
# -c 后面设置训练算法的yml配置文件
|
||||
# -o 配置可选参数
|
||||
# Architecture.Backbone.checkpoints 参数设置待转换的训练模型地址
|
||||
# Global.save_inference_dir 参数设置转换的模型将保存的地址
|
||||
|
||||
python3 tools/export_model.py -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml -o Architecture.Backbone.checkpoints=./output/re_vi_layoutxlm_xfund_zh/best_accuracy Global.save_inference_dir=./inference/re_vi_layoutxlm
|
||||
```
|
||||
|
||||
转换成功后，在目录下有三个文件：
|
||||
|
||||
```text linenums="1"
|
||||
inference/re_vi_layoutxlm/
|
||||
├── inference.pdiparams # inference模型的参数文件
|
||||
├── inference.pdiparams.info # inference模型的参数信息，可忽略
|
||||
└── inference.pdmodel # inference模型的模型结构文件
|
||||
```
|
||||
|
||||
### 4.2 模型推理
|
||||
|
||||
VI-LayoutXLM模型基于SER任务进行推理，可以执行如下命令：
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
python3 kie/predict_kie_token_ser.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--ser_model_dir=../inference/ser_vi_layoutxlm \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx"
|
||||
```
|
||||
|
||||
可视化SER结果结果默认保存到`./output`文件夹里面。结果示例如下：
|
||||
|
||||
|
||||
|
||||
VI-LayoutXLM模型基于RE任务进行推理，可以执行如下命令：
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
python3 kie/predict_kie_token_ser_re.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--re_model_dir=../inference/re_vi_layoutxlm \
|
||||
--ser_model_dir=../inference/ser_vi_layoutxlm \
|
||||
--use_visual_backbone=False \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx"
|
||||
```
|
||||
|
||||
RE可视化结果默认保存到`./output`文件夹里面，结果示例如下：
|
||||
|
||||
|
||||
|
||||
## 5. FAQ
|
||||
|
||||
Q1: 训练模型转inference 模型之后预测效果不一致？
|
||||
|
||||
**A**：该问题多是trained model预测时候的预处理、后处理参数和inference model预测的时候的预处理、后处理参数不一致导致的。可以对比训练使用的配置文件中的预处理、后处理和预测时是否存在差异。
|
||||
559
docs/version2.x/ppocr/model_train/recognition.en.md
Normal file
559
docs/version2.x/ppocr/model_train/recognition.en.md
Normal file
@@ -0,0 +1,559 @@
|
||||
---
|
||||
comments: true
|
||||
typora-copy-images-to: images
|
||||
---
|
||||
|
||||
# Text Recognition
|
||||
|
||||
This article provides a comprehensive guide for the PaddleOCR text recognition task, covering the entire workflow including data preparation, model training, fine-tuning, evaluation, and prediction, with detailed explanations for each phase.
|
||||
|
||||
## 1. Data Preparation
|
||||
|
||||
### 1.1. Prepare the Dataset
|
||||
|
||||
PaddleOCR supports two data formats:
|
||||
|
||||
- `lmdb`: Used for training with datasets stored in LMDB format (LMDBDataSet);
|
||||
- `General Data`: Used for training with datasets stored in text files (SimpleDataSet);
|
||||
|
||||
The default storage path for training data is `PaddleOCR/train_data`. If you already have a dataset on your disk, simply create a symbolic link to the dataset directory:
|
||||
|
||||
```bash
|
||||
# Linux and macOS
|
||||
ln -sf <path/to/dataset> <path/to/paddle_ocr>/train_data/dataset
|
||||
# Windows
|
||||
mklink /d <path/to/paddle_ocr>/train_data/dataset <path/to/dataset>
|
||||
```
|
||||
|
||||
### 1.2. Custom Dataset
|
||||
|
||||
Here, we will use a general dataset as an example to explain how to prepare the dataset:
|
||||
|
||||
- Training Dataset
|
||||
|
||||
It is recommended to place the training images in the same folder and record the image paths and labels in a txt file (`rec_gt_train.txt`). The content of the txt file should be as follows:
|
||||
|
||||
**Note:** In the txt file, please use `\t` to separate the image path and the label. Using any other separator will cause errors during training.
|
||||
|
||||
```text
|
||||
" Image Filename Image Label "
|
||||
|
||||
train_data/rec/train/word_001.jpg Simple and reliable
|
||||
train_data/rec/train/word_002.jpg Making the complex world simpler with technology
|
||||
...
|
||||
```
|
||||
|
||||
The final structure of the training dataset should look like this:
|
||||
|
||||
```text
|
||||
|-train_data
|
||||
|-rec
|
||||
|- rec_gt_train.txt
|
||||
|- train
|
||||
|- word_001.png
|
||||
|- word_002.jpg
|
||||
|- word_003.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
In addition to the single-image-per-line format described above, PaddleOCR also supports training on data augmented offline. To avoid sampling the same sample multiple times in the same batch, we can list image paths with the same label on one line. During training, PaddleOCR will randomly select one image from the list. The corresponding format of the label file is as follows:
|
||||
|
||||
```text
|
||||
["11.jpg", "12.jpg"] Simple and reliable
|
||||
["21.jpg", "22.jpg", "23.jpg"] Making the complex world simpler with technology
|
||||
3.jpg ocr
|
||||
```
|
||||
|
||||
In the above example, both "11.jpg" and "12.jpg" have the same label `Simple and reliable`. During training, one of these images will be randomly chosen for training.
|
||||
|
||||
- Validation Dataset
|
||||
|
||||
Similarly to the training dataset, the validation dataset should also provide a folder containing all the images (test) and a `rec_gt_test.txt` file. The structure of the validation dataset is as follows:
|
||||
|
||||
```text
|
||||
|-train_data
|
||||
|-rec
|
||||
|- rec_gt_test.txt
|
||||
|- test
|
||||
|- word_001.jpg
|
||||
|- word_002.jpg
|
||||
|- word_003.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
### 1.3. Data Download
|
||||
|
||||
- ICDAR2015
|
||||
|
||||
If you don't have a dataset locally, you can download the [ICDAR2015](http://rrc.cvc.uab.es/?ch=4&com=downloads) dataset from the official website for quick testing. You can also refer to [DTRB](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) to download the LMDB formatted dataset needed for benchmarking.
|
||||
|
||||
If you're using the public ICDAR2015 dataset, PaddleOCR provides a label file for training the ICDAR2015 dataset. You can download it as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
# Training set label
|
||||
wget -P ./train_data/ic15_data https://paddleocr.bj.bcebos.com/dataset/rec_gt_train.txt
|
||||
# Test Set Label
|
||||
wget -P ./train_data/ic15_data https://paddleocr.bj.bcebos.com/dataset/rec_gt_test.txt
|
||||
```
|
||||
|
||||
PaddleOCR also provides a data format conversion script, which can convert ICDAR official website label to a data format
|
||||
supported by PaddleOCR. The data conversion tool is in `ppocr/utils/gen_label.py`, here is the training set as an example:
|
||||
|
||||
```bash linenums="1"
|
||||
# convert the official gt to rec_gt_label.txt
|
||||
python gen_label.py --mode="rec" --input_path="{path/of/origin/label}" --output_label="rec_gt_label.txt"
|
||||
```
|
||||
|
||||
The data format is as follows, (a) is the original picture, (b) is the Ground Truth text file corresponding to each picture:
|
||||
|
||||
|
||||
|
||||
- Multilingual Datasets
|
||||
|
||||
The multi-language model training method is the same as the Chinese model. The training data set is 100w synthetic data. A small amount of fonts and test data can be downloaded using the following two methods.
|
||||
|
||||
- [Baidu Netdisk](https://pan.baidu.com/s/1bS_u207Rm7YbY33wOECKDA) ,Extraction code:frgi.
|
||||
- [Google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view)
|
||||
|
||||
### 1.4. Dictionary
|
||||
|
||||
Finally, a dictionary ({word_dict_name}.txt) needs to be provided so that when the model is trained, all the characters that appear can be mapped to the dictionary index.
|
||||
|
||||
Therefore, the dictionary needs to contain all the characters that you want to be recognized correctly. {word_dict_name}.txt needs to be written in the following format and saved in the `utf-8` encoding format:
|
||||
|
||||
```text linenums="1"
|
||||
l
|
||||
d
|
||||
a
|
||||
d
|
||||
r
|
||||
n
|
||||
```
|
||||
|
||||
In `word_dict.txt`, there is a single word in each line, which maps characters and numeric indexes together, e.g "and" will be mapped to [2 5 1]
|
||||
|
||||
PaddleOCR includes several built-in dictionaries that can be used as needed:
|
||||
|
||||
- `ppocr/utils/ppocr_keys_v1.txt`: A Chinese dictionary containing 6623 characters.
|
||||
- `ppocr/utils/ic15_dict.txt`: An English dictionary containing 36 characters.
|
||||
- `ppocr/utils/dict/french_dict.txt`: A French dictionary containing 118 characters.
|
||||
- `ppocr/utils/dict/japan_dict.txt`: A Japanese dictionary containing 4399 characters.
|
||||
- `ppocr/utils/dict/korean_dict.txt`: A Korean dictionary containing 3636 characters.
|
||||
- `ppocr/utils/dict/german_dict.txt`: A German dictionary containing 131 characters.
|
||||
- `ppocr/utils/en_dict.txt`: An English dictionary containing 96 characters.
|
||||
|
||||
Currently, the multilingual models are still in the demo stage, and we are continuously improving the models and adding new languages. **We highly welcome you to provide dictionaries and fonts for other languages**. If you are willing, you can submit your dictionary files to the [dict](../../ppocr/utils/dict) directory, and we will credit you in the repo.
|
||||
To customize the dict file, please modify the `character_dict_path` field in `configs/rec/rec_icdar15_train.yml`.
|
||||
|
||||
- Custom Dictionary
|
||||
|
||||
If you need to customize dic file, please add character_dict_path field in configs/rec/rec_icdar15_train.yml to point to your dictionary path. And set character_type to ch.
|
||||
|
||||
### 1.5. Add Space Category
|
||||
|
||||
To support recognition of the "space" category, set the `use_space_char` field in the YAML file to `True`.
|
||||
|
||||
### 1.6. Data Augmentation
|
||||
|
||||
PaddleOCR provides a variety of data augmentation methods. All the augmentation methods are enabled by default.
|
||||
|
||||
The default perturbation methods are: cvtColor, blur, jitter, Gasuss noise, random crop, perspective, color reverse, TIA augmentation.
|
||||
|
||||
Each disturbance method is selected with a 40% probability during the training process. For specific code implementation, please refer to: [rec_img_aug.py](../../ppocr/data/imaug/rec_img_aug.py)
|
||||
|
||||
## 2. Training
|
||||
|
||||
PaddleOCR provides training scripts, evaluation scripts, and prediction scripts. This section will take the PP-OCRv3 English recognition model as an example:
|
||||
|
||||
### 2.1. Start Training
|
||||
|
||||
First download the pretrain model, you can download the trained model to finetune on the icdar2015 data:
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/
|
||||
# Download the pre-trained model of en_PP-OCRv3
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_train.tar
|
||||
# Decompress model parameters
|
||||
cd pretrain_models
|
||||
tar -xf en_PP-OCRv3_rec_train.tar && rm -rf en_PP-OCRv3_rec_train.tar
|
||||
```
|
||||
|
||||
Start training:
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU training Support single card and multi-card training
|
||||
# Training icdar15 English data and The training log will be automatically saved as train.log under "{save_model_dir}"
|
||||
|
||||
#specify the single card training(Long training time, not recommended)
|
||||
python3 tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model=en_PP-OCRv3_rec_train/best_accuracy
|
||||
|
||||
#specify the card number through --gpus
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model=en_PP-OCRv3_rec_train/best_accuracy
|
||||
```
|
||||
|
||||
PaddleOCR supports alternating training and evaluation. You can modify `eval_batch_step` in `configs/rec/rec_icdar15_train.yml` to set the evaluation frequency. By default, it is evaluated every 500 iter and the best acc model is saved under `output/rec_CRNN/best_accuracy` during the evaluation process.
|
||||
|
||||
If the evaluation set is large, the test will be time-consuming. It is recommended to reduce the number of evaluations, or evaluate after training.
|
||||
|
||||
- Tip: You can use the `-c` parameter to select multiple model configurations under the `configs/rec/` path for training. The recognition algorithms supported at [rec_algorithm](../../algorithm/overview.en.md):
|
||||
|
||||
For training Chinese data, it is recommended to use
|
||||
[PP-OCRv3_mobile_rec_distillation.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/rec/PP-OCRv3/PP-OCRv3_mobile_rec_distillation.yml). If you want to try the result of other algorithms on the Chinese data set, please refer to the following instructions to modify the configuration file:
|
||||
|
||||
Take `PP-OCRv3_mobile_rec_distillation.yml` as an example:
|
||||
|
||||
```yaml linenums="1"
|
||||
Global:
|
||||
...
|
||||
# Add a custom dictionary, such as modify the dictionary, please point the path to the new dictionary
|
||||
character_dict_path: ppocr/utils/ppocr_keys_v1.txt
|
||||
# Modify character type
|
||||
...
|
||||
# Whether to recognize spaces
|
||||
use_space_char: True
|
||||
|
||||
|
||||
Optimizer:
|
||||
...
|
||||
# Add learning rate decay strategy
|
||||
lr:
|
||||
name: Cosine
|
||||
learning_rate: 0.001
|
||||
...
|
||||
|
||||
...
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
# Type of dataset，we support LMDBDataSet and SimpleDataSet
|
||||
name: SimpleDataSet
|
||||
# Path of dataset
|
||||
data_dir: ./train_data/
|
||||
# Path of train list
|
||||
label_file_list: ["./train_data/train_list.txt"]
|
||||
transforms:
|
||||
...
|
||||
- RecResizeImg:
|
||||
# Modify image_shape to fit long text
|
||||
image_shape: [3, 48, 320]
|
||||
...
|
||||
loader:
|
||||
...
|
||||
# Train batch_size for Single card
|
||||
batch_size_per_card: 256
|
||||
...
|
||||
|
||||
Eval:
|
||||
dataset:
|
||||
# Type of dataset，we support LMDBDataSet and SimpleDataSet
|
||||
name: SimpleDataSet
|
||||
# Path of dataset
|
||||
data_dir: ./train_data
|
||||
# Path of eval list
|
||||
label_file_list: ["./train_data/val_list.txt"]
|
||||
transforms:
|
||||
...
|
||||
- RecResizeImg:
|
||||
# Modify image_shape to fit long text
|
||||
image_shape: [3, 48, 320]
|
||||
...
|
||||
loader:
|
||||
# Eval batch_size for Single card
|
||||
batch_size_per_card: 256
|
||||
...
|
||||
```
|
||||
|
||||
**Note that the configuration file for prediction/evaluation must be consistent with the training.**
|
||||
|
||||
### 2.2 Load Trained Model and Continue Training
|
||||
|
||||
If you expect to load trained model and continue the training again, you can specify the parameter `Global.checkpoints` as the model path to be loaded.
|
||||
|
||||
For example:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/rec/rec_icdar15_train.yml -o Global.checkpoints=./your/trained/model
|
||||
```
|
||||
|
||||
**Note**: The priority of `Global.checkpoints` is higher than that of `Global.pretrained_model`, that is, when two parameters are specified at the same time, the model specified by `Global.checkpoints` will be loaded first. If the model path specified by `Global.checkpoints` is wrong, the one specified by `Global.pretrained_model` will be loaded.
|
||||
|
||||
### 2.3 Training with New Backbone
|
||||
|
||||
The network part completes the construction of the network, and PaddleOCR divides the network into four parts, which are under [ppocr/modeling](../../ppocr/modeling). The data entering the network will pass through these four parts in sequence(transforms->backbones->
|
||||
necks->heads).
|
||||
|
||||
```bash linenums="1"
|
||||
├── architectures # Code for building network
|
||||
├── transforms # Image Transformation Module
|
||||
├── backbones # Feature extraction module
|
||||
├── necks # Feature enhancement module
|
||||
└── heads # Output module
|
||||
```
|
||||
|
||||
If the Backbone to be replaced has a corresponding implementation in PaddleOCR, you can directly modify the parameters in the `Backbone` part of the configuration yml file.
|
||||
|
||||
However, if you want to use a new Backbone, an example of replacing the backbones is as follows:
|
||||
|
||||
1. Create a new file under the [ppocr/modeling/backbones](../../ppocr/modeling/backbones) folder, such as my_backbone.py.
|
||||
2. Add code in the my_backbone.py file, the sample code is as follows:
|
||||
|
||||
```python linenums="1"
|
||||
import paddle
|
||||
import paddle.nn as nn
|
||||
import paddle.nn.functional as F
|
||||
|
||||
|
||||
class MyBackbone(nn.Layer):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(MyBackbone, self).__init__()
|
||||
# your init code
|
||||
self.conv = nn.xxxx
|
||||
|
||||
def forward(self, inputs):
|
||||
# your network forward
|
||||
y = self.conv(inputs)
|
||||
return y
|
||||
```
|
||||
|
||||
3. Import the added module in the [ppocr/modeling/backbones/\__init\__.py](https://github.com/PaddlePaddle/PaddleOCR/blob/main/ppocr/modeling/backbones/__init__.py) file.
|
||||
|
||||
After adding the four-part modules of the network, you only need to configure them in the configuration file to use, such as:
|
||||
|
||||
```yaml linenums="1"
|
||||
Backbone:
|
||||
name: MyBackbone
|
||||
args1: args1
|
||||
```
|
||||
|
||||
**NOTE**: More details about replace Backbone and other module can be found in [doc](../../algorithm/add_new_algorithm.en.md).
|
||||
|
||||
### 2.4. Mixed Precision Training
|
||||
|
||||
If you want to speed up your training further, you can use [Auto Mixed Precision Training](https://www.paddlepaddle.org.cn/documentation/docs/en/guides/performance_improving/amp_en.html), taking a single machine and a single gpu as an example, the commands are as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/rec/rec_icdar15_train.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/rec_mv3_none_bilstm_ctc_v2.0_train \
|
||||
Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True
|
||||
```
|
||||
|
||||
### 2.5. Distributed Training
|
||||
|
||||
During multi-machine multi-gpu training, use the `--ips` parameter to set the used machine IP address, and the `--gpus` parameter to set the used GPU ID:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_icdar15_train.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/rec_mv3_none_bilstm_ctc_v2.0_train
|
||||
```
|
||||
|
||||
**Note:**
|
||||
1. When using multi-machine and multi-gpu training, you need to replace the ips value in the above command with the address of your machine, and the machines need to be able to ping each other.
|
||||
2. Training needs to be launched separately on multiple machines. The command to view the ip address of the machine is `ifconfig`.
|
||||
3. For more details about the distributed training speedup ratio, please refer to [Distributed Training Tutorial](../blog/distributed_training.en.md).
|
||||
|
||||
### 2.6. Training with Knowledge Distillation
|
||||
|
||||
Knowledge distillation is supported in PaddleOCR for text recognition training process. For more details, please refer to [doc](../model_compress/knowledge_distillation.en.md).
|
||||
|
||||
### 2.7. Multi-Language Model Training
|
||||
|
||||
Currently, the multi-language algorithms supported by PaddleOCR are:
|
||||
|
||||
| Configuration file | Algorithm name | backbone | trans | seq | pred | language |
|
||||
| :--------: | :-------: | :-------: | :-------: | :-----: | :-----: | :-----: |
|
||||
| rec_chinese_cht_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | chinese traditional |
|
||||
| rec_en_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | English(Case sensitive) |
|
||||
| rec_french_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | French |
|
||||
| rec_ger_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | German |
|
||||
| rec_japan_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Japanese |
|
||||
| rec_korean_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Korean |
|
||||
| rec_latin_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Latin |
|
||||
| rec_arabic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | arabic |
|
||||
| rec_cyrillic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | cyrillic |
|
||||
| rec_devanagari_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | devanagari |
|
||||
|
||||
For more supported languages, please refer to : [Multi-language model](../blog/multi_languages.en.md)
|
||||
|
||||
If you want to finetune on the basis of the existing model effect, please refer to the following instructions to modify the configuration file:
|
||||
|
||||
Take `rec_french_lite_train` as an example:
|
||||
|
||||
```yaml linenums="1"
|
||||
Global:
|
||||
...
|
||||
# Add a custom dictionary, such as modify the dictionary, please point the path to the new dictionary
|
||||
character_dict_path: ./ppocr/utils/dict/french_dict.txt
|
||||
...
|
||||
# Whether to recognize spaces
|
||||
use_space_char: True
|
||||
|
||||
...
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
# Type of dataset，we support LMDBDataSet and SimpleDataSet
|
||||
name: SimpleDataSet
|
||||
# Path of dataset
|
||||
data_dir: ./train_data/
|
||||
# Path of train list
|
||||
label_file_list: ["./train_data/french_train.txt"]
|
||||
...
|
||||
|
||||
Eval:
|
||||
dataset:
|
||||
# Type of dataset，we support LMDBDataSet and SimpleDataSet
|
||||
name: SimpleDataSet
|
||||
# Path of dataset
|
||||
data_dir: ./train_data
|
||||
# Path of eval list
|
||||
label_file_list: ["./train_data/french_val.txt"]
|
||||
...
|
||||
```
|
||||
|
||||
### 2.8 Training on other platform(Windows/macOS/Linux DCU)
|
||||
|
||||
- Windows GPU/CPU
|
||||
The Windows platform is slightly different from the Linux platform:
|
||||
Windows platform only supports `single gpu` training and inference, specify GPU for training `set CUDA_VISIBLE_DEVICES=0`
|
||||
On the Windows platform, DataLoader only supports single-process mode, so you need to set `num_workers` to 0;
|
||||
|
||||
- macOS
|
||||
GPU mode is not supported, you need to set `use_gpu` to False in the configuration file, and the rest of the training evaluation prediction commands are exactly the same as Linux GPU.
|
||||
|
||||
- Linux DCU
|
||||
Running on a DCU device requires setting the environment variable `export HIP_VISIBLE_DEVICES=0,1,2,3`, and the rest of the training and evaluation prediction commands are exactly the same as the Linux GPU.
|
||||
|
||||
## 2.9 Fine-tuning
|
||||
|
||||
In actual use, it is recommended to load the official pre-trained model and fine-tune it in your own data set. For the fine-tuning method of the recognition model, please refer to: [Model Fine-tuning Tutorial](./finetune.en.md).
|
||||
|
||||
## 3. Evaluation and Test
|
||||
|
||||
### 3.1. Evaluation
|
||||
|
||||
The model parameters during training are saved in the `Global.save_model_dir` directory by default. When evaluating indicators, you need to set `Global.checkpoints` to point to the saved parameter file. The evaluation dataset can be set by modifying the `Eval.dataset.label_file_list` field in the `configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml` file.
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU evaluation, Global.checkpoints is the weight to be tested
|
||||
python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.checkpoints={path/to/weights}/best_accuracy
|
||||
```
|
||||
|
||||
### 3.2 Test
|
||||
|
||||
Using the model trained by paddleocr, you can quickly get prediction through the following script.
|
||||
|
||||
The default prediction picture is stored in `infer_img`, and the trained weight is specified via `-o Global.checkpoints`:
|
||||
|
||||
According to the `save_model_dir` and `save_epoch_step` fields set in the configuration file, the following parameters will be saved:
|
||||
|
||||
```text linenums="1"
|
||||
output/rec/
|
||||
├── best_accuracy.pdopt
|
||||
├── best_accuracy.pdparams
|
||||
├── best_accuracy.states
|
||||
├── config.yml
|
||||
├── iter_epoch_3.pdopt
|
||||
├── iter_epoch_3.pdparams
|
||||
├── iter_epoch_3.states
|
||||
├── latest.pdopt
|
||||
├── latest.pdparams
|
||||
├── latest.states
|
||||
└── train.log
|
||||
```
|
||||
|
||||
Among them, best_accuracy._is the best model on the evaluation set; iter_epoch_x._ is the model saved at intervals of `save_epoch_step`; latest.* is the model of the last epoch.
|
||||
|
||||
```bash linenums="1"
|
||||
# Predict English results
|
||||
python3 tools/infer_rec.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
|
||||
```
|
||||
|
||||
Input image:
|
||||
|
||||
|
||||
|
||||
Get the prediction result of the input image:
|
||||
|
||||
```bash linenums="1"
|
||||
infer_img: doc/imgs_words/en/word_1.png
|
||||
result: ('joint', 0.9998967)
|
||||
```
|
||||
|
||||
The configuration file used for prediction must be consistent with the training. For example, you completed the training of the Chinese model with `python3 tools/train.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml`, you can use the following command to predict the Chinese model:
|
||||
|
||||
```bash linenums="1"
|
||||
# Predict Chinese results
|
||||
python3 tools/infer_rec.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/ch/word_1.jpg
|
||||
```
|
||||
|
||||
Input image:
|
||||
|
||||
|
||||
|
||||
Get the prediction result of the input image:
|
||||
|
||||
```bash linenums="1"
|
||||
infer_img: doc/imgs_words/ch/word_1.jpg
|
||||
result: ('韩国小馆', 0.997218)
|
||||
```
|
||||
|
||||
### 4. Model Export and Prediction
|
||||
|
||||
**Inference Model** (saved using `paddle.jit.save`)
|
||||
|
||||
The inference model is a "frozen" version of the model, where both the model structure and model parameters are saved in a file. It is typically used for prediction and deployment scenarios.
|
||||
In contrast, the **checkpoint model** only saves the model's parameters and is mostly used for training resumption, etc. Compared to the checkpoint model, the inference model also includes the model structure information, which makes it more efficient for deployment, inference acceleration, and flexible integration with systems.
|
||||
|
||||
The process of converting a recognition model to an inference model is similar to the detection model conversion, as shown below:
|
||||
|
||||
```bash linenums="1"
|
||||
# Enable old IR mode
|
||||
export FLAGS_enable_pir_api=0
|
||||
|
||||
# -c Set the training algorithm yml configuration file
|
||||
# -o Set optional parameters
|
||||
# Global.pretrained_model parameter Set the training model address to be converted without adding the file suffix .pdmodel, .pdopt or .pdparams.
|
||||
# Global.save_inference_dir Set the address where the converted model will be saved.
|
||||
|
||||
python3 tools/export_model.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model=en_PP-OCRv3_rec_train/best_accuracy Global.save_inference_dir=./inference/en_PP-OCRv3_mobile_rec/
|
||||
```
|
||||
|
||||
If you have a model trained on your own dataset with a different dictionary file, please make sure that you modify the `character_dict_path` in the configuration file to your dictionary file path.
|
||||
|
||||
After the conversion is successful, there are three files in the model save directory:
|
||||
|
||||
```text linenums="1"
|
||||
inference/en_PP-OCRv3_mobile_rec/
|
||||
├── inference.pdiparams # The parameter file of recognition inference model
|
||||
├── inference.pdiparams.info # The parameter information of recognition inference model, which can be ignored
|
||||
└── inference.pdmodel # The program file of recognition model
|
||||
```
|
||||
|
||||
**Note**: If you need to store the model in the new IR mode (i.e., `.json` format), use the following command to switch to the new IR mode:
|
||||
|
||||
```bash
|
||||
export FLAGS_enable_pir_api=1
|
||||
python3 tools/export_model.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy Global.save_inference_dir=./inference/en_PP-OCRv3_mobile_rec/
|
||||
```
|
||||
|
||||
Once successful, you will have two files in the directory:
|
||||
|
||||
```text
|
||||
inference/en_PP-OCRv3_mobile_rec/
|
||||
├── inference.pdiparams # Model parameter file for the inference model
|
||||
└── inference.json # Program file for the inference model
|
||||
```
|
||||
|
||||
### Custom Model Inference
|
||||
|
||||
If you modified the text dictionary during training, you must specify the path to the custom dictionary when using the inference model for prediction. For more information about configuring and explaining inference hyperparameters, refer to the [Inference Hyperparameters Explanation Tutorial](../blog/inference_args.md).
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./your_inference_model" --rec_image_shape="3, 48, 320" --rec_char_dict_path="your_text_dict_path"
|
||||
```
|
||||
|
||||
### 5. FAQ
|
||||
|
||||
**Q1:** Why is the prediction result inconsistent after converting a trained model to an inference model?
|
||||
|
||||
**A**: This is a common issue. It typically arises due to differences in the preprocessing and postprocessing parameters used during training and inference. To troubleshoot, check whether the preprocessing, postprocessing, and prediction settings in the configuration file used for training match those used during inference.
|
||||
586
docs/version2.x/ppocr/model_train/recognition.md
Normal file
586
docs/version2.x/ppocr/model_train/recognition.md
Normal file
@@ -0,0 +1,586 @@
|
||||
---
|
||||
comments: true
|
||||
typora-copy-images-to: images
|
||||
---
|
||||
|
||||
# 文字识别
|
||||
|
||||
本文提供了PaddleOCR文本识别任务的全流程指南，包括数据准备、模型训练、调优、评估、预测，各个阶段的详细说明：
|
||||
|
||||
## 1. 数据准备
|
||||
|
||||
### 1.1. 准备数据集
|
||||
|
||||
PaddleOCR 支持两种数据格式:
|
||||
|
||||
- `lmdb` 用于训练以lmdb格式存储的数据集(LMDBDataSet);
|
||||
- `通用数据` 用于训练以文本文件存储的数据集(SimpleDataSet);
|
||||
|
||||
训练数据的默认存储路径是 `PaddleOCR/train_data`,如果您的磁盘上已有数据集，只需创建软链接至数据集目录：
|
||||
|
||||
```bash linenums="1"
|
||||
# linux and mac os
|
||||
ln -sf <path/to/dataset> <path/to/paddle_ocr>/train_data/dataset
|
||||
# windows
|
||||
mklink /d <path/to/paddle_ocr>/train_data/dataset <path/to/dataset>
|
||||
```
|
||||
|
||||
### 1.2. 自定义数据集
|
||||
|
||||
下面以通用数据集为例， 介绍如何准备数据集：
|
||||
|
||||
- 训练集
|
||||
|
||||
建议将训练图片放入同一个文件夹，并用一个txt文件（rec_gt_train.txt）记录图片路径和标签，txt文件里的内容如下:
|
||||
|
||||
**注意：** txt文件中默认请将图片路径和图片标签用 \t 分割，如用其他方式分割将造成训练报错。
|
||||
|
||||
```text linenums="1"
|
||||
" 图像文件名 图像标注信息 "
|
||||
|
||||
train_data/rec/train/word_001.jpg 简单可依赖
|
||||
train_data/rec/train/word_002.jpg 用科技让复杂的世界更简单
|
||||
...
|
||||
```
|
||||
|
||||
最终训练集应有如下文件结构：
|
||||
|
||||
```text linenums="1"
|
||||
|-train_data
|
||||
|-rec
|
||||
|- rec_gt_train.txt
|
||||
|- train
|
||||
|- word_001.png
|
||||
|- word_002.jpg
|
||||
|- word_003.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
除上述单张图像为一行格式之外，PaddleOCR也支持对离线增广后的数据进行训练，为了防止相同样本在同一个batch中被多次采样，我们可以将相同标签对应的图片路径写在一行中，以列表的形式给出，在训练中，PaddleOCR会随机选择列表中的一张图片进行训练。对应地，标注文件的格式如下：
|
||||
|
||||
```text linenums="1"
|
||||
["11.jpg", "12.jpg"] 简单可依赖
|
||||
["21.jpg", "22.jpg", "23.jpg"] 用科技让复杂的世界更简单
|
||||
3.jpg ocr
|
||||
```
|
||||
|
||||
上述示例标注文件中，"11.jpg"和"12.jpg"的标签相同，都是`简单可依赖`，在训练的时候，对于该行标注，会随机选择其中的一张图片进行训练。
|
||||
|
||||
- 验证集
|
||||
|
||||
同训练集类似，验证集也需要提供一个包含所有图片的文件夹（test）和一个rec_gt_test.txt，验证集的结构如下所示：
|
||||
|
||||
```text linenums="1"
|
||||
|-train_data
|
||||
|-rec
|
||||
|- rec_gt_test.txt
|
||||
|- test
|
||||
|- word_001.jpg
|
||||
|- word_002.jpg
|
||||
|- word_003.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
### 1.3. 数据下载
|
||||
|
||||
- ICDAR2015
|
||||
|
||||
若您本地没有数据集，可以在官网下载 [ICDAR2015](http://rrc.cvc.uab.es/?ch=4&com=downloads) 数据，用于快速验证。也可以参考[DTRB](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) ，下载 benchmark 所需的lmdb格式数据集。
|
||||
|
||||
如果你使用的是icdar2015的公开数据集，PaddleOCR 提供了一份用于训练 ICDAR2015 数据集的标签文件，通过以下方式下载：
|
||||
|
||||
```bash linenums="1"
|
||||
# 训练集标签
|
||||
wget -P ./train_data/ic15_data https://paddleocr.bj.bcebos.com/dataset/rec_gt_train.txt
|
||||
# 测试集标签
|
||||
wget -P ./train_data/ic15_data https://paddleocr.bj.bcebos.com/dataset/rec_gt_test.txt
|
||||
```
|
||||
|
||||
PaddleOCR 也提供了数据格式转换脚本，可以将ICDAR官网 label 转换为PaddleOCR支持的数据格式。 数据转换工具在 `ppocr/utils/gen_label.py`, 这里以训练集为例：
|
||||
|
||||
```bash linenums="1"
|
||||
# 将官网下载的标签文件转换为 rec_gt_label.txt
|
||||
python gen_label.py --mode="rec" --input_path="{path/of/origin/label}" --output_label="rec_gt_label.txt"
|
||||
```
|
||||
|
||||
数据样式格式如下，(a)为原始图片,(b)为每张图片对应的 Ground Truth 文本文件：
|
||||
|
||||
|
||||
- 多语言数据集
|
||||
|
||||
多语言模型的训练数据集均为100w的合成数据，使用了开源合成工具 [text_renderer](https://github.com/Sanster/text_renderer) ，少量的字体可以通过下面两种方式下载。
|
||||
|
||||
- [百度网盘](https://pan.baidu.com/s/1bS_u207Rm7YbY33wOECKDA) 提取码：frgi
|
||||
- [google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view)
|
||||
|
||||
### 1.4. 字典
|
||||
|
||||
最后需要提供一个字典（{word_dict_name}.txt），使模型在训练时，可以将所有出现的字符映射为字典的索引。
|
||||
|
||||
因此字典需要包含所有希望被正确识别的字符，{word_dict_name}.txt需要写成如下格式，并以 `utf-8` 编码格式保存：
|
||||
|
||||
```text linenums="1"
|
||||
l
|
||||
d
|
||||
a
|
||||
d
|
||||
r
|
||||
n
|
||||
```
|
||||
|
||||
word_dict.txt 每行有一个单字，将字符与数字索引映射在一起，“and” 将被映射成 [2 5 1]
|
||||
|
||||
- 内置字典
|
||||
|
||||
PaddleOCR内置了一部分字典，可以按需使用。
|
||||
|
||||
`ppocr/utils/ppocr_keys_v1.txt` 是一个包含6623个字符的中文字典
|
||||
|
||||
`ppocr/utils/ic15_dict.txt` 是一个包含36个字符的英文字典
|
||||
|
||||
`ppocr/utils/dict/french_dict.txt` 是一个包含118个字符的法文字典
|
||||
|
||||
`ppocr/utils/dict/japan_dict.txt` 是一个包含4399个字符的日文字典
|
||||
|
||||
`ppocr/utils/dict/korean_dict.txt` 是一个包含3636个字符的韩文字典
|
||||
|
||||
`ppocr/utils/dict/german_dict.txt` 是一个包含131个字符的德文字典
|
||||
|
||||
`ppocr/utils/en_dict.txt` 是一个包含96个字符的英文字典
|
||||
|
||||
目前的多语言模型仍处在demo阶段，会持续优化模型并补充语种，**非常欢迎您为我们提供其他语言的字典和字体**，
|
||||
如您愿意可将字典文件提交至 [dict](../../ppocr/utils/dict)，我们会在Repo中感谢您。
|
||||
|
||||
- 自定义字典
|
||||
|
||||
如需自定义dic文件，请在 `configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml` 中添加 `character_dict_path` 字段, 指向您的字典路径。
|
||||
|
||||
### 1.5. 添加空格类别
|
||||
|
||||
如果希望支持识别"空格"类别, 请将yml文件中的 `use_space_char` 字段设置为 `True`。
|
||||
|
||||
### 1.6. 数据增强
|
||||
|
||||
PaddleOCR提供了多种数据增强方式，默认配置文件中已经添加了数据增广。
|
||||
|
||||
默认的扰动方式有：颜色空间转换(cvtColor)、模糊(blur)、抖动(jitter)、噪声(Gasuss noise)、随机切割(random crop)、透视(perspective)、颜色反转(reverse)、TIA数据增广。
|
||||
|
||||
训练过程中每种扰动方式以40%的概率被选择，具体代码实现请参考：[rec_img_aug.py](../../ppocr/data/imaug/rec_img_aug.py)
|
||||
|
||||
*由于OpenCV的兼容性问题，扰动操作暂时只支持Linux*
|
||||
|
||||
## 2. 开始训练
|
||||
|
||||
PaddleOCR提供了训练脚本、评估脚本和预测脚本，本节将以 PP-OCRv3 英文识别模型为例：
|
||||
|
||||
### 2.1. 启动训练
|
||||
|
||||
首先下载pretrain model，您可以下载训练好的模型在 icdar2015 数据上进行finetune
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/
|
||||
# 下载英文PP-OCRv3的预训练模型
|
||||
wget -P ./pretrain_models/ https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_train.tar
|
||||
# 解压模型参数
|
||||
cd pretrain_models
|
||||
tar -xf en_PP-OCRv3_rec_train.tar && rm -rf en_PP-OCRv3_rec_train.tar
|
||||
```
|
||||
|
||||
开始训练:
|
||||
|
||||
*如果您安装的是cpu版本，请将配置文件中的 `use_gpu` 字段修改为false*
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU训练 支持单卡，多卡训练
|
||||
# 训练icdar15英文数据 训练日志会自动保存为 "{save_model_dir}" 下的train.log
|
||||
|
||||
#单卡训练（训练周期长，不建议）
|
||||
python3 tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy
|
||||
|
||||
# 多卡训练，通过--gpus参数指定卡号
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy
|
||||
```
|
||||
|
||||
正常启动训练后，会看到以下log输出：
|
||||
|
||||
```bash linenums="1"
|
||||
[2022/02/22 07:58:05] root INFO: epoch: [1/800], iter: 10, lr: 0.000000, loss: 0.754281, acc: 0.000000, norm_edit_dis: 0.000008, reader_cost: 0.55541 s, batch_cost: 0.91654 s, samples: 1408, ips: 153.62133
|
||||
[2022/02/22 07:58:13] root INFO: epoch: [1/800], iter: 20, lr: 0.000001, loss: 0.924677, acc: 0.000000, norm_edit_dis: 0.000008, reader_cost: 0.00236 s, batch_cost: 0.28528 s, samples: 1280, ips: 448.68599
|
||||
[2022/02/22 07:58:23] root INFO: epoch: [1/800], iter: 30, lr: 0.000002, loss: 0.967231, acc: 0.000000, norm_edit_dis: 0.000008, reader_cost: 0.14527 s, batch_cost: 0.42714 s, samples: 1280, ips: 299.66507
|
||||
[2022/02/22 07:58:31] root INFO: epoch: [1/800], iter: 40, lr: 0.000003, loss: 0.895318, acc: 0.000000, norm_edit_dis: 0.000008, reader_cost: 0.00173 s, batch_cost: 0.27719 s, samples: 1280, ips: 461.77252
|
||||
```
|
||||
|
||||
log 中自动打印如下信息：
|
||||
|
||||
| 字段 | 含义 |
|
||||
| :----: | :------: |
|
||||
| epoch | 当前迭代轮次 |
|
||||
| iter | 当前迭代次数 |
|
||||
| lr | 当前学习率 |
|
||||
| loss | 当前损失函数 |
|
||||
| acc | 当前batch的准确率 |
|
||||
| norm_edit_dis | 当前 batch 的编辑距离 |
|
||||
| reader_cost | 当前 batch 数据处理耗时 |
|
||||
| batch_cost | 当前 batch 总耗时 |
|
||||
| samples | 当前 batch 内的样本数 |
|
||||
| ips | 每秒处理图片的数量 |
|
||||
|
||||
PaddleOCR支持训练和评估交替进行, 可以在 `configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml` 中修改 `eval_batch_step` 设置评估频率，默认每500个iter评估一次。评估过程中默认将最佳acc模型，保存为 `output/en_PP-OCRv3_mobile_rec/best_accuracy` 。
|
||||
|
||||
如果验证集很大，测试将会比较耗时，建议减少评估次数，或训练完再进行评估。
|
||||
|
||||
**提示：** 可通过 -c 参数选择 `configs/rec/` 路径下的多种模型配置进行训练，PaddleOCR支持的识别算法可以参考[前沿算法列表](../../algorithm/overview.md)：
|
||||
|
||||
训练中文数据，推荐使用[PP-OCRv3_mobile_rec_distillation.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/rec/PP-OCRv3/PP-OCRv3_mobile_rec_distillation.yml)，如您希望尝试其他算法在中文数据集上的效果，请参考下列说明修改配置文件：
|
||||
|
||||
以 `PP-OCRv3_mobile_rec_distillation.yml` 为例：
|
||||
|
||||
```yaml linenums="1"
|
||||
Global:
|
||||
...
|
||||
# 添加自定义字典，如修改字典请将路径指向新字典
|
||||
character_dict_path: ppocr/utils/ppocr_keys_v1.txt
|
||||
...
|
||||
# 识别空格
|
||||
use_space_char: True
|
||||
|
||||
|
||||
Optimizer:
|
||||
...
|
||||
# 添加学习率衰减策略
|
||||
lr:
|
||||
name: Cosine
|
||||
learning_rate: 0.001
|
||||
...
|
||||
|
||||
...
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
# 数据集格式，支持LMDBDataSet以及SimpleDataSet
|
||||
name: SimpleDataSet
|
||||
# 数据集路径
|
||||
data_dir: ./train_data/
|
||||
# 训练集标签文件
|
||||
label_file_list: ["./train_data/train_list.txt"]
|
||||
transforms:
|
||||
...
|
||||
- RecResizeImg:
|
||||
# 修改 image_shape 以适应长文本
|
||||
image_shape: [3, 48, 320]
|
||||
...
|
||||
loader:
|
||||
...
|
||||
# 单卡训练的batch_size
|
||||
batch_size_per_card: 256
|
||||
...
|
||||
|
||||
Eval:
|
||||
dataset:
|
||||
# 数据集格式，支持LMDBDataSet以及SimpleDataSet
|
||||
name: SimpleDataSet
|
||||
# 数据集路径
|
||||
data_dir: ./train_data
|
||||
# 验证集标签文件
|
||||
label_file_list: ["./train_data/val_list.txt"]
|
||||
transforms:
|
||||
...
|
||||
- RecResizeImg:
|
||||
# 修改 image_shape 以适应长文本
|
||||
image_shape: [3, 48, 320]
|
||||
...
|
||||
loader:
|
||||
# 单卡验证的batch_size
|
||||
batch_size_per_card: 256
|
||||
...
|
||||
```
|
||||
|
||||
**注意，预测/评估时的配置文件请务必与训练一致。**
|
||||
|
||||
### 2.2. 断点训练
|
||||
|
||||
如果训练程序中断，如果希望加载训练中断的模型从而恢复训练，可以通过指定Global.checkpoints指定要加载的模型路径：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.checkpoints=./your/trained/model
|
||||
```
|
||||
|
||||
**注意**：`Global.checkpoints`的优先级高于`Global.pretrained_model`的优先级，即同时指定两个参数时，优先加载`Global.checkpoints`指定的模型，如果`Global.checkpoints`指定的模型路径有误，会加载`Global.pretrained_model`指定的模型。
|
||||
|
||||
### 2.3. 更换Backbone 训练
|
||||
|
||||
PaddleOCR将网络划分为四部分，分别在[ppocr/modeling](../../ppocr/modeling)下。 进入网络的数据将按照顺序(transforms->backbones->necks->heads)依次通过这四个部分。
|
||||
|
||||
```bash linenums="1"
|
||||
├── architectures # 网络的组网代码
|
||||
├── transforms # 网络的图像变换模块
|
||||
├── backbones # 网络的特征提取模块
|
||||
├── necks # 网络的特征增强模块
|
||||
└── heads # 网络的输出模块
|
||||
```
|
||||
|
||||
如果要更换的Backbone 在PaddleOCR中有对应实现，直接修改配置yml文件中`Backbone`部分的参数即可。
|
||||
|
||||
如果要使用新的Backbone，更换backbones的例子如下:
|
||||
|
||||
1. 在 [ppocr/modeling/backbones](../../ppocr/modeling/backbones) 文件夹下新建文件，如my_backbone.py。
|
||||
2. 在 my_backbone.py 文件内添加相关代码，示例代码如下:
|
||||
|
||||
```python linenums="1"
|
||||
import paddle
|
||||
import paddle.nn as nn
|
||||
import paddle.nn.functional as F
|
||||
|
||||
|
||||
class MyBackbone(nn.Layer):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(MyBackbone, self).__init__()
|
||||
# your init code
|
||||
self.conv = nn.xxxx
|
||||
|
||||
def forward(self, inputs):
|
||||
# your network forward
|
||||
y = self.conv(inputs)
|
||||
return y
|
||||
```
|
||||
|
||||
3. 在 [ppocr/modeling/backbones/\_*init\_*.py](https://github.com/PaddlePaddle/PaddleOCR/blob/main/ppocr/modeling/backbones/__init__.py)文件内导入添加的`MyBackbone`模块，然后修改配置文件中Backbone进行配置即可使用，格式如下:
|
||||
|
||||
```yaml linenums="1"
|
||||
Backbone:
|
||||
name: MyBackbone
|
||||
args1: args1
|
||||
```
|
||||
|
||||
**注意**：如果要更换网络的其他模块，可以参考[文档](../../algorithm/add_new_algorithm.md)。
|
||||
|
||||
### 2.4. 混合精度训练
|
||||
|
||||
如果您想进一步加快训练速度，可以使用[自动混合精度训练](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/performance_improving/amp_cn.html)， 以单机单卡为例，命令如下：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy \
|
||||
Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True
|
||||
```
|
||||
|
||||
### 2.5. 分布式训练
|
||||
|
||||
多机多卡训练时，通过 `--ips` 参数设置使用的机器IP地址，通过 `--gpus` 参数设置使用的GPU ID：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m paddle.distributed.launch --ips="xx.xx.xx.xx,xx.xx.xx.xx" --gpus '0,1,2,3' tools/train.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml \
|
||||
-o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy
|
||||
```
|
||||
|
||||
**注意:** （1）采用多机多卡训练时，需要替换上面命令中的ips值为您机器的地址，机器之间需要能够相互ping通；（2）训练时需要在多个机器上分别启动命令。查看机器ip地址的命令为`ifconfig`；（3）更多关于分布式训练的性能优势等信息，请参考：[分布式训练教程](../blog/distributed_training.md)。
|
||||
|
||||
### 2.6. 知识蒸馏训练
|
||||
|
||||
PaddleOCR支持了基于知识蒸馏的文本识别模型训练过程，更多内容可以参考[知识蒸馏说明文档](../model_compress/knowledge_distillation.md)。
|
||||
|
||||
### 2.7. 多语言模型训练
|
||||
|
||||
PaddleOCR目前已支持80种（除中文外）语种识别，`configs/rec/multi_languages` 路径下提供了一个多语言的配置文件模版: [rec_multi_language_lite_train.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/rec/multi_language/rec_multi_language_lite_train.yml)。
|
||||
|
||||
按语系划分，目前PaddleOCR支持的语种有：
|
||||
|
||||
| 配置文件 | 算法名称 | backbone | trans | seq | pred | language |
|
||||
| :--------: | :-------: | :-------: | :-------: | :-----: | :-----: | :-----: |
|
||||
| rec_chinese_cht_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 中文繁体 |
|
||||
| rec_en_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 英语（区分大小写） |
|
||||
| rec_french_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 法语 |
|
||||
| rec_ger_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 德语 |
|
||||
| rec_japan_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 日语 |
|
||||
| rec_korean_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 韩语 |
|
||||
| rec_latin_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 拉丁字母 |
|
||||
| rec_arabic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 阿拉伯字母 |
|
||||
| rec_cyrillic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 斯拉夫字母 |
|
||||
| rec_devanagari_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 梵文字母 |
|
||||
|
||||
更多支持语种请参考: [多语言模型](../blog/multi_languages.md)
|
||||
|
||||
如您希望在现有模型效果的基础上调优，请参考下列说明修改配置文件：
|
||||
|
||||
以 `rec_french_lite_train` 为例：
|
||||
|
||||
```yaml linenums="1"
|
||||
Global:
|
||||
...
|
||||
# 添加自定义字典，如修改字典请将路径指向新字典
|
||||
character_dict_path: ./ppocr/utils/dict/french_dict.txt
|
||||
...
|
||||
# 识别空格
|
||||
use_space_char: True
|
||||
|
||||
...
|
||||
|
||||
Train:
|
||||
dataset:
|
||||
# 数据集格式，支持LMDBDataSet以及SimpleDataSet
|
||||
name: SimpleDataSet
|
||||
# 数据集路径
|
||||
data_dir: ./train_data/
|
||||
# 训练集标签文件
|
||||
label_file_list: ["./train_data/french_train.txt"]
|
||||
...
|
||||
|
||||
Eval:
|
||||
dataset:
|
||||
# 数据集格式，支持LMDBDataSet以及SimpleDataSet
|
||||
name: SimpleDataSet
|
||||
# 数据集路径
|
||||
data_dir: ./train_data
|
||||
# 验证集标签文件
|
||||
label_file_list: ["./train_data/french_val.txt"]
|
||||
...
|
||||
```
|
||||
|
||||
### 2.8. 其他训练环境
|
||||
|
||||
- Windows GPU/CPU
|
||||
在Windows平台上与Linux平台略有不同:
|
||||
Windows平台只支持`单卡`的训练与预测，指定GPU进行训练`set CUDA_VISIBLE_DEVICES=0`
|
||||
在Windows平台，DataLoader只支持单进程模式，因此需要设置 `num_workers` 为0;
|
||||
|
||||
- macOS
|
||||
不支持GPU模式，需要在配置文件中设置`use_gpu`为False，其余训练评估预测命令与Linux GPU完全相同。
|
||||
|
||||
- Linux DCU
|
||||
DCU设备上运行需要设置环境变量 `export HIP_VISIBLE_DEVICES=0,1,2,3`，其余训练评估预测命令与Linux GPU完全相同。
|
||||
|
||||
### 2.9 模型微调
|
||||
|
||||
实际使用过程中，建议加载官方提供的预训练模型，在自己的数据集中进行微调，关于识别模型的微调方法，请参考：[模型微调教程](./finetune.md)。
|
||||
|
||||
## 3. 模型评估与预测
|
||||
|
||||
### 3.1. 指标评估
|
||||
|
||||
训练中模型参数默认保存在`Global.save_model_dir`目录下。在评估指标时，需要设置`Global.checkpoints`指向保存的参数文件。评估数据集可以通过 `configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml` 修改Eval中的 `label_file_path` 设置。
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU 评估， Global.checkpoints 为待测权重
|
||||
python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.checkpoints={path/to/weights}/best_accuracy
|
||||
```
|
||||
|
||||
### 3.2. 测试识别效果
|
||||
|
||||
使用 PaddleOCR 训练好的模型，可以通过以下脚本进行快速预测。
|
||||
|
||||
默认预测图片存储在 `infer_img` 里，通过 `-o Global.checkpoints` 加载训练好的参数文件：
|
||||
|
||||
根据配置文件中设置的 `save_model_dir` 和 `save_epoch_step` 字段，会有以下几种参数被保存下来：
|
||||
|
||||
```text linenums="1"
|
||||
output/rec/
|
||||
├── best_accuracy.pdopt
|
||||
├── best_accuracy.pdparams
|
||||
├── best_accuracy.states
|
||||
├── config.yml
|
||||
├── iter_epoch_3.pdopt
|
||||
├── iter_epoch_3.pdparams
|
||||
├── iter_epoch_3.states
|
||||
├── latest.pdopt
|
||||
├── latest.pdparams
|
||||
├── latest.states
|
||||
└── train.log
|
||||
```
|
||||
|
||||
其中 best_accuracy.*是评估集上的最优模型；iter_epoch_x.* 是以 `save_epoch_step` 为间隔保存下来的模型；latest.* 是最后一个epoch的模型。
|
||||
|
||||
```bash linenums="1"
|
||||
# 预测英文结果
|
||||
python3 tools/infer_rec.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
|
||||
```
|
||||
|
||||
预测图片：
|
||||
|
||||
|
||||
|
||||
得到输入图像的预测结果：
|
||||
|
||||
```bash linenums="1"
|
||||
infer_img: doc/imgs_words/en/word_1.png
|
||||
result: ('joint', 0.9998967)
|
||||
```
|
||||
|
||||
预测使用的配置文件必须与训练一致，如您通过 `python3 tools/train.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml` 完成了中文模型的训练，
|
||||
您可以使用如下命令进行中文模型预测。
|
||||
|
||||
```bash linenums="1"
|
||||
# 预测中文结果
|
||||
python3 tools/infer_rec.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/ch/word_1.jpg
|
||||
```
|
||||
|
||||
预测图片：
|
||||
|
||||
|
||||
|
||||
得到输入图像的预测结果：
|
||||
|
||||
```bash linenums="1"
|
||||
infer_img: doc/imgs_words/ch/word_1.jpg
|
||||
result: ('韩国小馆', 0.997218)
|
||||
```
|
||||
|
||||
## 4. 模型导出与预测
|
||||
|
||||
inference 模型（`paddle.jit.save`保存的模型）
|
||||
一般是模型训练，把模型结构和模型参数保存在文件中的固化模型，多用于预测部署场景。
|
||||
训练过程中保存的模型是checkpoints模型，保存的只有模型的参数，多用于恢复训练等。
|
||||
与checkpoints模型相比，inference 模型会额外保存模型的结构信息，在预测部署、加速推理上性能优越，灵活方便，适合于实际系统集成。
|
||||
|
||||
识别模型转inference模型与检测的方式相同，如下：
|
||||
|
||||
```bash linenums="1"
|
||||
# 开启旧 IR 模式
|
||||
export FLAGS_enable_pir_api=0
|
||||
|
||||
# -c 后面设置训练算法的yml配置文件
|
||||
# -o 配置可选参数
|
||||
# Global.pretrained_model 参数设置待转换的训练模型地址，不用添加文件后缀 .pdmodel，.pdopt或.pdparams。
|
||||
# Global.save_inference_dir参数设置转换的模型将保存的地址。
|
||||
|
||||
python3 tools/export_model.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy Global.save_inference_dir=./inference/en_PP-OCRv3_mobile_rec/
|
||||
```
|
||||
|
||||
**注意：**如果您是在自己的数据集上训练的模型，并且调整了中文字符的字典文件，请注意修改配置文件中的`character_dict_path`为自定义字典文件。
|
||||
|
||||
转换成功后，在目录下有三个文件：
|
||||
|
||||
```text linenums="1"
|
||||
inference/en_PP-OCRv3_mobile_rec/
|
||||
├── inference.pdiparams # 识别inference模型的参数文件
|
||||
├── inference.pdiparams.info # 识别inference模型的参数信息，可忽略
|
||||
└── inference.pdmodel # 识别inference模型的program文件
|
||||
```
|
||||
|
||||
**注意：** 如果需要以新 IR 模式（`FLAGS_enable_pir_api=1`）存储 `.json` 文件，请执行以下命令切换到新 IR 模式：
|
||||
|
||||
```bash linenums="1"
|
||||
export FLAGS_enable_pir_api=1
|
||||
python3 tools/export_model.py -c configs/rec/PP-OCRv3/en_PP-OCRv3_mobile_rec.yml -o Global.pretrained_model=./pretrain_models/en_PP-OCRv3_rec_train/best_accuracy Global.save_inference_dir=./inference/en_PP-OCRv3_mobile_rec/
|
||||
```
|
||||
|
||||
转换成功后，在目录下有三个文件：
|
||||
|
||||
```text linenums="1"
|
||||
inference/en_PP-OCRv3_mobile_rec/
|
||||
├── inference.pdiparams # 识别inference模型的参数文件
|
||||
└── inference.json # 识别inference模型的program文件
|
||||
```
|
||||
|
||||
- 自定义模型推理
|
||||
|
||||
如果训练时修改了文本的字典，在使用inference模型预测时，需要通过`--rec_char_dict_path`指定使用的字典路径，更多关于推理超参数的配置与解释，请参考：[模型推理超参数解释教程](../blog/inference_args.md)。
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./your inference model" --rec_image_shape="3, 48, 320" --rec_char_dict_path="your text dict path"
|
||||
```
|
||||
|
||||
## 5. FAQ
|
||||
|
||||
Q1: 训练模型转inference 模型之后预测效果不一致？
|
||||
|
||||
**A**：此类问题出现较多，问题多是trained model预测时候的预处理、后处理参数和inference model预测的时候的预处理、后处理参数不一致导致的。可以对比训练使用的配置文件中的预处理、后处理和预测时是否存在差异。
|
||||
132
docs/version2.x/ppocr/model_train/training.en.md
Normal file
132
docs/version2.x/ppocr/model_train/training.en.md
Normal file
@@ -0,0 +1,132 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Model Training
|
||||
|
||||
This article will introduce the basic concepts that is necessary for model training and tuning.
|
||||
|
||||
At the same time, it will briefly introduce the structure of the training data and how to prepare the data to fine-tune model in vertical scenes.
|
||||
|
||||
## 1. Yml Configuration
|
||||
|
||||
The PaddleOCR uses configuration files to control network training and evaluation parameters. In the configuration file, you can set the model, optimizer, loss function, and pre- and post-processing parameters of the model. PaddleOCR reads these parameters from the configuration file, and then builds a complete training process to train the model. Fine-tuning can also be completed by modifying the parameters in the configuration file, which is simple and convenient.
|
||||
|
||||
For the complete configuration file description, please refer to [Configuration File](../blog/config.en.md).
|
||||
|
||||
## 2. Basic Concepts
|
||||
|
||||
During the model training process, some hyper-parameters can be manually specified to obtain the optimal result at the least cost. Different data volumes may require different hyper-parameters. When you want to fine-tune the model based on your own data, there are several parameter adjustment strategies for reference:
|
||||
|
||||
### 2.1 Learning Rate
|
||||
|
||||
The learning rate is one of the most important hyper-parameters for training neural networks. It represents the step length of the gradient moving towards the optimal solution of the loss function in each iteration.
|
||||
A variety of learning rate update strategies are provided by PaddleOCR, which can be specified in configuration files. For example:
|
||||
|
||||
```yaml linenums="1"
|
||||
Optimizer:
|
||||
...
|
||||
lr:
|
||||
name: Piecewise
|
||||
decay_epochs : [700, 800]
|
||||
values : [0.001, 0.0001]
|
||||
warmup_epoch: 5
|
||||
```
|
||||
|
||||
`Piecewise` stands for piece-wise constant attenuation. Different learning rates are specified in different learning stages, and the learning rate stay the same in each stage.
|
||||
|
||||
`warmup_epoch` means that in the first 5 epochs, the learning rate will be increased gradually from 0 to base_lr. For all strategies, please refer to the code [learning_rate.py](../../ppocr/optimizer/learning_rate.py).
|
||||
|
||||
### 2.2 Regularization
|
||||
|
||||
Regularization can effectively avoid algorithm over-fitting. PaddleOCR provides L1 and L2 regularization methods.
|
||||
L1 and L2 regularization are the most widely used regularization methods.
|
||||
L1 regularization adds a regularization term to the objective function to reduce the sum of absolute values of the parameters;
|
||||
while in L2 regularization, the purpose of adding a regularization term is to reduce the sum of squared parameters.
|
||||
The configuration method is as follows:
|
||||
|
||||
```yaml linenums="1"
|
||||
Optimizer:
|
||||
...
|
||||
regularizer:
|
||||
name: L2
|
||||
factor: 2.0e-05
|
||||
```
|
||||
|
||||
### 2.3 Evaluation Indicators
|
||||
|
||||
(1) Detection stage: First, evaluate according to the IOU of the detection frame and the labeled frame. If the IOU is greater than a certain threshold, it is judged that the detection is accurate. Here, the detection frame and the label frame are different from the general target detection frame, and they are represented by polygons. Detection accuracy: the percentage of the correct detection frame number in all detection frames is mainly used to judge the detection index. Detection recall rate: the percentage of correct detection frames in all marked frames, which is mainly an indicator of missed detection.
|
||||
|
||||
(2) Recognition stage: Character recognition accuracy, that is, the ratio of correctly recognized text lines to the number of marked text lines. Only the entire line of text recognition pairs can be regarded as correct recognition.
|
||||
|
||||
(3) End-to-end statistics: End-to-end recall rate: accurately detect and correctly identify the proportion of text lines in all labeled text lines; End-to-end accuracy rate: accurately detect and correctly identify the number of text lines in the detected text lines. The standard for accurate detection is that the IOU of the detection box and the labeled box is greater than a certain threshold, and the text in the correctly identified detection box is the same as the labeled text.
|
||||
|
||||
## 3. Data and Vertical Scenes
|
||||
|
||||
### 3.1 Training Data
|
||||
|
||||
The current open source models, data sets and magnitudes are as follows:
|
||||
|
||||
- Detection:
|
||||
- English data set, ICDAR2015
|
||||
- Chinese data set, LSVT street view data set training data 3w pictures
|
||||
|
||||
- Identification:
|
||||
- English data set, MJSynth and SynthText synthetic data, the data volume is tens of millions.
|
||||
- Chinese data set, LSVT street view data set crops the image according to the truth value, and performs position calibration, a total of 30w images. In addition, based on the LSVT corpus, 500w of synthesized data.
|
||||
- Small language data set, using different corpora and fonts, respectively generated 100w synthetic data set, and using ICDAR-MLT as the verification set.
|
||||
|
||||
Among them, the public data sets are all open source, users can search and download by themselves, or refer to [Chinese data set](../../datasets/datasets.en.md), synthetic data is not open source, users can use open source synthesis tools to synthesize by themselves. Synthesis tools include [text_renderer](https://github.com/Sanster/text_renderer), [SynthText](https://github.com/ankush-me/SynthText), [TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) etc.
|
||||
|
||||
### 3.2 Vertical Scene
|
||||
|
||||
PaddleOCR mainly focuses on general OCR. If you have vertical requirements, you can use PaddleOCR + vertical data to train yourself;
|
||||
If there is a lack of labeled data, or if you do not want to invest in research and development costs, it is recommended to directly call the open API, which covers some of the more common vertical categories.
|
||||
|
||||
### 3.3 Build Your Own Dataset
|
||||
|
||||
There are several experiences for reference when constructing the data set:
|
||||
|
||||
(1) The amount of data in the training set:
|
||||
|
||||
a. The data required for detection is relatively small. For Fine-tune based on the PaddleOCR model, 500 sheets are generally required to achieve good results.
|
||||
|
||||
b. Recognition is divided into English and Chinese. Generally, English scenarios require hundreds of thousands of data to achieve good results, while Chinese requires several million or more.
|
||||
|
||||
(2) When the amount of training data is small, you can try the following three ways to get more data:
|
||||
|
||||
a. Manually collect more training data, the most direct and effective way.
|
||||
|
||||
b. Basic image processing or transformation based on PIL and opencv. For example, the three modules of ImageFont, Image, ImageDraw in PIL write text into the background, opencv's rotating affine transformation, Gaussian filtering and so on.
|
||||
|
||||
c. Use data generation algorithms to synthesize data, such as algorithms such as pix2pix.
|
||||
|
||||
## 4. FAQ
|
||||
|
||||
**Q**: How to choose a suitable network input shape when training CRNN recognition?
|
||||
|
||||
A: The general height is 32, the longest width is selected, there are two methods:
|
||||
|
||||
(1) Calculate the aspect ratio distribution of training sample images. The selection of the maximum aspect ratio considers 80% of the training samples.
|
||||
|
||||
(2) Count the number of texts in training samples. The selection of the longest number of characters considers the training sample that satisfies 80%. Then the aspect ratio of Chinese characters is approximately considered to be 1, and that of English is 3:1, and the longest width is estimated.
|
||||
|
||||
**Q**: During the recognition training, the accuracy of the training set has reached 90, but the accuracy of the verification set has been kept at 70, what should I do?
|
||||
|
||||
A: If the accuracy of the training set is 90 and the test set is more than 70, it should be over-fitting. There are two methods to try:
|
||||
|
||||
(1) Add more augmentation methods or increase the [probability] of augmented prob (https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/rec_img_aug.py#L341), The default is 0.4.
|
||||
|
||||
(2) Increase the [l2 dcay value] of the system (https://github.com/PaddlePaddle/PaddleOCR/blob/a501603d54ff5513fc4fc760319472e59da25424/configs/rec/ch_ppocr_v1.1/rec_chinese_lite_train_v1.1.yml#L47)
|
||||
|
||||
**Q**: When the recognition model is trained, loss can drop normally, but acc is always 0
|
||||
|
||||
A: It is normal for the acc to be 0 at the beginning of the recognition model training, and the indicator will come up after a longer training period.
|
||||
|
||||
***
|
||||
|
||||
Click the following links for detailed training tutorial:
|
||||
|
||||
- [text detection model training](./detection.en.md)
|
||||
- [text recognition model training](./recognition.en.md)
|
||||
- [text direction classification model training](./angle_class.en.md)
|
||||
128
docs/version2.x/ppocr/model_train/training.md
Normal file
128
docs/version2.x/ppocr/model_train/training.md
Normal file
@@ -0,0 +1,128 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-OCR模型训练
|
||||
|
||||
本文将介绍模型训练时需掌握的基本概念，和训练时的调优方法。
|
||||
|
||||
同时会简单介绍PaddleOCR模型训练数据的组成部分，以及如何在垂类场景中准备数据finetune模型。
|
||||
|
||||
## 1. 配置文件说明
|
||||
|
||||
PaddleOCR模型使用配置文件管理网络训练、评估的参数。在配置文件中，可以设置组建模型、优化器、损失函数、模型前后处理的参数，PaddleOCR从配置文件中读取到这些参数，进而组建出完整的训练流程，完成模型训练，在需要对模型进行优化的时，可以通过修改配置文件中的参数完成配置，使用简单且方便修改。
|
||||
|
||||
完整的配置文件说明可以参考[配置文件](../blog/config.md)
|
||||
|
||||
## 2. 基本概念
|
||||
|
||||
模型训练过程中需要手动调整一些超参数，帮助模型以最小的代价获得最优指标。不同的数据量可能需要不同的超参，当您希望在自己的数据上finetune或对模型效果调优时，有以下几个参数调整策略可供参考：
|
||||
|
||||
### 2.1 学习率
|
||||
|
||||
学习率是训练神经网络的重要超参数之一，它代表在每一次迭代中梯度向损失函数最优解移动的步长。
|
||||
在PaddleOCR中提供了多种学习率更新策略,可以通过配置文件修改，例如：
|
||||
|
||||
```yaml linenums="1"
|
||||
Optimizer:
|
||||
...
|
||||
lr:
|
||||
name: Piecewise
|
||||
decay_epochs : [700, 800]
|
||||
values : [0.001, 0.0001]
|
||||
warmup_epoch: 5
|
||||
```
|
||||
|
||||
Piecewise 代表分段常数衰减，在不同的学习阶段指定不同的学习率，在每段内学习率相同。
|
||||
warmup_epoch 代表在前5个epoch中，学习率将逐渐从0增加到base_lr。全部策略可以参考代码[learning_rate.py](../../ppocr/optimizer/learning_rate.py) 。
|
||||
|
||||
### 2.2 正则化
|
||||
|
||||
正则化可以有效的避免算法过拟合，PaddleOCR中提供了L1、L2正则方法，L1 和 L2 正则化是最常用的正则化方法。L1 正则化向目标函数添加正则化项，以减少参数的绝对值总和；而 L2 正则化中，添加正则化项的目的在于减少参数平方的总和。配置方法如下：
|
||||
|
||||
```yaml linenums="1"
|
||||
Optimizer:
|
||||
...
|
||||
regularizer:
|
||||
name: L2
|
||||
factor: 2.0e-05
|
||||
```
|
||||
|
||||
### 2.3 评估指标
|
||||
|
||||
（1）检测阶段：先按照检测框和标注框的IOU评估，IOU大于某个阈值判断为检测准确。这里检测框和标注框不同于一般的通用目标检测框，是采用多边形进行表示。检测准确率：正确的检测框个数在全部检测框的占比，主要是判断检测指标。检测召回率：正确的检测框个数在全部标注框的占比，主要是判断漏检的指标。
|
||||
|
||||
（2）识别阶段： 字符识别准确率，即正确识别的文本行占标注的文本行数量的比例，只有整行文本识别对才算正确识别。
|
||||
|
||||
（3）端到端统计： 端对端召回率：准确检测并正确识别文本行在全部标注文本行的占比； 端到端准确率：准确检测并正确识别文本行在 检测到的文本行数量 的占比； 准确检测的标准是检测框与标注框的IOU大于某个阈值，正确识别的检测框中的文本与标注的文本相同。
|
||||
|
||||
## 3. 数据与垂类场景
|
||||
|
||||
### 3.1 训练数据
|
||||
|
||||
目前开源的模型，数据集和量级如下：
|
||||
|
||||
- 检测：
|
||||
- 英文数据集，ICDAR2015
|
||||
- 中文数据集，LSVT街景数据集训练数据3w张图片
|
||||
|
||||
- 识别：
|
||||
- 英文数据集，MJSynth和SynthText合成数据，数据量上千万。
|
||||
- 中文数据集，LSVT街景数据集根据真值将图crop出来，并进行位置校准，总共30w张图像。此外基于LSVT的语料，合成数据500w。
|
||||
- 小语种数据集，使用不同语料和字体，分别生成了100w合成数据集，并使用ICDAR-MLT作为验证集。
|
||||
|
||||
其中，公开数据集都是开源的，用户可自行搜索下载，也可参考[中文数据集](../../datasets/datasets.md)，合成数据暂不开源，用户可使用开源合成工具自行合成，可参考的合成工具包括[text_renderer](https://github.com/Sanster/text_renderer) 、[SynthText](https://github.com/ankush-me/SynthText) 、[TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) 等。
|
||||
|
||||
### 3.2 垂类场景
|
||||
|
||||
PaddleOCR主要聚焦通用OCR，如果有垂类需求，您可以用PaddleOCR+垂类数据自己训练；
|
||||
如果缺少带标注的数据，或者不想投入研发成本，建议直接调用开放的API，开放的API覆盖了目前比较常见的一些垂类。
|
||||
|
||||
### 3.3 自己构建数据集
|
||||
|
||||
在构建数据集时有几个经验可供参考：
|
||||
|
||||
（1） 训练集的数据量：
|
||||
|
||||
a. 检测需要的数据相对较少，在PaddleOCR模型的基础上进行Fine-tune，一般需要500张可达到不错的效果。
|
||||
|
||||
b. 识别分英文和中文，一般英文场景需要几十万数据可达到不错的效果，中文则需要几百万甚至更多。
|
||||
|
||||
（2）当训练数据量少时，可以尝试以下三种方式获取更多的数据：
|
||||
|
||||
a. 人工采集更多的训练数据，最直接也是最有效的方式。
|
||||
|
||||
b. 基于PIL和opencv基本图像处理或者变换。例如PIL中ImageFont, Image, ImageDraw三个模块将文字写到背景中，opencv的旋转仿射变换，高斯滤波等。
|
||||
|
||||
c. 利用数据生成算法合成数据，例如pix2pix或[StyleText](https://github.com/PFCCLab/StyleText)等算法。
|
||||
|
||||
## 4. 常见问题
|
||||
|
||||
**Q**：训练CRNN识别时，如何选择合适的网络输入shape？
|
||||
|
||||
A：一般高度采用32，最长宽度的选择，有两种方法：
|
||||
|
||||
（1）统计训练样本图像的宽高比分布。最大宽高比的选取考虑满足80%的训练样本。
|
||||
|
||||
（2）统计训练样本文字数目。最长字符数目的选取考虑满足80%的训练样本。然后中文字符长宽比近似认为是1，英文认为3：1，预估一个最长宽度。
|
||||
|
||||
**Q**：识别训练时，训练集精度已经到达90了，但验证集精度一直在70，涨不上去怎么办？
|
||||
|
||||
A：训练集精度90，测试集70多的话，应该是过拟合了，有两个可尝试的方法：
|
||||
|
||||
（1）加入更多的增广方式或者调大增广prob的[概率](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/rec_img_aug.py#L341)，默认为0.4。
|
||||
|
||||
（2）调大系统的[l2 dcay值](https://github.com/PaddlePaddle/PaddleOCR/blob/a501603d54ff5513fc4fc760319472e59da25424/configs/rec/ch_ppocr_v1.1/rec_chinese_lite_train_v1.1.yml#L47)
|
||||
|
||||
**Q**: 识别模型训练时，loss能正常下降，但acc一直为0
|
||||
|
||||
A：识别模型训练初期acc为0是正常的，多训一段时间指标就上来了。
|
||||
|
||||
***
|
||||
|
||||
具体的训练教程可点击下方链接跳转：
|
||||
|
||||
- [文本检测模型训练](./detection.md)
|
||||
- [文本识别模型训练](./recognition.md)
|
||||
- [文本方向分类器训练](./angle_class.md)
|
||||
- [知识蒸馏](../model_compress/knowledge_distillation.md)
|
||||
63
docs/version2.x/ppocr/model_train/tricks.en.md
Normal file
63
docs/version2.x/ppocr/model_train/tricks.en.md
Normal file
@@ -0,0 +1,63 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
Here we have sorted out some Chinese OCR training and prediction tricks, which are being updated continuously. You are welcome to contribute more OCR tricks!
|
||||
|
||||
#### 1. Replace Backbone Network
|
||||
|
||||
- **Problem Description**
|
||||
|
||||
At present, ResNet_vd series and MobileNetV3 series are the backbone networks used in PaddleOCR, whether replacing the other backbone networks will help to improve the accuracy? What should be paid attention to when replacing?
|
||||
|
||||
- **Tips**
|
||||
- Whether text detection or text recognition, the choice of backbone network is a trade-off between prediction effect and prediction efficiency. Generally, a larger backbone network is selected, e.g. ResNet101_vd, then the performance of the detection or recognition is more accurate, but the time cost will increase accordingly. And a smaller backbone network is selected, e.g. MobileNetV3_small_x0_35, the prediction speed is faster, but the accuracy of detection or recognition will be reduced. Fortunately, the detection or recognition effect of different backbone networks is positively correlated with the performance of ImageNet 1000 classification task. [**PaddleClas**](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.3/docs/en/models/models_intro_en.md) have sorted out the 23 series of classification network structures, such as ResNet_vd、Res2Net、HRNet、MobileNetV3、GhostNet. It provides the top1 accuracy of classification, the time cost of GPU(V100 and T4) and CPU(SD 855), and the 117 pretrained models [**download addresses**](https://paddleclas-en.readthedocs.io/en/latest/models/models_intro_en.html).
|
||||
|
||||
- Similar as the 4 stages of ResNet, the replacement of text detection backbone network is to determine those four stages to facilitate the integration of FPN like the object detection heads. In addition, for the text detection problem, the pre trained model in ImageNet1000 can accelerate the convergence and improve the accuracy.
|
||||
|
||||
- In order to replace the backbone network of text recognition, we need to pay attention to the descending position of network width and height stride. Since the ratio between width and height is large in chinese text recognition, the frequency of height decrease is less and the frequency of width decrease is more. You can refer the [modifies of MobileNetV3](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/ppocr/modeling/backbones/rec_mobilenet_v3.py) in PaddleOCR.
|
||||
|
||||
#### 2. Long Chinese Text Recognition
|
||||
|
||||
- **Problem Description**
|
||||
The maximum resolution of Chinese recognition model during training is [3,32,320], if the text image to be recognized is too long, as shown in the figure below, how to adapt?
|
||||
|
||||
|
||||
|
||||
- **Tips**
|
||||
|
||||
During the training, the training samples are not directly resized to [3,32,320]. At first, the height of samples are resized to 32 and keep the ratio between the width and the height. When the width is less than 320, the excess parts are padding 0. Besides, when the ratio between the width and the height of the samples is larger than 10, these samples will be ignored. When the prediction for one image, do as above, but do not limit the max ratio between the width and the height. When the prediction for an images batch, do as training, but the resized target width is the longest width of the images in the batch. [Code as following](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/tools/infer/predict_rec.py)：
|
||||
|
||||
```python linenums="1"
|
||||
def resize_norm_img(self, img, max_wh_ratio):
|
||||
imgC, imgH, imgW = self.rec_image_shape
|
||||
assert imgC == img.shape[2]
|
||||
if self.character_type == "ch":
|
||||
imgW = int((32 * max_wh_ratio))
|
||||
h, w = img.shape[:2]
|
||||
ratio = w / float(h)
|
||||
if math.ceil(imgH * ratio) > imgW:
|
||||
resized_w = imgW
|
||||
else:
|
||||
resized_w = int(math.ceil(imgH * ratio))
|
||||
resized_image = cv2.resize(img, (resized_w, imgH))
|
||||
resized_image = resized_image.astype('float32')
|
||||
resized_image = resized_image.transpose((2, 0, 1)) / 255
|
||||
resized_image -= 0.5
|
||||
resized_image /= 0.5
|
||||
padding_im = np.zeros((imgC, imgH, imgW), dtype=np.float32)
|
||||
padding_im[:, :, 0:resized_w] = resized_image
|
||||
return padding_im
|
||||
```
|
||||
|
||||
#### 3. Space Recognition
|
||||
|
||||
- **Problem Description**
|
||||
|
||||
As shown in the figure below, for Chinese and English mixed scenes, in order to facilitate reading and using the recognition results, it is often necessary to recognize the spaces between words. How can this situation be adapted?
|
||||
|
||||
|
||||
|
||||
- **Tips**
|
||||
|
||||
There are two possible methods for space recognition. (1) Optimize the text detection. For splitting the text at the space in detection results, it needs to divide the text line with space into many segments when label the data for detection. (2) Optimize the text recognition. The space character is introduced into the recognition dictionary. Label the blank line in the training data for text recognition. In addition, we can also concat multiple word lines to synthesize the training data with spaces. PaddleOCR currently uses the second method.
|
||||
63
docs/version2.x/ppocr/model_train/tricks.md
Normal file
63
docs/version2.x/ppocr/model_train/tricks.md
Normal file
@@ -0,0 +1,63 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
这里我们整理了一些中文OCR训练和预测技巧，持续更新中，欢迎大家贡献更多OCR技巧~
|
||||
|
||||
#### 1、更换骨干网络
|
||||
|
||||
- **问题描述**
|
||||
|
||||
目前PaddleOCR使用的主干网络为ResNet_vd系列和MobileNetV3系列，更换其他主干网络是否有助于提高准确率？更换时需要注意什么？
|
||||
|
||||
- **技巧**
|
||||
- 无论是文本检测还是文本识别，主干网络的选择都是预测效果和预测效率的权衡。一般选择较大的主干网络，如ResNet101_vd，则检测或识别的性能更准确，但时间成本也会相应增加。而选择较小的主干网络，如MobileNetV3_small_x0_35，预测速度更快，但检测或识别的准确率会降低。幸运的是，不同骨干网络的检测或识别效果与ImageNet 1000分类任务的性能呈正相关。[**PaddleClas**](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.3/docs/en/models/models_intro_en.md)整理了ResNet_vd、Res2Net、HRNet、MobileNetV3、GhostNet等23个系列的分类网络结构，提供了分类top1准确率、GPU(V100和T4)和CPU(SD 855)的时间成本，以及117个预训练模型[**下载地址**](https://paddleclas-en.readthedocs.io/en/latest/models/models_intro_en.html)。
|
||||
|
||||
- 和ResNet的4个阶段类似，文本检测骨干网络的更换就是确定这4个阶段，以便于像物体检测heads一样集成FPN。另外，对于文本检测问题，ImageNet1000中的预训练模型可以加速收敛并提高准确率。
|
||||
|
||||
- 更换文本识别骨干网络时，需要注意网络宽度和高度步长的下降位置。由于中文文本识别中宽度和高度的比值较大，因此高度下降的频率较少，宽度下降的频率较多。可以参考PaddleOCR中[MobileNetV3的修改](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/ppocr/modeling/backbones/rec_mobilenet_v3.py)。
|
||||
|
||||
#### 2、长中文文本识别
|
||||
|
||||
- **问题描述**
|
||||
中文识别模型在训练时的最大分辨率为[3,32,320]，如果待识别的文本图像过长，如下图所示，该如何适配？
|
||||
|
||||
|
||||
|
||||
- **小技巧**
|
||||
|
||||
在训练时，不要直接将训练样本resize到[3,32,320]，先将样本的高度resize为32，并保持宽高比，当宽度小于320时，超出部分用0填充。另外，当样本的宽高比大于10时，这些样本将被忽略。对一张图片进行预测时，同上，但不限制最大宽高比。对一批图像进行预测时，按照训练的方式进行，但调整后的目标宽度是该批图像的最长宽度。 [代码如下](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/tools/infer/predict_rec.py)：
|
||||
|
||||
```python linenums="1"
|
||||
def resize_norm_img(self, img, max_wh_ratio):
|
||||
imgC, imgH, imgW = self.rec_image_shape
|
||||
assert imgC == img.shape[2]
|
||||
if self.character_type == "ch":
|
||||
imgW = int((32 * max_wh_ratio))
|
||||
h, w = img.shape[:2]
|
||||
ratio = w / float(h)
|
||||
if math.ceil(imgH * ratio) > imgW:
|
||||
resized_w = imgW
|
||||
else:
|
||||
resized_w = int(math.ceil(imgH * ratio))
|
||||
resized_image = cv2.resize(img, (resized_w, imgH))
|
||||
resized_image = resized_image.astype('float32')
|
||||
resized_image = resized_image.transpose((2, 0, 1)) / 255
|
||||
resized_image -= 0.5
|
||||
resized_image /= 0.5
|
||||
padding_im = np.zeros((imgC, imgH, imgW), dtype=np.float32)
|
||||
padding_im[:, :, 0:resized_w] = resized_image
|
||||
return padding_im
|
||||
```
|
||||
|
||||
#### 3、空格识别
|
||||
|
||||
- **问题描述**
|
||||
|
||||
如下图所示，对于中英文混合场景，为了方便阅读和使用识别结果，经常需要识别单词之间的空格，这种情况该如何适配？
|
||||
|
||||
|
||||
|
||||
- **小技巧**
|
||||
|
||||
空格识别有两种可能的方法。（1）优化文本检测。为了将检测结果中的文本分割在空格处，在对数据进行标记时，需要将带有空格的文本行分成许多段
|
||||
Reference in New Issue
Block a user