This commit is contained in:
150
docs/version2.x/ppstructure/blog/how_to_do_kie.en.md
Normal file
150
docs/version2.x/ppstructure/blog/how_to_do_kie.en.md
Normal file
@@ -0,0 +1,150 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Key Information Extraction Pipeline
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
### 1.1 Background
|
||||
|
||||
Key information extraction (KIE) refers to extracting key information from text or images. As the downstream task of OCR, KIE of document image has many practical application scenarios, such as form recognition, ticket information extraction, ID card information extraction, etc. However, it is time-consuming and laborious to extract key information from these document images by manpower. It's challengable but also valuable to combine multi-modal features (visual, layout, text, etc) together and complete KIE tasks.
|
||||
|
||||
For the document images in a specific scene, the position and layout of the key information are relatively fixed. Therefore, in the early stage of the research, there are many methods based on template matching to extract the key information. This method is still widely used in many simple scenarios at present. However, it takes long time to adjust the template for different scenarios.
|
||||
|
||||
The KIE in the document image generally contains 2 subtasks, which is as shown follows.
|
||||
|
||||
* (1) SER: semantic entity recognition, which classifies each detected textline, such as dividing it into name and ID No. As shown in the red boxes in the following figure.
|
||||
|
||||
* (2) RE: relationship extraction, which matches the question and answer based on SER results. As shown in the figure below, the yellow arrows match the question and answer.
|
||||
|
||||
|
||||
|
||||
### 1.2 Mainstream Deep-learning Solutions
|
||||
|
||||
General KIE methods are based on Named Entity Recognition (NER), but such methods only use text information and ignore location and visual feature information, which leads to limited accuracy. In recent years, most scholars have started to combine mutil-modal features to improve the accuracy of KIE model. The main methods are as follows:
|
||||
|
||||
* (1) Grid based methods. These methods mainly focus on the fusion of multi-modal information at the image level. Most texts are of character granularity. The text and structure information embedding method is simple, such as the algorithm of chargrid [1].
|
||||
|
||||
* (2) Token based methods. These methods refer to the NLP methods such as Bert, which encode the position, vision and other feature information into the multi-modal model, and conduct pre-training on large-scale datasets, so that in downstream tasks, only a small amount of annotation data is required to obtain excellent results. The representative algorithms are layoutlm [2], layoutlmv2 [3], layoutxlm [4], structext [5], etc.
|
||||
|
||||
* (3) GCN based methods. These methods try to learn the structural information between images and characters, so as to solve the problem of extracting open set information (templates not seen in the training set), such as GCN [6], SDMGR [7] and other algorithms.
|
||||
|
||||
* (4) End to end based methods: these methods put the existing OCR character recognition and KIE information extraction tasks into a unified network for common learning, and strengthen each other in the learning process. Such as TRIE [8].
|
||||
|
||||
For more detailed introduction of the algorithms, please refer to Chapter 6 of [Diving into OCR](https://aistudio.baidu.com/aistudio/education/group/info/25207).
|
||||
|
||||
## 2. KIE Pipeline
|
||||
|
||||
Token based methods such as LayoutXLM are implemented in PaddleOCR. What's more, in PP-StructureV2, we simplify the LayoutXLM model and proposed VI-LayoutXLM, in which the visual feature extraction module is removed for speed-up. The textline sorting strategy conforming to the human reading order and UDML knowledge distillation strategy are utilized for higher model accuracy.
|
||||
|
||||
In the non end-to-end KIE method, KIE needs at least **2 steps**. Firstly, the OCR model is used to extract the text and its position. Secondly, the KIE model is used to extract the key information according to the image, text position and text content.
|
||||
|
||||
### 2.1 Train OCR Models
|
||||
|
||||
#### 2.1.1 Text Detection
|
||||
|
||||
##### (1) Data
|
||||
|
||||
Most of the models provided in PaddleOCR are general models. In the process of text detection, the detection of adjacent text lines is generally based on the distance of the position. As shown in the figure above, when using PP-OCRv3 general English detection model for text detection, it is easy to detect the two fields representing different properties as one. Therefore, it is suggested to finetune a detection model according to your scenario firstly during the KIE task.
|
||||
|
||||
During data annotation, the different key information needs to be separated. Otherwise, it will increase the difficulty of subsequent KIE tasks.
|
||||
|
||||
For downstream tasks, generally speaking, `200~300` training images can guarantee the basic training effect. If there is not too much prior knowledge, **`200~300`** images can be labeled firstly for subsequent text detection model training.
|
||||
|
||||
##### (2) Model
|
||||
|
||||
In terms of model selection, PP-OCRv3 detection model is recommended. For more information about the training methods of the detection model, please refer to: [Text detection tutorial](../../doc/doc_en/detection_en.md) and [PP-OCRv3 detection model tutorial](../../doc/doc_ch/PPOCRv3_det_train.md).
|
||||
|
||||
#### 2.1.2 Text recognition
|
||||
|
||||
Compared with the natural scene, the text recognition in the document image is generally relatively easier (the background is not too complex), so **it is suggested to** try the PP-OCRv3 general text recognition model provided in PaddleOCR ([PP-OCRv3 model list](../../doc/doc_en/models_list_en.md))
|
||||
|
||||
##### (1) Data
|
||||
|
||||
However, there are also some challenges in some document scenarios, such as rare words in ID card scenarios and special fonts in invoice and other scenarios. These problems will increase the difficulty of text recognition. At this time, if you want to ensure or further improve the model accuracy, it is recommended to load PP-OCRv3 model based on the text recognition dataset of specific document scenarios for finetuning.
|
||||
|
||||
In the process of model finetuning, it is recommended to prepare at least `5000` vertical scene text recognition images to ensure the basic model fine-tuning effect. If you want to improve the accuracy and generalization ability of the model, you can synthesize more text recognition images similar to the scene, collect general real text recognition data from the public data set, and add them to the text recognition training process. In the training process, it is suggested that the ratio of real data, synthetic data and general data of each epoch should be around `1:1:1`, which can be controlled by setting the sampling ratio of different data sources. If there are 3 training text files, including 10k, 20k and 50k pieces of data respectively, the data can be set in the configuration file as follows:
|
||||
|
||||
```yaml linenums="1" linenums="1"
|
||||
Train:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
data_dir: ./train_data/
|
||||
label_file_list:
|
||||
- ./train_data/train_list_10k.txt
|
||||
- ./train_data/train_list_10k.txt
|
||||
- ./train_data/train_list_50k.txt
|
||||
ratio_list: [1.0, 0.5, 0.2]
|
||||
...
|
||||
```
|
||||
|
||||
##### (2) Model
|
||||
|
||||
In terms of model selection, PP-OCRv3 recognition model is recommended. For more information about the training methods of the recognition model, please refer to: [Text recognition tutorial](../../doc/doc_en/recognition_en.md) and [PP-OCRv3 model list](../../doc/doc_en/models_list_en.md).
|
||||
|
||||
### 2.2 Train KIE Models
|
||||
|
||||
There are two main methods to extract the key information from the recognized texts.
|
||||
|
||||
(1) Directly use SER model to obtain the key information category. For example, in the ID card scenario, we mark "name" and "Geoff Sample" as "name_key" and "name_value", respectively. The **text field** corresponding to the category "name_value" finally identified is the key information we need.
|
||||
|
||||
(2) Joint use SER and RE models. For this case, we firstly use SER model to obtain all questions (keys) and questions (values) for the image text, and then use RE model to match all keys and values to find the relationship, so as to complete the extraction of key information.
|
||||
|
||||
#### 2.2.1 SER
|
||||
|
||||
Take the ID card scenario as an example. The key information generally includes `name`, `DOB`, etc. We can directly mark the corresponding fields as specific categories, as shown in the following figure.
|
||||
|
||||
<div align="center">
|
||||
<img src="https://user-images.githubusercontent.com/14270174/185728456-dc396f47-0880-4279-9c7c-c99601bf16a7.png" width="500">
|
||||
</div>
|
||||
|
||||
**Note:**
|
||||
|
||||
* In the labeling process, text content without key information about KIE shall be labeled as`other`, which is equivalent to background information. For example, in the ID card scenario, if we do not pay attention to `DOB` information, we can mark the categories of `DOB` and `Area manager` as `other`.
|
||||
* In the annotation process of, it is required to annotate the **textline** position rather than the character.
|
||||
|
||||
In terms of data, generally speaking, for relatively fixed scenes, **50** training images can achieve acceptable effects. You can refer to [PPOCRLabel](https://github.com/PFCCLab/PPOCRLabel/blob/main/README.md) for finish the labeling process.
|
||||
|
||||
In terms of model, it is recommended to use the VI-layoutXLM model proposed in PP-StructureV2. It is improved based on the LayoutXLM model, removing the visual feature extraction module, and further improving the model inference speed without the significant reduction on model accuracy. For more tutorials, please refer to [VI-LayoutXLM introduction](../../doc/doc_en/algorithm_kie_vi_layoutxlm_en.md) and [KIE tutorial](../../doc/doc_en/kie_en.md).
|
||||
|
||||
#### 2.2.2 SER + RE
|
||||
|
||||
The SER model is mainly used to identify all keys and values in the document image, and the RE model is mainly used to match all keys and values.
|
||||
|
||||
Taking the ID card scenario as an example, the key information generally includes key information such as `name`, `DOB`, etc. in the SER stage, we need to identify all questions (keys) and answers (values). The demo annotation is as follows. All keys can be annotated as `question`, and all values can be annotated as `answer`.
|
||||
|
||||
|
||||
|
||||
In the RE stage, the ID and connection information of each field need to be marked, as shown in the following figure.
|
||||
|
||||
|
||||
|
||||
For each textline, you need to add 'ID' and 'linking' field information. The 'ID' records the unique identifier of the textline. Different text contents in the same images cannot be repeated. The 'linking' is a list that records the connection information between different texts. If the ID of the field "name" is 0 and the ID of the field "Geoff Sample" is 1, then they all have [[0, 1]] 'linking' marks, indicating that the fields with `id=0` and `id=1` form a key value relationship (the fields such as DOB and Expires are similar, and will not be repeated here).
|
||||
|
||||
**Note:**
|
||||
|
||||
During annotation, if value is multiple text lines, a key-value pair can be added in linking, such as `[[0, 1], [0, 2]]`.
|
||||
|
||||
In terms of data, generally speaking, for relatively fixed scenes, about **50** training images can achieve acceptable effects.
|
||||
|
||||
In terms of model, it is recommended to use the VI-layoutXLM model proposed in PP-StructureV2. It is improved based on the LayoutXLM model, removing the visual feature extraction module, and further improving the model inference speed without the significant reduction on model accuracy. For more tutorials, please refer to [VI-LayoutXLM introduction](../../doc/doc_en/algorithm_kie_vi_layoutxlm_en.md) and [KIE tutorial](../../doc/doc_en/kie_en.md).
|
||||
|
||||
## 3. Reference
|
||||
|
||||
[1] Katti A R, Reisswig C, Guder C, et al. Chargrid: Towards understanding 2d documents[J]. arXiv preprint arXiv:1809.08799, 2018.
|
||||
|
||||
[2] Xu Y, Li M, Cui L, et al. Layoutlm: Pre-training of text and layout for document image understanding[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020: 1192-1200.
|
||||
|
||||
[3] Xu Y, Xu Y, Lv T, et al. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding[J]. arXiv preprint arXiv:2012.14740, 2020.
|
||||
|
||||
[4]: Xu Y, Lv T, Cui L, et al. Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding[J]. arXiv preprint arXiv:2104.08836, 2021.
|
||||
|
||||
[5] Li Y, Qian Y, Yu Y, et al. StrucTexT: Structured Text Understanding with Multi-Modal Transformers[C]//Proceedings of the 29th ACM International Conference on Multimedia. 2021: 1912-1920.
|
||||
|
||||
[6] Liu X, Gao F, Zhang Q, et al. Graph convolution for multimodal information extraction from visually rich documents[J]. arXiv preprint arXiv:1903.11279, 2019.
|
||||
|
||||
[7] Sun H, Kuang Z, Yue X, et al. Spatial Dual-Modality Graph Reasoning for Key Information Extraction[J]. arXiv preprint arXiv:2103.14470, 2021.
|
||||
|
||||
[8] Zhang P, Xu Y, Cheng Z, et al. Trie: End-to-end text reading and information extraction for document understanding[C]//Proceedings of the 28th ACM International Conference on Multimedia. 2020: 1413-1422.
|
||||
146
docs/version2.x/ppstructure/blog/how_to_do_kie.md
Normal file
146
docs/version2.x/ppstructure/blog/how_to_do_kie.md
Normal file
@@ -0,0 +1,146 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 怎样完成基于图像数据的信息抽取任务
|
||||
|
||||
## 1. 简介
|
||||
|
||||
### 1.1 背景
|
||||
|
||||
关键信息抽取 (Key Information Extraction, KIE)指的是是从文本或者图像中，抽取出关键的信息。针对文档图像的关键信息抽取任务作为OCR的下游任务，存在非常多的实际应用场景，如表单识别、车票信息抽取、身份证信息抽取等。然而，使用人力从这些文档图像中提取或者收集关键信息耗时费力，怎样自动化融合图像中的视觉、布局、文字等特征并完成关键信息抽取是一个价值与挑战并存的问题。
|
||||
|
||||
对于特定场景的文档图像，其中的关键信息位置、版式等较为固定，因此在研究早期有很多基于模板匹配的方法进行关键信息的抽取，考虑到其流程较为简单，该方法仍然被广泛应用在目前的很多场景中。但是这种基于模板匹配的方法在应用到不同的场景中时，需要耗费大量精力去调整与适配模板，迁移成本较高。
|
||||
|
||||
文档图像中的KIE一般包含2个子任务，示意图如下图所示。
|
||||
|
||||
* （1）SER: 语义实体识别 (Semantic Entity Recognition)，对每一个检测到的文本进行分类，如将其分为姓名，身份证。如下图中的黑色框和红色框。
|
||||
* （2）RE: 关系抽取 (Relation Extraction)，对每一个检测到的文本进行分类，如将其分为问题 (key) 和答案 (value) 。然后对每一个问题找到对应的答案，相当于完成key-value的匹配过程。如下图中的红色框和黑色框分别代表问题和答案，黄色线代表问题和答案之间的对应关系。
|
||||
|
||||
|
||||
|
||||
### 1.2 基于深度学习的主流方法
|
||||
|
||||
一般的KIE方法基于命名实体识别(Named Entity Recognition,NER)来展开研究，但是此类方法仅使用了文本信息而忽略了位置与视觉特征信息，因此精度受限。近几年大多学者开始融合多个模态的输入信息，进行特征融合，并对多模态信息进行处理，从而提升KIE的精度。主要方法有以下几种
|
||||
|
||||
* （1）基于Grid的方法：此类方法主要关注图像层面多模态信息的融合，文本大多大多为字符粒度，对文本与结构结构信息的嵌入方式较为简单，如Chargrid[1]等算法。
|
||||
* （2）基于Token的方法：此类方法参考NLP中的BERT等方法，将位置、视觉等特征信息共同编码到多模态模型中，并且在大规模数据集上进行预训练，从而在下游任务中，仅需要少量的标注数据便可以获得很好的效果。如LayoutLM[2], LayoutLMv2[3], LayoutXLM[4], StrucText[5]等算法。
|
||||
* （3）基于GCN的方法：此类方法尝试学习图像、文字之间的结构信息，从而可以解决开集信息抽取的问题（训练集中没有见过的模板），如GCN[6]、SDMGR[7]等算法。
|
||||
* （4）基于End-to-end的方法：此类方法将现有的OCR文字识别以及KIE信息抽取2个任务放在一个统一的网络中进行共同学习，并在学习过程中相互加强。如Trie[8]等算法。
|
||||
|
||||
更多关于该系列算法的详细介绍，请参考“动手学OCR·十讲”课程的课节六部分：[文档分析理论与实践](https://aistudio.baidu.com/aistudio/education/group/info/25207)。
|
||||
|
||||
## 2. 关键信息抽取任务流程
|
||||
|
||||
PaddleOCR中实现了LayoutXLM等算法（基于Token），同时，在PP-StructureV2中，对LayoutXLM多模态预训练模型的网络结构进行简化，去除了其中的Visual backbone部分，设计了视觉无关的VI-LayoutXLM模型，同时引入符合人类阅读顺序的排序逻辑以及UDML知识蒸馏策略，最终同时提升了关键信息抽取模型的精度与推理速度。
|
||||
|
||||
下面介绍怎样基于PaddleOCR完成关键信息抽取任务。
|
||||
|
||||
在非End-to-end的KIE方法中，完成关键信息抽取，至少需要**2个步骤**：首先使用OCR模型，完成文字位置与内容的提取，然后使用KIE模型，根据图像、文字位置以及文字内容，提取出其中的关键信息。
|
||||
|
||||
### 2.1 训练OCR模型
|
||||
|
||||
#### 2.1.1 文本检测
|
||||
|
||||
##### （1）数据
|
||||
|
||||
PaddleOCR中提供的模型大多数为通用模型，在进行文本检测的过程中，相邻文本行的检测一般是根据位置的远近进行区分，如上图，使用PP-OCRv3通用中英文检测模型进行文本检测时，容易将”民族“与“汉”这2个代表不同的字段检测到一起，从而增加后续KIE任务的难度。因此建议在做KIE任务的过程中，首先训练一个针对该文档数据集的检测模型。
|
||||
|
||||
在数据标注时，关键信息的标注需要隔开，比上图中的 “民族汉” 3个字相隔较近，此时需要将”民族“与”汉“标注为2个文本检测框，否则会增加后续KIE任务的难度。
|
||||
|
||||
对于下游任务，一般来说，`200~300`张的文本训练数据即可保证基本的训练效果，如果没有太多的先验知识，可以先标注 **`200~300`** 张图片，进行后续文本检测模型的训练。
|
||||
|
||||
##### （2）模型
|
||||
|
||||
在模型选择方面，推荐使用PP-OCRv3_det，关于更多关于检测模型的训练方法介绍，请参考：[OCR文本检测模型训练教程](../../ppocr/model_train/detection.md)。
|
||||
|
||||
#### 2.1.2 文本识别
|
||||
|
||||
相对自然场景，文档图像中的文本内容识别难度一般相对较低（背景相对不太复杂），因此**优先建议**尝试PaddleOCR中提供的PP-OCRv3通用文本识别模型([PP-OCRv3模型库链接](../../ppocr/model_list.md))。
|
||||
|
||||
##### （1）数据
|
||||
|
||||
然而，在部分文档场景中也会存在一些挑战，如身份证场景中存在着罕见字，在发票等场景中的字体比较特殊，这些问题都会增加文本识别的难度，此时如果希望保证或者进一步提升模型的精度，建议基于特定文档场景的文本识别数据集，加载PP-OCRv3模型进行微调。
|
||||
|
||||
在模型微调的过程中，建议准备至少`5000`张垂类场景的文本识别图像，可以保证基本的模型微调效果。如果希望提升模型的精度与泛化能力，可以合成更多与该场景类似的文本识别数据，从公开数据集中收集通用真实文本识别数据，一并添加到该场景的文本识别训练任务过程中。在训练过程中，建议每个epoch的真实垂类数据、合成数据、通用数据比例在`1:1:1`左右，这可以通过设置不同数据源的采样比例进行控制。如有3个训练文本文件，分别包含1W、2W、5W条数据，那么可以在配置文件中设置数据如下：
|
||||
|
||||
```yaml linenums="1" linenums="1"
|
||||
Train:
|
||||
dataset:
|
||||
name: SimpleDataSet
|
||||
data_dir: ./train_data/
|
||||
label_file_list:
|
||||
- ./train_data/train_list_1W.txt
|
||||
- ./train_data/train_list_2W.txt
|
||||
- ./train_data/train_list_5W.txt
|
||||
ratio_list: [1.0, 0.5, 0.2]
|
||||
...
|
||||
```
|
||||
|
||||
##### （2）模型
|
||||
|
||||
在模型选择方面，推荐使用通用中英文文本识别模型PP-OCRv3_rec，关于更多关于文本识别模型的训练方法介绍，请参考：[OCR文本识别模型训练教程](../../ppocr/model_train/recognition.md)。
|
||||
|
||||
### 2.2 训练KIE模型
|
||||
|
||||
对于识别得到的文字进行关键信息抽取，有2种主要的方法。
|
||||
|
||||
（1）直接使用SER，获取关键信息的类别：如身份证场景中，将“姓名“与”张三“分别标记为`name_key`与`name_value`。最终识别得到的类别为`name_value`对应的**文本字段**即为我们所需要的关键信息。
|
||||
|
||||
（2）联合SER与RE进行使用：这种方法中，首先使用SER，获取图像文字内容中所有的key与value，然后使用RE方法，对所有的key与value进行配对，找到映射关系，从而完成关键信息的抽取。
|
||||
|
||||
#### 2.2.1 SER
|
||||
|
||||
以身份证场景为例， 关键信息一般包含`姓名`、`性别`、`民族`等，我们直接将对应的字段标注为特定的类别即可，如下图所示。
|
||||
|
||||
|
||||
|
||||
**注意：**
|
||||
|
||||
* 标注过程中，对于无关于KIE关键信息的文本内容，均需要将其标注为`other`类别，相当于背景信息。如在身份证场景中，如果我们不关注性别信息，那么可以将“性别”与“男”这2个字段的类别均标注为`other`。
|
||||
* 标注过程中，需要以**文本行**为单位进行标注，无需标注单个字符的位置信息。
|
||||
|
||||
数据量方面，一般来说，对于比较固定的场景，**50张**左右的训练图片即可达到可以接受的效果，可以使用[PPOCRLabel](https://github.com/PFCCLab/PPOCRLabel/blob/main/README_ch.md)完成KIE的标注过程。
|
||||
|
||||
模型方面，推荐使用PP-StructureV2中提出的VI-LayoutXLM模型，它基于LayoutXLM模型进行改进，去除其中的视觉特征提取模块，在精度基本无损的情况下，进一步提升了模型推理速度。更多教程请参考：[VI-LayoutXLM算法介绍](../../algorithm/kie/algorithm_kie_layoutxlm.md)与[KIE关键信息抽取使用教程](../model_train/train_kie.md)。
|
||||
|
||||
#### 2.2.2 SER + RE
|
||||
|
||||
该过程主要包含SER与RE 2个过程。SER阶段主要用于识别出文档图像中的所有key与value，RE阶段主要用于对所有的key与value进行匹配。
|
||||
|
||||
以身份证场景为例， 关键信息一般包含`姓名`、`性别`、`民族`等关键信息，在SER阶段，我们需要识别所有的question (key) 与answer (value) 。标注如下所示。每个字段的类别信息（`label`字段）可以是question、answer或者other（与待抽取的关键信息无关的字段）
|
||||
|
||||
|
||||
|
||||
在RE阶段，需要标注每个字段的的id与连接信息，如下图所示。
|
||||
|
||||
|
||||
|
||||
每个文本行字段中，需要添加`id`与`linking`字段信息，`id`记录该文本行的唯一标识，同一张图片中的不同文本内容不能重复，`linking`是一个列表，记录了不同文本之间的连接信息。如字段“出生”的id为0，字段“1996年1月11日”的id为1，那么它们均有[[0, 1]]的`linking`标注，表示该id=0与id=1的字段构成key-value的关系（姓名、性别等字段类似，此处不再一一赘述）。
|
||||
|
||||
**注意：**
|
||||
|
||||
* 标注过程中，如果value是多个字符，那么linking中可以新增一个key-value对，如`[[0, 1], [0, 2]]`
|
||||
|
||||
数据量方面，一般来说，对于比较固定的场景，**50张**左右的训练图片即可达到可以接受的效果，可以使用PPOCRLabel完成KIE的标注过程。
|
||||
|
||||
模型方面，推荐使用PP-StructureV2中提出的VI-LayoutXLM模型，它基于LayoutXLM模型进行改进，去除其中的视觉特征提取模块，在精度基本无损的情况下，进一步提升了模型推理速度。更多教程请参考：[VI-LayoutXLM算法介绍](../../algorithm/kie/algorithm_kie_layoutxlm.md)与[KIE关键信息抽取使用教程](../model_train/train_kie.md)。
|
||||
|
||||
## 3. 参考文献
|
||||
|
||||
[1] Katti A R, Reisswig C, Guder C, et al. Chargrid: Towards understanding 2d documents[J]. arXiv preprint arXiv:1809.08799, 2018.
|
||||
|
||||
[2] Xu Y, Li M, Cui L, et al. Layoutlm: Pre-training of text and layout for document image understanding[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020: 1192-1200.
|
||||
|
||||
[3] Xu Y, Xu Y, Lv T, et al. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding[J]. arXiv preprint arXiv:2012.14740, 2020.
|
||||
|
||||
[4]: Xu Y, Lv T, Cui L, et al. Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding[J]. arXiv preprint arXiv:2104.08836, 2021.
|
||||
|
||||
[5] Li Y, Qian Y, Yu Y, et al. StrucTexT: Structured Text Understanding with Multi-Modal Transformers[C]//Proceedings of the 29th ACM International Conference on Multimedia. 2021: 1912-1920.
|
||||
|
||||
[6] Liu X, Gao F, Zhang Q, et al. Graph convolution for multimodal information extraction from visually rich documents[J]. arXiv preprint arXiv:1903.11279, 2019.
|
||||
|
||||
[7] Sun H, Kuang Z, Yue X, et al. Spatial Dual-Modality Graph Reasoning for Key Information Extraction[J]. arXiv preprint arXiv:2103.14470, 2021.
|
||||
|
||||
[8] Zhang P, Xu Y, Cheng Z, et al. Trie: End-to-end text reading and information extraction for document understanding[C]//Proceedings of the 28th ACM International Conference on Multimedia. 2020: 1413-1422.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 240 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 161 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 136 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 631 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 385 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 560 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 534 KiB |
100
docs/version2.x/ppstructure/blog/return_word_pos.en.md
Normal file
100
docs/version2.x/ppstructure/blog/return_word_pos.en.md
Normal file
@@ -0,0 +1,100 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Return recognition position
|
||||
|
||||
According to the horizontal document, the recognition model not only returns the recognized content, but also the position of each word.
|
||||
|
||||
## English document recovery
|
||||
|
||||
### Download the inference model first
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
|
||||
## download model
|
||||
mkdir inference && cd inference
|
||||
## Download the detection model of the ultra-lightweight English PP-OCRv3 model and unzip it
|
||||
https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar && tar xf en_PP-OCRv3_det_infer.tar
|
||||
## Download the recognition model of the ultra-lightweight English PP-OCRv3 model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/en_PP-OCRv3_mobile_rec_infer.tar && tar xf en_PP-OCRv3_mobile_rec_infer.tar
|
||||
## Download the ultra-lightweight English table inch model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
tar xf en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
## Download the layout model of publaynet dataset and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
tar xf picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
### Then use the following command inference in the /ppstructure/ directory
|
||||
|
||||
```bash linenums="1"
|
||||
python predict_system.py \
|
||||
--image_dir=../docs/ppstructure/images/table_1.png \
|
||||
--det_model_dir=inference/en_PP-OCRv3_det_infer \
|
||||
--rec_model_dir=inference/en_PP-OCRv3_mobile_rec_infer \
|
||||
--rec_char_dict_path=../ppocr/utils/en_dict.txt \
|
||||
--table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
|
||||
--layout_model_dir=inference/picodet_lcnet_x1_0_fgd_layout_infer \
|
||||
--layout_dict_path=../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--recovery=True \
|
||||
--output=../output/ \
|
||||
--return_word_box=True
|
||||
```
|
||||
|
||||
### View the visualization of the inference results under `../output/structure/table_1/show_0.jpg`, as shown below
|
||||
|
||||
|
||||
|
||||
## Recover Chinese documents
|
||||
|
||||
### Download the inference model first
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
|
||||
## download model
|
||||
cd inference
|
||||
## Download the detection model of the ultra-lightweight Chinese PP-OCRv3 model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_det_infer.tar && tar xf PP-OCRv3_mobile_det_infer.tar
|
||||
## Download the recognition model of the ultra-lightweight Chinese PP-OCRv3 model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_rec_infer.tar && tar xf PP-OCRv3_mobile_rec_infer.tar
|
||||
## Download the ultra-lightweight Chinese table inch model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
## Download the layout model of CDLA dataset and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar
|
||||
tar xf picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
### Upload the following test image "2.png" to the directory ./docs/table/
|
||||
|
||||
|
||||
|
||||
### Then use the following command inference in the /ppstructure/ directory
|
||||
|
||||
```bash linenums="1"
|
||||
python predict_system.py \
|
||||
--image_dir=./docs/table/2.png \
|
||||
--det_model_dir=inference/PP-OCRv3_mobile_det_infer \
|
||||
--rec_model_dir=inference/PP-OCRv3_mobile_rec_infer \
|
||||
--rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
|
||||
--table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
|
||||
--layout_model_dir=inference/picodet_lcnet_x1_0_fgd_layout_cdla_infer \
|
||||
--layout_dict_path=../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt \
|
||||
--vis_font_path=../doc/fonts/chinese_cht.ttf \
|
||||
--recovery=True \
|
||||
--output=../output/ \
|
||||
--return_word_box=True
|
||||
```
|
||||
|
||||
### View the visualization of the inference results under `../output/structure/2/show_0.jpg`, as shown below
|
||||
|
||||
|
||||
100
docs/version2.x/ppstructure/blog/return_word_pos.md
Normal file
100
docs/version2.x/ppstructure/blog/return_word_pos.md
Normal file
@@ -0,0 +1,100 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 返回识别位置
|
||||
|
||||
根据横排的文档，识别模型不仅返回识别的内容，还返回每个文字的位置。
|
||||
|
||||
## 英文文档恢复
|
||||
|
||||
### 先下载推理模型
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
|
||||
## download model
|
||||
mkdir inference && cd inference
|
||||
## Download the detection model of the ultra-lightweight English PP-OCRv3 model and unzip it
|
||||
https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar && tar xf en_PP-OCRv3_det_infer.tar
|
||||
## Download the recognition model of the ultra-lightweight English PP-OCRv3 model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/en_PP-OCRv3_mobile_rec_infer.tar && tar xf en_PP-OCRv3_mobile_rec_infer.tar
|
||||
## Download the ultra-lightweight English table inch model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
tar xf en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
## Download the layout model of publaynet dataset and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
tar xf picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
### 然后在/ppstructure/目录下使用下面的指令推理
|
||||
|
||||
```bash linenums="1"
|
||||
python predict_system.py \
|
||||
--image_dir=./docs/ppstructure/images/table_1.png \
|
||||
--det_model_dir=inference/en_PP-OCRv3_det_infer \
|
||||
--rec_model_dir=inference/en_PP-OCRv3_mobile_rec_infer \
|
||||
--rec_char_dict_path=../ppocr/utils/en_dict.txt \
|
||||
--table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
|
||||
--layout_model_dir=inference/picodet_lcnet_x1_0_fgd_layout_infer \
|
||||
--layout_dict_path=../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--recovery=True \
|
||||
--output=../output/ \
|
||||
--return_word_box=True
|
||||
```
|
||||
|
||||
### 在`../output/structure/table_1/show_0.jpg`下查看推理结果的可视化，如下图所示
|
||||
|
||||
|
||||
|
||||
## 针对中文文档恢复
|
||||
|
||||
### 先下载推理模型
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
|
||||
## download model
|
||||
cd inference
|
||||
## Download the detection model of the ultra-lightweight Chinese PP-OCRv3 model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_det_infer.tar && tar xf PP-OCRv3_mobile_det_infer.tar
|
||||
## Download the recognition model of the ultra-lightweight Chinese PP-OCRv3 model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_rec_infer.tar && tar xf PP-OCRv3_mobile_rec_infer.tar
|
||||
## Download the ultra-lightweight Chinese table inch model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
## Download the layout model of CDLA dataset and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar
|
||||
tar xf picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
### 上传下面的测试图片 "2.png" 至目录 ./docs/table/ 中
|
||||
|
||||
|
||||
|
||||
### 然后在/ppstructure/目录下使用下面的指令推理
|
||||
|
||||
```bash linenums="1"
|
||||
python predict_system.py \
|
||||
--image_dir=./docs/table/2.png \
|
||||
--det_model_dir=inference/PP-OCRv3_mobile_det_infer \
|
||||
--rec_model_dir=inference/PP-OCRv3_mobile_rec_infer \
|
||||
--rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
|
||||
--table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
|
||||
--layout_model_dir=inference/picodet_lcnet_x1_0_fgd_layout_cdla_infer \
|
||||
--layout_dict_path=../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt \
|
||||
--vis_font_path=../doc/fonts/chinese_cht.ttf \
|
||||
--recovery=True \
|
||||
--output=../output/ \
|
||||
--return_word_box=True
|
||||
```
|
||||
|
||||
### 在`../output/structure/2/show_0.jpg`下查看推理结果的可视化，如下图所示
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 234 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 143 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 125 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 347 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 247 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 189 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 215 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 221 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 205 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 344 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 372 KiB |
BIN
docs/version2.x/ppstructure/images/ppstructure.gif
Normal file
BIN
docs/version2.x/ppstructure/images/ppstructure.gif
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 2.5 MiB |
BIN
docs/version2.x/ppstructure/images/recovery.jpg
Normal file
BIN
docs/version2.x/ppstructure/images/recovery.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 385 KiB |
BIN
docs/version2.x/ppstructure/images/table_1.png
Normal file
BIN
docs/version2.x/ppstructure/images/table_1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 758 KiB |
433
docs/version2.x/ppstructure/infer_deploy/cpp_infer.en.md
Normal file
433
docs/version2.x/ppstructure/infer_deploy/cpp_infer.en.md
Normal file
@@ -0,0 +1,433 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Server-side C++ Inference
|
||||
|
||||
This chapter introduces the C++ deployment steps of the PaddleOCR model. C++ is better than Python in terms of performance. Therefore, in CPU and GPU deployment scenarios, C++ deployment is mostly used.
|
||||
This section will introduce how to configure the C++ environment and deploy PaddleOCR in Linux (CPU\GPU) environment. For Windows deployment please refer to [Windows](../../ppocr/infer_deploy/windows_vs2019_build.en.md) compilation guidelines.
|
||||
|
||||
## 1. Prepare the Environment
|
||||
|
||||
### 1.1 Environment
|
||||
|
||||
- Linux, docker is recommended.
|
||||
- Windows.
|
||||
|
||||
### 1.2 Compile OpenCV
|
||||
|
||||
- First of all, you need to download the source code compiled package in the Linux environment from the OpenCV official website. Taking OpenCV 3.4.7 as an example, the download command is as follows.
|
||||
|
||||
```bash linenums="1"
|
||||
cd deploy/cpp_infer
|
||||
wget https://paddleocr.bj.bcebos.com/libs/opencv/opencv-3.4.7.tar.gz
|
||||
tar -xf opencv-3.4.7.tar.gz
|
||||
```
|
||||
|
||||
Finally, you will see the folder of `opencv-3.4.7/` in the current directory.
|
||||
|
||||
- Compile OpenCV, the OpenCV source path (`root_path`) and installation path (`install_path`) should be set by yourself. Enter the OpenCV source code path and compile it in the following way.
|
||||
|
||||
```bash linenums="1"
|
||||
root_path=your_opencv_root_path
|
||||
install_path=${root_path}/opencv3
|
||||
|
||||
rm -rf build
|
||||
mkdir build
|
||||
cd build
|
||||
|
||||
cmake .. \
|
||||
-DCMAKE_INSTALL_PREFIX=${install_path} \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DBUILD_SHARED_LIBS=OFF \
|
||||
-DWITH_IPP=OFF \
|
||||
-DBUILD_IPP_IW=OFF \
|
||||
-DWITH_LAPACK=OFF \
|
||||
-DWITH_EIGEN=OFF \
|
||||
-DCMAKE_INSTALL_LIBDIR=lib64 \
|
||||
-DWITH_ZLIB=ON \
|
||||
-DBUILD_ZLIB=ON \
|
||||
-DWITH_JPEG=ON \
|
||||
-DBUILD_JPEG=ON \
|
||||
-DWITH_PNG=ON \
|
||||
-DBUILD_PNG=ON \
|
||||
-DWITH_TIFF=ON \
|
||||
-DBUILD_TIFF=ON
|
||||
|
||||
make -j
|
||||
make install
|
||||
```
|
||||
|
||||
In the above commands, `root_path` is the downloaded OpenCV source code path, and `install_path` is the installation path of OpenCV. After `make install` is completed, the OpenCV header file and library file will be generated in this folder for later OCR source code compilation.
|
||||
|
||||
The final file structure under the OpenCV installation path is as follows.
|
||||
|
||||
```
|
||||
opencv3/
|
||||
|-- bin
|
||||
|-- include
|
||||
|-- lib
|
||||
|-- lib64
|
||||
|-- share
|
||||
```
|
||||
|
||||
### 1.3 Compile or Download or the Paddle Inference Library
|
||||
|
||||
- There are 2 ways to obtain the Paddle inference library, described in detail below.
|
||||
|
||||
#### 1.3.1 Direct download and installation
|
||||
|
||||
[Paddle inference library official website](https://www.paddlepaddle.org.cn/inference/master/guides/install/download_lib.html#linux). You can review and select the appropriate version of the inference library on the official website.
|
||||
|
||||
- After downloading, use the following command to extract files.
|
||||
|
||||
```bash linenums="1"
|
||||
tar -xf paddle_inference.tgz
|
||||
```
|
||||
|
||||
Finally you will see the folder of `paddle_inference/` in the current path.
|
||||
|
||||
#### 1.3.2 Compile the inference source code
|
||||
|
||||
- If you want to get the latest Paddle inference library features, you can download the latest code from Paddle GitHub repository and compile the inference library from the source code. It is recommended to download the inference library with paddle version greater than or equal to 2.0.1.
|
||||
|
||||
- You can refer to [Paddle inference library](https://www.paddlepaddle.org.cn/documentation/docs/en/advanced_guide/inference_deployment/inference/build_and_install_lib_en.html) to get the Paddle source code from GitHub, and then compile To generate the latest inference library. The method of using git to access the code is as follows.
|
||||
|
||||
```bash linenums="1"
|
||||
git clone https://github.com/PaddlePaddle/Paddle.git
|
||||
git checkout develop
|
||||
```
|
||||
|
||||
- Enter the Paddle directory and run the following commands to compile the paddle inference library.
|
||||
|
||||
```bash linenums="1"
|
||||
rm -rf build
|
||||
mkdir build
|
||||
cd build
|
||||
|
||||
cmake .. \
|
||||
-DWITH_CONTRIB=OFF \
|
||||
-DWITH_MKL=ON \
|
||||
-DWITH_MKLDNN=ON \
|
||||
-DWITH_TESTING=OFF \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DWITH_INFERENCE_API_TEST=OFF \
|
||||
-DON_INFER=ON \
|
||||
-DWITH_PYTHON=ON
|
||||
make -j
|
||||
make inference_lib_dist
|
||||
```
|
||||
|
||||
For more compilation parameter options, please refer to the [document](https://www.paddlepaddle.org.cn/documentation/docs/zh/2.0/guides/05_inference_deployment/inference/build_and_install_lib_cn.html#congyuanmabianyi).
|
||||
|
||||
- After the compilation process, you can see the following files in the folder of `build/paddle_inference_install_dir/`.
|
||||
|
||||
```text linenums="1"
|
||||
build/paddle_inference_install_dir/
|
||||
|-- CMakeCache.txt
|
||||
|-- paddle
|
||||
|-- third_party
|
||||
|-- version.txt
|
||||
```
|
||||
|
||||
`paddle` is the Paddle library required for C++ prediction later, and `version.txt` contains the version information of the current inference library.
|
||||
|
||||
## 2. Compile and Run the Demo
|
||||
|
||||
### 2.1 Export the inference model
|
||||
|
||||
- You can refer to [Model inference](./python_infer.en.md) and export the inference model. After the model is exported, assuming it is placed in the `inference` directory, the directory structure is as follows.
|
||||
|
||||
```text linenums="1"
|
||||
inference/
|
||||
|-- det_db
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
|-- rec_rcnn
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
|-- cls
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
|-- table
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
|-- layout
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
```
|
||||
|
||||
### 2.2 Compile PaddleOCR C++ inference demo
|
||||
|
||||
- The compilation commands are as follows. The addresses of Paddle C++ inference library, opencv and other Dependencies need to be replaced with the actual addresses on your own machines.
|
||||
|
||||
```bash linenums="1"
|
||||
sh tools/build.sh
|
||||
```
|
||||
|
||||
Specifically, you should modify the paths in `tools/build.sh`. The related content is as follows.
|
||||
|
||||
```bash linenums="1"
|
||||
OPENCV_DIR=your_opencv_dir
|
||||
LIB_DIR=your_paddle_inference_dir
|
||||
CUDA_LIB_DIR=your_cuda_lib_dir
|
||||
CUDNN_LIB_DIR=your_cudnn_lib_dir
|
||||
```
|
||||
|
||||
`OPENCV_DIR` is the OpenCV installation path; `LIB_DIR` is the download (`paddle_inference` folder)
|
||||
or the generated Paddle inference library path (`build/paddle_inference_install_dir` folder);
|
||||
`CUDA_LIB_DIR` is the CUDA library file path, in docker; it is `/usr/local/cuda/lib64`; `CUDNN_LIB_DIR` is the cuDNN library file path, in docker it is `/usr/lib/x86_64-linux-gnu/`.
|
||||
|
||||
- After the compilation is completed, an executable file named `ppocr` will be generated in the `build` folder.
|
||||
|
||||
### 2.3 Run the demo
|
||||
|
||||
Execute the built executable file:
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr [--param1] [--param2] [...]
|
||||
```
|
||||
|
||||
**Note**:ppocr uses the `PP-OCRv3` model by default, and the input shape used by the recognition model is `3, 48, 320`, if you want to use the old version model, you should add the parameter `--rec_img_h=32`.
|
||||
|
||||
Specifically,
|
||||
|
||||
#### 1. det+cls+rec
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--rec_model_dir=inference/rec_rcnn \
|
||||
--cls_model_dir=inference/cls \
|
||||
--image_dir=../../doc/imgs/12.jpg \
|
||||
--use_angle_cls=true \
|
||||
--det=true \
|
||||
--rec=true \
|
||||
--cls=true \
|
||||
```
|
||||
|
||||
##### 2. det+rec
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--rec_model_dir=inference/rec_rcnn \
|
||||
--image_dir=../../doc/imgs/12.jpg \
|
||||
--use_angle_cls=false \
|
||||
--det=true \
|
||||
--rec=true \
|
||||
--cls=false \
|
||||
```
|
||||
|
||||
##### 3. det
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--image_dir=../../doc/imgs/12.jpg \
|
||||
--det=true \
|
||||
--rec=false
|
||||
```
|
||||
|
||||
##### 4. cls+rec
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --rec_model_dir=inference/rec_rcnn \
|
||||
--cls_model_dir=inference/cls \
|
||||
--image_dir=../../doc/imgs_words/ch/word_1.jpg \
|
||||
--use_angle_cls=true \
|
||||
--det=false \
|
||||
--rec=true \
|
||||
--cls=true \
|
||||
```
|
||||
|
||||
##### 5. rec
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --rec_model_dir=inference/rec_rcnn \
|
||||
--image_dir=../../doc/imgs_words/ch/word_1.jpg \
|
||||
--use_angle_cls=false \
|
||||
--det=false \
|
||||
--rec=true \
|
||||
--cls=false \
|
||||
```
|
||||
|
||||
##### 6. cls
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --cls_model_dir=inference/cls \
|
||||
--cls_model_dir=inference/cls \
|
||||
--image_dir=../../doc/imgs_words/ch/word_1.jpg \
|
||||
--use_angle_cls=true \
|
||||
--det=false \
|
||||
--rec=false \
|
||||
--cls=true \
|
||||
```
|
||||
|
||||
##### 7. layout+table
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--rec_model_dir=inference/rec_rcnn \
|
||||
--table_model_dir=inference/table \
|
||||
--image_dir=../../ppstructure/docs/table/table.jpg \
|
||||
--layout_model_dir=inference/layout \
|
||||
--type=structure \
|
||||
--table=true \
|
||||
--layout=true
|
||||
```
|
||||
|
||||
##### 8. layout
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --layout_model_dir=inference/layout \
|
||||
--image_dir=../../ppstructure/docs/table/1.png \
|
||||
--type=structure \
|
||||
--table=false \
|
||||
--layout=true \
|
||||
--det=false \
|
||||
--rec=false
|
||||
```
|
||||
|
||||
##### 9. table
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--rec_model_dir=inference/rec_rcnn \
|
||||
--table_model_dir=inference/table \
|
||||
--image_dir=../../ppstructure/docs/table/table.jpg \
|
||||
--type=structure \
|
||||
--table=true
|
||||
```
|
||||
|
||||
More parameters are as follows,
|
||||
|
||||
- Common parameters
|
||||
|
||||
|parameter|data type|default|meaning|
|
||||
| --- | --- | --- | --- |
|
||||
|use_gpu|bool|false|Whether to use GPU|
|
||||
|gpu_id|int|0|GPU id when use_gpu is true|
|
||||
|gpu_mem|int|4000|GPU memory requested|
|
||||
|cpu_math_library_num_threads|int|10|Number of threads when using CPU inference. When machine cores is enough, the large the value, the faster the inference speed|
|
||||
|enable_mkldnn|bool|true|Whether to use mkdlnn library|
|
||||
|output|str|./output|Path where visualization results are saved|
|
||||
|
||||
- forward
|
||||
|
||||
|parameter|data type|default|meaning|
|
||||
| :---: | :---: | :---: | :---: |
|
||||
|det|bool|true|Whether to perform text detection in the forward direction|
|
||||
|rec|bool|true|Whether to perform text recognition in the forward direction|
|
||||
|cls|bool|false|Whether to perform text direction classification in the forward direction|
|
||||
|
||||
- Detection related parameters
|
||||
|
||||
|parameter|data type|default|meaning|
|
||||
| --- | --- | --- | --- |
|
||||
|det_model_dir|string|-|Address of detection inference model|
|
||||
|max_side_len|int|960|Limit the maximum image height and width to 960|
|
||||
|det_db_thresh|float|0.3|Used to filter the binarized image of DB prediction, setting 0.-0.3 has no obvious effect on the result|
|
||||
|det_db_box_thresh|float|0.5|DB post-processing filter box threshold, if there is a missing box detected, it can be reduced as appropriate|
|
||||
|det_db_unclip_ratio|float|1.6|Indicates the compactness of the text box, the smaller the value, the closer the text box to the text|
|
||||
|det_db_score_mode|string|slow| slow: use polygon box to calculate bbox score, fast: use rectangle box to calculate. Use rectangular box to calculate faster, and polygonal box more accurate for curved text area.|
|
||||
|visualize|bool|true|Whether to visualize the results，when it is set as true, the prediction results will be saved in the folder specified by the `output` field on an image with the same name as the input image.|
|
||||
|
||||
- Classifier related parameters
|
||||
|
||||
|parameter|data type|default|meaning|
|
||||
| --- | --- | --- | --- |
|
||||
|use_angle_cls|bool|false|Whether to use the direction classifier|
|
||||
|cls_model_dir|string|-|Address of direction classifier inference model|
|
||||
|cls_thresh|float|0.9|Score threshold of the direction classifier|
|
||||
|cls_batch_num|int|1|batch size of classifier|
|
||||
|
||||
- Recognition related parameters
|
||||
|
||||
|parameter|data type|default|meaning|
|
||||
| --- | --- | --- | --- |
|
||||
|rec_model_dir|string|-|Address of recognition inference model|
|
||||
|rec_char_dict_path|string|../../ppocr/utils/ppocr_keys_v1.txt|dictionary file|
|
||||
|rec_batch_num|int|6|batch size of recognition|
|
||||
|rec_img_h|int|48|image height of recognition|
|
||||
|rec_img_w|int|320|image width of recognition|
|
||||
|
||||
- Layout related parameters
|
||||
|
||||
|parameter|data type|default|meaning|
|
||||
| :---: | :---: | :---: | :---: |
|
||||
|layout_model_dir|string|-| Address of layout inference model|
|
||||
|layout_dict_path|string|../../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt|dictionary file|
|
||||
|layout_score_threshold|float|0.5|Threshold of score.|
|
||||
|layout_nms_threshold|float|0.5|Threshold of nms.|
|
||||
|
||||
- Table recognition related parameters
|
||||
|
||||
|parameter|data type|default|meaning|
|
||||
| :---: | :---: | :---: | :---: |
|
||||
|table_model_dir|string|-|Address of table recognition inference model|
|
||||
|table_char_dict_path|string|../../ppocr/utils/dict/table_structure_dict.txt|dictionary file|
|
||||
|table_max_len|int|488|The size of the long side of the input image of the table recognition model, the final input image size of the network is（table_max_len，table_max_len）|
|
||||
|merge_no_span_structure|bool|true|Whether to merge <td> and </td> to <td></td|
|
||||
|
||||
- Multi-language inference is also supported in PaddleOCR, you can refer to [recognition tutorial](../../ppocr/blog/multi_languages.en.md) for more supported languages and models in PaddleOCR. Specifically, if you want to infer using multi-language models, you just need to modify values of `rec_char_dict_path` and `rec_model_dir`.
|
||||
|
||||
The detection results will be shown on the screen, which is as follows.
|
||||
|
||||
```bash linenums="1"
|
||||
predict img: ../../doc/imgs/12.jpg
|
||||
../../doc/imgs/12.jpg
|
||||
0 det boxes: [[74,553],[427,542],[428,571],[75,582]] rec text: 打浦路252935号 rec score: 0.947724
|
||||
1 det boxes: [[23,507],[513,488],[515,529],[24,548]] rec text: 绿洲仕格维花园公寓 rec score: 0.993728
|
||||
2 det boxes: [[187,456],[399,448],[400,480],[188,488]] rec text: 打浦路15号 rec score: 0.964994
|
||||
3 det boxes: [[42,413],[483,391],[484,428],[43,450]] rec text: 上海斯格威铂尔大酒店 rec score: 0.980086
|
||||
The detection visualized image saved in ./output//12.jpg
|
||||
```
|
||||
|
||||
- layout+table
|
||||
|
||||
```bash linenums="1"
|
||||
predict img: ../../ppstructure/docs/table/1.png
|
||||
0 type: text, region: [12,729,410,848], score: 0.781044, res: count of ocr result is : 7
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[4,1],[79,1],[79,12],[4,12]] rec text: CTW1500. rec score: 0.769472
|
||||
...
|
||||
6 det boxes: [[4,99],[391,99],[391,112],[4,112]] rec text: sate-of-the-artmethods[12.34.36l.ourapproachachieves rec score: 0.90414
|
||||
********** end print ocr result **********
|
||||
1 type: text, region: [69,342,342,359], score: 0.703666, res: count of ocr result is : 1
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[8,2],[269,2],[269,13],[8,13]] rec text: Table6.Experimentalresults on CTW-1500 rec score: 0.890454
|
||||
********** end print ocr result **********
|
||||
2 type: text, region: [70,316,706,332], score: 0.659738, res: count of ocr result is : 2
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[373,2],[630,2],[630,11],[373,11]] rec text: oroposals.andthegreencontoursarefinal rec score: 0.919729
|
||||
1 det boxes: [[8,3],[357,3],[357,11],[8,11]] rec text: Visualexperimentalresultshebluecontoursareboundar rec score: 0.915963
|
||||
********** end print ocr result **********
|
||||
3 type: text, region: [489,342,789,359], score: 0.630538, res: count of ocr result is : 1
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[8,2],[294,2],[294,14],[8,14]] rec text: Table7.Experimentalresults onMSRA-TD500 rec score: 0.942251
|
||||
********** end print ocr result **********
|
||||
4 type: text, region: [444,751,841,848], score: 0.607345, res: count of ocr result is : 5
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[19,3],[389,3],[389,17],[19,17]] rec text: Inthispaper,weproposeanovel adaptivebound rec score: 0.941031
|
||||
1 det boxes: [[4,22],[390,22],[390,36],[4,36]] rec text: aryproposalnetworkforarbitraryshapetextdetection rec score: 0.960172
|
||||
2 det boxes: [[4,42],[392,42],[392,56],[4,56]] rec text: whichadoptanboundaryproposalmodeltogeneratecoarse rec score: 0.934647
|
||||
3 det boxes: [[4,61],[389,61],[389,75],[4,75]] rec text: ooundaryproposals,andthenadoptanadaptiveboundary rec score: 0.946296
|
||||
4 det boxes: [[5,80],[387,80],[387,93],[5,93]] rec text: leformationmodelcombinedwithGCNandRNNtoper rec score: 0.952401
|
||||
********** end print ocr result **********
|
||||
5 type: title, region: [444,705,564,724], score: 0.785429, res: count of ocr result is : 1
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[6,2],[113,2],[113,14],[6,14]] rec text: 5.Conclusion rec score: 0.856903
|
||||
********** end print ocr result **********
|
||||
6 type: table, region: [14,360,402,711], score: 0.963643, res: <html><body><table><thead><tr><td>Methods</td><td>Ext</td><td>R</td><td>P</td><td>F</td><td>FPS</td></tr></thead><tbody><tr><td>TextSnake [18]</td><td>Syn</td><td>85.3</td><td>67.9</td><td>75.6</td><td></td></tr><tr><td>CSE [17]</td><td>MiLT</td><td>76.1</td><td>78.7</td><td>77.4</td><td>0.38</td></tr><tr><td>LOMO[40]</td><td>Syn</td><td>76.5</td><td>85.7</td><td>80.8</td><td>4.4</td></tr><tr><td>ATRR[35]</td><td>Sy-</td><td>80.2</td><td>80.1</td><td>80.1</td><td>-</td></tr><tr><td>SegLink++ [28]</td><td>Syn</td><td>79.8</td><td>82.8</td><td>81.3</td><td>-</td></tr><tr><td>TextField [37]</td><td>Syn</td><td>79.8</td><td>83.0</td><td>81.4</td><td>6.0</td></tr><tr><td>MSR[38]</td><td>Syn</td><td>79.0</td><td>84.1</td><td>81.5</td><td>4.3</td></tr><tr><td>PSENet-1s [33]</td><td>MLT</td><td>79.7</td><td>84.8</td><td>82.2</td><td>3.9</td></tr><tr><td>DB [12]</td><td>Syn</td><td>80.2</td><td>86.9</td><td>83.4</td><td>22.0</td></tr><tr><td>CRAFT [2]</td><td>Syn</td><td>81.1</td><td>86.0</td><td>83.5</td><td>-</td></tr><tr><td>TextDragon [5]</td><td>MLT+</td><td>82.8</td><td>84.5</td><td>83.6</td><td></td></tr><tr><td>PAN [34]</td><td>Syn</td><td>81.2</td><td>86.4</td><td>83.7</td><td>39.8</td></tr><tr><td>ContourNet [36]</td><td></td><td>84.1</td><td>83.7</td><td>83.9</td><td>4.5</td></tr><tr><td>DRRG [41]</td><td>MLT</td><td>83.02</td><td>85.93</td><td>84.45</td><td>-</td></tr><tr><td>TextPerception[23]</td><td>Syn</td><td>81.9</td><td>87.5</td><td>84.6</td><td></td></tr><tr><td>Ours</td><td> Syn</td><td>80.57</td><td>87.66</td><td>83.97</td><td>12.08</td></tr><tr><td>Ours</td><td></td><td>81.45</td><td>87.81</td><td>84.51</td><td>12.15</td></tr><tr><td>Ours</td><td>MLT</td><td>83.60</td><td>86.45</td><td>85.00</td><td>12.21</td></tr></tbody></table></body></html>
|
||||
The table visualized image saved in ./output//6_1.png
|
||||
7 type: table, region: [462,359,820,657], score: 0.953917, res: <html><body><table><thead><tr><td>Methods</td><td>R</td><td>P</td><td>F</td><td>FPS</td></tr></thead><tbody><tr><td>SegLink [26]</td><td>70.0</td><td>86.0</td><td>77.0</td><td>8.9</td></tr><tr><td>PixelLink [4]</td><td>73.2</td><td>83.0</td><td>77.8</td><td>-</td></tr><tr><td>TextSnake [18]</td><td>73.9</td><td>83.2</td><td>78.3</td><td>1.1</td></tr><tr><td>TextField [37]</td><td>75.9</td><td>87.4</td><td>81.3</td><td>5.2 </td></tr><tr><td>MSR[38]</td><td>76.7</td><td>87.4</td><td>81.7</td><td>-</td></tr><tr><td>FTSN[3]</td><td>77.1</td><td>87.6</td><td>82.0</td><td>:</td></tr><tr><td>LSE[30]</td><td>81.7</td><td>84.2</td><td>82.9</td><td></td></tr><tr><td>CRAFT [2]</td><td>78.2</td><td>88.2</td><td>82.9</td><td>8.6</td></tr><tr><td>MCN [16]</td><td>79</td><td>88</td><td>83</td><td>-</td></tr><tr><td>ATRR[35]</td><td>82.1</td><td>85.2</td><td>83.6</td><td>-</td></tr><tr><td>PAN [34]</td><td>83.8</td><td>84.4</td><td>84.1</td><td>30.2</td></tr><tr><td>DB[12]</td><td>79.2</td><td>91.5</td><td>84.9</td><td>32.0</td></tr><tr><td>DRRG [41]</td><td>82.30</td><td>88.05</td><td>85.08</td><td>-</td></tr><tr><td>Ours (SynText)</td><td>80.68</td><td>85.40</td><td>82.97</td><td>12.68</td></tr><tr><td>Ours (MLT-17)</td><td>84.54</td><td>86.62</td><td>85.57</td><td>12.31</td></tr></tbody></table></body></html>
|
||||
The table visualized image saved in ./output//7_1.png
|
||||
8 type: figure, region: [14,3,836,310], score: 0.969443, res: count of ocr result is : 26
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[506,14],[539,15],[539,22],[506,21]] rec text: E rec score: 0.318073
|
||||
...
|
||||
25 det boxes: [[680,290],[759,288],[759,303],[680,305]] rec text: (d) CTW1500 rec score: 0.95911
|
||||
********** end print ocr result **********
|
||||
```
|
||||
|
||||
## 3. FAQ
|
||||
|
||||
1. Encountered the error `unable to access 'https://github.com/LDOUBLEV/AutoLog.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated.`, change the github address in `deploy/cpp_infer/external-cmake/auto-log.cmake` to the <https://gitee.com/Double_V/AutoLog> address.
|
||||
443
docs/version2.x/ppstructure/infer_deploy/cpp_infer.md
Normal file
443
docs/version2.x/ppstructure/infer_deploy/cpp_infer.md
Normal file
@@ -0,0 +1,443 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 服务器端C++预测
|
||||
|
||||
本章节介绍PaddleOCR 模型的C++部署方法。C++在性能计算上优于Python，因此，在大多数CPU、GPU部署场景，多采用C++的部署方式，本节将介绍如何在Linux\Windows (CPU\GPU)环境下配置C++环境并完成PaddleOCR模型部署。
|
||||
|
||||
## 1. 准备环境
|
||||
|
||||
### 1.1 运行准备
|
||||
|
||||
- Linux环境，推荐使用docker。
|
||||
- Windows环境。
|
||||
|
||||
- 该文档主要介绍基于Linux环境的PaddleOCR C++预测流程，如果需要在Windows下基于预测库进行C++预测，具体编译方法请参考[Windows下编译教程](../../ppocr/infer_deploy/windows_vs2019_build.md)
|
||||
|
||||
### 1.2 编译opencv库
|
||||
|
||||
- 首先需要从opencv官网上下载在Linux环境下源码编译的包，以opencv3.4.7为例，下载命令如下：
|
||||
|
||||
```bash linenums="1"
|
||||
cd deploy/cpp_infer
|
||||
wget https://paddleocr.bj.bcebos.com/libs/opencv/opencv-3.4.7.tar.gz
|
||||
tar -xf opencv-3.4.7.tar.gz
|
||||
```
|
||||
|
||||
最终可以在当前目录下看到`opencv-3.4.7/`的文件夹。
|
||||
|
||||
- 编译opencv，设置opencv源码路径(`root_path`)以及安装路径(`install_path`)。进入opencv源码路径下，按照下面的方式进行编译。
|
||||
|
||||
```bash linenums="1"
|
||||
root_path="your_opencv_root_path"
|
||||
install_path=${root_path}/opencv3
|
||||
build_dir=${root_path}/build
|
||||
|
||||
rm -rf ${build_dir}
|
||||
mkdir ${build_dir}
|
||||
cd ${build_dir}
|
||||
|
||||
cmake .. \
|
||||
-DCMAKE_INSTALL_PREFIX=${install_path} \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DBUILD_SHARED_LIBS=OFF \
|
||||
-DWITH_IPP=OFF \
|
||||
-DBUILD_IPP_IW=OFF \
|
||||
-DWITH_LAPACK=OFF \
|
||||
-DWITH_EIGEN=OFF \
|
||||
-DCMAKE_INSTALL_LIBDIR=lib64 \
|
||||
-DWITH_ZLIB=ON \
|
||||
-DBUILD_ZLIB=ON \
|
||||
-DWITH_JPEG=ON \
|
||||
-DBUILD_JPEG=ON \
|
||||
-DWITH_PNG=ON \
|
||||
-DBUILD_PNG=ON \
|
||||
-DWITH_TIFF=ON \
|
||||
-DBUILD_TIFF=ON
|
||||
|
||||
make -j
|
||||
make install
|
||||
```
|
||||
|
||||
也可以直接修改`tools/build_opencv.sh`的内容，然后直接运行下面的命令进行编译。
|
||||
|
||||
```bash linenums="1"
|
||||
sh tools/build_opencv.sh
|
||||
```
|
||||
|
||||
其中`root_path`为下载的opencv源码路径，`install_path`为opencv的安装路径，`make install`完成之后，会在该文件夹下生成opencv头文件和库文件，用于后面的OCR代码编译。
|
||||
|
||||
最终在安装路径下的文件结构如下所示。
|
||||
|
||||
```text linenums="1"
|
||||
opencv3/
|
||||
|-- bin
|
||||
|-- include
|
||||
|-- lib
|
||||
|-- lib64
|
||||
|-- share
|
||||
```
|
||||
|
||||
### 1.3 下载或者编译Paddle预测库
|
||||
|
||||
可以选择直接下载安装或者从源码编译，下文分别进行具体说明。
|
||||
|
||||
#### 1.3.1 直接下载安装
|
||||
|
||||
[Paddle预测库官网](https://www.paddlepaddle.org.cn/inference/master/guides/install/download_lib.html#linux) 上提供了不同cuda版本的Linux预测库，可以在官网查看并选择合适的预测库版本（*建议选择paddle版本>=2.0.1版本的预测库* ）。
|
||||
|
||||
下载之后解压:
|
||||
|
||||
```bash linenums="1"
|
||||
tar -xf paddle_inference.tgz
|
||||
```
|
||||
|
||||
最终会在当前的文件夹中生成`paddle_inference/`的子文件夹。
|
||||
|
||||
#### 1.3.2 预测库源码编译
|
||||
|
||||
如果希望获取最新预测库特性，可以从github上克隆最新Paddle代码进行编译，生成最新的预测库。
|
||||
|
||||
- 使用git获取代码:
|
||||
|
||||
```bash linenums="1"
|
||||
git clone https://github.com/PaddlePaddle/Paddle.git
|
||||
git checkout develop
|
||||
```
|
||||
|
||||
- 进入Paddle目录，进行编译:
|
||||
|
||||
```bash linenums="1"
|
||||
rm -rf build
|
||||
mkdir build
|
||||
cd build
|
||||
|
||||
cmake .. \
|
||||
-DWITH_CONTRIB=OFF \
|
||||
-DWITH_MKL=ON \
|
||||
-DWITH_MKLDNN=ON \
|
||||
-DWITH_TESTING=OFF \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DWITH_INFERENCE_API_TEST=OFF \
|
||||
-DON_INFER=ON \
|
||||
-DWITH_PYTHON=ON
|
||||
make -j
|
||||
make inference_lib_dist
|
||||
```
|
||||
|
||||
更多编译参数选项介绍可以参考[Paddle预测库编译文档](https://www.paddlepaddle.org.cn/documentation/docs/zh/2.0/guides/05_inference_deployment/inference/build_and_install_lib_cn.html#congyuanmabianyi)。
|
||||
|
||||
- 编译完成之后，可以在`build/paddle_inference_install_dir/`文件下看到生成了以下文件及文件夹。
|
||||
|
||||
```
|
||||
build/paddle_inference_install_dir/
|
||||
|-- CMakeCache.txt
|
||||
|-- paddle
|
||||
|-- third_party
|
||||
|-- version.txt
|
||||
```
|
||||
|
||||
其中`paddle`就是C++预测所需的Paddle库，`version.txt`中包含当前预测库的版本信息。
|
||||
|
||||
## 2. 开始运行
|
||||
|
||||
### 2.1 准备模型
|
||||
|
||||
直接下载PaddleOCR提供的推理模型，或者参考[模型预测章节](../../ppocr/infer_deploy/python_infer.md)，将训练好的模型导出为推理模型。模型导出之后，假设放在`inference`目录下，则目录结构如下：
|
||||
|
||||
```text linenums="1"
|
||||
inference/
|
||||
|-- det_db
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
|-- rec_rcnn
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
|-- cls
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
|-- table
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
|-- layout
|
||||
| |--inference.pdiparams
|
||||
| |--inference.pdmodel
|
||||
```
|
||||
|
||||
### 2.2 编译PaddleOCR C++预测demo
|
||||
|
||||
编译命令如下，其中Paddle C++预测库、opencv等其他依赖库的地址需要换成自己机器上的实际地址。
|
||||
|
||||
```bash linenums="1"
|
||||
sh tools/build.sh
|
||||
```
|
||||
|
||||
具体的，需要修改`tools/build.sh`中环境路径，相关内容如下：
|
||||
|
||||
```bash linenums="1"
|
||||
OPENCV_DIR=your_opencv_dir
|
||||
LIB_DIR=your_paddle_inference_dir
|
||||
CUDA_LIB_DIR=your_cuda_lib_dir
|
||||
CUDNN_LIB_DIR=/your_cudnn_lib_dir
|
||||
```
|
||||
|
||||
其中，`OPENCV_DIR`为opencv编译安装的地址；`LIB_DIR`为下载(`paddle_inference`文件夹)或者编译生成的Paddle预测库地址(`build/paddle_inference_install_dir`文件夹)；`CUDA_LIB_DIR`为cuda库文件地址，在docker中为`/usr/local/cuda/lib64`；`CUDNN_LIB_DIR`为cudnn库文件地址，在docker中为`/usr/lib/x86_64-linux-gnu/`。**注意：以上路径都写绝对路径，不要写相对路径。**
|
||||
|
||||
编译完成之后，会在`build`文件夹下生成一个名为`ppocr`的可执行文件。
|
||||
|
||||
### 2.3 运行demo
|
||||
|
||||
本demo支持系统串联调用，也支持单个功能的调用，如，只使用检测或识别功能。
|
||||
|
||||
**注意** ppocr默认使用`PP-OCRv3`模型，识别模型使用的输入shape为`3,48,320`, 如需使用旧版本的PP-OCR模型，则需要设置参数`--rec_img_h=32`。
|
||||
|
||||
运行方式：
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr [--param1] [--param2] [...]
|
||||
```
|
||||
|
||||
具体命令如下：
|
||||
|
||||
#### 1. 检测+分类+识别
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--rec_model_dir=inference/rec_rcnn \
|
||||
--cls_model_dir=inference/cls \
|
||||
--image_dir=../../doc/imgs/12.jpg \
|
||||
--use_angle_cls=true \
|
||||
--det=true \
|
||||
--rec=true \
|
||||
--cls=true \
|
||||
```
|
||||
|
||||
##### 2. 检测+识别
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--rec_model_dir=inference/rec_rcnn \
|
||||
--image_dir=../../doc/imgs/12.jpg \
|
||||
--use_angle_cls=false \
|
||||
--det=true \
|
||||
--rec=true \
|
||||
--cls=false \
|
||||
```
|
||||
|
||||
##### 3. 检测
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--image_dir=../../doc/imgs/12.jpg \
|
||||
--det=true \
|
||||
--rec=false
|
||||
```
|
||||
|
||||
##### 4. 分类+识别
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --rec_model_dir=inference/rec_rcnn \
|
||||
--cls_model_dir=inference/cls \
|
||||
--image_dir=../../doc/imgs_words/ch/word_1.jpg \
|
||||
--use_angle_cls=true \
|
||||
--det=false \
|
||||
--rec=true \
|
||||
--cls=true \
|
||||
```
|
||||
|
||||
##### 5. 识别
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --rec_model_dir=inference/rec_rcnn \
|
||||
--image_dir=../../doc/imgs_words/ch/word_1.jpg \
|
||||
--use_angle_cls=false \
|
||||
--det=false \
|
||||
--rec=true \
|
||||
--cls=false \
|
||||
```
|
||||
|
||||
##### 6. 分类
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --cls_model_dir=inference/cls \
|
||||
--cls_model_dir=inference/cls \
|
||||
--image_dir=../../doc/imgs_words/ch/word_1.jpg \
|
||||
--use_angle_cls=true \
|
||||
--det=false \
|
||||
--rec=false \
|
||||
--cls=true \
|
||||
```
|
||||
|
||||
##### 7. 版面分析+表格识别
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--rec_model_dir=inference/rec_rcnn \
|
||||
--table_model_dir=inference/table \
|
||||
--image_dir=../../ppstructure/docs/table/table.jpg \
|
||||
--layout_model_dir=inference/layout \
|
||||
--type=structure \
|
||||
--table=true \
|
||||
--layout=true
|
||||
```
|
||||
|
||||
##### 8. 版面分析
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --layout_model_dir=inference/layout \
|
||||
--image_dir=../../ppstructure/docs/table/1.png \
|
||||
--type=structure \
|
||||
--table=false \
|
||||
--layout=true \
|
||||
--det=false \
|
||||
--rec=false
|
||||
```
|
||||
|
||||
##### 9. 表格识别
|
||||
|
||||
```bash linenums="1"
|
||||
./build/ppocr --det_model_dir=inference/det_db \
|
||||
--rec_model_dir=inference/rec_rcnn \
|
||||
--table_model_dir=inference/table \
|
||||
--image_dir=../../ppstructure/docs/table/table.jpg \
|
||||
--type=structure \
|
||||
--table=true
|
||||
```
|
||||
|
||||
更多支持的可调节参数解释如下：
|
||||
|
||||
- 通用参数
|
||||
|
||||
| 参数名称 | 类型 | 默认参数 | 意义 |
|
||||
| :--------------------------: | :---: | :------: | :---------------------------------------------------------------: |
|
||||
| use_gpu | bool | false | 是否使用GPU |
|
||||
| gpu_id | int | 0 | GPU id，使用GPU时有效 |
|
||||
| gpu_mem | int | 4000 | 申请的GPU内存 |
|
||||
| cpu_math_library_num_threads | int | 10 | CPU预测时的线程数，在机器核数充足的情况下，该值越大，预测速度越快 |
|
||||
| enable_mkldnn | bool | true | 是否使用mkldnn库 |
|
||||
| output | str | ./output | 可视化结果保存的路径 |
|
||||
|
||||
- 前向相关
|
||||
|
||||
| 参数名称 | 类型 | 默认参数 | 意义 |
|
||||
| :------: | :---: | :------: | :----------------------: |
|
||||
| det | bool | true | 前向是否执行文字检测 |
|
||||
| rec | bool | true | 前向是否执行文字识别 |
|
||||
| cls | bool | false | 前向是否执行文字方向分类 |
|
||||
|
||||
- 检测模型相关
|
||||
|
||||
| 参数名称 | 类型 | 默认参数 | 意义 |
|
||||
| :-----------------: | :----: | :------: | :----------------------------------------------------------------------------------------------------------: |
|
||||
| det_model_dir | string | - | 检测模型inference model地址 |
|
||||
| max_side_len | int | 960 | 输入图像长宽大于960时，等比例缩放图像，使得图像最长边为960 |
|
||||
| det_db_thresh | float | 0.3 | 用于过滤DB预测的二值化图像，设置为0.-0.3对结果影响不明显 |
|
||||
| det_db_box_thresh | float | 0.5 | DB后处理过滤box的阈值，如果检测存在漏框情况，可酌情减小 |
|
||||
| det_db_unclip_ratio | float | 1.6 | 表示文本框的紧致程度，越小则文本框更靠近文本 |
|
||||
| det_db_score_mode | string | slow | slow:使用多边形框计算bbox score，fast:使用矩形框计算。矩形框计算速度更快，多边形框对弯曲文本区域计算更准确。 |
|
||||
| visualize | bool | true | 是否对结果进行可视化，为1时，预测结果会保存在`output`字段指定的文件夹下和输入图像同名的图像上。 |
|
||||
|
||||
- 方向分类器相关
|
||||
|
||||
| 参数名称 | 类型 | 默认参数 | 意义 |
|
||||
| :-----------: | :----: | :------: | :---------------------------: |
|
||||
| use_angle_cls | bool | false | 是否使用方向分类器 |
|
||||
| cls_model_dir | string | - | 方向分类器inference model地址 |
|
||||
| cls_thresh | float | 0.9 | 方向分类器的得分阈值 |
|
||||
| cls_batch_num | int | 1 | 方向分类器batchsize |
|
||||
|
||||
- 文字识别模型相关
|
||||
|
||||
| 参数名称 | 类型 | 默认参数 | 意义 |
|
||||
| :----------------: | :----: | :---------------------------------: | :-----------------------------: |
|
||||
| rec_model_dir | string | - | 文字识别模型inference model地址 |
|
||||
| rec_char_dict_path | string | ../../ppocr/utils/ppocr_keys_v1.txt | 字典文件 |
|
||||
| rec_batch_num | int | 6 | 文字识别模型batchsize |
|
||||
| rec_img_h | int | 48 | 文字识别模型输入图像高度 |
|
||||
| rec_img_w | int | 320 | 文字识别模型输入图像宽度 |
|
||||
|
||||
- 版面分析模型相关
|
||||
|
||||
| 参数名称 | 类型 | 默认参数 | 意义 |
|
||||
| :--------------------: | :----: | :----------------------------------------------------------: | :-----------------------------: |
|
||||
| layout_model_dir | string | - | 版面分析模型inference model地址 |
|
||||
| layout_dict_path | string | ../../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt | 字典文件 |
|
||||
| layout_score_threshold | float | 0.5 | 检测框的分数阈值 |
|
||||
| layout_nms_threshold | float | 0.5 | nms的阈值 |
|
||||
|
||||
- 表格识别模型相关
|
||||
|
||||
| 参数名称 | 类型 | 默认参数 | 意义 |
|
||||
| :---------------------: | :----: | :------------------------------------------------: | :----------------------------------------------------------------------------------: |
|
||||
| table_model_dir | string | - | 表格识别模型inference model地址 |
|
||||
| table_char_dict_path | string | ../../ppocr/utils/dict/table_structure_dict_ch.txt | 字典文件 |
|
||||
| table_max_len | int | 488 | 表格识别模型输入图像长边大小，最终网络输入图像大小为（table_max_len，table_max_len） |
|
||||
| merge_no_span_structure | bool | true | 是否合并<td> 和 </td> 为<td></td> |
|
||||
|
||||
- PaddleOCR也支持多语言的预测，更多支持的语言和模型可以参考[识别文档](../../ppocr/blog/multi_languages.md)中的多语言字典与模型部分，如果希望进行多语言预测，只需将修改`rec_char_dict_path`（字典文件路径）以及`rec_model_dir`（inference模型路径）字段即可。
|
||||
|
||||
最终屏幕上会输出检测结果如下：
|
||||
|
||||
- ocr
|
||||
|
||||
```bash linenums="1"
|
||||
predict img: ../../doc/imgs/12.jpg
|
||||
../../doc/imgs/12.jpg
|
||||
0 det boxes: [[74,553],[427,542],[428,571],[75,582]] rec text: 打浦路252935号 rec score: 0.947724
|
||||
1 det boxes: [[23,507],[513,488],[515,529],[24,548]] rec text: 绿洲仕格维花园公寓 rec score: 0.993728
|
||||
2 det boxes: [[187,456],[399,448],[400,480],[188,488]] rec text: 打浦路15号 rec score: 0.964994
|
||||
3 det boxes: [[42,413],[483,391],[484,428],[43,450]] rec text: 上海斯格威铂尔大酒店 rec score: 0.980086
|
||||
The detection visualized image saved in ./output//12.jpg
|
||||
```
|
||||
|
||||
- layout+table
|
||||
|
||||
```bash linenums="1"
|
||||
predict img: ../../ppstructure/docs/table/1.png
|
||||
0 type: text, region: [12,729,410,848], score: 0.781044, res: count of ocr result is : 7
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[4,1],[79,1],[79,12],[4,12]] rec text: CTW1500. rec score: 0.769472
|
||||
...
|
||||
6 det boxes: [[4,99],[391,99],[391,112],[4,112]] rec text: sate-of-the-artmethods[12.34.36l.ourapproachachieves rec score: 0.90414
|
||||
********** end print ocr result **********
|
||||
1 type: text, region: [69,342,342,359], score: 0.703666, res: count of ocr result is : 1
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[8,2],[269,2],[269,13],[8,13]] rec text: Table6.Experimentalresults on CTW-1500 rec score: 0.890454
|
||||
********** end print ocr result **********
|
||||
2 type: text, region: [70,316,706,332], score: 0.659738, res: count of ocr result is : 2
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[373,2],[630,2],[630,11],[373,11]] rec text: oroposals.andthegreencontoursarefinal rec score: 0.919729
|
||||
1 det boxes: [[8,3],[357,3],[357,11],[8,11]] rec text: Visualexperimentalresultshebluecontoursareboundar rec score: 0.915963
|
||||
********** end print ocr result **********
|
||||
3 type: text, region: [489,342,789,359], score: 0.630538, res: count of ocr result is : 1
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[8,2],[294,2],[294,14],[8,14]] rec text: Table7.Experimentalresults onMSRA-TD500 rec score: 0.942251
|
||||
********** end print ocr result **********
|
||||
4 type: text, region: [444,751,841,848], score: 0.607345, res: count of ocr result is : 5
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[19,3],[389,3],[389,17],[19,17]] rec text: Inthispaper,weproposeanovel adaptivebound rec score: 0.941031
|
||||
1 det boxes: [[4,22],[390,22],[390,36],[4,36]] rec text: aryproposalnetworkforarbitraryshapetextdetection rec score: 0.960172
|
||||
2 det boxes: [[4,42],[392,42],[392,56],[4,56]] rec text: whichadoptanboundaryproposalmodeltogeneratecoarse rec score: 0.934647
|
||||
3 det boxes: [[4,61],[389,61],[389,75],[4,75]] rec text: ooundaryproposals,andthenadoptanadaptiveboundary rec score: 0.946296
|
||||
4 det boxes: [[5,80],[387,80],[387,93],[5,93]] rec text: leformationmodelcombinedwithGCNandRNNtoper rec score: 0.952401
|
||||
********** end print ocr result **********
|
||||
5 type: title, region: [444,705,564,724], score: 0.785429, res: count of ocr result is : 1
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[6,2],[113,2],[113,14],[6,14]] rec text: 5.Conclusion rec score: 0.856903
|
||||
********** end print ocr result **********
|
||||
6 type: table, region: [14,360,402,711], score: 0.963643, res: <html><body><table><thead><tr><td>Methods</td><td>Ext</td><td>R</td><td>P</td><td>F</td><td>FPS</td></tr></thead><tbody><tr><td>TextSnake [18]</td><td>Syn</td><td>85.3</td><td>67.9</td><td>75.6</td><td></td></tr><tr><td>CSE [17]</td><td>MiLT</td><td>76.1</td><td>78.7</td><td>77.4</td><td>0.38</td></tr><tr><td>LOMO[40]</td><td>Syn</td><td>76.5</td><td>85.7</td><td>80.8</td><td>4.4</td></tr><tr><td>ATRR[35]</td><td>Sy-</td><td>80.2</td><td>80.1</td><td>80.1</td><td>-</td></tr><tr><td>SegLink++ [28]</td><td>Syn</td><td>79.8</td><td>82.8</td><td>81.3</td><td>-</td></tr><tr><td>TextField [37]</td><td>Syn</td><td>79.8</td><td>83.0</td><td>81.4</td><td>6.0</td></tr><tr><td>MSR[38]</td><td>Syn</td><td>79.0</td><td>84.1</td><td>81.5</td><td>4.3</td></tr><tr><td>PSENet-1s [33]</td><td>MLT</td><td>79.7</td><td>84.8</td><td>82.2</td><td>3.9</td></tr><tr><td>DB [12]</td><td>Syn</td><td>80.2</td><td>86.9</td><td>83.4</td><td>22.0</td></tr><tr><td>CRAFT [2]</td><td>Syn</td><td>81.1</td><td>86.0</td><td>83.5</td><td>-</td></tr><tr><td>TextDragon [5]</td><td>MLT+</td><td>82.8</td><td>84.5</td><td>83.6</td><td></td></tr><tr><td>PAN [34]</td><td>Syn</td><td>81.2</td><td>86.4</td><td>83.7</td><td>39.8</td></tr><tr><td>ContourNet [36]</td><td></td><td>84.1</td><td>83.7</td><td>83.9</td><td>4.5</td></tr><tr><td>DRRG [41]</td><td>MLT</td><td>83.02</td><td>85.93</td><td>84.45</td><td>-</td></tr><tr><td>TextPerception[23]</td><td>Syn</td><td>81.9</td><td>87.5</td><td>84.6</td><td></td></tr><tr><td>Ours</td><td> Syn</td><td>80.57</td><td>87.66</td><td>83.97</td><td>12.08</td></tr><tr><td>Ours</td><td></td><td>81.45</td><td>87.81</td><td>84.51</td><td>12.15</td></tr><tr><td>Ours</td><td>MLT</td><td>83.60</td><td>86.45</td><td>85.00</td><td>12.21</td></tr></tbody></table></body></html>
|
||||
The table visualized image saved in ./output//6_1.png
|
||||
7 type: table, region: [462,359,820,657], score: 0.953917, res: <html><body><table><thead><tr><td>Methods</td><td>R</td><td>P</td><td>F</td><td>FPS</td></tr></thead><tbody><tr><td>SegLink [26]</td><td>70.0</td><td>86.0</td><td>77.0</td><td>8.9</td></tr><tr><td>PixelLink [4]</td><td>73.2</td><td>83.0</td><td>77.8</td><td>-</td></tr><tr><td>TextSnake [18]</td><td>73.9</td><td>83.2</td><td>78.3</td><td>1.1</td></tr><tr><td>TextField [37]</td><td>75.9</td><td>87.4</td><td>81.3</td><td>5.2 </td></tr><tr><td>MSR[38]</td><td>76.7</td><td>87.4</td><td>81.7</td><td>-</td></tr><tr><td>FTSN[3]</td><td>77.1</td><td>87.6</td><td>82.0</td><td>:</td></tr><tr><td>LSE[30]</td><td>81.7</td><td>84.2</td><td>82.9</td><td></td></tr><tr><td>CRAFT [2]</td><td>78.2</td><td>88.2</td><td>82.9</td><td>8.6</td></tr><tr><td>MCN [16]</td><td>79</td><td>88</td><td>83</td><td>-</td></tr><tr><td>ATRR[35]</td><td>82.1</td><td>85.2</td><td>83.6</td><td>-</td></tr><tr><td>PAN [34]</td><td>83.8</td><td>84.4</td><td>84.1</td><td>30.2</td></tr><tr><td>DB[12]</td><td>79.2</td><td>91.5</td><td>84.9</td><td>32.0</td></tr><tr><td>DRRG [41]</td><td>82.30</td><td>88.05</td><td>85.08</td><td>-</td></tr><tr><td>Ours (SynText)</td><td>80.68</td><td>85.40</td><td>82.97</td><td>12.68</td></tr><tr><td>Ours (MLT-17)</td><td>84.54</td><td>86.62</td><td>85.57</td><td>12.31</td></tr></tbody></table></body></html>
|
||||
The table visualized image saved in ./output//7_1.png
|
||||
8 type: figure, region: [14,3,836,310], score: 0.969443, res: count of ocr result is : 26
|
||||
********** print ocr result **********
|
||||
0 det boxes: [[506,14],[539,15],[539,22],[506,21]] rec text: E rec score: 0.318073
|
||||
...
|
||||
25 det boxes: [[680,290],[759,288],[759,303],[680,305]] rec text: (d) CTW1500 rec score: 0.95911
|
||||
********** end print ocr result **********
|
||||
```
|
||||
|
||||
## 3. FAQ
|
||||
|
||||
1. 遇到报错 `unable to access 'https://github.com/LDOUBLEV/AutoLog.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated.`， 将 `deploy/cpp_infer/external-cmake/auto-log.cmake` 中的github地址改为 <https://gitee.com/Double_V/AutoLog> 地址即可。
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 992 KiB |
21
docs/version2.x/ppstructure/infer_deploy/index.en.md
Normal file
21
docs/version2.x/ppstructure/infer_deploy/index.en.md
Normal file
@@ -0,0 +1,21 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-OCR Deployment
|
||||
|
||||
## Paddle Deployment Introduction
|
||||
|
||||
Paddle provides a variety of deployment schemes to meet the deployment requirements of different scenarios. Please choose according to the actual situation:
|
||||
|
||||
|
||||
|
||||
PP-OCR has supported multi deployment schemes. Click the link to get the specific tutorial.
|
||||
|
||||
- [Python Inference](./python_infer.en.md)
|
||||
- [C++ Inference](./cpp_infer.en.md)
|
||||
- [Serving (Python/C++)](./paddle_server.en.md)
|
||||
- [Paddle-Lite (ARM CPU/OpenCL ARM GPU)](../../ppocr/infer_deploy/lite.en.md)
|
||||
- [Paddle2ONNX](../../ppocr/infer_deploy/paddle2onnx.en.md)
|
||||
|
||||
If you need the deployment tutorial of academic algorithm models other than PP-OCR, please directly enter the main page of corresponding algorithms, [entrance](../../algorithm/overview.en.md)。
|
||||
24
docs/version2.x/ppstructure/infer_deploy/index.md
Normal file
24
docs/version2.x/ppstructure/infer_deploy/index.md
Normal file
@@ -0,0 +1,24 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-OCR 模型推理部署
|
||||
|
||||
## Paddle 推理部署方式简介
|
||||
|
||||
飞桨提供多种部署方案，以满足不同场景的部署需求，请根据实际情况进行选择：
|
||||
|
||||
|
||||
|
||||
## PP-OCR 推理部署
|
||||
|
||||
PP-OCR模型已打通多种场景部署方案，点击链接获取具体的使用教程。
|
||||
|
||||
- [Python 推理](./python_infer.md)
|
||||
- [C++ 推理](./cpp_infer.md)
|
||||
- [Serving 服务化部署（Python/C++）](./paddle_server.md)
|
||||
- [Paddle-Lite 端侧部署（ARM CPU/OpenCL ARM GPU）](../../ppocr/infer_deploy/lite.md)
|
||||
- [Paddle2ONNX 推理](../../ppocr/infer_deploy/paddle2onnx.md)
|
||||
|
||||
需要PP-OCR以外的学术算法模型的推理部署，请直接进入相应算法主页面，[入口](../../algorithm/overview.md)。
|
||||
273
docs/version2.x/ppstructure/infer_deploy/paddle_server.en.md
Executable file
273
docs/version2.x/ppstructure/infer_deploy/paddle_server.en.md
Executable file
@@ -0,0 +1,273 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
PaddleOCR provides 2 service deployment methods:
|
||||
|
||||
- Based on **PaddleHub Serving**: Code path is `./deploy/hubserving`. Please follow this tutorial.
|
||||
- Based on **PaddleServing**: Code path is `./deploy/pdserving`. Please refer to the [tutorial](../../ppocr/infer_deploy/paddle_server.en.md) for usage.
|
||||
|
||||
# Service deployment based on PaddleHub Serving
|
||||
|
||||
The hubserving service deployment directory includes seven service packages: text detection, text angle class, text recognition, text detection+text angle class+text recognition three-stage series connection, layout analysis, table recognition, and PP-Structure. Please select the corresponding service package to install and start the service according to your needs. The directory is as follows:
|
||||
|
||||
```text linenums="1"
|
||||
deploy/hubserving/
|
||||
└─ ocr_det text detection module service package
|
||||
└─ ocr_cls text angle class module service package
|
||||
└─ ocr_rec text recognition module service package
|
||||
└─ ocr_system text detection+text angle class+text recognition three-stage series connection service package
|
||||
└─ structure_layout layout analysis service package
|
||||
└─ structure_table table recognition service package
|
||||
└─ structure_system PP-Structure service package
|
||||
└─ kie_ser KIE(SER) service package
|
||||
└─ kie_ser_re KIE(SER+RE) service package
|
||||
```
|
||||
|
||||
Each service pack contains 3 files. Take the 2-stage series connection service package as an example, the directory is as follows:
|
||||
|
||||
```text linenums="1"
|
||||
deploy/hubserving/ocr_system/
|
||||
└─ __init__.py Empty file, required
|
||||
└─ config.json Configuration file, optional, passed in as a parameter when using configuration to start the service
|
||||
└─ module.py Main module file, required, contains the complete logic of the service
|
||||
└─ params.py Parameter file, required, including parameters such as model path, pre and post-processing parameters
|
||||
```
|
||||
|
||||
## 1. Update
|
||||
|
||||
- 2022.10.09 add KIE services.
|
||||
- 2022.08.23 add layout analysis services.
|
||||
- 2022.03.30 add PP-Structure and table recognition services.
|
||||
- 2022.05.05 add PP-OCRv3 text detection and recognition services.
|
||||
|
||||
## 2. Quick start service
|
||||
|
||||
The following steps take the 2-stage series service as an example. If only the detection service or recognition service is needed, replace the corresponding file path.
|
||||
|
||||
### 2.1 Install PaddleHub
|
||||
|
||||
```bash linenums="1"
|
||||
pip3 install paddlehub==2.1.0 --upgrade
|
||||
```
|
||||
|
||||
### 2.2 Download inference model
|
||||
|
||||
Before installing the service module, you need to prepare the inference model and put it in the correct path. By default, the PP-OCRv3 models are used, and the default model path is:
|
||||
|
||||
| Model | Path |
|
||||
| ------- | - |
|
||||
| text detection model | ./inference/PP-OCRv3_mobile_det_infer/ |
|
||||
| text recognition model | ./inference/PP-OCRv3_mobile_rec_infer/ |
|
||||
| text angle classifier | ./inference/ch_ppocr_mobile_v2.0_cls_infer/ |
|
||||
| layout parse model | ./inference/picodet_lcnet_x1_0_fgd_layout_infer/ |
|
||||
| tanle recognition | ./inference/ch_ppstructure_mobile_v2.0_SLANet_infer/ |
|
||||
| KIE(SER) | ./inference/ser_vi_layoutxlm_xfund_infer/ |
|
||||
| KIE(SER+RE) | ./inference/re_vi_layoutxlm_xfund_infer/ |
|
||||
|
||||
**The model path can be found and modified in `params.py`.**
|
||||
More models provided by PaddleOCR can be obtained from the [model library](../../ppocr/model_list.en.md). You can also use models trained by yourself.
|
||||
|
||||
### 2.3 Install Service Module
|
||||
|
||||
PaddleOCR provides 5 kinds of service modules, install the required modules according to your needs.
|
||||
|
||||
- On the Linux platform(replace `/` with `\` if using Windows), the examples are as the following table:
|
||||
|
||||
| Service model | Command |
|
||||
| text detection | `hub install deploy/hubserving/ocr_det` |
|
||||
| text angle class: | `hub install deploy/hubserving/ocr_cls` |
|
||||
| text recognition: | `hub install deploy/hubserving/ocr_rec` |
|
||||
| 2-stage series: | `hub install deploy/hubserving/ocr_system` |
|
||||
| table recognition | `hub install deploy/hubserving/structure_table` |
|
||||
| PP-Structure | `hub install deploy/hubserving/structure_system` |
|
||||
| KIE(SER) | `hub install deploy/hubserving/kie_ser` |
|
||||
| KIE(SER+RE) | `hub install deploy/hubserving/kie_ser_re` |
|
||||
|
||||
### 2.4 Start service
|
||||
|
||||
#### 2.4.1 Start with command line parameters (CPU only)
|
||||
|
||||
**start command:**
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving start --modules Module1==Version1, Module2==Version2, ... \
|
||||
--port 8866 \
|
||||
--use_multiprocess \
|
||||
--workers \
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|parameters|usage|
|
||||
|---|---|
|
||||
|`--modules`/`-m`|PaddleHub Serving pre-installed model, listed in the form of multiple Module==Version key-value pairs<br>**When Version is not specified, the latest version is selected by default**|
|
||||
|`--port`/`-p`|Service port, default is 8866|
|
||||
|`--use_multiprocess`|Enable concurrent mode, by default using the single-process mode, this mode is recommended for multi-core CPU machines<br>**Windows operating system only supports single-process mode**|
|
||||
|`--workers`|The number of concurrent tasks specified in concurrent mode, the default is `2*cpu_count-1`, where `cpu_count` is the number of CPU cores|
|
||||
|
||||
For example, start the 2-stage series service:
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving start -m ocr_system
|
||||
```
|
||||
|
||||
This completes the deployment of a service API, using the default port number 8866.
|
||||
|
||||
#### 2.4.2 Start with configuration file（CPU and GPU）
|
||||
|
||||
**start command:**
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving start --config/-c config.json
|
||||
```
|
||||
|
||||
In which the format of `config.json` is as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"modules_info": {
|
||||
"ocr_system": {
|
||||
"init_args": {
|
||||
"version": "1.0.0",
|
||||
"use_gpu": true
|
||||
},
|
||||
"predict_args": {
|
||||
}
|
||||
}
|
||||
},
|
||||
"port": 8868,
|
||||
"use_multiprocess": false,
|
||||
"workers": 2
|
||||
}
|
||||
```
|
||||
|
||||
- The configurable parameters in `init_args` are consistent with the `_initialize` function interface in `module.py`.
|
||||
|
||||
**When `use_gpu` is `true`, it means that the GPU is used to start the service**.
|
||||
- The configurable parameters in `predict_args` are consistent with the `predict` function interface in `module.py`.
|
||||
|
||||
**Note:**
|
||||
- When using the configuration file to start the service, other parameters will be ignored.
|
||||
- If you use GPU prediction (that is, `use_gpu` is set to `true`), you need to set the environment variable CUDA_VISIBLE_DEVICES before starting the service, such as:
|
||||
|
||||
```bash linenums="1"
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
```
|
||||
|
||||
- **`use_gpu` and `use_multiprocess` cannot be `true` at the same time.**
|
||||
|
||||
For example, use GPU card No. 3 to start the 2-stage series service:
|
||||
|
||||
```bash linenums="1"
|
||||
export CUDA_VISIBLE_DEVICES=3
|
||||
hub serving start -c deploy/hubserving/ocr_system/config.json
|
||||
```
|
||||
|
||||
## 3. Send prediction requests
|
||||
|
||||
After the service starts, you can use the following command to send a prediction request to obtain the prediction result:
|
||||
|
||||
```bash linenums="1"
|
||||
python tools/test_hubserving.py --server_url=server_url --image_dir=image_path
|
||||
```
|
||||
|
||||
Two parameters need to be passed to the script:
|
||||
|
||||
- **server_url**:service address, the format of which is
|
||||
`http://[ip_address]:[port]/predict/[module_name]`
|
||||
|
||||
For example, if using the configuration file to start the text angle classification, text detection, text recognition, detection+classification+recognition 3 stages, table recognition and PP-Structure service,
|
||||
|
||||
also modified the port for each service, then the `server_url` to send the request will be:
|
||||
|
||||
```text linenums="1"
|
||||
http://127.0.0.1:8865/predict/ocr_det
|
||||
http://127.0.0.1:8866/predict/ocr_cls
|
||||
http://127.0.0.1:8867/predict/ocr_rec
|
||||
http://127.0.0.1:8868/predict/ocr_system
|
||||
http://127.0.0.1:8869/predict/structure_table
|
||||
http://127.0.0.1:8870/predict/structure_system
|
||||
http://127.0.0.1:8870/predict/structure_layout
|
||||
http://127.0.0.1:8871/predict/kie_ser
|
||||
http://127.0.0.1:8872/predict/kie_ser_re
|
||||
```
|
||||
|
||||
- **image_dir**:Test image path, which can be a single image path or an image directory path
|
||||
- **visualize**:Whether to visualize the results, the default value is False
|
||||
- **output**:The folder to save the Visualization result, the default value is `./hubserving_result`
|
||||
|
||||
Example:
|
||||
|
||||
```bash linenums="1"
|
||||
python tools/test_hubserving.py --server_url=http://127.0.0.1:8868/predict/ocr_system --image_dir=./doc/imgs/ --visualize=false`
|
||||
```
|
||||
|
||||
## 4. Returned result format
|
||||
|
||||
The returned result is a list. Each item in the list is a dictionary which may contain three fields. The information is as follows:
|
||||
|
||||
|field name|data type|description|
|
||||
|----|----|----|
|
||||
|angle|str|angle|
|
||||
|text|str|text content|
|
||||
|confidence|float|text recognition confidence|
|
||||
|text_region|list|text location coordinates|
|
||||
|html|str|table HTML string|
|
||||
|regions|list|The result of layout analysis + table recognition + OCR, each item is a list<br>including `bbox` indicating area coordinates, `type` of area type and `res` of area results|
|
||||
|layout|list|The result of layout analysis, each item is a dict, including `bbox` indicating area coordinates, `label` of area type|
|
||||
|
||||
The fields returned by different modules are different. For example, the results returned by the text recognition service module do not contain `text_region`, detailed table is as follows:
|
||||
|
||||
|field name/module name |ocr_det |ocr_cls |ocr_rec |ocr_system |structure_table |structure_system |structure_layout |kie_ser |kie_re |
|
||||
|--- |--- |--- |--- |--- |--- |--- |--- |--- |--- |
|
||||
|angle | |✔ | |✔ | | | |
|
||||
|text | | |✔ |✔ | |✔ | |✔ |✔ |
|
||||
|confidence | |✔ |✔ |✔ | |✔ | |✔ |✔ |
|
||||
|text_region |✔ | | |✔ | |✔ | |✔ |✔ |
|
||||
|html | | | | |✔ |✔ | | | |
|
||||
|regions | | | | |✔ |✔ | | | |
|
||||
|layout | | | | | | |✔ | | |
|
||||
|ser_res | | | | | | | |✔ | |
|
||||
|re_res | | | | | | | | |✔ |
|
||||
|
||||
**Note:** If you need to add, delete or modify the returned fields, you can modify the file `module.py` of the corresponding module. For the complete process, refer to the user-defined modification service module in the next section.
|
||||
|
||||
## 5. User-defined service module modification
|
||||
|
||||
If you need to modify the service logic, the following steps are generally required (take the modification of `deploy/hubserving/ocr_system` for example):
|
||||
|
||||
1. Stop service:
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving stop --port/-p XXXX
|
||||
```
|
||||
|
||||
2. Modify the code in the corresponding files under `deploy/hubserving/ocr_system`, such as `module.py` and `params.py`, to your actual needs.
|
||||
|
||||
For example, if you need to replace the model used by the deployed service, you need to modify model path parameters `det_model_dir` and `rec_model_dir` in `params.py`. If you want to turn off the text direction classifier, set the parameter `use_angle_cls` to `False`.
|
||||
|
||||
Of course, other related parameters may need to be modified at the same time. Please modify and debug according to the actual situation.
|
||||
|
||||
**It is suggested to run `module.py` directly for debugging after modification before starting the service test.**
|
||||
|
||||
**Note** The image input shape used by the PPOCR-v3 recognition model is `3, 48, 320`, so you need to modify `cfg.rec_image_shape = "3, 48, 320"` in `params.py`, if you do not use the PPOCR-v3 recognition model, then there is no need to modify this parameter.
|
||||
3. (Optional) If you want to rename the module, the following lines should be modified:
|
||||
- [`ocr_system` within `from deploy.hubserving.ocr_system.params import read_params`](https://github.com/PaddlePaddle/PaddleOCR/blob/a923f35de57b5e378f8dd16e54d0a3e4f51267fd/deploy/hubserving/ocr_system/module.py#L35)
|
||||
- [`ocr_system` within `name="ocr_system",`](https://github.com/PaddlePaddle/PaddleOCR/blob/a923f35de57b5e378f8dd16e54d0a3e4f51267fd/deploy/hubserving/ocr_system/module.py#L39)
|
||||
4. (Optional) It may require you to delete the directory `__pycache__` to force flush build cache of CPython:
|
||||
|
||||
```bash linenums="1"
|
||||
find deploy/hubserving/ocr_system -name '__pycache__' -exec rm -r {} \;
|
||||
```
|
||||
|
||||
5. Install modified service module:
|
||||
|
||||
```bash linenums="1"
|
||||
hub install deploy/hubserving/ocr_system/
|
||||
```
|
||||
|
||||
6. Restart service:
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving start -m ocr_system
|
||||
```
|
||||
279
docs/version2.x/ppstructure/infer_deploy/paddle_server.md
Normal file
279
docs/version2.x/ppstructure/infer_deploy/paddle_server.md
Normal file
@@ -0,0 +1,279 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
PaddleOCR提供2种服务部署方式：
|
||||
|
||||
- 基于PaddleHub Serving的部署：代码路径为`./deploy/hubserving`，按照本教程使用；
|
||||
- 基于PaddleServing的部署：代码路径为`./deploy/pdserving`，使用方法参考[文档](../../ppocr/infer_deploy/paddle_server.md)。
|
||||
|
||||
# 基于PaddleHub Serving的服务部署
|
||||
|
||||
hubserving服务部署目录下包括文本检测、文本方向分类，文本识别、文本检测+文本方向分类+文本识别3阶段串联，版面分析、表格识别和PP-Structure七种服务包，请根据需求选择相应的服务包进行安装和启动。目录结构如下：
|
||||
|
||||
```text linenums="1"
|
||||
deploy/hubserving/
|
||||
└─ ocr_cls 文本方向分类模块服务包
|
||||
└─ ocr_det 文本检测模块服务包
|
||||
└─ ocr_rec 文本识别模块服务包
|
||||
└─ ocr_system 文本检测+文本方向分类+文本识别串联服务包
|
||||
└─ structure_layout 版面分析服务包
|
||||
└─ structure_table 表格识别服务包
|
||||
└─ structure_system PP-Structure服务包
|
||||
└─ kie_ser 关键信息抽取-SER服务包
|
||||
└─ kie_ser_re 关键信息抽取-SER+RE服务包
|
||||
```
|
||||
|
||||
每个服务包下包含3个文件。以2阶段串联服务包为例，目录如下：
|
||||
|
||||
```text linenums="1"
|
||||
deploy/hubserving/ocr_system/
|
||||
└─ __init__.py 空文件，必选
|
||||
└─ config.json 配置文件，可选，使用配置启动服务时作为参数传入
|
||||
└─ module.py 主模块，必选，包含服务的完整逻辑
|
||||
└─ params.py 参数文件，必选，包含模型路径、前后处理参数等参数
|
||||
```
|
||||
|
||||
## 1. 近期更新
|
||||
|
||||
- 2022.10.09 新增关键信息抽取服务。
|
||||
- 2022.08.23 新增版面分析服务。
|
||||
- 2022.05.05 新增PP-OCRv3检测和识别模型。
|
||||
- 2022.03.30 新增PP-Structure和表格识别两种服务。
|
||||
|
||||
## 2. 快速启动服务
|
||||
|
||||
以下步骤以检测+识别2阶段串联服务为例，如果只需要检测服务或识别服务，替换相应文件路径即可。
|
||||
|
||||
### 2.1 安装PaddleHub
|
||||
|
||||
paddlehub 需要 python>3.6.2
|
||||
|
||||
```bash linenums="1"
|
||||
pip3 install paddlehub==2.1.0 --upgrade -i https://mirror.baidu.com/pypi/simple
|
||||
```
|
||||
|
||||
### 2.2 下载推理模型
|
||||
|
||||
安装服务模块前，需要准备推理模型并放到正确路径。默认使用的是PP-OCRv3模型，默认模型路径为：
|
||||
|
||||
| 模型 | 路径 |
|
||||
| ------------------- | ------------------------------------------------------ |
|
||||
| 检测模型 | `./inference/PP-OCRv3_mobile_det_infer/` |
|
||||
| 识别模型 | `./inference/PP-OCRv3_mobile_rec_infer/` |
|
||||
| 方向分类器 | `./inference/ch_ppocr_mobile_v2.0_cls_infer/` |
|
||||
| 版面分析模型 | `./inference/picodet_lcnet_x1_0_fgd_layout_infer/` |
|
||||
| 表格结构识别模型 | `./inference/ch_ppstructure_mobile_v2.0_SLANet_infer/` |
|
||||
| 关键信息抽取SER模型 | `./inference/ser_vi_layoutxlm_xfund_infer/` |
|
||||
| 关键信息抽取RE模型 | `./inference/re_vi_layoutxlm_xfund_infer/` |
|
||||
|
||||
**模型路径可在`params.py`中查看和修改。**
|
||||
|
||||
更多模型可以从PaddleOCR提供的模型库[PP-OCR](../../ppocr/model_list.md)和[PP-Structure](../models_list.md)下载，也可以替换成自己训练转换好的模型。
|
||||
|
||||
### 2.3 安装服务模块
|
||||
|
||||
PaddleOCR提供5种服务模块，根据需要安装所需模块。
|
||||
|
||||
在Linux环境（Windows环境请将`/`替换为`\`）下，安装模块命令如下表：
|
||||
|
||||
| 服务模块 | 命令 |
|
||||
| ------------------ | ------------------------------------------------ |
|
||||
| 检测 | `hub install deploy/hubserving/ocr_det` |
|
||||
| 分类 | `hub install deploy/hubserving/ocr_cls` |
|
||||
| 识别 | `hub install deploy/hubserving/ocr_rec` |
|
||||
| 检测+识别串联 | `hub install deploy/hubserving/ocr_system` |
|
||||
| 表格识别 | `hub install deploy/hubserving/structure_table` |
|
||||
| PP-Structure | `hub install deploy/hubserving/structure_system` |
|
||||
| 版面分析 | `hub install deploy/hubserving/structure_layout` |
|
||||
| 关键信息抽取SER | `hub install deploy/hubserving/kie_ser` |
|
||||
| 关键信息抽取SER+RE | `hub install deploy/hubserving/kie_ser_re` |
|
||||
|
||||
### 2.4 启动服务
|
||||
|
||||
#### 2.4.1. 命令行命令启动（仅支持CPU）
|
||||
|
||||
**启动命令：**
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving start --modules Module1==Version1, Module2==Version2, ... \
|
||||
--port 8866 \
|
||||
--use_multiprocess \
|
||||
--workers \
|
||||
```
|
||||
|
||||
**参数：**
|
||||
|
||||
| 参数 | 用途 |
|
||||
| ----- | ---- |
|
||||
| `--modules`/`-m` | PaddleHub Serving预安装模型，以多个Module==Version键值对的形式列出<br>**当不指定Version时，默认选择最新版本** |
|
||||
| `--port`/`-p` | 服务端口，默认为8866 |
|
||||
| `--use_multiprocess` | 是否启用并发方式，默认为单进程方式，推荐多核CPU机器使用此方式<br>**Windows操作系统只支持单进程方式** |
|
||||
| `--workers` | 在并发方式下指定的并发任务数，默认为`2*cpu_count-1`，其中`cpu_count`为CPU核数 |
|
||||
|
||||
如启动串联服务：
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving start -m ocr_system
|
||||
```
|
||||
|
||||
这样就完成了一个服务化API的部署，使用默认端口号8866。
|
||||
|
||||
#### 2.4.2 配置文件启动（支持CPU、GPU）
|
||||
|
||||
**启动命令：**
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving start -c config.json
|
||||
```
|
||||
|
||||
其中，`config.json`格式如下：
|
||||
|
||||
```json
|
||||
{
|
||||
"modules_info": {
|
||||
"ocr_system": {
|
||||
"init_args": {
|
||||
"version": "1.0.0",
|
||||
"use_gpu": true
|
||||
},
|
||||
"predict_args": {
|
||||
}
|
||||
}
|
||||
},
|
||||
"port": 8868,
|
||||
"use_multiprocess": false,
|
||||
"workers": 2
|
||||
}
|
||||
```
|
||||
|
||||
- `init_args`中的可配参数与`module.py`中的`_initialize`函数接口一致。
|
||||
|
||||
**当`use_gpu`为`true`时，表示使用GPU启动服务。**
|
||||
- `predict_args`中的可配参数与`module.py`中的`predict`函数接口一致。
|
||||
|
||||
**注意：**
|
||||
|
||||
- 使用配置文件启动服务时，其他参数会被忽略。
|
||||
- 如果使用GPU预测(即，`use_gpu`置为`true`)，则需要在启动服务之前，设置CUDA_VISIBLE_DEVICES环境变量，如：
|
||||
|
||||
```bash linenums="1"
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
```
|
||||
|
||||
- **`use_gpu`不可与`use_multiprocess`同时为`true`**。
|
||||
|
||||
如，使用GPU 3号卡启动串联服务：
|
||||
|
||||
```bash linenums="1"
|
||||
export CUDA_VISIBLE_DEVICES=3
|
||||
hub serving start -c deploy/hubserving/ocr_system/config.json
|
||||
```
|
||||
|
||||
## 3. 发送预测请求
|
||||
|
||||
配置好服务端，可使用以下命令发送预测请求，获取预测结果：
|
||||
|
||||
```bash linenums="1"
|
||||
python tools/test_hubserving.py --server_url=server_url --image_dir=image_path
|
||||
```
|
||||
|
||||
需要给脚本传递2个参数：
|
||||
|
||||
- `server_url`：服务地址，格式为`http://[ip_address]:[port]/predict/[module_name]`
|
||||
|
||||
例如，如果使用配置文件启动分类，检测、识别，检测+分类+识别3阶段，表格识别和PP-Structure服务
|
||||
|
||||
并为每个服务修改了port，那么发送请求的url将分别是：
|
||||
|
||||
```text linenums="1"
|
||||
http://127.0.0.1:8865/predict/ocr_det
|
||||
http://127.0.0.1:8866/predict/ocr_cls
|
||||
http://127.0.0.1:8867/predict/ocr_rec
|
||||
http://127.0.0.1:8868/predict/ocr_system
|
||||
http://127.0.0.1:8869/predict/structure_table
|
||||
http://127.0.0.1:8870/predict/structure_system
|
||||
http://127.0.0.1:8870/predict/structure_layout
|
||||
http://127.0.0.1:8871/predict/kie_ser
|
||||
http://127.0.0.1:8872/predict/kie_ser_re
|
||||
```
|
||||
|
||||
- `image_dir`：测试图像路径，可以是单张图片路径，也可以是图像集合目录路径
|
||||
- `visualize`：是否可视化结果，默认为False
|
||||
- `output`：可视化结果保存路径，默认为`./hubserving_result`
|
||||
|
||||
访问示例：
|
||||
|
||||
```bash linenums="1"
|
||||
python tools/test_hubserving.py --server_url=http://127.0.0.1:8868/predict/ocr_system --image_dir=./doc/imgs/ --visualize=false
|
||||
```
|
||||
|
||||
## 4. 返回结果格式说明
|
||||
|
||||
返回结果为列表（list），列表中的每一项为词典（dict），词典一共可能包含3种字段，信息如下：
|
||||
|
||||
| 字段名称 | 数据类型 | 意义 |
|
||||
| ----------- | -------- | ----- |
|
||||
| angle | str | 文本角度 |
|
||||
| text | str | 文本内容 |
|
||||
| confidence | float | 文本识别置信度或文本角度分类置信度 |
|
||||
| text_region | list | 文本位置坐标 |
|
||||
| html | str | 表格的html字符串 |
|
||||
| regions | list | 版面分析+表格识别+OCR的结果，每一项为一个list<br>包含表示区域坐标的`bbox`，区域类型的`type`和区域结果的`res`三个字段 |
|
||||
| layout | list | 版面分析的结果，每一项一个dict，包含版面区域坐标的`bbox`，区域类型的`label` |
|
||||
|
||||
不同模块返回的字段不同，如，文本识别服务模块返回结果不含`text_region`字段，具体信息如下：
|
||||
|
||||
| 字段名/模块名 | ocr_det | ocr_cls | ocr_rec | ocr_system | structure_table | structure_system | structure_layout | kie_ser | kie_re |
|
||||
| ------------- | ------- | ------- | ------- | ---------- | --------------- | ---------------- | ---------------- | ------- | ------ |
|
||||
| angle | | ✔ | | ✔ | | | |
|
||||
| text | | | ✔ | ✔ | | ✔ | | ✔ | ✔ |
|
||||
| confidence | | ✔ | ✔ | ✔ | | ✔ | | ✔ | ✔ |
|
||||
| text_region | ✔ | | | ✔ | | ✔ | | ✔ | ✔ |
|
||||
| html | | | | | ✔ | ✔ | | | |
|
||||
| regions | | | | | ✔ | ✔ | | | |
|
||||
| layout | | | | | | | ✔ | | |
|
||||
| ser_res | | | | | | | | ✔ | |
|
||||
| re_res | | | | | | | | | ✔ |
|
||||
|
||||
**说明：** 如果需要增加、删除、修改返回字段，可在相应模块的`module.py`文件中进行修改，完整流程参考下一节自定义修改服务模块。
|
||||
|
||||
## 5. 自定义修改服务模块
|
||||
|
||||
如果需要修改服务逻辑，一般需要操作以下步骤（以修改`deploy/hubserving/ocr_system`为例）：
|
||||
|
||||
1. 停止服务：
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving stop --port/-p XXXX
|
||||
```
|
||||
|
||||
2. 到`deploy/hubserving/ocr_system`下的`module.py`和`params.py`等文件中根据实际需求修改代码。
|
||||
|
||||
例如，如果需要替换部署服务所用模型，则需要到`params.py`中修改模型路径参数`det_model_dir`和`rec_model_dir`，如果需要关闭文本方向分类器，则将参数`use_angle_cls`置为`False`
|
||||
|
||||
当然，同时可能还需要修改其他相关参数，请根据实际情况修改调试。
|
||||
|
||||
**强烈建议修改后先直接运行`module.py`调试，能正确运行预测后再启动服务测试。**
|
||||
|
||||
**注意：** PPOCR-v3识别模型使用的图片输入shape为`3,48,320`,因此需要修改`params.py`中的`cfg.rec_image_shape = "3, 48, 320"`，如果不使用PPOCR-v3识别模型，则无需修改该参数。
|
||||
3. （可选）如果想要重命名模块需要更改`module.py`文件中的以下行：
|
||||
- [`from deploy.hubserving.ocr_system.params import read_params`中的`ocr_system`](https://github.com/PaddlePaddle/PaddleOCR/blob/a923f35de57b5e378f8dd16e54d0a3e4f51267fd/deploy/hubserving/ocr_system/module.py#L35)
|
||||
- [`name="ocr_system",`中的`ocr_system`](https://github.com/PaddlePaddle/PaddleOCR/blob/a923f35de57b5e378f8dd16e54d0a3e4f51267fd/deploy/hubserving/ocr_system/module.py#L39)
|
||||
4. （可选）可能需要删除`__pycache__`目录以强制刷新CPython缓存：
|
||||
|
||||
```bash linenums="1"
|
||||
find deploy/hubserving/ocr_system -name '__pycache__' -exec rm -r {} \;
|
||||
```
|
||||
|
||||
5. 安装修改后的新服务包：
|
||||
|
||||
```bash linenums="1"
|
||||
hub install deploy/hubserving/ocr_system
|
||||
```
|
||||
|
||||
6. 重新启动服务：
|
||||
|
||||
```bash linenums="1"
|
||||
hub serving start -m ocr_system
|
||||
```
|
||||
116
docs/version2.x/ppstructure/infer_deploy/python_infer.en.md
Normal file
116
docs/version2.x/ppstructure/infer_deploy/python_infer.en.md
Normal file
@@ -0,0 +1,116 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Python Inference
|
||||
|
||||
## 1. Layout Structured Analysis
|
||||
|
||||
Go to the `ppstructure` directory
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
|
||||
# download model
|
||||
mkdir inference && cd inference
|
||||
# Download the PP-StructureV2 layout analysis model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_layout_infer.tar && tar xf picodet_lcnet_x1_0_layout_infer.tar
|
||||
# Download the PP-OCRv3 text detection model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_det_infer.tar && tar xf PP-OCRv3_mobile_det_infer.tar
|
||||
# Download the PP-OCRv3 text recognition model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_rec_infer.tar && tar xf PP-OCRv3_mobile_rec_infer.tar
|
||||
# Download the PP-StructureV2 form recognition model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
### 1.1 layout analysis + table recognition
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py --det_model_dir=inference/PP-OCRv3_mobile_det_infer \
|
||||
--rec_model_dir=inference/PP-OCRv3_mobile_rec_infer \
|
||||
--table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--layout_model_dir=inference/picodet_lcnet_x1_0_layout_infer \
|
||||
--image_dir=./docs/table/1.png \
|
||||
--rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
|
||||
--output=../output \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf
|
||||
```
|
||||
|
||||
After the operation is completed, each image will have a directory with the same name in the `structure` directory under the directory specified by the `output` field. Each table in the image will be stored as an excel, and the picture area will be cropped and saved. The filename of excel and picture is their coordinates in the image. Detailed results are stored in the `res.txt` file.
|
||||
|
||||
### 1.2 layout analysis
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py --layout_model_dir=inference/picodet_lcnet_x1_0_layout_infer \
|
||||
--image_dir=./docs/table/1.png \
|
||||
--output=../output \
|
||||
--table=false \
|
||||
--ocr=false
|
||||
```
|
||||
|
||||
After the operation is completed, each image will have a directory with the same name in the `structure` directory under the directory specified by the `output` field. Each picture in image will be cropped and saved. The filename of picture area is their coordinates in the image. Layout analysis results will be stored in the `res.txt` file
|
||||
|
||||
### 1.3 table recognition
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py --det_model_dir=inference/PP-OCRv3_mobile_det_infer \
|
||||
--rec_model_dir=inference/PP-OCRv3_mobile_rec_infer \
|
||||
--table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--image_dir=./docs/table/table.jpg \
|
||||
--rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
|
||||
--output=../output \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--layout=false
|
||||
```
|
||||
|
||||
After the operation is completed, each image will have a directory with the same name in the `structure` directory under the directory specified by the `output` field. Each table in the image will be stored as an excel. The filename of excel is their coordinates in the image.
|
||||
|
||||
## 2. Key Information Extraction
|
||||
|
||||
### 2.1 SER
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
|
||||
mkdir inference && cd inference
|
||||
# download model
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
|
||||
cd ..
|
||||
python3 predict_system.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--ser_model_dir=./inference/ser_vi_layoutxlm_xfund_infer \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../ppocr/utils/dict/kie_dict/xfund_class_list.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx" \
|
||||
--mode=kie
|
||||
```
|
||||
|
||||
After the operation is completed, each image will store the visualized image in the `kie` directory under the directory specified by the `output` field, and the image name is the same as the input image name.
|
||||
|
||||
### 2.2 RE+SER
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
|
||||
mkdir inference && cd inference
|
||||
# download model
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_infer.tar && tar -xf re_vi_layoutxlm_xfund_infer.tar
|
||||
cd ..
|
||||
|
||||
python3 predict_system.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--re_model_dir=./inference/re_vi_layoutxlm_xfund_infer \
|
||||
--ser_model_dir=./inference/ser_vi_layoutxlm_xfund_infer \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../ppocr/utils/dict/kie_dict/xfund_class_list.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx" \
|
||||
--mode=kie
|
||||
```
|
||||
|
||||
After the operation is completed, each image will have a directory with the same name in the `kie` directory under the directory specified by the `output` field, where the visual images and prediction results are stored.
|
||||
119
docs/version2.x/ppstructure/infer_deploy/python_infer.md
Normal file
119
docs/version2.x/ppstructure/infer_deploy/python_infer.md
Normal file
@@ -0,0 +1,119 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 基于Python预测引擎推理
|
||||
|
||||
## 1. 版面信息抽取
|
||||
|
||||
进入`ppstructure`目录
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
```
|
||||
|
||||
下载模型
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir inference && cd inference
|
||||
# 下载PP-StructureV2版面分析模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_layout_infer.tar && tar xf picodet_lcnet_x1_0_layout_infer.tar
|
||||
# 下载PP-OCRv3文本检测模型并解压
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_det_infer.tar && tar xf PP-OCRv3_mobile_det_infer.tar
|
||||
# 下载PP-OCRv3文本识别模型并解压
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_rec_infer.tar && tar xf PP-OCRv3_mobile_rec_infer.tar
|
||||
# 下载PP-StructureV2表格识别模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
### 1.1 版面分析+表格识别
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py --det_model_dir=inference/PP-OCRv3_mobile_det_infer \
|
||||
--rec_model_dir=inference/PP-OCRv3_mobile_rec_infer \
|
||||
--table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--layout_model_dir=inference/picodet_lcnet_x1_0_layout_infer \
|
||||
--image_dir=./docs/table/1.png \
|
||||
--rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
|
||||
--output=../output \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf
|
||||
```
|
||||
|
||||
运行完成后，每张图片会在`output`字段指定的目录下的`structure`目录下有一个同名目录，图片里的每个表格会存储为一个excel，图片区域会被裁剪之后保存下来，excel文件和图片名为表格在图片里的坐标。详细的结果会存储在`res.txt`文件中。
|
||||
|
||||
### 1.2 版面分析
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py --layout_model_dir=inference/picodet_lcnet_x1_0_layout_infer \
|
||||
--image_dir=./docs/table/1.png \
|
||||
--output=../output \
|
||||
--table=false \
|
||||
--ocr=false
|
||||
```
|
||||
|
||||
运行完成后，每张图片会在`output`字段指定的目录下的`structure`目录下有一个同名目录，图片区域会被裁剪之后保存下来，图片名为表格在图片里的坐标。版面分析结果会存储在`res.txt`文件中。
|
||||
|
||||
### 1.3 表格识别
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py --det_model_dir=inference/PP-OCRv3_mobile_det_infer \
|
||||
--rec_model_dir=inference/PP-OCRv3_mobile_rec_infer \
|
||||
--table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--image_dir=./docs/table/table.jpg \
|
||||
--rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
|
||||
--output=../output \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--layout=false
|
||||
```
|
||||
|
||||
运行完成后，每张图片会在`output`字段指定的目录下的`structure`目录下有一个同名目录，表格会存储为一个excel，excel文件名为`[0,0,img_h,img_w]`。
|
||||
|
||||
## 2. 关键信息抽取
|
||||
|
||||
### 2.1 SER
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
|
||||
mkdir inference && cd inference
|
||||
# 下载SER XFUND 模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
|
||||
cd ..
|
||||
python3 predict_system.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--ser_model_dir=./inference/ser_vi_layoutxlm_xfund_infer \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../ppocr/utils/dict/kie_dict/xfund_class_list.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx" \
|
||||
--mode=kie
|
||||
```
|
||||
|
||||
运行完成后，每张图片会在`output`字段指定的目录下的`kie`目录下存放可视化之后的图片，图片名和输入图片名一致。
|
||||
|
||||
### 2.2 RE+SER
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
|
||||
mkdir inference && cd inference
|
||||
# 下载RE SER XFUND 模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_infer.tar && tar -xf re_vi_layoutxlm_xfund_infer.tar
|
||||
cd ..
|
||||
|
||||
python3 predict_system.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--re_model_dir=./inference/re_vi_layoutxlm_xfund_infer \
|
||||
--ser_model_dir=./inference/ser_vi_layoutxlm_xfund_infer \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../ppocr/utils/dict/kie_dict/xfund_class_list.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx" \
|
||||
--mode=kie
|
||||
```
|
||||
|
||||
运行完成后，每张图片会在`output`字段指定的目录下的`kie`目录下有一个同名目录，目录中存放可视化图片和预测结果。
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 234 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 143 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 125 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 347 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 404 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 247 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 565 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 736 KiB |
BIN
docs/version2.x/ppstructure/model_train/images/layout.jpg
Normal file
BIN
docs/version2.x/ppstructure/model_train/images/layout.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 535 KiB |
BIN
docs/version2.x/ppstructure/model_train/images/layout_res.jpg
Normal file
BIN
docs/version2.x/ppstructure/model_train/images/layout_res.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 451 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 385 KiB |
BIN
docs/version2.x/ppstructure/model_train/images/recovery_ch.jpg
Normal file
BIN
docs/version2.x/ppstructure/model_train/images/recovery_ch.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 315 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 369 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 772 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 467 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 552 KiB |
BIN
docs/version2.x/ppstructure/model_train/images/zh_val_42_ser.jpg
Normal file
BIN
docs/version2.x/ppstructure/model_train/images/zh_val_42_ser.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 692 KiB |
184
docs/version2.x/ppstructure/model_train/recovery_to_doc.en.md
Normal file
184
docs/version2.x/ppstructure/model_train/recovery_to_doc.en.md
Normal file
@@ -0,0 +1,184 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Layout Recovery
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
The layout recovery module is used to restore the image or pdf to an
|
||||
editable Word file consistent with the original image layout.
|
||||
|
||||
Two layout recovery methods are provided, you can choose by PDF format:
|
||||
|
||||
- **Standard PDF parse(the input is standard PDF)**: Python based PDF to word library [pdf2docx](https://github.com/dothinking/pdf2docx) is optimized, the method extracts data from PDF with PyMuPDF, then parse layout with rule, finally, generate docx with python-docx.
|
||||
|
||||
- **Image format PDF parse(the input can be standard PDF or image format PDF)**: Layout recovery combines [layout analysis](./train_layout.en.md)、[table recognition](./train_table.en.md) to better recover images, tables, titles, etc. supports input files in PDF and document image formats in Chinese and English.
|
||||
|
||||
The input formats and application scenarios of the two methods are as follows:
|
||||
|
||||
| method | input formats | application scenarios/problem |
|
||||
| :-----: | :----------: | :----------------------------------------------------------: |
|
||||
| Standard PDF parse | pdf | Advantages: Better recovery for non-paper documents, each page remains on the same page after restoration<br>Disadvantages: English characters in some Chinese documents are garbled, some contents are still beyond the current page, the whole page content is restored to the table format, and the recovery effect of some pictures is not good |
|
||||
| Image format PDF parse( | pdf、picture | Advantages: More suitable for paper document content recovery, OCR recognition effect is more good<br>Disadvantages: Currently, the recovery is based on rules, the effect of content typesetting (spacing, fonts, etc.) need to be further improved, and the effect of layout recovery depends on layout analysis |
|
||||
|
||||
The following figure shows the effect of restoring the layout of documents by using PDF parse:
|
||||
|
||||
|
||||
|
||||
The following figures show the effect of restoring the layout of English and Chinese documents by using OCR technique:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 2. Install
|
||||
|
||||
### 2.1 Install PaddlePaddle
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install --upgrade pip
|
||||
|
||||
# If you have cuda9 or cuda10 installed on your machine, please run the following command to install
|
||||
python3 -m pip install "paddlepaddle-gpu" -i https://mirror.baidu.com/pypi/simple
|
||||
|
||||
# CPU installation
|
||||
python3 -m pip install "paddlepaddle" -i https://mirror.baidu.com/pypi/simple
|
||||
````
|
||||
|
||||
For more requirements, please refer to the instructions in [Installation Documentation](https://www.paddlepaddle.org.cn/en/install/quick?docurl=/documentation/docs/en/install/pip/macos-pip_en.html).
|
||||
|
||||
### 2.2 Install PaddleOCR
|
||||
|
||||
- **(1) Download source code**
|
||||
|
||||
```bash linenums="1"
|
||||
[Recommended] git clone https://github.com/PaddlePaddle/PaddleOCR
|
||||
|
||||
# If the pull cannot be successful due to network problems, you can also choose to use the hosting on the code cloud:
|
||||
git clone https://gitee.com/paddlepaddle/PaddleOCR
|
||||
|
||||
# Note: Code cloud hosting code may not be able to synchronize the update of this github project in real time, there is a delay of 3 to 5 days, please use the recommended method first.
|
||||
````
|
||||
|
||||
- **(2) Install recovery `requirements`**
|
||||
|
||||
The layout restoration is exported as docx files, so python-docx API need to be installed, and PyMuPDF api([requires Python >= 3.7](https://pypi.org/project/PyMuPDF/)) need to be installed to process the input files in pdf format.
|
||||
|
||||
Install all the libraries by running the following command:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install -r ppstructure/recovery/requirements.txt
|
||||
````
|
||||
|
||||
And if using pdf parse method, we need to install pdf2docx api.
|
||||
|
||||
```bash linenums="1"
|
||||
wget https://paddleocr.bj.bcebos.com/whl/pdf2docx-0.0.0-py3-none-any.whl
|
||||
pip3 install pdf2docx-0.0.0-py3-none-any.whl
|
||||
```
|
||||
|
||||
## 3. Quick Start using standard PDF parse
|
||||
|
||||
`use_pdf2docx_api` use PDF parse for layout recovery, The whl package is also provided for quick use, follow the above code, for more information please refer to [quickstart](../quick_start.en.md) for details.
|
||||
|
||||
```bash linenums="1"
|
||||
# install paddleocr
|
||||
pip3 install "paddleocr>=2.6"
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --use_pdf2docx_api=true
|
||||
```
|
||||
|
||||
Command line:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py \
|
||||
--image_dir=ppstructure/docs/recovery/UnrealText.pdf \
|
||||
--recovery=True \
|
||||
--use_pdf2docx_api=True \
|
||||
--output=../output/
|
||||
```
|
||||
|
||||
## 4. Quick Start using image format PDF parse
|
||||
|
||||
Through layout analysis, we divided the image/PDF documents into regions, located the key regions, such as text, table, picture, etc., and recorded the location, category, and regional pixel value information of each region. Different regions are processed separately, where:
|
||||
|
||||
- OCR detection and recognition is performed in the text area, and the coordinates of the OCR detection box and the text content information are added on the basis of the previous information
|
||||
|
||||
- The table area identifies tables and records html and text information of tables
|
||||
- Save the image directly
|
||||
|
||||
We can restore the test picture through the layout information, OCR detection and recognition structure, table information, and saved pictures.
|
||||
|
||||
The whl package is also provided for quick use, follow the above code, for more information please refer to [quickstart](../quick_start.en.md) for details.
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --recovery=true --lang='en'
|
||||
```
|
||||
|
||||
### 4.1 Download models
|
||||
|
||||
If input is English document, download English models:
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
|
||||
# download model
|
||||
mkdir inference && cd inference
|
||||
# Download the detection model of the ultra-lightweight English PP-OCRv3 model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar && tar xf en_PP-OCRv3_det_infer.tar
|
||||
# Download the recognition model of the ultra-lightweight English PP-OCRv3 model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/en_PP-OCRv3_mobile_rec_infer.tar && tar xf en_PP-OCRv3_mobile_rec_infer.tar
|
||||
# Download the ultra-lightweight English table inch model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
tar xf en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
# Download the layout model of publaynet dataset and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
tar xf picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
If input is Chinese document，download Chinese models:
|
||||
[Chinese and English ultra-lightweight PP-OCRv3 model](../../ppocr/model_list.md)
|
||||
|
||||
### 4.2 Layout recovery
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py \
|
||||
--image_dir=./docs/table/1.png \
|
||||
--det_model_dir=inference/en_PP-OCRv3_det_infer \
|
||||
--rec_model_dir=inference/en_PP-OCRv3_mobile_rec_infer \
|
||||
--rec_char_dict_path=../ppocr/utils/en_dict.txt \
|
||||
--table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
|
||||
--layout_model_dir=inference/picodet_lcnet_x1_0_fgd_layout_infer \
|
||||
--layout_dict_path=../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--recovery=True \
|
||||
--output=../output/
|
||||
```
|
||||
|
||||
After running, the docx of each picture will be saved in the directory specified by the output field
|
||||
|
||||
Field：
|
||||
|
||||
- image_dir：test file， can be picture, picture directory, pdf file, pdf file directory
|
||||
- det_model_dir：OCR detection model path
|
||||
- rec_model_dir：OCR recognition model path
|
||||
- rec_char_dict_path：OCR recognition dict path. If the Chinese model is used, change to "../ppocr/utils/ppocr_keys_v1.txt". And if you trained the model on your own dataset, change to the trained dictionary
|
||||
- table_model_dir：table recognition model path
|
||||
- table_char_dict_path：table recognition dict path. If the Chinese model is used, no need to change
|
||||
- layout_model_dir：layout analysis model path
|
||||
- layout_dict_path：layout analysis dict path. If the Chinese model is used, change to "../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt"
|
||||
- recovery：whether to enable layout of recovery, default False
|
||||
- output：save the recovery result path
|
||||
|
||||
## 5. More
|
||||
|
||||
For training, evaluation and inference tutorial for text detection models, please refer to [text detection doc](../../ppocr/model_train/detection.en.md).
|
||||
|
||||
For training, evaluation and inference tutorial for text recognition models, please refer to [text recognition doc](../../ppocr/model_train/recognition.en.md).
|
||||
|
||||
For training, evaluation and inference tutorial for layout analysis models, please refer to [layout analysis doc](./train_layout.en.md)
|
||||
|
||||
For training, evaluation and inference tutorial for table recognition models, please refer to [table recognition doc](./train_table.en.md)
|
||||
191
docs/version2.x/ppstructure/model_train/recovery_to_doc.md
Normal file
191
docs/version2.x/ppstructure/model_train/recovery_to_doc.md
Normal file
@@ -0,0 +1,191 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 版面恢复
|
||||
|
||||
## 1. 简介
|
||||
|
||||
版面恢复就是将输入的图片、pdf内容仍然像原文档那样排列着，段落不变、顺序不变的输出到word文档中等。
|
||||
|
||||
提供了2种版面恢复方法，可根据输入PDF的格式进行选择：
|
||||
|
||||
- **标准PDF解析(输入须为标准PDF)**：基于Python的pdf转word库[pdf2docx](https://github.com/dothinking/pdf2docx)进行优化，该方法通过PyMuPDF获取页面元素，然后利用规则解析章节、段落、表格等布局及样式，最后通过python-docx将解析的内容元素重建到word文档中。
|
||||
- **图片格式PDF解析(输入可为标准PDF或图片格式PDF)**：结合[版面分析](./train_layout.md)、[表格识别](./train_table.md)技术，从而更好地恢复图片、表格、标题等内容，支持中、英文pdf文档、文档图片格式的输入文件。
|
||||
|
||||
2种方法输入格式、适用场景如下：
|
||||
|
||||
| 方法 | 支持输入文件 | 适用场景/存在问题 |
|
||||
| :-------------: | :----------: | :----------------------------------------------------------: |
|
||||
| 标准PDF解析 | pdf | 优点：非论文文档恢复效果更优、每一页内容恢复后仍在同一页<br>缺点：有些中文文档中的英文乱码、仍存在内容超出当前页面的情况、整页内容恢复为表格格式、部分图片恢复效果不佳 |
|
||||
| 图片格式PDF解析 | pdf、图片 | 优点：更适合论文文档正文内容的恢复、中英文文档OCR识别效果好<br>缺点：目前内容恢复基于规则，内容排版效果(间距、字体等)待进一步提升、版面恢复效果依赖于版面分析效果 |
|
||||
|
||||
下图展示了通过PDF解析版面恢复效果：
|
||||
|
||||
|
||||
|
||||
下图分别展示了通过OCR技术，英文文档和中文文档版面恢复的效果：
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 2. 安装
|
||||
|
||||
### 2.1 安装PaddlePaddle
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install --upgrade pip
|
||||
|
||||
# 您的机器安装的是CUDA9或CUDA10，请运行以下命令安装
|
||||
python3 -m pip install "paddlepaddle-gpu" -i https://mirror.baidu.com/pypi/simple
|
||||
|
||||
# 您的机器是CPU，请运行以下命令安装
|
||||
python3 -m pip install "paddlepaddle" -i https://mirror.baidu.com/pypi/simple
|
||||
```
|
||||
|
||||
更多需求，请参照[安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。
|
||||
|
||||
### 2.2 安装PaddleOCR
|
||||
|
||||
- **（1）下载版面恢复源码**
|
||||
|
||||
```bash linenums="1"
|
||||
【推荐】git clone https://github.com/PaddlePaddle/PaddleOCR
|
||||
|
||||
# 如果因为网络问题无法pull成功，也可选择使用码云上的托管：
|
||||
git clone https://gitee.com/paddlepaddle/PaddleOCR
|
||||
|
||||
# 注：码云托管代码可能无法实时同步本github项目更新，存在3~5天延时，请优先使用推荐方式。
|
||||
```
|
||||
|
||||
- **（2）安装recovery的`requirements`**
|
||||
|
||||
版面恢复导出为docx文件，所以需要安装Python处理word文档的python-docx API，同时处理pdf格式的输入文件，需要安装PyMuPDF API([要求Python >= 3.7](https://pypi.org/project/PyMuPDF/))。
|
||||
|
||||
通过如下命令安装全部库：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install -r ppstructure/recovery/requirements.txt
|
||||
```
|
||||
|
||||
使用pdf2docx库解析的方式恢复文档需要安装优化的pdf2docx。
|
||||
|
||||
```bash linenums="1"
|
||||
wget https://paddleocr.bj.bcebos.com/whl/pdf2docx-0.0.0-py3-none-any.whl
|
||||
pip3 install pdf2docx-0.0.0-py3-none-any.whl
|
||||
```
|
||||
|
||||
## 3.使用标准PDF解析进行版面恢复
|
||||
|
||||
`use_pdf2docx_api`表示使用PDF解析的方式进行版面恢复，通过whl包的形式方便快速使用，代码如下，更多信息详见 [quickstart](../quick_start.md)。
|
||||
|
||||
```bash linenums="1"
|
||||
# 安装 paddleocr，推荐使用2.6版本
|
||||
pip3 install "paddleocr>=2.6"
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --use_pdf2docx_api=true
|
||||
```
|
||||
|
||||
通过命令行的方式：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py \
|
||||
--image_dir=ppstructure/docs/recovery/UnrealText.pdf \
|
||||
--recovery=True \
|
||||
--use_pdf2docx_api=True \
|
||||
--output=../output/
|
||||
```
|
||||
|
||||
## 4.使用图片格式PDF解析进行版面恢复
|
||||
|
||||
我们通过版面分析对图片/pdf形式的文档进行区域划分，定位其中的关键区域，如文字、表格、图片等，记录每个区域的位置、类别、区域像素值信息。对不同的区域分别处理，其中：
|
||||
|
||||
- 文字区域直接进行OCR检测和识别，在之前信息基础上增加OCR检测框坐标和文本内容信息
|
||||
- 表格区域进行表格识别，记录表格html和文字信息
|
||||
- 图片直接保存
|
||||
|
||||
我们通过版面信息、OCR检测和识别结构、表格信息、保存的图片，对测试图片进行恢复即可。
|
||||
|
||||
提供如下代码实现版面恢复，也提供了whl包的形式方便快速使用，代码如下，更多信息详见 [quickstart](../quick_start.md)。
|
||||
|
||||
```bash linenums="1"
|
||||
# 安装 paddleocr，推荐使用2.6版本
|
||||
pip3 install "paddleocr>=2.6"
|
||||
# 中文测试图
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --recovery=true
|
||||
# 英文测试图
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --recovery=true --lang='en'
|
||||
# pdf测试文件
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --lang='en'
|
||||
```
|
||||
|
||||
### 4.1 下载模型
|
||||
|
||||
如果输入为英文文档类型，下载OCR检测和识别、版面分析、表格识别的英文模型
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
|
||||
# 下载模型
|
||||
mkdir inference && cd inference
|
||||
# 下载英文超轻量PP-OCRv3检测模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar && tar xf en_PP-OCRv3_det_infer.tar
|
||||
# 下载英文超轻量PP-OCRv3识别模型并解压
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/en_PP-OCRv3_mobile_rec_infer.tar && tar xf en_PP-OCRv3_mobile_rec_infer.tar
|
||||
# 下载英文表格识别模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
tar xf en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
# 下载英文版面分析模型
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
tar xf picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
如果输入为中文文档类型，在下述链接中下载中文模型即可：
|
||||
|
||||
[PP-OCRv3中英文超轻量文本检测和识别模型](../../ppocr/model_list.md)
|
||||
|
||||
### 4.2 版面恢复
|
||||
|
||||
使用下载的模型恢复给定文档的版面，以英文模型为例，执行如下命令：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 predict_system.py \
|
||||
--image_dir=./docs/table/1.png \
|
||||
--det_model_dir=inference/en_PP-OCRv3_det_infer \
|
||||
--rec_model_dir=inference/en_PP-OCRv3_mobile_rec_infer \
|
||||
--rec_char_dict_path=../ppocr/utils/en_dict.txt \
|
||||
--table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
|
||||
--layout_model_dir=inference/picodet_lcnet_x1_0_fgd_layout_infer \
|
||||
--layout_dict_path=../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--recovery=True \
|
||||
--output=../output/
|
||||
```
|
||||
|
||||
运行完成后，恢复版面的docx文档会保存到`output`字段指定的目录下
|
||||
|
||||
字段含义：
|
||||
|
||||
- image_dir：测试文件，可以是图片、图片目录、pdf文件、pdf文件目录
|
||||
- det_model_dir：OCR检测模型路径
|
||||
- rec_model_dir：OCR识别模型路径
|
||||
- rec_char_dict_path：OCR识别字典，如果更换为中文模型，需要更改为"../ppocr/utils/ppocr_keys_v1.txt"，如果您在自己的数据集上训练的模型，则更改为训练的字典的文件
|
||||
- table_model_dir：表格识别模型路径
|
||||
- table_char_dict_path：表格识别字典，如果更换为中文模型，不需要更换字典
|
||||
- layout_model_dir：版面分析模型路径
|
||||
- layout_dict_path：版面分析字典，如果更换为中文模型，需要更改为"../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt"
|
||||
- recovery：是否进行版面恢复，默认False
|
||||
- output：版面恢复结果保存路径
|
||||
|
||||
## 5. 更多
|
||||
|
||||
关于OCR检测模型的训练评估与推理，请参考：[文本检测教程](../../ppocr/model_train/detection.md)
|
||||
|
||||
关于OCR识别模型的训练评估与推理，请参考：[文本识别教程](../../ppocr/model_train/recognition.md)
|
||||
|
||||
关于版面分析模型的训练评估与推理，请参考：[版面分析教程](./train_layout.md)
|
||||
|
||||
关于表格识别模型的训练评估与推理，请参考：[表格识别教程](./train_table.md)
|
||||
245
docs/version2.x/ppstructure/model_train/train_kie.en.md
Normal file
245
docs/version2.x/ppstructure/model_train/train_kie.en.md
Normal file
@@ -0,0 +1,245 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Key Information Extraction (KIE)
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
Key information extraction (KIE) refers to extracting key information from text or images. As downstream task of OCR, the key information extraction task of document image has many practical application scenarios, such as form recognition, ticket information extraction, ID card information extraction, etc.
|
||||
|
||||
PP-Structure conducts research based on the LayoutXLM multi-modal, and proposes the VI-LayoutXLM, which gets rid of visual features when finetuning the downstream tasks. An textline sorting method is also utilized to fit in reading order. What's more, UDML knowledge distillation is used for higher accuracy. Finally, the accuracy and inference speed of VI-LayoutXLM surpass those of LayoutXLM.
|
||||
|
||||
The main features of the key information extraction module in PP-Structure are as follows.
|
||||
|
||||
- Integrate multi-modal methods such as [LayoutXLM](https://arxiv.org/pdf/2104.08836.pdf), VI-LayoutXLM, and PP-OCR inference engine.
|
||||
- Supports Semantic Entity Recognition (SER) and Relation Extraction (RE) tasks based on multimodal methods. Based on the SER task, the text recognition and classification in the image can be completed; based on the RE task, the relationship extraction of the text content in the image can be completed, such as judging the problem pair (pair).
|
||||
- Supports custom training for SER tasks and RE tasks.
|
||||
- Supports end-to-end system prediction and evaluation of OCR+SER.
|
||||
- Supports end-to-end system prediction of OCR+SER+RE.
|
||||
- Support SER model export and inference using PaddleInference.
|
||||
|
||||
## 2. Performance
|
||||
|
||||
We evaluate the methods on the Chinese dataset of [XFUND](https://github.com/doc-analysis/XFUND), and the performance is as follows
|
||||
|
||||
|Model | Backbone | Task | Config file | Hmean | Inference time (ms) | Download link|
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
|VI-LayoutXLM| VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|**93.19%**| 15.49|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)|
|
||||
|LayoutXLM| LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%| 19.49 | [trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)|
|
||||
|VI-LayoutXLM| VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|**83.92%**| 15.49|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)|
|
||||
|LayoutXLM| LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%| 19.49|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)|
|
||||
|
||||
- Note：Inference environment：V100 GPU + cuda10.2 + cudnn8.1.1 + TensorRT 7.2.3.4，tested using fp16.
|
||||
|
||||
For more KIE models in PaddleOCR, please refer to [KIE model zoo](../../algorithm/overview.en.md).
|
||||
|
||||
## 3. Visualization
|
||||
|
||||
There are two main solutions to the key information extraction task based on VI-LayoutXLM series model.
|
||||
|
||||
(1) Text detection + text recognition + semantic entity recognition (SER)
|
||||
|
||||
(2) Text detection + text recognition + semantic entity recognition (SER) + relationship extraction (RE)
|
||||
|
||||
The following images are demo results of the SER and RE models. For more detailed introduction to the above solutions, please refer to [KIE Guide](../blog/how_to_do_kie.en.md).
|
||||
|
||||
### 3.1 SER
|
||||
|
||||
Demo results for SER task are as follows.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
**Note:** test pictures are from [xfund dataset](https://github.com/doc-analysis/XFUND), [invoice dataset](https://aistudio.baidu.com/aistudio/datasetdetail/165561) and a composite ID card dataset.
|
||||
|
||||
Boxes of different colors in the image represent different categories.
|
||||
|
||||
The invoice and application form images have three categories: `request`, `answer` and `header`. The `question` and `answer` can be used to extract the relationship.
|
||||
|
||||
For the ID card image, the model can directly identify the key information such as `name`, `gender`, `nationality`, so that the subsequent relationship extraction process is not required, and the key information extraction task can be completed using only one model.
|
||||
|
||||
### 3.2 RE
|
||||
|
||||
Demo results for RE task are as follows.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Red boxes are questions, blue boxes are answers. The green lines means the two connected objects are a pair.
|
||||
|
||||
## 4. Usage
|
||||
|
||||
### 4.1 Prepare for the environment
|
||||
|
||||
Use the following command to install KIE dependencies.
|
||||
|
||||
```bash linenums="1"
|
||||
git clone https://github.com/PaddlePaddle/PaddleOCR.git
|
||||
cd PaddleOCR
|
||||
pip install -r requirements.txt
|
||||
pip install -r ppstructure/kie/requirements.txt
|
||||
# 安装PaddleOCR引擎用于预测
|
||||
pip install "paddleocr<3.0"
|
||||
```
|
||||
|
||||
NOTE: For KIE tasks, it is necessary to downgrade the Paddle framework version (Paddle<2.6) and the PaddleNLP version (PaddleNLP<2.6).
|
||||
|
||||
The visualized results of SER are saved in the `./output` folder by default. Examples of results are as follows.
|
||||
|
||||
|
||||
|
||||
### 4.2 Quick start
|
||||
|
||||
Here we use XFUND dataset to quickly experience the SER model and RE model.
|
||||
|
||||
#### 4.2.1 Prepare for the dataset
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir train_data
|
||||
cd train_data
|
||||
# download and uncompress the dataset
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar && tar -xf XFUND.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
#### 4.2.2 Predict images using the trained model
|
||||
|
||||
Use the following command to download the models.
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir pretrained_model
|
||||
cd pretrained_model
|
||||
# download and uncompress the SER trained model
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar && tar -xf ser_vi_layoutxlm_xfund_pretrained.tar
|
||||
|
||||
# download and uncompress the RE trained model
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar && tar -xf re_vi_layoutxlm_xfund_pretrained.tar
|
||||
```
|
||||
|
||||
If you want to use OCR engine to obtain end-to-end prediction results, you can use the following command to predict.
|
||||
|
||||
```bash linenums="1"
|
||||
# just predict using SER trained model
|
||||
python3 tools/infer_kie_token_ser.py \
|
||||
-c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy \
|
||||
Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg
|
||||
|
||||
# predict using SER and RE trained model at the same time
|
||||
python3 ./tools/infer_kie_token_ser_re.py \
|
||||
-c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrained_model/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/image/zh_val_42.jpg \
|
||||
-c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o_ser Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
|
||||
```
|
||||
|
||||
The visual result images and the predicted text file will be saved in the `Global.save_res_path` directory.
|
||||
|
||||
If you want to use a custom ocr model, you can set it through the following fields
|
||||
|
||||
- `Global.kie_det_model_dir`: the detection inference model path
|
||||
- `Global.kie_rec_model_dir`: the recognition inference model path
|
||||
|
||||
If you want to load the text detection and recognition results collected before, you can use the following command to predict.
|
||||
|
||||
```bash linenums="1"
|
||||
# just predict using SER trained model
|
||||
python3 tools/infer_kie_token_ser.py \
|
||||
-c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/val.json \
|
||||
Global.infer_mode=False
|
||||
|
||||
# predict using SER and RE trained model at the same time
|
||||
python3 ./tools/infer_kie_token_ser_re.py \
|
||||
-c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrained_model/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/val.json \
|
||||
Global.infer_mode=False \
|
||||
-c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o_ser Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
|
||||
```
|
||||
|
||||
#### 4.2.3 Inference using PaddleInference
|
||||
|
||||
Firstly, download the inference SER inference model.
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir inference
|
||||
cd inference
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_infer.tar && tar -xf re_vi_layoutxlm_xfund_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
- SER
|
||||
|
||||
Use the following command for inference.
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
python3 kie/predict_kie_token_ser.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--ser_model_dir=../inference/ser_vi_layoutxlm_xfund_infer \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx"
|
||||
```
|
||||
|
||||
The visual results and text file will be saved in directory `output`.
|
||||
|
||||
- RE
|
||||
|
||||
Use the following command for inference.
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
python3 kie/predict_kie_token_ser_re.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--re_model_dir=../inference/re_vi_layoutxlm_xfund_infer \
|
||||
--ser_model_dir=../inference/ser_vi_layoutxlm_xfund_infer \
|
||||
--use_visual_backbone=False \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx"
|
||||
```
|
||||
|
||||
The visual results and text file will be saved in directory `output`.
|
||||
|
||||
If you want to use a custom ocr model, you can set it through the following fields
|
||||
|
||||
- `--det_model_dir`: the detection inference model path
|
||||
- `--rec_model_dir`: the recognition inference model path
|
||||
|
||||
### 4.3 More
|
||||
|
||||
For training, evaluation and inference tutorial for KIE models, please refer to [KIE doc](../model_train/train_kie.en.md).
|
||||
|
||||
For training, evaluation and inference tutorial for text detection models, please refer to [text detection doc](../../ppocr/model_train/detection.en.md).
|
||||
|
||||
For training, evaluation and inference tutorial for text recognition models, please refer to [text recognition doc](../../ppocr/model_train/recognition.en.md).
|
||||
|
||||
To complete the key information extraction task in your own scenario from data preparation to model selection, please refer to: [Guide to End-to-end KIE](../blog/how_to_do_kie.en.md)。
|
||||
|
||||
## 5. Reference
|
||||
|
||||
- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding, <https://arxiv.org/pdf/2104.08836.pdf>
|
||||
- microsoft/unilm/layoutxlm, <https://github.com/microsoft/unilm/tree/master/layoutxlm>
|
||||
- XFUND dataset, <https://github.com/doc-analysis/XFUND>
|
||||
|
||||
## 6. License
|
||||
|
||||
The content of this project itself is licensed under the [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
|
||||
240
docs/version2.x/ppstructure/model_train/train_kie.md
Normal file
240
docs/version2.x/ppstructure/model_train/train_kie.md
Normal file
@@ -0,0 +1,240 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 关键信息抽取
|
||||
|
||||
## 1. 简介
|
||||
|
||||
关键信息抽取 (Key Information Extraction, KIE)指的是是从文本或者图像中，抽取出关键的信息。针对文档图像的关键信息抽取任务作为OCR的下游任务，存在非常多的实际应用场景，如表单识别、车票信息抽取、身份证信息抽取等。
|
||||
|
||||
PP-Structure 基于 LayoutXLM 文档多模态系列方法进行研究与优化，设计了视觉特征无关的多模态模型结构VI-LayoutXLM，同时引入符合阅读顺序的文本行排序方法以及UDML联合互学习蒸馏方法，最终在精度与速度均超越LayoutXLM。
|
||||
|
||||
PP-Structure中关键信息抽取模块的主要特性如下：
|
||||
|
||||
- 集成[LayoutXLM](https://arxiv.org/pdf/2104.08836.pdf)、VI-LayoutXLM等多模态模型以及PP-OCR预测引擎。
|
||||
- 支持基于多模态方法的语义实体识别 (Semantic Entity Recognition, SER) 以及关系抽取 (Relation Extraction, RE) 任务。基于 SER 任务，可以完成对图像中的文本识别与分类；基于 RE 任务，可以完成对图象中的文本内容的关系提取，如判断问题对(pair)。
|
||||
- 支持SER任务和RE任务的自定义训练。
|
||||
- 支持OCR+SER的端到端系统预测与评估。
|
||||
- 支持OCR+SER+RE的端到端系统预测。
|
||||
- 支持SER模型的动转静导出与基于PaddleInfernece的模型推理。
|
||||
|
||||
## 2. 精度与性能
|
||||
|
||||
我们在 [XFUND](https://github.com/doc-analysis/XFUND) 的中文数据集上对算法进行了评估，SER与RE上的任务性能如下
|
||||
|
||||
| 模型 | 骨干网络 | 任务 | 配置文件 | hmean | 预测耗时(ms) | 下载链接 |
|
||||
| ------------ | ----------------- | ---- | ------- | ---------- | ------ | ------ |
|
||||
| VI-LayoutXLM | VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml) | **93.19%** | 15.49 | [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar) |
|
||||
| LayoutXLM | LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml) | 90.38% | 19.49 | [训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar) |
|
||||
| VI-LayoutXLM | VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml) | **83.92%** | 15.49 | [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar) |
|
||||
| LayoutXLM | LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml) | 74.83% | 19.49 | [训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar) |
|
||||
|
||||
- 注：预测耗时测试条件：V100 GPU + cuda10.2 + cudnn8.1.1 + TensorRT 7.2.3.4，使用FP16进行测试。
|
||||
|
||||
更多关于PaddleOCR中关键信息抽取模型的介绍，请参考[关键信息抽取模型库](../../algorithm/overview.md)。
|
||||
|
||||
## 3. 效果演示
|
||||
|
||||
基于多模态模型的关键信息抽取任务有2种主要的解决方案。
|
||||
|
||||
（1）文本检测 + 文本识别 + 语义实体识别(SER)
|
||||
（2）文本检测 + 文本识别 + 语义实体识别(SER) + 关系抽取(RE)
|
||||
|
||||
下面给出SER与RE任务的示例效果，关于上述解决方案的详细介绍，请参考[关键信息抽取全流程指南](../blog/how_to_do_kie.md)。
|
||||
|
||||
### 3.1 SER
|
||||
|
||||
对于SER任务，效果如下所示。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
**注意：** 测试图片来源于[XFUND数据集](https://github.com/doc-analysis/XFUND)、[发票数据集](https://aistudio.baidu.com/aistudio/datasetdetail/165561)以及合成的身份证数据集。
|
||||
|
||||
图中不同颜色的框表示不同的类别。
|
||||
|
||||
图中的发票以及申请表图像，有`QUESTION`, `ANSWER`, `HEADER` 3种类别，识别的`QUESTION`, `ANSWER`可以用于后续的问题与答案的关系抽取。
|
||||
|
||||
图中的身份证图像，则直接识别出其中的`姓名`、`性别`、`民族`等关键信息，这样就无需后续的关系抽取过程，一个模型即可完成关键信息抽取。
|
||||
|
||||
### 3.2 RE
|
||||
|
||||
对于RE任务，效果如下所示。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
红色框是问题，蓝色框是答案。绿色线条表示连接的两端为一个key-value的pair。
|
||||
|
||||
## 4. 使用
|
||||
|
||||
### 4.1 准备环境
|
||||
|
||||
使用下面的命令安装运行SER与RE关键信息抽取的依赖。
|
||||
|
||||
```bash linenums="1"
|
||||
git clone https://github.com/PaddlePaddle/PaddleOCR.git
|
||||
cd PaddleOCR
|
||||
pip install -r requirements.txt
|
||||
pip install -r ppstructure/kie/requirements.txt
|
||||
# 安装PaddleOCR引擎用于预测
|
||||
pip install "paddleocr<3.0"
|
||||
```
|
||||
|
||||
NOTE: 对于KIE任务需要降低Paddle框架版本（Paddle<2.6），和PaddleNLP版本（PaddleNLP<2.6）。
|
||||
|
||||
### 4.2 快速开始
|
||||
|
||||
下面XFUND数据集，快速体验SER模型与RE模型。
|
||||
|
||||
#### 4.2.1 准备数据
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir train_data
|
||||
cd train_data
|
||||
# 下载与解压数据
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar && tar -xf XFUND.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
#### 4.2.2 基于动态图的预测
|
||||
|
||||
首先下载模型。
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir pretrained_model
|
||||
cd pretrained_model
|
||||
# 下载并解压SER预训练模型
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar && tar -xf ser_vi_layoutxlm_xfund_pretrained.tar
|
||||
|
||||
# 下载并解压RE预训练模型
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar && tar -xf re_vi_layoutxlm_xfund_pretrained.tar
|
||||
```
|
||||
|
||||
如果希望使用OCR引擎，获取端到端的预测结果，可以使用下面的命令进行预测。
|
||||
|
||||
```bash linenums="1"
|
||||
# 仅预测SER模型
|
||||
python3 tools/infer_kie_token_ser.py \
|
||||
-c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy \
|
||||
Global.infer_img=./ppstructure/docs/kie/input/zh_val_42.jpg
|
||||
|
||||
# SER + RE模型串联
|
||||
python3 ./tools/infer_kie_token_ser_re.py \
|
||||
-c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrained_model/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/image/zh_val_42.jpg \
|
||||
-c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o_ser Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
|
||||
```
|
||||
|
||||
`Global.save_res_path`目录中会保存可视化的结果图像以及预测的文本文件。
|
||||
|
||||
如果想使用自定义OCR模型，可通过如下字段进行设置
|
||||
|
||||
- `Global.kie_det_model_dir`: 设置检测inference模型地址
|
||||
- `Global.kie_rec_model_dir`: 设置识别inference模型地址
|
||||
|
||||
如果希望加载标注好的文本检测与识别结果，仅预测可以使用下面的命令进行预测。
|
||||
|
||||
```bash linenums="1"
|
||||
# 仅预测SER模型
|
||||
python3 tools/infer_kie_token_ser.py \
|
||||
-c configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/val.json \
|
||||
Global.infer_mode=False
|
||||
|
||||
# SER + RE模型串联
|
||||
python3 ./tools/infer_kie_token_ser_re.py \
|
||||
-c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
|
||||
-o Architecture.Backbone.checkpoints=./pretrained_model/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
|
||||
Global.infer_img=./train_data/XFUND/zh_val/val.json \
|
||||
Global.infer_mode=False \
|
||||
-c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
|
||||
-o_ser Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
|
||||
```
|
||||
|
||||
#### 4.2.3 基于PaddleInference的预测
|
||||
|
||||
首先下载SER和RE的推理模型。
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir inference
|
||||
cd inference
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar && tar -xf ser_vi_layoutxlm_xfund_infer.tar
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_infer.tar && tar -xf re_vi_layoutxlm_xfund_infer.tar
|
||||
cd ..
|
||||
```
|
||||
|
||||
- SER
|
||||
|
||||
执行下面的命令进行预测。
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
python3 kie/predict_kie_token_ser.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--ser_model_dir=../inference/ser_vi_layoutxlm_xfund_infer \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx"
|
||||
```
|
||||
|
||||
可视化结果保存在`output`目录下。
|
||||
|
||||
- RE
|
||||
|
||||
执行下面的命令进行预测。
|
||||
|
||||
```bash linenums="1"
|
||||
cd ppstructure
|
||||
python3 kie/predict_kie_token_ser_re.py \
|
||||
--kie_algorithm=LayoutXLM \
|
||||
--re_model_dir=../inference/re_vi_layoutxlm_xfund_infer \
|
||||
--ser_model_dir=../inference/ser_vi_layoutxlm_xfund_infer \
|
||||
--use_visual_backbone=False \
|
||||
--image_dir=./docs/kie/input/zh_val_42.jpg \
|
||||
--ser_dict_path=../train_data/XFUND/class_list_xfun.txt \
|
||||
--vis_font_path=../doc/fonts/simfang.ttf \
|
||||
--ocr_order_method="tb-yx"
|
||||
```
|
||||
|
||||
可视化结果保存在`output`目录下。
|
||||
|
||||
如果想使用自定义OCR模型，可通过如下字段进行设置
|
||||
|
||||
- `--det_model_dir`: 设置检测inference模型地址
|
||||
- `--rec_model_dir`: 设置识别inference模型地址
|
||||
|
||||
### 4.3 更多
|
||||
|
||||
关于KIE模型的训练评估与推理，请参考：[关键信息抽取教程](./train_kie.md)。
|
||||
|
||||
关于文本检测模型的训练评估与推理，请参考：[文本检测教程](../../ppocr/model_train/detection.md)。
|
||||
|
||||
关于文本识别模型的训练评估与推理，请参考：[文本识别教程](../../ppocr/model_train/recognition.md)。
|
||||
|
||||
关于怎样在自己的场景中完成关键信息抽取任务，请参考：[关键信息抽取全流程指南](../blog/how_to_do_kie.md)。
|
||||
|
||||
## 5. 参考链接
|
||||
|
||||
- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding, <https://arxiv.org/pdf/2104.08836.pdf>
|
||||
- microsoft/unilm/layoutxlm, <https://github.com/microsoft/unilm/tree/master/layoutxlm>
|
||||
- XFUND dataset, <https://github.com/doc-analysis/XFUND>
|
||||
|
||||
## 6. License
|
||||
|
||||
The content of this project itself is licensed under the [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
|
||||
444
docs/version2.x/ppstructure/model_train/train_layout.en.md
Normal file
444
docs/version2.x/ppstructure/model_train/train_layout.en.md
Normal file
@@ -0,0 +1,444 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Layout analysis
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
Layout analysis refers to the regional division of documents in the form of pictures and the positioning of key areas, such as text, title, table, picture, etc. The layout analysis algorithm is based on the lightweight model PP-picodet of [PaddleDetection]( https://github.com/PaddlePaddle/PaddleDetection ), including English layout analysis, Chinese layout analysis and table layout analysis models. English layout analysis models can detect document layout elements such as text, title, table, figure, list. Chinese layout analysis models can detect document layout elements such as text, figure, figure caption, table, table caption, header, footer, reference, and equation. Table layout analysis models can detect table regions.
|
||||
|
||||
|
||||
|
||||
## 2. Quick start
|
||||
|
||||
PP-Structure currently provides layout analysis models in Chinese, English and table documents. For the model link, see [models_list](../models_list.en.md). The whl package is also provided for quick use, see [quickstart](../quick_start.en.md) for details.
|
||||
|
||||
## 3. Install
|
||||
|
||||
### 3.1. Install PaddlePaddle
|
||||
|
||||
- **（1) Install PaddlePaddle**
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install --upgrade pip
|
||||
|
||||
# GPU Install
|
||||
python3 -m pip install "paddlepaddle-gpu>=2.3" -i https://mirror.baidu.com/pypi/simple
|
||||
|
||||
# CPU Install
|
||||
python3 -m pip install "paddlepaddle>=2.3" -i https://mirror.baidu.com/pypi/simple
|
||||
```
|
||||
|
||||
For more requirements, please refer to the instructions in the [Install file](https://www.paddlepaddle.org.cn/install/quick)。
|
||||
|
||||
### 3.2. Install PaddleDetection
|
||||
|
||||
- **（1）Download PaddleDetection Source code**
|
||||
|
||||
```bash linenums="1"
|
||||
git clone https://github.com/PaddlePaddle/PaddleDetection.git
|
||||
```
|
||||
|
||||
- **（2）Install third-party libraries**
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleDetection
|
||||
python3 -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## 4. Data preparation
|
||||
|
||||
If you want to experience the prediction process directly, you can skip data preparation and download the pre-training model.
|
||||
|
||||
### 4.1. English data set
|
||||
|
||||
Download document analysis data set [PubLayNet](https://developer.ibm.com/exchanges/data/all/publaynet/)（Dataset 96G），contains 5 classes：`{0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"}`
|
||||
|
||||
```
|
||||
# Download data
|
||||
wget https://dax-cdn.cdn.appdomain.cloud/dax-publaynet/1.0.0/publaynet.tar.gz
|
||||
# Decompress data
|
||||
tar -xvf publaynet.tar.gz
|
||||
```
|
||||
|
||||
Uncompressed **directory structure：**
|
||||
|
||||
```
|
||||
|-publaynet
|
||||
|- test
|
||||
|- PMC1277013_00004.jpg
|
||||
|- PMC1291385_00002.jpg
|
||||
| ...
|
||||
|- train.json
|
||||
|- train
|
||||
|- PMC1291385_00002.jpg
|
||||
|- PMC1277013_00004.jpg
|
||||
| ...
|
||||
|- val.json
|
||||
|- val
|
||||
|- PMC538274_00004.jpg
|
||||
|- PMC539300_00004.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
**data distribution：**
|
||||
|
||||
| File or Folder | Description | num |
|
||||
| :------------- | :------------- | ------- |
|
||||
| `train/` | Training set pictures | 335,703 |
|
||||
| `val/` | Verification set pictures | 11,245 |
|
||||
| `test/` | Test set pictures | 11,405 |
|
||||
| `train.json` | Training set annotation files | - |
|
||||
| `val.json` | Validation set dimension files | - |
|
||||
|
||||
**Data Annotation**
|
||||
|
||||
The JSON file contains the annotations of all images, and the data is stored in a dictionary nested manner.Contains the following keys：
|
||||
|
||||
- info，represents the dimension file info。
|
||||
|
||||
- licenses，represents the dimension file licenses。
|
||||
|
||||
- images，represents the list of image information in the annotation file，each element is the information of an image。The information of one of the images is as follows:
|
||||
|
||||
```
|
||||
{
|
||||
'file_name': 'PMC4055390_00006.jpg', # file_name
|
||||
'height': 601, # image height
|
||||
'width': 792, # image width
|
||||
'id': 341427 # image id
|
||||
}
|
||||
```
|
||||
|
||||
- annotations， represents the list of annotation information of the target object in the annotation file，each element is the annotation information of a target object。The following is the annotation information of one of the target objects:
|
||||
|
||||
```
|
||||
{
|
||||
|
||||
'segmentation': # Segmentation annotation of objects
|
||||
'area': 60518.099043117836, # Area of object
|
||||
'iscrowd': 0, # iscrowd
|
||||
'image_id': 341427, # image id
|
||||
'bbox': [50.58, 490.86, 240.15, 252.16], # bbox [x1,y1,w,h]
|
||||
'category_id': 1, # category_id
|
||||
'id': 3322348 # image id
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2. More datasets
|
||||
|
||||
We provide CDLA(Chinese layout analysis), TableBank(Table layout analysis)etc. data set download links，process to the JSON format of the above annotation file，that is, the training can be conducted in the same way。
|
||||
|
||||
| dataset | 简介 |
|
||||
| ------------------------------------------------------------ | ------------------------------------------------------------ |
|
||||
| [cTDaR2019_cTDaR](https://cndplab-founder.github.io/cTDaR2019/) | For form detection (TRACKA) and form identification (TRACKB).Image types include historical data sets (beginning with cTDaR_t0, such as CTDAR_T00872.jpg) and modern data sets (beginning with cTDaR_t1, CTDAR_T10482.jpg). |
|
||||
| [IIIT-AR-13K](http://cvit.iiit.ac.in/usodi/iiitar13k.php) | Data sets constructed by manually annotating figures or pages from publicly available annual reports, containing 5 categories:table, figure, natural image, logo, and signature. |
|
||||
| [TableBank](https://github.com/doc-analysis/TableBank) | For table detection and recognition of large datasets, including Word and Latex document formats |
|
||||
| [CDLA](https://github.com/buptlihang/CDLA) | Chinese document layout analysis data set, for Chinese literature (paper) scenarios, including 10 categories:Text, Title, Figure, Figure caption, Table, Table caption, Header, Footer, Reference, Equation |
|
||||
| [DocBank](https://github.com/doc-analysis/DocBank) | Large-scale dataset (500K document pages) constructed using weakly supervised methods for document layout analysis, containing 12 categories:Author, Caption, Date, Equation, Figure, Footer, List, Paragraph, Reference, Section, Table, Title |
|
||||
|
||||
## 5. Start training
|
||||
|
||||
Training scripts, evaluation scripts, and prediction scripts are provided, and the PubLayNet pre-training model is used as an example in this section.
|
||||
|
||||
If you do not want training and directly experience the following process of model evaluation, prediction, motion to static, and inference, you can download the provided pre-trained model (PubLayNet dataset) and skip this part.
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir pretrained_model
|
||||
cd pretrained_model
|
||||
# Download PubLayNet pre-training model（Direct experience model evaluates, predicts, and turns static）
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout.pdparams
|
||||
# Download the PubLaynet inference model（Direct experience model reasoning）
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
```
|
||||
|
||||
If the test image is Chinese, the pre-trained model of Chinese CDLA dataset can be downloaded to identify 10 types of document regions：Table, Figure, Figure caption, Table, Table caption, Header, Footer, Reference, Equation，Download the training model and inference model of Model 'picodet_lcnet_x1_0_fgd_layout_cdla' in [layout analysis model](../models_list.en.md)。If only the table area in the image is detected, you can download the pre-trained model of the table dataset, and download the training model and inference model of the 'picodet_LCnet_x1_0_FGd_layout_table' model in [Layout Analysis model](../models_list.en.md)
|
||||
|
||||
### 5.1. Train
|
||||
|
||||
Start training with the PaddleDetection [layout analysis profile](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/picodet/legacy_model/application/layout_analysis)
|
||||
|
||||
- Modify Profile
|
||||
|
||||
If you want to train your own data set, you need to modify the data configuration and the number of categories in the configuration file.
|
||||
|
||||
Using 'configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml' as an example, the change is as follows:
|
||||
|
||||
```yaml linenums="1"
|
||||
metric: COCO
|
||||
# Number of categories
|
||||
num_classes: 5
|
||||
|
||||
TrainDataset:
|
||||
!COCODataSet
|
||||
# Modify to your own training data directory
|
||||
image_dir: train
|
||||
# Modify to your own training data label file
|
||||
anno_path: train.json
|
||||
# Modify to your own training data root directory
|
||||
dataset_dir: /root/publaynet/
|
||||
data_fields: ['image', 'gt_bbox', 'gt_class', 'is_crowd']
|
||||
|
||||
EvalDataset:
|
||||
!COCODataSet
|
||||
# Modify to your own validation data directory
|
||||
image_dir: val
|
||||
# Modify to your own validation data label file
|
||||
anno_path: val.json
|
||||
# Modify to your own validation data root
|
||||
dataset_dir: /root/publaynet/
|
||||
|
||||
TestDataset:
|
||||
!ImageFolder
|
||||
# Modify to your own test data label file
|
||||
anno_path: /root/publaynet/val.json
|
||||
```
|
||||
|
||||
- Start training. During training, PP picodet pre training model will be downloaded by default. There is no need to download in advance.
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU training supports single-card and multi-card training
|
||||
# The training log is automatically saved to the log directory
|
||||
|
||||
# Single card training
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
python3 tools/train.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--eval
|
||||
|
||||
# Multi-card training, with the -- GPUS parameter specifying the card number
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--eval
|
||||
```
|
||||
|
||||
**Attention：**If the video memory is out during training, adjust Batch_size in TrainReader and base_LR in LearningRate. The published config is obtained by 8-card training. If the number of GPU cards is changed to 1, then the base_LR needs to be reduced by 8 times.
|
||||
|
||||
After starting training normally, you will see the following log output:
|
||||
|
||||
```
|
||||
[08/15 04:02:30] ppdet.utils.checkpoint INFO: Finish loading model weights: /root/.cache/paddle/weights/LCNet_x1_0_pretrained.pdparams
|
||||
[08/15 04:02:46] ppdet.engine INFO: Epoch: [0] [ 0/1929] learning_rate: 0.040000 loss_vfl: 1.216707 loss_bbox: 1.142163 loss_dfl: 0.544196 loss: 2.903065 eta: 17 days, 13:50:26 batch_cost: 15.7452 data_cost: 2.9112 ips: 1.5243 images/s
|
||||
[08/15 04:03:19] ppdet.engine INFO: Epoch: [0] [ 20/1929] learning_rate: 0.064000 loss_vfl: 1.180627 loss_bbox: 0.939552 loss_dfl: 0.442436 loss: 2.628206 eta: 2 days, 12:18:53 batch_cost: 1.5770 data_cost: 0.0008 ips: 15.2184 images/s
|
||||
[08/15 04:03:47] ppdet.engine INFO: Epoch: [0] [ 40/1929] learning_rate: 0.088000 loss_vfl: 0.543321 loss_bbox: 1.071401 loss_dfl: 0.457817 loss: 2.057003 eta: 2 days, 0:07:03 batch_cost: 1.3190 data_cost: 0.0007 ips: 18.1954 images/s
|
||||
[08/15 04:04:12] ppdet.engine INFO: Epoch: [0] [ 60/1929] learning_rate: 0.112000 loss_vfl: 0.630989 loss_bbox: 0.859183 loss_dfl: 0.384702 loss: 1.883143 eta: 1 day, 19:01:29 batch_cost: 1.2177 data_cost: 0.0006 ips: 19.7087 images/s
|
||||
```
|
||||
|
||||
- `--eval` indicates that the best model is saved as `output/picodet_lcnet_x1_0_layout/best_accuracy` by default during the evaluation process 。
|
||||
|
||||
**Note that the configuration file for prediction / evaluation must be consistent with the training.**
|
||||
|
||||
### 5.2. FGD Distillation Training
|
||||
|
||||
PaddleDetection supports FGD-based [Focal and Global Knowledge Distillation for Detectors]( https://arxiv.org/abs/2111.11837v1) The training process of the target detection model of distillation, FGD distillation is divided into two parts `Focal` and `Global`. `Focal` Distillation separates the foreground and background of the image, allowing the student model to focus on the key pixels of the foreground and background features of the teacher model respectively;`Global`Distillation section reconstructs the relationships between different pixels and transfers them from the teacher to the student to compensate for the global information lost in `Focal`Distillation.
|
||||
|
||||
Change the dataset and modify the data configuration and number of categories in the [TODO] configuration, referring to 4.1. Start training:
|
||||
|
||||
```bash linenums="1"
|
||||
# Single Card Training
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
python3 tools/train.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
|
||||
--eval
|
||||
```
|
||||
|
||||
- `-c`: Specify the model configuration file.
|
||||
- `--slim_config`: Specify the compression policy profile.
|
||||
|
||||
## 6. Model evaluation and prediction
|
||||
|
||||
### 6.1. Indicator evaluation
|
||||
|
||||
Model parameters in training are saved by default in `output/picodet_ Lcnet_ X1_ 0_ Under the layout` directory. When evaluating indicators, you need to set `weights` to point to the saved parameter file.Assessment datasets can be accessed via `configs/picodet/legacy_ Model/application/layout_ Analysis/picodet_ Lcnet_ X1_ 0_ Layout. Yml` . Modify `EvalDataset` : `img_dir`,`anno_ Path`and`dataset_dir` setting.
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU evaluation, weights as weights to be measured
|
||||
python3 tools/eval.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
-o weights=./output/picodet_lcnet_x1_0_layout/best_model
|
||||
```
|
||||
|
||||
The following information will be printed out, such as mAP, AP0.5, etc.
|
||||
|
||||
```python linenums="1"
|
||||
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.935
|
||||
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.979
|
||||
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.956
|
||||
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.404
|
||||
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.782
|
||||
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.969
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.539
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.938
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.949
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.495
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.818
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.978
|
||||
[08/15 07:07:09] ppdet.engine INFO: Total sample number: 11245, averge FPS: 24.405059207157436
|
||||
[08/15 07:07:09] ppdet.engine INFO: Best test bbox ap is 0.935.
|
||||
```
|
||||
|
||||
If you use the provided pre-training model for evaluation or the FGD distillation training model, replace the `weights` model path and execute the following command for evaluation:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/eval.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
|
||||
-o weights=output/picodet_lcnet_x2_5_layout/best_model
|
||||
```
|
||||
|
||||
- `-c`: Specify the model configuration file.
|
||||
- `--slim_config`: Specify the distillation policy profile.
|
||||
- `-o weights`: Specify the model path trained by the distillation algorithm.
|
||||
|
||||
### 6.2. Test Layout Analysis Results
|
||||
|
||||
The profile predicted to be used must be consistent with the training, for example, if you pass `python3 tools/train'. Py-c configs/picodet/legacy_ Model/application/layout_ Analysis/picodet_ Lcnet_ X1_ 0_ Layout. Yml` completed the training process for the model.
|
||||
|
||||
With trained PaddleDetection model, you can use the following commands to make model predictions.
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
-o weights='output/picodet_lcnet_x1_0_layout/best_model.pdparams' \
|
||||
--infer_img='docs/images/layout.jpg' \
|
||||
--output_dir=output_dir/ \
|
||||
--draw_threshold=0.5
|
||||
```
|
||||
|
||||
- `--infer_img`: Reasoning for a single picture can also be done via `--infer_ Dir`Inform all pictures in the file.
|
||||
- `--output_dir`: Specify the path to save the visualization results.
|
||||
- `--draw_threshold`:Specify the NMS threshold for drawing the result box.
|
||||
|
||||
If you use the provided pre-training model for prediction or the FGD distillation training model, change the `weights` model path and execute the following command to make the prediction:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
|
||||
-o weights='output/picodet_lcnet_x2_5_layout/best_model.pdparams' \
|
||||
--infer_img='docs/images/layout.jpg' \
|
||||
--output_dir=output_dir/ \
|
||||
--draw_threshold=0.5
|
||||
```
|
||||
|
||||
## 7. Model Export and Inference
|
||||
|
||||
### 7.1 Model Export
|
||||
|
||||
The inference model (the model saved by `paddle.jit.save`) is generally a solidified model saved after the model training is completed, and is mostly used to give prediction in deployment.
|
||||
|
||||
The model saved during the training process is the checkpoints model, which saves the parameters of the model and is mostly used to resume training.
|
||||
|
||||
Compared with the checkpoints model, the inference model will additionally save the structural information of the model. Therefore, it is easier to deploy because the model structure and model parameters are already solidified in the inference model file, and is suitable for integration with actual systems.
|
||||
|
||||
Layout analysis model to inference model steps are as follows：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/export_model.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
-o weights=output/picodet_lcnet_x1_0_layout/best_model \
|
||||
--output_dir=output_inference/
|
||||
```
|
||||
|
||||
- If no post-export processing is required, specify：`-o export.benchmark=True`（If -o already exists, delete -o here）
|
||||
- If you do not need to export NMS, specify：`-o export.nms=False`
|
||||
|
||||
After successful conversion, there are three files in the directory:
|
||||
|
||||
```
|
||||
output_inference/picodet_lcnet_x1_0_layout/
|
||||
├── model.pdiparams # inference Parameter file for model
|
||||
├── model.pdiparams.info # inference Model parameter information, ignorable
|
||||
└── model.pdmodel # inference Model Structure File for Model
|
||||
```
|
||||
|
||||
If you change the `weights` model path using the provided pre-training model to the Inference model, or using the FGD distillation training model, the model to inference model steps are as follows:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/export_model.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
|
||||
-o weights=./output/picodet_lcnet_x2_5_layout/best_model \
|
||||
--output_dir=output_inference/
|
||||
```
|
||||
|
||||
### 7.2 Model inference
|
||||
|
||||
Replace model_with the provided inference training model for inference or the FGD distillation training `model_dir`Inference model path, execute the following commands for inference:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 deploy/python/infer.py \
|
||||
--model_dir=output_inference/picodet_lcnet_x1_0_layout/ \
|
||||
--image_file=docs/images/layout.jpg \
|
||||
--device=CPU
|
||||
```
|
||||
|
||||
- --device：Specify the GPU or CPU device
|
||||
|
||||
When model inference is complete, you will see the following log output:
|
||||
|
||||
```
|
||||
------------------------------------------
|
||||
----------- Model Configuration -----------
|
||||
Model Arch: PicoDet
|
||||
Transform Order:
|
||||
--transform op: Resize
|
||||
--transform op: NormalizeImage
|
||||
--transform op: Permute
|
||||
--transform op: PadStride
|
||||
--------------------------------------------
|
||||
class_id:0, confidence:0.9921, left_top:[20.18,35.66],right_bottom:[341.58,600.99]
|
||||
class_id:0, confidence:0.9914, left_top:[19.77,611.42],right_bottom:[341.48,901.82]
|
||||
class_id:0, confidence:0.9904, left_top:[369.36,375.10],right_bottom:[691.29,600.59]
|
||||
class_id:0, confidence:0.9835, left_top:[369.60,608.60],right_bottom:[691.38,736.72]
|
||||
class_id:0, confidence:0.9830, left_top:[369.58,805.38],right_bottom:[690.97,901.80]
|
||||
class_id:0, confidence:0.9716, left_top:[383.68,271.44],right_bottom:[688.93,335.39]
|
||||
class_id:0, confidence:0.9452, left_top:[370.82,34.48],right_bottom:[688.10,63.54]
|
||||
class_id:1, confidence:0.8712, left_top:[370.84,771.03],right_bottom:[519.30,789.13]
|
||||
class_id:3, confidence:0.9856, left_top:[371.28,67.85],right_bottom:[685.73,267.72]
|
||||
save result to: output/layout.jpg
|
||||
Test iter 0
|
||||
------------------ Inference Time Info ----------------------
|
||||
total_time(ms): 2196.0, img_num: 1
|
||||
average latency time(ms): 2196.00, QPS: 0.455373
|
||||
preprocess_time(ms): 2172.50, inference_time(ms): 11.90, postprocess_time(ms): 11.60
|
||||
```
|
||||
|
||||
- Model：model structure
|
||||
- Transform Order：Preprocessing operation
|
||||
- class_id, confidence, left_top, right_bottom：Indicates category id, confidence level, upper left coordinate, lower right coordinate, respectively
|
||||
- save result to：Save path of visual layout analysis results, default save to ./output folder
|
||||
- inference time info：Inference time, where preprocess_time represents the preprocessing time, Inference_time represents the model prediction time, and postprocess_time represents the post-processing time
|
||||
|
||||
The result of visualization layout is shown in the following figure
|
||||
|
||||
|
||||
|
||||
## Citations
|
||||
|
||||
```bibtex
|
||||
@inproceedings{zhong2019publaynet,
|
||||
title={PubLayNet: largest dataset ever for document layout analysis},
|
||||
author={Zhong, Xu and Tang, Jianbin and Yepes, Antonio Jimeno},
|
||||
booktitle={2019 International Conference on Document Analysis and Recognition (ICDAR)},
|
||||
year={2019},
|
||||
volume={},
|
||||
number={},
|
||||
pages={1015-1022},
|
||||
doi={10.1109/ICDAR.2019.00166},
|
||||
ISSN={1520-5363},
|
||||
month={Sep.},
|
||||
organization={IEEE}
|
||||
}
|
||||
|
||||
@inproceedings{yang2022focal,
|
||||
title={Focal and global knowledge distillation for detectors},
|
||||
author={Yang, Zhendong and Li, Zhe and Jiang, Xiaohu and Gong, Yuan and Yuan, Zehuan and Zhao, Danpei and Yuan, Chun},
|
||||
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
|
||||
pages={4643--4652},
|
||||
year={2022}
|
||||
}
|
||||
```
|
||||
441
docs/version2.x/ppstructure/model_train/train_layout.md
Normal file
441
docs/version2.x/ppstructure/model_train/train_layout.md
Normal file
@@ -0,0 +1,441 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 版面分析
|
||||
|
||||
## 1. 简介
|
||||
|
||||
版面分析指的是对图片形式的文档进行区域划分，定位其中的关键区域，如文字、标题、表格、图片等。版面分析算法基于[PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection)的轻量模型PP-PicoDet进行开发，包含英文、中文、表格版面分析3类模型。其中，英文模型支持Text、Title、Tale、Figure、List5类区域的检测，中文模型支持Text、Title、Figure、Figure caption、Table、Table caption、Header、Footer、Reference、Equation10类区域的检测，表格版面分析支持Table区域的检测，版面分析效果如下图所示：
|
||||
|
||||
|
||||
|
||||
## 2. 快速开始
|
||||
|
||||
PP-Structure目前提供了中文、英文、表格三类文档版面分析模型，模型链接见 [models_list](../models_list.md)。也提供了whl包的形式方便快速使用，详见 [quickstart](../quick_start.md)。
|
||||
|
||||
## 3. 安装
|
||||
|
||||
### 3.1. 安装PaddlePaddle
|
||||
|
||||
- **（1) 安装PaddlePaddle**
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install --upgrade pip
|
||||
|
||||
# GPU安装
|
||||
python3 -m pip install "paddlepaddle-gpu>=2.3" -i https://mirror.baidu.com/pypi/simple
|
||||
|
||||
# CPU安装
|
||||
python3 -m pip install "paddlepaddle>=2.3" -i https://mirror.baidu.com/pypi/simple
|
||||
```
|
||||
|
||||
更多需求，请参照[安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。
|
||||
|
||||
### 3.2. 安装PaddleDetection
|
||||
|
||||
- **（1）下载PaddleDetection源码**
|
||||
|
||||
```bash linenums="1"
|
||||
git clone https://github.com/PaddlePaddle/PaddleDetection.git
|
||||
```
|
||||
|
||||
- **（2）安装其他依赖**
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleDetection
|
||||
python3 -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## 4. 数据准备
|
||||
|
||||
如果希望直接体验预测过程，可以跳过数据准备，下载我们提供的预训练模型。
|
||||
|
||||
### 4.1. 英文数据集
|
||||
|
||||
下载文档分析数据集[PubLayNet](https://developer.ibm.com/exchanges/data/all/publaynet/)（数据集96G），包含5个类：`{0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"}`
|
||||
|
||||
```
|
||||
# 下载数据
|
||||
wget https://dax-cdn.cdn.appdomain.cloud/dax-publaynet/1.0.0/publaynet.tar.gz
|
||||
# 解压数据
|
||||
tar -xvf publaynet.tar.gz
|
||||
```
|
||||
|
||||
解压之后的**目录结构：**
|
||||
|
||||
```
|
||||
|-publaynet
|
||||
|- test
|
||||
|- PMC1277013_00004.jpg
|
||||
|- PMC1291385_00002.jpg
|
||||
| ...
|
||||
|- train.json
|
||||
|- train
|
||||
|- PMC1291385_00002.jpg
|
||||
|- PMC1277013_00004.jpg
|
||||
| ...
|
||||
|- val.json
|
||||
|- val
|
||||
|- PMC538274_00004.jpg
|
||||
|- PMC539300_00004.jpg
|
||||
| ...
|
||||
```
|
||||
|
||||
**数据分布：**
|
||||
|
||||
| File or Folder | Description | num |
|
||||
| :------------- | :------------- | ------- |
|
||||
| `train/` | 训练集图片 | 335,703 |
|
||||
| `val/` | 验证集图片 | 11,245 |
|
||||
| `test/` | 测试集图片 | 11,405 |
|
||||
| `train.json` | 训练集标注文件 | - |
|
||||
| `val.json` | 验证集标注文件 | - |
|
||||
|
||||
**标注格式：**
|
||||
|
||||
json文件包含所有图像的标注，数据以字典嵌套的方式存放，包含以下key：
|
||||
|
||||
- info，表示标注文件info。
|
||||
|
||||
- licenses，表示标注文件licenses。
|
||||
|
||||
- images，表示标注文件中图像信息列表，每个元素是一张图像的信息。如下为其中一张图像的信息：
|
||||
|
||||
```
|
||||
{
|
||||
'file_name': 'PMC4055390_00006.jpg', # file_name
|
||||
'height': 601, # image height
|
||||
'width': 792, # image width
|
||||
'id': 341427 # image id
|
||||
}
|
||||
```
|
||||
|
||||
- annotations，表示标注文件中目标物体的标注信息列表，每个元素是一个目标物体的标注信息。如下为其中一个目标物体的标注信息：
|
||||
|
||||
```
|
||||
{
|
||||
|
||||
'segmentation': # 物体的分割标注
|
||||
'area': 60518.099043117836, # 物体的区域面积
|
||||
'iscrowd': 0, # iscrowd
|
||||
'image_id': 341427, # image id
|
||||
'bbox': [50.58, 490.86, 240.15, 252.16], # bbox [x1,y1,w,h]
|
||||
'category_id': 1, # category_id
|
||||
'id': 3322348 # image id
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2. 更多数据集
|
||||
|
||||
我们提供了CDLA(中文版面分析)、TableBank(表格版面分析)等数据集的下连接，处理为上述标注文件json格式，即可以按相同方式进行训练。
|
||||
|
||||
| dataset | 简介 |
|
||||
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| [cTDaR2019_cTDaR](https://cndplab-founder.github.io/cTDaR2019/) | 用于表格检测(TRACKA)和表格识别(TRACKB)。图片类型包含历史数据集(以cTDaR_t0开头，如cTDaR_t00872.jpg)和现代数据集(以cTDaR_t1开头，cTDaR_t10482.jpg)。 |
|
||||
| [IIIT-AR-13K](http://cvit.iiit.ac.in/usodi/iiitar13k.php) | 手动注释公开的年度报告中的图形或页面而构建的数据集，包含5类：table, figure, natural image, logo, and signature |
|
||||
| [CDLA](https://github.com/buptlihang/CDLA) | 中文文档版面分析数据集，面向中文文献类（论文）场景，包含10类：Text、Title、Figure、Figure caption、Table、Table caption、Header、Footer、Reference、Equation |
|
||||
| [TableBank](https://github.com/doc-analysis/TableBank) | 用于表格检测和识别大型数据集，包含Word和Latex2种文档格式 |
|
||||
| [DocBank](https://github.com/doc-analysis/DocBank) | 使用弱监督方法构建的大规模数据集(500K文档页面)，用于文档布局分析，包含12类：Author、Caption、Date、Equation、Figure、Footer、List、Paragraph、Reference、Section、Table、Title |
|
||||
|
||||
## 5. 开始训练
|
||||
|
||||
提供了训练脚本、评估脚本和预测脚本，本节将以PubLayNet预训练模型为例进行讲解。
|
||||
|
||||
如果不希望训练，直接体验后面的模型评估、预测、动转静、推理的流程，可以下载提供的预训练模型(PubLayNet数据集)，并跳过5.1和5.2。
|
||||
|
||||
```bash linenums="1"
|
||||
mkdir pretrained_model
|
||||
cd pretrained_model
|
||||
# 下载PubLayNet预训练模型（直接体验模型评估、预测、动转静）
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout.pdparams
|
||||
# 下载PubLaynet推理模型（直接体验模型推理）
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar
|
||||
```
|
||||
|
||||
如果测试图片为中文，可以下载中文CDLA数据集的预训练模型，识别10类文档区域：Table、Figure、Figure caption、Table、Table caption、Header、Footer、Reference、Equation，在[版面分析模型](../models_list.md)中下载`picodet_lcnet_x1_0_fgd_layout_cdla`模型的训练模型和推理模型。如果只检测图片中的表格区域，可以下载表格数据集的预训练模型，在[版面分析模型](../models_list.md)中下载`picodet_lcnet_x1_0_fgd_layout_table`模型的训练模型和推理模型。
|
||||
|
||||
### 5.1. 启动训练
|
||||
|
||||
使用PaddleDetection[版面分析配置文件](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/picodet/legacy_model/application/layout_analysis)启动训练
|
||||
|
||||
- 修改配置文件
|
||||
|
||||
如果你希望训练自己的数据集，需要修改配置文件中的数据配置、类别数。
|
||||
|
||||
以`configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml` 为例，修改的内容如下所示。
|
||||
|
||||
```yaml linenums="1"
|
||||
metric: COCO
|
||||
# 类别数
|
||||
num_classes: 5
|
||||
|
||||
TrainDataset:
|
||||
!COCODataSet
|
||||
# 修改为你自己的训练数据目录
|
||||
image_dir: train
|
||||
# 修改为你自己的训练数据标签文件
|
||||
anno_path: train.json
|
||||
# 修改为你自己的训练数据根目录
|
||||
dataset_dir: /root/publaynet/
|
||||
data_fields: ['image', 'gt_bbox', 'gt_class', 'is_crowd']
|
||||
|
||||
EvalDataset:
|
||||
!COCODataSet
|
||||
# 修改为你自己的验证数据目录
|
||||
image_dir: val
|
||||
# 修改为你自己的验证数据标签文件
|
||||
anno_path: val.json
|
||||
# 修改为你自己的验证数据根目录
|
||||
dataset_dir: /root/publaynet/
|
||||
|
||||
TestDataset:
|
||||
!ImageFolder
|
||||
# 修改为你自己的测试数据标签文件
|
||||
anno_path: /root/publaynet/val.json
|
||||
```
|
||||
|
||||
- 开始训练，在训练时，会默认下载PP-PicoDet预训练模型，这里无需预先下载。
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU训练 支持单卡，多卡训练
|
||||
# 训练日志会自动保存到 log 目录中
|
||||
|
||||
# 单卡训练
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
python3 tools/train.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--eval
|
||||
|
||||
# 多卡训练，通过--gpus参数指定卡号
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--eval
|
||||
```
|
||||
|
||||
**注意：**如果训练时显存out memory，将TrainReader中batch_size调小，同时LearningRate中base_lr等比例减小。发布的config均由8卡训练得到，如果改变GPU卡数为1，那么base_lr需要减小8倍。
|
||||
|
||||
正常启动训练后，会看到以下log输出：
|
||||
|
||||
```bash linenums="1"
|
||||
[08/15 04:02:30] ppdet.utils.checkpoint INFO: Finish loading model weights: /root/.cache/paddle/weights/LCNet_x1_0_pretrained.pdparams
|
||||
[08/15 04:02:46] ppdet.engine INFO: Epoch: [0] [ 0/1929] learning_rate: 0.040000 loss_vfl: 1.216707 loss_bbox: 1.142163 loss_dfl: 0.544196 loss: 2.903065 eta: 17 days, 13:50:26 batch_cost: 15.7452 data_cost: 2.9112 ips: 1.5243 images/s
|
||||
[08/15 04:03:19] ppdet.engine INFO: Epoch: [0] [ 20/1929] learning_rate: 0.064000 loss_vfl: 1.180627 loss_bbox: 0.939552 loss_dfl: 0.442436 loss: 2.628206 eta: 2 days, 12:18:53 batch_cost: 1.5770 data_cost: 0.0008 ips: 15.2184 images/s
|
||||
[08/15 04:03:47] ppdet.engine INFO: Epoch: [0] [ 40/1929] learning_rate: 0.088000 loss_vfl: 0.543321 loss_bbox: 1.071401 loss_dfl: 0.457817 loss: 2.057003 eta: 2 days, 0:07:03 batch_cost: 1.3190 data_cost: 0.0007 ips: 18.1954 images/s
|
||||
[08/15 04:04:12] ppdet.engine INFO: Epoch: [0] [ 60/1929] learning_rate: 0.112000 loss_vfl: 0.630989 loss_bbox: 0.859183 loss_dfl: 0.384702 loss: 1.883143 eta: 1 day, 19:01:29 batch_cost: 1.2177 data_cost: 0.0006 ips: 19.7087 images/s
|
||||
```
|
||||
|
||||
- `--eval`表示训练的同时，进行评估， 评估过程中默认将最佳模型，保存为 `output/picodet_lcnet_x1_0_layout/best_accuracy` 。
|
||||
|
||||
**注意，预测/评估时的配置文件请务必与训练一致。**
|
||||
|
||||
### 5.2. FGD蒸馏训练
|
||||
|
||||
PaddleDetection支持了基于FGD([Focal and Global Knowledge Distillation for Detectors](https://arxiv.org/abs/2111.11837v1))蒸馏的目标检测模型训练过程，FGD蒸馏分为两个部分`Focal`和`Global`。`Focal`蒸馏分离图像的前景和背景，让学生模型分别关注教师模型的前景和背景部分特征的关键像素；`Global`蒸馏部分重建不同像素之间的关系并将其从教师转移到学生，以补偿`Focal`蒸馏中丢失的全局信息。
|
||||
|
||||
更换数据集，修改【TODO】配置中的数据配置、类别数，具体可以参考4.1。启动训练：
|
||||
|
||||
```bash linenums="1"
|
||||
# 单卡训练
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
python3 tools/train.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
|
||||
--eval
|
||||
```
|
||||
|
||||
- `-c`: 指定模型配置文件。
|
||||
- `--slim_config`: 指定压缩策略配置文件。
|
||||
|
||||
## 6. 模型评估与预测
|
||||
|
||||
### 6.1. 指标评估
|
||||
|
||||
训练中模型参数默认保存在`output/picodet_lcnet_x1_0_layout`目录下。在评估指标时，需要设置`weights`指向保存的参数文件。评估数据集可以通过 `configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml` 修改`EvalDataset`中的 `image_dir`、`anno_path`和`dataset_dir` 设置。
|
||||
|
||||
```bash linenums="1"
|
||||
# GPU 评估， weights 为待测权重
|
||||
python3 tools/eval.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
-o weights=./output/picodet_lcnet_x1_0_layout/best_model
|
||||
```
|
||||
|
||||
会输出以下信息，打印出mAP、AP0.5等信息。
|
||||
|
||||
```python linenums="1"
|
||||
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.935
|
||||
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.979
|
||||
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.956
|
||||
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.404
|
||||
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.782
|
||||
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.969
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.539
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.938
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.949
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.495
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.818
|
||||
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.978
|
||||
[08/15 07:07:09] ppdet.engine INFO: Total sample number: 11245, averge FPS: 24.405059207157436
|
||||
[08/15 07:07:09] ppdet.engine INFO: Best test bbox ap is 0.935.
|
||||
```
|
||||
|
||||
若使用**提供的预训练模型进行评估**，或使用**FGD蒸馏训练的模型**，更换`weights`模型路径，执行如下命令进行评估：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/eval.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
|
||||
-o weights=output/picodet_lcnet_x2_5_layout/best_model
|
||||
```
|
||||
|
||||
- `-c`: 指定模型配置文件。
|
||||
- `--slim_config`: 指定蒸馏策略配置文件。
|
||||
- `-o weights`: 指定蒸馏算法训好的模型路径。
|
||||
|
||||
### 6.2 测试版面分析结果
|
||||
|
||||
预测使用的配置文件必须与训练一致，如您通过 `python3 tools/train.py -c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml` 完成了模型的训练过程。
|
||||
|
||||
使用 PaddleDetection 训练好的模型，您可以使用如下命令进行模型预测。
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
-o weights='output/picodet_lcnet_x1_0_layout/best_model.pdparams' \
|
||||
--infer_img='docs/images/layout.jpg' \
|
||||
--output_dir=output_dir/ \
|
||||
--draw_threshold=0.5
|
||||
```
|
||||
|
||||
- `--infer_img`: 推理单张图片，也可以通过`--infer_dir`推理文件中的所有图片。
|
||||
- `--output_dir`: 指定可视化结果保存路径。
|
||||
- `--draw_threshold`:指定绘制结果框的NMS阈值。
|
||||
|
||||
若使用**提供的预训练模型进行预测**，或使用**FGD蒸馏训练的模型**，更换`weights`模型路径，执行如下命令进行预测：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/infer.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
|
||||
-o weights='output/picodet_lcnet_x2_5_layout/best_model.pdparams' \
|
||||
--infer_img='docs/images/layout.jpg' \
|
||||
--output_dir=output_dir/ \
|
||||
--draw_threshold=0.5
|
||||
```
|
||||
|
||||
## 7. 模型导出与预测
|
||||
|
||||
### 7.1 模型导出
|
||||
|
||||
inference 模型（`paddle.jit.save`保存的模型） 一般是模型训练，把模型结构和模型参数保存在文件中的固化模型，多用于预测部署场景。 训练过程中保存的模型是checkpoints模型，保存的只有模型的参数，多用于恢复训练等。 与checkpoints模型相比，inference 模型会额外保存模型的结构信息，在预测部署、加速推理上性能优越，灵活方便，适合于实际系统集成。
|
||||
|
||||
版面分析模型转inference模型步骤如下：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/export_model.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
-o weights=output/picodet_lcnet_x1_0_layout/best_model \
|
||||
--output_dir=output_inference/
|
||||
```
|
||||
|
||||
- 如无需导出后处理，请指定：`-o export.benchmark=True`（如果-o已出现过，此处删掉-o）
|
||||
- 如无需导出NMS，请指定：`-o export.nms=False`
|
||||
|
||||
转换成功后，在目录下有三个文件：
|
||||
|
||||
```
|
||||
output_inference/picodet_lcnet_x1_0_layout/
|
||||
├── model.pdiparams # inference模型的参数文件
|
||||
├── model.pdiparams.info # inference模型的参数信息，可忽略
|
||||
└── model.pdmodel # inference模型的模型结构文件
|
||||
```
|
||||
|
||||
若使用**提供的预训练模型转Inference模型**，或使用**FGD蒸馏训练的模型**，更换`weights`模型路径，模型转inference模型步骤如下：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 tools/export_model.py \
|
||||
-c configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x1_0_layout.yml \
|
||||
--slim_config configs/picodet/legacy_model/application/layout_analysis/picodet_lcnet_x2_5_layout.yml \
|
||||
-o weights=./output/picodet_lcnet_x2_5_layout/best_model \
|
||||
--output_dir=output_inference/
|
||||
```
|
||||
|
||||
### 7.2 模型推理
|
||||
|
||||
若使用**提供的推理训练模型推理**，或使用**FGD蒸馏训练的模型**，更换`model_dir`推理模型路径，执行如下命令进行推理：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 deploy/python/infer.py \
|
||||
--model_dir=output_inference/picodet_lcnet_x1_0_layout/ \
|
||||
--image_file=docs/images/layout.jpg \
|
||||
--device=CPU
|
||||
```
|
||||
|
||||
- --device：指定GPU、CPU设备
|
||||
|
||||
模型推理完成，会看到以下log输出
|
||||
|
||||
```
|
||||
------------------------------------------
|
||||
----------- Model Configuration -----------
|
||||
Model Arch: PicoDet
|
||||
Transform Order:
|
||||
--transform op: Resize
|
||||
--transform op: NormalizeImage
|
||||
--transform op: Permute
|
||||
--transform op: PadStride
|
||||
--------------------------------------------
|
||||
class_id:0, confidence:0.9921, left_top:[20.18,35.66],right_bottom:[341.58,600.99]
|
||||
class_id:0, confidence:0.9914, left_top:[19.77,611.42],right_bottom:[341.48,901.82]
|
||||
class_id:0, confidence:0.9904, left_top:[369.36,375.10],right_bottom:[691.29,600.59]
|
||||
class_id:0, confidence:0.9835, left_top:[369.60,608.60],right_bottom:[691.38,736.72]
|
||||
class_id:0, confidence:0.9830, left_top:[369.58,805.38],right_bottom:[690.97,901.80]
|
||||
class_id:0, confidence:0.9716, left_top:[383.68,271.44],right_bottom:[688.93,335.39]
|
||||
class_id:0, confidence:0.9452, left_top:[370.82,34.48],right_bottom:[688.10,63.54]
|
||||
class_id:1, confidence:0.8712, left_top:[370.84,771.03],right_bottom:[519.30,789.13]
|
||||
class_id:3, confidence:0.9856, left_top:[371.28,67.85],right_bottom:[685.73,267.72]
|
||||
save result to: output/layout.jpg
|
||||
Test iter 0
|
||||
------------------ Inference Time Info ----------------------
|
||||
total_time(ms): 2196.0, img_num: 1
|
||||
average latency time(ms): 2196.00, QPS: 0.455373
|
||||
preprocess_time(ms): 2172.50, inference_time(ms): 11.90, postprocess_time(ms): 11.60
|
||||
```
|
||||
|
||||
- Model：模型结构
|
||||
- Transform Order：预处理操作
|
||||
- class_id、confidence、left_top、right_bottom：分别表示类别id、置信度、左上角坐标、右下角坐标
|
||||
- save result to：可视化版面分析结果保存路径，默认保存到`./output`文件夹
|
||||
- Inference Time Info：推理时间，其中preprocess_time表示预处理耗时，inference_time表示模型预测耗时，postprocess_time表示后处理耗时
|
||||
|
||||
可视化版面结果如下图所示
|
||||
|
||||
|
||||
|
||||
## Citations
|
||||
|
||||
```bibtex
|
||||
@inproceedings{zhong2019publaynet,
|
||||
title={PubLayNet: largest dataset ever for document layout analysis},
|
||||
author={Zhong, Xu and Tang, Jianbin and Yepes, Antonio Jimeno},
|
||||
booktitle={2019 International Conference on Document Analysis and Recognition (ICDAR)},
|
||||
year={2019},
|
||||
volume={},
|
||||
number={},
|
||||
pages={1015-1022},
|
||||
doi={10.1109/ICDAR.2019.00166},
|
||||
ISSN={1520-5363},
|
||||
month={Sep.},
|
||||
organization={IEEE}
|
||||
}
|
||||
|
||||
@inproceedings{yang2022focal,
|
||||
title={Focal and global knowledge distillation for detectors},
|
||||
author={Yang, Zhendong and Li, Zhe and Jiang, Xiaohu and Gong, Yuan and Yuan, Zehuan and Zhao, Danpei and Yuan, Chun},
|
||||
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
|
||||
pages={4643--4652},
|
||||
year={2022}
|
||||
}
|
||||
```
|
||||
164
docs/version2.x/ppstructure/model_train/train_table.en.md
Normal file
164
docs/version2.x/ppstructure/model_train/train_table.en.md
Normal file
@@ -0,0 +1,164 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Table Recognition
|
||||
|
||||
## 1. pipeline
|
||||
|
||||
The table recognition mainly contains three models
|
||||
|
||||
1. Single line text detection-DB
|
||||
2. Single line text recognition-CRNN
|
||||
3. Table structure and cell coordinate prediction-SLANet
|
||||
|
||||
The table recognition flow chart is as follows
|
||||
|
||||
|
||||
|
||||
1. The coordinates of single-line text is detected by DB model, and then sends it to the recognition model to get the recognition result.
|
||||
2. The table structure and cell coordinates is predicted by SLANet model.
|
||||
3. The recognition result of the cell is combined by the coordinates, recognition result of the single line and the coordinates of the cell.
|
||||
4. The cell recognition result and the table structure together construct the html string of the table.
|
||||
|
||||
## 2. Performance
|
||||
|
||||
We evaluated the algorithm on the PubTabNet<sup>[1]</sup> eval dataset, and the performance is as follows:
|
||||
|
||||
|Method|Acc|[TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src)|Speed|
|
||||
| --- | --- | --- | ---|
|
||||
| EDD<sup>[2]</sup> |x| 88.30% |x|
|
||||
| TableRec-RARE(ours) | 71.73%| 93.88% |779ms|
|
||||
| SLANet(ours) | 76.31%| 95.89%|766ms|
|
||||
|
||||
The performance indicators are explained as follows:
|
||||
|
||||
- Acc: The accuracy of the table structure in each image, a wrong token is considered an error.
|
||||
- TEDS: The accuracy of the model's restoration of table information. This indicator evaluates not only the table structure, but also the text content in the table.
|
||||
- Speed: The inference speed of a single image when the model runs on the CPU machine and MKL is enabled.
|
||||
|
||||
## 3. Result
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 4. How to use
|
||||
|
||||
### 4.1 Quick start
|
||||
|
||||
PP-Structure currently provides table recognition models in both Chinese and English. For the model link, see [models_list](../models_list.en.md). The whl package is also provided for quick use, see [quickstart](../quick_start.en.md) for details.
|
||||
|
||||
The following takes the Chinese table recognition model as an example to introduce how to recognize a table.
|
||||
|
||||
Use the following commands to quickly complete the identification of a table.
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
|
||||
# download model
|
||||
mkdir inference && cd inference
|
||||
# Download the PP-OCRv3 text detection model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_det_infer.tar && tar xf PP-OCRv3_mobile_det_infer.tar
|
||||
# Download the PP-OCRv3 text recognition model and unzip it
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_rec_infer.tar && tar xf PP-OCRv3_mobile_rec_infer.tar
|
||||
# Download the PP-StructureV2 form recognition model and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
cd ..
|
||||
# run
|
||||
python3 table/predict_table.py \
|
||||
--det_model_dir=inference/PP-OCRv3_mobile_det_infer \
|
||||
--rec_model_dir=inference/PP-OCRv3_mobile_rec_infer \
|
||||
--table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
|
||||
--image_dir=docs/table/table.jpg \
|
||||
--output=../output/table
|
||||
|
||||
```
|
||||
|
||||
After the operation is completed, the excel table of each image will be saved to the directory specified by the output field, and an html file will be produced in the directory to visually view the cell coordinates and the recognized table.
|
||||
|
||||
**NOTE**
|
||||
|
||||
1. If you want to use the English table recognition model, you need to download the English text detection and recognition model and the English table recognition model in [models_list](../models_list.en.md), and replace `table_structure_dict_ch.txt` with `table_structure_dict.txt`.
|
||||
2. To use the TableRec-RARE model, you need to replace `table_structure_dict_ch.txt` with `table_structure_dict.txt`, and add parameter `--merge_no_span_structure=False`
|
||||
|
||||
### 4.2 Training, Evaluation and Inference
|
||||
|
||||
The training, evaluation and inference process of the text detection model can be referred to [detection](../../ppocr/model_train/detection.en.md)
|
||||
|
||||
The training, evaluation and inference process of the text recognition model can be referred to [recognition](../../ppocr/model_train/recognition.en.md)
|
||||
|
||||
The training, evaluation and inference process of the table recognition model can be referred to [table_recognition](./train_table.en.md)
|
||||
|
||||
### 4.3 Calculate TEDS
|
||||
|
||||
The table uses [TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src) as the evaluation metric of the model. Before the model evaluation, the three models in the pipeline need to be exported as inference models (we have provided them), and the gt for evaluation needs to be prepared. Examples of gt are as follows:
|
||||
|
||||
```txt
|
||||
PMC5755158_010_01.png <html><body><table><thead><tr><td></td><td><b>Weaning</b></td><td><b>Week 15</b></td><td><b>Off-test</b></td></tr></thead><tbody><tr><td>Weaning</td><td>–</td><td>–</td><td>–</td></tr><tr><td>Week 15</td><td>–</td><td>0.17 ± 0.08</td><td>0.16 ± 0.03</td></tr><tr><td>Off-test</td><td>–</td><td>0.80 ± 0.24</td><td>0.19 ± 0.09</td></tr></tbody></table></body></html>
|
||||
```
|
||||
|
||||
Each line in gt consists of the file name and the html string of the table. The file name and the html string of the table are separated by `\t`.
|
||||
|
||||
You can also use the following command to generate an evaluation gt file from the annotation file:
|
||||
|
||||
```bash linenums="1"
|
||||
python3 ppstructure/table/convert_label2html.py --ori_gt_path /path/to/your_label_file --save_path /path/to/save_file
|
||||
```
|
||||
|
||||
Use the following command to evaluate. After the evaluation is completed, the teds indicator will be output.
|
||||
|
||||
```bash linenums="1"
|
||||
python3 table/eval_table.py \
|
||||
--det_model_dir=path/to/det_model_dir \
|
||||
--rec_model_dir=path/to/rec_model_dir \
|
||||
--table_model_dir=path/to/table_model_dir \
|
||||
--image_dir=docs/table/table.jpg \
|
||||
--rec_char_dict_path=../ppocr/utils/dict/table_dict.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
|
||||
--det_limit_side_len=736 \
|
||||
--det_limit_type=min \
|
||||
--gt_path=path/to/gt.txt
|
||||
```
|
||||
|
||||
Evaluate on the PubLatNet dataset using the English model
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
# Download the model
|
||||
mkdir inference && cd inference
|
||||
# Download the text detection model trained on the PubTabNet dataset and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar && tar xf en_ppocr_mobile_v2.0_table_det_infer.tar
|
||||
# Download the text recognition model trained on the PubTabNet dataset and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar && tar xf en_ppocr_mobile_v2.0_table_rec_infer.tar
|
||||
# Download the table recognition model trained on the PubTabNet dataset and unzip it
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
cd ..
|
||||
|
||||
python3 table/eval_table.py \
|
||||
--det_model_dir=inference/en_ppocr_mobile_v2.0_table_det_infer \
|
||||
--rec_model_dir=inference/en_ppocr_mobile_v2.0_table_rec_infer \
|
||||
--table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--image_dir=train_data/table/pubtabnet/val/ \
|
||||
--rec_char_dict_path=../ppocr/utils/dict/table_dict.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
|
||||
--det_limit_side_len=736 \
|
||||
--det_limit_type=min \
|
||||
--rec_image_shape=3,32,320 \
|
||||
--gt_path=path/to/gt.txt
|
||||
```
|
||||
|
||||
output is
|
||||
|
||||
```bash linenums="1"
|
||||
teds: 95.89
|
||||
```
|
||||
|
||||
## 5. Reference
|
||||
|
||||
1. <https://github.com/ibm-aur-nlp/PubTabNet>
|
||||
2. <https://arxiv.org/pdf/1911.10683>
|
||||
167
docs/version2.x/ppstructure/model_train/train_table.md
Normal file
167
docs/version2.x/ppstructure/model_train/train_table.md
Normal file
@@ -0,0 +1,167 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# 表格识别
|
||||
|
||||
## 1. 表格识别 pipeline
|
||||
|
||||
表格识别主要包含三个模型
|
||||
|
||||
1. 单行文本检测-DB
|
||||
2. 单行文本识别-CRNN
|
||||
3. 表格结构和cell坐标预测-SLANet
|
||||
|
||||
具体流程图如下
|
||||
|
||||
|
||||
|
||||
流程说明:
|
||||
|
||||
1. 图片由单行文字检测模型检测到单行文字的坐标，然后送入识别模型拿到识别结果。
|
||||
2. 图片由SLANet模型拿到表格的结构信息和单元格的坐标信息。
|
||||
3. 由单行文字的坐标、识别结果和单元格的坐标一起组合出单元格的识别结果。
|
||||
4. 单元格的识别结果和表格结构一起构造表格的html字符串。
|
||||
|
||||
## 2. 性能
|
||||
|
||||
我们在 PubTabNet<sup>[1]</sup> 评估数据集上对算法进行了评估，性能如下
|
||||
|
||||
| 算法 | Acc | [TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src) | Speed |
|
||||
| ------------------- | ------ | ----------------------------------------------------------------------------------------------------- | ----- |
|
||||
| EDD<sup>[2]</sup> | x | 88.30% | x |
|
||||
| TableRec-RARE(ours) | 71.73% | 93.88% | 779ms |
|
||||
| SLANet(ours) | 76.31% | 95.89% | 766ms |
|
||||
|
||||
性能指标解释如下：
|
||||
|
||||
- Acc: 模型对每张图像里表格结构的识别准确率，错一个token就算错误。
|
||||
- TEDS: 模型对表格信息还原的准确度，此指标评价内容不仅包含表格结构，还包含表格内的文字内容。
|
||||
- Speed: 模型在CPU机器上，开启MKL的情况下，单张图片的推理速度。
|
||||
|
||||
## 3. 效果演示
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 4. 使用
|
||||
|
||||
### 4.1 快速开始
|
||||
|
||||
PP-Structure目前提供了中英文两种语言的表格识别模型，模型链接见 [models_list](../models_list.md)。也提供了whl包的形式方便快速使用，详见 [quickstart](../quick_start.md)。
|
||||
|
||||
下面以中文表格识别模型为例，介绍如何识别一张表格。
|
||||
|
||||
使用如下命令即可快速完成一张表格的识别。
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
|
||||
# 下载模型
|
||||
mkdir inference && cd inference
|
||||
# 下载PP-OCRv3文本检测模型并解压
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_det_infer.tar && tar xf PP-OCRv3_mobile_det_infer.tar
|
||||
# 下载PP-OCRv3文本识别模型并解压
|
||||
wget https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-OCRv3_mobile_rec_infer.tar && tar xf PP-OCRv3_mobile_rec_infer.tar
|
||||
# 下载PP-StructureV2中文表格识别模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf ch_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
cd ..
|
||||
# 执行表格识别
|
||||
python table/predict_table.py \
|
||||
--det_model_dir=inference/PP-OCRv3_mobile_det_infer \
|
||||
--rec_model_dir=inference/PP-OCRv3_mobile_rec_infer \
|
||||
--table_model_dir=inference/ch_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
|
||||
--image_dir=docs/table/table.jpg \
|
||||
--output=../output/table
|
||||
```
|
||||
|
||||
运行完成后，每张图片的excel表格会保存到output字段指定的目录下，同时在该目录下回生产一个html文件，用于可视化查看单元格坐标和识别的表格。
|
||||
|
||||
**NOTE**
|
||||
|
||||
1. 如果想使用英文模型，需要在 [models_list](../../ppocr/model_list.md) 中下载英文文字检测识别模型和英文表格识别模型，同时替换`table_structure_dict_ch.txt`为`table_structure_dict.txt`即可。
|
||||
2. 如需使用TableRec-RARE模型，需要替换`table_structure_dict_ch.txt`为`table_structure_dict.txt`，同时参数`--merge_no_span_structure=False`
|
||||
|
||||
### 4.2 模型训练、评估与推理
|
||||
|
||||
文本检测模型的训练、评估和推理流程可参考 [detection](../../ppocr/model_train/detection.md)
|
||||
|
||||
文本识别模型的训练、评估和推理流程可参考 [recognition](../../ppocr/model_train/recognition.md)
|
||||
|
||||
表格识别模型的训练、评估和推理流程可参考 [table_recognition](../../ppstructure/model_train/train_table.md)
|
||||
|
||||
### 4.3 计算TEDS
|
||||
|
||||
表格使用 [TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src) 作为模型的评估指标。在进行模型评估之前，需要将pipeline中的三个模型分别导出为inference模型(我们已经提供好)，还需要准备评估的gt， gt示例如下:
|
||||
|
||||
```txt
|
||||
PMC5755158_010_01.png <html><body><table><thead><tr><td></td><td><b>Weaning</b></td><td><b>Week 15</b></td><td><b>Off-test</b></td></tr></thead><tbody><tr><td>Weaning</td><td>–</td><td>–</td><td>–</td></tr><tr><td>Week 15</td><td>–</td><td>0.17 ± 0.08</td><td>0.16 ± 0.03</td></tr><tr><td>Off-test</td><td>–</td><td>0.80 ± 0.24</td><td>0.19 ± 0.09</td></tr></tbody></table></body></html>
|
||||
```
|
||||
|
||||
gt每一行都由文件名和表格的html字符串组成，文件名和表格的html字符串之间使用`\t`分隔。
|
||||
|
||||
也可使用如下命令，由标注文件生成评估的gt文件：
|
||||
|
||||
```bash linenums="1"
|
||||
python3 ppstructure/table/convert_label2html.py --ori_gt_path /path/to/your_label_file --save_path /path/to/save_file
|
||||
```
|
||||
|
||||
准备完成后使用如下命令进行评估，评估完成后会输出teds指标。
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
python3 table/eval_table.py \
|
||||
--det_model_dir=path/to/det_model_dir \
|
||||
--rec_model_dir=path/to/rec_model_dir \
|
||||
--table_model_dir=path/to/table_model_dir \
|
||||
--image_dir=docs/table/table.jpg \
|
||||
--rec_char_dict_path=../ppocr/utils/dict/table_dict.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
|
||||
--det_limit_side_len=736 \
|
||||
--det_limit_type=min \
|
||||
--gt_path=path/to/gt.txt
|
||||
```
|
||||
|
||||
如使用英文表格识别模型在PubLatNet数据集上进行评估
|
||||
|
||||
```bash linenums="1"
|
||||
cd PaddleOCR/ppstructure
|
||||
# 下载模型
|
||||
mkdir inference && cd inference
|
||||
# 下载基于PubTabNet数据集训练的文本检测模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar && tar xf en_ppocr_mobile_v2.0_table_det_infer.tar
|
||||
# 下载基于PubTabNet数据集训练的文本识别模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar && tar xf en_ppocr_mobile_v2.0_table_rec_infer.tar
|
||||
# 下载基于PubTabNet数据集训练的表格识别模型并解压
|
||||
wget https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar && tar xf en_ppstructure_mobile_v2.0_SLANet_infer.tar
|
||||
cd ..
|
||||
|
||||
python3 table/eval_table.py \
|
||||
--det_model_dir=inference/en_ppocr_mobile_v2.0_table_det_infer \
|
||||
--rec_model_dir=inference/en_ppocr_mobile_v2.0_table_rec_infer \
|
||||
--table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
|
||||
--image_dir=train_data/table/pubtabnet/val/ \
|
||||
--rec_char_dict_path=../ppocr/utils/dict/table_dict.txt \
|
||||
--table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
|
||||
--det_limit_side_len=736 \
|
||||
--det_limit_type=min \
|
||||
--rec_image_shape=3,32,320 \
|
||||
--gt_path=path/to/gt.txt
|
||||
```
|
||||
|
||||
将会输出
|
||||
|
||||
```bash linenums="1"
|
||||
teds: 95.89
|
||||
```
|
||||
|
||||
## 5. Reference
|
||||
|
||||
1. <https://github.com/ibm-aur-nlp/PubTabNet>
|
||||
2. <https://arxiv.org/pdf/1911.10683>
|
||||
132
docs/version2.x/ppstructure/model_train/training.en.md
Normal file
132
docs/version2.x/ppstructure/model_train/training.en.md
Normal file
@@ -0,0 +1,132 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# Model Training
|
||||
|
||||
This article will introduce the basic concepts that is necessary for model training and tuning.
|
||||
|
||||
At the same time, it will briefly introduce the structure of the training data and how to prepare the data to fine-tune model in vertical scenes.
|
||||
|
||||
## 1. Yml Configuration
|
||||
|
||||
The PaddleOCR uses configuration files to control network training and evaluation parameters. In the configuration file, you can set the model, optimizer, loss function, and pre- and post-processing parameters of the model. PaddleOCR reads these parameters from the configuration file, and then builds a complete training process to train the model. Fine-tuning can also be completed by modifying the parameters in the configuration file, which is simple and convenient.
|
||||
|
||||
For the complete configuration file description, please refer to [Configuration File](../../ppocr/blog/config.en.md)
|
||||
|
||||
## 2. Basic Concepts
|
||||
|
||||
During the model training process, some hyper-parameters can be manually specified to obtain the optimal result at the least cost. Different data volumes may require different hyper-parameters. When you want to fine-tune the model based on your own data, there are several parameter adjustment strategies for reference:
|
||||
|
||||
### 2.1 Learning Rate
|
||||
|
||||
The learning rate is one of the most important hyper-parameters for training neural networks. It represents the step length of the gradient moving towards the optimal solution of the loss function in each iteration.
|
||||
A variety of learning rate update strategies are provided by PaddleOCR, which can be specified in configuration files. For example,
|
||||
|
||||
```yaml linenums="1"
|
||||
Optimizer:
|
||||
...
|
||||
lr:
|
||||
name: Piecewise
|
||||
decay_epochs : [700, 800]
|
||||
values : [0.001, 0.0001]
|
||||
warmup_epoch: 5
|
||||
```
|
||||
|
||||
`Piecewise` stands for piece-wise constant attenuation. Different learning rates are specified in different learning stages, and the learning rate stay the same in each stage.
|
||||
|
||||
`warmup_epoch` means that in the first 5 epochs, the learning rate will be increased gradually from 0 to base_lr. For all strategies, please refer to the code [learning_rate.py](../../ppocr/optimizer/learning_rate.py).
|
||||
|
||||
### 2.2 Regularization
|
||||
|
||||
Regularization can effectively avoid algorithm over-fitting. PaddleOCR provides L1 and L2 regularization methods.
|
||||
L1 and L2 regularization are the most widely used regularization methods.
|
||||
L1 regularization adds a regularization term to the objective function to reduce the sum of absolute values of the parameters;
|
||||
while in L2 regularization, the purpose of adding a regularization term is to reduce the sum of squared parameters.
|
||||
The configuration method is as follows:
|
||||
|
||||
```yaml linenums="1"
|
||||
Optimizer:
|
||||
...
|
||||
regularizer:
|
||||
name: L2
|
||||
factor: 2.0e-05
|
||||
```
|
||||
|
||||
### 2.3 Evaluation Indicators
|
||||
|
||||
(1) Detection stage: First, evaluate according to the IOU of the detection frame and the labeled frame. If the IOU is greater than a certain threshold, it is judged that the detection is accurate. Here, the detection frame and the label frame are different from the general general target detection frame, and they are represented by polygons. Detection accuracy: the percentage of the correct detection frame number in all detection frames is mainly used to judge the detection index. Detection recall rate: the percentage of correct detection frames in all marked frames, which is mainly an indicator of missed detection.
|
||||
|
||||
(2) Recognition stage: Character recognition accuracy, that is, the ratio of correctly recognized text lines to the number of marked text lines. Only the entire line of text recognition pairs can be regarded as correct recognition.
|
||||
|
||||
(3) End-to-end statistics: End-to-end recall rate: accurately detect and correctly identify the proportion of text lines in all labeled text lines; End-to-end accuracy rate: accurately detect and correctly identify the number of text lines in the detected text lines The standard for accurate detection is that the IOU of the detection box and the labeled box is greater than a certain threshold, and the text in the correctly identified detection box is the same as the labeled text.
|
||||
|
||||
## 3. Data and Vertical Scenes
|
||||
|
||||
### 3.1 Training Data
|
||||
|
||||
The current open source models, data sets and magnitudes are as follows:
|
||||
|
||||
- Detection:
|
||||
- English data set, ICDAR2015
|
||||
- Chinese data set, LSVT street view data set training data 3w pictures
|
||||
|
||||
- Identification:
|
||||
- English data set, MJSynth and SynthText synthetic data, the data volume is tens of millions.
|
||||
- Chinese data set, LSVT street view data set crops the image according to the truth value, and performs position calibration, a total of 30w images. In addition, based on the LSVT corpus, 500w of synthesized data.
|
||||
- Small language data set, using different corpora and fonts, respectively generated 100w synthetic data set, and using ICDAR-MLT as the verification set.
|
||||
|
||||
Among them, the public data sets are all open source, users can search and download by themselves, or refer to [Chinese data set](../../datasets/datasets.en.md), synthetic data is not open source, users can use open source synthesis tools to synthesize by themselves. Synthesis tools include [text_renderer](https://github.com/Sanster/text_renderer), [SynthText](https://github.com/ankush-me/SynthText), [TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) etc.
|
||||
|
||||
### 3.2 Vertical Scene
|
||||
|
||||
PaddleOCR mainly focuses on general OCR. If you have vertical requirements, you can use PaddleOCR + vertical data to train yourself;
|
||||
If there is a lack of labeled data, or if you do not want to invest in research and development costs, it is recommended to directly call the open API, which covers some of the more common vertical categories.
|
||||
|
||||
### 3.3 Build Your Own Dataset
|
||||
|
||||
There are several experiences for reference when constructing the data set:
|
||||
|
||||
(1) The amount of data in the training set:
|
||||
|
||||
a. The data required for detection is relatively small. For Fine-tune based on the PaddleOCR model, 500 sheets are generally required to achieve good results.
|
||||
|
||||
b. Recognition is divided into English and Chinese. Generally, English scenarios require hundreds of thousands of data to achieve good results, while Chinese requires several million or more.
|
||||
|
||||
(2) When the amount of training data is small, you can try the following three ways to get more data:
|
||||
|
||||
a. Manually collect more training data, the most direct and effective way.
|
||||
|
||||
b. Basic image processing or transformation based on PIL and opencv. For example, the three modules of ImageFont, Image, ImageDraw in PIL write text into the background, opencv's rotating affine transformation, Gaussian filtering and so on.
|
||||
|
||||
c. Use data generation algorithms to synthesize data, such as algorithms such as pix2pix.
|
||||
|
||||
## 4. FAQ
|
||||
|
||||
**Q**: How to choose a suitable network input shape when training CRNN recognition?
|
||||
|
||||
A: The general height is 32, the longest width is selected, there are two methods:
|
||||
|
||||
(1) Calculate the aspect ratio distribution of training sample images. The selection of the maximum aspect ratio considers 80% of the training samples.
|
||||
|
||||
(2) Count the number of texts in training samples. The selection of the longest number of characters considers the training sample that satisfies 80%. Then the aspect ratio of Chinese characters is approximately considered to be 1, and that of English is 3:1, and the longest width is estimated.
|
||||
|
||||
**Q**: During the recognition training, the accuracy of the training set has reached 90, but the accuracy of the verification set has been kept at 70, what should I do?
|
||||
|
||||
A: If the accuracy of the training set is 90 and the test set is more than 70, it should be over-fitting. There are two methods to try:
|
||||
|
||||
(1) Add more augmentation methods or increase the [probability] of augmented prob (https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/rec_img_aug.py#L341), The default is 0.4.
|
||||
|
||||
(2) Increase the [l2 dcay value] of the system (https://github.com/PaddlePaddle/PaddleOCR/blob/a501603d54ff5513fc4fc760319472e59da25424/configs/rec/ch_ppocr_v1.1/rec_chinese_lite_train_v1.1.yml#L47)
|
||||
|
||||
**Q**: When the recognition model is trained, loss can drop normally, but acc is always 0
|
||||
|
||||
A: It is normal for the acc to be 0 at the beginning of the recognition model training, and the indicator will come up after a longer training period.
|
||||
|
||||
***
|
||||
|
||||
Click the following links for detailed training tutorial:
|
||||
|
||||
- [text detection model training](./detection.en.md)
|
||||
- [text recognition model training](./recognition.en.md)
|
||||
- [text direction classification model training](./angle_class.en.md)
|
||||
128
docs/version2.x/ppstructure/model_train/training.md
Normal file
128
docs/version2.x/ppstructure/model_train/training.md
Normal file
@@ -0,0 +1,128 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-OCR模型训练
|
||||
|
||||
本文将介绍模型训练时需掌握的基本概念，和训练时的调优方法。
|
||||
|
||||
同时会简单介绍PaddleOCR模型训练数据的组成部分，以及如何在垂类场景中准备数据finetune模型。
|
||||
|
||||
## 1. 配置文件说明
|
||||
|
||||
PaddleOCR模型使用配置文件管理网络训练、评估的参数。在配置文件中，可以设置组建模型、优化器、损失函数、模型前后处理的参数，PaddleOCR从配置文件中读取到这些参数，进而组建出完整的训练流程，完成模型训练，在需要对模型进行优化的时，可以通过修改配置文件中的参数完成配置，使用简单且方便修改。
|
||||
|
||||
完整的配置文件说明可以参考[配置文件](../../ppocr/blog/config.md)
|
||||
|
||||
## 2. 基本概念
|
||||
|
||||
模型训练过程中需要手动调整一些超参数，帮助模型以最小的代价获得最优指标。不同的数据量可能需要不同的超参，当您希望在自己的数据上finetune或对模型效果调优时，有以下几个参数调整策略可供参考：
|
||||
|
||||
### 2.1 学习率
|
||||
|
||||
学习率是训练神经网络的重要超参数之一，它代表在每一次迭代中梯度向损失函数最优解移动的步长。
|
||||
在PaddleOCR中提供了多种学习率更新策略,可以通过配置文件修改，例如：
|
||||
|
||||
```yaml linenums="1"
|
||||
Optimizer:
|
||||
...
|
||||
lr:
|
||||
name: Piecewise
|
||||
decay_epochs : [700, 800]
|
||||
values : [0.001, 0.0001]
|
||||
warmup_epoch: 5
|
||||
```
|
||||
|
||||
Piecewise 代表分段常数衰减，在不同的学习阶段指定不同的学习率，在每段内学习率相同。
|
||||
warmup_epoch 代表在前5个epoch中，学习率将逐渐从0增加到base_lr。全部策略可以参考代码[learning_rate.py](../../ppocr/optimizer/learning_rate.py) 。
|
||||
|
||||
### 2.2 正则化
|
||||
|
||||
正则化可以有效的避免算法过拟合，PaddleOCR中提供了L1、L2正则方法，L1 和 L2 正则化是最常用的正则化方法。L1 正则化向目标函数添加正则化项，以减少参数的绝对值总和；而 L2 正则化中，添加正则化项的目的在于减少参数平方的总和。配置方法如下：
|
||||
|
||||
```yaml linenums="1"
|
||||
Optimizer:
|
||||
...
|
||||
regularizer:
|
||||
name: L2
|
||||
factor: 2.0e-05
|
||||
```
|
||||
|
||||
### 2.3 评估指标
|
||||
|
||||
（1）检测阶段：先按照检测框和标注框的IOU评估，IOU大于某个阈值判断为检测准确。这里检测框和标注框不同于一般的通用目标检测框，是采用多边形进行表示。检测准确率：正确的检测框个数在全部检测框的占比，主要是判断检测指标。检测召回率：正确的检测框个数在全部标注框的占比，主要是判断漏检的指标。
|
||||
|
||||
（2）识别阶段： 字符识别准确率，即正确识别的文本行占标注的文本行数量的比例，只有整行文本识别对才算正确识别。
|
||||
|
||||
（3）端到端统计： 端对端召回率：准确检测并正确识别文本行在全部标注文本行的占比； 端到端准确率：准确检测并正确识别文本行在 检测到的文本行数量 的占比； 准确检测的标准是检测框与标注框的IOU大于某个阈值，正确识别的检测框中的文本与标注的文本相同。
|
||||
|
||||
## 3. 数据与垂类场景
|
||||
|
||||
### 3.1 训练数据
|
||||
|
||||
目前开源的模型，数据集和量级如下：
|
||||
|
||||
- 检测：
|
||||
- 英文数据集，ICDAR2015
|
||||
- 中文数据集，LSVT街景数据集训练数据3w张图片
|
||||
|
||||
- 识别：
|
||||
- 英文数据集，MJSynth和SynthText合成数据，数据量上千万。
|
||||
- 中文数据集，LSVT街景数据集根据真值将图crop出来，并进行位置校准，总共30w张图像。此外基于LSVT的语料，合成数据500w。
|
||||
- 小语种数据集，使用不同语料和字体，分别生成了100w合成数据集，并使用ICDAR-MLT作为验证集。
|
||||
|
||||
其中，公开数据集都是开源的，用户可自行搜索下载，也可参考[中文数据集](../../datasets/datasets.md)，合成数据暂不开源，用户可使用开源合成工具自行合成，可参考的合成工具包括[text_renderer](https://github.com/Sanster/text_renderer) 、[SynthText](https://github.com/ankush-me/SynthText) 、[TextRecognitionDataGenerator](https://github.com/Belval/TextRecognitionDataGenerator) 等。
|
||||
|
||||
### 3.2 垂类场景
|
||||
|
||||
PaddleOCR主要聚焦通用OCR，如果有垂类需求，您可以用PaddleOCR+垂类数据自己训练；
|
||||
如果缺少带标注的数据，或者不想投入研发成本，建议直接调用开放的API，开放的API覆盖了目前比较常见的一些垂类。
|
||||
|
||||
### 3.3 自己构建数据集
|
||||
|
||||
在构建数据集时有几个经验可供参考：
|
||||
|
||||
（1） 训练集的数据量：
|
||||
|
||||
a. 检测需要的数据相对较少，在PaddleOCR模型的基础上进行Fine-tune，一般需要500张可达到不错的效果。
|
||||
|
||||
b. 识别分英文和中文，一般英文场景需要几十万数据可达到不错的效果，中文则需要几百万甚至更多。
|
||||
|
||||
（2）当训练数据量少时，可以尝试以下三种方式获取更多的数据：
|
||||
|
||||
a. 人工采集更多的训练数据，最直接也是最有效的方式。
|
||||
|
||||
b. 基于PIL和opencv基本图像处理或者变换。例如PIL中ImageFont, Image, ImageDraw三个模块将文字写到背景中，opencv的旋转仿射变换，高斯滤波等。
|
||||
|
||||
c. 利用数据生成算法合成数据，例如pix2pix或[StyleText](https://github.com/PFCCLab/StyleText)等算法。
|
||||
|
||||
## 4. 常见问题
|
||||
|
||||
**Q**：训练CRNN识别时，如何选择合适的网络输入shape？
|
||||
|
||||
A：一般高度采用32，最长宽度的选择，有两种方法：
|
||||
|
||||
（1）统计训练样本图像的宽高比分布。最大宽高比的选取考虑满足80%的训练样本。
|
||||
|
||||
（2）统计训练样本文字数目。最长字符数目的选取考虑满足80%的训练样本。然后中文字符长宽比近似认为是1，英文认为3：1，预估一个最长宽度。
|
||||
|
||||
**Q**：识别训练时，训练集精度已经到达90了，但验证集精度一直在70，涨不上去怎么办？
|
||||
|
||||
A：训练集精度90，测试集70多的话，应该是过拟合了，有两个可尝试的方法：
|
||||
|
||||
（1）加入更多的增广方式或者调大增广prob的[概率](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/rec_img_aug.py#L341)，默认为0.4。
|
||||
|
||||
（2）调大系统的[l2 dcay值](https://github.com/PaddlePaddle/PaddleOCR/blob/a501603d54ff5513fc4fc760319472e59da25424/configs/rec/ch_ppocr_v1.1/rec_chinese_lite_train_v1.1.yml#L47)
|
||||
|
||||
**Q**: 识别模型训练时，loss能正常下降，但acc一直为0
|
||||
|
||||
A：识别模型训练初期acc为0是正常的，多训一段时间指标就上来了。
|
||||
|
||||
***
|
||||
|
||||
具体的训练教程可点击下方链接跳转：
|
||||
|
||||
- [文本检测模型训练](./detection.md)
|
||||
- [文本识别模型训练](./recognition.md)
|
||||
- [文本方向分类器训练](./angle_class.md)
|
||||
- [知识蒸馏](../model_compress/knowledge_distillation.md)
|
||||
57
docs/version2.x/ppstructure/models_list.en.md
Normal file
57
docs/version2.x/ppstructure/models_list.en.md
Normal file
@@ -0,0 +1,57 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-Structure Model list
|
||||
|
||||
## 1. Layout Analysis
|
||||
|
||||
|model name| description | inference model size |download|dict path|
|
||||
| --- |----| --- | --- | --- |
|
||||
| picodet_lcnet_x1_0_fgd_layout | The layout analysis English model trained on the PubLayNet dataset based on PicoDet LCNet_x1_0 and FGD . the model can recognition 5 types of areas such as **Text, Title, Table, Picture and List** | 9.7M | [inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout.pdparams) | [PubLayNet dict](../../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt) |
|
||||
| ppyolov2_r50vd_dcn_365e_publaynet | The layout analysis English model trained on the PubLayNet dataset based on PP-YOLOv2 | 221.0M | [inference_moel](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) / [trained model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet_pretrained.pdparams) | same as above |
|
||||
| picodet_lcnet_x1_0_fgd_layout_cdla | The layout analysis Chinese model trained on the CDLA dataset, the model can recognition 10 types of areas such as **Table、Figure、Figure caption、Table、Table caption、Header、Footer、Reference、Equation** | 9.7M | [inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla.pdparams) | [CDLA dict](../../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt) |
|
||||
| picodet_lcnet_x1_0_fgd_layout_table | The layout analysis model trained on the table dataset, the model can detect tables in Chinese and English documents | 9.7M | [inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table.pdparams) | [Table dict](../../ppocr/utils/dict/layout_dict/layout_table_dict.txt) |
|
||||
| ppyolov2_r50vd_dcn_365e_tableBank_word | The layout analysis model trained on the TableBank Word dataset based on PP-YOLOv2, the model can detect tables in English documents | 221.0M | [inference model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) | same as above |
|
||||
| ppyolov2_r50vd_dcn_365e_tableBank_latex | The layout analysis model trained on the TableBank Latex dataset based on PP-YOLOv2, the model can detect tables in English documents | 221.0M | [inference model](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) | same as above |
|
||||
|
||||
## 2. OCR and Table Recognition
|
||||
|
||||
### 2.1 OCR
|
||||
|
||||
|model name| description | inference model size |download|
|
||||
| --- |---|---| --- |
|
||||
|en_ppocr_mobile_v2.0_table_det| Text detection model of English table scenes trained on PubTabNet dataset | 4.7M |[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_det_train.tar) |
|
||||
|en_ppocr_mobile_v2.0_table_rec| Text recognition model of English table scenes trained on PubTabNet dataset | 6.9M |[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_rec_train.tar) |
|
||||
|
||||
If you need to use other OCR models, you can download the model in [PP-OCR model_list](../ppocr/model_list.en.md) or use the model you trained yourself to configure to `det_model_dir`, `rec_model_dir` field.
|
||||
|
||||
### 2.2 Table Recognition
|
||||
|
||||
|model| description |inference model size|download|
|
||||
| --- |-----| --- | --- |
|
||||
|en_ppocr_mobile_v2.0_table_structure| English table recognition model trained on PubTabNet dataset based on TableRec-RARE |6.8M|[inference model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
|
||||
|en_ppstructure_mobile_v2.0_SLANet|English table recognition model trained on PubTabNet dataset based on SLANet|9.2M|[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_train.tar) |
|
||||
|ch_ppstructure_mobile_v2.0_SLANet|Chinese table recognition model based on SLANet|9.3M|[inference model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
|
||||
|
||||
## 3. KIE
|
||||
|
||||
On XFUND_zh dataset, Accuracy and time cost of different models on V100 GPU are as follows.
|
||||
|
||||
|Model|Backbone|Task|Config|Hmean|Time cost(ms)|Download link|
|
||||
| --- | --- | --- | --- | --- | --- |--- |
|
||||
|VI-LayoutXLM| VI-LayoutXLM-base | SER | [ser_vi_layoutxlm_xfund_zh_udml.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh_udml.yml)|**93.19%**| 15.49| [trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar)|
|
||||
|LayoutXLM| LayoutXLM-base | SER | [ser_layoutxlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/ser_layoutxlm_xfund_zh.yml)|90.38%| 19.49 |[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar)|
|
||||
|LayoutLM| LayoutLM-base | SER | [ser_layoutlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml)|77.31%|-|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar)|
|
||||
|LayoutLMv2| LayoutLMv2-base | SER | [ser_layoutlmv2_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/ser_layoutlmv2_xfund_zh.yml)|85.44%|31.46|[trained model](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar)|
|
||||
|VI-LayoutXLM| VI-LayoutXLM-base | RE | [re_vi_layoutxlm_xfund_zh_udml.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh_udml.yml)|**83.92%**|15.49|[trained model](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar)|
|
||||
|LayoutXLM| LayoutXLM-base | RE | [re_layoutxlm_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/re_layoutxlm_xfund_zh.yml)|74.83%|19.49|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar)|
|
||||
|LayoutLMv2| LayoutLMv2-base | RE | [re_layoutlmv2_xfund_zh.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/layoutlm_series/re_layoutlmv2_xfund_zh.yml)|67.77%|31.46|[trained model](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar)|
|
||||
|
||||
* Note: The above time cost information just considers inference time without preprocess or postprocess, test environment: `V100 GPU + CUDA 10.2 + CUDNN 8.1.1 + TRT 7.2.3.4`
|
||||
|
||||
On wildreceipt dataset, the algorithm result is as follows:
|
||||
|
||||
|Model|Backbone|Config|Hmean|Download link|
|
||||
| --- | --- | --- | --- | --- |
|
||||
|SDMGR|VGG6|[configs/kie/sdmgr/kie_unet_sdmgr.yml](https://github.com/PaddlePaddle/PaddleOCR/tree/main/configs/kie/sdmgr/kie_unet_sdmgr.yml)|86.70%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)|
|
||||
57
docs/version2.x/ppstructure/models_list.md
Normal file
57
docs/version2.x/ppstructure/models_list.md
Normal file
@@ -0,0 +1,57 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-Structure 系列模型列表
|
||||
|
||||
## 1. 版面分析模型
|
||||
|
||||
|模型名称|模型简介|推理模型大小|下载地址|dict path|
|
||||
| --- | --- | --- | --- | --- |
|
||||
| picodet_lcnet_x1_0_fgd_layout | 基于PicoDet LCNet_x1_0和FGD蒸馏在PubLayNet 数据集训练的英文版面分析模型，可以划分**文字、标题、表格、图片以及列表**5类区域 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout.pdparams) | [PubLayNet dict](../../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt) |
|
||||
| ppyolov2_r50vd_dcn_365e_publaynet | 基于PP-YOLOv2在PubLayNet数据集上训练的英文版面分析模型 | 221.0M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) / [训练模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet_pretrained.pdparams) | 同上 |
|
||||
| picodet_lcnet_x1_0_fgd_layout_cdla | CDLA数据集训练的中文版面分析模型，可以划分为**表格、图片、图片标题、表格、表格标题、页眉、脚本、引用、公式**10类区域 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla.pdparams) | [CDLA dict](../../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt) |
|
||||
| picodet_lcnet_x1_0_fgd_layout_table | 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table.pdparams) | [Table dict](../../ppocr/utils/dict/layout_dict/layout_table_dict.txt) |
|
||||
| ppyolov2_r50vd_dcn_365e_tableBank_word | 基于PP-YOLOv2在TableBank Word 数据集训练的版面分析模型，支持英文文档表格区域的检测 | 221.0M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) | 同上 |
|
||||
| ppyolov2_r50vd_dcn_365e_tableBank_latex | 基于PP-YOLOv2在TableBank Latex数据集训练的版面分析模型，支持英文文档表格区域的检测 | 221.0M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) | 同上 |
|
||||
|
||||
## 2. OCR和表格识别模型
|
||||
|
||||
### 2.1 OCR
|
||||
|
||||
|模型名称|模型简介|推理模型大小|下载地址|
|
||||
| --- | --- | --- | --- |
|
||||
|en_ppocr_mobile_v2.0_table_det|PubTabNet数据集训练的英文表格场景的文字检测|4.7M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_det_train.tar) |
|
||||
|en_ppocr_mobile_v2.0_table_rec|PubTabNet数据集训练的英文表格场景的文字识别|6.9M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_rec_train.tar) |
|
||||
|
||||
如需要使用其他OCR模型，可以在 [PP-OCR model_list](../ppocr/model_list.md) 下载模型或者使用自己训练好的模型配置到 `det_model_dir`, `rec_model_dir`两个字段即可。
|
||||
|
||||
### 2.2 表格识别模型
|
||||
|
||||
|模型名称|模型简介|推理模型大小|下载地址|
|
||||
| --- | --- | --- | --- |
|
||||
|en_ppocr_mobile_v2.0_table_structure|基于TableRec-RARE在PubTabNet数据集上训练的英文表格识别模型|6.8M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
|
||||
|en_ppstructure_mobile_v2.0_SLANet|基于SLANet在PubTabNet数据集上训练的英文表格识别模型|9.2M|[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_train.tar) |
|
||||
|ch_ppstructure_mobile_v2.0_SLANet|基于SLANet的中文表格识别模型|9.3M|[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
|
||||
|
||||
## 3. KIE模型
|
||||
|
||||
在XFUND_zh数据集上，不同模型的精度与V100 GPU上速度信息如下所示。
|
||||
|
||||
|模型名称|模型简介 | 推理模型大小| 精度(hmean) | 预测耗时(ms) | 下载地址|
|
||||
| --- | --- | --- |--- |--- | --- |
|
||||
|ser_VI-LayoutXLM_xfund_zh|基于VI-LayoutXLM在xfund中文数据集上训练的SER模型|1.1G| 93.19% | 15.49 | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar) |
|
||||
|re_VI-LayoutXLM_xfund_zh|基于VI-LayoutXLM在xfund中文数据集上训练的RE模型|1.1G| 83.92% | 15.49 |[推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar) |
|
||||
|ser_LayoutXLM_xfund_zh|基于LayoutXLM在xfund中文数据集上训练的SER模型|1.4G| 90.38% | 19.49 |[推理模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar) |
|
||||
|re_LayoutXLM_xfund_zh|基于LayoutXLM在xfund中文数据集上训练的RE模型|1.4G| 74.83% | 19.49 |[推理模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar) |
|
||||
|ser_LayoutLMv2_xfund_zh|基于LayoutLMv2在xfund中文数据集上训练的SER模型|778.0M| 85.44% | 31.46 |[推理模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar) |
|
||||
|re_LayoutLMv2_xfund_zh|基于LayoutLMv2在xfun中文数据集上训练的RE模型|765.0M| 67.77% | 31.46 |[推理模型 coming soon]() / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar) |
|
||||
|ser_LayoutLM_xfund_zh|基于LayoutLM在xfund中文数据集上训练的SER模型|430.0M| 77.31% | - |[推理模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar) |
|
||||
|
||||
* 注：上述预测耗时信息仅包含了inference模型的推理耗时，没有统计预处理与后处理耗时，测试环境为`V100 GPU + CUDA 10.2 + CUDNN 8.1.1 + TRT 7.2.3.4`。
|
||||
|
||||
在wildreceipt数据集上，SDMGR模型精度与下载地址如下所示。
|
||||
|
||||
|模型名称|模型简介|模型大小|精度|下载地址|
|
||||
| --- | --- | --- |--- | --- |
|
||||
|SDMGR|关键信息提取模型|78.0M| 86.70% | [推理模型 coming soon]() / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar)|
|
||||
104
docs/version2.x/ppstructure/overview.en.md
Normal file
104
docs/version2.x/ppstructure/overview.en.md
Normal file
@@ -0,0 +1,104 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-Structure
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
PP-Structure is an intelligent document analysis system developed by the PaddleOCR team, which aims to help developers better complete tasks related to document understanding such as layout analysis and table recognition.
|
||||
|
||||
The pipeline of PP-StructureV2 system is shown below. The document image first passes through the image direction correction module to identify the direction of the entire image and complete the direction correction. Then, two tasks of layout information analysis and key information extraction can be completed.
|
||||
|
||||
- In the layout analysis task, the image first goes through the layout analysis model to divide the image into different areas such as text, table, and figure, and then analyze these areas separately. For example, the table area is sent to the form recognition module for structured recognition, and the text area is sent to the OCR engine for text recognition. Finally, the layout recovery module restores it to a word or pdf file with the same layout as the original image;
|
||||
- In the key information extraction task, the OCR engine is first used to extract the text content, and then the SER(semantic entity recognition) module obtains the semantic entities in the image, and finally the RE(relationship extraction) module obtains the correspondence between the semantic entities, thereby extracting the required key information.
|
||||
|
||||
|
||||
|
||||
More technical details: 👉 [PP-StructureV2 Technical Report](https://arxiv.org/abs/2210.05391)
|
||||
|
||||
PP-StructureV2 supports independent use or flexible collocation of each module. For example, you can use layout analysis alone or table recognition alone. Click the corresponding link below to get the tutorial for each independent module:
|
||||
|
||||
- [Layout Analysis](./model_train/train_layout.en.md)
|
||||
- [Table Recognition](./model_train/train_table.en.md)
|
||||
- [Key Information Extraction](./model_train/train_kie.en.md)
|
||||
- [Layout Recovery](./model_train/recovery_to_doc.en.md)
|
||||
|
||||
## 2. Features
|
||||
|
||||
The main features of PP-StructureV2 are as follows:
|
||||
|
||||
- Support layout analysis of documents in the form of images/pdfs, which can be divided into areas such as **text, titles, tables, figures, formulas, etc.**;
|
||||
- Support common Chinese and English **table detection** tasks;
|
||||
- Support structured table recognition, and output the final result to **Excel file**;
|
||||
- Support multimodal-based Key Information Extraction (KIE) tasks - **Semantic Entity Recognition** (SER) and **Relation Extraction** (RE);
|
||||
- Support **layout recovery**, that is, restore the document in word or pdf format with the same layout as the original image;
|
||||
- Support customized training and multiple inference deployment methods such as python whl package quick start;
|
||||
- Connect with the semi-automatic data labeling tool PPOCRLabel, which supports the labeling of layout analysis, table recognition, and SER.
|
||||
|
||||
## 3. Results
|
||||
|
||||
PP-StructureV2 supports the independent use or flexible collocation of each module. For example, layout analysis can be used alone, or table recognition can be used alone. Only the visualization effects of several representative usage methods are shown here.
|
||||
|
||||
### 3.1 Layout analysis and table recognition
|
||||
|
||||
The figure shows the pipeline of layout analysis + table recognition. The image is first divided into four areas of image, text, title and table by layout analysis, and then OCR detection and recognition is performed on the three areas of image, text and title, and the table is performed table recognition, where the image will also be stored for use.
|
||||
|
||||
|
||||
|
||||
### 3.1.1 Layout recognition returns the coordinates of a single word
|
||||
|
||||
The following figure shows the result of layout analysis on single word，please refer to the [doc](./blog/return_word_pos.en.md).
|
||||
|
||||
|
||||
|
||||
### 3.2 Layout recovery
|
||||
|
||||
The following figure shows the effect of layout recovery based on the results of layout analysis and table recognition in the previous section.
|
||||
|
||||
|
||||
|
||||
### 3.3 KIE
|
||||
|
||||
- SER
|
||||
|
||||
Different colored boxes in the figure represent different categories.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
- RE
|
||||
|
||||
In the figure, the red box represents `Question`, the blue box represents `Answer`, and `Question` and `Answer` are connected by green lines.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 4. Quick start
|
||||
|
||||
Start from [Quick Start](./quick_start.en.md).
|
||||
|
||||
## 5. Model List
|
||||
|
||||
Some tasks need to use both the structured analysis models and the OCR models. For example, the table recognition task needs to use the table recognition model for structured analysis, and the OCR model to recognize the text in the table. Please select the appropriate models according to your specific needs.
|
||||
|
||||
For structural analysis related model downloads, please refer to:
|
||||
|
||||
- [PP-Structure Model Zoo](./models_list.en.md)
|
||||
|
||||
For OCR related model downloads, please refer to:
|
||||
|
||||
- [PP-OCR Model Zoo](../ppocr/model_list.en.md)
|
||||
104
docs/version2.x/ppstructure/overview.md
Normal file
104
docs/version2.x/ppstructure/overview.md
Normal file
@@ -0,0 +1,104 @@
|
||||
---
|
||||
typora-copy-images-to: images
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-Structure 文档分析
|
||||
|
||||
## 1. 简介
|
||||
|
||||
PP-Structure是PaddleOCR团队自研的智能文档分析系统，旨在帮助开发者更好的完成版面分析、表格识别等文档理解相关任务。
|
||||
|
||||
PP-StructureV2系统流程图如下所示，文档图像首先经过图像矫正模块，判断整图方向并完成转正，随后可以完成版面信息分析与关键信息抽取2类任务。
|
||||
|
||||
- 版面分析任务中，图像首先经过版面分析模型，将图像划分为文本、表格、图像等不同区域，随后对这些区域分别进行识别，如，将表格区域送入表格识别模块进行结构化识别，将文本区域送入OCR引擎进行文字识别，最后使用版面恢复模块将其恢复为与原始图像布局一致的word或者pdf格式的文件；
|
||||
- 关键信息抽取任务中，首先使用OCR引擎提取文本内容，然后由语义实体识别模块获取图像中的语义实体，最后经关系抽取模块获取语义实体之间的对应关系，从而提取需要的关键信息。
|
||||
|
||||
|
||||
|
||||
更多技术细节：👉 PP-StructureV2技术报告 [中文版](docs/PP-StructureV2_introduction.md)，[英文版](https://arxiv.org/abs/2210.05391)。
|
||||
|
||||
PP-StructureV2支持各个模块独立使用或灵活搭配，如，可以单独使用版面分析，或单独使用表格识别，点击下面相应链接获取各个独立模块的使用教程：
|
||||
|
||||
- [版面分析](model_train/train_layout.md)
|
||||
- [表格识别](model_train/train_table.md)
|
||||
- [关键信息抽取](model_train/train_kie.md)
|
||||
- [版面复原](model_train/recovery_to_doc.md)
|
||||
|
||||
## 2. 特性
|
||||
|
||||
PP-StructureV2的主要特性如下：
|
||||
|
||||
- 支持对图片/pdf形式的文档进行版面分析，可以划分**文字、标题、表格、图片、公式等**区域；
|
||||
- 支持通用的中英文**表格检测**任务；
|
||||
- 支持表格区域进行结构化识别，最终结果输出**Excel文件**；
|
||||
- 支持基于多模态的关键信息抽取(Key Information Extraction，KIE)任务-**语义实体识别**(Semantic Entity Recognition，SER)和**关系抽取**(Relation Extraction，RE)；
|
||||
- 支持**版面复原**，即恢复为与原始图像布局一致的word或者pdf格式的文件；
|
||||
- 支持自定义训练及python whl包调用等多种推理部署方式，简单易用；
|
||||
- 与半自动数据标注工具PPOCRLabel打通，支持版面分析、表格识别、SER三种任务的标注。
|
||||
|
||||
## 3. 效果展示
|
||||
|
||||
PP-StructureV2支持各个模块独立使用或灵活搭配，如，可以单独使用版面分析，或单独使用表格识别，这里仅展示几种代表性使用方式的可视化效果。
|
||||
|
||||
### 3.1 版面分析和表格识别
|
||||
|
||||
下图展示了版面分析+表格识别的整体流程，图片先有版面分析划分为图像、文本、标题和表格四种区域，然后对图像、文本和标题三种区域进行OCR的检测识别，对表格进行表格识别，其中图像还会被存储下来以便使用。
|
||||
|
||||
|
||||
|
||||
### 3.1.1 版面识别返回单字坐标
|
||||
|
||||
下图展示了基于上一节版面分析对文字进行定位的效果， 可参考[文档](blog/return_word_pos.md)。
|
||||
|
||||
|
||||
|
||||
### 3.2 版面恢复
|
||||
|
||||
下图展示了基于上一节版面分析和表格识别的结果进行版面恢复的效果。
|
||||
|
||||
|
||||
|
||||
### 3.3 关键信息抽取
|
||||
|
||||
- SER
|
||||
|
||||
图中不同颜色的框表示不同的类别。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
- RE
|
||||
|
||||
图中红色框表示`问题`，蓝色框表示`答案`，`问题`和`答案`之间使用绿色线连接。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 4. 快速体验
|
||||
|
||||
请参考[快速使用](./quick_start.md)教程。
|
||||
|
||||
## 5. 模型库
|
||||
|
||||
部分任务需要同时用到结构化分析模型和OCR模型，如表格识别需要使用表格识别模型进行结构化解析，同时也要用到OCR模型对表格内的文字进行识别，请根据具体需求选择合适的模型。
|
||||
|
||||
结构化分析相关模型下载可以参考：
|
||||
|
||||
- [PP-Structure 模型库](./models_list.md)
|
||||
|
||||
OCR相关模型下载可以参考：
|
||||
|
||||
- [PP-OCR 模型库](../ppocr/model_list.md)
|
||||
60
docs/version2.x/ppstructure/ppstructure_model.md
Normal file
60
docs/version2.x/ppstructure/ppstructure_model.md
Normal file
@@ -0,0 +1,60 @@
|
||||
---
|
||||
comments: true
|
||||
hide:
|
||||
- toc
|
||||
---
|
||||
|
||||
|
||||
# PP-Structure 系列模型列表
|
||||
|
||||
## 1. 版面分析模型
|
||||
|
||||
| 模型名称 | 模型简介 | 推理模型大小 | 下载地址 | dict path |
|
||||
| --------------------------------------- | ----- | ------------ | ------------------- | ------ |
|
||||
| picodet_lcnet_x1_0_fgd_layout | 基于PicoDet LCNet_x1_0和FGD蒸馏在PubLayNet 数据集训练的英文版面分析模型，可以划分**文字、标题、表格、图片以及列表**5类区域 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout.pdparams) | [PubLayNet dict](../../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt) |
|
||||
| ppyolov2_r50vd_dcn_365e_publaynet | 基于PP-YOLOv2在PubLayNet数据集上训练的英文版面分析模型 | 221.0M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) / [训练模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet_pretrained.pdparams) | 同上 |
|
||||
| picodet_lcnet_x1_0_fgd_layout_cdla | CDLA数据集训练的中文版面分析模型，可以划分为**表格、图片、图片标题、表格、表格标题、页眉、脚本、引用、公式**10类区域 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_cdla.pdparams) | [CDLA dict](../../ppocr/utils/dict/layout_dict/layout_cdla_dict.txt) |
|
||||
| picodet_lcnet_x1_0_fgd_layout_table | 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测 | 9.7M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/layout/picodet_lcnet_x1_0_fgd_layout_table.pdparams) | [Table dict](../../ppocr/utils/dict/layout_dict/layout_table_dict.txt) |
|
||||
| ppyolov2_r50vd_dcn_365e_tableBank_word | 基于PP-YOLOv2在TableBank Word 数据集训练的版面分析模型，支持英文文档表格区域的检测 | 221.0M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) | 同上 |
|
||||
| ppyolov2_r50vd_dcn_365e_tableBank_latex | 基于PP-YOLOv2在TableBank Latex数据集训练的版面分析模型，支持英文文档表格区域的检测 | 221.0M | [推理模型](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) | 同上 |
|
||||
|
||||
## 2. OCR和表格识别模型
|
||||
|
||||
### 2.1 OCR
|
||||
|
||||
| 模型名称 | 模型简介 | 推理模型大小 | 下载地址 |
|
||||
| ------------------------------ | ------------------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| en_ppocr_mobile_v2.0_table_det | PubTabNet数据集训练的英文表格场景的文字检测 | 4.7M | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_det_train.tar) |
|
||||
| en_ppocr_mobile_v2.0_table_rec | PubTabNet数据集训练的英文表格场景的文字识别 | 6.9M | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_rec_train.tar) |
|
||||
|
||||
如需要使用其他OCR模型，可以在 [PP-OCR model_list](../ppocr/model_list.md) 下载模型或者使用自己训练好的模型配置到 `det_model_dir`, `rec_model_dir`两个字段即可。
|
||||
|
||||
### 2.2 表格识别模型
|
||||
|
||||
| 模型名称 | 模型简介 | 推理模型大小 | 下载地址 |
|
||||
| ------------------------------------ | ---------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| en_ppocr_mobile_v2.0_table_structure | 基于TableRec-RARE在PubTabNet数据集上训练的英文表格识别模型 | 6.8M | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
|
||||
| en_ppstructure_mobile_v2.0_SLANet | 基于SLANet在PubTabNet数据集上训练的英文表格识别模型 | 9.2M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/en_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/en_ppstructure_mobile_v2.0_SLANet_train.tar) |
|
||||
| ch_ppstructure_mobile_v2.0_SLANet | 基于SLANet的中文表格识别模型 | 9.3M | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/paddle3.0b2/ch_ppstructure_mobile_v2.0_SLANet_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/slanet/ch_ppstructure_mobile_v2.0_SLANet_train.tar) |
|
||||
|
||||
## 3. KIE模型
|
||||
|
||||
在XFUND_zh数据集上，不同模型的精度与V100 GPU上速度信息如下所示。
|
||||
|
||||
| 模型名称 | 模型简介 | 推理模型大小 | 精度(hmean) | 预测耗时(ms) | 下载地址 |
|
||||
| ------------------------- | ------------------------------------------------ | ------------ | ----------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| ser_VI-LayoutXLM_xfund_zh | 基于VI-LayoutXLM在xfund中文数据集上训练的SER模型 | 1.1G | 93.19% | 15.49 | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/ser_vi_layoutxlm_xfund_pretrained.tar) |
|
||||
| re_VI-LayoutXLM_xfund_zh | 基于VI-LayoutXLM在xfund中文数据集上训练的RE模型 | 1.1G | 83.92% | 15.49 | [推理模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/ppstructure/models/vi_layoutxlm/re_vi_layoutxlm_xfund_pretrained.tar) |
|
||||
| ser_LayoutXLM_xfund_zh | 基于LayoutXLM在xfund中文数据集上训练的SER模型 | 1.4G | 90.38% | 19.49 | [推理模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar) |
|
||||
| re_LayoutXLM_xfund_zh | 基于LayoutXLM在xfund中文数据集上训练的RE模型 | 1.4G | 74.83% | 19.49 | [推理模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar) |
|
||||
| ser_LayoutLMv2_xfund_zh | 基于LayoutLMv2在xfund中文数据集上训练的SER模型 | 778.0M | 85.44% | 31.46 | [推理模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar) |
|
||||
| re_LayoutLMv2_xfund_zh | 基于LayoutLMv2在xfun中文数据集上训练的RE模型 | 765.0M | 67.77% | 31.46 | [推理模型 coming soon]() / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar) |
|
||||
| ser_LayoutLM_xfund_zh | 基于LayoutLM在xfund中文数据集上训练的SER模型 | 430.0M | 77.31% | - | [推理模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar) |
|
||||
|
||||
* 注：上述预测耗时信息仅包含了inference模型的推理耗时，没有统计预处理与后处理耗时，测试环境为`V100 GPU + CUDA 10.2 + CUDNN 8.1.1 + TRT 7.2.3.4`。
|
||||
|
||||
在wildreceipt数据集上，SDMGR模型精度与下载地址如下所示。
|
||||
|
||||
| 模型名称 | 模型简介 | 模型大小 | 精度 | 下载地址 |
|
||||
| -------- | ---------------- | -------- | ------ | ----------------------------------------------------------------------------------------------------- |
|
||||
| SDMGR | 关键信息提取模型 | 78.0M | 86.70% | [推理模型 coming soon]() / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/kie/kie_vgg16.tar) |
|
||||
395
docs/version2.x/ppstructure/quick_start.en.md
Normal file
395
docs/version2.x/ppstructure/quick_start.en.md
Normal file
@@ -0,0 +1,395 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-Structure Quick Start
|
||||
|
||||
## 1. Environment Preparation
|
||||
|
||||
### 1.1 Install PaddlePaddle
|
||||
|
||||
> If you do not have a Python environment, please refer to [Environment Preparation](../ppocr/environment.en.md).
|
||||
|
||||
- PaddlePaddle with CUDA 11.8
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install "paddlepaddle-gpu<=2.6" -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
|
||||
```
|
||||
|
||||
- PaddlePaddle with CUDA 12.3
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install "paddlepaddle-gpu<=2.6" -i https://www.paddlepaddle.org.cn/packages/stable/cu123/
|
||||
```
|
||||
|
||||
- If your machine does not have an available GPU, please run the following command to install the CPU version
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install "paddlepaddle<=2.6" -i https://www.paddlepaddle.org.cn/packages/stable/cpu/
|
||||
```
|
||||
|
||||
For more software version requirements, please refer to the instructions in the [Installation Document](https://www.paddlepaddle.org.cn/en/install/quick).
|
||||
|
||||
### 1.2 Install PaddleOCR Whl Package
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install "paddleocr<3.0"
|
||||
|
||||
# Install the image direction classification dependency package paddleclas (if you do not use the image direction classification, you can skip it)
|
||||
python3 -m pip install paddleclas
|
||||
```
|
||||
|
||||
## 2. Quick Use
|
||||
|
||||
### 2.1 Use by command line
|
||||
|
||||
#### 2.1.1 image orientation + layout analysis + table recognition
|
||||
|
||||
```bash linenums="1"
|
||||
# Temporarily disable the new IR feature
|
||||
export FLAGS_enable_pir_api=0
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --image_orientation=true
|
||||
```
|
||||
|
||||
#### 2.1.2 layout analysis + table recognition
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure
|
||||
```
|
||||
|
||||
#### 2.1.3 layout analysis
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --table=false --ocr=false
|
||||
```
|
||||
|
||||
#### 2.1.4 table recognition
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/table/table.jpg --type=structure --layout=false
|
||||
```
|
||||
|
||||
#### 2.1.5 Key Information Extraction
|
||||
|
||||
Key information extraction does not currently support use by the whl package. For detailed usage tutorials, please refer to: [Key Information Extraction](../ppocr/model_train/kie.en.md).
|
||||
|
||||
#### 2.1.6 layout recovery
|
||||
|
||||
Two layout recovery methods are provided, For detailed usage tutorials, please refer to: [Layout Recovery](./model_train/recovery_to_doc.en.md).
|
||||
|
||||
- PDF parse
|
||||
- OCR
|
||||
|
||||
Recovery by using PDF parse (only support pdf as input):
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --use_pdf2docx_api=true
|
||||
```
|
||||
|
||||
Recovery by using OCR：
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --recovery=true --lang='en'
|
||||
```
|
||||
|
||||
#### 2.1.7 layout recovery(PDF to Markdown)
|
||||
|
||||
Do not use LaTeXCOR model for formula recognition：
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --recovery_to_markdown=true --lang='en'
|
||||
```
|
||||
|
||||
Use LaTeXCOR model for formula recognition, where Chinese layout model must be used：
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --formula=true --recovery_to_markdown=true --lang='ch'
|
||||
```
|
||||
|
||||
### 2.2 Use by python script
|
||||
|
||||
#### 2.2.1 image orientation + layout analysis + table recognition
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,draw_structure_result,save_structure_res
|
||||
|
||||
table_engine = PPStructure(show_log=True, image_orientation=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder,os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
|
||||
from PIL import Image
|
||||
|
||||
font_path = 'doc/fonts/simfang.ttf' # PaddleOCR下提供字体包
|
||||
image = Image.open(img_path).convert('RGB')
|
||||
im_show = draw_structure_result(image, result,font_path=font_path)
|
||||
im_show = Image.fromarray(im_show)
|
||||
im_show.save('result.jpg')
|
||||
```
|
||||
|
||||
#### 2.2.2 layout analysis + table recognition
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,draw_structure_result,save_structure_res
|
||||
|
||||
table_engine = PPStructure(show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder,os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
|
||||
from PIL import Image
|
||||
|
||||
font_path = 'doc/fonts/simfang.ttf' # font provided in PaddleOCR
|
||||
image = Image.open(img_path).convert('RGB')
|
||||
im_show = draw_structure_result(image, result,font_path=font_path)
|
||||
im_show = Image.fromarray(im_show)
|
||||
im_show.save('result.jpg')
|
||||
```
|
||||
|
||||
#### 2.2.3 layout analysis
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
|
||||
table_engine = PPStructure(table=False, ocr=False, show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
```
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
|
||||
ocr_engine = PPStructure(table=False, ocr=True, show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/recovery/UnrealText.pdf'
|
||||
result = ocr_engine(img_path)
|
||||
for index, res in enumerate(result):
|
||||
save_structure_res(res, save_folder, os.path.basename(img_path).split('.')[0], index)
|
||||
|
||||
for res in result:
|
||||
for line in res:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
```
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
import numpy as np
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
from paddle.utils import try_import
|
||||
from PIL import Image
|
||||
|
||||
ocr_engine = PPStructure(table=False, ocr=True, show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/recovery/UnrealText.pdf'
|
||||
|
||||
fitz = try_import("fitz")
|
||||
imgs = []
|
||||
with fitz.open(img_path) as pdf:
|
||||
for pg in range(0, pdf.page_count):
|
||||
page = pdf[pg]
|
||||
mat = fitz.Matrix(2, 2)
|
||||
pm = page.get_pixmap(matrix=mat, alpha=False)
|
||||
|
||||
# if width or height > 2000 pixels, don't enlarge the image
|
||||
if pm.width > 2000 or pm.height > 2000:
|
||||
pm = page.get_pixmap(matrix=fitz.Matrix(1, 1), alpha=False)
|
||||
|
||||
img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples)
|
||||
img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
|
||||
imgs.append(img)
|
||||
|
||||
for index, img in enumerate(imgs):
|
||||
result = ocr_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0], index)
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
```
|
||||
|
||||
#### 2.2.4 table recognition
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
|
||||
table_engine = PPStructure(layout=False, show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/table.jpg'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
```
|
||||
|
||||
#### 2.2.5 Key Information Extraction
|
||||
|
||||
Key information extraction does not currently support use by the whl package. For detailed usage tutorials, please refer to: [Inference](../infer_deploy/python_infer.en.md).
|
||||
|
||||
#### 2.2.6 layout recovery
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
from paddleocr.ppstructure.recovery.recovery_to_doc import sorted_layout_boxes, convert_info_docx
|
||||
|
||||
# Chinese image
|
||||
table_engine = PPStructure(recovery=True)
|
||||
# English image
|
||||
# table_engine = PPStructure(recovery=True, lang='en')
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
|
||||
h, w, _ = img.shape
|
||||
res = sorted_layout_boxes(result, w)
|
||||
convert_info_docx(img, res, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
```
|
||||
|
||||
#### 2.2.7 layout recovery(PDF to Markdown)
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
from paddleocr.ppstructure.recovery.recovery_to_doc import sorted_layout_boxes
|
||||
from paddleocr.ppstructure.recovery.recovery_to_markdown import convert_info_markdown
|
||||
|
||||
# Chinese image
|
||||
table_engine = PPStructure(recovery=True)
|
||||
# English image
|
||||
# table_engine = PPStructure(recovery=True, lang='en')
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
|
||||
h, w, _ = img.shape
|
||||
res = sorted_layout_boxes(result, w)
|
||||
convert_info_markdown(res, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
```
|
||||
|
||||
### 2.3 Result description
|
||||
|
||||
The return of PP-Structure is a list of dicts, the example is as follows:
|
||||
|
||||
#### 2.3.1 layout analysis + table recognition
|
||||
|
||||
```bash linenums="1"
|
||||
[
|
||||
{ 'type': 'Text',
|
||||
'bbox': [34, 432, 345, 462],
|
||||
'res': ([[36.0, 437.0, 341.0, 437.0, 341.0, 446.0, 36.0, 447.0], [41.0, 454.0, 125.0, 453.0, 125.0, 459.0, 41.0, 460.0]],
|
||||
[('Tigure-6. The performance of CNN and IPT models using difforen', 0.90060663), ('Tent ', 0.465441)])
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Each field in dict is described as follows:
|
||||
|
||||
| field | description |
|
||||
| --- |---|
|
||||
|type| Type of image area. |
|
||||
|bbox| The coordinates of the image area in the original image, respectively [upper left corner x, upper left corner y, lower right corner x, lower right corner y]. |
|
||||
|res| OCR or table recognition result of the image area. <br> table: a dict with field descriptions as follows: <br>        `html`: html str of table.<br>        In the code usage mode, set return_ocr_result_in_table=True whrn call can get the detection and recognition results of each text in the table area, corresponding to the following fields: <br>        `boxes`: text detection boxes.<br>        `rec_res`: text recognition results.<br> OCR: A tuple containing the detection boxes and recognition results of each single text. |
|
||||
|
||||
After the recognition is completed, each image will have a directory with the same name under the directory specified by the `output` field. Each table in the image will be stored as an excel, and the picture area will be cropped and saved. The filename of excel and picture is their coordinates in the image.
|
||||
|
||||
```text linenums="1"
|
||||
/output/table/1/
|
||||
└─ res.txt
|
||||
└─ [454, 360, 824, 658].xlsx table recognition result
|
||||
└─ [16, 2, 828, 305].jpg picture in Image
|
||||
└─ [17, 361, 404, 711].xlsx table recognition result
|
||||
```
|
||||
|
||||
#### 2.3.2 Key Information Extraction
|
||||
|
||||
Please refer to: [Key Information Extraction](../ppocr/model_train/kie.en.md) .
|
||||
|
||||
### 2.4 Parameter Description
|
||||
|
||||
| field | description | default |
|
||||
|-------------------------|----------------------------------------------------------------------------------------------------------------------------|---|
|
||||
| output | result save path | ./output/table |
|
||||
| table_max_len | long side of the image resize in table structure model | 488 |
|
||||
| table_model_dir | Table structure model inference model path | None |
|
||||
| table_char_dict_path | The dictionary path of table structure model | ../ppocr/utils/dict/table_structure_dict.txt |
|
||||
| merge_no_span_structure | In the table recognition model, whether to merge '\<td>' and '\</td>' | False |
|
||||
| formula_model_dir | Formula recognition model inference model path | None |
|
||||
| formula_char_dict_path | The dictionary path of formula recognition model | ../ppocr/utils/dict/latex_ocr_tokenizer.json |
|
||||
| layout_model_dir | Layout analysis model inference model path | None |
|
||||
| layout_dict_path | The dictionary path of layout analysis model | ../ppocr/utils/dict/layout_publaynet_dict.txt |
|
||||
| layout_score_threshold | The box threshold path of layout analysis model | 0.5|
|
||||
| layout_nms_threshold | The nms threshold path of layout analysis model | 0.5|
|
||||
| kie_algorithm | kie model algorithm | LayoutXLM|
|
||||
| ser_model_dir | Ser model inference model path | None|
|
||||
| ser_dict_path | The dictionary path of Ser model | ../train_data/XFUND/class_list_xfun.txt|
|
||||
| mode | structure or kie | structure |
|
||||
| image_orientation | Whether to perform image orientation classification in forward | False |
|
||||
| layout | Whether to perform layout analysis in forward | True |
|
||||
| table | Whether to perform table recognition in forward | True |
|
||||
| formula | Whether to perform formula recognition in forward | False |
|
||||
| ocr | Whether to perform ocr for non-table areas in layout analysis. When layout is False, it will be automatically set to False | True |
|
||||
| recovery | Whether to perform layout recovery in forward | False |
|
||||
| recovery_to_markdown | Whether to convert the layout recovery results into a markdown file | False |
|
||||
| save_pdf | Whether to convert docx to pdf when recovery | False |
|
||||
| structure_version | Structure version, optional PP-structure and PP-structurev2 | PP-structure |
|
||||
|
||||
Most of the parameters are consistent with the PaddleOCR whl package, see [whl package documentation](../ppocr/blog/whl.en.md)
|
||||
|
||||
## 3. Summary
|
||||
|
||||
Through the content in this section, you can master the use of PP-Structure related functions through PaddleOCR whl package. Please refer to [documentation tutorial](../index.en.md) for more detailed usage tutorials including model training, inference and deployment, etc.
|
||||
400
docs/version2.x/ppstructure/quick_start.md
Normal file
400
docs/version2.x/ppstructure/quick_start.md
Normal file
@@ -0,0 +1,400 @@
|
||||
---
|
||||
comments: true
|
||||
---
|
||||
|
||||
# PP-Structure 快速开始
|
||||
|
||||
## 1. 准备环境
|
||||
|
||||
### 1.1 安装PaddlePaddle
|
||||
>
|
||||
> 如果您没有基础的Python运行环境，请参考[运行环境准备](../ppocr/environment.md)。
|
||||
|
||||
- CUDA11.8 的 PaddlePaddle
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install "paddlepaddle-gpu<=2.6" -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
|
||||
```
|
||||
|
||||
- CUDA12.3 的 PaddlePaddle
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install "paddlepaddle-gpu<=2.6" -i https://www.paddlepaddle.org.cn/packages/stable/cu123/
|
||||
```
|
||||
|
||||
- 您的机器是CPU，请运行以下命令安装
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install "paddlepaddle<=2.6" -i https://www.paddlepaddle.org.cn/packages/stable/cpu/
|
||||
```
|
||||
|
||||
更多的版本需求，请参照[飞桨官网安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。
|
||||
|
||||
### 1.2 安装PaddleOCR whl包
|
||||
|
||||
```bash linenums="1"
|
||||
python3 -m pip install "paddleocr<3.0"
|
||||
|
||||
# 安装 图像方向分类依赖包paddleclas（如不需要图像方向分类功能，可跳过）
|
||||
python3 -m pip install paddleclas
|
||||
```
|
||||
|
||||
## 2. 便捷使用
|
||||
|
||||
### 2.1 命令行使用
|
||||
|
||||
#### 2.1.1 图像方向分类+版面分析+表格识别
|
||||
|
||||
```bash linenums="1"
|
||||
# 暂时关闭新 IR 功能
|
||||
export FLAGS_enable_pir_api=0
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --image_orientation=true
|
||||
```
|
||||
|
||||
#### 2.1.2 版面分析+表格识别
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure
|
||||
```
|
||||
|
||||
#### 2.1.3 版面分析
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --table=false --ocr=false
|
||||
```
|
||||
|
||||
#### 2.1.4 表格识别
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/table/table.jpg --type=structure --layout=false
|
||||
```
|
||||
|
||||
#### 2.1.5 关键信息抽取
|
||||
|
||||
关键信息抽取暂不支持通过whl包调用，详细使用教程请参考：[关键信息抽取教程](../ppocr/model_train/kie.md)。
|
||||
|
||||
#### 2.1.6 版面恢复
|
||||
|
||||
版面恢复分为2种方法，详细介绍请参考：[版面恢复教程](./model_train/recovery_to_doc.md)：
|
||||
|
||||
- PDF解析
|
||||
- OCR技术
|
||||
|
||||
通过PDF解析(只支持pdf格式的输入)：
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --use_pdf2docx_api=true
|
||||
```
|
||||
|
||||
通过OCR技术：
|
||||
|
||||
```bash linenums="1"
|
||||
# 中文测试图
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --recovery=true
|
||||
# 英文测试图
|
||||
paddleocr --image_dir=ppstructure/docs/table/1.png --type=structure --recovery=true --lang='en'
|
||||
# pdf测试文件
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --lang='en'
|
||||
```
|
||||
|
||||
#### 2.1.7 版面恢复+转换为markdown文件
|
||||
|
||||
不使用LaTeXOCR模型进行公式识别：
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --recovery_to_markdown=true --lang='en'
|
||||
```
|
||||
|
||||
使用LaTeXOCR模型进行公式识别，其中必须使用中文layout模型：
|
||||
|
||||
```bash linenums="1"
|
||||
paddleocr --image_dir=ppstructure/docs/recovery/UnrealText.pdf --type=structure --recovery=true --formula=true --recovery_to_markdown=true --lang='ch'
|
||||
```
|
||||
|
||||
### 2.2 Python脚本使用
|
||||
|
||||
#### 2.2.1 图像方向分类+版面分析+表格识别
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,draw_structure_result,save_structure_res
|
||||
|
||||
table_engine = PPStructure(show_log=True, image_orientation=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder,os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
|
||||
from PIL import Image
|
||||
|
||||
font_path = 'doc/fonts/simfang.ttf' # PaddleOCR下提供字体包
|
||||
image = Image.open(img_path).convert('RGB')
|
||||
im_show = draw_structure_result(image, result,font_path=font_path)
|
||||
im_show = Image.fromarray(im_show)
|
||||
im_show.save('result.jpg')
|
||||
```
|
||||
|
||||
#### 2.2.2 版面分析+表格识别
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,draw_structure_result,save_structure_res
|
||||
|
||||
table_engine = PPStructure(show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder,os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
|
||||
from PIL import Image
|
||||
|
||||
font_path = 'doc/fonts/simfang.ttf' # PaddleOCR下提供字体包
|
||||
image = Image.open(img_path).convert('RGB')
|
||||
im_show = draw_structure_result(image, result,font_path=font_path)
|
||||
im_show = Image.fromarray(im_show)
|
||||
im_show.save('result.jpg')
|
||||
```
|
||||
|
||||
#### 2.2.3 版面分析
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
|
||||
table_engine = PPStructure(table=False, ocr=False, show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
```
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
|
||||
ocr_engine = PPStructure(table=False, ocr=True, show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/recovery/UnrealText.pdf'
|
||||
result = ocr_engine(img_path)
|
||||
for index, res in enumerate(result):
|
||||
save_structure_res(res, save_folder, os.path.basename(img_path).split('.')[0], index)
|
||||
|
||||
for res in result:
|
||||
for line in res:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
```
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
import numpy as np
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
from paddle.utils import try_import
|
||||
from PIL import Image
|
||||
|
||||
ocr_engine = PPStructure(table=False, ocr=True, show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/recovery/UnrealText.pdf'
|
||||
|
||||
fitz = try_import("fitz")
|
||||
imgs = []
|
||||
with fitz.open(img_path) as pdf:
|
||||
for pg in range(0, pdf.page_count):
|
||||
page = pdf[pg]
|
||||
mat = fitz.Matrix(2, 2)
|
||||
pm = page.get_pixmap(matrix=mat, alpha=False)
|
||||
|
||||
# if width or height > 2000 pixels, don't enlarge the image
|
||||
if pm.width > 2000 or pm.height > 2000:
|
||||
pm = page.get_pixmap(matrix=fitz.Matrix(1, 1), alpha=False)
|
||||
|
||||
img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples)
|
||||
img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
|
||||
imgs.append(img)
|
||||
|
||||
for index, img in enumerate(imgs):
|
||||
result = ocr_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0], index)
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
```
|
||||
|
||||
#### 2.2.4 表格识别
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
|
||||
table_engine = PPStructure(layout=False, show_log=True)
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/table.jpg'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
```
|
||||
|
||||
#### 2.2.5 关键信息抽取
|
||||
|
||||
关键信息抽取暂不支持通过whl包调用，详细使用教程请参考：[inference文档](./infer_deploy/python_infer.md)。
|
||||
|
||||
#### 2.2.6 版面恢复
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
from paddleocr.ppstructure.recovery.recovery_to_doc import sorted_layout_boxes, convert_info_docx
|
||||
|
||||
# 中文测试图
|
||||
table_engine = PPStructure(recovery=True)
|
||||
# 英文测试图
|
||||
# table_engine = PPStructure(recovery=True, lang='en')
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
|
||||
h, w, _ = img.shape
|
||||
res = sorted_layout_boxes(result, w)
|
||||
convert_info_docx(img, res, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
```
|
||||
|
||||
#### 2.2.7 版面恢复+转换为markdown文件
|
||||
|
||||
```python linenums="1"
|
||||
import os
|
||||
import cv2
|
||||
from paddleocr import PPStructure,save_structure_res
|
||||
from paddleocr.ppstructure.recovery.recovery_to_doc import sorted_layout_boxes
|
||||
from paddleocr.ppstructure.recovery.recovery_to_markdown import convert_info_markdown
|
||||
|
||||
# 中文测试图
|
||||
table_engine = PPStructure(recovery=True)
|
||||
# 英文测试图
|
||||
# table_engine = PPStructure(recovery=True, lang='en')
|
||||
|
||||
save_folder = './output'
|
||||
img_path = 'ppstructure/docs/table/1.png'
|
||||
img = cv2.imread(img_path)
|
||||
result = table_engine(img)
|
||||
save_structure_res(result, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
|
||||
for line in result:
|
||||
line.pop('img')
|
||||
print(line)
|
||||
|
||||
h, w, _ = img.shape
|
||||
res = sorted_layout_boxes(result, w)
|
||||
convert_info_markdown(res, save_folder, os.path.basename(img_path).split('.')[0])
|
||||
```
|
||||
|
||||
### 2.3 返回结果说明
|
||||
|
||||
PP-Structure的返回结果为一个dict组成的list，示例如下：
|
||||
|
||||
#### 2.3.1 版面分析+表格识别
|
||||
|
||||
```bash linenums="1"
|
||||
[
|
||||
{ 'type': 'Text',
|
||||
'bbox': [34, 432, 345, 462],
|
||||
'res': ([[36.0, 437.0, 341.0, 437.0, 341.0, 446.0, 36.0, 447.0], [41.0, 454.0, 125.0, 453.0, 125.0, 459.0, 41.0, 460.0]],
|
||||
[('Tigure-6. The performance of CNN and IPT models using difforen', 0.90060663), ('Tent ', 0.465441)])
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
dict 里各个字段说明如下：
|
||||
|
||||
| 字段 | 说明 |
|
||||
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| type | 图片区域的类型 |
|
||||
| bbox | 图片区域的在原图的坐标，分别[左上角x，左上角y，右下角x，右下角y] |
|
||||
| res | 图片区域的OCR或表格识别结果。<br> 表格: 一个dict，字段说明如下<br>        `html`: 表格的HTML字符串<br>        在代码使用模式下，前向传入return_ocr_result_in_table=True可以拿到表格中每个文本的检测识别结果，对应为如下字段: <br>        `boxes`: 文本检测坐标<br>        `rec_res`: 文本识别结果。<br> OCR: 一个包含各个单行文字的检测坐标和识别结果的元组 |
|
||||
|
||||
运行完成后，每张图片会在`output`字段指定的目录下有一个同名目录，图片里的每个表格会存储为一个excel，图片区域会被裁剪之后保存下来，excel文件和图片名为表格在图片里的坐标。
|
||||
|
||||
```
|
||||
/output/table/1/
|
||||
└─ res.txt
|
||||
└─ [454, 360, 824, 658].xlsx 表格识别结果
|
||||
└─ [16, 2, 828, 305].jpg 被裁剪出的图片区域
|
||||
└─ [17, 361, 404, 711].xlsx 表格识别结果
|
||||
```
|
||||
|
||||
#### 2.3.2 关键信息抽取
|
||||
|
||||
请参考：[关键信息抽取教程](../ppocr/model_train/kie.md)。
|
||||
|
||||
### 2.4 参数说明
|
||||
|
||||
| 字段 | 说明 | 默认值 |
|
||||
|-------------------------|-------------------------------------------------| ------ |
|
||||
| output | 结果保存地址 | ./output/table |
|
||||
| table_max_len | 表格结构模型预测时，图像的长边resize尺度 | 488 |
|
||||
| table_model_dir | 表格结构模型 inference 模型地址 | None |
|
||||
| table_char_dict_path | 表格结构模型所用字典地址 | ../ppocr/utils/dict/table_structure_dict.txt |
|
||||
| merge_no_span_structure | 表格识别模型中，是否对'\<td>'和'\</td>' 进行合并 | False |
|
||||
| formula_model_dir | 公式识别模型 inference 模型地址 | None |
|
||||
| formula_char_dict_path | 公式识别模型所用字典地址 | ../ppocr/utils/dict/latex_ocr_tokenizer.json |
|
||||
| layout_model_dir | 版面分析模型 inference 模型地址 | None |
|
||||
| layout_dict_path | 版面分析模型字典 | ../ppocr/utils/dict/layout_publaynet_dict.txt |
|
||||
| layout_score_threshold | 版面分析模型检测框阈值 | 0.5 |
|
||||
| layout_nms_threshold | 版面分析模型nms阈值 | 0.5 |
|
||||
| kie_algorithm | kie模型算法 | LayoutXLM |
|
||||
| ser_model_dir | ser模型 inference 模型地址 | None |
|
||||
| ser_dict_path | ser模型字典 | ../train_data/XFUND/class_list_xfun.txt |
|
||||
| mode | structure or kie | structure |
|
||||
| image_orientation | 前向中是否执行图像方向分类 | False |
|
||||
| layout | 前向中是否执行版面分析 | True |
|
||||
| table | 前向中是否执行表格识别 | True |
|
||||
| formula | 前向中是否执行公式识别 | False |
|
||||
| ocr | 对于版面分析中的非表格区域，是否执行ocr。当layout为False时会被自动设置为False | True |
|
||||
| recovery | 前向中是否执行版面恢复 | False |
|
||||
| recovery_to_markdown | 是否将版面恢复结果转换为markdown文件 | False |
|
||||
| save_pdf | 版面恢复导出docx文件的同时，是否导出pdf文件 | False |
|
||||
| structure_version | 模型版本，可选 PP-structure和PP-structurev2 | PP-structure |
|
||||
|
||||
大部分参数和PaddleOCR whl包保持一致，见 [whl包文档](../ppocr/blog/whl.md)
|
||||
|
||||
## 3. 小结
|
||||
|
||||
通过本节内容，相信您已经熟练掌握通过PaddleOCR whl包调用PP-Structure相关功能的使用方法，您可以参考[文档教程](../index.md)，获取包括模型训练、推理部署等更详细的使用教程。
|
||||
Reference in New Issue
Block a user