first commit

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.trie filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,12 +1,165 @@
----
-title: ReadReview
-emoji: 📊
-colorFrom: red
-colorTo: gray
-sdk: gradio
-sdk_version: 4.32.0
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# MedicalReviewAgent 不想看文献
+## 项目概述
+- 整一个帮我写综述的Agent，希望他能完成文献内容的收集，文本分类和总结，科学事实对比，撰写综述等功能
+- 计划用到RAG, function calling等技术
+- 还在不断摸索中，欢迎大佬指导！
+
+## 流程图
+基本上就是在上海AIlab的茴香豆上面改的 这里主要讲解使用流程 架构和茴香豆一样 [茴香豆架构](https://github.com/InternLM/HuixiangDou/blob/main/docs/architecture_zh.md)
+### 文献库和知识库构建
+![image](https://github.com/jabberwockyang/MedicalReviewAgent/assets/52541128/d70a2ec1-7a20-4b5b-a91c-bf649f657319)
+
+### 人机合作写文章
+<img width="847" alt="image" src="https://github.com/jabberwockyang/MedicalReviewAgent/assets/52541128/fc394d8b-1668-4349-9adc-1c4c0a7e0a8b">
+
+
+## 功能
+
+1. **模型服务配置**
+   - **远程模型选择**：允许用户选择使用远程大模型或本地模型。提供多种大模型提供商选择，如kimi、deepseek、zhipuai、gpt。
+
+2. **文献查找+数据库生成**
+   - **文献查找**：
+     - 用户可以输入感兴趣的关键词，设置查找数量，并进行PubMed PMC文献查找。
+     - 支持用户上传已有的PDF文献文件，可处理复杂PDF结构。
+   - **文献库管理**：
+     - 支持删除现有文献库。
+     - 提供文献库概况的实时更新。
+   - **数据库生成**：
+     - 用户可以设置块大小用于构建数据库
+     - 用户可以设置聚类数量用于文本聚类
+   - **数据库管理**：      
+     - 支持生成新的数据库，删除现有数据库，并查看数据库概况。
+
+3. **写综述**
+   - **抽样标注文章聚类**：
+     - 用户可以选择特定的块大小和聚类数，设置抽样标注比例，并开始标注过程。
+   - **获取灵感**：
+     - 基于标注的文章聚类，大模型提供灵感，帮助用户生成综述所需的问题框架。
+   - **综述生成**：
+     - 用户可以输入想写的内容或主题，点击生成综述按钮，系统将自动生成综述文本并提供参考文献。
+
+## 亮点
+
+1. **高效的文献查找和管理**
+   - 通过关键词快速查找相关文献，支持上传已有PDF文献，方便文献库的构建和管理。
+
+2. **灵活的数据库生成**
+   - 提供灵活的数据库生成参数设置，支持多次生成和更新数据库，保证数据的及时性和准确性。
+
+3. **智能的综述生成**
+   - 基于先进的大模型技术，提供自动化的文章聚类标注和灵感生成功能，帮助用户快速生成高质量的综述文本。
+
+4. **用户友好界面**
+   - 直观的操作界面和详细的使用指导，让用户能够轻松上手和使用各项功能。
+
+5. **远程和本地模型支持**
+   - 支持多种大模型提供商的选择，满足不同用户的需求，无论是本地模型还是远程大模型，都能灵活配置和使用。
+
+## 安装运行
+   新建conda环境
+
+   ```bash
+conda create --name ReviewAgent python=3.10.14
+conda activate ReviewAgent
+   ```
+   拉取github仓库 
+
+   ```bash
+git clone https://github.com/jabberwockyang/MedicalReviewAgent.git
+cd MedicalReviewAgent
+pip install -r requirements.txt
+   ```
+   huggingface-cli下载模型
+
+   ```bash
+cd /root && mkdir models
+cd /root/models
+# login required
+huggingface-cli download Qwen/Qwen1.5-7B-Chat --local-dir /root/models/Qwen1.5-7B-Chat
+huggingface-cli download maidalun1020/bce-embedding-base_v1 --local-dir /root/models/bce-embedding-base_v1
+huggingface-cli download maidalun1020/bce-reranker-base_v1 --local-dir /root/models/bce-reranker-base_v1
+   ```
+   启动服务
+
+   ```bash
+conda activate ReviewAgent
+cd MedicalReviewAgent
+python3 app.py
+   ```
+   gradio在本地7860端口运行
+
+## 技术要点
+
+基于茴香豆加了几个功能
+
+1. 本地和远程模型灵活配置
+   - 本地模型是qwen1.5 7b chat
+   - 用户也可以在前端界面填写自己的API，本地远程模型任意切换
+     
+3. 文献搜索和文本清洗
+   - 用户键入文献检索关键词，自动从PubMed公开数据库上搜索并下载文献全文
+   - xml到txt的文本清洗，去除reference 等无关信息
+  
+4. 基于[Ragflow](https://github.com/infiniflow/ragflow/blob/main/README_zh.md)的deepdoc库的PDF识别
+   - 输出文献中的文字和表格，其中文字存储为 txt, 表格存储为图片，json, html三个格式
+   - 目前工作流中仅利用文字，表格数据的利用开发中
+     
+5. chunk size可调的数据库生成
+   -  default 1024 [ref](https://www.llamaindex.ai/blog/evaluating-the-ideal-chunk-size-for-a-rag-system-using-llamaindex-6207e5d3fec5)
+     
+6. 嵌入kmeans聚类
+   - 基于Faiss库
+   - k可调
+     
+7. 基于LLM的聚类内容标注
+   - 为节省算力，可以抽样标注
+   - 标注后本地储存避免重复标注
+     
+8. 基于LLM的子问题生成
+   - 聚类标注内容作为context，生成对应的子问题
+     
+9. 基于LLM的综述生成
+   - 输入可以是用户自己的问题，也可以参考之前llm生成的子问题
+   - 为了比较同一个科学问题的不同来源的观点，修改了一部分茴香豆的Retriever逻辑
+      - retreiver 优化 [ref](https://medium.aiplanet.com/evaluating-naive-rag-and-advanced-rag-pipeline-using-langchain-v-0-1-0-and-ragas-17d24e74e5cf)
+      - 返回topk =10的分段
+      - 由于LLM捞针能力在头尾两端较靠谱，用langchain_community.document_transformers.LongContextReorder 将相关性较高的文本分布在头尾两端
+     
+10. gradio前端
+     
+<div style="display: flex;">
+    <img src="https://github.com/jabberwockyang/MedicalReviewAgent/assets/52541128/83a1cefe-ebe6-499a-9ca7-214e90089815" style="width: 30%;" />
+    <img src="https://github.com/jabberwockyang/MedicalReviewAgent/assets/52541128/c369aeaa-6749-4d56-b71a-d62ff1cb780f" style="width: 30%;" />
+    <img src="https://github.com/jabberwockyang/MedicalReviewAgent/assets/52541128/27f1d440-2b79-4cbb-9b15-a5e2fa037d33" style="width: 30%;" />
+</div>
+<div style="display: flex;">
+    <img src="https://github.com/jabberwockyang/MedicalReviewAgent/assets/52541128/db2443ff-b6a2-4c35-83e6-21e478c39eba" style="width: 30%;" />
+    <img src="https://github.com/jabberwockyang/MedicalReviewAgent/assets/52541128/77496f38-f1e6-4919-a439-c06b4fd52aab" style="width: 30%;" />
+</div>
+
+   
+## TODO 
+1. 自然语言到文献搜索参数的functional call功能
+   - 比如：
+       - 输入：帮我搜索近五年特应性皮炎相关的孟德尔随机化文章，不要综述
+       - 输出：
+         ```json
+            {"keywords":["atopic dermatitis","mendelian randomisation"],
+             "min-year":2019,
+             "max-year":2024,
+             "include-type":null,
+             "exclude-type":"review"
+             }
+         ```
+2. 摸索适用于不同需求的chunk size和 k值
+   - 比方说用来找某个实验方法用多大浓度的试剂，和总结某研究领域的前沿进展所用到的chunksize应该是不一样的吧🤔
+   - 后者的大小其实取决于前者叭🤔
+3. 表格数据利用
+     
+## 感谢
+1. [茴香豆](https://github.com/InternLM/HuixiangDou)
+2. [E-utilities](https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMCID)
+3. [Ragflow](https://github.com/infiniflow/ragflow/blob/main/README_zh.md)
+4. [Advanced RAG pipeline](https://medium.aiplanet.com/evaluating-naive-rag-and-advanced-rag-pipeline-using-langchain-v-0-1-0-and-ragas-17d24e74e5cf)
+   

app.py

@@ -0,0 +1,640 @@
+import argparse
+import json
+import time
+import os
+import glob
+import random
+import shutil
+from enum import Enum
+from threading import Thread
+from multiprocessing import Process, Value
+
+import gradio as gr
+import pytoml
+from loguru import logger
+
+from huixiangdou.service import Worker, llm_serve, ArticleRetrieval, CacheRetriever, FeatureStore, FileOperation
+
+class PARAM_CODE(Enum):
+    """Parameter code."""
+    SUCCESS = 0
+    FAILED = 1
+    ERROR = 2
+
+def parse_args():
+    """Parse args."""
+    parser = argparse.ArgumentParser(description='Worker.')
+    parser.add_argument('--work_dir',
+                        type=str,
+                        default='workdir',
+                        help='Working directory.')
+    parser.add_argument('--repo_dir',
+                        type=str,
+                        default='repodir',
+                        help='Repository directory.')
+    parser.add_argument(
+        '--config_path',
+        default='config.ini',
+        type=str,
+        help='Worker configuration path. Default value is config.ini')
+    parser.add_argument('--standalone',
+                        action='store_true',
+                        default=True,
+                        help='Auto deploy required Hybrid LLM Service.')
+    args = parser.parse_args()
+    return args
+
+def update_remote_buttons(remote):
+    if remote:
+        return [
+                gr.Markdown("[如何配置API]('https://github.com/jabberwockyang/MedicalReviewAgent/blob/main/README.md')",
+                            visible=True),
+                gr.Dropdown(["kimi", "deepseek", "zhipuai",'gpt'],
+                                                label="选择大模型提供商",
+                                                interactive=True,visible=True),
+                gr.Textbox(label="您的API",lines = 1,
+                        interactive=True,visible=True),
+                gr.Dropdown([],label="选择模型",
+                            interactive=True,visible=True)
+        ]
+    else:
+        return [
+                gr.Markdown("[如何配置API]('https://github.com/jabberwockyang/MedicalReviewAgent/blob/main/README.md')",
+                            visible=False),
+                gr.Dropdown(["kimi", "deepseek", "zhipuai",'gpt'],
+                                                label="选择大模型提供商",
+                                                interactive=False,visible=False),
+                gr.Textbox(label="您的API",lines = 1,
+                        interactive=False,visible=False),
+                gr.Dropdown([],label="选择模型",
+                            interactive=False,visible=False)
+        ]
+
+def udate_model_dropdown(remote_company):
+    model_choices = {
+        'kimi': ['moonshot-v1-128k'],
+        'deepseek': ['deepseek-chat'],
+        'zhipuai': ['glm-4'],
+        'gpt': ['gpt-4-32k-0613','gpt-3.5-turbo']
+    }
+    return gr.Dropdown(choices= model_choices[remote_company])
+
+def update_remote_config(remote_ornot,remote_company = None,api = None,model = None):
+    with open(CONFIG_PATH, encoding='utf8') as f:
+        config = pytoml.load(f)
+         
+        if remote_ornot:
+            if remote_company == None or api == None or model == None:
+                raise ValueError('remote_company, api, model not provided')
+            config['llm']['enable_local'] = 0
+            config['llm']['enable_remote'] = 1
+            config['llm']['server']['remote_type'] = remote_company
+            config['llm']['server']['remote_api_key'] = api
+            config['llm']['server']['remote_llm_model'] = model
+        else:
+            config['llm']['enable_local'] = 1
+            config['llm']['enable_remote'] = 0
+    with open(CONFIG_PATH, 'w') as f:
+        pytoml.dump(config, f)
+    return gr.Button("配置已保存")
+
+
+def get_ready(query:str,chunksize=None,k=None):
+    
+    with open(CONFIG_PATH, encoding='utf8') as f:
+        config = pytoml.load(f)
+    workdir = config['feature_store']['work_dir']
+    repodir = config['feature_store']['repo_dir']
+
+    if query == 'repo_work': # no need to return assistant
+        return repodir, workdir, config
+    theme = ''
+    try:
+        with open(os.path.join(config['feature_store']['repo_dir'],'config.json'), 'r') as f:
+            repo_config = json.load(f)
+        theme = ' '.join(repo_config['keywords'])
+    except:
+        pass
+
+    if query == 'annotation':
+        if not chunksize or not k:
+            raise ValueError('chunksize or k not provided')
+        chunkdir = os.path.join(workdir, f'chunksize_{chunksize}')
+        clusterdir = os.path.join(chunkdir, 'cluster_features', f'cluster_features_{k}')
+        assistant = Worker(work_dir=chunkdir, config_path=CONFIG_PATH,language='en')
+        samples_json = os.path.join(clusterdir,'samples.json')
+        with open(samples_json, 'r') as f:
+            samples = json.load(f)
+            f.close()
+        return clusterdir, samples, assistant, theme
+    
+    elif query == 'inspiration':
+        if not chunksize or not k:
+            raise ValueError('chunksize or k not provided')
+        
+        chunkdir = os.path.join(workdir, f'chunksize_{chunksize}')
+        clusterdir = os.path.join(chunkdir, 'cluster_features', f'cluster_features_{k}')
+        assistant = Worker(work_dir=chunkdir, config_path=CONFIG_PATH,language='en')
+        annofile = os.path.join(clusterdir,'annotation.jsonl')
+        with open(annofile, 'r') as f:
+            annoresult = f.readlines()
+
+            f.close()
+        annoresult = [json.loads(obj) for obj in annoresult]
+        return clusterdir, annoresult, assistant, theme
+    elif query == 'summarize': # no need for params k
+        if not chunksize:
+            raise ValueError('chunksize not provided')
+        chunkdir = os.path.join(workdir, f'chunksize_{chunksize}')
+        assistant = Worker(work_dir=chunkdir, config_path=CONFIG_PATH,language='en')
+        return assistant,theme
+
+    else:
+        raise ValueError('query not recognized')
+            
+def update_repo_info():
+    with open(CONFIG_PATH, encoding='utf8') as f:
+        config = pytoml.load(f)
+    repodir = config['feature_store']['repo_dir']
+    if os.path.exists(repodir):
+        pdffiles = glob.glob(os.path.join(repodir, '*.pdf'))
+        number_of_pdf = len(pdffiles)
+        if os.path.exists(os.path.join(repodir,'config.json')):
+                
+            with open(os.path.join(repodir,'config.json'), 'r') as f:
+                repo_config = json.load(f)
+
+            keywords = repo_config['keywords']
+            length = repo_config['len']
+            retmax = repo_config['retmax']
+
+            return keywords,length,retmax,number_of_pdf
+        else:
+            return None,None,None,number_of_pdf
+    else:
+        return None,None,None,None
+               
+def upload_file(files):
+    repodir, workdir, _ = get_ready('repo_work')
+    if not os.path.exists(repodir):
+        os.makedirs(repodir)
+
+    for file in files:
+        destination_path = os.path.join(repodir, os.path.basename(file.name))
+
+        shutil.copy(file.name, destination_path)
+    
+
+    return files
+
+def generate_articles_repo(keywords:str,retmax:int):
+    keys= [k.strip() for k in keywords.split('\n')]
+    repodir, _, _ = get_ready('repo_work')
+
+    articelfinder = ArticleRetrieval(keywords = keys,
+                                     repo_dir = repodir,
+                                     retmax = retmax)
+    articelfinder.initiallize()
+    return update_repo()
+def delete_articles_repo():
+    # 在这里运行生成数据库的函数
+    repodir, workdir, _ = get_ready('repo_work')
+    if os.path.exists(repodir):
+        shutil.rmtree(repodir)
+    if os.path.exists(workdir):
+        shutil.rmtree(workdir)
+
+    return gr.Textbox(label="文献库概况",lines =3,
+                      value = '文献库和相关数据库已删除',
+                      visible = True)
+
+def update_repo():
+    keys,len,retmax,pdflen = update_repo_info()
+    if keys:
+        newinfo = f"搜索得到文献：\n    关键词：{keys}\n    文献数量：{len}\n    获取上限：{retmax}\n\n上传文献：\n    数量：{pdflen}"
+    else:
+        if pdflen:
+            newinfo = f'搜索得到文献：无\n上传文献：\n    数量：{pdflen}'
+        else:
+            newinfo = '目前还没有文献库'
+
+    return gr.Textbox(label="文献库概况",lines =1,
+                      value = newinfo,
+                      visible = True)
+
+def update_database_info():
+    with open(CONFIG_PATH, encoding='utf8') as f:
+        config = pytoml.load(f)
+    workdir = config['feature_store']['work_dir']
+    chunkdirs = glob.glob(os.path.join(workdir, 'chunksize_*'))
+    chunkdirs.sort()
+    list_of_chunksize = [int(chunkdir.split('_')[-1]) for chunkdir in chunkdirs]
+    # print(list_of_chunksize)
+    jsonobj = {}
+    for chunkdir in chunkdirs:
+        k_dir = glob.glob(os.path.join(chunkdir, 'cluster_features','cluster_features_*'))
+        k_dir.sort()
+        list_of_k = [int(k.split('_')[-1]) for k in k_dir]
+        jsonobj[int(chunkdir.split('_')[-1])] = list_of_k
+        
+
+    new_options = [f"chunksize:{chunksize}, k:{k}" for chunksize in list_of_chunksize for k in jsonobj[chunksize]]
+    
+    return new_options, jsonobj
+
+
+def generate_database(chunksize:int,nclusters:str|list[str]):
+    # 在这里运行生成数据库的函数
+    repodir, workdir, _ = get_ready('repo_work')
+    if not os.path.exists(repodir):
+        return gr.Textbox(label="数据库已生成",value = '请先生成文献库',visible = True)
+    nclusters = [int(i) for i in nclusters]
+    # 文献库和数据库的覆盖删除逻辑待定 
+    # 理论上 文献库只能生成一次 所以每次生成文献库都要删除之前的文献库和数据库
+    # 数据库可以根据文献库多次生成 暂不做删除 目前没有节省算力的逻辑 重复计算后覆盖 以后优化 
+    # 不同的chunksize和nclusters会放在不同的��件夹下 不会互相覆盖
+    # if os.path.exists(workdir):
+    #     shutil.rmtree(workdir)
+
+    cache = CacheRetriever(config_path=CONFIG_PATH)
+    fs_init = FeatureStore(embeddings=cache.embeddings,
+                           reranker=cache.reranker,
+                            chunk_size=chunksize,
+                            n_clusters=nclusters,
+                           config_path=CONFIG_PATH)
+
+    # walk all files in repo dir
+    file_opr = FileOperation()
+    files = file_opr.scan_dir(repo_dir=repodir)
+    fs_init.initialize(files=files, work_dir=workdir,file_opr=file_opr)
+    file_opr.summarize(files)
+    del fs_init
+    cache.pop('default')
+    texts, _ = update_database_info()
+    return gr.Textbox(label="数据库概况",value = '\n'.join(texts) ,visible = True)
+
+def delete_database():
+    _, workdir, _ = get_ready('repo_work')
+    if os.path.exists(workdir):
+        shutil.rmtree(workdir)
+
+    return  gr.Textbox(label="数据库概况",lines =3,value = '数据库已删除',visible = True)
+def update_database_textbox():
+    texts, _ = update_database_info()
+    if texts == []:
+        return gr.Textbox(label="数据库概况",value = '目前还没有数据库',visible = True)
+    else:
+        return gr.Textbox(label="数据库概况",value = '\n'.join(texts),visible = True)
+
+def update_chunksize_dropdown():
+    _, jsonobj = update_database_info()
+    return gr.Dropdown(choices= jsonobj.keys())
+
+def update_ncluster_dropdown(chunksize:int):
+    _, jsonobj = update_database_info()
+    nclusters = jsonobj[chunksize]
+    return gr.Dropdown(choices= nclusters)
+
+def annotation(n,chunksize:int,nclusters:int,remote_ornot:bool):
+    '''
+    use llm to annotate cluster
+    n: percentage of clusters to annotate
+    '''
+    query = 'annotation'
+    if remote_ornot:
+        backend = 'remote'
+    else:
+        backend = 'local'
+
+    clusterdir, samples, assistant, theme = get_ready('annotation',chunksize,nclusters)
+    new_obj_list = []
+    n = round(n * len(samples.keys()))
+    for cluster_no in random.sample(samples.keys(), n):
+        chunk = '\n'.join(samples[cluster_no]['samples'][:10])
+
+        code, reply, cluster_no = assistant.annotate_cluster(
+                                                theme = theme,
+                                                cluster_no=cluster_no,
+                                                chunk=chunk,
+                                                history=[],
+                                                groupname='',
+                                                backend=backend)
+        references = f"cluster_no: {cluster_no}"
+        new_obj = {
+            'cluster_no': cluster_no,
+            'chunk': chunk,
+            'annotation': reply
+        }
+        new_obj_list.append(new_obj)
+        logger.info(f'{code}, {query}, {reply}, {references}')
+
+        with open(os.path.join(clusterdir, 'annotation.jsonl'), 'a') as f:
+            json.dump(new_obj, f, ensure_ascii=False)
+            f.write('\n')
+            
+    return '\n\n'.join([obj['annotation'] for obj in new_obj_list])
+
+
+def inspiration(annotation:str,chunksize:int,nclusters:int,remote_ornot:bool):
+    query = 'inspiration'
+    if remote_ornot:
+        backend = 'remote'
+    else:
+        backend = 'local'
+        
+    clusterdir, annoresult, assistant, theme = get_ready('inspiration',chunksize,nclusters)
+    new_obj_list = []
+
+    if annotation is not None: # if the user wants to get inspiration from specific clusters only  
+        annoresult = [obj for obj in annoresult if obj['annotation'] in [txt.strip() for txt in annotation.split('\n')]]
+    
+    for index in random.sample(range(len(annoresult)), min(5, len(annoresult))):
+        cluster_no = annoresult[index]['cluster_no']
+        chunks = annoresult[index]['annotation']
+        
+        code, reply = assistant.getinspiration(
+                                                theme = theme,
+                                                annotations = chunks,
+                                                history=[], 
+                                                groupname='',backend=backend)
+        new_obj = {
+            'inspiration': reply,
+            'cluster_no': cluster_no
+        }
+        new_obj_list.append(new_obj)
+        logger.info(f'{code}, {query}, {cluster_no},{reply}')
+
+        with open(os.path.join(clusterdir, 'inspiration.jsonl'), 'a') as f:
+            json.dump(new_obj, f, ensure_ascii=False)
+        with open(os.path.join(clusterdir, 'inspiration.txt'), 'a') as f:
+            f.write(f'{reply}\n')
+            
+    return '\n\n'.join(list(set([obj['inspiration'] for obj in new_obj_list])))
+
+
+def getpmcurls(references):
+    urls = []
+    for ref in references:
+        if ref.startswith('PMC'):
+            
+            refid = ref.replace('.txt','')
+            urls.append(f'https://www.ncbi.nlm.nih.gov/pmc/articles/{refid}/')
+        else:
+            urls.append(ref)
+    return urls
+
+def summarize_text(query,chunksize:int,remote_ornot:bool):
+    if remote_ornot:
+        backend = 'remote'
+    else:
+        backend = 'local'
+        
+    assistant,_ = get_ready('summarize',chunksize=chunksize,k=None)
+    code, reply, references = assistant.generate(query=query,
+                                                history=[],
+                                                groupname='',backend = backend)
+      
+    logger.info(f'{code}, {query}, {reply}, {references}')
+    urls = getpmcurls(references)
+    mds = '\n'.join([f'[{ref}]({url})' for ref,url in zip(references,urls)])
+    return reply, gr.Markdown(label="参考文献",value = mds) 
+
+def main_interface():   
+    with gr.Blocks() as demo:
+        with gr.Row():
+            gr.Markdown(
+                """
+                # 医学文献综述助手 (又名 不想看文献)
+                """
+            )
+
+        with gr.Tab("模型服务配置"):
+            gr.Markdown("""
+            #### 配置模型服务 🛠️
+
+            1. **是否使用远程大模型**
+            - 勾选此项，如果你想使用远程的大模型服务。
+            - 如果不勾选，将默认使用本地模型服务。
+
+            2. **API配置**
+            - 配置大模型提供商和API，确保模型服务能够正常运行。
+            - 提供商选择：kimi、deepseek、zhipuai、gpt。
+            - 输入您的API密钥和选择对应模型。
+            - 点击“保存配置”按钮以保存您的设置。
+
+            📝 **备注**：请参考[如何使用]('https://github.com/jabberwockyang/MedicalReviewAgent/blob/main/README.md')获取更多信息。
+
+            """)
+
+            remote_ornot = gr.Checkbox(label="是否使用远程大模型")
+            with gr.Accordion("API配置", open=True):
+                apimd = gr.Markdown("[如何配置API]('https://github.com/jabberwockyang/MedicalReviewAgent/blob/main/README.md')",visible=False)
+                remote_company = gr.Dropdown(["kimi", "deepseek", "zhipuai",'gpt'],
+                                            label="选择大模型提供商",interactive=False,visible=False)
+                api = gr.Textbox(label="您的API",lines = 1,interactive=False,visible=False)
+                model = gr.Dropdown([],label="选择模型",interactive=False,visible=False)
+            
+            confirm_button = gr.Button("保存配置")
+
+            remote_ornot.change(update_remote_buttons, inputs=[remote_ornot],outputs=[apimd,remote_company,api,model])
+            remote_company.change(udate_model_dropdown, inputs=[remote_company],outputs=[model])
+            confirm_button.click(update_remote_config, inputs=[remote_ornot,remote_company,api,model],outputs=[confirm_button])
+
+
+        with gr.Tab("文献查找+数据库生成"):
+            gr.Markdown("""
+#### 查找文献 📚
+
+1. **输入关键词批量PubMed PMC文献**
+   - 在“感兴趣的关键词”框中输入您感兴趣的关键词，每行一个。
+   - 设置查找数量（0-1000）。
+   - 点击“搜索PubMed PMC”按钮进行文献查找。
+
+2. **上传PDF**
+   - 通过“上传PDF”按钮上传您已有的PDF文献文件。
+
+3. **更新文献库情况 删除文献库**
+   - 点击“更新文献库情况”按钮，查看当前文献库的概况。
+   - 如果需要重置或删除现有文献库，点击“删除文献库”按钮。
+
+
+#### 生成数据库 🗂️
+
+1. **设置数据库构建参数 生成数据库**
+   - 选择块大小（Chunk Size）和聚类数（Number of Clusters）。
+   - 提供选项用于选择合适的块大小和聚类数。
+   - 点击“生成数据库”按钮开始数据库生成过程。
+
+2. **更新数据库情况 删除数据库**
+   - 点击“更新数据库情况”按钮，查看当前数据库的概况。
+   - 点击“删除数据库”按钮移除现有数据库。
+
+📝 **备注**：请参考[如何选择数据库构建参数]('https://github.com/jabberwockyang/MedicalReviewAgent/tree/main')获取更多信息。
+""")
+            with gr.Row(equal_height=True):
+                with gr.Column(scale=1):
+                    input_keys = gr.Textbox(label="感兴趣的关键词",
+                                                    lines = 5)
+                    retmax = gr.Slider(
+                            minimum=0,
+                            maximum=1000,
+                            value=500,
+                            interactive=True,
+                            label="查多少",
+                        )
+                    generate_repo_button = gr.Button("搜索PubMed PMC")
+                with gr.Column(scale=2):
+                    file_output = gr.File(scale=2)
+                    upload_button = gr.UploadButton("上传PDF", 
+                                    file_types=[".pdf",".csv",".doc"], 
+                                    file_count="multiple",scale=0)
+                    
+            with gr.Row(equal_height=True):
+                with gr.Column(scale=0):
+                    delete_repo_button = gr.Button("删除文献库")
+                    update_repo_button = gr.Button("更新文献库情况")
+                with gr.Column(scale=2):
+
+                    repo_summary =gr.Textbox(label= '文献库概况', value="目前还没有文献库")
+
+            generate_repo_button.click(generate_articles_repo, 
+                                inputs=[input_keys,retmax],
+                                outputs = [repo_summary])
+            
+            
+            delete_repo_button.click(delete_articles_repo, inputs=None,
+                                outputs = repo_summary)
+            update_repo_button.click(update_repo, inputs=None,
+                                outputs = repo_summary)
+            upload_button.upload(upload_file, upload_button, file_output)
+            
+            with gr.Accordion("数据库构建参数", open=True):
+                gr.Markdown("[如何选择数据库构建参数]('https://github.com/jabberwockyang/MedicalReviewAgent/tree/main')")
+                chunksize = gr.Slider(label="Chunk Size",
+                                      info= 'How long you want the chunk to be?',
+                                        minimum=128, maximum=4096,value=1024,step=1,
+                                        interactive=True)
+                ncluster = gr.CheckboxGroup(["10", "20", "50", '100','200','500','1000'], 
+                                            # default=["20", "50", '100'],
+                                            label="Number of Clusters", 
+                                            info="How many Clusters you want to generate")
+
+            with gr.Row():
+                gene_database_button = gr.Button("生成数据库")
+                delete_database_button = gr.Button("删除数据库")
+                update_database_button = gr.Button("更新数据库情况")
+
+            database_summary = gr.Textbox(label="数据库概况",lines = 1,value="目前还没有数据库")
+            
+
+            gene_database_button.click(generate_database, inputs=[chunksize,ncluster],
+                                outputs = database_summary)
+            
+            update_database_button.click(update_database_textbox,inputs=None,
+                                outputs = [database_summary])
+                                         
+            delete_database_button.click(delete_database, inputs=None,
+                                outputs = database_summary)
+        with gr.Tab("写综述"):
+            gr.Markdown("""
+#### 写综述 ✍️
+
+1. **更新数据库情况**
+   - 点击“更新数据库情况”按钮，确保使用最新的数据库信息。
+
+2. **选择块大小和聚类数**
+   - 从下拉菜单中选择合适的块大小和聚类数。
+
+3. **抽样标注文章聚类**
+   - 设置抽样标注比例（0-1）。
+   - 点击“抽样标注文章聚类”按钮开始标注过程。
+
+4. **获取灵感**
+   - 如果不知道写什么，点击“获取灵感”按钮。
+   - 系统将基于标注的文章聚类提供相应的综述子问题。
+
+5. **写综述**
+   - 输入您想写的内容或主题。
+   - 点击“写综述”按钮，生成综述文本。
+
+6. **查看生成结果**
+   - 生成的综述文本将显示在“看看”文本框中。
+   - 参考文献将显示在“参考文献”框中。
+
+📝 **备注**：可以尝试不同的参数进行标注和灵感获取，有助于提高综述的质量和相关性。
+            """)
+
+            with gr.Accordion("聚类标注相关参数", open=True):
+                with gr.Row():
+                    update_options = gr.Button("更新数据库情况", scale=0)
+                    chunksize = gr.Dropdown([], label="选择块大小", scale=0)
+                    nclusters = gr.Dropdown([], label="选择聚类数", scale=0)
+                    ntoread = gr.Slider(
+                            minimum=0,maximum=1,value=0.5,
+                            interactive=True,
+                            label="抽样标注比例",
+                        )
+
+            annotation_button = gr.Button("抽样标注文章聚类")
+            annotation_output =  gr.Textbox(label="文章聚类标注/片段摘要",
+                                            lines = 5, 
+                                            interactive= True,
+                                            show_copy_button=True)
+            inspiration_button = gr.Button("获取灵感")
+            inspiration_output = gr.Textbox(label="灵光一现",
+                                            lines = 5,
+                                            show_copy_button=True)
+
+
+            query = gr.Textbox(label="想写什么")
+            
+            write_button = gr.Button("写综述")
+            output_text = gr.Textbox(label="看看",lines=10)
+            output_references = gr.Markdown(label="参考文献")
+            
+            update_options.click(update_chunksize_dropdown,
+                                outputs=[chunksize])
+            
+            chunksize.change(update_ncluster_dropdown, 
+                             inputs=[chunksize],
+                             outputs= [nclusters])
+            
+            annotation_button.click(annotation, 
+                                    inputs = [ntoread, chunksize, nclusters,remote_ornot],
+                                    outputs=[annotation_output])
+            
+            inspiration_button.click(inspiration, 
+                                     inputs= [annotation_output, chunksize, nclusters,remote_ornot],
+                                     outputs=[inspiration_output])
+            
+            write_button.click(summarize_text,
+                                inputs=[query, chunksize,remote_ornot],
+                                outputs =[output_text,output_references])
+
+    demo.launch(share=False, server_name='0.0.0.0', debug=True,show_error=True,allowed_paths=['img_0.jpg']) 
+ 
+# start service
+if __name__ == '__main__':
+    args = parse_args()
+    # copy config from config-bak
+    shutil.copy('config-bak.ini', args.config_path) # yyj
+    CONFIG_PATH = args.config_path
+
+    if args.standalone is True:
+        # hybrid llm serve
+        server_ready = Value('i', 0)
+        server_process = Process(target=llm_serve,
+                                args=(args.config_path, server_ready))
+        server_process.start()
+        while True:
+            if server_ready.value == 0:
+                logger.info('waiting for server to be ready..')
+                time.sleep(3)
+            elif server_ready.value == 1:
+                break
+            else:
+                logger.error('start local LLM server failed, quit.')
+                raise Exception('local LLM path')
+        logger.info('Hybrid LLM Server start.')
+
+    main_interface()

config-bak.ini

@@ -0,0 +1,63 @@
+[feature_store]
+reject_throttle = 0
+embedding_model_path = "/root/models/bce-embedding-base_v1"
+reranker_model_path = "/root/models/bce-reranker-base_v1"
+repo_dir = "repodir"
+work_dir = "workdir"
+n_clusters = [20, 50]
+chunk_size = 1024
+
+[web_search]
+x_api_key = "${YOUR-API-KEY}"
+domain_partial_order = ["openai.com", "pytorch.org", "readthedocs.io", "nvidia.com", "stackoverflow.com", "juejin.cn", "zhuanlan.zhihu.com", "www.cnblogs.com"]
+save_dir = "logs/web_search_result"
+
+[llm]
+enable_local = 1
+enable_remote = 1
+client_url = "http://127.0.0.1:8888/inference"
+
+[llm.server]
+local_llm_path = "/root/models/Qwen1.5-7B-Chat"
+local_llm_max_text_length = 32000
+local_llm_bind_port = 8888
+remote_type = ""
+remote_api_key = ""
+remote_llm_max_text_length = 32000
+remote_llm_model = ""
+rpm = 500
+
+[worker]
+enable_sg_search = 0
+save_path = "logs/work.txt"
+
+[worker.time]
+start = "00:00:00"
+end = "23:59:59"
+has_weekday = 1
+
+[sg_search]
+binary_src_path = "/usr/local/bin/src"
+src_access_token = "${YOUR-SRC-ACCESS-TOKEN}"
+
+[sg_search.opencompass]
+github_repo_id = "open-compass/opencompass"
+introduction = "用于评测大型语言模型（LLM）. 它提供了完整的开源可复现的评测框架，支持大语言模型、多模态模型的一站式评测，基于分布式技术，对大参数量模型亦能实现高效评测。评测方向汇总为知识、语言、理解、推理、考试五大能力维度，整合集纳了超过70个评测数据集，合计提供了超过40万个模型评测问题，并提供长文本、安全、代码3类大模型特色技术能力评测。"
+
+[sg_search.lmdeploy]
+github_repo_id = "internlm/lmdeploy"
+introduction = "lmdeploy 是一个用于压缩、部署和服务 LLM（Large Language Model）的工具包。是一个服务端场景下，transformer 结构 LLM 部署工具，支持 GPU 服务端部署，速度有保障，支持 Tensor Parallel，多并发优化，功能全面，包括模型转换、缓存历史会话的 cache feature 等. 它还提供了 WebUI、命令行和 gRPC 客户端接入。"
+
+[frontend]
+type = "none"
+webhook_url = "https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxx"
+message_process_policy = "immediate"
+
+[frontend.lark_group]
+app_id = "cli_a53a34dcb778500e"
+app_secret = "2ajhg1ixSvlNm1bJkH4tJhPfTCsGGHT1"
+encrypt_key = "abc"
+verification_token = "def"
+
+[frontend.wechat_personal]
+bind_port = 9527

config.ini

@@ -0,0 +1,63 @@
+[feature_store]
+reject_throttle = 0
+embedding_model_path = "/root/models/bce-embedding-base_v1"
+reranker_model_path = "/root/models/bce-reranker-base_v1"
+repo_dir = "repodir"
+work_dir = "workdir"
+n_clusters = [20, 50]
+chunk_size = 1024
+
+[web_search]
+x_api_key = "${YOUR-API-KEY}"
+domain_partial_order = ["openai.com", "pytorch.org", "readthedocs.io", "nvidia.com", "stackoverflow.com", "juejin.cn", "zhuanlan.zhihu.com", "www.cnblogs.com"]
+save_dir = "logs/web_search_result"
+
+[llm]
+enable_local = 1
+enable_remote = 1
+client_url = "http://127.0.0.1:8888/inference"
+
+[llm.server]
+local_llm_path = "/root/models/Qwen1.5-7B-Chat"
+local_llm_max_text_length = 32000
+local_llm_bind_port = 8888
+remote_type = ""
+remote_api_key = ""
+remote_llm_max_text_length = 32000
+remote_llm_model = ""
+rpm = 500
+
+[worker]
+enable_sg_search = 0
+save_path = "logs/work.txt"
+
+[worker.time]
+start = "00:00:00"
+end = "23:59:59"
+has_weekday = 1
+
+[sg_search]
+binary_src_path = "/usr/local/bin/src"
+src_access_token = "${YOUR-SRC-ACCESS-TOKEN}"
+
+[sg_search.opencompass]
+github_repo_id = "open-compass/opencompass"
+introduction = "用于评测大型语言模型（LLM）. 它提供了完整的开源可复现的评测框架，支持大语言模型、多模态模型的一站式评测，基于分布式技术，对大参数量模型亦能实现高效评测。评测方向汇总为知识、语言、理解、推理、考试五大能力维度，整合集纳了超过70个评测数据集，合计提供了超过40万个模型评测问题，并提供长文本、安全、代码3类大模型特色技术能力评测。"
+
+[sg_search.lmdeploy]
+github_repo_id = "internlm/lmdeploy"
+introduction = "lmdeploy 是一个用于压缩、部署和服务 LLM（Large Language Model）的工具包。是一个服务端场景下，transformer 结构 LLM 部署工具，支持 GPU 服务端部署，速度有保障，支持 Tensor Parallel，多并发优化，功能全面，包括模型转换、缓存历史会话的 cache feature 等. 它还提供了 WebUI、命令行和 gRPC 客户端接入。"
+
+[frontend]
+type = "none"
+webhook_url = "https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxx"
+message_process_policy = "immediate"
+
+[frontend.lark_group]
+app_id = "cli_a53a34dcb778500e"
+app_secret = "2ajhg1ixSvlNm1bJkH4tJhPfTCsGGHT1"
+encrypt_key = "abc"
+verification_token = "def"
+
+[frontend.wechat_personal]
+bind_port = 9527

deepdoc/README.md

@@ -0,0 +1,122 @@
+English | [简体中文](./README_zh.md)
+
+# *Deep*Doc
+
+- [1. Introduction](#1)
+- [2. Vision](#2)
+- [3. Parser](#3)
+
+<a name="1"></a>
+## 1. Introduction
+
+With a bunch of documents from various domains with various formats and along with diverse retrieval requirements, 
+an accurate analysis becomes a very challenge task. *Deep*Doc is born for that purpose.
+There are 2 parts in *Deep*Doc so far: vision and parser. 
+You can run the flowing test programs if you're interested in our results of OCR, layout recognition and TSR.
+```bash
+python deepdoc/vision/t_ocr.py -h
+usage: t_ocr.py [-h] --inputs INPUTS [--output_dir OUTPUT_DIR]
+
+options:
+  -h, --help            show this help message and exit
+  --inputs INPUTS       Directory where to store images or PDFs, or a file path to a single image or PDF
+  --output_dir OUTPUT_DIR
+                        Directory where to store the output images. Default: './ocr_outputs'
+```
+```bash
+python deepdoc/vision/t_recognizer.py -h
+usage: t_recognizer.py [-h] --inputs INPUTS [--output_dir OUTPUT_DIR] [--threshold THRESHOLD] [--mode {layout,tsr}]
+
+options:
+  -h, --help            show this help message and exit
+  --inputs INPUTS       Directory where to store images or PDFs, or a file path to a single image or PDF
+  --output_dir OUTPUT_DIR
+                        Directory where to store the output images. Default: './layouts_outputs'
+  --threshold THRESHOLD
+                        A threshold to filter out detections. Default: 0.5
+  --mode {layout,tsr}   Task mode: layout recognition or table structure recognition
+```
+
+Our models are served on HuggingFace. If you have trouble downloading HuggingFace models, this might help!!
+```bash
+export HF_ENDPOINT=https://hf-mirror.com
+```
+
+<a name="2"></a>
+## 2. Vision
+
+We use vision information to resolve problems as human being.
+  - OCR. Since a lot of documents presented as images or at least be able to transform to image, 
+    OCR is a very essential and fundamental or even universal solution for text extraction.
+    ```bash
+        python deepdoc/vision/t_ocr.py --inputs=path_to_images_or_pdfs --output_dir=path_to_store_result
+     ```
+    The inputs could be directory to images or PDF, or a image or PDF. 
+    You can look into the folder 'path_to_store_result' where has images which demonstrate the positions of results,
+    txt files which contain the OCR text.
+    <div align="center" style="margin-top:20px;margin-bottom:20px;">
+    <img src="https://github.com/infiniflow/ragflow/assets/12318111/f25bee3d-aaf7-4102-baf5-d5208361d110" width="900"/>
+    </div>
+
+  - Layout recognition. Documents from different domain may have various layouts, 
+    like, newspaper, magazine, book and résumé are distinct in terms of layout. 
+    Only when machine have an accurate layout analysis, it can decide if these text parts are successive or not, 
+    or this part needs Table Structure Recognition(TSR) to process, or this part is a figure and described with this caption.
+    We have 10 basic layout components which covers most cases:
+      - Text
+      - Title
+      - Figure
+      - Figure caption
+      - Table
+      - Table caption
+      - Header
+      - Footer
+      - Reference
+      - Equation
+      
+     Have a try on the following command to see the layout detection results.
+     ```bash
+        python deepdoc/vision/t_recognizer.py --inputs=path_to_images_or_pdfs --threshold=0.2 --mode=layout --output_dir=path_to_store_result
+     ```
+    The inputs could be directory to images or PDF, or a image or PDF. 
+    You can look into the folder 'path_to_store_result' where has images which demonstrate the detection results as following:
+    <div align="center" style="margin-top:20px;margin-bottom:20px;">
+    <img src="https://github.com/infiniflow/ragflow/assets/12318111/07e0f625-9b28-43d0-9fbb-5bf586cd286f" width="1000"/>
+    </div>
+  
+  - Table Structure Recognition(TSR). Data table is a frequently used structure to present data including numbers or text.
+    And the structure of a table might be very complex, like hierarchy headers, spanning cells and projected row headers.
+    Along with TSR, we also reassemble the content into sentences which could be well comprehended by LLM. 
+    We have five labels for TSR task:
+      - Column
+      - Row
+      - Column header
+      - Projected row header
+      - Spanning cell
+      
+    Have a try on the following command to see the layout detection results.
+     ```bash
+        python deepdoc/vision/t_recognizer.py --inputs=path_to_images_or_pdfs --threshold=0.2 --mode=tsr --output_dir=path_to_store_result
+     ```
+    The inputs could be directory to images or PDF, or a image or PDF. 
+    You can look into the folder 'path_to_store_result' where has both images and html pages which demonstrate the detection results as following:
+    <div align="center" style="margin-top:20px;margin-bottom:20px;">
+    <img src="https://github.com/infiniflow/ragflow/assets/12318111/cb24e81b-f2ba-49f3-ac09-883d75606f4c" width="1000"/>
+    </div>
+        
+<a name="3"></a>
+## 3. Parser
+
+Four kinds of document formats as PDF, DOCX, EXCEL and PPT have their corresponding parser. 
+The most complex one is PDF parser since PDF's flexibility. The output of PDF parser includes:
+  - Text chunks with their own positions in PDF(page number and rectangular positions).
+  - Tables with cropped image from the PDF, and contents which has already translated into natural language sentences.
+  - Figures with caption and text in the figures.
+  
+### Résumé
+
+The résumé is a very complicated kind of document. A résumé which is composed of unstructured text 
+with various layouts could be resolved into structured data composed of nearly a hundred of fields.
+We haven't opened the parser yet, as we open the processing method after parsing procedure.
+
+    

deepdoc/README_zh.md

@@ -0,0 +1,116 @@
+[English](./README.md) | 简体中文
+
+# *Deep*Doc
+
+- [*Deep*Doc](#deepdoc)
+  - [1. 介绍](#1-介绍)
+  - [2. 视觉处理](#2-视觉处理)
+  - [3. 解析器](#3-解析器)
+    - [简历](#简历)
+
+<a name="1"></a>
+## 1. 介绍
+
+对于来自不同领域、具有不同格式和不同检索要求的大量文档，准确的分析成为一项极具挑战性的任务。*Deep*Doc 就是为了这个目的而诞生的。到目前为止，*Deep*Doc 中有两个组成部分：视觉处理和解析器。如果您对我们的OCR、布局识别和TSR结果感兴趣，您可以运行下面的测试程序。
+
+```bash
+python deepdoc/vision/t_ocr.py -h
+usage: t_ocr.py [-h] --inputs INPUTS [--output_dir OUTPUT_DIR]
+
+options:
+  -h, --help            show this help message and exit
+  --inputs INPUTS       Directory where to store images or PDFs, or a file path to a single image or PDF
+  --output_dir OUTPUT_DIR
+                        Directory where to store the output images. Default: './ocr_outputs'
+```
+
+```bash
+python deepdoc/vision/t_recognizer.py -h
+usage: t_recognizer.py [-h] --inputs INPUTS [--output_dir OUTPUT_DIR] [--threshold THRESHOLD] [--mode {layout,tsr}]
+
+options:
+  -h, --help            show this help message and exit
+  --inputs INPUTS       Directory where to store images or PDFs, or a file path to a single image or PDF
+  --output_dir OUTPUT_DIR
+                        Directory where to store the output images. Default: './layouts_outputs'
+  --threshold THRESHOLD
+                        A threshold to filter out detections. Default: 0.5
+  --mode {layout,tsr}   Task mode: layout recognition or table structure recognition
+```
+
+HuggingFace为我们的模型提供服务。如果你在下载HuggingFace模型时遇到问题，这可能会有所帮助！！
+
+```bash
+export HF_ENDPOINT=https://hf-mirror.com
+```
+
+<a name="2"></a>
+## 2. 视觉处理
+
+作为人类，我们使用视觉信息来解决问题。
+
+  - **OCR（Optical Character Recognition，光学字符识别）**。由于许多文档都是以图像形式呈现的，或者至少能够转换为图像，因此OCR是文本提取的一个非常重要、基本，甚至通用的解决方案。
+
+    ```bash
+    python deepdoc/vision/t_ocr.py --inputs=path_to_images_or_pdfs --output_dir=path_to_store_result
+    ```
+
+    输入可以是图像或PDF的目录，或者单个图像、PDF文件。您可以查看文件夹 `path_to_store_result` ，其中有演示结果位置的图像，以及包含OCR文本的txt文件。
+    
+    <div align="center" style="margin-top:20px;margin-bottom:20px;">
+    <img src="https://github.com/infiniflow/ragflow/assets/12318111/f25bee3d-aaf7-4102-baf5-d5208361d110" width="900"/>
+    </div>
+
+  - 布局识别（Layout recognition）。来自不同领域的文件可能有不同的布局，如报纸、杂志、书籍和简历在布局方面是不同的。只有当机器有准确的布局分析时，它才能决定这些文本部分是连续的还是不连续的，或者这个部分需要表结构识别（Table Structure Recognition，TSR）来处理，或者这个部件是一个图形并用这个标题来描述。我们有10个基本布局组件，涵盖了大多数情况：
+      - 文本
+      - 标题
+      - 配图
+      - 配图标题
+      - 表格
+      - 表格标题
+      - 页头
+      - 页尾
+      - 参考引用
+      - 公式
+      
+     请尝试以下命令以查看布局检测结果。
+
+    ```bash
+    python deepdoc/vision/t_recognizer.py --inputs=path_to_images_or_pdfs --threshold=0.2 --mode=layout --output_dir=path_to_store_result
+    ```
+
+    输入可以是图像或PDF的目录，或者单个图像、PDF文件。您可以查看文件夹 `path_to_store_result` ，其中有显示检测结果的图像，如下所示：
+    <div align="center" style="margin-top:20px;margin-bottom:20px;">
+    <img src="https://github.com/infiniflow/ragflow/assets/12318111/07e0f625-9b28-43d0-9fbb-5bf586cd286f" width="1000"/>
+    </div>
+  
+  - **TSR（Table Structure Recognition，表结构识别）**。数据表是一种常用的结构，用于表示包括数字或文本在内的数据。表的结构可能非常复杂，比如层次结构标题、跨单元格和投影行标题。除了TSR，我们还将内容重新组合成LLM可以很好理解的句子。TSR任务有五个标签：
+      - 列
+      - 行
+      - 列标题
+      - 行标题
+      - 合并单元格
+      
+    请尝试以下命令以查看布局检测结果。
+
+    ```bash
+    python deepdoc/vision/t_recognizer.py --inputs=path_to_images_or_pdfs --threshold=0.2 --mode=tsr --output_dir=path_to_store_result
+    ```
+
+    输入可以是图像或PDF的目录，或者单个图像、PDF文件。您可以查看文件夹 `path_to_store_result` ，其中包含图像和html页面，这些页面展示了以下检测结果：
+
+    <div align="center" style="margin-top:20px;margin-bottom:20px;">
+    <img src="https://github.com/infiniflow/ragflow/assets/12318111/cb24e81b-f2ba-49f3-ac09-883d75606f4c" width="1000"/>
+    </div>
+        
+<a name="3"></a>
+## 3. 解析器
+
+PDF、DOCX、EXCEL和PPT四种文档格式都有相应的解析器。最复杂的是PDF解析器，因为PDF具有灵活性。PDF解析器的输出包括：
+  - 在PDF中有自己位置的文本块（页码和矩形位置）。
+  - 带有PDF裁剪图像的表格，以及已经翻译成自然语言句子的内容。
+  - 图中带标题和文字的图。
+  
+### 简历
+
+简历是一种非常复杂的文件。一份由各种布局的非结构化文本组成的简历可以分解为由近百个字段组成的结构化数据。我们还没有打开解析器，因为我们在解析过程之后打开了处理方法。

deepdoc/__init__.py

@@ -0,0 +1,6 @@
+from .parser import RAGFlowPdfParser
+from .parser import PlainParser
+from .parser import RAGFlowDocxParser
+from .parser import RAGFlowExcelParser
+from .parser import RAGFlowPptParser
+

deepdoc/parser/__init__.py

@@ -0,0 +1,8 @@
+
+
+from .pdf_parser import RAGFlowPdfParser 
+from .pdf_parser import PlainParser
+from .docx_parser import RAGFlowDocxParser
+from .excel_parser import RAGFlowExcelParser
+from .ppt_parser import RAGFlowPptParser
+

deepdoc/parser/docx_parser.py

@@ -0,0 +1,119 @@
+# -*- coding: utf-8 -*-
+from docx import Document
+import re
+import pandas as pd
+from collections import Counter
+from deepdoc.utils import rag_tokenizer
+from io import BytesIO
+
+
+class RAGFlowDocxParser:
+
+    def __extract_table_content(self, tb):
+        df = []
+        for row in tb.rows:
+            df.append([c.text for c in row.cells])
+        return self.__compose_table_content(pd.DataFrame(df))
+
+    def __compose_table_content(self, df):
+
+        def blockType(b):
+            patt = [
+                ("^(20|19)[0-9]{2}[年/-][0-9]{1,2}[月/-][0-9]{1,2}日*$", "Dt"),
+                (r"^(20|19)[0-9]{2}年$", "Dt"),
+                (r"^(20|19)[0-9]{2}[年/-][0-9]{1,2}月*$", "Dt"),
+                ("^[0-9]{1,2}[月/-][0-9]{1,2}日*$", "Dt"),
+                (r"^第*[一二三四1-4]季度$", "Dt"),
+                (r"^(20|19)[0-9]{2}年*[一二三四1-4]季度$", "Dt"),
+                (r"^(20|19)[0-9]{2}[ABCDE]$", "DT"),
+                ("^[0-9.,+%/ -]+$", "Nu"),
+                (r"^[0-9A-Z/\._~-]+$", "Ca"),
+                (r"^[A-Z]*[a-z' -]+$", "En"),
+                (r"^[0-9.,+-]+[0-9A-Za-z/$￥%<>（）()' -]+$", "NE"),
+                (r"^.{1}$", "Sg")
+            ]
+            for p, n in patt:
+                if re.search(p, b):
+                    return n
+            tks = [t for t in rag_tokenizer.tokenize(b).split(" ") if len(t) > 1]
+            if len(tks) > 3:
+                if len(tks) < 12:
+                    return "Tx"
+                else:
+                    return "Lx"
+
+            if len(tks) == 1 and rag_tokenizer.tag(tks[0]) == "nr":
+                return "Nr"
+
+            return "Ot"
+
+        if len(df) < 2:
+            return []
+        max_type = Counter([blockType(str(df.iloc[i, j])) for i in range(
+            1, len(df)) for j in range(len(df.iloc[i, :]))])
+        max_type = max(max_type.items(), key=lambda x: x[1])[0]
+
+        colnm = len(df.iloc[0, :])
+        hdrows = [0]  # header is not nessesarily appear in the first line
+        if max_type == "Nu":
+            for r in range(1, len(df)):
+                tys = Counter([blockType(str(df.iloc[r, j]))
+                              for j in range(len(df.iloc[r, :]))])
+                tys = max(tys.items(), key=lambda x: x[1])[0]
+                if tys != max_type:
+                    hdrows.append(r)
+
+        lines = []
+        for i in range(1, len(df)):
+            if i in hdrows:
+                continue
+            hr = [r - i for r in hdrows]
+            hr = [r for r in hr if r < 0]
+            t = len(hr) - 1
+            while t > 0:
+                if hr[t] - hr[t - 1] > 1:
+                    hr = hr[t:]
+                    break
+                t -= 1
+            headers = []
+            for j in range(len(df.iloc[i, :])):
+                t = []
+                for h in hr:
+                    x = str(df.iloc[i + h, j]).strip()
+                    if x in t:
+                        continue
+                    t.append(x)
+                t = ",".join(t)
+                if t:
+                    t += ": "
+                headers.append(t)
+            cells = []
+            for j in range(len(df.iloc[i, :])):
+                if not str(df.iloc[i, j]):
+                    continue
+                cells.append(headers[j] + str(df.iloc[i, j]))
+            lines.append(";".join(cells))
+
+        if colnm > 3:
+            return lines
+        return ["\n".join(lines)]
+
+    def __call__(self, fnm, from_page=0, to_page=100000):
+        self.doc = Document(fnm) if isinstance(
+            fnm, str) else Document(BytesIO(fnm))
+        pn = 0
+        secs = []
+        for p in self.doc.paragraphs:
+            if pn > to_page:
+                break
+            if from_page <= pn < to_page and p.text.strip():
+                secs.append((p.text, p.style.name))
+            for run in p.runs:
+                if 'lastRenderedPageBreak' in run._element.xml:
+                    pn += 1
+                    continue
+                if 'w:br' in run._element.xml and 'type="page"' in run._element.xml:
+                    pn += 1
+
+        tbls = [self.__extract_table_content(tb) for tb in self.doc.tables]
+        return secs, tbls

deepdoc/parser/excel_parser.py

@@ -0,0 +1,101 @@
+# -*- coding: utf-8 -*-
+from openpyxl import load_workbook
+import sys
+from io import BytesIO
+
+def find_codec(blob):
+    global all_codecs
+    for c in all_codecs:
+        try:
+            blob[:1024].decode(c)
+            return c
+        except Exception as e:
+            pass
+        try:
+            blob.decode(c)
+            return c
+        except Exception as e:
+            pass
+
+    return "utf-8"
+
+
+class RAGFlowExcelParser:
+    def html(self, fnm, chunk_rows=256):
+        if isinstance(fnm, str):
+            wb = load_workbook(fnm)
+        else:
+            wb = load_workbook(BytesIO(fnm))
+
+        tb_chunks = []
+        for sheetname in wb.sheetnames:
+            ws = wb[sheetname]
+            rows = list(ws.rows)
+            if not rows: continue
+
+            tb_rows_0 = "<tr>"
+            for t in list(rows[0]):
+                tb_rows_0 += f"<th>{t.value}</th>"
+            tb_rows_0 += "</tr>"
+
+            for chunk_i in range((len(rows) - 1) // chunk_rows + 1):
+                tb = ""
+                tb += f"<table><caption>{sheetname}</caption>"
+                tb += tb_rows_0
+                for r in list(rows[1 + chunk_i * chunk_rows:1 + (chunk_i + 1) * chunk_rows]):
+                    tb += "<tr>"
+                    for i, c in enumerate(r):
+                        if c.value is None:
+                            tb += "<td></td>"
+                        else:
+                            tb += f"<td>{c.value}</td>"
+                    tb += "</tr>"
+                tb += "</table>\n"
+                tb_chunks.append(tb)
+
+        return tb_chunks
+
+    def __call__(self, fnm):
+        if isinstance(fnm, str):
+            wb = load_workbook(fnm)
+        else:
+            wb = load_workbook(BytesIO(fnm))
+        res = []
+        for sheetname in wb.sheetnames:
+            ws = wb[sheetname]
+            rows = list(ws.rows)
+            if not rows:continue
+            ti = list(rows[0])
+            for r in list(rows[1:]):
+                l = []
+                for i, c in enumerate(r):
+                    if not c.value:
+                        continue
+                    t = str(ti[i].value) if i < len(ti) else ""
+                    t += ("：" if t else "") + str(c.value)
+                    l.append(t)
+                l = "; ".join(l)
+                if sheetname.lower().find("sheet") < 0:
+                    l += " ——" + sheetname
+                res.append(l)
+        return res
+
+    @staticmethod
+    def row_number(fnm, binary):
+        if fnm.split(".")[-1].lower().find("xls") >= 0:
+            wb = load_workbook(BytesIO(binary))
+            total = 0
+            for sheetname in wb.sheetnames:
+                ws = wb[sheetname]
+                total += len(list(ws.rows))
+                return total
+
+        if fnm.split(".")[-1].lower() in ["csv", "txt"]:
+            encoding = find_codec(binary)
+            txt = binary.decode(encoding, errors="ignore")
+            return len(txt.split("\n"))
+
+
+if __name__ == "__main__":
+    psr = RAGFlowExcelParser()
+    psr(sys.argv[1])

deepdoc/parser/pdf_parser.py

@@ -0,0 +1,1161 @@
+# -*- coding: utf-8 -*-
+import os
+import random
+
+import xgboost as xgb
+from io import BytesIO
+import torch
+import re
+import pdfplumber
+import logging
+from PIL import Image, ImageDraw
+import numpy as np
+from timeit import default_timer as timer
+from PyPDF2 import PdfReader as pdf2_read
+
+from deepdoc.utils.file_utils import get_project_base_directory
+from deepdoc.vision import OCR, Recognizer, LayoutRecognizer, TableStructureRecognizer
+from deepdoc.utils import rag_tokenizer
+from copy import deepcopy
+from huggingface_hub import snapshot_download
+
+logging.getLogger("pdfminer").setLevel(logging.WARNING)
+
+
+class RAGFlowPdfParser:
+    def __init__(self):
+        self.ocr = OCR()
+        if hasattr(self, "model_speciess"):
+            self.layouter = LayoutRecognizer("layout." + self.model_speciess)
+        else:
+            self.layouter = LayoutRecognizer("layout")
+        self.tbl_det = TableStructureRecognizer()
+
+        self.updown_cnt_mdl = xgb.Booster()
+        if torch.cuda.is_available():
+            self.updown_cnt_mdl.set_param({"device": "cuda"})
+        try:
+            model_dir = os.path.join(
+                get_project_base_directory(),
+                "rag/res/deepdoc")
+            self.updown_cnt_mdl.load_model(os.path.join(
+                model_dir, "updown_concat_xgb.model"))
+        except Exception as e:
+            model_dir = snapshot_download(
+                repo_id="InfiniFlow/text_concat_xgb_v1.0",
+                local_dir=os.path.join(get_project_base_directory(), "rag/res/deepdoc"),
+                local_dir_use_symlinks=False)
+            self.updown_cnt_mdl.load_model(os.path.join(
+                model_dir, "updown_concat_xgb.model"))
+
+        self.page_from = 0
+        """
+        If you have trouble downloading HuggingFace models, -_^ this might help!!
+
+        For Linux:
+        export HF_ENDPOINT=https://hf-mirror.com
+
+        For Windows:
+        Good luck
+        ^_-
+
+        """
+
+    def __char_width(self, c):
+        return (c["x1"] - c["x0"]) // max(len(c["text"]), 1)
+
+    def __height(self, c):
+        return c["bottom"] - c["top"]
+
+    def _x_dis(self, a, b):
+        return min(abs(a["x1"] - b["x0"]), abs(a["x0"] - b["x1"]),
+                   abs(a["x0"] + a["x1"] - b["x0"] - b["x1"]) / 2)
+
+    def _y_dis(
+            self, a, b):
+        return (
+                       b["top"] + b["bottom"] - a["top"] - a["bottom"]) / 2
+
+    def _match_proj(self, b):
+        proj_patt = [
+            r"第[零一二三四五六七八九十百]+章",
+            r"第[零一二三四五六七八九十百]+[条节]",
+            r"[零一二三四五六七八九十百]+[、是 　]",
+            r"[\(（][零一二三四五六七八九十百]+[）\)]",
+            r"[\(（][0-9]+[）\)]",
+            r"[0-9]+(、|\.[　 ]|）|\.[^0-9./a-zA-Z_%><-]{4,})",
+            r"[0-9]+\.[0-9.]+(、|\.[ 　])",
+            r"[⚫•➢①② ]",
+        ]
+        return any([re.match(p, b["text"]) for p in proj_patt])
+
+    def _updown_concat_features(self, up, down):
+        w = max(self.__char_width(up), self.__char_width(down))
+        h = max(self.__height(up), self.__height(down))
+        y_dis = self._y_dis(up, down)
+        LEN = 6
+        tks_down = rag_tokenizer.tokenize(down["text"][:LEN]).split(" ")
+        tks_up = rag_tokenizer.tokenize(up["text"][-LEN:]).split(" ")
+        tks_all = up["text"][-LEN:].strip() \
+                  + (" " if re.match(r"[a-zA-Z0-9]+",
+                                     up["text"][-1] + down["text"][0]) else "") \
+                  + down["text"][:LEN].strip()
+        tks_all = rag_tokenizer.tokenize(tks_all).split(" ")
+        fea = [
+            up.get("R", -1) == down.get("R", -1),
+            y_dis / h,
+            down["page_number"] - up["page_number"],
+            up["layout_type"] == down["layout_type"],
+            up["layout_type"] == "text",
+            down["layout_type"] == "text",
+            up["layout_type"] == "table",
+            down["layout_type"] == "table",
+            True if re.search(
+                r"([。？！；!?;+)）]|[a-z]\.)$",
+                up["text"]) else False,
+            True if re.search(r"[，：‘“、0-9（+-]$", up["text"]) else False,
+            True if re.search(
+                r"(^.?[/,?;:\]，。；：’”？！》】）-])",
+                down["text"]) else False,
+            True if re.match(r"[\(（][^\(\)（）]+[）\)]$", up["text"]) else False,
+            True if re.search(r"[，,][^。.]+$", up["text"]) else False,
+            True if re.search(r"[，,][^。.]+$", up["text"]) else False,
+            True if re.search(r"[\(（][^\)）]+$", up["text"])
+                    and re.search(r"[\)）]", down["text"]) else False,
+            self._match_proj(down),
+            True if re.match(r"[A-Z]", down["text"]) else False,
+            True if re.match(r"[A-Z]", up["text"][-1]) else False,
+            True if re.match(r"[a-z0-9]", up["text"][-1]) else False,
+            True if re.match(r"[0-9.%,-]+$", down["text"]) else False,
+            up["text"].strip()[-2:] == down["text"].strip()[-2:] if len(up["text"].strip()
+                                                                        ) > 1 and len(
+                down["text"].strip()) > 1 else False,
+            up["x0"] > down["x1"],
+            abs(self.__height(up) - self.__height(down)) / min(self.__height(up),
+                                                               self.__height(down)),
+            self._x_dis(up, down) / max(w, 0.000001),
+            (len(up["text"]) - len(down["text"])) /
+            max(len(up["text"]), len(down["text"])),
+            len(tks_all) - len(tks_up) - len(tks_down),
+            len(tks_down) - len(tks_up),
+            tks_down[-1] == tks_up[-1],
+            max(down["in_row"], up["in_row"]),
+            abs(down["in_row"] - up["in_row"]),
+            len(tks_down) == 1 and rag_tokenizer.tag(tks_down[0]).find("n") >= 0,
+            len(tks_up) == 1 and rag_tokenizer.tag(tks_up[0]).find("n") >= 0
+        ]
+        return fea
+
+    @staticmethod
+    def sort_X_by_page(arr, threashold):
+        # sort using y1 first and then x1
+        arr = sorted(arr, key=lambda r: (r["page_number"], r["x0"], r["top"]))
+        for i in range(len(arr) - 1):
+            for j in range(i, -1, -1):
+                # restore the order using th
+                if abs(arr[j + 1]["x0"] - arr[j]["x0"]) < threashold \
+                        and arr[j + 1]["top"] < arr[j]["top"] \
+                        and arr[j + 1]["page_number"] == arr[j]["page_number"]:
+                    tmp = arr[j]
+                    arr[j] = arr[j + 1]
+                    arr[j + 1] = tmp
+        return arr
+
+    def _has_color(self, o):
+        if o.get("ncs", "") == "DeviceGray":
+            if o["stroking_color"] and o["stroking_color"][0] == 1 and o["non_stroking_color"] and \
+                    o["non_stroking_color"][0] == 1:
+                if re.match(r"[a-zT_\[\]\(\)-]+", o.get("text", "")):
+                    return False
+        return True
+
+    def _table_transformer_job(self, ZM):
+        logging.info("Table processing...")
+        imgs, pos = [], []
+        tbcnt = [0]
+        MARGIN = 10
+        self.tb_cpns = []
+        assert len(self.page_layout) == len(self.page_images)
+        for p, tbls in enumerate(self.page_layout):  # for page
+            tbls = [f for f in tbls if f["type"] == "table"]
+            tbcnt.append(len(tbls))
+            if not tbls:
+                continue
+            for tb in tbls:  # for table
+                left, top, right, bott = tb["x0"] - MARGIN, tb["top"] - MARGIN, \
+                                         tb["x1"] + MARGIN, tb["bottom"] + MARGIN
+                left *= ZM
+                top *= ZM
+                right *= ZM
+                bott *= ZM
+                pos.append((left, top))
+                imgs.append(self.page_images[p].crop((left, top, right, bott)))
+
+        assert len(self.page_images) == len(tbcnt) - 1
+        if not imgs:
+            return
+        recos = self.tbl_det(imgs)
+        tbcnt = np.cumsum(tbcnt)
+        for i in range(len(tbcnt) - 1):  # for page
+            pg = []
+            for j, tb_items in enumerate(
+                    recos[tbcnt[i]: tbcnt[i + 1]]):  # for table
+                poss = pos[tbcnt[i]: tbcnt[i + 1]]
+                for it in tb_items:  # for table components
+                    it["x0"] = (it["x0"] + poss[j][0])
+                    it["x1"] = (it["x1"] + poss[j][0])
+                    it["top"] = (it["top"] + poss[j][1])
+                    it["bottom"] = (it["bottom"] + poss[j][1])
+                    for n in ["x0", "x1", "top", "bottom"]:
+                        it[n] /= ZM
+                    it["top"] += self.page_cum_height[i]
+                    it["bottom"] += self.page_cum_height[i]
+                    it["pn"] = i
+                    it["layoutno"] = j
+                    pg.append(it)
+            self.tb_cpns.extend(pg)
+
+        def gather(kwd, fzy=10, ption=0.6):
+            eles = Recognizer.sort_Y_firstly(
+                [r for r in self.tb_cpns if re.match(kwd, r["label"])], fzy)
+            eles = Recognizer.layouts_cleanup(self.boxes, eles, 5, ption)
+            return Recognizer.sort_Y_firstly(eles, 0)
+
+        # add R,H,C,SP tag to boxes within table layout
+        headers = gather(r".*header$")
+        rows = gather(r".* (row|header)")
+        spans = gather(r".*spanning")
+        clmns = sorted([r for r in self.tb_cpns if re.match(
+            r"table column$", r["label"])], key=lambda x: (x["pn"], x["layoutno"], x["x0"]))
+        clmns = Recognizer.layouts_cleanup(self.boxes, clmns, 5, 0.5)
+        for b in self.boxes:
+            if b.get("layout_type", "") != "table":
+                continue
+            ii = Recognizer.find_overlapped_with_threashold(b, rows, thr=0.3)
+            if ii is not None:
+                b["R"] = ii
+                b["R_top"] = rows[ii]["top"]
+                b["R_bott"] = rows[ii]["bottom"]
+
+            ii = Recognizer.find_overlapped_with_threashold(
+                b, headers, thr=0.3)
+            if ii is not None:
+                b["H_top"] = headers[ii]["top"]
+                b["H_bott"] = headers[ii]["bottom"]
+                b["H_left"] = headers[ii]["x0"]
+                b["H_right"] = headers[ii]["x1"]
+                b["H"] = ii
+
+            ii = Recognizer.find_horizontally_tightest_fit(b, clmns)
+            if ii is not None:
+                b["C"] = ii
+                b["C_left"] = clmns[ii]["x0"]
+                b["C_right"] = clmns[ii]["x1"]
+
+            ii = Recognizer.find_overlapped_with_threashold(b, spans, thr=0.3)
+            if ii is not None:
+                b["H_top"] = spans[ii]["top"]
+                b["H_bott"] = spans[ii]["bottom"]
+                b["H_left"] = spans[ii]["x0"]
+                b["H_right"] = spans[ii]["x1"]
+                b["SP"] = ii
+
+    def __ocr(self, pagenum, img, chars, ZM=3):
+        bxs = self.ocr.detect(np.array(img))
+        if not bxs:
+            self.boxes.append([])
+            return
+        bxs = [(line[0], line[1][0]) for line in bxs]
+        bxs = Recognizer.sort_Y_firstly(
+            [{"x0": b[0][0] / ZM, "x1": b[1][0] / ZM,
+              "top": b[0][1] / ZM, "text": "", "txt": t,
+              "bottom": b[-1][1] / ZM,
+              "page_number": pagenum} for b, t in bxs if b[0][0] <= b[1][0] and b[0][1] <= b[-1][1]],
+            self.mean_height[-1] / 3
+        )
+
+        # merge chars in the same rect
+        for c in Recognizer.sort_X_firstly(
+                chars, self.mean_width[pagenum - 1] // 4):
+            ii = Recognizer.find_overlapped(c, bxs)
+            if ii is None:
+                self.lefted_chars.append(c)
+                continue
+            ch = c["bottom"] - c["top"]
+            bh = bxs[ii]["bottom"] - bxs[ii]["top"]
+            if abs(ch - bh) / max(ch, bh) >= 0.7 and c["text"] != ' ':
+                self.lefted_chars.append(c)
+                continue
+            if c["text"] == " " and bxs[ii]["text"]:
+                if re.match(r"[0-9a-zA-Z,.?;:!%%]", bxs[ii]["text"][-1]):
+                    bxs[ii]["text"] += " "
+            else:
+                bxs[ii]["text"] += c["text"]
+
+        for b in bxs:
+            if not b["text"]:
+                left, right, top, bott = b["x0"] * ZM, b["x1"] * \
+                                         ZM, b["top"] * ZM, b["bottom"] * ZM
+                b["text"] = self.ocr.recognize(np.array(img),
+                                               np.array([[left, top], [right, top], [right, bott], [left, bott]],
+                                                        dtype=np.float32))
+            del b["txt"]
+        bxs = [b for b in bxs if b["text"]]
+        if self.mean_height[-1] == 0:
+            self.mean_height[-1] = np.median([b["bottom"] - b["top"]
+                                              for b in bxs])
+        self.boxes.append(bxs)
+
+    def _layouts_rec(self, ZM, drop=True):
+        assert len(self.page_images) == len(self.boxes)
+        self.boxes, self.page_layout = self.layouter(
+            self.page_images, self.boxes, ZM, drop=drop)
+        # cumlative Y
+        for i in range(len(self.boxes)):
+            self.boxes[i]["top"] += \
+                self.page_cum_height[self.boxes[i]["page_number"] - 1]
+            self.boxes[i]["bottom"] += \
+                self.page_cum_height[self.boxes[i]["page_number"] - 1]
+
+    def _text_merge(self):
+        # merge adjusted boxes
+        bxs = self.boxes
+
+        def end_with(b, txt):
+            txt = txt.strip()
+            tt = b.get("text", "").strip()
+            return tt and tt.find(txt) == len(tt) - len(txt)
+
+        def start_with(b, txts):
+            tt = b.get("text", "").strip()
+            return tt and any([tt.find(t.strip()) == 0 for t in txts])
+
+        # horizontally merge adjacent box with the same layout
+        i = 0
+        while i < len(bxs) - 1:
+            b = bxs[i]
+            b_ = bxs[i + 1]
+            if b.get("layoutno", "0") != b_.get("layoutno", "1") or b.get("layout_type", "") in ["table", "figure",
+                                                                                                 "equation"]:
+                i += 1
+                continue
+            if abs(self._y_dis(b, b_)
+                   ) < self.mean_height[bxs[i]["page_number"] - 1] / 3:
+                # merge
+                bxs[i]["x1"] = b_["x1"]
+                bxs[i]["top"] = (b["top"] + b_["top"]) / 2
+                bxs[i]["bottom"] = (b["bottom"] + b_["bottom"]) / 2
+                bxs[i]["text"] += b_["text"]
+                bxs.pop(i + 1)
+                continue
+            i += 1
+            continue
+
+            dis_thr = 1
+            dis = b["x1"] - b_["x0"]
+            if b.get("layout_type", "") != "text" or b_.get(
+                    "layout_type", "") != "text":
+                if end_with(b, "，") or start_with(b_, "（，"):
+                    dis_thr = -8
+                else:
+                    i += 1
+                    continue
+
+            if abs(self._y_dis(b, b_)) < self.mean_height[bxs[i]["page_number"] - 1] / 5 \
+                    and dis >= dis_thr and b["x1"] < b_["x1"]:
+                # merge
+                bxs[i]["x1"] = b_["x1"]
+                bxs[i]["top"] = (b["top"] + b_["top"]) / 2
+                bxs[i]["bottom"] = (b["bottom"] + b_["bottom"]) / 2
+                bxs[i]["text"] += b_["text"]
+                bxs.pop(i + 1)
+                continue
+            i += 1
+        self.boxes = bxs
+
+    def _naive_vertical_merge(self):
+        bxs = Recognizer.sort_Y_firstly(
+            self.boxes, np.median(
+                self.mean_height) / 3)
+        i = 0
+        while i + 1 < len(bxs):
+            b = bxs[i]
+            b_ = bxs[i + 1]
+            if b["page_number"] < b_["page_number"] and re.match(
+                    r"[0-9  •一—-]+$", b["text"]):
+                bxs.pop(i)
+                continue
+            if not b["text"].strip():
+                bxs.pop(i)
+                continue
+            concatting_feats = [
+                b["text"].strip()[-1] in ",;:'\"，、‘“；：-",
+                len(b["text"].strip()) > 1 and b["text"].strip(
+                )[-2] in ",;:'\"，‘“、；：",
+                b["text"].strip()[0] in "。；？！?”）),，、：",
+            ]
+            # features for not concating
+            feats = [
+                b.get("layoutno", 0) != b_.get("layoutno", 0),
+                b["text"].strip()[-1] in "。？！?",
+                self.is_english and b["text"].strip()[-1] in ".!?",
+                b["page_number"] == b_["page_number"] and b_["top"] -
+                b["bottom"] > self.mean_height[b["page_number"] - 1] * 1.5,
+                b["page_number"] < b_["page_number"] and abs(
+                    b["x0"] - b_["x0"]) > self.mean_width[b["page_number"] - 1] * 4,
+            ]
+            # split features
+            detach_feats = [b["x1"] < b_["x0"],
+                            b["x0"] > b_["x1"]]
+            if (any(feats) and not any(concatting_feats)) or any(detach_feats):
+                print(
+                    b["text"],
+                    b_["text"],
+                    any(feats),
+                    any(concatting_feats),
+                    any(detach_feats))
+                i += 1
+                continue
+            # merge up and down
+            b["bottom"] = b_["bottom"]
+            b["text"] += b_["text"]
+            b["x0"] = min(b["x0"], b_["x0"])
+            b["x1"] = max(b["x1"], b_["x1"])
+            bxs.pop(i + 1)
+        self.boxes = bxs
+
+    def _concat_downward(self, concat_between_pages=True):
+        # count boxes in the same row as a feature
+        for i in range(len(self.boxes)):
+            mh = self.mean_height[self.boxes[i]["page_number"] - 1]
+            self.boxes[i]["in_row"] = 0
+            j = max(0, i - 12)
+            while j < min(i + 12, len(self.boxes)):
+                if j == i:
+                    j += 1
+                    continue
+                ydis = self._y_dis(self.boxes[i], self.boxes[j]) / mh
+                if abs(ydis) < 1:
+                    self.boxes[i]["in_row"] += 1
+                elif ydis > 0:
+                    break
+                j += 1
+
+        # concat between rows
+        boxes = deepcopy(self.boxes)
+        blocks = []
+        while boxes:
+            chunks = []
+
+            def dfs(up, dp):
+                chunks.append(up)
+                i = dp
+                while i < min(dp + 12, len(boxes)):
+                    ydis = self._y_dis(up, boxes[i])
+                    smpg = up["page_number"] == boxes[i]["page_number"]
+                    mh = self.mean_height[up["page_number"] - 1]
+                    mw = self.mean_width[up["page_number"] - 1]
+                    if smpg and ydis > mh * 4:
+                        break
+                    if not smpg and ydis > mh * 16:
+                        break
+                    down = boxes[i]
+                    if not concat_between_pages and down["page_number"] > up["page_number"]:
+                        break
+
+                    if up.get("R", "") != down.get(
+                            "R", "") and up["text"][-1] != "，":
+                        i += 1
+                        continue
+
+                    if re.match(r"[0-9]{2,3}/[0-9]{3}$", up["text"]) \
+                            or re.match(r"[0-9]{2,3}/[0-9]{3}$", down["text"]) \
+                            or not down["text"].strip():
+                        i += 1
+                        continue
+
+                    if not down["text"].strip():
+                        i += 1
+                        continue
+
+                    if up["x1"] < down["x0"] - 10 * \
+                            mw or up["x0"] > down["x1"] + 10 * mw:
+                        i += 1
+                        continue
+
+                    if i - dp < 5 and up.get("layout_type") == "text":
+                        if up.get("layoutno", "1") == down.get(
+                                "layoutno", "2"):
+                            dfs(down, i + 1)
+                            boxes.pop(i)
+                            return
+                        i += 1
+                        continue
+
+                    fea = self._updown_concat_features(up, down)
+                    if self.updown_cnt_mdl.predict(
+                            xgb.DMatrix([fea]))[0] <= 0.5:
+                        i += 1
+                        continue
+                    dfs(down, i + 1)
+                    boxes.pop(i)
+                    return
+
+            dfs(boxes[0], 1)
+            boxes.pop(0)
+            if chunks:
+                blocks.append(chunks)
+
+        # concat within each block
+        boxes = []
+        for b in blocks:
+            if len(b) == 1:
+                boxes.append(b[0])
+                continue
+            t = b[0]
+            for c in b[1:]:
+                t["text"] = t["text"].strip()
+                c["text"] = c["text"].strip()
+                if not c["text"]:
+                    continue
+                if t["text"] and re.match(
+                        r"[0-9\.a-zA-Z]+$", t["text"][-1] + c["text"][-1]):
+                    t["text"] += " "
+                t["text"] += c["text"]
+                t["x0"] = min(t["x0"], c["x0"])
+                t["x1"] = max(t["x1"], c["x1"])
+                t["page_number"] = min(t["page_number"], c["page_number"])
+                t["bottom"] = c["bottom"]
+                if not t["layout_type"] \
+                        and c["layout_type"]:
+                    t["layout_type"] = c["layout_type"]
+            boxes.append(t)
+
+        self.boxes = Recognizer.sort_Y_firstly(boxes, 0)
+
+    def _filter_forpages(self):
+        if not self.boxes:
+            return
+        findit = False
+        i = 0
+        while i < len(self.boxes):
+            if not re.match(r"(contents|目录|目次|table of contents|致谢|acknowledge)$",
+                            re.sub(r"( | |\u3000)+", "", self.boxes[i]["text"].lower())):
+                i += 1
+                continue
+            findit = True
+            eng = re.match(
+                r"[0-9a-zA-Z :'.-]{5,}",
+                self.boxes[i]["text"].strip())
+            self.boxes.pop(i)
+            if i >= len(self.boxes):
+                break
+            prefix = self.boxes[i]["text"].strip()[:3] if not eng else " ".join(
+                self.boxes[i]["text"].strip().split(" ")[:2])
+            while not prefix:
+                self.boxes.pop(i)
+                if i >= len(self.boxes):
+                    break
+                prefix = self.boxes[i]["text"].strip()[:3] if not eng else " ".join(
+                    self.boxes[i]["text"].strip().split(" ")[:2])
+            self.boxes.pop(i)
+            if i >= len(self.boxes) or not prefix:
+                break
+            for j in range(i, min(i + 128, len(self.boxes))):
+                if not re.match(prefix, self.boxes[j]["text"]):
+                    continue
+                for k in range(i, j):
+                    self.boxes.pop(i)
+                break
+        if findit:
+            return
+
+        page_dirty = [0] * len(self.page_images)
+        for b in self.boxes:
+            if re.search(r"(··|··|··)", b["text"]):
+                page_dirty[b["page_number"] - 1] += 1
+        page_dirty = set([i + 1 for i, t in enumerate(page_dirty) if t > 3])
+        if not page_dirty:
+            return
+        i = 0
+        while i < len(self.boxes):
+            if self.boxes[i]["page_number"] in page_dirty:
+                self.boxes.pop(i)
+                continue
+            i += 1
+
+    def _merge_with_same_bullet(self):
+        i = 0
+        while i + 1 < len(self.boxes):
+            b = self.boxes[i]
+            b_ = self.boxes[i + 1]
+            if not b["text"].strip():
+                self.boxes.pop(i)
+                continue
+            if not b_["text"].strip():
+                self.boxes.pop(i + 1)
+                continue
+
+            if b["text"].strip()[0] != b_["text"].strip()[0] \
+                    or b["text"].strip()[0].lower() in set("qwertyuopasdfghjklzxcvbnm") \
+                    or rag_tokenizer.is_chinese(b["text"].strip()[0]) \
+                    or b["top"] > b_["bottom"]:
+                i += 1
+                continue
+            b_["text"] = b["text"] + "\n" + b_["text"]
+            b_["x0"] = min(b["x0"], b_["x0"])
+            b_["x1"] = max(b["x1"], b_["x1"])
+            b_["top"] = b["top"]
+            self.boxes.pop(i)
+
+    def _extract_table_figure(self, need_image, ZM,
+                              return_html, need_position):
+        tables = {}
+        figures = {}
+        # extract figure and table boxes
+        i = 0
+        lst_lout_no = ""
+        nomerge_lout_no = []
+        while i < len(self.boxes):
+            if "layoutno" not in self.boxes[i]:
+                i += 1
+                continue
+            lout_no = str(self.boxes[i]["page_number"]) + \
+                      "-" + str(self.boxes[i]["layoutno"])
+            if TableStructureRecognizer.is_caption(self.boxes[i]) or self.boxes[i]["layout_type"] in ["table caption",
+                                                                                                      "title",
+                                                                                                      "figure caption",
+                                                                                                      "reference"]:
+                nomerge_lout_no.append(lst_lout_no)
+            if self.boxes[i]["layout_type"] == "table":
+                if re.match(r"(数据|资料|图表)*来源[:： ]", self.boxes[i]["text"]):
+                    self.boxes.pop(i)
+                    continue
+                if lout_no not in tables:
+                    tables[lout_no] = []
+                tables[lout_no].append(self.boxes[i])
+                self.boxes.pop(i)
+                lst_lout_no = lout_no
+                continue
+            if need_image and self.boxes[i]["layout_type"] == "figure":
+                if re.match(r"(数据|资料|图表)*来源[:： ]", self.boxes[i]["text"]):
+                    self.boxes.pop(i)
+                    continue
+                if lout_no not in figures:
+                    figures[lout_no] = []
+                figures[lout_no].append(self.boxes[i])
+                self.boxes.pop(i)
+                lst_lout_no = lout_no
+                continue
+            i += 1
+
+        # merge table on different pages
+        nomerge_lout_no = set(nomerge_lout_no)
+        tbls = sorted([(k, bxs) for k, bxs in tables.items()],
+                      key=lambda x: (x[1][0]["top"], x[1][0]["x0"]))
+
+        i = len(tbls) - 1
+        while i - 1 >= 0:
+            k0, bxs0 = tbls[i - 1]
+            k, bxs = tbls[i]
+            i -= 1
+            if k0 in nomerge_lout_no:
+                continue
+            if bxs[0]["page_number"] == bxs0[0]["page_number"]:
+                continue
+            if bxs[0]["page_number"] - bxs0[0]["page_number"] > 1:
+                continue
+            mh = self.mean_height[bxs[0]["page_number"] - 1]
+            if self._y_dis(bxs0[-1], bxs[0]) > mh * 23:
+                continue
+            tables[k0].extend(tables[k])
+            del tables[k]
+
+        def x_overlapped(a, b):
+            return not any([a["x1"] < b["x0"], a["x0"] > b["x1"]])
+
+        # find captions and pop out
+        i = 0
+        while i < len(self.boxes):
+            c = self.boxes[i]
+            # mh = self.mean_height[c["page_number"]-1]
+            if not TableStructureRecognizer.is_caption(c):
+                i += 1
+                continue
+
+            # find the nearest layouts
+            def nearest(tbls):
+                nonlocal c
+                mink = ""
+                minv = 1000000000
+                for k, bxs in tbls.items():
+                    for b in bxs:
+                        if b.get("layout_type", "").find("caption") >= 0:
+                            continue
+                        y_dis = self._y_dis(c, b)
+                        x_dis = self._x_dis(
+                            c, b) if not x_overlapped(
+                            c, b) else 0
+                        dis = y_dis * y_dis + x_dis * x_dis
+                        if dis < minv:
+                            mink = k
+                            minv = dis
+                return mink, minv
+
+            tk, tv = nearest(tables)
+            fk, fv = nearest(figures)
+            # if min(tv, fv) > 2000:
+            #    i += 1
+            #    continue
+            if tv < fv and tk:
+                tables[tk].insert(0, c)
+                logging.debug(
+                    "TABLE:" +
+                    self.boxes[i]["text"] +
+                    "; Cap: " +
+                    tk)
+            elif fk:
+                figures[fk].insert(0, c)
+                logging.debug(
+                    "FIGURE:" +
+                    self.boxes[i]["text"] +
+                    "; Cap: " +
+                    tk)
+            self.boxes.pop(i)
+
+        res = []
+        positions = []
+
+        def cropout(bxs, ltype, poss):
+            nonlocal ZM
+            pn = set([b["page_number"] - 1 for b in bxs])
+            if len(pn) < 2:
+                pn = list(pn)[0]
+                ht = self.page_cum_height[pn]
+                b = {
+                    "x0": np.min([b["x0"] for b in bxs]),
+                    "top": np.min([b["top"] for b in bxs]) - ht,
+                    "x1": np.max([b["x1"] for b in bxs]),
+                    "bottom": np.max([b["bottom"] for b in bxs]) - ht
+                }
+                louts = [l for l in self.page_layout[pn] if l["type"] == ltype]
+                ii = Recognizer.find_overlapped(b, louts, naive=True)
+                if ii is not None:
+                    b = louts[ii]
+                else:
+                    logging.warn(
+                        f"Missing layout match: {pn + 1},%s" %
+                        (bxs[0].get(
+                            "layoutno", "")))
+
+                left, top, right, bott = b["x0"], b["top"], b["x1"], b["bottom"]
+                if right < left: right = left + 1
+                poss.append((pn + self.page_from, left, right, top, bott))
+                return self.page_images[pn] \
+                    .crop((left * ZM, top * ZM,
+                           right * ZM, bott * ZM))
+            pn = {}
+            for b in bxs:
+                p = b["page_number"] - 1
+                if p not in pn:
+                    pn[p] = []
+                pn[p].append(b)
+            pn = sorted(pn.items(), key=lambda x: x[0])
+            imgs = [cropout(arr, ltype, poss) for p, arr in pn]
+            pic = Image.new("RGB",
+                            (int(np.max([i.size[0] for i in imgs])),
+                             int(np.sum([m.size[1] for m in imgs]))),
+                            (245, 245, 245))
+            height = 0
+            for img in imgs:
+                pic.paste(img, (0, int(height)))
+                height += img.size[1]
+            return pic
+
+        # crop figure out and add caption
+        for k, bxs in figures.items():
+            txt = "\n".join([b["text"] for b in bxs])
+            if not txt:
+                continue
+
+            poss = []
+            res.append(
+                (cropout(
+                    bxs,
+                    "figure", poss),
+                 [txt]))
+            positions.append(poss)
+
+        for k, bxs in tables.items():
+            if not bxs:
+                continue
+            bxs = Recognizer.sort_Y_firstly(bxs, np.mean(
+                [(b["bottom"] - b["top"]) / 2 for b in bxs]))
+            poss = []
+            res.append((cropout(bxs, "table", poss),
+                        self.tbl_det.construct_table(bxs, html=return_html, is_english=self.is_english)))
+            positions.append(poss)
+
+        assert len(positions) == len(res)
+
+        if need_position:
+            return list(zip(res, positions))
+        return res
+
+    def proj_match(self, line):
+        if len(line) <= 2:
+            return
+        if re.match(r"[0-9 ().,%%+/-]+$", line):
+            return False
+        for p, j in [
+            (r"第[零一二三四五六七八九十百]+章", 1),
+            (r"第[零一二三四五六七八九十百]+[条节]", 2),
+            (r"[零一二三四五六七八九十百]+[、 　]", 3),
+            (r"[\(（][零一二三四五六七八九十百]+[）\)]", 4),
+            (r"[0-9]+(、|\.[　 ]|\.[^0-9])", 5),
+            (r"[0-9]+\.[0-9]+(、|[. 　]|[^0-9])", 6),
+            (r"[0-9]+\.[0-9]+\.[0-9]+(、|[ 　]|[^0-9])", 7),
+            (r"[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+(、|[ 　]|[^0-9])", 8),
+            (r".{,48}[：:?？]$", 9),
+            (r"[0-9]+）", 10),
+            (r"[\(（][0-9]+[）\)]", 11),
+            (r"[零一二三四五六七八九十百]+是", 12),
+            (r"[⚫•➢✓]", 12)
+        ]:
+            if re.match(p, line):
+                return j
+        return
+
+    def _line_tag(self, bx, ZM):
+        pn = [bx["page_number"]]
+        top = bx["top"] - self.page_cum_height[pn[0] - 1]
+        bott = bx["bottom"] - self.page_cum_height[pn[0] - 1]
+        page_images_cnt = len(self.page_images)
+        if pn[-1] - 1 >= page_images_cnt: return ""
+        while bott * ZM > self.page_images[pn[-1] - 1].size[1]:
+            bott -= self.page_images[pn[-1] - 1].size[1] / ZM
+            pn.append(pn[-1] + 1)
+            if pn[-1] - 1 >= page_images_cnt:
+                return ""
+
+        return "@@{}\t{:.1f}\t{:.1f}\t{:.1f}\t{:.1f}##" \
+            .format("-".join([str(p) for p in pn]),
+                    bx["x0"], bx["x1"], top, bott)
+
+    def __filterout_scraps(self, boxes, ZM):
+
+        def width(b):
+            return b["x1"] - b["x0"]
+
+        def height(b):
+            return b["bottom"] - b["top"]
+
+        def usefull(b):
+            if b.get("layout_type"):
+                return True
+            if width(
+                    b) > self.page_images[b["page_number"] - 1].size[0] / ZM / 3:
+                return True
+            if b["bottom"] - b["top"] > self.mean_height[b["page_number"] - 1]:
+                return True
+            return False
+
+        res = []
+        while boxes:
+            lines = []
+            widths = []
+            pw = self.page_images[boxes[0]["page_number"] - 1].size[0] / ZM
+            mh = self.mean_height[boxes[0]["page_number"] - 1]
+            mj = self.proj_match(
+                boxes[0]["text"]) or boxes[0].get(
+                "layout_type",
+                "") == "title"
+
+            def dfs(line, st):
+                nonlocal mh, pw, lines, widths
+                lines.append(line)
+                widths.append(width(line))
+                width_mean = np.mean(widths)
+                mmj = self.proj_match(
+                    line["text"]) or line.get(
+                    "layout_type",
+                    "") == "title"
+                for i in range(st + 1, min(st + 20, len(boxes))):
+                    if (boxes[i]["page_number"] - line["page_number"]) > 0:
+                        break
+                    if not mmj and self._y_dis(
+                            line, boxes[i]) >= 3 * mh and height(line) < 1.5 * mh:
+                        break
+
+                    if not usefull(boxes[i]):
+                        continue
+                    if mmj or \
+                            (self._x_dis(boxes[i], line) < pw / 10): \
+                            # and abs(width(boxes[i])-width_mean)/max(width(boxes[i]),width_mean)<0.5):
+                        # concat following
+                        dfs(boxes[i], i)
+                        boxes.pop(i)
+                        break
+
+            try:
+                if usefull(boxes[0]):
+                    dfs(boxes[0], 0)
+                else:
+                    logging.debug("WASTE: " + boxes[0]["text"])
+            except Exception as e:
+                pass
+            boxes.pop(0)
+            mw = np.mean(widths)
+            if mj or mw / pw >= 0.35 or mw > 200:
+                res.append(
+                    "\n".join([c["text"] + self._line_tag(c, ZM) for c in lines]))
+            else:
+                logging.debug("REMOVED: " +
+                              "<<".join([c["text"] for c in lines]))
+
+        return "\n\n".join(res)
+
+    @staticmethod
+    def total_page_number(fnm, binary=None):
+        try:
+            pdf = pdfplumber.open(
+                fnm) if not binary else pdfplumber.open(BytesIO(binary))
+            return len(pdf.pages)
+        except Exception as e:
+            logging.error(str(e))
+
+    def __images__(self, fnm, zoomin=3, page_from=0,
+                   page_to=299, callback=None):
+        self.lefted_chars = []
+        self.mean_height = []
+        self.mean_width = []
+        self.boxes = []
+        self.garbages = {}
+        self.page_cum_height = [0]
+        self.page_layout = []
+        self.page_from = page_from
+        st = timer()
+        try:
+            self.pdf = pdfplumber.open(fnm) if isinstance(
+                fnm, str) else pdfplumber.open(BytesIO(fnm))
+            self.page_images = [p.to_image(resolution=72 * zoomin).annotated for i, p in
+                                enumerate(self.pdf.pages[page_from:page_to])]
+            self.page_chars = [[c for c in page.chars if self._has_color(c)] for page in
+                               self.pdf.pages[page_from:page_to]]
+            self.total_page = len(self.pdf.pages)
+        except Exception as e:
+            logging.error(str(e))
+
+        self.outlines = []
+        try:
+            self.pdf = pdf2_read(fnm if isinstance(fnm, str) else BytesIO(fnm))
+            outlines = self.pdf.outline
+
+            def dfs(arr, depth):
+                for a in arr:
+                    if isinstance(a, dict):
+                        self.outlines.append((a["/Title"], depth))
+                        continue
+                    dfs(a, depth + 1)
+
+            dfs(outlines, 0)
+        except Exception as e:
+            logging.warning(f"Outlines exception: {e}")
+        if not self.outlines:
+            logging.warning(f"Miss outlines")
+
+        logging.info("Images converted.")
+        self.is_english = [re.search(r"[a-zA-Z0-9,/¸;:'\[\]\(\)!@#$%^&*\"?<>._-]{30,}", "".join(
+            random.choices([c["text"] for c in self.page_chars[i]], k=min(100, len(self.page_chars[i]))))) for i in
+                           range(len(self.page_chars))]
+        if sum([1 if e else 0 for e in self.is_english]) > len(
+                self.page_images) / 2:
+            self.is_english = True
+        else:
+            self.is_english = False
+        self.is_english = False
+
+        st = timer()
+        for i, img in enumerate(self.page_images):
+            chars = self.page_chars[i] if not self.is_english else []
+            self.mean_height.append(
+                np.median(sorted([c["height"] for c in chars])) if chars else 0
+            )
+            self.mean_width.append(
+                np.median(sorted([c["width"] for c in chars])) if chars else 8
+            )
+            self.page_cum_height.append(img.size[1] / zoomin)
+            j = 0
+            while j + 1 < len(chars):
+                if chars[j]["text"] and chars[j + 1]["text"] \
+                        and re.match(r"[0-9a-zA-Z,.:;!%]+", chars[j]["text"] + chars[j + 1]["text"]) \
+                        and chars[j + 1]["x0"] - chars[j]["x1"] >= min(chars[j + 1]["width"],
+                                                                       chars[j]["width"]) / 2:
+                    chars[j]["text"] += " "
+                j += 1
+
+            self.__ocr(i + 1, img, chars, zoomin)
+            if callback and i % 6 == 5:
+                callback(prog=(i + 1) * 0.6 / len(self.page_images), msg="")
+        # print("OCR:", timer()-st)
+
+        if not self.is_english and not any(
+                [c for c in self.page_chars]) and self.boxes:
+            bxes = [b for bxs in self.boxes for b in bxs]
+            self.is_english = re.search(r"[\na-zA-Z0-9,/¸;:'\[\]\(\)!@#$%^&*\"?<>._-]{30,}",
+                                        "".join([b["text"] for b in random.choices(bxes, k=min(30, len(bxes)))]))
+
+        logging.info(f"Is it English: {self.is_english}")
+
+        self.page_cum_height = np.cumsum(self.page_cum_height)
+        assert len(self.page_cum_height) == len(self.page_images) + 1
+
+    def __call__(self, fnm, need_image=False, zoomin=3, return_html=False):
+        self.__images__(fnm, zoomin)
+        self._layouts_rec(zoomin)
+        self._table_transformer_job(zoomin)
+        self._text_merge()
+        self._concat_downward()
+        self._filter_forpages()
+        tbls = self._extract_table_figure(
+            need_image, zoomin, return_html, False)
+        return self.__filterout_scraps(deepcopy(self.boxes), zoomin), tbls
+
+    def remove_tag(self, txt):
+        return re.sub(r"@@[\t0-9.-]+?##", "", txt)
+
+    def crop(self, text, ZM=3, need_position=False):
+        imgs = []
+        poss = []
+        for tag in re.findall(r"@@[0-9-]+\t[0-9.\t]+##", text):
+            pn, left, right, top, bottom = tag.strip(
+                "#").strip("@").split("\t")
+            left, right, top, bottom = float(left), float(
+                right), float(top), float(bottom)
+            poss.append(([int(p) - 1 for p in pn.split("-")],
+                         left, right, top, bottom))
+        if not poss:
+            if need_position:
+                return None, None
+            return
+
+        max_width = max(
+            np.max([right - left for (_, left, right, _, _) in poss]), 6)
+        GAP = 6
+        pos = poss[0]
+        poss.insert(0, ([pos[0][0]], pos[1], pos[2], max(
+            0, pos[3] - 120), max(pos[3] - GAP, 0)))
+        pos = poss[-1]
+        poss.append(([pos[0][-1]], pos[1], pos[2], min(self.page_images[pos[0][-1]].size[1] / ZM, pos[4] + GAP),
+                     min(self.page_images[pos[0][-1]].size[1] / ZM, pos[4] + 120)))
+
+        positions = []
+        for ii, (pns, left, right, top, bottom) in enumerate(poss):
+            right = left + max_width
+            bottom *= ZM
+            for pn in pns[1:]:
+                bottom += self.page_images[pn - 1].size[1]
+            imgs.append(
+                self.page_images[pns[0]].crop((left * ZM, top * ZM,
+                                               right *
+                                               ZM, min(
+                    bottom, self.page_images[pns[0]].size[1])
+                                               ))
+            )
+            if 0 < ii < len(poss) - 1:
+                positions.append((pns[0] + self.page_from, left, right, top, min(
+                    bottom, self.page_images[pns[0]].size[1]) / ZM))
+            bottom -= self.page_images[pns[0]].size[1]
+            for pn in pns[1:]:
+                imgs.append(
+                    self.page_images[pn].crop((left * ZM, 0,
+                                               right * ZM,
+                                               min(bottom,
+                                                   self.page_images[pn].size[1])
+                                               ))
+                )
+                if 0 < ii < len(poss) - 1:
+                    positions.append((pn + self.page_from, left, right, 0, min(
+                        bottom, self.page_images[pn].size[1]) / ZM))
+                bottom -= self.page_images[pn].size[1]
+
+        if not imgs:
+            if need_position:
+                return None, None
+            return
+        height = 0
+        for img in imgs:
+            height += img.size[1] + GAP
+        height = int(height)
+        width = int(np.max([i.size[0] for i in imgs]))
+        pic = Image.new("RGB",
+                        (width, height),
+                        (245, 245, 245))
+        height = 0
+        for ii, img in enumerate(imgs):
+            if ii == 0 or ii + 1 == len(imgs):
+                img = img.convert('RGBA')
+                overlay = Image.new('RGBA', img.size, (0, 0, 0, 0))
+                overlay.putalpha(128)
+                img = Image.alpha_composite(img, overlay).convert("RGB")
+            pic.paste(img, (0, int(height)))
+            height += img.size[1] + GAP
+
+        if need_position:
+            return pic, positions
+        return pic
+
+    def get_position(self, bx, ZM):
+        poss = []
+        pn = bx["page_number"]
+        top = bx["top"] - self.page_cum_height[pn - 1]
+        bott = bx["bottom"] - self.page_cum_height[pn - 1]
+        poss.append((pn, bx["x0"], bx["x1"], top, min(
+            bott, self.page_images[pn - 1].size[1] / ZM)))
+        while bott * ZM > self.page_images[pn - 1].size[1]:
+            bott -= self.page_images[pn - 1].size[1] / ZM
+            top = 0
+            pn += 1
+            poss.append((pn, bx["x0"], bx["x1"], top, min(
+                bott, self.page_images[pn - 1].size[1] / ZM)))
+        return poss
+
+
+class PlainParser(object):
+    def __call__(self, filename, from_page=0, to_page=100000, **kwargs):
+        self.outlines = []
+        lines = []
+        try:
+            self.pdf = pdf2_read(
+                filename if isinstance(
+                    filename, str) else BytesIO(filename))
+            for page in self.pdf.pages[from_page:to_page]:
+                lines.extend([t for t in page.extract_text().split("\n")])
+
+            outlines = self.pdf.outline
+
+            def dfs(arr, depth):
+                for a in arr:
+                    if isinstance(a, dict):
+                        self.outlines.append((a["/Title"], depth))
+                        continue
+                    dfs(a, depth + 1)
+
+            dfs(outlines, 0)
+        except Exception as e:
+            logging.warning(f"Outlines exception: {e}")
+        if not self.outlines:
+            logging.warning(f"Miss outlines")
+
+        return [(l, "") for l in lines], []
+
+    def crop(self, ck, need_position):
+        raise NotImplementedError
+
+    @staticmethod
+    def remove_tag(txt):
+        raise NotImplementedError
+
+
+if __name__ == "__main__":
+    pass

deepdoc/parser/ppt_parser.py

@@ -0,0 +1,60 @@
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+from io import BytesIO
+from pptx import Presentation
+
+
+class RAGFlowPptParser(object):
+    def __init__(self):
+        super().__init__()
+
+    def __extract(self, shape):
+        if shape.shape_type == 19:
+            tb = shape.table
+            rows = []
+            for i in range(1, len(tb.rows)):
+                rows.append("; ".join([tb.cell(
+                    0, j).text + ": " + tb.cell(i, j).text for j in range(len(tb.columns)) if tb.cell(i, j)]))
+            return "\n".join(rows)
+
+        if shape.has_text_frame:
+            return shape.text_frame.text
+
+        if shape.shape_type == 6:
+            texts = []
+            for p in sorted(shape.shapes, key=lambda x: (x.top // 10, x.left)):
+                t = self.__extract(p)
+                if t:
+                    texts.append(t)
+            return "\n".join(texts)
+
+    def __call__(self, fnm, from_page, to_page, callback=None):
+        ppt = Presentation(fnm) if isinstance(
+            fnm, str) else Presentation(
+            BytesIO(fnm))
+        txts = []
+        self.total_page = len(ppt.slides)
+        for i, slide in enumerate(ppt.slides):
+            if i < from_page:
+                continue
+            if i >= to_page:
+                break
+            texts = []
+            for shape in sorted(
+                    slide.shapes, key=lambda x: (x.top // 10, x.left)):
+                txt = self.__extract(shape)
+                if txt:
+                    texts.append(txt)
+            txts.append("\n".join(texts))
+
+        return txts

deepdoc/parser/readme_parse.md

@@ -0,0 +1,27 @@
+使用 `RAGFlowPdfParser` 类可以从 PDF 文件中解析出文本和表格等内容。以下是如何使用这个类的步骤：
+
+1. **初始化解析器**：
+   创建 `RAGFlowPdfParser` 的实例。
+   ```python
+   pdf_parser = RAGFlowPdfParser()
+   ```
+
+2. **调用解析器**：
+   使用 PDF 文件的路径或文件对象调用解析器。你可以指定是否需要图像、放大比例、是否返回 HTML 格式的表格等参数。
+   ```python
+   pdf_file_path = 'path_to_your_pdf_file.pdf'
+   text_content, tables = pdf_parser(pdf_file_path, need_image=True, zoomin=3, return_html=False)
+   ```
+
+   这里 `text_content` 将包含从 PDF 提取的文本，而 `tables` 将包含提取的表格。
+
+3. **处理结果**：
+   `text_content` 和 `tables` 包含解析结果，可以根据需要进一步处理。例如，可以打印文本内容或者处理表格数据。
+
+4. **高级功能**：
+   - 使用 `crop` 方法从 PDF 中裁剪特定文本或区域。
+   - 使用 `remove_tag` 方法移除从文本中提取的特定标记。
+
+这个类高度依赖外部库和自定义的模型，包括 OCR、布局识别和表格结构识别等。确保这些依赖和相关资源都已正确安装和配置。
+
+此外，如果有特定的需求或需要处理的特殊 PDF 文件格式，可能还需要对解析器进行适当的调整或优化。

deepdoc/parser/readpdf.py

@@ -0,0 +1,62 @@
+from deepdoc.parser import RAGFlowPdfParser, PlainParser
+import os
+from PIL import Image
+import json
+class PDFprocess():
+    def __init__(self, repodir, workdir):
+        self.repodir = repodir
+        self.preprocessdir = os.path.join(workdir,'preprocess')
+        self.pdf_parser = RAGFlowPdfParser()
+        # self.plain_parser = PlainParser()
+            
+    def _save_image(self,image, path, name):
+        """ 保存图片到指定路径 """
+        if not os.path.exists(path):
+            os.makedirs(path)
+        image_path = os.path.join(path, name)
+        image.save(image_path)
+        return image_path
+
+    def save_all_image(self,preprocessdir,tables):
+        image_folder = os.path.join(preprocessdir,'saved_images')
+        # 假设 res 中包含了图片对象和其他数据
+        for index, data in enumerate(tables):
+            image, text = data  # 假设 data 结构是这样的
+            image_path = self._save_image(image, image_folder, f'image_{index}.png')
+            # relative_path = os.path.relpath(image_path, preprocessdir)
+            tables[index] = (image_path, text)  # 更新 res 中的图片对象为图片路径
+        return tables
+
+    def create_html_file(self,tables, html_file_path):
+        html_content = '<html><body>\n'
+        for index, data in enumerate(tables):
+            image_path, text = data
+            # 创建图片链接和文本
+            html_content += f'<img src="{image_path}" alt="Image">\n{text}\n'
+        html_content += '</body></html>'
+        
+        # 写入 HTML 文件
+        with open(html_file_path, 'w') as file:
+            file.write(html_content)
+    def process(self, pdffilename):
+        pdf_file_path = os.path.join(self.repodir,pdffilename)
+        text_content, tables = self.pdf_parser(pdf_file_path, need_image=False, zoomin=3, return_html=True)
+
+        text_file_path = os.path.join(self.preprocessdir,pdffilename.replace('.pdf','.txt'))
+        with open(text_file_path, 'w') as f:
+            f.write(text_content)
+
+        image_folder = os.path.join(self.preprocessdir,f'{pdffilename}_images')
+        tables = self.save_all_image(image_folder,tables)
+
+        html_file_path = os.path.join(self.preprocessdir,pdffilename.replace('.pdf','.html'))
+        self.create_html_file(tables, html_file_path)
+
+        json_file_path = os.path.join(self.preprocessdir,pdffilename.replace('.pdf','.json'))
+        with open(json_file_path, 'w') as f:
+            json.dump(tables, f, indent=4, ensure_ascii=False)
+if __name__ == '__main__':
+    repodir = '/Users/chen/Downloads/ReviewAgent2'
+    preprocessdir = '/Users/chen/Downloads/ReviewAgent2/preprocess'
+    pdfprocess = PDFprocess(repodir, preprocessdir)
+    pdfprocess.process('test.pdf')

deepdoc/parser/resume/__init__.py

@@ -0,0 +1,52 @@
+import datetime
+
+
+def refactor(cv):
+    for n in ["raw_txt", "parser_name", "inference", "ori_text", "use_time", "time_stat"]:
+        if n in cv and cv[n] is not None: del cv[n]
+    cv["is_deleted"] = 0
+    if "basic" not in cv: cv["basic"] = {}
+    if cv["basic"].get("photo2"): del cv["basic"]["photo2"]
+
+    for n in ["education", "work", "certificate", "project", "language", "skill", "training"]:
+        if n not in cv or cv[n] is None: continue
+        if type(cv[n]) == type({}): cv[n] = [v for _, v in cv[n].items()]
+        if type(cv[n]) != type([]):
+            del cv[n]
+            continue
+        vv = []
+        for v in cv[n]:
+            if "external" in v and v["external"] is not None: del v["external"]
+            vv.append(v)
+        cv[n] = {str(i): vv[i] for i in range(len(vv))}
+
+    basics = [
+        ("basic_salary_month", "salary_month"),
+        ("expect_annual_salary_from", "expect_annual_salary"),
+    ]
+    for n, t in basics:
+        if cv["basic"].get(n):
+            cv["basic"][t] = cv["basic"][n]
+            del cv["basic"][n]
+
+    work = sorted([v for _, v in cv.get("work", {}).items()], key=lambda x: x.get("start_time", ""))
+    edu = sorted([v for _, v in cv.get("education", {}).items()], key=lambda x: x.get("start_time", ""))
+
+    if work:
+        cv["basic"]["work_start_time"] = work[0].get("start_time", "")
+        cv["basic"]["management_experience"] = 'Y' if any(
+            [w.get("management_experience", '') == 'Y' for w in work]) else 'N'
+        cv["basic"]["annual_salary"] = work[-1].get("annual_salary_from", "0")
+
+        for n in ["annual_salary_from", "annual_salary_to", "industry_name", "position_name", "responsibilities",
+                  "corporation_type", "scale", "corporation_name"]:
+            cv["basic"][n] = work[-1].get(n, "")
+
+    if edu:
+        for n in ["school_name", "discipline_name"]:
+            if n in edu[-1]: cv["basic"][n] = edu[-1][n]
+
+    cv["basic"]["updated_at"] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+    if "contact" not in cv: cv["contact"] = {}
+    if not cv["contact"].get("name"): cv["contact"]["name"] = cv["basic"].get("name", "")
+    return cv

deepdoc/parser/resume/entities/corporations.py

@@ -0,0 +1,80 @@
+import re,json,os
+import pandas as pd
+from rag.nlp import rag_tokenizer
+from . import regions
+current_file_path = os.path.dirname(os.path.abspath(__file__))
+GOODS = pd.read_csv(os.path.join(current_file_path, "res/corp_baike_len.csv"), sep="\t", header=0).fillna(0)
+GOODS["cid"] = GOODS["cid"].astype(str)
+GOODS = GOODS.set_index(["cid"])
+CORP_TKS = json.load(open(os.path.join(current_file_path, "res/corp.tks.freq.json"), "r"))
+GOOD_CORP = json.load(open(os.path.join(current_file_path, "res/good_corp.json"), "r"))
+CORP_TAG = json.load(open(os.path.join(current_file_path, "res/corp_tag.json"), "r"))
+
+def baike(cid, default_v=0):
+    global GOODS
+    try:
+        return GOODS.loc[str(cid), "len"]
+    except Exception as e:
+        pass
+    return default_v
+
+
+def corpNorm(nm, add_region=True):
+    global CORP_TKS
+    if not nm or type(nm)!=type(""):return ""
+    nm = rag_tokenizer.tradi2simp(rag_tokenizer.strQ2B(nm)).lower()
+    nm = re.sub(r"&", "&", nm)
+    nm = re.sub(r"[\(\)（）\+'\"\t \*\\【】-]+", " ", nm)
+    nm = re.sub(r"([—-]+.*| +co\..*|corp\..*| +inc\..*| +ltd.*)", "", nm, 10000, re.IGNORECASE)
+    nm = re.sub(r"(计算机|技术|(技术|科技|网络)*有限公司|公司|有限|研发中心|中国|总部)$", "", nm, 10000, re.IGNORECASE)
+    if not nm or (len(nm)<5 and not regions.isName(nm[0:2])):return nm
+
+    tks = rag_tokenizer.tokenize(nm).split(" ")
+    reg = [t for i,t in enumerate(tks) if regions.isName(t) and (t != "中国" or i > 0)]
+    nm = ""
+    for t in tks:
+        if regions.isName(t) or t in CORP_TKS:continue
+        if re.match(r"[0-9a-zA-Z\\,.]+", t) and re.match(r".*[0-9a-zA-Z\,.]+$", nm):nm += " "
+        nm += t
+
+    r = re.search(r"^([^a-z0-9 \(\)&]{2,})[a-z ]{4,}$", nm.strip())
+    if r:nm = r.group(1)
+    r = re.search(r"^([a-z ]{3,})[^a-z0-9 \(\)&]{2,}$", nm.strip())
+    if r:nm = r.group(1)
+    return nm.strip() + (("" if not reg else "(%s)"%reg[0]) if add_region else "")
+
+
+def rmNoise(n):
+    n = re.sub(r"[\(（][^()（）]+[)）]", "", n)
+    n = re.sub(r"[,. &（）()]+", "", n)
+    return n
+
+GOOD_CORP = set([corpNorm(rmNoise(c), False) for c in GOOD_CORP])
+for c,v in CORP_TAG.items():
+    cc = corpNorm(rmNoise(c), False)
+    if not cc: print (c)
+CORP_TAG = {corpNorm(rmNoise(c), False):v for c,v in CORP_TAG.items()}
+
+def is_good(nm):
+    global GOOD_CORP
+    if nm.find("外派")>=0:return False
+    nm = rmNoise(nm)
+    nm = corpNorm(nm, False)
+    for n in GOOD_CORP:
+        if re.match(r"[0-9a-zA-Z]+$", n):
+            if n == nm: return True
+        elif nm.find(n)>=0:return True
+    return False
+
+def corp_tag(nm):
+    global CORP_TAG
+    nm = rmNoise(nm)
+    nm = corpNorm(nm, False)
+    for n in CORP_TAG.keys():
+        if re.match(r"[0-9a-zA-Z., ]+$", n):
+            if n == nm: return CORP_TAG[n]
+        elif nm.find(n)>=0:
+            if len(n)<3 and len(nm)/len(n)>=2:continue
+            return CORP_TAG[n]
+    return []
+

deepdoc/parser/resume/entities/degrees.py

@@ -0,0 +1,24 @@
+TBL = {"94":"EMBA",
+"6":"MBA",
+"95":"MPA",
+"92":"专升本",
+"4":"专科",
+"90":"中专",
+"91":"中技",
+"86":"初中",
+"3":"博士",
+"10":"博士后",
+"1":"本科",
+"2":"硕士",
+"87":"职高",
+"89":"高中"
+}
+
+TBL_ = {v:k for k,v in TBL.items()}
+
+def get_name(id):
+    return TBL.get(str(id), "")
+
+def get_id(nm):
+    if not nm:return ""
+    return TBL_.get(nm.upper().strip(), "")

deepdoc/parser/resume/entities/industries.py

@@ -0,0 +1,692 @@
+
+TBL = {"1":{"name":"IT/通信/电子","parent":"0"},
+"2":{"name":"互联网","parent":"0"},
+"3":{"name":"电子商务","parent":"2"},
+"4":{"name":"互联网金融","parent":"2"},
+"5":{"name":"网络游戏","parent":"2"},
+"6":{"name":"社交网络平台","parent":"2"},
+"7":{"name":"视频音乐","parent":"2"},
+"9":{"name":"安全","parent":"2"},
+"10":{"name":"云计算","parent":"2"},
+"12":{"name":"工具类客户端应用","parent":"2"},
+"13":{"name":"互联网广告","parent":"2"},
+"14":{"name":"企业互联网服务","parent":"2"},
+"16":{"name":"在线教育","parent":"2"},
+"17":{"name":"在线医疗","parent":"2"},
+"19":{"name":"B2B","parent":"3"},
+"20":{"name":"B2C","parent":"3"},
+"21":{"name":"C2C","parent":"3"},
+"22":{"name":"生活信息本地化","parent":"3"},
+"23":{"name":"在线旅游","parent":"2"},
+"24":{"name":"第三方支付","parent":"4"},
+"26":{"name":"客户端游戏","parent":"5"},
+"27":{"name":"网页游戏","parent":"5"},
+"28":{"name":"手机游戏","parent":"5"},
+"29":{"name":"微博","parent":"6"},
+"30":{"name":"社交网站","parent":"6"},
+"31":{"name":"在线视频","parent":"7"},
+"32":{"name":"在线音乐","parent":"7"},
+"35":{"name":"企业安全","parent":"9"},
+"36":{"name":"个人安全","parent":"9"},
+"37":{"name":"企业级云服务","parent":"10"},
+"38":{"name":"个人级云服务","parent":"10"},
+"43":{"name":"输入法","parent":"12"},
+"44":{"name":"浏览器","parent":"12"},
+"45":{"name":"词典","parent":"12"},
+"46":{"name":"播放器","parent":"12"},
+"47":{"name":"下载器","parent":"12"},
+"48":{"name":"IM","parent":"12"},
+"49":{"name":"广告服务","parent":"13"},
+"50":{"name":"第三方广告网络平台","parent":"13"},
+"51":{"name":"媒体代理","parent":"13"},
+"52":{"name":"创意代理","parent":"13"},
+"53":{"name":"IT-综合","parent":"1"},
+"71":{"name":"团购","parent":"3"},
+"72":{"name":"地图","parent":"2"},
+"73":{"name":"数据存储","parent":"2"},
+"414":{"name":"计算机软件","parent":"1"},
+"415":{"name":"计算机硬件","parent":"1"},
+"416":{"name":"计算机服务(系统、数据服务、维修)","parent":"1"},
+"417":{"name":"通信/电信/网络设备","parent":"1"},
+"418":{"name":"通信/电信运营、增值服务","parent":"1"},
+"419":{"name":"电子技术/半导体/集成电路","parent":"1"},
+"472":{"name":"P2P网贷","parent":"4"},
+"473":{"name":"互联网理财","parent":"4"},
+"474":{"name":"婚恋","parent":"6"},
+"476":{"name":"虚拟化","parent":"10"},
+"477":{"name":"邮箱","parent":"12"},
+"478":{"name":"商业智能","parent":"14"},
+"479":{"name":"企业建站","parent":"14"},
+"480":{"name":"安防","parent":"14"},
+"481":{"name":"网络营销","parent":"2"},
+"487":{"name":"智能终端","parent":"2"},
+"488":{"name":"移动互联网","parent":"2"},
+"489":{"name":"数字城市","parent":"2"},
+"490":{"name":"大数据","parent":"2"},
+"491":{"name":"互联网人力资源","parent":"2"},
+"492":{"name":"舆情监控","parent":"2"},
+"493":{"name":"移动营销","parent":"481"},
+"494":{"name":"微博营销","parent":"481"},
+"495":{"name":"精准营销","parent":"481"},
+"496":{"name":"海外营销","parent":"481"},
+"497":{"name":"微信营销","parent":"481"},
+"498":{"name":"智能手机","parent":"487"},
+"499":{"name":"可穿戴设备","parent":"487"},
+"500":{"name":"智能电视","parent":"487"},
+"501":{"name":"WAP","parent":"488"},
+"502":{"name":"物联网","parent":"489"},
+"503":{"name":"O2O","parent":"489"},
+"504":{"name":"数字出版","parent":"489"},
+"505":{"name":"搜索","parent":"2"},
+"506":{"name":"垂直搜索","parent":"505"},
+"507":{"name":"无线搜索","parent":"505"},
+"508":{"name":"网页搜索","parent":"505"},
+"509":{"name":"网址导航","parent":"2"},
+"510":{"name":"门户","parent":"2"},
+"511":{"name":"网络文学","parent":"2"},
+"512":{"name":"自媒体","parent":"2"},
+"513":{"name":"金融","parent":"0"},
+"514":{"name":"建筑与房地产","parent":"0"},
+"515":{"name":"专业服务","parent":"0"},
+"516":{"name":"教育培训","parent":"0"},
+"517":{"name":"文化传媒","parent":"0"},
+"518":{"name":"消费品","parent":"0"},
+"519":{"name":"工业","parent":"0"},
+"520":{"name":"交通物流","parent":"0"},
+"521":{"name":"贸易","parent":"0"},
+"522":{"name":"医药","parent":"0"},
+"523":{"name":"医疗器械","parent":"522"},
+"524":{"name":"保健品","parent":"518"},
+"525":{"name":"服务业","parent":"0"},
+"526":{"name":"能源/矿产/环保","parent":"0"},
+"527":{"name":"化工","parent":"0"},
+"528":{"name":"政府","parent":"0"},
+"529":{"name":"公共事业","parent":"0"},
+"530":{"name":"非盈利机构","parent":"0"},
+"531":{"name":"农业","parent":"1131"},
+"532":{"name":"林业","parent":"1131"},
+"533":{"name":"畜牧业","parent":"1131"},
+"534":{"name":"渔业","parent":"1131"},
+"535":{"name":"学术科研","parent":"0"},
+"536":{"name":"零售","parent":"0"},
+"537":{"name":"银行","parent":"513"},
+"538":{"name":"保险","parent":"513"},
+"539":{"name":"证券","parent":"513"},
+"540":{"name":"基金","parent":"513"},
+"541":{"name":"信托","parent":"513"},
+"542":{"name":"担保","parent":"513"},
+"543":{"name":"典当","parent":"513"},
+"544":{"name":"拍卖","parent":"513"},
+"545":{"name":"投资/融资","parent":"513"},
+"546":{"name":"期货","parent":"513"},
+"547":{"name":"房地产开发","parent":"514"},
+"548":{"name":"工程施工","parent":"514"},
+"549":{"name":"建筑设计","parent":"514"},
+"550":{"name":"房地产代理","parent":"514"},
+"551":{"name":"物业管理","parent":"514"},
+"552":{"name":"室内设计","parent":"514"},
+"553":{"name":"装修装潢","parent":"514"},
+"554":{"name":"市政工程","parent":"514"},
+"555":{"name":"工程造价","parent":"514"},
+"556":{"name":"工程监理","parent":"514"},
+"557":{"name":"环境工程","parent":"514"},
+"558":{"name":"园林景观","parent":"514"},
+"559":{"name":"法律","parent":"515"},
+"560":{"name":"人力资源","parent":"515"},
+"561":{"name":"会计","parent":"1125"},
+"562":{"name":"审计","parent":"515"},
+"563":{"name":"检测认证","parent":"515"},
+"565":{"name":"翻译","parent":"515"},
+"566":{"name":"中介","parent":"515"},
+"567":{"name":"咨询","parent":"515"},
+"568":{"name":"外包服务","parent":"515"},
+"569":{"name":"家教","parent":"516"},
+"570":{"name":"早教","parent":"516"},
+"571":{"name":"职业技能培训","parent":"516"},
+"572":{"name":"外语培训","parent":"516"},
+"573":{"name":"设计培训","parent":"516"},
+"574":{"name":"IT培训","parent":"516"},
+"575":{"name":"文艺体育培训","parent":"516"},
+"576":{"name":"学历教育","parent":"516"},
+"577":{"name":"管理培训","parent":"516"},
+"578":{"name":"民办基础教育","parent":"516"},
+"579":{"name":"广告","parent":"517"},
+"580":{"name":"媒体","parent":"517"},
+"581":{"name":"会展","parent":"517"},
+"582":{"name":"公关","parent":"517"},
+"583":{"name":"影视","parent":"517"},
+"584":{"name":"艺术","parent":"517"},
+"585":{"name":"文化传播","parent":"517"},
+"586":{"name":"娱乐","parent":"517"},
+"587":{"name":"体育","parent":"517"},
+"588":{"name":"出版","parent":"517"},
+"589":{"name":"休闲","parent":"517"},
+"590":{"name":"动漫","parent":"517"},
+"591":{"name":"市场推广","parent":"517"},
+"592":{"name":"市场研究","parent":"517"},
+"593":{"name":"食品","parent":"1129"},
+"594":{"name":"饮料","parent":"1129"},
+"595":{"name":"烟草","parent":"1129"},
+"596":{"name":"酒品","parent":"518"},
+"597":{"name":"服饰","parent":"518"},
+"598":{"name":"纺织","parent":"518"},
+"599":{"name":"化妆品","parent":"1129"},
+"600":{"name":"日用品","parent":"1129"},
+"601":{"name":"家电","parent":"518"},
+"602":{"name":"家具","parent":"518"},
+"603":{"name":"办公用品","parent":"518"},
+"604":{"name":"奢侈品","parent":"518"},
+"605":{"name":"珠宝","parent":"518"},
+"606":{"name":"数码产品","parent":"518"},
+"607":{"name":"玩具","parent":"518"},
+"608":{"name":"图书","parent":"518"},
+"609":{"name":"音像","parent":"518"},
+"610":{"name":"钟表","parent":"518"},
+"611":{"name":"箱包","parent":"518"},
+"612":{"name":"母婴","parent":"518"},
+"613":{"name":"营养保健","parent":"518"},
+"614":{"name":"户外用品","parent":"518"},
+"615":{"name":"健身器材","parent":"518"},
+"616":{"name":"乐器","parent":"518"},
+"617":{"name":"汽车用品","parent":"518"},
+"619":{"name":"厨具","parent":"518"},
+"620":{"name":"机械制造","parent":"519"},
+"621":{"name":"流体控制","parent":"519"},
+"622":{"name":"自动化控制","parent":"519"},
+"623":{"name":"仪器仪表","parent":"519"},
+"624":{"name":"航空/航天","parent":"519"},
+"625":{"name":"交通设施","parent":"519"},
+"626":{"name":"工业电子","parent":"519"},
+"627":{"name":"建材","parent":"519"},
+"628":{"name":"五金材料","parent":"519"},
+"629":{"name":"汽车","parent":"519"},
+"630":{"name":"印刷","parent":"519"},
+"631":{"name":"造纸","parent":"519"},
+"632":{"name":"包装","parent":"519"},
+"633":{"name":"原材料及加工","parent":"519"},
+"634":{"name":"物流","parent":"520"},
+"635":{"name":"仓储","parent":"520"},
+"636":{"name":"客运","parent":"520"},
+"637":{"name":"快递","parent":"520"},
+"638":{"name":"化学药","parent":"522"},
+"639":{"name":"中药","parent":"522"},
+"640":{"name":"生物制药","parent":"522"},
+"641":{"name":"兽药","parent":"522"},
+"642":{"name":"农药","parent":"522"},
+"643":{"name":"CRO","parent":"522"},
+"644":{"name":"消毒","parent":"522"},
+"645":{"name":"医药商业","parent":"522"},
+"646":{"name":"医疗服务","parent":"522"},
+"647":{"name":"医疗器械","parent":"523"},
+"648":{"name":"制药设备","parent":"523"},
+"649":{"name":"医用耗材","parent":"523"},
+"650":{"name":"手术器械","parent":"523"},
+"651":{"name":"保健器材","parent":"524"},
+"652":{"name":"性保健品","parent":"524"},
+"653":{"name":"医药保养","parent":"524"},
+"654":{"name":"医用保健","parent":"524"},
+"655":{"name":"酒店","parent":"525"},
+"656":{"name":"餐饮","parent":"525"},
+"657":{"name":"旅游","parent":"525"},
+"658":{"name":"生活服务","parent":"525"},
+"659":{"name":"保健服务","parent":"525"},
+"660":{"name":"运动健身","parent":"525"},
+"661":{"name":"家政服务","parent":"525"},
+"662":{"name":"婚庆服务","parent":"525"},
+"663":{"name":"租赁服务","parent":"525"},
+"664":{"name":"维修服务","parent":"525"},
+"665":{"name":"石油天然气","parent":"526"},
+"666":{"name":"电力","parent":"526"},
+"667":{"name":"新能源","parent":"526"},
+"668":{"name":"水利","parent":"526"},
+"669":{"name":"矿产","parent":"526"},
+"670":{"name":"采掘业","parent":"526"},
+"671":{"name":"冶炼","parent":"526"},
+"672":{"name":"环保","parent":"526"},
+"673":{"name":"无机化工原料","parent":"527"},
+"674":{"name":"有机化工原料","parent":"527"},
+"675":{"name":"精细化学品","parent":"527"},
+"676":{"name":"化工设备","parent":"527"},
+"677":{"name":"化工工程","parent":"527"},
+"678":{"name":"资产管理","parent":"513"},
+"679":{"name":"金融租赁","parent":"513"},
+"680":{"name":"征信及信评机构","parent":"513"},
+"681":{"name":"资产评估机构","parent":"513"},
+"683":{"name":"金融监管机构","parent":"513"},
+"684":{"name":"国际贸易","parent":"521"},
+"685":{"name":"海关","parent":"521"},
+"686":{"name":"购物中心","parent":"536"},
+"687":{"name":"超市","parent":"536"},
+"688":{"name":"便利店","parent":"536"},
+"689":{"name":"专卖店","parent":"536"},
+"690":{"name":"专业店","parent":"536"},
+"691":{"name":"百货店","parent":"536"},
+"692":{"name":"杂货店","parent":"536"},
+"693":{"name":"个人银行","parent":"537"},
+"695":{"name":"私人银行","parent":"537"},
+"696":{"name":"公司银行","parent":"537"},
+"697":{"name":"投资银行","parent":"537"},
+"698":{"name":"政策性银行","parent":"537"},
+"699":{"name":"中央银行","parent":"537"},
+"700":{"name":"人寿险","parent":"538"},
+"701":{"name":"财产险","parent":"538"},
+"702":{"name":"再保险","parent":"538"},
+"703":{"name":"养老险","parent":"538"},
+"704":{"name":"保险代理公司","parent":"538"},
+"705":{"name":"公募基金","parent":"540"},
+"707":{"name":"私募基金","parent":"540"},
+"708":{"name":"第三方理财","parent":"679"},
+"709":{"name":"资产管理公司","parent":"679"},
+"711":{"name":"房产中介","parent":"566"},
+"712":{"name":"职业中介","parent":"566"},
+"713":{"name":"婚姻中介","parent":"566"},
+"714":{"name":"战略咨询","parent":"567"},
+"715":{"name":"投资咨询","parent":"567"},
+"716":{"name":"心理咨询","parent":"567"},
+"717":{"name":"留学移民咨询","parent":"567"},
+"718":{"name":"工商注册代理","parent":"568"},
+"719":{"name":"商标专利代理","parent":"568"},
+"720":{"name":"财务代理","parent":"568"},
+"721":{"name":"工程机械","parent":"620"},
+"722":{"name":"农业机械","parent":"620"},
+"723":{"name":"海工设备","parent":"620"},
+"724":{"name":"包装机械","parent":"620"},
+"725":{"name":"印刷机械","parent":"620"},
+"726":{"name":"数控机床","parent":"620"},
+"727":{"name":"矿山机械","parent":"620"},
+"728":{"name":"水泵","parent":"621"},
+"729":{"name":"管道","parent":"621"},
+"730":{"name":"阀门","parent":"621"},
+"732":{"name":"压缩机","parent":"621"},
+"733":{"name":"集散控制系统","parent":"622"},
+"734":{"name":"远程控制","parent":"622"},
+"735":{"name":"液压系统","parent":"622"},
+"736":{"name":"楼宇智能化","parent":"622"},
+"737":{"name":"飞机制造","parent":"624"},
+"738":{"name":"航空公司","parent":"624"},
+"739":{"name":"发动机","parent":"624"},
+"740":{"name":"复合材料","parent":"624"},
+"741":{"name":"高铁","parent":"625"},
+"742":{"name":"地铁","parent":"625"},
+"743":{"name":"信号传输","parent":"625"},
+"745":{"name":"结构材料","parent":"627"},
+"746":{"name":"装饰材料","parent":"627"},
+"747":{"name":"专用材料","parent":"627"},
+"749":{"name":"经销商集团","parent":"629"},
+"750":{"name":"整车制造","parent":"629"},
+"751":{"name":"汽车零配件","parent":"629"},
+"752":{"name":"外型设计","parent":"629"},
+"753":{"name":"平版印刷","parent":"630"},
+"754":{"name":"凸版印刷","parent":"630"},
+"755":{"name":"凹版印刷","parent":"630"},
+"756":{"name":"孔版印刷","parent":"630"},
+"757":{"name":"印刷用纸","parent":"631"},
+"758":{"name":"书写、制图及复制用纸","parent":"631"},
+"759":{"name":"包装用纸","parent":"631"},
+"760":{"name":"生活、卫生及装饰用纸","parent":"631"},
+"761":{"name":"技术用纸","parent":"631"},
+"762":{"name":"加工纸原纸","parent":"631"},
+"763":{"name":"食品包装","parent":"632"},
+"764":{"name":"医药包装","parent":"632"},
+"765":{"name":"日化包装","parent":"632"},
+"766":{"name":"物流包装","parent":"632"},
+"767":{"name":"礼品包装","parent":"632"},
+"768":{"name":"电子五金包装","parent":"632"},
+"769":{"name":"汽车服务","parent":"525"},
+"770":{"name":"汽车保养","parent":"769"},
+"771":{"name":"租车","parent":"769"},
+"773":{"name":"出租车","parent":"769"},
+"774":{"name":"代驾","parent":"769"},
+"775":{"name":"发电","parent":"666"},
+"777":{"name":"输配电","parent":"666"},
+"779":{"name":"风电","parent":"667"},
+"780":{"name":"光伏/太阳能","parent":"667"},
+"781":{"name":"生物质发电","parent":"667"},
+"782":{"name":"煤化工","parent":"667"},
+"783":{"name":"垃圾发电","parent":"667"},
+"784":{"name":"核电","parent":"667"},
+"785":{"name":"能源矿产","parent":"669"},
+"786":{"name":"金属矿产","parent":"669"},
+"787":{"name":"非金属矿产","parent":"669"},
+"788":{"name":"水气矿产","parent":"669"},
+"789":{"name":"锅炉","parent":"775"},
+"790":{"name":"发电机","parent":"775"},
+"791":{"name":"汽轮机","parent":"775"},
+"792":{"name":"燃机","parent":"775"},
+"793":{"name":"冷却","parent":"775"},
+"794":{"name":"电力设计院","parent":"775"},
+"795":{"name":"高压输配电","parent":"777"},
+"796":{"name":"中压输配电","parent":"777"},
+"797":{"name":"低压输配电","parent":"777"},
+"798":{"name":"继电保护","parent":"777"},
+"799":{"name":"智能电网","parent":"777"},
+"800":{"name":"小学","parent":"516"},
+"801":{"name":"电动车","parent":"519"},
+"802":{"name":"皮具箱包","parent":"518"},
+"803":{"name":"医药制造","parent":"522"},
+"804":{"name":"电器销售","parent":"536"},
+"805":{"name":"塑料制品","parent":"527"},
+"806":{"name":"公益基金会","parent":"530"},
+"807":{"name":"美发服务","parent":"525"},
+"808":{"name":"农业养殖","parent":"531"},
+"809":{"name":"金融服务","parent":"513"},
+"810":{"name":"商业地产综合体","parent":"514"},
+"811":{"name":"美容服务","parent":"525"},
+"812":{"name":"灯饰","parent":"518"},
+"813":{"name":"油墨颜料产品","parent":"527"},
+"814":{"name":"眼镜制造","parent":"518"},
+"815":{"name":"农业生物技术","parent":"531"},
+"816":{"name":"体育用品","parent":"518"},
+"817":{"name":"保健用品","parent":"524"},
+"818":{"name":"化学化工产品","parent":"527"},
+"819":{"name":"饲料","parent":"531"},
+"821":{"name":"保安服务","parent":"525"},
+"822":{"name":"干细胞技术","parent":"522"},
+"824":{"name":"农药化肥","parent":"527"},
+"825":{"name":"卫生洁具","parent":"518"},
+"826":{"name":"体育器材、场馆","parent":"518"},
+"827":{"name":"饲料加工","parent":"531"},
+"828":{"name":"测绘服务","parent":"529"},
+"830":{"name":"金属船舶制造","parent":"519"},
+"831":{"name":"基因工程","parent":"522"},
+"832":{"name":"花卉服务","parent":"536"},
+"833":{"name":"农业种植","parent":"531"},
+"834":{"name":"皮革制品","parent":"518"},
+"835":{"name":"地理信息加工服务","parent":"529"},
+"836":{"name":"机器人","parent":"519"},
+"837":{"name":"礼品","parent":"518"},
+"838":{"name":"理发及美容服务","parent":"525"},
+"839":{"name":"其他清洁服务","parent":"525"},
+"840":{"name":"硅胶材料","parent":"527"},
+"841":{"name":"茶叶销售","parent":"518"},
+"842":{"name":"彩票活动","parent":"529"},
+"843":{"name":"化妆培训","parent":"516"},
+"844":{"name":"鞋业","parent":"518"},
+"845":{"name":"酒店用品","parent":"518"},
+"846":{"name":"复合材料","parent":"527"},
+"847":{"name":"房地产工程建设","parent":"548"},
+"848":{"name":"知识产权服务","parent":"559"},
+"849":{"name":"新型建材","parent":"627"},
+"850":{"name":"企业投资咨询","parent":"567"},
+"851":{"name":"含乳饮料和植物蛋白饮料制造","parent":"594"},
+"852":{"name":"汽车检测设备","parent":"629"},
+"853":{"name":"手机通讯器材","parent":"417"},
+"854":{"name":"环保材料","parent":"672"},
+"855":{"name":"交通设施","parent":"554"},
+"856":{"name":"电子器件","parent":"419"},
+"857":{"name":"啤酒","parent":"594"},
+"858":{"name":"生态旅游","parent":"657"},
+"859":{"name":"自动化设备","parent":"626"},
+"860":{"name":"软件开发","parent":"414"},
+"861":{"name":"葡萄酒销售","parent":"594"},
+"862":{"name":"钢材","parent":"633"},
+"863":{"name":"餐饮培训","parent":"656"},
+"864":{"name":"速冻食品","parent":"593"},
+"865":{"name":"空气环保","parent":"672"},
+"866":{"name":"互联网房地产经纪服务","parent":"550"},
+"867":{"name":"食品添加剂","parent":"593"},
+"868":{"name":"演艺传播","parent":"585"},
+"869":{"name":"信用卡","parent":"537"},
+"870":{"name":"报纸期刊广告","parent":"579"},
+"871":{"name":"摄影","parent":"525"},
+"872":{"name":"手机软件","parent":"414"},
+"873":{"name":"地坪建材","parent":"627"},
+"874":{"name":"企业管理咨询","parent":"567"},
+"875":{"name":"幼儿教育","parent":"570"},
+"876":{"name":"系统集成","parent":"416"},
+"877":{"name":"皮革服饰","parent":"597"},
+"878":{"name":"保健食品","parent":"593"},
+"879":{"name":"叉车","parent":"620"},
+"880":{"name":"厨卫电器","parent":"601"},
+"882":{"name":"地暖设备","parent":"627"},
+"883":{"name":"钢结构制造","parent":"548"},
+"884":{"name":"投影机","parent":"606"},
+"885":{"name":"啤酒销售","parent":"594"},
+"886":{"name":"度假村旅游","parent":"657"},
+"887":{"name":"电力元件设备","parent":"626"},
+"888":{"name":"管理软件","parent":"414"},
+"889":{"name":"轴承","parent":"628"},
+"890":{"name":"餐饮设备","parent":"656"},
+"891":{"name":"肉制品及副产品加工","parent":"593"},
+"892":{"name":"艺术收藏品投资交易","parent":"584"},
+"893":{"name":"净水器","parent":"601"},
+"894":{"name":"进口食品","parent":"593"},
+"895":{"name":"娱乐文化传播","parent":"585"},
+"896":{"name":"文化传播","parent":"585"},
+"897":{"name":"商旅传媒","parent":"580"},
+"898":{"name":"广告设计制作","parent":"579"},
+"899":{"name":"金属丝绳及其制品制造","parent":"627"},
+"900":{"name":"建筑涂料","parent":"627"},
+"901":{"name":"抵押贷款","parent":"543"},
+"902":{"name":"早教","parent":"570"},
+"903":{"name":"电影放映","parent":"583"},
+"904":{"name":"内衣服饰","parent":"597"},
+"905":{"name":"无线网络通信","parent":"418"},
+"906":{"name":"记忆卡","parent":"415"},
+"907":{"name":"女装服饰","parent":"597"},
+"908":{"name":"建筑机械","parent":"620"},
+"909":{"name":"制冷电器","parent":"601"},
+"910":{"name":"通信设备","parent":"417"},
+"911":{"name":"空调设备","parent":"601"},
+"912":{"name":"建筑装饰","parent":"553"},
+"913":{"name":"办公设备","parent":"603"},
+"916":{"name":"数据处理软件","parent":"414"},
+"917":{"name":"葡萄酒贸易","parent":"594"},
+"918":{"name":"通讯器材","parent":"417"},
+"919":{"name":"铜业","parent":"633"},
+"920":{"name":"食堂","parent":"656"},
+"921":{"name":"糖果零食","parent":"593"},
+"922":{"name":"文化艺术传播","parent":"584"},
+"923":{"name":"太阳能电器","parent":"601"},
+"924":{"name":"药品零售","parent":"645"},
+"925":{"name":"果蔬食品","parent":"593"},
+"926":{"name":"文化活动策划","parent":"585"},
+"928":{"name":"汽车广告","parent":"657"},
+"929":{"name":"条码设备","parent":"630"},
+"930":{"name":"建筑石材","parent":"627"},
+"931":{"name":"贵金属","parent":"545"},
+"932":{"name":"体育","parent":"660"},
+"933":{"name":"金融信息服务","parent":"414"},
+"934":{"name":"玻璃建材","parent":"627"},
+"935":{"name":"家教","parent":"569"},
+"936":{"name":"歌舞厅娱乐活动","parent":"586"},
+"937":{"name":"计算机服务器","parent":"415"},
+"938":{"name":"管道","parent":"627"},
+"939":{"name":"婴幼儿服饰","parent":"597"},
+"940":{"name":"热水器","parent":"601"},
+"941":{"name":"计算机及零部件制造","parent":"415"},
+"942":{"name":"钢铁贸易","parent":"633"},
+"944":{"name":"包装材料","parent":"632"},
+"945":{"name":"计算机办公设备","parent":"603"},
+"946":{"name":"白酒","parent":"594"},
+"948":{"name":"发动机","parent":"620"},
+"949":{"name":"快餐服务","parent":"656"},
+"950":{"name":"酒类销售","parent":"594"},
+"951":{"name":"电子产品、机电设备","parent":"626"},
+"952":{"name":"激光设备","parent":"626"},
+"953":{"name":"餐饮策划","parent":"656"},
+"954":{"name":"饮料、食品","parent":"594"},
+"955":{"name":"文化娱乐经纪","parent":"585"},
+"956":{"name":"天然气","parent":"665"},
+"957":{"name":"农副食品","parent":"593"},
+"958":{"name":"艺术表演","parent":"585"},
+"959":{"name":"石膏、水泥制品及类似制品制造","parent":"627"},
+"960":{"name":"橱柜","parent":"602"},
+"961":{"name":"管理培训","parent":"577"},
+"962":{"name":"男装服饰","parent":"597"},
+"963":{"name":"化肥制造","parent":"675"},
+"964":{"name":"童装服饰","parent":"597"},
+"965":{"name":"电源电池","parent":"626"},
+"966":{"name":"家电维修","parent":"664"},
+"967":{"name":"光电子器件","parent":"419"},
+"968":{"name":"旅行社服务","parent":"657"},
+"969":{"name":"电线、电缆制造","parent":"626"},
+"970":{"name":"软件开发、信息系统集成","parent":"419"},
+"971":{"name":"白酒制造","parent":"594"},
+"973":{"name":"甜品服务","parent":"656"},
+"974":{"name":"糕点、面包制造","parent":"593"},
+"975":{"name":"木工机械","parent":"620"},
+"976":{"name":"酒吧服务","parent":"656"},
+"977":{"name":"火腿肠","parent":"593"},
+"978":{"name":"广告策划推广","parent":"579"},
+"979":{"name":"新能源产品和生产装备制造","parent":"667"},
+"980":{"name":"调味品","parent":"593"},
+"981":{"name":"礼仪表演","parent":"585"},
+"982":{"name":"劳务派遣","parent":"560"},
+"983":{"name":"建材零售","parent":"627"},
+"984":{"name":"商品交易中心","parent":"545"},
+"985":{"name":"体育推广","parent":"585"},
+"986":{"name":"茶饮料及其他饮料制造","parent":"594"},
+"987":{"name":"金属建材","parent":"627"},
+"988":{"name":"职业技能培训","parent":"571"},
+"989":{"name":"网吧活动","parent":"586"},
+"990":{"name":"洗衣服务","parent":"658"},
+"991":{"name":"管道工程","parent":"554"},
+"992":{"name":"通信工程","parent":"417"},
+"993":{"name":"电子元器件","parent":"626"},
+"994":{"name":"电子设备","parent":"419"},
+"995":{"name":"茶馆服务","parent":"656"},
+"996":{"name":"旅游开发","parent":"657"},
+"997":{"name":"视频通讯","parent":"417"},
+"998":{"name":"白酒销售","parent":"594"},
+"1000":{"name":"咖啡馆服务","parent":"656"},
+"1001":{"name":"食品零售","parent":"593"},
+"1002":{"name":"健康疗养旅游","parent":"655"},
+"1003":{"name":"粮油食品","parent":"593"},
+"1004":{"name":"儿童教育影视","parent":"583"},
+"1005":{"name":"新能源发电","parent":"667"},
+"1006":{"name":"旅游策划","parent":"657"},
+"1007":{"name":"绘画","parent":"575"},
+"1008":{"name":"方便面及其他方便食品","parent":"593"},
+"1009":{"name":"房地产经纪","parent":"550"},
+"1010":{"name":"母婴家政","parent":"661"},
+"1011":{"name":"居家养老健康服务","parent":"661"},
+"1012":{"name":"文化艺术投资","parent":"545"},
+"1013":{"name":"运动健身","parent":"660"},
+"1014":{"name":"瓶（罐）装饮用水制造","parent":"594"},
+"1015":{"name":"金属门窗","parent":"627"},
+"1016":{"name":"机动车检测","parent":"563"},
+"1017":{"name":"货物运输","parent":"634"},
+"1018":{"name":"服饰专卖","parent":"690"},
+"1019":{"name":"酒店服装","parent":"597"},
+"1020":{"name":"通讯软件","parent":"417"},
+"1021":{"name":"消防工程","parent":"554"},
+"1022":{"name":"嵌入式电子系统","parent":"419"},
+"1023":{"name":"航空票务","parent":"636"},
+"1024":{"name":"电气设备","parent":"626"},
+"1025":{"name":"酒业贸易","parent":"594"},
+"1027":{"name":"其他饮料及冷饮服务","parent":"656"},
+"1028":{"name":"乳制品","parent":"593"},
+"1029":{"name":"新闻期刊出版","parent":"588"},
+"1030":{"name":"水污染治理","parent":"672"},
+"1031":{"name":"谷物食品","parent":"593"},
+"1032":{"name":"数字动漫设计制造服务","parent":"590"},
+"1033":{"name":"医院","parent":"646"},
+"1034":{"name":"旅游广告","parent":"657"},
+"1035":{"name":"办公家具","parent":"602"},
+"1036":{"name":"房地产营销策划","parent":"550"},
+"1037":{"name":"保洁家政","parent":"661"},
+"1038":{"name":"水泥制造","parent":"627"},
+"1039":{"name":"市场研究咨询","parent":"567"},
+"1040":{"name":"驾校","parent":"571"},
+"1041":{"name":"正餐服务","parent":"656"},
+"1043":{"name":"机动车燃油","parent":"665"},
+"1044":{"name":"食品","parent":"593"},
+"1045":{"name":"新能源汽车","parent":"629"},
+"1046":{"name":"手机无线网络推广","parent":"417"},
+"1047":{"name":"环保设备","parent":"672"},
+"1048":{"name":"通讯工程","parent":"418"},
+"1049":{"name":"半导体集成电路","parent":"419"},
+"1050":{"name":"航空服务","parent":"636"},
+"1051":{"name":"电机设备","parent":"626"},
+"1052":{"name":"档案软件","parent":"414"},
+"1053":{"name":"冷链物流服务","parent":"634"},
+"1054":{"name":"小吃服务","parent":"656"},
+"1055":{"name":"水产品加工","parent":"593"},
+"1056":{"name":"图书出版","parent":"588"},
+"1057":{"name":"固体废物治理","parent":"672"},
+"1059":{"name":"坚果食品","parent":"593"},
+"1060":{"name":"广告传媒","parent":"579"},
+"1061":{"name":"电梯","parent":"622"},
+"1062":{"name":"社区医疗与卫生院","parent":"646"},
+"1063":{"name":"广告、印刷包装","parent":"630"},
+"1064":{"name":"婚纱礼服","parent":"662"},
+"1065":{"name":"地毯","parent":"602"},
+"1066":{"name":"互联网物业","parent":"551"},
+"1067":{"name":"跨境电商","parent":"3"},
+"1068":{"name":"信息安全、系统集成","parent":"9"},
+"1069":{"name":"专用汽车制造","parent":"750"},
+"1070":{"name":"商品贸易","parent":"3"},
+"1071":{"name":"墙壁装饰材料","parent":"746"},
+"1072":{"name":"窗帘装饰材料","parent":"746"},
+"1073":{"name":"电子商务、本地生活服务","parent":"3"},
+"1075":{"name":"白酒电子商务","parent":"3"},
+"1076":{"name":"商品贸易、电子商务","parent":"3"},
+"1077":{"name":"木质装饰材料","parent":"746"},
+"1078":{"name":"电子商务、汽车电商交易平台","parent":"3"},
+"1079":{"name":"汽车轮胎","parent":"751"},
+"1080":{"name":"气体压缩机械制造","parent":"732"},
+"1081":{"name":"家装家具电子商务","parent":"3"},
+"1082":{"name":"化妆品电子商务","parent":"3"},
+"1083":{"name":"汽车销售","parent":"749"},
+"1084":{"name":"新闻资讯网站","parent":"510"},
+"1085":{"name":"母婴电商","parent":"3"},
+"1086":{"name":"电商商务、收藏品交易","parent":"3"},
+"1088":{"name":"电子商务、数码产品","parent":"3"},
+"1089":{"name":"二手车交易","parent":"749"},
+"1090":{"name":"游戏制作服务","parent":"5"},
+"1091":{"name":"母婴服务","parent":"510"},
+"1092":{"name":"家具电子商务","parent":"3"},
+"1093":{"name":"汽车配件电子商务","parent":"3"},
+"1094":{"name":"输配电设备","parent":"777"},
+"1095":{"name":"矿山设备","parent":"727"},
+"1096":{"name":"机床机械","parent":"726"},
+"1097":{"name":"农产品电商","parent":"3"},
+"1098":{"name":"陶瓷装饰材料","parent":"746"},
+"1099":{"name":"车载联网设备","parent":"487"},
+"1100":{"name":"汽车销售电子商务","parent":"3"},
+"1101":{"name":"石油设备","parent":"730"},
+"1102":{"name":"智能家居","parent":"487"},
+"1103":{"name":"散热器","parent":"751"},
+"1104":{"name":"电力工程","parent":"775"},
+"1105":{"name":"生鲜电商","parent":"3"},
+"1106":{"name":"互联网数据服务","parent":"490"},
+"1107":{"name":"房车、商务车销售","parent":"749"},
+"1108":{"name":"茶叶电子商务","parent":"3"},
+"1109":{"name":"酒类电子商务","parent":"3"},
+"1110":{"name":"阀门","parent":"730"},
+"1111":{"name":"食品电商","parent":"3"},
+"1112":{"name":"儿童摄影","parent":"871"},
+"1113":{"name":"广告摄影","parent":"871"},
+"1114":{"name":"婚纱摄影","parent":"871"},
+"1115":{"name":"模具制造","parent":"620"},
+"1116":{"name":"汽车模具","parent":"629"},
+"1117":{"name":"认证咨询","parent":"567"},
+"1118":{"name":"数字视觉制作服务","parent":"590"},
+"1119":{"name":"牙科及医疗器械","parent":"646"},
+"1120":{"name":"猎头招聘","parent":"560"},
+"1121":{"name":"家居","parent":"518"},
+"1122":{"name":"收藏品","parent":"518"},
+"1123":{"name":"首饰","parent":"518"},
+"1124":{"name":"工艺品","parent":"518"},
+"1125":{"name":"财务","parent":"515"},
+"1126":{"name":"税务","parent":"515"},
+"1127":{"name":"分类信息","parent":"2"},
+"1128":{"name":"宠物","parent":"0"},
+"1129":{"name":"快消品","parent":"518"},
+"1130":{"name":"人工智能","parent":"2"},
+"1131":{"name":"农/林/牧/渔","parent":"0"}
+}
+
+def get_names(id):
+    id = str(id)
+    nms = []
+    d = TBL.get(id)
+    if not d:return []
+    nms.append(d["name"])
+    p = get_names(d["parent"])
+    if p: nms.extend(p)
+    return nms
+
+if __name__ == "__main__":
+    print(get_names("1119"))

deepdoc/parser/resume/entities/regions.py

@@ -0,0 +1,762 @@
+TBL = {
+"2":{"name":"北京","parent":"1"},
+"3":{"name":"天津","parent":"1"},
+"4":{"name":"河北","parent":"1"},
+"5":{"name":"山西","parent":"1"},
+"6":{"name":"内蒙古","parent":"1"},
+"7":{"name":"辽宁","parent":"1"},
+"8":{"name":"吉林","parent":"1"},
+"9":{"name":"黑龙江","parent":"1"},
+"10":{"name":"上海","parent":"1"},
+"11":{"name":"江苏","parent":"1"},
+"12":{"name":"浙江","parent":"1"},
+"13":{"name":"安徽","parent":"1"},
+"14":{"name":"福建","parent":"1"},
+"15":{"name":"江西","parent":"1"},
+"16":{"name":"山东","parent":"1"},
+"17":{"name":"河南","parent":"1"},
+"18":{"name":"湖北","parent":"1"},
+"19":{"name":"湖南","parent":"1"},
+"20":{"name":"广东","parent":"1"},
+"21":{"name":"广西","parent":"1"},
+"22":{"name":"海南","parent":"1"},
+"23":{"name":"重庆","parent":"1"},
+"24":{"name":"四川","parent":"1"},
+"25":{"name":"贵州","parent":"1"},
+"26":{"name":"云南","parent":"1"},
+"27":{"name":"西藏","parent":"1"},
+"28":{"name":"陕西","parent":"1"},
+"29":{"name":"甘肃","parent":"1"},
+"30":{"name":"青海","parent":"1"},
+"31":{"name":"宁夏","parent":"1"},
+"32":{"name":"新疆","parent":"1"},
+"33":{"name":"北京市","parent":"2"},
+"34":{"name":"天津市","parent":"3"},
+"35":{"name":"石家庄市","parent":"4"},
+"36":{"name":"唐山市","parent":"4"},
+"37":{"name":"秦皇岛市","parent":"4"},
+"38":{"name":"邯郸市","parent":"4"},
+"39":{"name":"邢台市","parent":"4"},
+"40":{"name":"保定市","parent":"4"},
+"41":{"name":"张家口市","parent":"4"},
+"42":{"name":"承德市","parent":"4"},
+"43":{"name":"沧州市","parent":"4"},
+"44":{"name":"廊坊市","parent":"4"},
+"45":{"name":"衡水市","parent":"4"},
+"46":{"name":"太原市","parent":"5"},
+"47":{"name":"大同市","parent":"5"},
+"48":{"name":"阳泉市","parent":"5"},
+"49":{"name":"长治市","parent":"5"},
+"50":{"name":"晋城市","parent":"5"},
+"51":{"name":"朔州市","parent":"5"},
+"52":{"name":"晋中市","parent":"5"},
+"53":{"name":"运城市","parent":"5"},
+"54":{"name":"忻州市","parent":"5"},
+"55":{"name":"临汾市","parent":"5"},
+"56":{"name":"吕梁市","parent":"5"},
+"57":{"name":"呼和浩特市","parent":"6"},
+"58":{"name":"包头市","parent":"6"},
+"59":{"name":"乌海市","parent":"6"},
+"60":{"name":"赤峰市","parent":"6"},
+"61":{"name":"通辽市","parent":"6"},
+"62":{"name":"鄂尔多斯市","parent":"6"},
+"63":{"name":"呼伦贝尔市","parent":"6"},
+"64":{"name":"巴彦淖尔市","parent":"6"},
+"65":{"name":"乌兰察布市","parent":"6"},
+"66":{"name":"兴安盟","parent":"6"},
+"67":{"name":"锡林郭勒盟","parent":"6"},
+"68":{"name":"阿拉善盟","parent":"6"},
+"69":{"name":"沈阳市","parent":"7"},
+"70":{"name":"大连市","parent":"7"},
+"71":{"name":"鞍山市","parent":"7"},
+"72":{"name":"抚顺市","parent":"7"},
+"73":{"name":"本溪市","parent":"7"},
+"74":{"name":"丹东市","parent":"7"},
+"75":{"name":"锦州市","parent":"7"},
+"76":{"name":"营口市","parent":"7"},
+"77":{"name":"阜新市","parent":"7"},
+"78":{"name":"辽阳市","parent":"7"},
+"79":{"name":"盘锦市","parent":"7"},
+"80":{"name":"铁岭市","parent":"7"},
+"81":{"name":"朝阳市","parent":"7"},
+"82":{"name":"葫芦岛市","parent":"7"},
+"83":{"name":"长春市","parent":"8"},
+"84":{"name":"吉林市","parent":"8"},
+"85":{"name":"四平市","parent":"8"},
+"86":{"name":"辽源市","parent":"8"},
+"87":{"name":"通化市","parent":"8"},
+"88":{"name":"白山市","parent":"8"},
+"89":{"name":"松原市","parent":"8"},
+"90":{"name":"白城市","parent":"8"},
+"91":{"name":"延边朝鲜族自治州","parent":"8"},
+"92":{"name":"哈尔滨市","parent":"9"},
+"93":{"name":"齐齐哈尔市","parent":"9"},
+"94":{"name":"鸡西市","parent":"9"},
+"95":{"name":"鹤岗市","parent":"9"},
+"96":{"name":"双鸭山市","parent":"9"},
+"97":{"name":"大庆市","parent":"9"},
+"98":{"name":"伊春市","parent":"9"},
+"99":{"name":"佳木斯市","parent":"9"},
+"100":{"name":"七台河市","parent":"9"},
+"101":{"name":"牡丹江市","parent":"9"},
+"102":{"name":"黑河市","parent":"9"},
+"103":{"name":"绥化市","parent":"9"},
+"104":{"name":"大兴安岭地区","parent":"9"},
+"105":{"name":"上海市","parent":"10"},
+"106":{"name":"南京市","parent":"11"},
+"107":{"name":"无锡市","parent":"11"},
+"108":{"name":"徐州市","parent":"11"},
+"109":{"name":"常州市","parent":"11"},
+"110":{"name":"苏州市","parent":"11"},
+"111":{"name":"南通市","parent":"11"},
+"112":{"name":"连云港市","parent":"11"},
+"113":{"name":"淮安市","parent":"11"},
+"114":{"name":"盐城市","parent":"11"},
+"115":{"name":"扬州市","parent":"11"},
+"116":{"name":"镇江市","parent":"11"},
+"117":{"name":"泰州市","parent":"11"},
+"118":{"name":"宿迁市","parent":"11"},
+"119":{"name":"杭州市","parent":"12"},
+"120":{"name":"宁波市","parent":"12"},
+"121":{"name":"温州市","parent":"12"},
+"122":{"name":"嘉兴市","parent":"12"},
+"123":{"name":"湖州市","parent":"12"},
+"124":{"name":"绍兴市","parent":"12"},
+"125":{"name":"金华市","parent":"12"},
+"126":{"name":"衢州市","parent":"12"},
+"127":{"name":"舟山市","parent":"12"},
+"128":{"name":"台州市","parent":"12"},
+"129":{"name":"丽水市","parent":"12"},
+"130":{"name":"合肥市","parent":"13"},
+"131":{"name":"芜湖市","parent":"13"},
+"132":{"name":"蚌埠市","parent":"13"},
+"133":{"name":"淮南市","parent":"13"},
+"134":{"name":"马鞍山市","parent":"13"},
+"135":{"name":"淮北市","parent":"13"},
+"136":{"name":"铜陵市","parent":"13"},
+"137":{"name":"安庆市","parent":"13"},
+"138":{"name":"黄山市","parent":"13"},
+"139":{"name":"滁州市","parent":"13"},
+"140":{"name":"阜阳市","parent":"13"},
+"141":{"name":"宿州市","parent":"13"},
+"143":{"name":"六安市","parent":"13"},
+"144":{"name":"亳州市","parent":"13"},
+"145":{"name":"池州市","parent":"13"},
+"146":{"name":"宣城市","parent":"13"},
+"147":{"name":"福州市","parent":"14"},
+"148":{"name":"厦门市","parent":"14"},
+"149":{"name":"莆田市","parent":"14"},
+"150":{"name":"三明市","parent":"14"},
+"151":{"name":"泉州市","parent":"14"},
+"152":{"name":"漳州市","parent":"14"},
+"153":{"name":"南平市","parent":"14"},
+"154":{"name":"龙岩市","parent":"14"},
+"155":{"name":"宁德市","parent":"14"},
+"156":{"name":"南昌市","parent":"15"},
+"157":{"name":"景德镇市","parent":"15"},
+"158":{"name":"萍乡市","parent":"15"},
+"159":{"name":"九江市","parent":"15"},
+"160":{"name":"新余市","parent":"15"},
+"161":{"name":"鹰潭市","parent":"15"},
+"162":{"name":"赣州市","parent":"15"},
+"163":{"name":"吉安市","parent":"15"},
+"164":{"name":"宜春市","parent":"15"},
+"165":{"name":"抚州市","parent":"15"},
+"166":{"name":"上饶市","parent":"15"},
+"167":{"name":"济南市","parent":"16"},
+"168":{"name":"青岛市","parent":"16"},
+"169":{"name":"淄博市","parent":"16"},
+"170":{"name":"枣庄市","parent":"16"},
+"171":{"name":"东营市","parent":"16"},
+"172":{"name":"烟台市","parent":"16"},
+"173":{"name":"潍坊市","parent":"16"},
+"174":{"name":"济宁市","parent":"16"},
+"175":{"name":"泰安市","parent":"16"},
+"176":{"name":"威海市","parent":"16"},
+"177":{"name":"日照市","parent":"16"},
+"179":{"name":"临沂市","parent":"16"},
+"180":{"name":"德州市","parent":"16"},
+"181":{"name":"聊城市","parent":"16"},
+"182":{"name":"滨州市","parent":"16"},
+"183":{"name":"菏泽市","parent":"16"},
+"184":{"name":"郑州市","parent":"17"},
+"185":{"name":"开封市","parent":"17"},
+"186":{"name":"洛阳市","parent":"17"},
+"187":{"name":"平顶山市","parent":"17"},
+"188":{"name":"安阳市","parent":"17"},
+"189":{"name":"鹤壁市","parent":"17"},
+"190":{"name":"新乡市","parent":"17"},
+"191":{"name":"焦作市","parent":"17"},
+"192":{"name":"濮阳市","parent":"17"},
+"193":{"name":"许昌市","parent":"17"},
+"194":{"name":"漯河市","parent":"17"},
+"195":{"name":"三门峡市","parent":"17"},
+"196":{"name":"南阳市","parent":"17"},
+"197":{"name":"商丘市","parent":"17"},
+"198":{"name":"信阳市","parent":"17"},
+"199":{"name":"周口市","parent":"17"},
+"200":{"name":"驻马店市","parent":"17"},
+"201":{"name":"武汉市","parent":"18"},
+"202":{"name":"黄石市","parent":"18"},
+"203":{"name":"十堰市","parent":"18"},
+"204":{"name":"宜昌市","parent":"18"},
+"205":{"name":"襄阳市","parent":"18"},
+"206":{"name":"鄂州市","parent":"18"},
+"207":{"name":"荆门市","parent":"18"},
+"208":{"name":"孝感市","parent":"18"},
+"209":{"name":"荆州市","parent":"18"},
+"210":{"name":"黄冈市","parent":"18"},
+"211":{"name":"咸宁市","parent":"18"},
+"212":{"name":"随州市","parent":"18"},
+"213":{"name":"恩施土家族苗族自治州","parent":"18"},
+"215":{"name":"长沙市","parent":"19"},
+"216":{"name":"株洲市","parent":"19"},
+"217":{"name":"湘潭市","parent":"19"},
+"218":{"name":"衡阳市","parent":"19"},
+"219":{"name":"邵阳市","parent":"19"},
+"220":{"name":"岳阳市","parent":"19"},
+"221":{"name":"常德市","parent":"19"},
+"222":{"name":"张家界市","parent":"19"},
+"223":{"name":"益阳市","parent":"19"},
+"224":{"name":"郴州市","parent":"19"},
+"225":{"name":"永州市","parent":"19"},
+"226":{"name":"怀化市","parent":"19"},
+"227":{"name":"娄底市","parent":"19"},
+"228":{"name":"湘西土家族苗族自治州","parent":"19"},
+"229":{"name":"广州市","parent":"20"},
+"230":{"name":"韶关市","parent":"20"},
+"231":{"name":"深圳市","parent":"20"},
+"232":{"name":"珠海市","parent":"20"},
+"233":{"name":"汕头市","parent":"20"},
+"234":{"name":"佛山市","parent":"20"},
+"235":{"name":"江门市","parent":"20"},
+"236":{"name":"湛江市","parent":"20"},
+"237":{"name":"茂名市","parent":"20"},
+"238":{"name":"肇庆市","parent":"20"},
+"239":{"name":"惠州市","parent":"20"},
+"240":{"name":"梅州市","parent":"20"},
+"241":{"name":"汕尾市","parent":"20"},
+"242":{"name":"河源市","parent":"20"},
+"243":{"name":"阳江市","parent":"20"},
+"244":{"name":"清远市","parent":"20"},
+"245":{"name":"东莞市","parent":"20"},
+"246":{"name":"中山市","parent":"20"},
+"247":{"name":"潮州市","parent":"20"},
+"248":{"name":"揭阳市","parent":"20"},
+"249":{"name":"云浮市","parent":"20"},
+"250":{"name":"南宁市","parent":"21"},
+"251":{"name":"柳州市","parent":"21"},
+"252":{"name":"桂林市","parent":"21"},
+"253":{"name":"梧州市","parent":"21"},
+"254":{"name":"北海市","parent":"21"},
+"255":{"name":"防城港市","parent":"21"},
+"256":{"name":"钦州市","parent":"21"},
+"257":{"name":"贵港市","parent":"21"},
+"258":{"name":"玉林市","parent":"21"},
+"259":{"name":"百色市","parent":"21"},
+"260":{"name":"贺州市","parent":"21"},
+"261":{"name":"河池市","parent":"21"},
+"262":{"name":"来宾市","parent":"21"},
+"263":{"name":"崇左市","parent":"21"},
+"264":{"name":"海口市","parent":"22"},
+"265":{"name":"三亚市","parent":"22"},
+"267":{"name":"重庆市","parent":"23"},
+"268":{"name":"成都市","parent":"24"},
+"269":{"name":"自贡市","parent":"24"},
+"270":{"name":"攀枝花市","parent":"24"},
+"271":{"name":"泸州市","parent":"24"},
+"272":{"name":"德阳市","parent":"24"},
+"273":{"name":"绵阳市","parent":"24"},
+"274":{"name":"广元市","parent":"24"},
+"275":{"name":"遂宁市","parent":"24"},
+"276":{"name":"内江市","parent":"24"},
+"277":{"name":"乐山市","parent":"24"},
+"278":{"name":"南充市","parent":"24"},
+"279":{"name":"眉山市","parent":"24"},
+"280":{"name":"宜宾市","parent":"24"},
+"281":{"name":"广安市","parent":"24"},
+"282":{"name":"达州市","parent":"24"},
+"283":{"name":"雅安市","parent":"24"},
+"284":{"name":"巴中市","parent":"24"},
+"285":{"name":"资阳市","parent":"24"},
+"286":{"name":"阿坝藏族羌族自治州","parent":"24"},
+"287":{"name":"甘孜藏族自治州","parent":"24"},
+"288":{"name":"凉山彝族自治州","parent":"24"},
+"289":{"name":"贵阳市","parent":"25"},
+"290":{"name":"六盘水市","parent":"25"},
+"291":{"name":"遵义市","parent":"25"},
+"292":{"name":"安顺市","parent":"25"},
+"293":{"name":"铜仁市","parent":"25"},
+"294":{"name":"黔西南布依族苗族自治州","parent":"25"},
+"295":{"name":"毕节市","parent":"25"},
+"296":{"name":"黔东南苗族侗族自治州","parent":"25"},
+"297":{"name":"黔南布依族苗族自治州","parent":"25"},
+"298":{"name":"昆明市","parent":"26"},
+"299":{"name":"曲靖市","parent":"26"},
+"300":{"name":"玉溪市","parent":"26"},
+"301":{"name":"保山市","parent":"26"},
+"302":{"name":"昭通市","parent":"26"},
+"303":{"name":"丽江市","parent":"26"},
+"304":{"name":"普洱市","parent":"26"},
+"305":{"name":"临沧市","parent":"26"},
+"306":{"name":"楚雄彝族自治州","parent":"26"},
+"307":{"name":"红河哈尼族彝族自治州","parent":"26"},
+"308":{"name":"文山壮族苗族自治州","parent":"26"},
+"309":{"name":"西双版纳傣族自治州","parent":"26"},
+"310":{"name":"大理白族自治州","parent":"26"},
+"311":{"name":"德宏傣族景颇族自治州","parent":"26"},
+"312":{"name":"怒江傈僳族自治州","parent":"26"},
+"313":{"name":"迪庆藏族自治州","parent":"26"},
+"314":{"name":"拉萨市","parent":"27"},
+"315":{"name":"昌都市","parent":"27"},
+"316":{"name":"山南市","parent":"27"},
+"317":{"name":"日喀则市","parent":"27"},
+"318":{"name":"那曲市","parent":"27"},
+"319":{"name":"阿里地区","parent":"27"},
+"320":{"name":"林芝市","parent":"27"},
+"321":{"name":"西安市","parent":"28"},
+"322":{"name":"铜川市","parent":"28"},
+"323":{"name":"宝鸡市","parent":"28"},
+"324":{"name":"咸阳市","parent":"28"},
+"325":{"name":"渭南市","parent":"28"},
+"326":{"name":"延安市","parent":"28"},
+"327":{"name":"汉中市","parent":"28"},
+"328":{"name":"榆林市","parent":"28"},
+"329":{"name":"安康市","parent":"28"},
+"330":{"name":"商洛市","parent":"28"},
+"331":{"name":"兰州市","parent":"29"},
+"332":{"name":"嘉峪关市","parent":"29"},
+"333":{"name":"金昌市","parent":"29"},
+"334":{"name":"白银市","parent":"29"},
+"335":{"name":"天水市","parent":"29"},
+"336":{"name":"武威市","parent":"29"},
+"337":{"name":"张掖市","parent":"29"},
+"338":{"name":"平凉市","parent":"29"},
+"339":{"name":"酒泉市","parent":"29"},
+"340":{"name":"庆阳市","parent":"29"},
+"341":{"name":"定西市","parent":"29"},
+"342":{"name":"陇南市","parent":"29"},
+"343":{"name":"临夏回族自治州","parent":"29"},
+"344":{"name":"甘南藏族自治州","parent":"29"},
+"345":{"name":"西宁市","parent":"30"},
+"346":{"name":"海东市","parent":"30"},
+"347":{"name":"海北藏族自治州","parent":"30"},
+"348":{"name":"黄南藏族自治州","parent":"30"},
+"349":{"name":"海南藏族自治州","parent":"30"},
+"350":{"name":"果洛藏族自治州","parent":"30"},
+"351":{"name":"玉树藏族自治州","parent":"30"},
+"352":{"name":"海西蒙古族藏族自治州","parent":"30"},
+"353":{"name":"银川市","parent":"31"},
+"354":{"name":"石嘴山市","parent":"31"},
+"355":{"name":"吴忠市","parent":"31"},
+"356":{"name":"固原市","parent":"31"},
+"357":{"name":"中卫市","parent":"31"},
+"358":{"name":"乌鲁木齐市","parent":"32"},
+"359":{"name":"克拉玛依市","parent":"32"},
+"360":{"name":"吐鲁番市","parent":"32"},
+"361":{"name":"哈密市","parent":"32"},
+"362":{"name":"昌吉回族自治州","parent":"32"},
+"363":{"name":"博尔塔拉蒙古自治州","parent":"32"},
+"364":{"name":"巴音郭楞蒙古自治州","parent":"32"},
+"365":{"name":"阿克苏地区","parent":"32"},
+"366":{"name":"克孜勒苏柯尔克孜自治州","parent":"32"},
+"367":{"name":"喀什地区","parent":"32"},
+"368":{"name":"和田地区","parent":"32"},
+"369":{"name":"伊犁哈萨克自治州","parent":"32"},
+"370":{"name":"塔城地区","parent":"32"},
+"371":{"name":"阿勒泰地区","parent":"32"},
+"372":{"name":"新疆省直辖行政单位","parent":"32"},
+"373":{"name":"可克达拉市","parent":"32"},
+"374":{"name":"昆玉市","parent":"32"},
+"375":{"name":"胡杨河市","parent":"32"},
+"376":{"name":"双河市","parent":"32"},
+"3560":{"name":"北票市","parent":"7"},
+"3615":{"name":"高州市","parent":"20"},
+"3651":{"name":"济源市","parent":"17"},
+"3662":{"name":"胶南市","parent":"16"},
+"3683":{"name":"老河口市","parent":"18"},
+"3758":{"name":"沙河市","parent":"4"},
+"3822":{"name":"宜城市","parent":"18"},
+"3842":{"name":"枣阳市","parent":"18"},
+"3850":{"name":"肇东市","parent":"9"},
+"3905":{"name":"澳门","parent":"1"},
+"3906":{"name":"澳门","parent":"3905"},
+"3907":{"name":"香港","parent":"1"},
+"3908":{"name":"香港","parent":"3907"},
+"3947":{"name":"仙桃市","parent":"18"},
+"3954":{"name":"台湾","parent":"1"},
+"3955":{"name":"台湾","parent":"3954"},
+"3956":{"name":"海外","parent":"1"},
+"3957":{"name":"海外","parent":"3956"},
+"3958":{"name":"美国","parent":"3956"},
+"3959":{"name":"加拿大","parent":"3956"},
+"3961":{"name":"日本","parent":"3956"},
+"3962":{"name":"韩国","parent":"3956"},
+"3963":{"name":"德国","parent":"3956"},
+"3964":{"name":"英国","parent":"3956"},
+"3965":{"name":"意大利","parent":"3956"},
+"3966":{"name":"西班牙","parent":"3956"},
+"3967":{"name":"法国","parent":"3956"},
+"3968":{"name":"澳大利亚","parent":"3956"},
+"3969":{"name":"东城区","parent":"2"},
+"3970":{"name":"西城区","parent":"2"},
+"3971":{"name":"崇文区","parent":"2"},
+"3972":{"name":"宣武区","parent":"2"},
+"3973":{"name":"朝阳区","parent":"2"},
+"3974":{"name":"海淀区","parent":"2"},
+"3975":{"name":"丰台区","parent":"2"},
+"3976":{"name":"石景山区","parent":"2"},
+"3977":{"name":"门头沟区","parent":"2"},
+"3978":{"name":"房山区","parent":"2"},
+"3979":{"name":"通州区","parent":"2"},
+"3980":{"name":"顺义区","parent":"2"},
+"3981":{"name":"昌平区","parent":"2"},
+"3982":{"name":"大兴区","parent":"2"},
+"3983":{"name":"平谷区","parent":"2"},
+"3984":{"name":"怀柔区","parent":"2"},
+"3985":{"name":"密云区","parent":"2"},
+"3986":{"name":"延庆区","parent":"2"},
+"3987":{"name":"黄浦区","parent":"10"},
+"3988":{"name":"徐汇区","parent":"10"},
+"3989":{"name":"长宁区","parent":"10"},
+"3990":{"name":"静安区","parent":"10"},
+"3991":{"name":"普陀区","parent":"10"},
+"3992":{"name":"闸北区","parent":"10"},
+"3993":{"name":"虹口区","parent":"10"},
+"3994":{"name":"杨浦区","parent":"10"},
+"3995":{"name":"宝山区","parent":"10"},
+"3996":{"name":"闵行区","parent":"10"},
+"3997":{"name":"嘉定区","parent":"10"},
+"3998":{"name":"浦东新区","parent":"10"},
+"3999":{"name":"松江区","parent":"10"},
+"4000":{"name":"金山区","parent":"10"},
+"4001":{"name":"青浦区","parent":"10"},
+"4002":{"name":"奉贤区","parent":"10"},
+"4003":{"name":"崇明区","parent":"10"},
+"4004":{"name":"和平区","parent":"3"},
+"4005":{"name":"河东区","parent":"3"},
+"4006":{"name":"河西区","parent":"3"},
+"4007":{"name":"南开区","parent":"3"},
+"4008":{"name":"红桥区","parent":"3"},
+"4009":{"name":"河北区","parent":"3"},
+"4010":{"name":"滨海新区","parent":"3"},
+"4011":{"name":"东丽区","parent":"3"},
+"4012":{"name":"西青区","parent":"3"},
+"4013":{"name":"北辰区","parent":"3"},
+"4014":{"name":"津南区","parent":"3"},
+"4015":{"name":"武清区","parent":"3"},
+"4016":{"name":"宝坻区","parent":"3"},
+"4017":{"name":"静海区","parent":"3"},
+"4018":{"name":"宁河区","parent":"3"},
+"4019":{"name":"蓟州区","parent":"3"},
+"4020":{"name":"渝中区","parent":"23"},
+"4021":{"name":"江北区","parent":"23"},
+"4022":{"name":"南岸区","parent":"23"},
+"4023":{"name":"沙坪坝区","parent":"23"},
+"4024":{"name":"九龙坡区","parent":"23"},
+"4025":{"name":"大渡口区","parent":"23"},
+"4026":{"name":"渝北区","parent":"23"},
+"4027":{"name":"巴南区","parent":"23"},
+"4028":{"name":"北碚区","parent":"23"},
+"4029":{"name":"万州区","parent":"23"},
+"4030":{"name":"黔江区","parent":"23"},
+"4031":{"name":"永川区","parent":"23"},
+"4032":{"name":"涪陵区","parent":"23"},
+"4033":{"name":"江津区","parent":"23"},
+"4034":{"name":"合川区","parent":"23"},
+"4035":{"name":"双桥区","parent":"23"},
+"4036":{"name":"万盛区","parent":"23"},
+"4037":{"name":"荣昌区","parent":"23"},
+"4038":{"name":"大足区","parent":"23"},
+"4039":{"name":"璧山区","parent":"23"},
+"4040":{"name":"铜梁区","parent":"23"},
+"4041":{"name":"潼南区","parent":"23"},
+"4042":{"name":"綦江区","parent":"23"},
+"4043":{"name":"忠县","parent":"23"},
+"4044":{"name":"开州区","parent":"23"},
+"4045":{"name":"云阳县","parent":"23"},
+"4046":{"name":"梁平区","parent":"23"},
+"4047":{"name":"垫江县","parent":"23"},
+"4048":{"name":"丰都县","parent":"23"},
+"4049":{"name":"奉节县","parent":"23"},
+"4050":{"name":"巫山县","parent":"23"},
+"4051":{"name":"巫溪县","parent":"23"},
+"4052":{"name":"城口县","parent":"23"},
+"4053":{"name":"武隆区","parent":"23"},
+"4054":{"name":"石柱土家族自治县","parent":"23"},
+"4055":{"name":"秀山土家族苗族自治县","parent":"23"},
+"4056":{"name":"酉阳土家族苗族自治县","parent":"23"},
+"4057":{"name":"彭水苗族土家族自治县","parent":"23"},
+"4058":{"name":"潜江市","parent":"18"},
+"4059":{"name":"三沙市","parent":"22"},
+"4060":{"name":"石河子市","parent":"32"},
+"4061":{"name":"阿拉尔市","parent":"32"},
+"4062":{"name":"图木舒克市","parent":"32"},
+"4063":{"name":"五家渠市","parent":"32"},
+"4064":{"name":"北屯市","parent":"32"},
+"4065":{"name":"铁门关市","parent":"32"},
+"4066":{"name":"儋州市","parent":"22"},
+"4067":{"name":"五指山市","parent":"22"},
+"4068":{"name":"文昌市","parent":"22"},
+"4069":{"name":"琼海市","parent":"22"},
+"4070":{"name":"万宁市","parent":"22"},
+"4072":{"name":"定安县","parent":"22"},
+"4073":{"name":"屯昌县","parent":"22"},
+"4074":{"name":"澄迈县","parent":"22"},
+"4075":{"name":"临高县","parent":"22"},
+"4076":{"name":"琼中黎族苗族自治县","parent":"22"},
+"4077":{"name":"保亭黎族苗族自治县","parent":"22"},
+"4078":{"name":"白沙黎族自治县","parent":"22"},
+"4079":{"name":"昌江黎族自治县","parent":"22"},
+"4080":{"name":"乐东黎族自治县","parent":"22"},
+"4081":{"name":"陵水黎族自治县","parent":"22"},
+"4082":{"name":"马来西亚","parent":"3956"},
+"6047":{"name":"长寿区","parent":"23"},
+"6857":{"name":"阿富汗","parent":"3956"},
+"6858":{"name":"阿尔巴尼亚","parent":"3956"},
+"6859":{"name":"阿尔及利亚","parent":"3956"},
+"6860":{"name":"美属萨摩亚","parent":"3956"},
+"6861":{"name":"安道尔","parent":"3956"},
+"6862":{"name":"安哥拉","parent":"3956"},
+"6863":{"name":"安圭拉","parent":"3956"},
+"6864":{"name":"南极洲","parent":"3956"},
+"6865":{"name":"安提瓜和巴布达","parent":"3956"},
+"6866":{"name":"阿根廷","parent":"3956"},
+"6867":{"name":"亚美尼亚","parent":"3956"},
+"6869":{"name":"奥地利","parent":"3956"},
+"6870":{"name":"阿塞拜疆","parent":"3956"},
+"6871":{"name":"巴哈马","parent":"3956"},
+"6872":{"name":"巴林","parent":"3956"},
+"6873":{"name":"孟加拉国","parent":"3956"},
+"6874":{"name":"巴巴多斯","parent":"3956"},
+"6875":{"name":"白俄罗斯","parent":"3956"},
+"6876":{"name":"比利时","parent":"3956"},
+"6877":{"name":"伯利兹","parent":"3956"},
+"6878":{"name":"贝宁","parent":"3956"},
+"6879":{"name":"百慕大","parent":"3956"},
+"6880":{"name":"不丹","parent":"3956"},
+"6881":{"name":"玻利维亚","parent":"3956"},
+"6882":{"name":"波黑","parent":"3956"},
+"6883":{"name":"博茨瓦纳","parent":"3956"},
+"6884":{"name":"布维岛","parent":"3956"},
+"6885":{"name":"巴西","parent":"3956"},
+"6886":{"name":"英属印度洋领土","parent":"3956"},
+"6887":{"name":"文莱","parent":"3956"},
+"6888":{"name":"保加利亚","parent":"3956"},
+"6889":{"name":"布基纳法索","parent":"3956"},
+"6890":{"name":"布隆迪","parent":"3956"},
+"6891":{"name":"柬埔寨","parent":"3956"},
+"6892":{"name":"喀麦隆","parent":"3956"},
+"6893":{"name":"佛得角","parent":"3956"},
+"6894":{"name":"开曼群岛","parent":"3956"},
+"6895":{"name":"中非","parent":"3956"},
+"6896":{"name":"乍得","parent":"3956"},
+"6897":{"name":"智利","parent":"3956"},
+"6898":{"name":"圣诞岛","parent":"3956"},
+"6899":{"name":"科科斯（基林）群岛","parent":"3956"},
+"6900":{"name":"哥伦比亚","parent":"3956"},
+"6901":{"name":"科摩罗","parent":"3956"},
+"6902":{"name":"刚果（布）","parent":"3956"},
+"6903":{"name":"刚果（金）","parent":"3956"},
+"6904":{"name":"库克群岛","parent":"3956"},
+"6905":{"name":"哥斯达黎加","parent":"3956"},
+"6906":{"name":"科特迪瓦","parent":"3956"},
+"6907":{"name":"克罗地亚","parent":"3956"},
+"6908":{"name":"古巴","parent":"3956"},
+"6909":{"name":"塞浦路斯","parent":"3956"},
+"6910":{"name":"捷克","parent":"3956"},
+"6911":{"name":"丹麦","parent":"3956"},
+"6912":{"name":"吉布提","parent":"3956"},
+"6913":{"name":"多米尼克","parent":"3956"},
+"6914":{"name":"多米尼加共和国","parent":"3956"},
+"6915":{"name":"东帝汶","parent":"3956"},
+"6916":{"name":"厄瓜多尔","parent":"3956"},
+"6917":{"name":"埃及","parent":"3956"},
+"6918":{"name":"萨尔瓦多","parent":"3956"},
+"6919":{"name":"赤道几内亚","parent":"3956"},
+"6920":{"name":"厄立特里亚","parent":"3956"},
+"6921":{"name":"爱沙尼亚","parent":"3956"},
+"6922":{"name":"埃塞俄比亚","parent":"3956"},
+"6923":{"name":"福克兰群岛（马尔维纳斯）","parent":"3956"},
+"6924":{"name":"法罗群岛","parent":"3956"},
+"6925":{"name":"斐济","parent":"3956"},
+"6926":{"name":"芬兰","parent":"3956"},
+"6927":{"name":"法属圭亚那","parent":"3956"},
+"6928":{"name":"法属波利尼西亚","parent":"3956"},
+"6929":{"name":"法属南部领土","parent":"3956"},
+"6930":{"name":"加蓬","parent":"3956"},
+"6931":{"name":"冈比亚","parent":"3956"},
+"6932":{"name":"格鲁吉亚","parent":"3956"},
+"6933":{"name":"加纳","parent":"3956"},
+"6934":{"name":"直布罗陀","parent":"3956"},
+"6935":{"name":"希腊","parent":"3956"},
+"6936":{"name":"格陵兰","parent":"3956"},
+"6937":{"name":"格林纳达","parent":"3956"},
+"6938":{"name":"瓜德罗普","parent":"3956"},
+"6939":{"name":"关岛","parent":"3956"},
+"6940":{"name":"危地马拉","parent":"3956"},
+"6941":{"name":"几内亚","parent":"3956"},
+"6942":{"name":"几内亚比绍","parent":"3956"},
+"6943":{"name":"圭亚那","parent":"3956"},
+"6944":{"name":"海地","parent":"3956"},
+"6945":{"name":"赫德岛和麦克唐纳岛","parent":"3956"},
+"6946":{"name":"洪都拉斯","parent":"3956"},
+"6947":{"name":"匈牙利","parent":"3956"},
+"6948":{"name":"冰岛","parent":"3956"},
+"6949":{"name":"印度","parent":"3956"},
+"6950":{"name":"印度尼西亚","parent":"3956"},
+"6951":{"name":"伊朗","parent":"3956"},
+"6952":{"name":"伊拉克","parent":"3956"},
+"6953":{"name":"爱尔兰","parent":"3956"},
+"6954":{"name":"以色列","parent":"3956"},
+"6955":{"name":"牙买加","parent":"3956"},
+"6956":{"name":"约旦","parent":"3956"},
+"6957":{"name":"哈萨克斯坦","parent":"3956"},
+"6958":{"name":"肯尼亚","parent":"3956"},
+"6959":{"name":"基里巴斯","parent":"3956"},
+"6960":{"name":"朝鲜","parent":"3956"},
+"6961":{"name":"科威特","parent":"3956"},
+"6962":{"name":"吉尔吉斯斯坦","parent":"3956"},
+"6963":{"name":"老挝","parent":"3956"},
+"6964":{"name":"拉脱维亚","parent":"3956"},
+"6965":{"name":"黎巴嫩","parent":"3956"},
+"6966":{"name":"莱索托","parent":"3956"},
+"6967":{"name":"利比里亚","parent":"3956"},
+"6968":{"name":"利比亚","parent":"3956"},
+"6969":{"name":"列支敦士登","parent":"3956"},
+"6970":{"name":"立陶宛","parent":"3956"},
+"6971":{"name":"卢森堡","parent":"3956"},
+"6972":{"name":"前南马其顿","parent":"3956"},
+"6973":{"name":"马达加斯加","parent":"3956"},
+"6974":{"name":"马拉维","parent":"3956"},
+"6975":{"name":"马尔代夫","parent":"3956"},
+"6976":{"name":"马里","parent":"3956"},
+"6977":{"name":"马耳他","parent":"3956"},
+"6978":{"name":"马绍尔群岛","parent":"3956"},
+"6979":{"name":"马提尼克","parent":"3956"},
+"6980":{"name":"毛里塔尼亚","parent":"3956"},
+"6981":{"name":"毛里求斯","parent":"3956"},
+"6982":{"name":"马约特","parent":"3956"},
+"6983":{"name":"墨西哥","parent":"3956"},
+"6984":{"name":"密克罗尼西亚联邦","parent":"3956"},
+"6985":{"name":"摩尔多瓦","parent":"3956"},
+"6986":{"name":"摩纳哥","parent":"3956"},
+"6987":{"name":"蒙古","parent":"3956"},
+"6988":{"name":"蒙特塞拉特","parent":"3956"},
+"6989":{"name":"摩洛哥","parent":"3956"},
+"6990":{"name":"莫桑比克","parent":"3956"},
+"6991":{"name":"缅甸","parent":"3956"},
+"6992":{"name":"纳米比亚","parent":"3956"},
+"6993":{"name":"瑙鲁","parent":"3956"},
+"6994":{"name":"尼泊尔","parent":"3956"},
+"6995":{"name":"荷兰","parent":"3956"},
+"6996":{"name":"荷属安的列斯","parent":"3956"},
+"6997":{"name":"新喀里多尼亚","parent":"3956"},
+"6998":{"name":"新西兰","parent":"3956"},
+"6999":{"name":"尼加拉瓜","parent":"3956"},
+"7000":{"name":"尼日尔","parent":"3956"},
+"7001":{"name":"尼日利亚","parent":"3956"},
+"7002":{"name":"纽埃","parent":"3956"},
+"7003":{"name":"诺福克岛","parent":"3956"},
+"7004":{"name":"北马里亚纳","parent":"3956"},
+"7005":{"name":"挪威","parent":"3956"},
+"7006":{"name":"阿曼","parent":"3956"},
+"7007":{"name":"巴基斯坦","parent":"3956"},
+"7008":{"name":"帕劳","parent":"3956"},
+"7009":{"name":"巴勒斯坦","parent":"3956"},
+"7010":{"name":"巴拿马","parent":"3956"},
+"7011":{"name":"巴布亚新几内亚","parent":"3956"},
+"7012":{"name":"巴拉圭","parent":"3956"},
+"7013":{"name":"秘鲁","parent":"3956"},
+"7014":{"name":"菲律宾","parent":"3956"},
+"7015":{"name":"皮特凯恩群岛","parent":"3956"},
+"7016":{"name":"波兰","parent":"3956"},
+"7017":{"name":"葡萄牙","parent":"3956"},
+"7018":{"name":"波多黎各","parent":"3956"},
+"7019":{"name":"卡塔尔","parent":"3956"},
+"7020":{"name":"留尼汪","parent":"3956"},
+"7021":{"name":"罗马尼亚","parent":"3956"},
+"7022":{"name":"俄罗斯联邦","parent":"3956"},
+"7023":{"name":"卢旺达","parent":"3956"},
+"7024":{"name":"圣赫勒拿","parent":"3956"},
+"7025":{"name":"圣基茨和尼维斯","parent":"3956"},
+"7026":{"name":"圣卢西亚","parent":"3956"},
+"7027":{"name":"圣皮埃尔和密克隆","parent":"3956"},
+"7028":{"name":"圣文森特和格林纳丁斯","parent":"3956"},
+"7029":{"name":"萨摩亚","parent":"3956"},
+"7030":{"name":"圣马力诺","parent":"3956"},
+"7031":{"name":"圣多美和普林西比","parent":"3956"},
+"7032":{"name":"沙特阿拉伯","parent":"3956"},
+"7033":{"name":"塞内加尔","parent":"3956"},
+"7034":{"name":"塞舌尔","parent":"3956"},
+"7035":{"name":"塞拉利昂","parent":"3956"},
+"7036":{"name":"新加坡","parent":"3956"},
+"7037":{"name":"斯洛伐克","parent":"3956"},
+"7038":{"name":"斯洛文尼亚","parent":"3956"},
+"7039":{"name":"所罗门群岛","parent":"3956"},
+"7040":{"name":"索马里","parent":"3956"},
+"7041":{"name":"南非","parent":"3956"},
+"7042":{"name":"南乔治亚岛和南桑德韦奇岛","parent":"3956"},
+"7043":{"name":"斯里兰卡","parent":"3956"},
+"7044":{"name":"苏丹","parent":"3956"},
+"7045":{"name":"苏里南","parent":"3956"},
+"7046":{"name":"斯瓦尔巴群岛","parent":"3956"},
+"7047":{"name":"斯威士兰","parent":"3956"},
+"7048":{"name":"瑞典","parent":"3956"},
+"7049":{"name":"瑞士","parent":"3956"},
+"7050":{"name":"叙利亚","parent":"3956"},
+"7051":{"name":"塔吉克斯坦","parent":"3956"},
+"7052":{"name":"坦桑尼亚","parent":"3956"},
+"7053":{"name":"泰国","parent":"3956"},
+"7054":{"name":"多哥","parent":"3956"},
+"7055":{"name":"托克劳","parent":"3956"},
+"7056":{"name":"汤加","parent":"3956"},
+"7057":{"name":"特立尼达和多巴哥","parent":"3956"},
+"7058":{"name":"突尼斯","parent":"3956"},
+"7059":{"name":"土耳其","parent":"3956"},
+"7060":{"name":"土库曼斯坦","parent":"3956"},
+"7061":{"name":"特克斯科斯群岛","parent":"3956"},
+"7062":{"name":"图瓦卢","parent":"3956"},
+"7063":{"name":"乌干达","parent":"3956"},
+"7064":{"name":"乌克兰","parent":"3956"},
+"7065":{"name":"阿联酋","parent":"3956"},
+"7066":{"name":"美国本土外小岛屿","parent":"3956"},
+"7067":{"name":"乌拉圭","parent":"3956"},
+"7068":{"name":"乌兹别克斯坦","parent":"3956"},
+"7069":{"name":"瓦努阿图","parent":"3956"},
+"7070":{"name":"梵蒂冈","parent":"3956"},
+"7071":{"name":"委内瑞拉","parent":"3956"},
+"7072":{"name":"越南","parent":"3956"},
+"7073":{"name":"英属维尔京群岛","parent":"3956"},
+"7074":{"name":"美属维尔京群岛","parent":"3956"},
+"7075":{"name":"瓦利斯和富图纳","parent":"3956"},
+"7076":{"name":"西撒哈拉","parent":"3956"},
+"7077":{"name":"也门","parent":"3956"},
+"7078":{"name":"南斯拉夫","parent":"3956"},
+"7079":{"name":"赞比亚","parent":"3956"},
+"7080":{"name":"津巴布韦","parent":"3956"},
+"7081":{"name":"塞尔维亚","parent":"3956"},
+"7082":{"name":"雄安新区","parent":"4"},
+"7084":{"name":"天门市","parent":"18"}
+}
+
+NM_SET = set([v["name"] for _,v in TBL.items()])
+
+def get_names(id):
+    if not id or str(id).lower() == "none":return []
+    id = str(id)
+    if not re.match("[0-9]+$", id.strip()):return [id]
+    nms = []
+    d = TBL.get(id)
+    if not d:return[]
+    nms.append(d["name"])
+    p = get_names(d["parent"])
+    if p: nms.extend(p)
+    return nms
+
+import re
+def isName(nm):
+    if nm in NM_SET:return True
+    if nm + "市" in NM_SET:return True
+    if re.sub(r"(省|(回族|壮族|维吾尔)*自治区)$", "", nm) in NM_SET:return True
+    return False

deepdoc/parser/resume/entities/res/corp.tks.freq.json

@@ -0,0 +1,65 @@
+[
+        "科技",
+        "集团",
+        "网络科技",
+        "技术",
+        "信息",
+        "分公司",
+        "信息技术",
+        "发展",
+        "科技股份",
+        "网络",
+        "贸易",
+        "商贸",
+        "工程",
+        "企业",
+        "集团股份",
+        "商务",
+        "工业",
+        "控股集团",
+        "国际贸易",
+        "软件技术",
+        "数码科技",
+        "软件开发",
+        "有限",
+        "经营",
+        "科技开发",
+        "股份公司",
+        "电子技术",
+        "实业集团",
+        "责任",
+        "无限",
+        "工程技术",
+        "上市公司",
+        "技术开发",
+        "软件系统",
+        "总公司",
+        "网络服务",
+        "ltd.",
+        "technology",
+        "company",
+        "服务公司",
+        "计算机技术",
+        "计算机软件",
+        "电子信息",
+        "corporation",
+        "计算机服务",
+        "计算机系统",
+	"有限公司",
+	"事业部",
+	"公司",
+	"股份",
+	"有限责任",
+	"软件",
+	"控股",
+	"高科技",
+	"房地产",
+	"事业群",
+	"部门",
+	"电子商务",
+	"人力资源顾问",
+	"人力资源",
+	"株式会社",
+	"网络营销"
+]
+

deepdoc/parser/resume/entities/res/good_corp.json

@@ -0,0 +1,911 @@
+[
+    "google assistant investments",
+    "amazon",
+    "dingtalk china information",
+    "zhejiang alibaba communication",
+    "yunos",
+    "腾讯云",
+    "新浪新闻",
+    "网邻通",
+    "蚂蚁集团",
+    "大疆",
+    "恒生股份",
+    "sf express",
+    "智者天下",
+    "shanghai hema network",
+    "papayamobile",
+    "lexinfintech",
+    "industrial consumer finance",
+    "360搜索",
+    "世纪光速",
+    "迅雷区块链",
+    "赛盒科技",
+    "齐力电子商务",
+    "平安养老险",
+    "平安证券",
+    "平安好贷",
+    "五八新服",
+    "呯嘭智能",
+    "阿里妈妈",
+    "mdt",
+    "tencent",
+    "weibo",
+    "浪潮软件",
+    "阿里巴巴广告",
+    "mashang consumer finance",
+    "维沃",
+    "hqg , limited",
+    "moodys",
+    "搜狐支付",
+    "百度秀",
+    "新浪服务",
+    "零售通",
+    "同城艺龙",
+    "虾米音乐",
+    "贝壳集团",
+    "小米有品",
+    "滴滴自动驾驶",
+    "图记",
+    "阿里影业",
+    "卓联软件",
+    "zhejiang tmall",
+    "谷歌中国",
+    "hithink flush",
+    "时装科技",
+    "程会玩国际旅行社",
+    "amazon china holding limited",
+    "中信消金",
+    "当当比特物流",
+    "新浪新媒体咨询",
+    "tongcheng network",
+    "金山在线",
+    "shopping cart",
+    "犀互动",
+    "五八",
+    "bilibili",
+    "阿里星球",
+    "滴滴金科服务",
+    "美团",
+    "哈啰出行",
+    "face",
+    "平安健康",
+    "招商银行",
+    "连亚",
+    "盒马网络",
+    "b站",
+    "华为机器",
+    "shanghai mdt infotech",
+    "ping an healthkonnect",
+    "beijing home link real estate broker",
+    "花海仓",
+    "beijing jingdong shangke information",
+    "微影智能",
+    "酷狗游戏",
+    "health.pingan.com",
+    "众安",
+    "陌陌",
+    "海康威视数字",
+    "同程网",
+    "艾丁金融",
+    "知乎",
+    " lu",
+    "国际商业机器公司",
+    "捷信消费金融",
+    "恒生利融",
+    "china merchants bank",
+    "企鹅电竞",
+    "捷信信驰",
+    "360智能家居",
+    "小桔车服",
+    "homecredit",
+    "皮皮虾",
+    "畅游",
+    "聚爱聊",
+    "suning.com",
+    "途牛旅游网",
+    "花呗",
+    "盈店通",
+    "sina",
+    "阿里巴巴音乐",
+    "华为技术有限公司",
+    "国付宝",
+    "shanghai lianshang network",
+    "oppo",
+    "华为投资控股",
+    "beijing sohu new media information",
+    "times square",
+    "菜鸟物流",
+    "lingxing",
+    "jd digits",
+    "同程旅游",
+    "分期乐",
+    "火锅视频",
+    "天天快报",
+    "猎豹移动",
+    "五八人力资源",
+    "宝宝树",
+    "顺丰科技",
+    "上海西翠",
+    "诗程文化传播",
+    "dewu",
+    "领星网络",
+    "aliexpress",
+    "贝塔通科技",
+    "链家",
+    "花小猪",
+    "趣输入",
+    "搜狐新媒体",
+    "一淘",
+    "56",
+    "qq阅读",
+    "青桔单车",
+    "iflytek",
+    "每日优鲜电子商务",
+    "腾讯觅影",
+    "微医",
+    "松果网",
+    "paypal",
+    "递瑞供应链管理",
+    "领星",
+    "qunar",
+    "三快",
+    "lu.com",
+    "携程旅行网",
+    "新潮传媒",
+    "链家经纪",
+    "景域文化",
+    "阿里健康",
+    "pingpeng",
+    "聚划算",
+    "零机科技",
+    "街兔电单车",
+    "快乐购",
+    "华为数字能源",
+    "搜狐",
+    "陆家嘴国际金融资产交易市场",
+    "nanjing tuniu",
+    "亚马逊",
+    "苏宁易购",
+    "携程旅游",
+    "苏宁金服",
+    "babytree",
+    "悟空问答",
+    "同花顺",
+    "eastmoney",
+    "浪潮信息",
+    "滴滴智慧交通",
+    "beijing ruixun lingtong",
+    "平安综合金融服务",
+    "爱奇艺",
+    "小米集团",
+    "华为云",
+    "微店",
+    "恒生集团",
+    "网易有道",
+    "boccfc",
+    "世纪思速科技",
+    "海康消防",
+    "beijing xiaomi",
+    "众安科技",
+    "五八同城",
+    "霆程汽车租赁",
+    "云卖分销",
+    "乐信集团",
+    "蚂蚁",
+    "舶乐蜜电子商务",
+    "支付宝中国",
+    "砖块消消消",
+    "vivo",
+    "阿里互娱",
+    "中国平安",
+    "lingxihudong",
+    "百度网盘",
+    "1号店",
+    "字节跳动",
+    "京东科技",
+    "驴妈妈兴旅国际旅行社",
+    "hangzhou alibaba music",
+    "xunlei",
+    "灵犀互动娱乐",
+    "快手",
+    "youtube",
+    "连尚慧眼",
+    "腾讯体育",
+    "爱商在线",
+    "酷我音乐",
+    "金融壹账通",
+    "搜狗服务",
+    "banma information",
+    "a站",
+    "罗汉堂",
+    "薇仕网络",
+    "搜狐新闻",
+    "贝宝",
+    "薇仕",
+    "口袋时尚科技",
+    "穆迪咨询",
+    "新狐投资管理",
+    "hikvision",
+    "alimama china holding limited",
+    "超聚变数字",
+    "腾讯视频",
+    "恒生电子",
+    "百度游戏",
+    "绿洲",
+    "木瓜移动",
+    "红袖添香",
+    "店匠科技",
+    "易贝",
+    "一淘网",
+    "博览群书",
+    "唯品会",
+    "lazglobal",
+    "amap",
+    "芒果网",
+    "口碑",
+    "海康慧影",
+    "腾讯音乐娱乐",
+    "网易严选",
+    "微信",
+    "shenzhen lexin holding",
+    "hangzhou pingpeng intelligent",
+    "连尚网络",
+    "海思",
+    "isunor",
+    "蝉翼",
+    "阿里游戏",
+    "广州优视",
+    "优视",
+    "腾讯征信",
+    "识装",
+    "finserve.pingan.com",
+    "papaya",
+    "阅文",
+    "平安健康保险",
+    "考拉海购",
+    "网易印象",
+    "wifi万能钥匙",
+    "新浪互联服务",
+    "亚马逊云科技",
+    "迅雷看看",
+    "华为朗新科技",
+    "adyen hong kong limited",
+    "谷歌",
+    "得物",
+    "网心",
+    "cainiao network",
+    "沐瞳",
+    "linkedln",
+    "hundsun",
+    "阿里旅行",
+    "珍爱网",
+    "阿里巴巴通信",
+    "金山奇剑",
+    "tongtool",
+    "华为安捷信电气",
+    "快乐时代",
+    "平安寿险",
+    "微博",
+    "微跳蚤",
+    "oppo移动通信",
+    "毒",
+    "alimama",
+    "shoplazza",
+    "shenzhen dianjiang science and",
+    "众鸣世科",
+    "平安金融",
+    "狐友",
+    "维沃移动通信",
+    "tobosoft",
+    "齐力电商",
+    "ali",
+    "诚信通",
+    "行吟",
+    "跳舞的线",
+    "橙心优选",
+    "众安健康",
+    "亚马逊中国投资",
+    "德絮投资管理中心合伙",
+    "招联消费金融",
+    "百度文学",
+    "芝麻信用",
+    "阿里零售通",
+    "时装",
+    "花样直播",
+    "sogou",
+    "uc",
+    "海思半导体",
+    "zhongan online p&c insurance",
+    "新浪数字",
+    "驴妈妈旅游网",
+    "华为数字能源技术",
+    "京东数科",
+    "oracle",
+    "xiaomi",
+    "nyse",
+    "阳光消费金融",
+    "天天动听",
+    "大众点评",
+    "上海瑞家",
+    "trustpass",
+    "hundsun technologies",
+    "美团小贷",
+    "ebay",
+    "通途",
+    "tcl",
+    "鸿蒙",
+    "酷狗计算机",
+    "品诺保险",
+    "capitalg",
+    "康盛创想",
+    "58同城",
+    "闲鱼",
+    "微软",
+    "吉易付科技",
+    "理财通",
+    "ctrip",
+    "yy",
+    "华为数字",
+    "kingsoft",
+    "孙宁金融",
+    "房江湖经纪",
+    "youku",
+    "ant financial services group",
+    "盒马",
+    "sensetime",
+    "伊千网络",
+    "小豹ai翻译棒",
+    "shopify",
+    "前海微众银行",
+    "qd",
+    "gmail",
+    "pingpong",
+    "alibaba group holding limited",
+    "捷信时空电子商务",
+    "orientsec",
+    "乔戈里管理咨询",
+    "ant",
+    "锐讯灵通",
+    "兴业消费金融",
+    "京东叁佰陆拾度电子商务",
+    "新浪",
+    "优酷土豆",
+    "海康机器人",
+    "美团单车",
+    "海康存储",
+    "领英",
+    "阿里全球速卖通",
+    "美菜网",
+    "京邦达",
+    "安居客",
+    "阿里体育",
+    "相互宝",
+    "cloudwalk",
+    "百度智能云",
+    "贝壳",
+    "酷狗",
+    "sunshine consumer finance",
+    "掌宜",
+    "奇酷网",
+    "核新同花顺",
+    "阿里巴巴影业",
+    "节创",
+    "学而思网校",
+    "速途",
+    "途牛",
+    "阿里云计算",
+    "beijing sensetime",
+    "alibaba cloud",
+    "西瓜视频",
+    "美团优选",
+    "orient securities limited",
+    "华为朗新",
+    "店匠",
+    "shanghai weishi network",
+    "友盟",
+    "飞猪旅行",
+    "滴滴出行",
+    "alipay",
+    "mogu",
+    "dangdang",
+    "大麦网",
+    "汉军智能系统",
+    "百度地图",
+    "货车帮",
+    "狐狸金服",
+    "众安在线保险经纪",
+    "华为通信",
+    "新浪支付",
+    "zhihu",
+    "alibaba cloud computing",
+    "沙发视频",
+    "金山软件",
+    "ping an good doctor",
+    "携程",
+    "脉脉",
+    "youku information beijing",
+    "zhongan",
+    "艾丁软件",
+    "乒乓智能",
+    "蘑菇街",
+    "taobao",
+    "华为技术服务",
+    "仕承文化传播",
+    "安捷信",
+    "狐狸互联网小额贷款",
+    "节点迅捷",
+    "中国银行",
+    "搜镇",
+    "众安在线",
+    "dingtalk",
+    "云从科技",
+    "beijing jingbangda trade",
+    "moody s",
+    "滚动的天空",
+    "yl.pingan.com",
+    "奇虎",
+    "alihealth",
+    "芒果tv",
+    "lufax",
+    "美团打车",
+    "小桔",
+    "贝壳找房网",
+    "小米科技",
+    "vips",
+    "kindle",
+    "亚马逊服务",
+    "citic consumer finance",
+    "微众",
+    "搜狗智慧互联网医院",
+    "盒马鲜生",
+    "life.pinan.com",
+    "ph.com.cn",
+    "银联",
+    "cmbchina",
+    "平安金融科技咨询",
+    "微保",
+    "甲骨文中国",
+    "飞书",
+    "koubei shanghai information",
+    "企鹅辅导",
+    "斑马",
+    "平安租赁",
+    "云从",
+    "马上消费",
+    "hangzhou ali baba advertising",
+    "金山",
+    "赛盒",
+    "科大讯飞",
+    "金星创业投资",
+    "平安国际融资租赁",
+    "360你财富",
+    "西山居",
+    "shenzhen qianhai fourth paradigm data",
+    "海思光电子",
+    "猎户星空",
+    "网易公司",
+    "浪潮",
+    "粒粒橙传媒",
+    "招联金融",
+    "100. me",
+    "捷信信驰咨询",
+    "唯品仓",
+    "orient",
+    "趣拿",
+    "摩拜单车",
+    "天猫精灵",
+    "菜鸟",
+    "豹小贩",
+    "去哪儿",
+    "米家",
+    "哈啰单车",
+    "搜狐体育",
+    "shopify payments usa",
+    "高德软件",
+    "讯联智付",
+    "乐信",
+    "唯你搭",
+    "第四范式",
+    "菜鸟网络",
+    "同程",
+    "yy语音",
+    "浪潮云",
+    "东财",
+    "淘宝",
+    "寻梦",
+    "citic securities limited",
+    "青橙之旅",
+    "阿里巴巴",
+    "番茄小说",
+    "上海亿贝",
+    "inspur",
+    "babytree inc",
+    "海康智慧产业股权投资基金合伙合伙",
+    "adyen",
+    "艺龙",
+    "蚂蚁金服",
+    "平安金服",
+    "百度百科",
+    "unionpay",
+    "当当",
+    "阅文集团",
+    "东方财富",
+    "东方证券",
+    "哈罗单车",
+    "优酷",
+    "海康",
+    "alipay china network",
+    "网商银行",
+    "钧正",
+    "property.pingan.com",
+    "豹咖啡",
+    "网易",
+    "我爱cba",
+    "theduapp",
+    "360",
+    "金山数字娱乐",
+    "新浪阅读",
+    "alibabagames",
+    "顺丰",
+    "支点商贸",
+    "同程旅行",
+    "citic securities",
+    "ele.com",
+    "tal",
+    "fresh hema",
+    "运满满",
+    "贝壳网",
+    "酷狗音乐",
+    "鲜城",
+    "360健康",
+    "浪潮世科",
+    "迅雷网络",
+    "哔哩哔哩",
+    "华为电动",
+    "淘友天下",
+    "华多网络",
+    "xunlei networking technologies",
+    "云杉",
+    "当当网电子商务",
+    "津虹网络",
+    "wedoc cloud hangzhou holdings",
+    "alisports shanghai",
+    "旷视金智",
+    "钉钉中国",
+    "微影",
+    "金山快快",
+    "亿贝",
+    "wedoc",
+    "autonavi",
+    "哈啰助力车",
+    "google cloud",
+    "新浪乐居",
+    "京东股票",
+    "搜狗智慧远程医疗中心",
+    "中银消金",
+    "merchants union consumer finance",
+    "王者荣耀",
+    "百度手机",
+    "美团民宿",
+    "kaola",
+    "小屋",
+    "金山网络",
+    "来往",
+    "顺丰速运",
+    "腾讯课堂",
+    "百度在线网络",
+    "美团买菜",
+    "威视汽车",
+    "uc mobile",
+    "来赞达",
+    "平安健康医疗",
+    "豹小秘",
+    "尚网",
+    "哈勃投资",
+    " ping an insurance group of china ,",
+    "小米",
+    "360好药",
+    "qq音乐",
+    "lingxigames",
+    "faceu激萌",
+    "搜狗",
+    "sohu",
+    "满帮",
+    "vipshop",
+    "wishpost",
+    "金山世游",
+    "shanghai yibaimi network",
+    "1688",
+    "海康汽车",
+    "顺丰控股",
+    "华为",
+    "妙镜vr",
+    "paybkj.com",
+    "hellobike",
+    "豹来电",
+    "京东",
+    "驴妈妈",
+    "momo",
+    "平安健康险",
+    "哈勃科技",
+    "美菜",
+    "众安在线财产保险",
+    "海康威视",
+    "east money information",
+    "阿里云",
+    "蝉游记",
+    "余额宝",
+    "屋客",
+    "滴滴",
+    "shopify international limited",
+    "百度",
+    "阿里健康中国",
+    "阿里通信",
+    "微梦创科",
+    "微医云",
+    "轻颜相机",
+    "搜易居",
+    "趣店集团",
+    "美团云",
+    "ant group",
+    "金山云",
+    "beijing express hand",
+    "觅觅",
+    "支付宝",
+    "滴滴承信科技咨询服务",
+    "拼多多",
+    "众安运动",
+    "乞力电商",
+    "youcash",
+    "唯品金融",
+    "陆金所",
+    "本地生活",
+    "sz dji",
+    "海康智能",
+    "魔方网聘",
+    "青藤大学",
+    "international business machines",
+    "学而思",
+    "beijing zhongming century science and",
+    "猎豹清理大师",
+    "asinking",
+    "高德",
+    "苏宁",
+    "优酷网",
+    "艾丁",
+    "中银消费金融",
+    "京东健康",
+    "五八教育",
+    "pingpongx",
+    "搜狐时尚",
+    "阿里广告",
+    "平安财险",
+    "中邮消金",
+    "etao",
+    "怕怕",
+    "nyse:cmcm",
+    "华为培训中心",
+    "高德地图",
+    "云狐天下征信",
+    "大疆创新",
+    "连尚",
+    "壹佰米",
+    "康健公司",
+    "iqiyi.com",
+    "360安全云盘",
+    "馒头直播",
+    "淘友网",
+    "东方赢家",
+    "bank of china",
+    "微众银行",
+    "阿里巴巴国际站",
+    "虾米",
+    "去哪儿网",
+    "ctrip travel network shanghai",
+    "潇湘书院",
+    "腾讯",
+    "快乐阳光互动娱乐传媒",
+    "迅雷",
+    "weidian",
+    "滴滴货运",
+    "ping an puhui enterprise management",
+    "新浪仓石基金销售",
+    "搜狐焦点",
+    "alibaba pictures",
+    "wps",
+    "平安",
+    "lazmall",
+    "百度开放平台",
+    "兴业消金",
+    " 珍爱网",
+    "京东云",
+    "小红书",
+    "1688. com",
+    "如视智数",
+    "missfresh",
+    "pazl.pingan.cn",
+    "平安集团",
+    "kugou",
+    "懂车帝",
+    "斑马智行",
+    "浪潮集团",
+    "netease hangzhou network",
+    "pagd.net",
+    "探探",
+    "chinaliterature",
+    "amazon亚马逊",
+    "alphabet",
+    "当当文创手工艺品电子商务",
+    "五八邦",
+    "shenzhen zhenai network information",
+    "lingshoutong",
+    "字节",
+    "lvmama",
+    "金山办公",
+    "众安保险",
+    "时装信息",
+    "优视科技",
+    "guangzhou kugou",
+    "ibm",
+    "滴滴打车",
+    "beijing sogou information service",
+    "megvii",
+    "健谈哥",
+    "cloudwalk group",
+    "蜂联科技",
+    "冬云",
+    "京东尚科",
+    "钢琴块2",
+    "京东世纪",
+    "商汤",
+    "众鸣世纪",
+    "腾讯音乐",
+    "迅雷网文化",
+    "华为云计算技术",
+    "live.me",
+    "全球速卖通",
+    "快的打车",
+    "hello group inc",
+    "美丽说",
+    "suning",
+    "opengauss",
+    "lazada",
+    "tmall",
+    "acfun",
+    "当当网",
+    "中银",
+    "旷视科技",
+    "百度钱包",
+    "淘宝网",
+    "新浪微博",
+    "迅雷集团",
+    "中信消费金融",
+    "学而思教育",
+    "平安普惠",
+    "悟空跨境",
+    "irobotbox",
+    "平安产险",
+    "inspur group",
+    "世纪卓越快递服务",
+    "奇虎360",
+    "webank",
+    "偶藻",
+    "唯品支付",
+    "腾讯云计算",
+    "众安服务",
+    "亿之唐",
+    "beijing 58 information ttechnology",
+    "平安好医生",
+    "迅雷之锤",
+    "旅行小账本",
+    "芒果游戏",
+    "新浪传媒",
+    "旷镜博煊",
+    "全民k歌",
+    "滴滴支付",
+    "北京网心科技",
+    "挂号网",
+    "萤石",
+    "chinavision media group limited",
+    "猎豹安全大师",
+    "cmcm",
+    "趣店",
+    "蚂蚁财富",
+    "商汤科技",
+    "甲骨文",
+    "百度云",
+    "百度apollo",
+    "19 pay",
+    "stock.pingan.com",
+    "tiktok",
+    "alibaba pictures group limited",
+    "ele",
+    "考拉",
+    "天猫",
+    "腾讯优图",
+    "起点中文网",
+    "百度视频",
+    "shanghai bili bili",
+    "京东物流",
+    "ebay marketplaces gmbh",
+    "alibaba sport",
+    "wish",
+    "阿里巴巴中国",
+    "中国银联",
+    "alibaba china network",
+    "china ping an property insurance",
+    "百度糯米网",
+    "微软中国",
+    "一九付",
+    "4 paradigm",
+    "叮咚买菜",
+    "umeng",
+    "众鸣科技",
+    "平安财富通",
+    "google",
+    "巨量引擎",
+    "百度贴吧",
+    "beijing jingdong century information",
+    "讯飞",
+    "beijing yunshan information",
+    "满运软件",
+    "中邮消费金融",
+    "饿了么",
+    "alios",
+    "腾讯ai实验室",
+    "第四范式智能",
+    "瀚星创业投资",
+    "gradient ventures",
+    "microsoft",
+    "哈啰共享汽车",
+    "乞力电子商务",
+    "mscf",
+    "网易影业文化",
+    "铁友旅游咨询",
+    "kilimall",
+    "云企互联投资",
+    "ping an financial consulting",
+    "beijng jingdong century commerce",
+    "高德威智能交通系统",
+    "中友信息",
+    "平安医疗健康管理",
+    "eciticcfc",
+    "中信证券",
+    "fliggy",
+    "电子湾",
+    "旷云金智",
+    "微粒贷",
+    "rsi",
+    "滴滴云计算",
+    "google ventures",
+    "箐程",
+    "每日优鲜",
+    "音兔",
+    "拉扎斯",
+    "今日头条",
+    "乐信控股",
+    "猎豹浏览器",
+    "细微咨询",
+    "好未来",
+    "我乐",
+    "绘声绘色",
+    "抖音",
+    "搜狐新时代",
+    "飞猪",
+    "鹅厂",
+    "贝壳找房",
+    "tuniu",
+    "红马传媒文化",
+    "钉钉",
+    "马上消费金融",
+    "360手机",
+    "平安医保",
+    "快途",
+    "alibaba",
+    "小哈换电",
+    "大麦",
+    "恒睿人工智能研究院",
+    "谷歌资本",
+    "猎豹",
+    "穆迪信息"
+]

deepdoc/parser/resume/entities/res/good_sch.json

@@ -0,0 +1,595 @@
+[
+"中国科技大学",
+"国防科学技术大学",
+"清华大学",
+"清华",
+"tsinghua university",
+"thu",
+"北京大学",
+"北大",
+"beijing university",
+"pku",
+"中国科学技术大学",
+"中国科大",
+"中科大",
+"china science & technology university",
+"ustc",
+"复旦大学",
+"复旦",
+"fudan university",
+"fdu",
+"中国人民大学",
+"人大",
+"人民大学",
+"renmin university of china",
+"ruc",
+"上海交通大学",
+"上海交大",
+"shanghai jiao tong university",
+"sjtu",
+"南京大学",
+"南大",
+"nanjing university",
+"nju",
+"同济大学",
+"同济",
+"tongji university",
+"tongji",
+"浙江大学",
+"浙大",
+"zhejiang university",
+"zju",
+"南开大学",
+"南开",
+"nankai university",
+"nku",
+"北京航空航天大学",
+"北航",
+"beihang university",
+"buaa",
+"北京师范大学",
+"北师",
+"北师大",
+"beijing normal university",
+"bnu",
+"武汉大学",
+"武大",
+"wuhan university",
+"whu",
+"西安交通大学",
+"西安交大",
+"xi’an jiaotong university",
+"xjtu",
+"天津大学",
+"天大",
+"university of tianjin",
+"tju",
+"华中科技大学",
+"华中大",
+"central china university science and technology",
+"hust",
+"北京理工大学",
+"北理",
+"beijing institute of technology",
+"bit",
+"东南大学",
+"东大",
+"southeast china university",
+"seu",
+"中山大学",
+"中大",
+"zhongshan university",
+"sysu",
+"华东师范大学",
+"华师大",
+"east china normal university",
+"ecnu",
+"哈尔滨工业大学",
+"哈工大",
+"harbin institute of technology",
+"hit",
+"厦门大学",
+"厦大",
+"xiamen university",
+"xmu",
+"西北工业大学",
+"西工大",
+"西北工大",
+"northwestern polytechnical university",
+"npu",
+"中南大学",
+"中南",
+"middle and southern university",
+"csu",
+"大连理工大学",
+"大工",
+"institute of technology of dalian",
+"dut",
+"四川大学",
+"川大",
+"sichuan university",
+"scu",
+"电子科技大学",
+"电子科大",
+"university of electronic science and technology of china",
+"uestc",
+"华南理工大学",
+"华南理工",
+"institutes of technology of south china",
+"scut",
+"吉林大学",
+"吉大",
+"jilin university",
+"jlu",
+"湖南大学",
+"湖大",
+"hunan university",
+"hnu",
+"重庆大学",
+"重大",
+"university of chongqing",
+"cqu",
+"山东大学",
+"山大",
+"shandong university",
+"sdu",
+"中国农业大学",
+"中国农大",
+"china agricultural university",
+"cau",
+"中国海洋大学",
+"中国海大",
+"chinese marine university",
+"ouc",
+"中央民族大学",
+"中央民大",
+"central university for nationalities",
+"muc",
+"东北大学",
+"东北工学院",
+"northeastern university",
+"neu 或 nu",
+"兰州大学",
+"兰大",
+"lanzhou university",
+"lzu",
+"西北农林科技大学",
+"西农","西北农大",
+"northwest a&f university",
+"nwafu",
+"中国人民解放军国防科技大学",
+"国防科技大学","国防科大",
+"national university of defense technology",
+"nudt",
+"郑州大学",
+"郑大",
+"zhengzhou university",
+"zzu",
+"云南大学",
+"云大",
+"yunnan university",
+"ynu",
+"新疆大学",
+"新大",
+"xinjiang university",
+"xju",
+"北京交通大学",
+"北京交大",
+"beijing jiaotong university",
+"bjtu",
+"北京工业大学",
+"北工大",
+"beijing university of technology",
+"bjut",
+"北京科技大学",
+"北科大","北京科大",
+"university of science and technology beijing",
+"ustb",
+"北京化工大学",
+"北化",
+"beijing university of chemical technology",
+"buct",
+"北京邮电大学",
+"北邮",
+"beijing university of posts and telecommunications",
+"beijing university of post and telecommunications",
+"beijing university of post and telecommunication",
+"beijing university of posts and telecommunication",
+"bupt",
+"北京林业大学",
+"北林",
+"beijing forestry university",
+"bfu",
+"北京协和医学院",
+"协和医学院",
+"peking union medical college",
+"pumc",
+"北京中医药大学",
+"北中医",
+"beijing university of chinese medicine",
+"bucm",
+"首都师范大学",
+"首师大",
+"capital normal university",
+"cnu",
+"北京外国语大学",
+"北外",
+"beijing foreign studies university",
+"bfsu",
+"中国传媒大学",
+"中媒",
+"中传",
+"北京广播学院",
+"communication university of china",
+"cuc",
+"中央财经大学",
+"中央财大",
+"中财大",
+"the central university of finance and economics",
+"cufe",
+"对外经济贸易大学",
+"对外经贸大学",
+"贸大",
+"university of international business and economics",
+"uibe",
+"外交学院",
+"外院",
+"china foreign affairs university",
+"cfau",
+"中国人民公安大学",
+"公安大学",
+"people's public security university of china",
+"ppsuc",
+"北京体育大学",
+"北体大",
+"beijing sport university",
+"bsu",
+"中央音乐学院",
+"央音",
+"中央院",
+"central conservatory of music",
+"ccom",
+"中国音乐学院",
+"国音",
+"中国院",
+"china conservatory of music",
+"ccmusic",
+"中央美术学院",
+"央美",
+"central academy of fine art",
+"cafa",
+"中央���剧学院",
+"中戏",
+"the central academy of drama",
+"tcad",
+"中国政法大学",
+"法大",
+"china university of political science and law",
+"zuc",
+"cupl",
+"中国科学院大学",
+"国科大",
+"科院大",
+"university of chinese academy of sciences",
+"ucas",
+"福州大学",
+"福大",
+"university of fuzhou",
+"fzu",
+"暨南大学",
+"暨大",
+"ji'nan university",
+"jnu",
+"广州中医药大学",
+"广中医",
+"traditional chinese medicine university of guangzhou",
+"gucm",
+"华南师范大学",
+"华南师大",
+"south china normal university",
+"scnu",
+"广西大学",
+"西大",
+"guangxi university",
+"gxu",
+"贵州大学",
+"贵大",
+"guizhou university",
+"gzu",
+"海南大学",
+"海大",
+"university of hainan",
+"hainu",
+"河南大学",
+"河大",
+"he'nan university",
+"henu",
+"哈尔滨工程大学",
+"哈工程",
+"harbin engineering university",
+"heu",
+"东北农业大学",
+"东北农大",
+"northeast agricultural university",
+"neau",
+"东北林业大学",
+"东北林大",
+"northeast forestry university",
+"nefu",
+"中国地质大学",
+"地大",
+"china university of geosciences",
+"cug",
+"武汉理工大学",
+"武汉理工",
+"wuhan university of technology",
+"wut",
+"华中农业大学",
+"华中农大",
+"华农",
+"central china agricultural university",
+"hzau",
+"华中师范大学",
+"华中师大",
+"华大",
+"central china normal university",
+"ccnu",
+"中南财经政法大学",
+"中南大",
+"zhongnan university of economics & law",
+"zuel",
+"湖南师范大学",
+"湖南师大",
+"hunan normal university",
+"hunnu",
+"延边大学",
+"延大",
+"yanbian university",
+"ybu",
+"东北师范大学",
+"东北师大",
+"northeast normal university",
+"nenu",
+"苏州大学",
+"苏大",
+"soochow university",
+"suda",
+"南京航空航天大学",
+"南航",
+"nanjing aero-space university",
+"nuaa",
+"南京理工大学",
+"南理工",
+"institutes of technology of nanjing",
+"njust",
+"中国矿业大学",
+"中国矿大",
+"china mining university",
+"cumt",
+"南京邮电大学",
+"南邮",
+"nanjing university of posts and telecommunications",
+"njupt",
+"河海大学",
+"河海",
+"river sea university",
+"hhu",
+"江南大学",
+"江南大",
+"jiangnan university",
+"jiangnan",
+"南京林业大学",
+"南林",
+"nanjing forestry university",
+"njfu",
+"南京信息工程大学",
+"南信大",
+"nanjing university of information science and technology",
+"nuist",
+"南京农业大学",
+"南农",
+"南农大",
+"南京农大",
+"agricultural university of nanjing",
+"njau",
+"nau",
+"南京中医药大学",
+"南中医",
+"nanjing university of chinese medicine",
+"njucm",
+"中国药科大学",
+"中国药大",
+"china medicine university",
+"cpu",
+"南京师范大学",
+"南京师大",
+"南师大",
+"南师",
+"nanjing normal university",
+"nnu",
+"南昌大学",
+"昌大",
+"university of nanchang","nanchang university",
+"ncu",
+"辽宁大学",
+"辽大",
+"liaoning university",
+"lnu",
+"大连海事大学",
+"大连海大",
+"海大",
+"maritime affairs university of dalian",
+"dmu",
+"内蒙古大学",
+"内大",
+"university of the inner mongol","inner mongolia university",
+"imu",
+"宁夏大学",
+"宁大",
+"ningxia university",
+"nxu",
+"青海大学",
+"清大",
+"qinghai university",
+"qhu",
+"中国石油大学",
+"中石大",
+"china university of petroleum beijing",
+"upc",
+"太原理工大学",
+"太原理工",
+"institutes of technology of taiyuan","taiyuan university of technology",
+"tyut",
+"西北大学",
+"西大",
+"northwest university",
+"nwu",
+"西安电子科技大学",
+"西电",
+"xidian university",
+"xdu",
+"长安大学",
+"长大",
+"chang`an university",
+"chu",
+"陕西师范大学",
+"陕西师大",
+"陕师大",
+"shaanxi normal university",
+"snnu",
+"第四军医大学",
+"空军军医大学","四医大",
+"air force medical university",
+"fmmu",
+"华东理工大学",
+"华理",
+"east china university of science",
+"ecust",
+"东华大学",
+"东华",
+"donghua university",
+"dhu",
+"上海海洋大学",
+"上海海大",
+"shanghai ocean university",
+"shou",
+"上海中医药大学",
+"上中医",
+"shanghai university of traditional chinese medicine",
+"shutcm",
+"上海外国语大学",
+"上外",
+"shanghai international studies university",
+"sisu",
+"上海财经大学",
+"上海财大",
+"上财",
+"shanghai university of finance",
+"sufe",
+"上海体育学院",
+"shanghai university of sport",
+"上海音乐学院",
+"上音",
+"shanghai conservatory of music",
+"shcm",
+"上海大学",
+"上大",
+"shanghai university",
+"第二军医大学",
+"海军军医大学",
+"naval medical university",
+"西南交通大学",
+"西南交大",
+"southwest jiaotong university",
+"swjtu",
+"西南石油大学",
+"西南石大",
+"southwest petroleum university",
+"swpu",
+"成都理工大学",
+"成都理工",
+"chengdu university of technology",
+"cdut ",
+"四川农业大学",
+"川农",
+"川农大",
+"sichuan agricultural university",
+"sicau",
+"成都中医药大学",
+"成中医",
+"chengdu university of tcm",
+"cdutcm",
+"西南财经大学",
+"西南财大",
+"西财",
+"southwestern university of finance and economics",
+"swufe",
+"天津工业大学",
+"天工大",
+"tianjin university of technology",
+"tgu",
+"天津医科大学",
+"天津医大",
+"medical university of tianjin",
+"tmu",
+"天津中医药大学",
+"天中",
+"tianjin university of traditional chinese medicine",
+"tutcm",
+"华北电力大学",
+"华电",
+"north china electric power university",
+"ncepu",
+"河北工业大学",
+"河工大",
+"hebei university of technology",
+"hebut",
+"西藏大学",
+"藏大",
+"tibet university",
+"tu",
+"石河子大学",
+"石大",
+"shihezi university",
+"中国美术学院",
+"中国美院",
+"国美",
+"china academy of art",
+"caa",
+"宁波大学",
+"宁大",
+"ningbo university",
+"nbu",
+"西南大学",
+"西大",
+"southwest university",
+"swu",
+"安徽大学",
+"安大",
+"university of anhui",
+"ahu",
+"合肥工业大学",
+"合肥工大",
+"合工大",
+"hefei university of technology",
+"hfut",
+"中国地质大学",
+"地大",
+"china university of geosciences",
+"cug",
+"中国地质大学",
+"地大",
+"北京地大",
+"cugb",
+"中国矿业大学",
+"中国矿大",
+"china university of mining & technology",
+"cumtb",
+"中国石油大学",
+"中石大",
+"石大",
+"china university of petroleum",
+"cup",
+"中国石油大学",
+"中石大",
+"cup"]

deepdoc/parser/resume/entities/res/school.rank.csv

@@ -0,0 +1,1627 @@
+清华大学,2,985,清华
+清华大学,2,985,Tsinghua University
+清华大学,2,985,THU
+北京大学,1,985,北大
+北京大学,1,985,Beijing University
+北京大学,1,985,PKU
+中国科学技术大学,14,985,中国科大
+中国科学技术大学,14,985,中科大
+中国科学技术大学,14,985,China Science & Technology University
+中国科学技术大学,14,985,USTC
+复旦大学,5,985,复旦
+复旦大学,5,985,Fudan University
+复旦大学,5,985,FDU
+中国人民大学,15,985,人大
+中国人民大学,15,985,人民大学
+中国人民大学,15,985,Renmin University Of China
+中国人民大学,15,985,RUC
+上海交通大学,4,985,上海交大
+上海交通大学,4,985,Shanghai Jiao Tong University
+上海交通大学,4,985,SJTU
+南京大学,11,985,南大
+南京大学,11,985,Nanjing University
+南京大学,11,985,NJU
+同济大学,17,985,同济
+同济大学,17,985,Tongji University
+同济大学,17,985,Tongji
+浙江大学,3,985,浙大
+浙江大学,3,985,Zhejiang University
+浙江大学,3,985,ZJU
+南开大学,27,985,南开
+南开大学,27,985,Nankai University
+南开大学,27,985,NKU
+北京航空航天大学,21,985,北航
+北京航空航天大学,21,985,Beihang University
+北京航空航天大学,21,985,BUAA
+北京师范大学,23,985,北师
+北京师范大学,23,985,北师大
+北京师范大学,23,985,Beijing Normal University
+北京师范大学,23,985,BNU
+武汉大学,6,985,武大
+武汉大学,6,985,Wuhan University
+武汉大学,6,985,WHU
+西安交通大学,13,985,西安交大
+西安交通大学,13,985,Xi’an Jiaotong University
+西安交通大学,13,985,XJTU
+天津大学,22,985,天大
+天津大学,22,985,University Of Tianjin
+天津大学,22,985,TJU
+华中科技大学,8,985,华中大
+华中科技大学,8,985,Central China University Science and Technology
+华中科技大学,8,985,HUST
+北京理工大学,24,985,北理
+北京理工大学,24,985,Beijing Institute of Technology
+北京理工大学,24,985,BIT
+东南大学,20,985,东大
+东南大学,20,985,Southeast China University
+东南大学,20,985,SEU
+中山大学,9,985,中大
+中山大学,9,985,Zhongshan University
+中山大学,9,985,SYSU
+华东师范大学,30,985,华师大
+华东师范大学,30,985,East China Normal University
+华东师范大学,30,985,ECNU
+哈尔滨工业大学,10,985,哈工大
+哈尔滨工业大学,10,985,Harbin Institute Of Technology
+哈尔滨工业大学,10,985,HIT
+厦门大学,28,985,厦大
+厦门大学,28,985,Xiamen University
+厦门大学,28,985,XMU
+西北工业大学,31,985,西工大
+西北工业大学,31,985,西北工大
+西北工业大学,31,985,Northwestern Polytechnical University
+西北工业大学,31,985,NPU
+中南大学,19,985,中南
+中南大学,19,985,Middle and Southern University
+中南大学,19,985,CSU
+大连理工大学,26,985,大工
+大连理工大学,26,985,Institute Of Technology Of Dalian
+大连理工大学,26,985,DUT
+四川大学,7,985,川大
+四川大学,7,985,Sichuan University
+四川大学,7,985,SCU
+电子科技大学,32,985,电子科大
+电子科技大学,32,985,University of Electronic Science and Technology of China
+电子科技大学,32,985,UESTC
+华南理工大学,25,985,华南理工
+华南理工大学,25,985,Institutes Of Technology Of South China
+华南理工大学,25,985,SCUT
+吉林大学,12,985,吉大
+吉林大学,12,985,Jilin University
+吉林大学,12,985,JLU
+湖南大学,38,985,湖大
+湖南大学,38,985,Hunan University
+湖南大学,38,985,HNU
+重庆大学,29,985,重大
+重庆大学,29,985,University Of Chongqing
+重庆大学,29,985,CQU
+山东大学,16,985,山大
+山东大学,16,985,Shandong University
+山东大学,16,985,SDU
+中国农业大学,37,985,中国农大
+中国农业大学,37,985,China Agricultural University
+中国农业大学,37,985,CAU
+中国海洋大学,60,985,中国海大
+中国海洋大学,60,985,Chinese Marine University
+中国海洋大学,60,985,OUC
+中央民族大学,106,985,中央民大
+中央民族大学,106,985,Central University For Nationalities
+中央民族大学,106,985,MUC
+东北大学,41,985,东北工学院
+东北大学,41,985,Northeastern University
+东北大学,41,985,NEU
+东北大学,41,985,NU
+兰州大学,39,985,兰大
+兰州大学,39,985,Lanzhou University
+兰州大学,39,985,LZU
+西北农林科技大学,78,985,西农、西北农大
+西北农林科技大学,78,985,Northwest A&F University
+西北农林科技大学,78,985,NWAFU
+中国人民解放军国防科技大学,89,985,国防科技大学、国防科大、国防科学技术大学
+中国人民解放军国防科技大学,89,985,National University of Defense Technology
+中国人民解放军国防科技大学,89,985,NUDT
+郑州大学,34,211,郑大
+郑州大学,34,211,Zhengzhou University
+郑州大学,34,211,ZZU
+云南大学,75,211,云大
+云南大学,75,211,Yunnan University
+云南大学,75,211,YNU
+新疆大学,114,211,新大
+新疆大学,114,211,Xinjiang University
+新疆大学,114,211,XJU
+北京交通大学,49,211,北京交大
+北京交通大学,49,211,Beijing Jiaotong University
+北京交通大学,49,211,BJTU
+北京工业大学,63,211,北工大
+北京工业大学,63,211,Beijing University Of Technology
+北京工业大学,63,211,BJUT
+北京科技大学,48,211,北科大、北京科大
+北京科技大学,48,211,University Of Science and Technology Beijing
+北京科技大学,48,211,USTB
+北京化工大学,82,211,北化
+北京化工大学,82,211,Beijing University of Chemical Technology
+北京化工大学,82,211,BUCT
+北京邮电大学,76,211,北邮
+北京邮电大学,76,211,Beijing University Of Posts and Telecommunications
+北京邮电大学,76,211,BUPT
+北京林业大学,88,211,北林
+北京林业大学,88,211,Beijing Forestry University
+北京林业大学,88,211,BFU
+北京协和医学院,,双一流,协和医学院
+北京协和医学院,,双一流,Peking Union Medical College
+北京协和医学院,,双一流,PUMC
+北京中医药大学,121,211,北中医
+北京中医药大学,121,211,Beijing University Of Chinese Medicine
+北京中医药大学,121,211,BUCM
+首都师范大学,,双一流,首师大
+首都师范大学,,双一流,Capital Normal University
+首都师范大学,,双一流,CNU
+北京外国语大学,124,211,北外
+北京外国语大学,124,211,Beijing Foreign Studies University
+北京外国语大学,124,211,BFSU
+中国传媒大学,96,211,中媒
+中国传媒大学,96,211,中传
+中国传媒大学,96,211,北京广播学院
+中国传媒大学,96,211,Communication University Of China
+中国传媒大学,96,211,CUC
+中央财经大学,79,211,中央财大
+中央财经大学,79,211,中财大
+中央财经大学,79,211,The Central University Of Finance and Economics
+中央财经大学,79,211,CUFE
+对外经济贸易大学,99,211,对外经贸大学
+对外经济贸易大学,99,211,贸大
+对外经济贸易大学,99,211,University Of International Business and Economics
+对外经济贸易大学,99,211,UIBE
+外交学院,,双一流,外院
+外交学院,,双一流,China Foreign Affairs University
+外交学院,,双一流,CFAU
+中国人民公安大学,,双一流,公安大学
+中国人民公安大学,,双一流,People's Public Security University of China
+中国人民公安大学,,双一流,PPSUC
+北京体育大学,122,211,北体大
+北京体育大学,122,211,Beijing Sport University
+北京体育大学,122,211,BSU
+中央音乐学院,103,211,央音
+中央音乐学院,103,211,中央院
+中央音乐学院,103,211,Central Conservatory Of Music
+中央音乐学院,103,211,CCOM
+中国音乐学院,,双一流,国音
+中国音乐学院,,双一流,中国院
+中国音乐学院,,双一流,China Conservatory of Music
+中国音乐学院,,双一流,CCMUSIC
+中央美术学院,104,双一流,央美
+中央美术学院,104,双一流,Central Academy of Fine Art
+中央美术学院,104,双一流,CAFA
+中央戏剧学院,133,双一流,中戏
+中央戏剧学院,133,双一流,The Central Academy Of Drama
+中央戏剧学院,133,双一流,TCAD
+中国政法大学,90,211,法大
+中国政法大学,90,211,China University Of Political Science and Law
+中国政法大学,90,211,ZUC
+中国政法大学,90,211,CUPL
+中国科学院大学,18,双一流,国科大
+中国科学院大学,18,双一流,科院大
+中国科学院大学,18,双一流,University of Chinese Academy of Sciences
+中国科学院大学,18,双一流,UCAS
+福州大学,72,211,福大
+福州大学,72,211,University Of Fuzhou
+福州大学,72,211,FZU
+暨南大学,44,211,暨大
+暨南大学,44,211,Ji'nan University
+暨南大学,44,211,JNU
+广州中医药大学,,双一流,广中医
+广州中医药大学,,双一流,Traditional Chinese Medicine University Of Guangzhou
+广州中医药大学,,双一流,GUCM
+华南师范大学,55,211,华南师大
+华南师范大学,55,211,South China Normal University
+华南师范大学,55,211,SCNU
+广西大学,71,211,广西大
+广西大学,71,211,Guangxi University
+广西大学,71,211,GXU
+贵州大学,94,211,贵大
+贵州大学,94,211,Guizhou University
+贵州大学,94,211,GZU
+海南大学,101,211,海大
+海南大学,101,211,University Of Hainan
+海南大学,101,211,HAINU
+河南大学,85,双一流,河大
+河南大学,85,双一流,He'nan University
+河南大学,85,双一流,HENU
+哈尔滨工程大学,65,211,哈工程
+哈尔滨工程大学,65,211,Harbin Engineering University
+哈尔滨工程大学,65,211,HEU
+东北农业大学,98,211,东北农大
+东北农业大学,98,211,Northeast Agricultural University
+东北农业大学,98,211,NEAU
+东北林业大学,93,211,东北林大
+东北林业大学,93,211,Northeast Forestry University
+东北林业大学,93,211,NEFU
+中国地质大学,80,211,地大
+中国地质大学,80,211,China University Of Geosciences
+中国地质大学,80,211,CUG
+武汉理工大学,40,211,武汉理工
+武汉理工大学,40,211,Wuhan University of Technology
+武汉理工大学,40,211,WUT
+华中农业大学,52,211,华中农大
+华中农业大学,52,211,华农
+华中农业大学,52,211,Central China Agricultural University
+华中农业大学,52,211,HZAU
+华中师范大学,58,211,华中师大
+华中师范大学,58,211,华大
+华中师范大学,58,211,Central China Normal University
+华中师范大学,58,211,CCNU
+中南财经政法大学,105,211,中南大
+中南财经政法大学,105,211,Zhongnan University Of Economics & Law
+中南财经政法大学,105,211,ZUEL
+湖南师范大学,68,211,湖南师大
+湖南师范大学,68,211,Hunan Normal University
+湖南师范大学,68,211,HUNNU
+延边大学,130,211,延大
+延边大学,130,211,Yanbian University
+延边大学,130,211,YBU
+东北师范大学,69,211,东北师大
+东北师范大学,69,211,Northeast Normal University
+东北师范大学,69,211,NENU
+苏州大学,35,211,苏大
+苏州大学,35,211,Soochow University
+苏州大学,35,211,SUDA
+南京航空航天大学,33,211,南航
+南京航空航天大学,33,211,Nanjing Aero-Space University
+南京航空航天大学,33,211,NUAA
+南京理工大学,42,211,南理工
+南京理工大学,42,211,Institutes Of Technology Of Nanjing
+南京理工大学,42,211,NJUST
+中国矿业大学,61,211,中国矿大
+中国矿业大学,61,211,China Mining University
+中国矿业大学,61,211,CUMT
+南京邮电大学,,双一流,南邮
+南京邮电大学,,双一流,Nanjing University of Posts and Telecommunications
+南京邮电大学,,双一流,NJUPT
+河海大学,54,211,河海
+河海大学,54,211,River Sea University
+河海大学,54,211,HHU
+江南大学,57,211,江南大
+江南大学,57,211,Jiangnan University
+江南大学,57,211,Jiangnan
+南京林业大学,,双一流,南林
+南京林业大学,,双一流,Nanjing Forestry University
+南京林业大学,,双一流,NJFU
+南京信息工程大学,91,双一流,南信大
+南京信息工程大学,91,双一流,Nanjing University of Information Science and Technology
+南京信息工程大学,91,双一流,NUIST
+南京农业大学,53,211,南农
+南京农业大学,53,211,南农大
+南京农业大学,53,211,南京农大
+南京农业大学,53,211,Agricultural University Of Nanjing
+南京农业大学,53,211,NJAU
+南京农业大学,53,211,NAU
+南京中医药大学,,双一流,南中医
+南京中医药大学,,双一流,Nanjing University Of Chinese Medicine
+南京中医药大学,,双一流,NJUCM
+中国药科大学,119,211,中国药大
+中国药科大学,119,211,China Medicine University
+中国药科大学,119,211,CPU
+南京师范大学,56,211,南京师大
+南京师范大学,56,211,南师大
+南京师范大学,56,211,南师
+南京师范大学,56,211,Nanjing Normal University
+南京师范大学,56,211,NNU
+南昌大学,47,211,昌大
+南昌大学,47,211,University Of Nanchang、Nanchang University
+南昌大学,47,211,NCU
+辽宁大学,118,211,辽大
+辽宁大学,118,211,Liaoning University
+辽宁大学,118,211,LNU
+大连海事大学,111,211,大连海大
+大连海事大学,111,211,Maritime Affairs University Of Dalian
+大连海事大学,111,211,DMU
+内蒙古大学,116,211,内大
+内蒙古大学,116,211,University Of The Inner Mongol、Inner Mongolia University
+内蒙古大学,116,211,IMU
+宁夏大学,125,211,Ningxia University
+宁夏大学,125,211,NXU
+青海大学,129,211,清大
+青海大学,129,211,Qinghai University
+青海大学,129,211,QHU
+中国石油大学,77,双一流,中石大
+中国石油大学,77,双一流,China University Of Petroleum Beijing
+中国石油大学,77,双一流,UPC
+太原理工大学,84,211,太原理工
+太原理工大学,84,211,Institutes Of Technology Of Taiyuan、Taiyuan University of Technology
+太原理工大学,84,211,TYUT
+西北大学,59,211,西北大
+西北大学,59,211,Northwest University
+西北大学,59,211,NWU
+西安电子科技大学,50,211,西电
+西安电子科技大学,50,211,Xidian University
+西安电子科技大学,50,211,XDU
+长安大学,83,211,长大
+长安大学,83,211,Chang`an University
+长安大学,83,211,CHU
+陕西师范大学,67,211,陕西师大
+陕西师范大学,67,211,陕师大
+陕西师范大学,67,211,Shaanxi Normal University
+陕西师范大学,67,211,SNNU
+第四军医大学,,211,空军军医大学、四医大
+第四军医大学,,211,Air Force Medical University
+第四军医大学,,211,FMMU
+华东理工大学,45,211,华理
+华东理工大学,45,211,East China University Of Science
+华东理工大学,45,211,ECUST
+东华大学,74,211,东华
+东华大学,74,211,Donghua University
+东华大学,74,211,DHU
+上海海洋大学,,双一流,上海海大
+上海海洋大学,,双一流,Shanghai Ocean University
+上海海洋大学,,双一流,SHOU
+上海中医药大学,,211,上中医
+上海中医药大学,,211,Shanghai University of Traditional Chinese Medicine
+上海中医药大学,,211,SHUTCM
+上海外国语大学,123,211,上外
+上海外国语大学,123,211,Shanghai International Studies University
+上海外国语大学,123,211,SISU
+上海财经大学,95,211,上海财大
+上海财经大学,95,211,上财
+上海财经大学,95,211,Shanghai University Of Finance
+上海财经大学,95,211,SUFE
+上海体育学院,,双一流,Shanghai University Of Sport
+上海音乐学院,,双一流,上音
+上海音乐学院,,双一流,Shanghai Conservatory Of Music
+上海音乐学院,,双一流,SHCM
+上海大学,43,211,上大
+上海大学,43,211,Shanghai University
+第二军医大学,,211,海军军医大学
+第二军医大学,,211,Naval Medical University
+西南交通大学,36,211,西南交大
+西南交通大学,36,211,Southwest Jiaotong University
+西南交通大学,36,211,SWJTU
+西南石油大学,,双一流,西南石大
+西南石油大学,,双一流,Southwest Petroleum University
+西南石油大学,,双一流,SWPU
+成都理工大学,,双一流,成都理工
+成都理工大学,,双一流,Chengdu University Of Technology
+成都理工大学,,双一流,CDUT
+四川农业大学,113,211,川农
+四川农业大学,113,211,川农大
+四川农业大学,113,211,Sichuan Agricultural University
+四川农业大学,113,211,SICAU
+成都中医药大学,,双一流,成中医
+成都中医药大学,,双一流,Chengdu University of TCM
+成都中医药大学,,双一流,CDUTCM
+西南财经大学,97,211,西南财大
+西南财经大学,97,211,西财
+西南财经大学,97,211,Southwestern University Of Finance And Economics
+西南财经大学,97,211,SWUFE
+天津工业大学,,双一流,天工大
+天津工业大学,,双一流,Tianjin University of Technology
+天津工业大学,,双一流,TGU
+天津医科大学,107,211,天津医大
+天津医科大学,107,211,Medical University Of Tianjin
+天津医科大学,107,211,TMU
+天津中医药大学,,双一流,天中
+天津中医药大学,,双一流,Tianjin University of Traditional Chinese Medicine
+天津中医药大学,,双一流,TUTCM
+华北电力大学,73,211,华电
+华北电力大学,73,211,North China Electric Power University
+华北电力大学,73,211,NCEPU
+河北工业大学,92,211,河工大
+河北工业大学,92,211,Hebei University of Technology
+河北工业大学,92,211,HEBUT
+西藏大学,135,211,藏大
+西藏大学,135,211,Tibet University
+西藏大学,135,211,TU
+石河子大学,117,211,石大
+石河子大学,117,211,Shihezi University
+中国美术学院,,双一流,中国美院
+中国美术学院,,双一流,国美
+中国美术学院,,双一流,China Academy of Art
+中国美术学院,,双一流,CAA
+宁波大学,70,双一流,Ningbo University
+宁波大学,70,双一流,NBU
+西南大学,46,985,西南大
+西南大学,46,985,Southwest University
+西南大学,46,985,SWU
+安徽大学,81,211,安大
+安徽大学,81,211,University Of Anhui
+安徽大学,81,211,AHU
+合肥工业大学,51,211,合肥工大
+合肥工业大学,51,211,合工大
+合肥工业大学,51,211,HeFei University of Technology
+合肥工业大学,51,211,HFUT
+麻省理工学院,1,海外名校,Massachusetts Institute of Technology
+麻省理工学院,1,海外名校,MIT
+麻省理工学院,1,海外名校,麻省
+麻省理工学院,1,海外名校,马萨诸塞理工学院
+牛津大学,2,海外名校,University of Oxford
+牛津大学,2,海外名校,Oxford
+牛津大学,2,海外名校,Oxon
+牛津大学,2,海外名校,牛津
+斯坦福大学,3,海外名校,Stanford University
+斯坦福大学,3,海外名校,Leland Stanford Junior University
+斯坦福大学,3,海外名校,斯坦福
+斯坦福大学,3,海外名校,Stanford
+剑桥大学,3,海外名校,University of Cambridge
+剑桥大学,3,海外名校,剑桥
+哈佛大学,5,海外名校,Harvard University
+哈佛大学,5,海外名校,哈佛
+加州理工学院,6,海外名校,California Institute of Technology
+加州理工学院,6,海外名校,Caltech
+加州理工学院,6,海外名校,加州理工
+帝国理工学院,,海外名校,Imperial College London
+帝国理工学院,,海外名校,帝国理工医学院
+帝国理工学院,,海外名校,Imperial College of Science
+帝国理工学院,,海外名校,Technology and Medicine
+帝国理工学院,,海外名校,帝国学院
+帝国理工学院,,海外名校,帝国理工
+帝国理工学院,,海外名校,Imperial
+帝国理工学院,,海外名校,IC
+苏黎世联邦理工学院,8,海外名校,苏黎世理工
+苏黎世联邦理工学院,8,海外名校,ETH
+苏黎世联邦理工学院,8,海外名校,Swiss Federal Institute of Technology in Zurich
+伦敦大学学院,8,海外名校,UCL
+伦敦大学学院,8,海外名校,University College London
+芝加哥大学,10,海外名校,University of Chicago
+芝加哥大学,10,海外名校,芝大
+芝加哥大学,10,海外名校,UChicago
+新加坡国立大学,11,海外名校,National University of Singapore
+新加坡国立大学,11,海外名校,NUS
+南洋理工大学,12,海外名校,Nanyang Technological University
+南洋理工大学,12,海外名校,Singapore
+南洋理工大学,12,海外名校,NTU
+宾夕法尼亚大学,13,海外名校,University of Pennsylvania
+宾夕法尼亚大学,13,海外名校,UPenn
+宾夕法尼亚大学,13,海外名校,宾大
+洛桑联邦理工学院,14,海外名校,EPFL
+洛桑联邦理工学院,14,海外名校,Swiss federal Institute of Technology in Lausanne
+耶鲁大学,14,海外名校,Yale University
+耶鲁大学,14,海外名校,耶鲁
+耶鲁大学,14,海外名校,Yale
+爱丁堡大学,16,海外名校,The University of Edinburgh
+爱丁堡大学,16,海外名校,Edin
+爱丁堡大学,16,海外名校,爱大
+哥伦比亚大学,19,海外名校,Columbia University
+哥伦比亚大学,19,海外名校,哥大
+哥伦比亚大学,19,海外名校,Columbia University in the City of New York
+普林斯顿大学,20,海外名校,Princeton University
+普林斯顿大学,20,海外名校,普林斯顿
+康奈尔大学,21,海外名校,Cornell University
+康奈尔大学,21,海外名校,Cornell
+康奈尔大学,21,海外名校,康奈尔
+香港大学,22,海外名校,The University of Hong Kong
+香港大学,22,海外名校,HKU
+香港大学,22,海外名校,港大
+东京大学,23,海外名校,The University of Tokyo
+东京大学,23,海外名校,東京大学
+东京大学,23,海外名校,UTokyo
+密歇根大学安娜堡分校,,海外名校,University of Michigan-Ann Arbor
+密歇根大学安娜堡分校,,海外名校,UMich
+约翰霍普金斯大学,25,海外名校,Johns Hopkins University
+约翰霍普金斯大学,25,海外名校,JHU
+约翰霍普金斯大学,25,海外名校,Hopkins
+约翰霍普金斯大学,25,海外名校,霍普金斯大学
+多伦多大学,26,海外名校,University of Toronto
+多伦多大学,26,海外名校,UofT
+多伦多大学,26,海外名校,UToronto
+麦吉尔大学,27,海外名校,McGill University
+麦吉尔大学,27,海外名校,McGill
+澳洲国立大学,,海外名校,The Australian National University
+澳洲国立大学,,海外名校,ANU
+曼彻斯特大学,27,海外名校,The University of Manchester
+曼彻斯特大学,27,海外名校,UoM
+京都大学,33,海外名校,Kyoto University
+香港科技大学,34,海外名校,The Hong Kong University of Science and Technology
+香港科技大学,34,海外名校,HKUST
+伦敦大学国王学院,,海外名校,King's College London
+伦敦大学国王学院,,海外名校,King's
+伦敦大学国王学院,,海外名校,KCL
+首尔国立大学,36,海外名校,Seoul National University
+首尔国立大学,36,海外名校,SNU
+墨尔本大学,37,海外名校,The University of Melbourne
+墨尔本大学,37,海外名校,UniMelb
+墨尔本大学,37,海外名校,Melb Uni
+悉尼大学,38,海外名校,The University of Sydney
+悉尼大学,38,海外名校,USYD
+悉尼大学,38,海外名校,Sydney U
+香港中文大学,39,海外名校,The Chinese University of Hong Kong
+香港中文大学,39,海外名校,CUHK
+韩国高等科技学院,,海外名校,Korea Advanced Institute of Science And Technology
+韩国高等科技学院,,海外名校,KAIST
+纽约大学,42,海外名校,New York University
+纽约大学,42,海外名校,NYU
+新南威尔士大学,43,海外名校,The University of New South Wales
+新南威尔士大学,43,海外名校,UNSW
+巴黎第九大学,,海外名校,University Paris Dauphine
+英属哥伦比亚大学,,海外名校,University of British Columbia
+英属哥伦比亚大学,,海外名校,UBC
+昆士兰大学,47,海外名校,The University of Queensland
+昆士兰大学,47,海外名校,UQ
+巴黎理工学院,49,海外名校,Institut Polytechnique de Paris
+巴黎理工学院,49,海外名校,IP-Paris
+伦敦政治经济学院,49,海外名校,The London School of Economics and Political Science
+伦敦政治经济学院,49,海外名校,LSE
+伦敦政治经济学院,49,海外名校,London School of Economics
+慕尼黑工业大学,,海外名校,Technical University of Munich
+杜克大学,52,海外名校,Duke University
+卡耐基梅隆大学,,海外名校,Carnegie Mellon University
+卡耐基梅隆大学,,海外名校,TU München
+卡耐基梅隆大学,,海外名校,TUM
+香港城市大学,53,海外名校,City University of Hong Kong
+香港城市大学,53,海外名校,CityU
+阿姆斯特丹大学,55,海外名校,University of Amsterdam
+阿姆斯特丹大学,55,海外名校,UvA
+东京工业大学,56,海外名校,Tokyo Institute of Technology
+东京工业大学,56,海外名校,Tokyo Tech
+代尔夫特理工大学,57,海外名校,Delft University of Technology
+莫纳什大学,,海外名校,Monash University
+莫纳什大学,,海外名校,Monash
+莫纳什大学,,海外名校,蒙纳士
+莫纳什大学,,海外名校,莫纳什
+莫纳什大学,,海外名校,莫大
+布朗大学,60,海外名校,Brown University
+布朗大学,60,海外名校,Brown
+布朗大学,60,海外名校,布朗
+华威大学,61,海外名校,The University of Warwick
+华威大学,61,海外名校,华威
+华威大学,61,海外名校,Warwick
+华威大学,61,海外名校,UoW
+布里斯托大学,62,海外名校,University of Bristol
+布里斯托大学,62,海外名校,布大
+海德堡大学,63,海外名校,Heidelberg University
+海德堡大学,63,海外名校,海德堡大学
+慕尼黑大学,64,海外名校,LMU
+慕尼黑大学,64,海外名校,University of Munich
+马来亚大学,65,海外名校,University of Malaya
+马来亚大学,65,海外名校,马大
+香港理工大学,66,海外名校,The Hong Kong Polytechnic University
+香港理工大学,66,海外名校,PolyU
+香港理工大学,66,海外名校,理大
+香港理工大学,66,海外名校,港理工
+德克萨斯大学奥斯汀分校,,海外名校,University of Texas at Austin
+德克萨斯大学奥斯汀分校,,海外名校,得州大学
+德克萨斯大学奥斯汀分校,,海外名校,UT-Austin
+国立台湾大学,68,海外名校,National Taiwan University
+国立台湾大学,68,海外名校,台大
+布宜诺斯艾利斯大学,69,海外名校,Universidad de Buenos Aires
+布宜诺斯艾利斯大学,69,海外名校,UBA
+鲁汶大学（荷语）,,海外名校,University of Leuven
+鲁汶大学（荷语）,,海外名校,KU Leuven
+苏黎世大学,70,海外名校,University of Zurich
+苏黎世大学,70,海外名校,UZH
+索邦大学,72,海外名校,Sorbonne University
+索邦大学,72,海外名校,索邦
+索邦大学,72,海外名校,Sorbonne
+格拉斯哥大学,73,海外名校,University of Glasgow
+格拉斯哥大学,73,海外名校,格大
+高丽大学,,海外名校,Korea University
+高丽大学,,海外名校,高丽大
+大阪大学,75,海外名校,Osaka University
+大阪大学,75,海外名校,阪大
+威斯康辛大学麦迪逊分校,,海外名校,University of Wisconsin-Madison
+威斯康辛大学麦迪逊分校,,海外名校,UW-Madison
+南安普敦大学,,海外名校,University of Southampton
+南安普敦大学,,海外名校,Soton
+莫斯科国立大学,,海外名校,Lomonosov Moscow State University
+莫斯科国立大学,,海外名校,莫斯科大学
+哥本哈根大学,79,海外名校,University of Copenhagen
+哥本哈根大学,79,海外名校,UCPH
+延世大学,79,海外名校,Yonsei University
+延世大学,79,海外名校,연세대학교
+浦项科技大学,81,海外名校,Pohang University of Science And Technology
+浦项科技大学,81,海外名校,POSTECH
+杜伦大学,,海外名校,Durham University
+杜伦大学,,海外名校,Durham
+伊利诺伊大学厄巴纳-香槟分校,,海外名校,University of Illinois at Urbana-Champaign
+伊利诺伊大学厄巴纳-香槟分校,,海外名校,UIUC
+奥克兰大学,85,海外名校,The University of Auckland
+奥克兰大学,85,海外名校,UoA
+华盛顿大学,85,海外名校,University of Washington
+华盛顿大学,85,海外名校,UWashington
+华盛顿大学,85,海外名校,UW
+巴黎萨克雷大学,86,海外名校,Université Paris-Saclay
+巴黎萨克雷大学,86,海外名校,UPSaclay
+巴黎萨克雷大学,86,海外名校,UPS
+隆德大学,87,海外名校,Lund University
+佐治亚理工学院,88,海外名校,Georgia Institute of Technology
+佐治亚理工学院,88,海外名校,Georgia Tech
+佐治亚理工学院,88,海外名校,Gatech
+佐治亚理工学院,88,海外名校,GT
+瑞典皇家理工学院,89,海外名校,KTH Royal Institute of Technology
+瑞典皇家理工学院,89,海外名校,皇家理工学院
+瑞典皇家理工学院,89,海外名校,KTH
+伯明翰大学,90,海外名校,University of Birmingham
+伯明翰大学,90,海外名校,伯大
+伯明翰大学,90,海外名校,UoB
+圣安德鲁斯大学,91,海外名校,University of St Andrews
+圣安德鲁斯大学,91,海外名校,St Andrews
+圣安德鲁斯大学,91,海外名校,St And
+圣安德鲁斯大学,91,海外名校,圣安
+利兹大学,92,海外名校,University of Leeds
+西澳大学,93,海外名校,The University of Western Australia
+西澳大学,93,海外名校,UWA
+莱斯大学,94,海外名校,Rice University
+莱斯大学,94,海外名校,Rice
+谢菲尔德大学,95,海外名校,The University of Sheffield
+谢菲尔德大学,95,海外名校,谢菲
+谢菲尔德大学,95,海外名校,谢大
+谢菲尔德大学,95,海外名校,TUoS
+宾州州立大学公园分校,,海外名校,Pennsylvania State University
+宾州州立大学公园分校,,海外名校,University Park
+成均馆大学,97,海外名校,Sungkyunkwan University(SKKU)
+成均馆大学,97,海外名校,成均馆大
+成均馆大学,97,海外名校,SKKU
+丹麦技术大学,99,海外名校,Technical University of Denmark
+丹麦技术大学,99,海外名校,DTU
+北卡罗来纳大学教堂山分校,100,海外名校,University of North Carolina at Chapel Hill
+北卡罗来纳大学教堂山分校,100,海外名校,UNC
+都柏林三一学院,,海外名校,Trinity College Dublin
+都柏林三一学院,,海外名校,The University of Dublin
+都柏林三一学院,,海外名校,TCD
+奥斯陆大学,102,海外名校,University of Oslo
+奥斯陆大学,102,海外名校,UiO
+奥斯陆大学,102,海外名校,奥大
+诺丁汉大学,103,海外名校,University of Nottingham
+诺丁汉大学,103,海外名校,UoN
+诺丁汉大学,103,海外名校,诺大
+赫尔辛基大学,104,海外名校,University of Helsinki
+赫尔辛基大学,104,海外名校,UH
+墨西哥国立自治大学,105,海外名校,Universidad Nacional Autónoma de México
+墨西哥国立自治大学,105,海外名校,墨国大
+墨西哥国立自治大学,105,海外名校,UNAM
+日内瓦大学,105,海外名校,University of Geneva
+日内瓦大学,105,海外名校,UNIGE
+圣路易斯华盛顿大学,107,海外名校,Washington University in St. Louis
+圣路易斯华盛顿大学,107,海外名校,WashU
+圣路易斯华盛顿大学,107,海外名校,WUSTL
+阿德雷德大学,,海外名校,The University of Adelaide
+阿德雷德大学,,海外名校,阿大
+阿卜杜勒阿齐兹国王大学,109,海外名校,King Abdulaziz University (KAU)
+乌得勒支大学,,��外名校,Utrecht University
+乌得勒支大学,,海外名校,UU
+蒙特利尔大学,111,海外名校,Université de Montréal
+蒙特利尔大学,111,海外名校,UdeM
+蒙特利尔大学,111,海外名校,蒙大
+阿尔托大学,112,海外名校,Aalto University
+阿尔托大学,112,海外名校,Aalto
+波士顿大学,112,海外名校,Boston University
+波士顿大学,112,海外名校,BU
+莱顿大学,112,海外名校,Leiden University
+南加州大学,11,海外名校,University of Southern California
+南加州大学,11,海外名校,南加州大学
+南加州大学,11,海外名校,南加大
+南加州大学,11,海外名校,USC
+普渡大学西拉法叶分校,116,海外名校,Purdue University
+普渡大学西拉法叶分校,116,海外名校,普渡大学
+伦敦大学玛丽女王学院,,海外名校,Queen Mary University of London
+伦敦大学玛丽女王学院,,海外名校,QMUL
+伦敦大学玛丽女王学院,,海外名校,Queen Mary
+名古屋大学,118,海外名校,Nagoya University
+名古屋大学,118,海外名校,名大
+伯尔尼大学,119,海外名校,University of Bern
+伯尔尼大学,119,海外名校,UniBe
+俄亥俄州立大学,120,海外名校,The Ohio State University
+俄亥俄州立大学,120,海外名校,Ohio State
+俄亥俄州立大学,120,海外名校,OSU
+查尔姆斯理工大学,121,海外名校,Chalmers University of Technology
+查尔姆斯理工大学,121,海外名校,Chalmers
+圣保罗大学,121,海外名校,Universidade de São Paulo
+圣保罗大学,121,海外名校,USP
+圣保罗大学,121,海外名校,圣大
+瓦格宁根大学,,海外名校,Wageningen University & Research
+瓦格宁根大学,,海外名校,Wageningen UR
+乌普萨拉大学,124,海外名校,Uppsala University
+乌普萨拉大学,124,海外名校,乌大
+埃因霍温理工大学,,海外名校,Eindhoven University of Technology
+埃因霍温理工大学,,海外名校,TU/e
+埃因霍温理工大学,,海外名校,TU Eindhoven
+柏林自由大学,127,海外名校,Freie Universitaet Berlin
+柏林自由大学,127,海外名校,FU Berlin
+柏林洪堡大学,128,海外名校,Humboldt-Universität zu Berlin
+柏林洪堡大学,128,海外名校,洪堡大学
+柏林洪堡大学,128,海外名校,柏林大学
+格罗宁根大学,128,海外名校,University of Groningen
+格罗宁根大学,128,海外名校,RuG
+里昂高等师范学院,,海外名校,École Normale Supérieure de Lyon
+里昂高等师范学院,,海外名校,ENS Lyon
+兰卡斯特大学,,海外名校,Lancaster University
+兰卡斯特大学,,海外名校,兰卡
+悉尼科技大学,133,海外名校,University of Technology Sydney
+悉尼科技大学,133,海外名校,UTS
+悉尼科技大学,133,海外名校,悉尼科大
+纽卡斯尔大学,134,海外名校,Newcastle University
+纽卡斯尔大学,134,海外名校,NCL
+纽卡斯尔大学,134,海外名校,纽大
+智利天主大学,,海外名校,Pontificia Universidad Católica de Chile (UC)
+卡尔斯鲁厄理工学院,136,海外名校,KIT
+卡尔斯鲁厄理工学院,136,海外名校,Karlsruhe Institute of Technology
+九州大学,137,海外名校,Kyushu University
+九州大学,137,海外名校,九大
+巴塞尔大学,138,海外名校,University of Basel
+巴塞尔大学,138,海外名校,Unibasel
+麦克马斯特大学,140,海外名校,McMaster University
+麦克马斯特大学,140,海外名校,麦马
+根特大学,141,海外名校,Ghent University
+根特大学,141,海外名校,UGent
+米兰理工大学,142,海外名校,Politecnico di Milano
+米兰理工大学,142,海外名校,POLIMI
+米兰理工大学,142,海外名校,米理
+米兰理工大学,142,海外名校,米兰理工
+马来西亚博特拉大学,143,海外名校,Universiti Putra Malaysia (UPM)
+马来西亚博特拉大学,143,海外名校,UPM
+马来西亚国民大学,144,海外名校,Universiti Kebangsaan Malaysia (UKM)
+马来西亚国民大学,144,海外名校,UKM
+北海道大学,145,海外名校,Hokkaido University
+北海道大学,145,海外名校,北大（ほくだい）
+马来西亚理科大学,147,海外名校,Universiti Sains Malaysia (USM)
+马来西亚理科大学,147,海外名校,USM
+马来西亚理科大学,147,海外名校,槟城理大
+斯德哥尔摩大学,148,海外名校,Stockholm University
+斯德哥尔摩大学,148,海外名校,SU
+埃克塞特大学,149,海外名校,The University of Exeter
+埃克塞特大学,149,海外名校,Exon
+滑铁卢大学,149,海外名校,University of Waterloo
+滑铁卢大学,149,海外名校,Waterloo（UW）
+卡迪夫大学,151,海外名校,Cardiff University
+卡迪夫大学,151,海外名校,卡大
+维也纳大学,151,海外名校,University of Vienna
+维也纳大学,151,海外名校,univie
+约克大学（英国）,,海外名校,University of York
+约克大学（英国）,,海外名校,York
+罗切斯特大学,,海外名校,University of Rochester
+罗切斯特大学,,海外名校,U of R
+奥胡斯大学,155,海外名校,Aarhus University
+奥胡斯大学,155,海外名校,AU
+汉阳大学,156,海外名校,Hanyang University
+汉阳大学,156,海外名校,汉阳大
+密歇根州立大学,157,海外名��,Michigan State University
+密歇根州立大学,157,海外名校,MSU
+马里兰大学学院公园分校,,海外名校,University of Maryland
+马里兰大学学院公园分校,,海外名校,College Park
+马里兰大学学院公园分校,,海外名校,UMD
+马里兰大学学院公园分校,,海外名校,UMCP
+柏林工业大学,159,海外名校,Technische Universität Berlin (TU Berlin)
+柏林工业大学,159,海外名校,TUB
+柏林工业大学,159,海外名校,TU Berlin
+埃默里大学,160,海外名校,Emory University
+埃默里大学,160,海外名校,Emory
+凯斯西储大学,161,海外名校,Case Western Reserve University
+凯斯西储大学,161,海外名校,CWRU
+凯斯西储大学,161,海外名校,CASE
+蒙特雷理工学院,,海外名校,Tecnológico de Monterrey
+法赫德国王石油与矿产大学,,海外名校,King Fahd University of Petroleum & Minerals
+法赫德国王石油与矿产大学,,海外名校,KFUPM
+匹兹堡大学,163,海外名校,University of Pittsburgh
+匹兹堡大学,163,海外名校,PITT
+匹兹堡大学,163,海外名校,匹大
+亚琛工业大学,165,海外名校,RWTH Aachen University
+亚琛工业大学,165,海外名校,RWTH Aachen
+亚琛工业大学,165,海外名校,RWTH
+博洛尼亚大学,166,海外名校,Alma Mater Studiorum - University of Bologna
+博洛尼亚大学,166,海外名校,Unibo
+博洛尼亚大学,166,海外名校,博大
+巴斯大学,166,海外名校,University of Bath
+德州农工大学,168,海外名校,Texas A&M University
+德州农工大学,168,海外名校,TAMU
+巴塞罗那大学,168,海外名校,Universitat de Barcelona
+西安大略大学,,海外名校,Western University
+西安大略大学,,海外名校,韦仕敦大学
+西安大略大学,,海外名校,UWO
+罗马第一大学,,海外名校,Sapienza University of Rome
+罗马第一大学,,海外名校,罗马一大
+弗莱堡大学,,海外名校,Albert-Ludwigs-Universitaet Freiburg
+都柏林大学学院,,海外名校,University College Dublin
+都柏林大学学院,,海外名校,UCD
+都柏林大学学院,,海外名校,UC Dublin
+佛罗里达大学,173,海外名校,University of Florida
+佛罗里达大学,173,海外名校,UF
+佛罗里达大学,173,海外名校,UFL
+国立哈萨克大学,,海外名校,Al-Farabi Kazakh National University
+洛桑大学,176,海外名校,University of Lausanne
+蒂宾根大学,177,海外名校,Eberhard Karls Universität Tübingen
+印度理工学院孟买分校,,海外名校,Indian Institute of Technology Bombay
+印度理工学院孟买分校,,海外名校,IITB
+伊拉斯姆斯大学,,海外名校,Erasmus University Rotterdam
+国立清华大学,180,海外名校,National Tsing Hua University
+维也纳技术大学,,海外名校,Technische Universität Wien
+哥德堡大学,180,海外名校,University of Gothenburg
+哈里发大学,,海外名校,Khalifa University
+智利大学,183,海外名校,Universidad de Chile
+印度理工学院德里分校,,海外名校,Indian Institute of Technology Delhi
+印度理工学院德里分校,,海外名校,IITD
+印度科学学院,,海外名校,Indian Institute of Science
+明尼苏达大学双城分校,186,海外名校,University of Minnesota Twin Cities
+鲁汶大学（法语）,,海外名校,Université catholique de Louvain
+鲁汶大学（法语）,,海外名校,UCLouvain
+利物浦大学,189,海外名校,University of Liverpool
+特文特大学,,海外名校,University of Twente
+达特茅斯学院,191,海外名校,Dartmouth College
+马来西亚理工大学,,海外名校,Universiti Teknologi Malaysia
+卧龙岗大学,193,海外名校,University of Wollongong
+科廷大学,194,海外名校,Curtin University
+德累斯顿工业大学,194,海外名校,Technische Universität Dresden
+奥塔戈大学,,海外名校,University of Otago
+纽卡斯尔大学（澳洲）,,海外名校,The University of Newcastle
+纽卡斯尔大学（澳洲）,,海外名校,Australia
+纽卡斯尔大学（澳洲）,,海外名校,UON
+耶路撒冷希伯来大学,198,海外名校,The Hebrew University of Jerusalem
+卑尔根大学,199,海外名校,University of Bergen
+麦考瑞大学,200,海外名校,Macquarie University
+加州大学伯克利分校,32,海外名校,"University of California, Berkeley"
+加州大学伯克利分校,32,海外名校,UC Berkeley
+加州大学伯克利分校,32,海外名校,Cal
+加州大学圣地亚哥分校,48,海外名校,"University of California, San Diego"
+加州大学圣地亚哥分校,48,海外名校,UCSD
+加州大学圣地亚哥分校,48,海外名校,UC San Diego
+加州大学戴维斯分校,138,海外名校,"University of California, Davis"
+加州大学戴维斯分校,138,海外名校,UC Davis
+加州大学圣塔芭芭拉分校,146,海外名校,"University of California, Santa Barbara"
+加州大学圣塔芭芭拉分校,146,海外名校,UCSB
+加州大学圣塔芭芭拉分校,146,海外名校,UC Santa Barbara
+上海体育学院,,双一流,SUS
+密歇根大学-安娜堡,23,海外名校,University of Michigan - Ann Arbor
+伦敦国王学院,35,海外名校,King’s College London
+加州大学洛杉矶分校,40,海外名校,"University of California, Los Angeles"
+韩国科学技术院,41,海外名校,KAIST - Korea Advanced Institute of Science & Technology
+巴黎文理研究大学,44,海外名校,PSL University
+德克萨斯州大学奥斯汀分校,67,海外名校,The University of Texas at Austin
+威斯康星大学-麦迪逊,75,海外名校,University of Wisconsin - Madison
+宾夕法尼亚州立大学,96,海外名校,Penn State (Main campus)
+里昂高等师范学校,130,海外名校,Ecole Normale Superieure - Lyon
+智利天主教大学,135,海外名校,Pontifical Catholic University of Chile
+约克大学,494,海外名校,York University
+马里兰大学帕克分校,158,海外名校,"University of Maryland, College Park"
+蒙特雷技术学院,161,海外名校,Tecnologico de Monterrey
+天主教鲁汶大学（法语区）,188,海外名校,Université Catholique de Louvain
+澳大利亚纽卡斯尔大学,197,海外名校,"The University of Newcastle, Australia"
+庆应义塾大学,201,海外名校,Keio University
+雷丁大学,202,海外名校,University of Reading
+早稻田大学,203,海外名校,Waseda University
+哥廷根大学,204,海外名校,University of Göttingen
+阿伯丁大学,205,海外名校,University of Aberdeen
+墨尔本皇家理工大学,206,海外名校,RMIT University
+马德里自治大学,207,海外名校,Autonomous University of Madrid
+布鲁塞尔自由大学,207,海外名校,Universite libre de Bruxelles
+阿姆斯特丹自由大学,209,海外名校,Vrije Universiteit Amsterdam
+巴塞罗那自治大学,209,海外名校,Autonomous University of Barcelona
+蔚山国立科学技术研究所,212,海外名校,Ulsan National Institute of Science & Technology
+昆士兰科技大学,213,海外名校,Queensland University of Technology
+汉堡大学,214,海外名校,Universität Hamburg
+朱拉隆功大学,215,海外名校,Chulalongkorn University
+亚利桑那州立大学,216,海外名校,Arizona State University
+贝尔法斯特女王大学,216,海外名校,Queen’s University Belfast
+范德堡大学,218,海外名校,Vanderbilt University
+坎皮纳斯州立大学,219,海外名校,State University of Campinas
+拉德堡德大学,220,海外名校,Radboud University
+布鲁塞尔大学,221,海外名校,Vrije Universiteit Brussel
+圣母大学,701,海外名校,Notre Dame University-Louaize NDU
+马德里康普顿斯大学,223,海外名校,Complutense University of Madrid
+卡塔尔大学,224,海外名校,Qatar University
+波恩大学,226,海外名校,Rheinische Friedrich-Wilhelms-Universität Bonn
+开普敦大学,226,海外名校,University of Cape Town
+萨塞克斯大学,226,海外名校,University of Sussex
+弗吉尼亚大学,226,海外名校,University of Virginia
+渥太华大学,230,海外名校,University of Ottawa
+拉夫堡大学,231,海外名校,Loughborough University
+加州大学欧文分校,232,海外名校,"University of California, Irvine"
+安特卫普大学,233,海外名校,University of Antwerp
+马斯特里赫特大学,233,海外名校,Maastricht University
+卡尔加里大学,235,海外名校,University of Calgary
+安德斯大学,236,海外名校,Universidad de los Andes
+莱斯特大学,236,海外名校,University of Leicester
+维多利亚大学,601,海外名校,Victoria University
+金斯敦女王大学,240,海外名校,Queen's University at Kingston
+瑞士提契诺大学,240,海外名校,USI - Università della Svizzera italiana
+贝鲁特美国大学,242,海外名校,American University of Beirut
+帕多瓦大学,242,海外名校,Università di Padova
+圣彼得堡国立大学,242,海外名校,Saint Petersburg State University
+巴黎高科桥梁学院,245,海外名校,Ecole des Ponts ParisTech
+新西伯利亚州立大学,246,海外名校,Novosibirsk State University
+马萨诸塞大学安姆斯特分校,246,海外名校,University of Massachusetts
+乔治敦大学,248,海外名校,Georgetown University
+庞培法布拉大学,248,海外名校,Pompeu Fabra University
+文莱达鲁萨兰国大学,250,海外名校,Universiti Brunei Darussalam (UBD)
+科罗拉多大学-玻尔得,251,海外名校,University of Colorado at Boulder
+国立成功大学,252,海外名校,National Cheng Kung University
+叶史瓦大学,252,海外名校,Yeshiva University
+加查马达大学,254,海外名校,Gadjah Mada University
+印度马德拉斯技术学院,255,海外名校,Indian Institute of Technology Madras
+玛希隆大学,255,海外名校,Mahidol University
+特拉维夫大学,255,海外名校,Tel Aviv University
+哥伦比亚国立大学,258,海外名校,National University of Colombia
+爱尔兰国立大学戈尔韦分校,258,海外名校,National University of Ireland Galway
+坎特伯雷大学,258,海外名校,University of Canterbury
+科学宝,261,海外名校,Sciences Po
+巴黎大学,261,海外名校,Université de Paris
+庆熙大学,264,海外名校,Kyung Hee University
+罗格斯大学,264,海外名校,Rutgers University–New Brunswick
+纳瓦拉大学,266,海外名校,University of Navarra
+布拉格查理大学,266,海外名校,Charles University
+国立阳明大学,268,海外名校,National Yang Ming Chiao Tung University
+亚利桑那大学,268,海外名校,The University of Arizona
+达姆施塔特工业大学,269,海外名校,Technical University Darmstadt
+赫瑞瓦特大学,270,海外名校,Heriot Watt University
+达尔豪斯大学,272,海外名校,Dalhousie University
+托木斯克州立大学,272,海外名校,Tomsk State University
+萨里大学,272,海外名校,University of Surrey
+南方科技大学,275,海外名校,Southern University of Science & Technology
+塔夫茨大学,275,海外名校,Tufts University
+格拉茨科技大学,277,海外名校,Graz University of Technology
+印度理工学院坎普尔,277,海外名校,Indian Institute of Technology Kanpur
+沙特国王大学,277,海外名校,King Saud University
+印度理工学院卡拉格普尔分校,280,海外名校,Indian Institute of Technology Kharagpur
+鲍曼莫斯科国立技术大学,281,海外名校,Bauman Moscow State Technical University
+因斯布鲁克大学,281,海外名校,University of Innsbruck
+迪肯大学,283,海外名校,Deakin University
+梅西大学,284,海外名校,Massey University
+伊利诺伊大学芝加哥分校,285,海外名校,University of Illinois at Chicago
+筑波大学,285,海外名校,University of Tsukuba
+香港浸会大学,287,海外名校,Hong Kong Baptist University
+阿联酋大学,288,海外名校,United Arab Emirates University
+印度尼西亚大学,290,海外名校,University of Indonesia
+格里菲斯大学,290,海外名校,Griffith University
+莫斯科物理科学与技术学院,290,海外名校,Moscow Institute of Physics and Technology
+巴黎第一大学,290,海外名校,Panthéon-Sorbonne University – Paris 1
+白俄罗斯国立大学,295,海外名校,Belarusian State University
+波尔图大学,295,海外名校,University of Porto
+图尔库大学,295,海外名校,University of Turku
+科克大学,298,海外名校,University College Cork
+西蒙弗雷泽大学,298,海外名校,Simon Fraser University
+北卡罗来纳州立大学,300,海外名校,North Carolina State University
+塔尔图大学,300,海外名校,University of Tartu
+斯特拉斯克莱德大学,302,海外名校,University of Strathclyde
+塔斯马尼亚大学,303,海外名校,University of Tasmania
+万隆理工学院（ITB）,303,海外名校,Bandung Institute of Technology (ITB)
+光州科学技术学院,305,海外名校,Gwangju Institute of Science and Technology
+俄罗斯国立高等经济大学,305,海外名校,HSE University
+东英吉利大学,307,海外名校,University of East Anglia
+华沙大学,308,海外名校,University of Warsaw
+南丹麦大学,309,海外名校,University of Southern Denmark
+雅盖隆大学,309,海外名校,Jagiellonian University
+印第安纳大学伯明顿分校,311,海外名校,Indiana University
+科隆大学,311,海外名校,University of Cologne
+迈阿密大学,311,海外名校,University of Miami
+格勒诺布尔阿尔卑斯大学,314,海外名校,Grenoble Alpes University
+国立台湾科技大学,314,海外名校,National Taiwan University of Science and Technology
+米兰大学,316,海外名校,University of Milan
+俄罗斯人民友谊大学,317,海外名校,RUDN University
+IE大学,317,海外名校,IE University
+国立核研究大学-莫斯科工程物理学院,319,海外名校,National Research Nuclear University MEPhI
+加泰罗尼亚理工大学,319,海外名校,Polytechnic University of Catalonia
+斯威本科技大学,321,海外名校,Swinburne University of Technology
+阿根廷天主教大学,322,海外名校,Pontificia Universidad Católica Argentina
+邓迪大学,322,海外名校,University of Dundee
+澳门大学,322,海外名校,University of Macau
+埃尔朗根-纽伦堡大学,322,海外名校,Friedrich-Alexander-Universität Erlangen-Nürnberg
+奥尔堡大学,326,海外名校,Aalborg University
+南澳大利亚大学,326,海外名校,University of South Australia
+国立欧亚大学,328,海外名校,L.N. Gumilyov Eurasian National University
+林雪平大学,329,海外名校,Linkoping University
+以色列理工学院,330,海外名校,Technion - Israel Institute of Technology
+伦敦大学城市学院,330,海外名校,City University London
+伦敦大学伯贝克学院,332,海外名校,Birkbeck University London
+圣光机大学,365,海外名校,ITMO University
+国立台湾师范大学,334,海外名校,National Taiwan Normal University
+都灵理工大学,334,海外名校,Politecnico di Torino
+伦敦大学皇家霍洛威学院,334,海外名校,Royal Holloway University of London
+法兰克福大学,340,海外名校,Goethe University Frankfurt
+夏威夷大学马诺阿分校,340,海外名校,University of Hawai’i at Mānoa
+广岛大学,343,海外名校,Hiroshima University
+耶拿大学,344,海外名校,Universität Jena
+文莱大学,344,海外名校,Universiti Teknologi Brunei
+弗吉尼亚理工,346,海外名校,Virginia Polytechnic Institute and State University
+斯图加特大学,347,海外名校,Universität Stuttgart
+加州大学圣克鲁兹分校,347,海外名校,"University of California, Santa Cruz"
+喀山联邦大学,347,海外名校,Kazan Federal University
+斯特雅大学,347,海外名校,UCSI University
+马德里卡洛斯三世大学,351,海外名校,Universidad Carlos III de Madrid
+乌拉尔联邦大学,351,海外名校,Ural Federal University
+伦敦布鲁内尔大学,351,海外名校,Brunel University London
+约翰内斯开普勒大学林茨,354,海外名校,Johannes Kepler University Linz
+乔治华盛顿大学,355,海外名校,George Washington University
+里斯本大学,356,海外名校,University of Lisbon
+蒂尔堡大学,356,海外名校,Tilburg University
+国立科技大学-巴基斯坦,358,海外名校,National University of Sciences & Technology - Pakistan
+于韦斯屈莱大学,358,海外名校,University of Jyvaskyla
+犹他大学,358,海外名校,University of Utah
+拉筹伯大学,362,海外名校,La Trobe University
+莫斯科国立国际关系学院,362,海外名校,MGIMO University
+梨花女子大学,362,海外名校,Ewha Womans University
+乌尔姆大学,365,海外名校,Ulm University
+于默奥大学,365,海外名校,Umea University
+苏丹卡布斯大学,368,海外名校,Sultan Qaboos University
+挪威科技大学,369,海外名校,Norwegian University of Science and Technology
+里约热内卢联邦大学,369,海外名校,Federal University of Rio de Janeiro
+瓦伦西亚理工大学,371,海外名校,Polytechnic University of Valencia
+林肯大学,801,海外名校,University of Lincoln
+布拉格化工大学,373,海外名校,"University of Chemistry & Technology, Prague"
+康涅狄格大学,373,海外名校,University of Connecticut
+怀卡托大学,373,海外名校,University of Waikato
+奥卢大学,377,海外名校,University of Oulu
+维尔茨堡大学,378,海外名校,Julius-Maximilians-Universität Würzburg
+真纳大学,378,海外名校,Quaid I Azam University
+纽约州立大学石溪分校,378,海外名校,Stony Brook University
+谢里夫科技大学,381,海外名校,Sharif University of Technology
+东京医科齿科大学,381,海外名校,Tokyo Medical and Dental University
+美国沙迦大学,383,海外名校,American University of Sharjah
+肯特大学,383,海外名校,University of Kent
+神户大学,386,海外名校,Kobe University
+堪萨斯大学,387,海外名校,University of Kansas
+纽约州立大学布法罗分校,388,海外名校,University at Buffalo
+比萨大学,388,海外名校,University of Pisa
+圣拉斐尔生命健康大学,390,海外名校,Università Vita-Salute San Raffaele
+伦敦大学亚非学院,391,海外名校,SOAS University of London
+巴勒莫大学,801,海外名校,University of Palermo
+彼得大帝圣彼得堡理工大学,393,海外名校,Peter the Great St Petersburg Polytechnic University
+波鸿大学,393,海外名校,University of Bochum
+国立研究托木斯克理工大学,395,海外名校,National Research Tomsk Polytechnic University
+秘鲁天主教大学,395,海外名校,Pontifical Catholic University of Peru
+印度理工学院古瓦哈提,395,海外名校,Indian Institute of Technology Guwahati
+巴基斯坦工程与应用科学研究所,398,海外名校,Pakistan Institute of Engineering and Applied Sciences (PIEAS)
+菲律宾大学,399,海外名校,University of the Philippines
+南方大学,400,海外名校,Universidad Austral
+印度理工学院鲁尔基校区,400,海外名校,Indian Institute of Technology Roorkee
+维尔纽斯大学,400,海外名校,Vilnius University
+加州大学河滨分校,403,海外名校,"University of California, Riverside"
+布拉格捷克技术大学,403,海外名校,Czech Technical University in Prague
+弗林德斯大学,407,海外名校,Flinders University
+台北医科大学,407,海外名校,Taipei Medical University
+明斯特大学,411,海外名校,University of Muenster
+国立中山大学,412,海外名校,National Sun Yat-Sen University
+哈韦里亚纳大学,412,海外名校,Pontificia Universidad Javeriana
+拉彭兰塔理工大学,414,海外名校,Lappeenranta University of Technology
+拉瓦尔大学,414,海外名校,Université Laval
+国油大学,414,海外名校,Universiti Teknologi Petronas
+科罗拉多大学-丹佛,414,海外名校,University of Colorado at Denver
+坦佩雷大学,414,海外名校,University of Tampere
+中央大学,414,海外名校,Chung-Ang University
+斯特拉斯堡大学,421,海外名校,Université de Strasbourg
+雅典国立技术大学,421,海外名校,National Technical University of Athens
+曼海姆大学,423,海外名校,Universität Mannheim
+那不勒斯菲里德里克第二大学,424,海外名校,University of Naples - Federico II
+威特沃特斯兰德大学,424,海外名校,University of the Witwatersrand
+詹姆斯库克大学,424,海外名校,James Cook University
+约翰内斯古腾堡美因茨大学,427,海外名校,Johannes Gutenberg University of Mainz
+华盛顿州立大学,427,海外名校,Washington State University
+维克森林大学,429,海外名校,Wake Forest University
+牛津布鲁克斯大学,429,海外名校,Oxford Brookes University
+新里斯本大学,431,海外名校,NOVA University Lisbon
+伦斯勒理工学院,431,海外名校,Rensselaer Polytechnic Institute
+科罗拉多州立大学,431,海外名校,Colorado State University
+圣保罗联邦大学,434,海外名校,Federal University of Sao Paulo
+约翰内斯堡大学,434,海外名校,University of Johannesburg
+堪培拉大学,436,海外名校,University of Canberra
+杜兰大学,436,海外名校,Tulane University
+埃塞克斯大学,439,海外名校,University of Essex
+塞浦路斯大学,440,海外名校,University of Cyprus
+特伦托大学,440,海外名校,University of Trento
+特罗姆瑟大学挪威北极大学,440,海外名校,University of Tromsø The Arctic University of Norway
+斯旺西大学,440,海外名校,Swansea University
+伊利诺伊理工学院,444,海外名校,Illinois Institute of Technology
+韩国外国语大学,445,海外名校,HUFS - Hankuk (Korea) University of Foreign Studies
+开罗美国大学,445,海外名校,The American University in Cairo
+萨尔大学,447,海外名校,Saarland University
+乌姆古拉大学,447,海外名校,Umm Al-Qura University
+莱比锡大学,447,海外名校,Universität  Leipzig
+米兰 - 比可卡大学,450,海外名校,University of Milan-Bicocca
+佛罗伦萨大学,451,海外名校,University of Florence
+奥克兰理工大学,451,海外名校,Auckland University of Technology
+邦德大学,451,海外名校,Bond University
+科英布拉大学,455,海外名校,University of Coimbra
+爱荷华大学,455,海外名校,University of Iowa
+布兰迪斯大学,455,海外名校,Brandeis University
+萨斯喀彻温大学,458,海外名校,University of Saskatchewan
+马德里理工大学,459,海外名校,Universidad Politécnica de Madrid
+圣加仑大学,459,海外名校,University of St Gallen
+马丁·路德·哈勒-威登堡大学,461,海外名校,Martin Luther University of Halle-Wittenberg
+科罗拉多矿业大学,461,海外名校,Colorado School of Mines
+远东联邦大学,461,海外名校,Far Eastern Federal University
+伦敦大学金史密斯学院,461,海外名校,Goldsmiths University London
+爱兰加大学,465,海外名校,Airlangga University
+阿米尔卡比尔理工大学,465,海外名校,Amirkabir University of Technology
+贝尔格拉诺大学,465,海外名校,Universidad de Belgrano
+国立台北科技大学,469,海外名校,National Taipei University of Technology
+乌拉圭蒙得维的亚大学,469,海外名校,Universidad de Montevideo (UM)
+乌拉圭大学,471,海外名校,Universidad ORT Uruguay
+斯特灵大学,471,海外名校,University of Stirling
+内盖夫本古里安大学,471,海外名校,Ben-Gurion University of the Negev
+哈瑟尔特大学,471,海外名校,Hasselt University
+佛罗里达州立大学,475,海外名校,Florida State University
+密苏里大学,476,海外名校,Mizzou - University of Missouri
+德克萨斯大学-达拉斯,477,海外名校,The University of Texas at Dallas
+巴伊兰大学,477,海外名校,Bar-Ilan University
+千叶大学,477,海外名校,Chiba University
+长庚大学,480,海外名校,Chang Gung University
+列日大学,480,海外名校,University of Liege
+南哈萨克斯坦大学,482,海外名校,Auezov South Kazakhstan University (SKU)
+斯泰伦博斯大学,482,海外名校,Stellenbosch University
+韩国天主教大学,482,海外名校,The Catholic University of Korea
+都灵大学,485,海外名校,University of Turin
+阿斯顿大学,485,海外名校,Aston University
+俄罗斯国立科技大学,487,海外名校,The National University of Science and Technology MISIS
+横滨市立大学,487,海外名校,Yokohama City University
+智利圣地亚哥大学,487,海外名校,Universidad de Santiago de Chile
+都柏林城市大学,490,海外名校,Dublin City University
+贾森·利比希大学,490,海外名校,Justus Liebig University Giessen
+圣保罗州立大学,492,海外名校,UNESP
+格拉纳达大学,492,海外名校,University of Granada
+罗马第二大学,494,海外名校,University of Roma - Tor Vergata
+西悉尼大学,494,海外名校,University of Western Sydney
+波士顿学院,494,海外名校,Boston College
+基尔大学,751,海外名校,Keele University
+东国大学,494,海外名校,Dongguk University
+爱荷华州立大学,494,海外名校,Iowa State University
+西江大学,494,海外名校,Sogang University
+华沙工业大学,501,海外名校,Warsaw University of Technology
+艾克斯 - 马赛大学,501,海外名校,Aix-Marseille University
+哈瓦那大学,501,海外名校,Universidad de La Habana
+康斯坦茨大学,501,海外名校,Universität Konstanz
+波尔多大学,501,海外名校,Universite de Bordeaux
+德里大学,501,海外名校,University of Delhi
+克拉根福大学,501,海外名校,University of Klagenfurt
+利默里克大学,501,海外名校,University of Limerick
+萨特巴耶夫大学,501,海外名校,Satbayev University
+密苏里科技大学,511,海外名校,Missouri University of Science & Technology
+新加坡管理大学,511,海外名校,Singapore Management University
+茂物农业大学,511,海外名校,IPB University
+科奇大学,511,海外名校,Koç University
+哈尔科夫大学,511,海外名校,V. N. Karazin Kharkiv National University
+韦恩州立大学,511,海外名校,Wayne State University
+阿尔卡拉大学,511,海外名校,Universidad de Alcala
+马来西亚北方大学,511,海外名校,Universiti Utara Malaysia
+巴拉曼德大学,511,海外名校,University of Balamand
+中央大学“阿伯勒玛塔”德拉斯别墅,521,海外名校,"Universidad Central ""Marta Abreu"" de Las Villas"
+拜罗伊特大学,521,海外名校,University of Bayreuth
+东芬兰大学,521,海外名校,University of Eastern Finland
+德黑兰大学,521,海外名校,University of Tehran
+康考迪亚大学,521,海外名校,Concordia University
+阿卜杜勒拉曼费萨尔大学,521,海外名校,Imam Abdulrahman Bin Faisal University
+蒙彼利埃大学,521,海外名校,Montpellier University
+国立中央大学,521,海外名校,National Central University
+理海大学,531,海外名校,Lehigh University
+长崎大学,531,海外名校,Nagasaki University
+国家研究萨拉托夫州立大学,531,海外名校,National Research Saratov State University
+俄勒冈州立大学,531,海外名校,Oregon State University
+贝鲁特圣约瑟夫大学,531,海外名校,Saint Joseph University of Beirut (USJ)
+南方联邦大学,531,海外名校,Southern Federal University
+韩国亚洲大学,531,海外名校,Ajou University
+一桥大学,531,海外名校,Hitotsubashi University
+里昂国家科学研究院,531,海外名校,Institut National des Sciences Appliquées de Lyon (INSA)
+哥斯达黎加大学,531,海外名校,Universidad de Costa Rica
+萨拉戈萨大学,531,海外名校,Universidad de Zaragoza
+特拉华大学,531,海外名校,University of Delaware
+蔚山大学,541,海外名校,University of Ulsan
+阿伯里斯特威斯大学,541,海外名校,Aberystwyth University
+伊朗科技大学,541,海外名校,Iran University of Science & Technology
+庆北国立大学,541,海外名校,Kyungpook National University
+新泻大学,541,海外名校,Niigata University
+萨班哲大学,541,海外名校,Sabanci University
+佐治亚大学,541,海外名校,The University of Georgia
+田纳西大学,541,海外名校,"The University of Tennessee, Knoxville"
+马萨里克大学,551,海外名校,Masaryk University
+中东技术大学,551,海外名校,Middle East Technical University
+图卢兹第三大学,551,海外名校,Paul Sabatier University (Toulouse 3)
+阿拜哈萨克斯坦国立师范大学,551,海外名校,Abai Kazakh National Pedagogical University
+塞萨洛尼基亚里士多德大学,551,海外名校,Aristotle University of Thessaloniki
+哈萨克国立农业大学,551,海外名校,Kazakh National Agrarian University KazNAU
+塞维利亚大学,551,海外名校,Universidad de Sevilla
+泛美大学,551,海外名校,Universidad Panamericana (UP)
+米兰圣心天主教大学,551,海外名校,Università Cattolica del Sacro Cuore
+塞格德大学,551,海外名校,University of Szeged
+帕维亚大学研究学院,561,海外名校,Università degli Studi di Pavia
+弗里堡大学,561,海外名校,Université de Fribourg
+阿尔泰州立大学,561,海外名校,Altai State University
+哈林大学,561,海外名校,Hallym University
+仁荷大学,561,海外名校,Inha University
+贾瓦哈拉尔尼赫鲁大学,561,海外名校,Jawaharlal Nehru University
+布伦瑞克工业大学,561,海外名校,Technical University of Braunschweig
+新学院大学,561,海外名校,The New School
+大阪市立大学,571,海外名校,Osaka City University
+开罗大学,571,海外名校,Cairo University
+卡斯里克圣灵大学,571,海外名校,Holy Spirit University of Kaslik
+伊万第比利斯国立大学,571,海外名校,Ivane Javakhishvili Tbilisi State University
+全北国立大学,571,海外名校,Jeonbuk National University
+不来梅大学,571,海外名校,Universität Bremen
+瓦伦西亚大学,571,海外名校,Universitat de Valencia
+米尼奥大学,571,海外名校,University of Minho
+内布拉斯加大学-林肯,571,海外名校,University of Nebraska - Lincoln
+拉普拉塔国立大学,581,海外名校,Universidad Nacional de La Plata (UNLP)
+圭尔夫大学,581,海外名校,University of Guelph
+南佛罗里达大学,581,海外名校,University of South Florida
+黎巴嫩美国大学,581,海外名校,Lebanese American University
+香港岭南大学,581,海外名校,"Lingnan University, Hong Kong"
+默多克大学,581,海外名校,Murdoch University
+冈山大学,581,海外名校,Okayama University
+萨马拉国立研究大学（Samara University）,581,海外名校,Samara National Research University (Samara University)
+国立政治大学,591,海外名校,National Chengchi University
+Savitribai普鲁大学,591,海外名校,Savitribai Phule Pune University
+索非亚大学,591,海外名校,Sofia University
+哥伦比亚外事大学,591,海外名校,Universidad Externado de Colombia
+雷根斯堡大学,591,海外名校,Universität Regensburg
+魁北克大学,591,海外名校,Université du Québec
+德布勒森大学,591,海外名校,University of Debrecen
+卢布尔雅那大学,591,海外名校,University of Ljubljana
+巴林应用科学大学,591,海外名校,Applied Science University - Bahrain
+比尔肯特大学,591,海外名校,Bilkent University
+印度理工学院海德拉巴,591,海外名校,Indian Institute of Technology Hyderabad
+熊本大学,591,海外名校,Kumamoto University
+阿博大学,601,海外名校,Abo Akademi University
+迪拜美国大学,601,海外名校,American University in Dubai
+马尼拉雅典大学,601,海外名校,Ateneo de Manila University
+班戈大学,601,海外名校,Bangor University
+迪拜加拿大大学,601,海外名校,Canadian University Dubai
+卡尔顿大学,601,海外名校,Carleton University
+中央昆士兰大学,601,海外名校,Central Queensland University
+清迈大学,601,海外名校,Chiang Mai University
+克拉克大学,601,海外名校,Clark University
+科米利亚斯宗座大学,601,海外名校,Comillas Pontifical University
+考文垂大学,601,海外名校,Coventry University
+岐阜大学,601,海外名校,Gifu University
+金泽大学,601,海外名校,Kanazawa University
+金斯顿大学,601,海外名校,Kingston University
+建国大学,601,海外名校,Konkuk University
+墨西哥阿那瓦克大学,601,海外名校,Universidad Anáhuac México
+康塞普西翁大学,601,海外名校,Universidad de Concepción
+布宜诺斯艾利斯省中央大学 (UNICEN,601,海外名校,Universidad Nacional del Centro de la Provincia de Buenos Aires (UNICEN)
+拉蒙鲁尔大学,601,海外名校,Universitat Ramon Llull
+霍恩海姆大学,601,海外名校,University Hohenheim
+阿威罗大学,601,海外名校,University of Aveiro
+辛辛那提大学,601,海外名校,University of Cincinnati
+热那亚大学,601,海外名校,University of Genoa
+曼尼托巴大学,601,海外名校,University of Manitoba
+普利茅斯大学,601,海外名校,University of Plymouth
+比勒陀利亚大学,601,海外名校,University of Pretoria
+沙迦大学,601,海外名校,University of Sharjah
+锡耶纳大学,601,海外名校,University of Siena
+南卡罗来纳大学,601,海外名校,University of South Carolina
+汉诺威莱布尼兹大学,601,海外名校,Leibniz University Hannover
+管理科学大学,601,海外名校,Management and Science University
+南京科技大学,601,海外名校,Nanjing University of Science & Technology
+帕拉茨基大学,601,海外名校,Palacky University Olomouc
+Pavol JozefŠafárikUniversity位于科希策,601,海外名校,Pavol Jozef Šafárik University in Košice
+釜山国立大学,601,海外名校,Pusan National University
+S.D.阿斯芬迪亚罗夫哈萨克国立医科大学,601,海外名校,S.D. Asfendiyarov Kazakh National Medical University
+世宗大学,601,海外名校,Sejong University
+深圳大学,601,海外名校,Shenzhen University
+史密斯学院,601,海外名校,Smith College
+基辅国立大学,601,海外名校,Taras Shevchenko National University Kiev
+法政大学,601,海外名校,Thammasat University
+约旦大学,601,海外名校,The University of Jordan
+东京农工大学,601,海外名校,Tokyo University of Agriculture and Technology
+巴黎第二大学,601,海外名校,University Paris 2 Panthéon-Assas
+扎耶德大学,651,海外名校,Zayed University
+安蒂奥基亚大学,651,海外名校,Universidad de Antioquia
+ICESI大学,651,海外名校,Universidad ICESI
+秘鲁秘鲁卡耶塔诺大学,651,海外名校,Universidad Peruana Cayetano Heredia
+玻利瓦尔纳天主教大学,651,海外名校,Universidad Pontificia Bolivariana
+罗斯托克大学,651,海外名校,Universität Rostock
+克劳德·伯纳德·里昂大学,651,海外名校,Université Claude Bernard Lyon 1
+玛拉工艺大学,651,海外名校,Universiti Teknologi MARA - UiTM
+克里特岛大学,651,海外名校,University of Crete
+赫尔大学,651,海外名校,University of Hull
+海德拉巴大学,651,海外名校,University of Hyderabad
+肯塔基大学,651,海外名校,University of Kentucky
+马萨诸塞大学波士顿分校,651,海外名校,University of Massachusetts Boston
+蒙斯大学,651,海外名校,University of Mons
+新不伦瑞克大学,651,海外名校,University of New Brunswick
+俄克拉荷马大学,651,海外名校,University of Oklahoma
+俄勒冈大学,651,海外名校,University of Oregon
+佩奇大学,651,海外名校,University of Pecs
+朴茨茅斯大学,651,海外名校,University of Portsmouth
+萨拉曼卡大学,651,海外名校,University of Salamanca
+的里雅斯特大学,651,海外名校,University of Trieste
+佛蒙特大学,651,海外名校,University of Vermont
+美利坚大学,651,海外名校,American University
+威廉与玛丽学院,651,海外名校,College of William and Mary
+布拉迪斯拉发的夸美纽斯大学,651,海外名校,Comenius University in Bratislava
+德雷塞尔大学,651,海外名校,Drexel University
+伊迪丝科文大学,651,海外名校,Edith Cowan University
+罗兰大学,651,海外名校,Eotvos Lorand University
+米纳斯吉拉斯州联邦大学,651,海外名校,Universidade Federal de Minas Gerais
+博尔扎诺自由大学,651,海外名校,Free University of Bozen-Bolzano
+霍华德大学,651,海外名校,Howard University
+伊曼纽尔·康德波罗的海联邦大学,651,海外名校,Immanuel Kant Baltic Federal University
+马来西亚国际伊斯兰大学,651,海外名校,International Islamic University Malaysia
+迦达浦大学,651,海外名校,Jadavpur University
+鹿儿岛大学,651,海外名校,Kagoshima University
+卡尔·弗朗岑斯大学格拉茨大学,651,海外名校,Karl-Franzens-Universitaet Graz
+哈立德国王大学,651,海外名校,King Khalid University
+拉合尔管理科学大学,651,海外名校,Lahore University of Management Sciences
+洛巴切夫斯基大学,651,海外名校,Lobachevsky University
+澳门科技大学,651,海外名校,Macau University of Science & Technology
+密歇根理工大学,651,海外名校,Michigan Technological University
+雅典大学,651,海外名校,National and Kapodistrian University of Athens
+国立中兴大学,651,海外名校,National Chung Hsing University
+国立哈尔科夫工业大学,651,海外名校,"National Technical University  ""Kharkiv Polytechnic Institute"""
+里约热内卢天主教大学,651,海外名校,Pontifícia Universidade Católica do Rio de Janeiro
+谢切诺夫大学,651,海外名校,Sechenov University
+双威大学,651,海外名校,Sunway University
+雪城大学,651,海外名校,Syracuse University
+新墨西哥大学-阿尔伯克基,651,海外名校,The University of New Mexico - Albuquerque
+德岛大学,651,海外名校,Tokushima University
+阿尔斯特大学,651,海外名校,Ulster University
+黎巴嫩大学,701,海外名校,Lebanese University
+米德尔塞克斯大学,701,海外名校,Middlesex University
+乌克兰国立技术大学 – 伊戈尔·西科斯基基辅理工学院,701,海外名校,National Technical University of Ukraine – Igor Sikorsky Kyiv Poly
+新泽西理工大学,701,海外名校,New Jersey Institute of Technology
+诺桑比亚大学,701,海外名校,Northumbria University
+印度金德而全球大学,701,海外名校,O.P. Jindal Global University
+大阪府大学,701,海外名校,Osaka Prefecture University
+马尔堡菲尔兹大学,701,海外名校,Philipps University of Marburg
+普列汉诺夫俄罗斯经济大学,701,海外名校,Plekhanov Russian University of Economics
+巴里理工大学,701,海外名校,Politecnico di Bari
+穆罕默德·本·法赫德王子大学,701,海外名校,Prince Mohammad Bin Fahd university
+罗维拉-威尔吉利大学,701,海外名校,Rovira i Virgili University
+圣彼得堡国立电子科技大学,701,海外名校,Saint Petersburg Electrotechnical University ETU-LETI
+史蒂文斯理工学院,701,海外名校,Stevens Institute of Technology
+苏美州立大学,701,海外名校,Sumy State University
+东京都立大学,701,海外名校,Tokyo Metropolitan University
+乌法国立航空技术大学,701,海外名校,Ufa State Aviation Technical University
+委内瑞拉中央大学,701,海外名校,Universidad Central de Venezuela
+萨瓦纳大学,701,海外名校,Universidad de La Sabana
+圣安德烈斯大学,701,海外名校,Universidad de San Andrés - UdeSA
+伊比利亚美洲大学,701,海外名校,Universidad Iberoamericana IBERO
+圣地亚哥德孔波斯特拉大学,701,海外名校,Universidade de Santiago de Compostela
+波茨坦大学,701,海外名校,Universität Potsdam
+蔚蓝海岸大学,701,海外名校,Université Côte d'Azur
+苏塞大学,701,海外名校,Université de Sousse
+布拉德福德大学,701,海外名校,University of Bradford
+海法大学,701,海外名校,University of Haifa
+休斯敦大学,701,海外名校,University of Houston
+哈德斯菲尔德大学,701,海外名校,University of Huddersfield
+摩德纳大学和雷焦艾米利亚,701,海外名校,University of Modena and Reggio Emilia
+南昆士兰大学,701,海外名校,University of Southern Queensland
+巴斯克大学,701,海外名校,University of the Basque Country
+威斯敏斯特大学,701,海外名校,University of Westminster
+温莎大学,701,海外名校,University of Windsor
+弗吉尼亚联邦大学,701,海外名校,Virginia Commonwealth University
+阿布扎比大学,701,海外名校,Abu Dhabi University
+阿治曼大学,701,海外名校,Ajman University
+艾因大学,701,海外名校,Al Ain University
+海峡大学,701,海外名校,Bogaziçi Üniversitesi
+布尔诺工业大学,701,海外名校,Brno University of Technology
+查尔斯达尔文大学,701,海外名校,Charles Darwin University
+纽约城市大学,701,海外名校,City University of New York
+群马大学,701,海外名校,Gunma University
+印度理工学院布巴内斯瓦尔,701,海外名校,Indian Institute of Technology Bhubaneswar
+布宜诺斯艾利斯技术大学,701,海外名校,Instituto Tecnológico de Buenos Aires (ITBA)
+伊斯坦布尔理工大学,701,海外名校,Istanbul Technical University
+约夫大学,701,海外名校,Jouf University
+阿道夫·伊巴涅斯大学,751,海外名校,Adolfo Ibáñez University
+中东美国大学,751,海外名校,American University of the Middle East
+白俄罗斯国立技术大学（BNTU）,751,海外名校,Belarusian National Technical University (BNTU)
+布雷西亚大学,751,海外名校,Brescia University
+全南国立大学,751,海外名校,Chonnam National University
+忠南国立大学,751,海外名校,Chungnam National University
+克拉克森大学,751,海外名校,Clarkson University
+檀国大学,751,海外名校,Dankook University
+南里奥格兰德联邦大学,751,海外名校,Federal University of Rio Grande do Sul
+佐治亚州立大学,751,海外名校,Georgia State University
+杜塞尔多夫大学,751,海外名校,Heinrich Heine University Duesseldorf
+印尼泗水理工大学,751,海外名校,Institut Teknologi Sepuluh Nopember
+国立理工学院 (IPN),751,海外名校,Instituto Politécnico Nacional (IPN)
+墨西哥自治技术学院,751,海外名校,Instituto Tecnológico Autónomo de México (ITAM)
+印度新德里国立伊斯兰大学,751,海外名校,Jamia Millia Islamia
+卡拉干达国立技术大学,751,海外名校,Karaganda State Technical University
+República大学（乌德拉尔）,751,海外名校,Universidad de la República (Udelar)
+罗萨里奥大学,751,海外名校,Universidad del Rosario
+基多圣弗朗西斯科大学,751,海外名校,Universidad San Francisco de Quito
+托拉卡托迪泰拉大学,751,海外名校,Universidad Torcuato Di Tella
+佩鲁贾研究大学,751,海外名校,Università degli Studi di Perugia
+舍布鲁克大学,751,海外名校,Université de Sherbrooke
+国能大学,751,海外名校,Universiti Tenaga Nasional (UNITEN)
+中佛罗里达大学,751,海外名校,University of Central Florida
+丹佛大学,751,海外名校,University of Denver
+格林威治大学,751,海外名校,University of Greenwich
+马里兰大学巴尔的摩,751,海外名校,"University of Maryland, Baltimore County"
+萨格勒布大学,751,海外名校,University of Zagreb
+曼尼珀尔高等教育学院,751,海外名校,Manipal Academy of Higher Education
+梅努斯大学,751,海外名校,Maynooth University
+纽芬兰纪念大学,751,海外名校,Memorial University of Newfoundland
+瓦尔帕莱索天主教大学,751,海外名校,Pontificia Universidad Catolica de Valparaiso
+努拉·本·公主阿卜杜勒拉曼大学,751,海外名校,Princess Nourah bint Abdulrahman University
+里加技术大学,751,海外名校,Riga Technical University
+立命馆大学,751,海外名校,Ritsumeikan University
+设拉子大学,751,海外名校,Shiraz University
+南十字大学,751,海外名校,Southern Cross University
+纽约州立大学奥尔巴尼分校,751,海外名校,State University of New York Albany
+塔林科技大学,751,海外名校,Tallinn University of Technology
+天普大学,751,海外名校,Temple University
+维尔纽斯Gediminas技术大学,751,海外名校,Vilnius Gediminas Technical University
+伍斯特理工学院,751,海外名校,Worcester Polytechnic Institute
+越南国立大学胡志明市,801,海外名校,Viet Nam National University Ho Chi Minh City (VNU-HCM)
+越南河内国立大学,801,海外名校,Vietnam National University Hanoi
+维陶塔斯·马格努斯大学,801,海外名校,Vytautas Magnus University
+西弗吉尼亚大学,801,海外名校,West Virginia University
+弗罗茨瓦夫科技大学,801,海外名校,Wroclaw University of Science and Technology
+西交利物浦大学,801,海外名校,Xi‘an Jiaotong-Liverpool University
+山口大学,801,海外名校,Yamaguchi University
+埃里温州立大学,801,海外名校,Yerevan State University
+岭南大学,801,海外名校,Yeungnam University
+横滨国立大学,801,海外名校,Yokohama National University
+查平戈自治大学,801,海外名校,Universidad Autónoma de Chapingo
+伊达尔戈自治大学,801,海外名校,Universidad Autónoma del Estado de Hidalgo
+墨西哥自治大学,801,海外名校,Universidad Autonoma del Estado de Mexico
+大都会大学（UAM）,801,海外名校,Universidad Autónoma Metropolitana (UAM)
+卡塔利卡大学安德烈斯·贝洛大学,801,海外名校,Universidad Católica Andres Bello
+乌拉圭卡托利卡大学（UCU）,801,海外名校,Universidad Católica del Uruguay (UCU)
+瓜达拉哈拉大学,801,海外名校,Universidad de Guadalajara
+美洲普埃布拉大学（UDLAP）,801,海外名校,Universidad de las Américas Puebla (UDLAP)
+洛斯安第斯大学-（ULA）梅里达,801,海外名校,Universidad de Los Andes - (ULA) Mérida
+洛斯安第斯大学-智利,801,海外名校,Universidad de los Andes - Chile
+奥维耶多大学,801,海外名校,Universidad de Oviedo
+瓦莱大学,801,海外名校,Universidad del Valle
+圣地亚哥大学门户网站,801,海外名校,University Diego Portales
+EAFIT大学,801,海外名校,Universidad EAFIT
+国立圣路易斯大学,801,海外名校,Universidad Nacional de San Luis
+圣马科斯国立大学市长,801,海外名校,Universidad Nacional Mayor de San Marcos
+西蒙玻利瓦尔大学（USB）,801,海外名校,Universidad Simón Bolívar (USB)
+巴拿马巴拿马技术大学（UTP）,801,海外名校,Universidad Tecnológica de Panamá (UTP)
+全国技术大学（UTN）,801,海外名校,Universidad Tecnológica Nacional (UTN)
+卡图里卡葡萄牙大学-UCP,801,海外名校,Universidade Católica Portuguesa - UCP
+拉科鲁尼亚大学,801,海外名校,Universidade da Coruna
+巴西利亚大学,801,海外名校,Universidade de Brasília
+费拉拉大学,801,海外名校,Universita' degli Studi di Ferrara
+乌迪内大学研究,801,海外名校,Università degli Studi di Udine
+意大利马尔凯理工大学,801,海外名校,Universita' Politecnica delle Marche
+帕贾扎兰大学,801,海外名校,Universitas Padjadjaran
+杜伊斯堡 - 埃森大学,801,海外名校,University of Duisburg-Essen
+里尔大学,801,海外名校,Université de Lille
+洛林大学,801,海外名校,Universite de Lorraine
+南特大学,801,海外名校,Université de Nantes
+雷恩第一大学,801,海外名校,Université de Rennes 1
+图卢兹大学1 Capitole,801,海外名校,Université Toulouse 1 Capitole
+吉隆坡大学,801,海外名校,Universiti Kuala Lumpur
+马来西亚彭亨大学,801,海外名校,Universiti Malaysia Pahang
+马来西亚玻璃市大学,801,海外名校,Universiti Malaysia Perlis
+拉曼大学,801,海外名校,Universiti Tunku Abdul Rahman (UTAR)
+阿利坎特大学,801,海外名校,University of Alicante
+巴格达大学,801,海外名校,University of Baghdad
+巴林大学,801,海外名校,University of Bahrain
+巴里大学,801,海外名校,University of Bari
+布莱顿大学,801,海外名校,University of Brighton
+加尔各答大学,801,海外名校,University of Calcutta
+中央兰开夏大学,801,海外名校,University of Central Lancashire
+达卡大学,801,海外名校,University of Dhaka
+东伦敦大学,801,海外名校,University of East London
+拉合尔工程技术大学（UET）,801,海外名校,University of Engineering & Technology (UET) Lahore
+格但斯克大学,801,海外名校,University of Gdansk
+哈特福德大学,801,海外名校,University of Hartford
+赫特福德大学,801,海外名校,University of Hertfordshire
+赫拉德茨克拉洛夫大学,801,海外名校,University of Hradec Kralove
+夸祖鲁纳塔尔大学,801,海外名校,University of KwaZulu Natal
+罗兹大学,801,海外名校,University of Lodz
+马耳他大学,801,海外名校,University of Malta
+马里博尔大学,801,海外名校,University of Maribor
+墨西拿大学,801,海外名校,University of Messina
+密西西比大学,801,海外名校,University of Mississippi
+密苏里大学堪萨斯城分校,801,海外名校,"University of Missouri, Kansas City"
+穆尔西亚大学,801,海外名校,University of Murcia
+新英格兰大学,801,海外名校,University of New England
+新罕布什尔大学,801,海外名校,University of New Hampshire
+帕尔马大学,801,海外名校,University of Parma
+帕特雷大学,801,海外名校,University of Patras
+萨勒诺大学,801,海外名校,University of Salerno
+索尔福德大学,801,海外名校,University of Salford
+首尔大学,801,海外名校,University of Seoul
+旁遮普大学,801,海外名校,University of the Punjab
+英格兰西部大学,801,海外名校,University of the West of England
+塔尔萨大学,801,海外名校,University of Tulsa
+秋明大学,801,海外名校,University of Tyumen
+维罗纳大学,801,海外名校,University of Verona
+威斯康星大学-密尔沃基,801,海外名校,University of Wisconsin - Milwaukee
+弗罗茨瓦夫大学,801,海外名校,University of Wroclaw
+怀俄明大学,801,海外名校,University of Wyoming
+齐里纳大学,801,海外名校,University of Žilina
+Y.A.院士布克托夫卡拉干达大学,801,海外名校,Academician Y.A. Buketov Karaganda University
+亚当·米基维奇大学,801,海外名校,Adam Mickiewicz University
+AGH科技大学,801,海外名校,AGH University of Science & Technology
+艾恩夏姆斯大学,801,海外名校,Ain Shams University
+安卡拉大学,801,海外名校,Ankara Üniversitesi
+安娜大学,801,海外名校,Anna University
+奥本大学,801,海外名校,Auburn University
+澳洲天主教大学,801,海外名校,Australian Catholic University
+孟加拉国工程技术大学,801,海外名校,Bangladesh University of Engineering and Technology
+贝鲁特阿拉伯大学,801,海外名校,Beirut Arab University
+比勒费尔德大学,801,海外名校,Bielefeld University
+纽约州立大学宾汉姆顿大学,801,海外名校,Binghamton University
+布达佩斯科技经济大学,801,海外名校,Budapest University of Technology & Economics
+威尼斯福斯卡里宫大学,801,海外名校,Ca’ Foscari University of Venice
+卡塔尼亚大学,801,海外名校,Catania University
+查尔斯斯特大学,801,海外名校,Charles Sturt University
+克莱姆森大学,801,海外名校,Clemson University
+布达佩斯科维努斯大学,801,海外名校,Corvinus University of Budapest
+克拉科夫工业大学,801,海外名校,Cracow University of Technology
+CY塞尔吉巴黎大学,801,海外名校,CY Cergy Paris Université
+布拉格捷克生命科学大学,801,海外名校,Czech University of Life Sciences in Prague
+德拉萨大学,801,海外名校,De La Salle University
+德蒙福特大学,801,海外名校,De Montfort University
+多特蒙德工业大学,801,海外名校,Technical University of Dortmund
+爱丁堡纳皮尔大学,801,海外名校,Edinburgh Napier University
+巴拉那联邦大学,801,海外名校,Federal University of Parana
+伯南布哥联邦大学,801,海外名校,Federal University of Pernambuco
+圣卡塔琳娜联邦大学,801,海外名校,Federal University of Santa Catarina
+圣卡洛斯联邦大学,801,海外名校,Federal University of Sao Carlos
+费德里科圣玛丽亚技术大学,801,海外名校,Federico Santa María Technical University
+俄罗斯联邦政府金融大学,801,海外名校,Financial University under the Government of the Russian Federation
+佛罗里达国际大学,801,海外名校,Florida International University
+福特汉姆大学,801,海外名校,Fordham University
+格但斯克工业大学,801,海外名校,Gdansk University of Technology
+乔治梅森大学,801,海外名校,George Mason University
+德国约旦大学,801,海外名校,German Jordanian University
+海湾科技大学,801,海外名校,Gulf University for Science and Technology
+哈斯特帕大学,801,海外名校,Hacettepe University
+印第安纳大学-印第安纳波利斯,801,海外名校,Indiana University-Purdue University at Indianapolis
+国际基督教大学,801,海外名校,International Christian University
+麦地那伊斯兰大学,801,海外名校,Islamic University of Madinah
+伊斯坦布尔大学,801,海外名校,Istanbul University
+约旦科技大学,801,海外名校,Jordan University of Science & Technology
+堪萨斯州立大学,801,海外名校,Kansas State University
+卡塞萨特大学,801,海外名校,Kasetsart University
+考纳斯工业大学,801,海外名校,Kaunas University of Technology
+哈萨克阿布赖汗国际关系与世界语言大学,801,海外名校,Kazakh Ablai Khan University of International Relations and World Languages
+哈萨克斯坦-英国技术大学,801,海外名校,Kazakh-British Technical University
+喀山国立研究技术大学,801,海外名校,Kazan National Research Technical University
+孔敬大学,801,海外名校,Khon Kaen University
+费萨尔国王大学,801,海外名校,King Faisal University
+泰国国王科技大学,801,海外名校,King Mongkut’s University of Technology Thonburi
+京都工业大学,801,海外名校,Kyoto Institute of Technology
+九州工业大学,801,海外名校,Kyushu Institute of Technology
+利物浦约翰摩尔斯大学,801,海外名校,Liverpool John Moores University
+罗兹工业大学,801,海外名校,Lodz University of Technology
+伦敦都会大学,801,海外名校,London Metropolitan University
+伦敦南岸大学,801,海外名校,London South Bank University
+路易斯安那州立大学,801,海外名校,Louisiana State University
+芝加哥洛约拉大学,801,海外名校,Loyola University Chicago
+利沃夫理工大学,801,海外名校,Lviv Polytechnic National University
+曼彻斯特城市大学,801,海外名校,Manchester Metropolitan University
+布尔诺的孟德尔大学,801,海外名校,Mendel University Brno
+门捷列夫化工大学,801,海外名校,Mendeleev University of Chemical Technology
+国立中正大学,801,海外名校,National Chung Cheng University
+科尔多瓦国立大学,801,海外名校,National University of Cordoba
+罗萨里奥国立大学,801,海外名校,National University of Rosario
+尼古拉·哥白尼大学,801,海外名校,Nicolaus Copernicus University
+NJSC KIMEP大学,801,海外名校,NJSC KIMEP University
+诺丁汉特伦特大学,801,海外名校,Nottingham Trent University
+新西伯利亚国立技术大学,801,海外名校,Novosibirsk State Technical University
+俄克拉荷马州立大学,801,海外名校,Oklahoma State University
+萨尔茨堡巴黎罗德隆大学,801,海外名校,Paris Lodron University of Salzburg
+那不勒斯帕斯诺普大学,801,海外名校,Parthenope University Naples
+彼尔姆国立研究大学,801,海外名校,Perm State National Research University
+本地治里大学,801,海外名校,Pondicherry University
+圣保罗天主教大学,801,海外名校,Pontifícia Universidade Católica de São Paulo
+波兹南理工大学,801,海外名校,Poznan University of Technology
+宋卡王子大学,801,海外名校,Prince of Songkla University
+苏美亚公主科技大学,801,海外名校,Princess Sumaya University for Technology
+卡西姆大学,801,海外名校,Qassim University
+英国爱丁堡玛格丽特女王大学,801,海外名校,"Queen Margaret University , Edinburgh"
+罗得大学,801,海外名校,Rhodes University
+里加斯特拉丁斯大学,801,海外名校,Riga Stradins University
+罗伯特·戈登大学,801,海外名校,Robert Gordon University
+罗马特雷大学,801,海外名校,Roma Tre University
+俄罗斯总统国民经济与公共行政学院,801,海外名校,Russian Presidential Academy of National Economy and Public Admini
+罗格斯大学纽瓦克分校,801,海外名校,Rutgers University - Newark
+瑞尔森大学,801,海外名校,Ryerson University
+西雅图大学,801,海外名校,Seattle University
+信州大学,801,海外名校,Shinshu University
+Siksha'O'Anusandhan,801,海外名校,Siksha ‘O’ Anusandhan
+西里西亚工业大学,801,海外名校,Silesian University of Technology
+斯洛伐克工业大学布拉迪斯拉发,801,海外名校,Slovak University of Technology Bratislava
+上智大学,801,海外名校,Sophia University
+南乌拉尔州立大学,801,海外名校,South Ural State University
+南卫理公会大学,801,海外名校,Southern Methodist University
+塞切尼伊斯特万大学,801,海外名校,Széchenyi István University
+圣伊斯特万大学,801,海外名校,Szent Istvan University
+科希策技术大学,801,海外名校,Technical University of Kosice
+利贝雷茨技术大学,801,海外名校,Technical University of Liberec
+都柏林科技大学,801,海外名校,Technological University Dublin
+哥斯达黎加技术中心（TECN）,801,海外名校,Tecnológico de Costa Rica -TEC
+德克萨斯理工大学,801,海外名校,Texas Tech University
+阿拉巴马大学,801,海外名校,The University of Alabama
+阿尔伯塔大学,126,海外名校,
+加州艺术学院,1,海外名校,California Institute of the Arts
+萨凡纳艺术学院,2,海外名校,Savannah College of Art and Design
+瑞格林艺术设计学院,3,海外名校,Ringling College of Art and Design
+纽约视觉艺术学院,4,海外名校,School of Visual Art
+谢尔丹学院,6,海外名校,Sheridan College
+伯恩茅斯大学,7,海外名校,Bournemouth University
+提赛德大学,8,海外名校,Teesside University
+高布兰学院,9,海外名校,Gobelins

deepdoc/parser/resume/entities/schools.py

@@ -0,0 +1,62 @@
+# -*- coding: UTF-8 -*-
+import os, json,re,copy
+import pandas as pd
+current_file_path = os.path.dirname(os.path.abspath(__file__))
+TBL = pd.read_csv(os.path.join(current_file_path, "res/schools.csv"), sep="\t", header=0).fillna("")
+TBL["name_en"] = TBL["name_en"].map(lambda x: x.lower().strip())
+GOOD_SCH = json.load(open(os.path.join(current_file_path, "res/good_sch.json"), "r"))
+GOOD_SCH = set([re.sub(r"[,. &（）()]+", "", c) for c in GOOD_SCH])
+
+
+def loadRank(fnm):
+    global TBL
+    TBL["rank"] = 1000000
+    with open(fnm, "r",encoding='UTF-8') as f:
+        while True:
+            l = f.readline()
+            if not l:break
+            l = l.strip("\n").split(",")
+            try:
+                nm,rk = l[0].strip(),int(l[1])
+                #assert len(TBL[((TBL.name_cn == nm) | (TBL.name_en == nm))]),f"<{nm}>"
+                TBL.loc[((TBL.name_cn == nm) | (TBL.name_en == nm)), "rank"] = rk
+            except Exception as e:
+                pass
+
+
+loadRank(os.path.join(current_file_path, "res/school.rank.csv"))
+
+
+def split(txt):
+    tks = []
+    for t in re.sub(r"[ \t]+", " ",txt).split(" "):
+        if tks and re.match(r".*[a-zA-Z]$", tks[-1]) and \
+           re.match(r"[a-zA-Z]", t) and tks:
+            tks[-1] = tks[-1] + " " + t
+        else:tks.append(t)
+    return tks
+
+
+def select(nm):
+    global TBL
+    if not nm:return 
+    if isinstance(nm, list):nm = str(nm[0])
+    nm = split(nm)[0]
+    nm = str(nm).lower().strip()
+    nm = re.sub(r"[(（][^()（）]+[)）]", "", nm.lower())
+    nm = re.sub(r"(^the |[,.&（）();；·]+|^(英国|美国|瑞士))", "", nm)
+    nm = re.sub(r"大学.*学院", "大学", nm)
+    tbl = copy.deepcopy(TBL)
+    tbl["hit_alias"] = tbl["alias"].map(lambda x:nm in set(x.split("+")))
+    res = tbl[((tbl.name_cn == nm) | (tbl.name_en == nm) | (tbl.hit_alias == True))]
+    if res.empty:return
+
+    return json.loads(res.to_json(orient="records"))[0]
+
+
+def is_good(nm):
+    global GOOD_SCH
+    nm = re.sub(r"[(（][^()（）]+[)）]", "", nm.lower())
+    nm = re.sub(r"[''`‘’“”,. &（）();；]+", "", nm)
+    return nm in GOOD_SCH
+

deepdoc/parser/resume/step_one.py

@@ -0,0 +1,174 @@
+# -*- coding: utf-8 -*-
+import json
+from deepdoc.parser.resume.entities import degrees, regions, industries
+
+FIELDS = [
+"address STRING",
+"annual_salary int",
+"annual_salary_from int",
+"annual_salary_to int",
+"birth STRING",
+"card STRING",
+"certificate_obj string",
+"city STRING",
+"corporation_id int",
+"corporation_name STRING",
+"corporation_type STRING",
+"degree STRING",
+"discipline_name STRING",
+"education_obj string",
+"email STRING",
+"expect_annual_salary int",
+"expect_city_names string",
+"expect_industry_name STRING",
+"expect_position_name STRING",
+"expect_salary_from int",
+"expect_salary_to int",
+"expect_type STRING",
+"gender STRING",
+"industry_name STRING",
+"industry_names STRING",
+"is_deleted STRING",
+"is_fertility STRING",
+"is_house STRING",
+"is_management_experience STRING",
+"is_marital STRING",
+"is_oversea STRING",
+"language_obj string",
+"name STRING",
+"nation STRING",
+"phone STRING",
+"political_status STRING",
+"position_name STRING",
+"project_obj string",
+"responsibilities string",
+"salary_month int",
+"scale STRING",
+"school_name STRING",
+"self_remark string",
+"skill_obj string",
+"title_name STRING",
+"tob_resume_id STRING",
+"updated_at Timestamp",
+"wechat STRING",
+"work_obj string",
+"work_experience int",
+"work_start_time BIGINT"
+]
+
+def refactor(df):
+    def deal_obj(obj, k, kk):
+        if not isinstance(obj, type({})):
+            return ""
+        obj = obj.get(k, {})
+        if not isinstance(obj, type({})):
+            return ""
+        return obj.get(kk, "")
+
+    def loadjson(line):
+        try:
+            return json.loads(line)
+        except Exception as e:
+            pass
+        return {}
+
+    df["obj"] = df["resume_content"].map(lambda x: loadjson(x))
+    df.fillna("", inplace=True)
+
+    clms = ["tob_resume_id", "updated_at"]
+
+    def extract(nms, cc=None):
+        nonlocal clms
+        clms.extend(nms)
+        for c in nms:
+            if cc:
+                df[c] = df["obj"].map(lambda x: deal_obj(x, cc, c))
+            else:
+                df[c] = df["obj"].map(
+                    lambda x: json.dumps(
+                        x.get(
+                            c,
+                            {}),
+                        ensure_ascii=False) if isinstance(
+                        x,
+                        type(
+                            {})) and (
+                        isinstance(
+                            x.get(c),
+                            type(
+                                {})) or not x.get(c)) else str(x).replace(
+                                    "None",
+                        ""))
+
+    extract(["education", "work", "certificate", "project", "language",
+             "skill"])
+    extract(["wechat", "phone", "is_deleted",
+            "name", "tel", "email"], "contact")
+    extract(["nation", "expect_industry_name", "salary_month",
+             "industry_ids", "is_house", "birth", "annual_salary_from",
+             "annual_salary_to", "card",
+             "expect_salary_to", "expect_salary_from",
+             "expect_position_name", "gender", "city",
+             "is_fertility", "expect_city_names",
+             "political_status", "title_name", "expect_annual_salary",
+             "industry_name", "address", "position_name", "school_name",
+             "corporation_id",
+             "is_oversea", "responsibilities",
+             "work_start_time", "degree", "management_experience",
+             "expect_type", "corporation_type", "scale", "corporation_name",
+             "self_remark", "annual_salary", "work_experience",
+             "discipline_name", "marital", "updated_at"], "basic")
+
+    df["degree"] = df["degree"].map(lambda x: degrees.get_name(x))
+    df["address"] = df["address"].map(lambda x: " ".join(regions.get_names(x)))
+    df["industry_names"] = df["industry_ids"].map(lambda x: " ".join([" ".join(industries.get_names(i)) for i in
+                                                                      str(x).split(",")]))
+    clms.append("industry_names")
+
+    def arr2str(a):
+        if not a:
+            return ""
+        if isinstance(a, list):
+            a = " ".join([str(i) for i in a])
+        return str(a).replace(",", " ")
+
+    df["expect_industry_name"] = df["expect_industry_name"].map(
+        lambda x: arr2str(x))
+    df["gender"] = df["gender"].map(
+        lambda x: "男" if x == 'M' else (
+            "女" if x == 'F' else ""))
+    for c in ["is_fertility", "is_oversea", "is_house",
+              "management_experience", "marital"]:
+        df[c] = df[c].map(
+            lambda x: '是' if x == 'Y' else (
+                '否' if x == 'N' else ""))
+    df["is_management_experience"] = df["management_experience"]
+    df["is_marital"] = df["marital"]
+    clms.extend(["is_management_experience", "is_marital"])
+
+    df.fillna("", inplace=True)
+    for i in range(len(df)):
+        if not df.loc[i, "phone"].strip() and df.loc[i, "tel"].strip():
+            df.loc[i, "phone"] = df.loc[i, "tel"].strip()
+
+    for n in ["industry_ids", "management_experience", "marital", "tel"]:
+        for i in range(len(clms)):
+            if clms[i] == n:
+                del clms[i]
+                break
+
+    clms = list(set(clms))
+
+    df = df.reindex(sorted(clms), axis=1)
+    #print(json.dumps(list(df.columns.values)), "LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL")
+    for c in clms:
+        df[c] = df[c].map(
+            lambda s: str(s).replace(
+                "\t",
+                " ").replace(
+                "\n",
+                "\\n").replace(
+                "\r",
+                "\\n"))
+    # print(df.values.tolist())
+    return dict(zip([n.split(" ")[0] for n in FIELDS], df.values.tolist()[0]))

deepdoc/parser/resume/step_two.py

@@ -0,0 +1,580 @@
+# -*- coding: utf-8 -*-
+import re, copy, time, datetime, demjson3, \
+    traceback, signal
+import numpy as np
+from deepdoc.parser.resume.entities import degrees, schools, corporations
+from rag.nlp import rag_tokenizer, surname
+from xpinyin import Pinyin
+from contextlib import contextmanager
+
+
+class TimeoutException(Exception): pass
+
+
+@contextmanager
+def time_limit(seconds):
+    def signal_handler(signum, frame):
+        raise TimeoutException("Timed out!")
+
+    signal.signal(signal.SIGALRM, signal_handler)
+    signal.alarm(seconds)
+    try:
+        yield
+    finally:
+        signal.alarm(0)
+
+
+ENV = None
+PY = Pinyin()
+
+
+def rmHtmlTag(line):
+    return re.sub(r"<[a-z0-9.\"=';,:\+_/ -]+>", " ", line, 100000, re.IGNORECASE)
+
+
+def highest_degree(dg):
+    if not dg: return ""
+    if type(dg) == type(""): dg = [dg]
+    m = {"初中": 0, "高中": 1, "中专": 2, "大专": 3, "专升本": 4, "本科": 5, "硕士": 6, "博士": 7, "博士后": 8}
+    return sorted([(d, m.get(d, -1)) for d in dg], key=lambda x: x[1] * -1)[0][0]
+
+
+def forEdu(cv):
+    if not cv.get("education_obj"):
+        cv["integerity_flt"] *= 0.8
+        return cv
+
+    first_fea, fea, maj, fmaj, deg, fdeg, sch, fsch, st_dt, ed_dt = [], [], [], [], [], [], [], [], [], []
+    edu_nst = []
+    edu_end_dt = ""
+    cv["school_rank_int"] = 1000000
+    for ii, n in enumerate(sorted(cv["education_obj"], key=lambda x: x.get("start_time", "3"))):
+        e = {}
+        if n.get("end_time"):
+            if n["end_time"] > edu_end_dt: edu_end_dt = n["end_time"]
+            try:
+                dt = n["end_time"]
+                if re.match(r"[0-9]{9,}", dt): dt = turnTm2Dt(dt)
+                y, m, d = getYMD(dt)
+                ed_dt.append(str(y))
+                e["end_dt_kwd"] = str(y)
+            except Exception as e:
+                pass
+        if n.get("start_time"):
+            try:
+                dt = n["start_time"]
+                if re.match(r"[0-9]{9,}", dt): dt = turnTm2Dt(dt)
+                y, m, d = getYMD(dt)
+                st_dt.append(str(y))
+                e["start_dt_kwd"] = str(y)
+            except Exception as e:
+                pass
+
+        r = schools.select(n.get("school_name", ""))
+        if r:
+            if str(r.get("type", "")) == "1": fea.append("211")
+            if str(r.get("type", "")) == "2": fea.append("211")
+            if str(r.get("is_abroad", "")) == "1": fea.append("留学")
+            if str(r.get("is_double_first", "")) == "1": fea.append("双一流")
+            if str(r.get("is_985", "")) == "1": fea.append("985")
+            if str(r.get("is_world_known", "")) == "1": fea.append("海外知名")
+            if r.get("rank") and cv["school_rank_int"] > r["rank"]: cv["school_rank_int"] = r["rank"]
+
+        if n.get("school_name") and isinstance(n["school_name"], str):
+            sch.append(re.sub(r"(211|985|重点大学|[,&;；-])", "", n["school_name"]))
+            e["sch_nm_kwd"] = sch[-1]
+        fea.append(rag_tokenizer.fine_grained_tokenize(rag_tokenizer.tokenize(n.get("school_name", ""))).split(" ")[-1])
+
+        if n.get("discipline_name") and isinstance(n["discipline_name"], str):
+            maj.append(n["discipline_name"])
+            e["major_kwd"] = n["discipline_name"]
+
+        if not n.get("degree") and "985" in fea and not first_fea: n["degree"] = "1"
+
+        if n.get("degree"):
+            d = degrees.get_name(n["degree"])
+            if d: e["degree_kwd"] = d
+            if d == "本科" and ("专科" in deg or "专升本" in deg or "中专" in deg or "大专" in deg or re.search(r"(成人|自考|自学考试)",
+                                                                                                     n.get(
+                                                                                                         "school_name",
+                                                                                                         ""))): d = "专升本"
+            if d: deg.append(d)
+
+            # for first degree
+            if not fdeg and d in ["中专", "专升本", "专科", "本科", "大专"]:
+                fdeg = [d]
+                if n.get("school_name"): fsch = [n["school_name"]]
+                if n.get("discipline_name"): fmaj = [n["discipline_name"]]
+                first_fea = copy.deepcopy(fea)
+
+        edu_nst.append(e)
+
+    cv["sch_rank_kwd"] = []
+    if cv["school_rank_int"] <= 20 \
+            or ("海外名校" in fea and cv["school_rank_int"] <= 200):
+        cv["sch_rank_kwd"].append("顶尖学校")
+    elif cv["school_rank_int"] <= 50 and cv["school_rank_int"] > 20 \
+            or ("海外名校" in fea and cv["school_rank_int"] <= 500 and \
+                cv["school_rank_int"] > 200):
+        cv["sch_rank_kwd"].append("精英学校")
+    elif cv["school_rank_int"] > 50 and ("985" in fea or "211" in fea) \
+            or ("海外名校" in fea and cv["school_rank_int"] > 500):
+        cv["sch_rank_kwd"].append("优质学校")
+    else:
+        cv["sch_rank_kwd"].append("一般学校")
+
+    if edu_nst: cv["edu_nst"] = edu_nst
+    if fea: cv["edu_fea_kwd"] = list(set(fea))
+    if first_fea: cv["edu_first_fea_kwd"] = list(set(first_fea))
+    if maj: cv["major_kwd"] = maj
+    if fsch: cv["first_school_name_kwd"] = fsch
+    if fdeg: cv["first_degree_kwd"] = fdeg
+    if fmaj: cv["first_major_kwd"] = fmaj
+    if st_dt: cv["edu_start_kwd"] = st_dt
+    if ed_dt: cv["edu_end_kwd"] = ed_dt
+    if ed_dt: cv["edu_end_int"] = max([int(t) for t in ed_dt])
+    if deg:
+        if "本科" in deg and "专科" in deg:
+            deg.append("专升本")
+            deg = [d for d in deg if d != '本科']
+        cv["degree_kwd"] = deg
+        cv["highest_degree_kwd"] = highest_degree(deg)
+    if edu_end_dt:
+        try:
+            if re.match(r"[0-9]{9,}", edu_end_dt): edu_end_dt = turnTm2Dt(edu_end_dt)
+            if edu_end_dt.strip("\n") == "至今": edu_end_dt = cv.get("updated_at_dt", str(datetime.date.today()))
+            y, m, d = getYMD(edu_end_dt)
+            cv["work_exp_flt"] = min(int(str(datetime.date.today())[0:4]) - int(y), cv.get("work_exp_flt", 1000))
+        except Exception as e:
+            print("EXCEPTION: ", e, edu_end_dt, cv.get("work_exp_flt"))
+    if sch:
+        cv["school_name_kwd"] = sch
+        if (len(cv.get("degree_kwd", [])) >= 1 and "本科" in cv["degree_kwd"]) \
+                or all([c.lower() in ["硕士", "博士", "mba", "博士后"] for c in cv.get("degree_kwd", [])]) \
+                or not cv.get("degree_kwd"):
+            for c in sch:
+                if schools.is_good(c):
+                    if "tag_kwd" not in cv: cv["tag_kwd"] = []
+                    cv["tag_kwd"].append("好学校")
+                    cv["tag_kwd"].append("好学历")
+                    break
+        if (len(cv.get("degree_kwd", [])) >= 1 and \
+            "本科" in cv["degree_kwd"] and \
+            any([d.lower() in ["硕士", "博士", "mba", "博士"] for d in cv.get("degree_kwd", [])])) \
+                or all([d.lower() in ["硕士", "博士", "mba", "博士后"] for d in cv.get("degree_kwd", [])]) \
+                or any([d in ["mba", "emba", "博士后"] for d in cv.get("degree_kwd", [])]):
+            if "tag_kwd" not in cv: cv["tag_kwd"] = []
+            if "好学历" not in cv["tag_kwd"]: cv["tag_kwd"].append("好学历")
+
+    if cv.get("major_kwd"): cv["major_tks"] = rag_tokenizer.tokenize(" ".join(maj))
+    if cv.get("school_name_kwd"): cv["school_name_tks"] = rag_tokenizer.tokenize(" ".join(sch))
+    if cv.get("first_school_name_kwd"): cv["first_school_name_tks"] = rag_tokenizer.tokenize(" ".join(fsch))
+    if cv.get("first_major_kwd"): cv["first_major_tks"] = rag_tokenizer.tokenize(" ".join(fmaj))
+
+    return cv
+
+
+def forProj(cv):
+    if not cv.get("project_obj"): return cv
+
+    pro_nms, desc = [], []
+    for i, n in enumerate(
+            sorted(cv.get("project_obj", []), key=lambda x: str(x.get("updated_at", "")) if type(x) == type({}) else "",
+                   reverse=True)):
+        if n.get("name"): pro_nms.append(n["name"])
+        if n.get("describe"): desc.append(str(n["describe"]))
+        if n.get("responsibilities"): desc.append(str(n["responsibilities"]))
+        if n.get("achivement"): desc.append(str(n["achivement"]))
+
+    if pro_nms:
+        # cv["pro_nms_tks"] = rag_tokenizer.tokenize(" ".join(pro_nms))
+        cv["project_name_tks"] = rag_tokenizer.tokenize(pro_nms[0])
+    if desc:
+        cv["pro_desc_ltks"] = rag_tokenizer.tokenize(rmHtmlTag(" ".join(desc)))
+        cv["project_desc_ltks"] = rag_tokenizer.tokenize(rmHtmlTag(desc[0]))
+
+    return cv
+
+
+def json_loads(line):
+    return demjson3.decode(re.sub(r": *(True|False)", r": '\1'", line))
+
+
+def forWork(cv):
+    if not cv.get("work_obj"):
+        cv["integerity_flt"] *= 0.7
+        return cv
+
+    flds = ["position_name", "corporation_name", "corporation_id", "responsibilities",
+            "industry_name", "subordinates_count"]
+    duas = []
+    scales = []
+    fea = {c: [] for c in flds}
+    latest_job_tm = ""
+    goodcorp = False
+    goodcorp_ = False
+    work_st_tm = ""
+    corp_tags = []
+    for i, n in enumerate(
+            sorted(cv.get("work_obj", []), key=lambda x: str(x.get("start_time", "")) if type(x) == type({}) else "",
+                   reverse=True)):
+        if type(n) == type(""):
+            try:
+                n = json_loads(n)
+            except Exception as e:
+                continue
+
+        if n.get("start_time") and (not work_st_tm or n["start_time"] < work_st_tm): work_st_tm = n["start_time"]
+        for c in flds:
+            if not n.get(c) or str(n[c]) == '0':
+                fea[c].append("")
+                continue
+            if c == "corporation_name":
+                n[c] = corporations.corpNorm(n[c], False)
+                if corporations.is_good(n[c]):
+                    if i == 0:
+                        goodcorp = True
+                    else:
+                        goodcorp_ = True
+                ct = corporations.corp_tag(n[c])
+                if i == 0:
+                    corp_tags.extend(ct)
+                elif ct and ct[0] != "软外":
+                    corp_tags.extend([f"{t}(曾)" for t in ct])
+
+            fea[c].append(rmHtmlTag(str(n[c]).lower()))
+
+        y, m, d = getYMD(n.get("start_time"))
+        if not y or not m: continue
+        st = "%s-%02d-%02d" % (y, int(m), int(d))
+        latest_job_tm = st
+
+        y, m, d = getYMD(n.get("end_time"))
+        if (not y or not m) and i > 0: continue
+        if not y or not m or int(y) > 2022:  y, m, d = getYMD(str(n.get("updated_at", "")))
+        if not y or not m: continue
+        ed = "%s-%02d-%02d" % (y, int(m), int(d))
+
+        try:
+            duas.append((datetime.datetime.strptime(ed, "%Y-%m-%d") - datetime.datetime.strptime(st, "%Y-%m-%d")).days)
+        except Exception as e:
+            print("kkkkkkkkkkkkkkkkkkkk", n.get("start_time"), n.get("end_time"))
+
+        if n.get("scale"):
+            r = re.search(r"^([0-9]+)", str(n["scale"]))
+            if r: scales.append(int(r.group(1)))
+
+    if goodcorp:
+        if "tag_kwd" not in cv: cv["tag_kwd"] = []
+        cv["tag_kwd"].append("好公司")
+    if goodcorp_:
+        if "tag_kwd" not in cv: cv["tag_kwd"] = []
+        cv["tag_kwd"].append("好公司(曾)")
+
+    if corp_tags:
+        if "tag_kwd" not in cv: cv["tag_kwd"] = []
+        cv["tag_kwd"].extend(corp_tags)
+        cv["corp_tag_kwd"] = [c for c in corp_tags if re.match(r"(综合|行业)", c)]
+
+    if latest_job_tm: cv["latest_job_dt"] = latest_job_tm
+    if fea["corporation_id"]: cv["corporation_id"] = fea["corporation_id"]
+
+    if fea["position_name"]:
+        cv["position_name_tks"] = rag_tokenizer.tokenize(fea["position_name"][0])
+        cv["position_name_sm_tks"] = rag_tokenizer.fine_grained_tokenize(cv["position_name_tks"])
+        cv["pos_nm_tks"] = rag_tokenizer.tokenize(" ".join(fea["position_name"][1:]))
+
+    if fea["industry_name"]:
+        cv["industry_name_tks"] = rag_tokenizer.tokenize(fea["industry_name"][0])
+        cv["industry_name_sm_tks"] = rag_tokenizer.fine_grained_tokenize(cv["industry_name_tks"])
+        cv["indu_nm_tks"] = rag_tokenizer.tokenize(" ".join(fea["industry_name"][1:]))
+
+    if fea["corporation_name"]:
+        cv["corporation_name_kwd"] = fea["corporation_name"][0]
+        cv["corp_nm_kwd"] = fea["corporation_name"]
+        cv["corporation_name_tks"] = rag_tokenizer.tokenize(fea["corporation_name"][0])
+        cv["corporation_name_sm_tks"] = rag_tokenizer.fine_grained_tokenize(cv["corporation_name_tks"])
+        cv["corp_nm_tks"] = rag_tokenizer.tokenize(" ".join(fea["corporation_name"][1:]))
+
+    if fea["responsibilities"]:
+        cv["responsibilities_ltks"] = rag_tokenizer.tokenize(fea["responsibilities"][0])
+        cv["resp_ltks"] = rag_tokenizer.tokenize(" ".join(fea["responsibilities"][1:]))
+
+    if fea["subordinates_count"]: fea["subordinates_count"] = [int(i) for i in fea["subordinates_count"] if
+                                                               re.match(r"[^0-9]+$", str(i))]
+    if fea["subordinates_count"]: cv["max_sub_cnt_int"] = np.max(fea["subordinates_count"])
+
+    if type(cv.get("corporation_id")) == type(1): cv["corporation_id"] = [str(cv["corporation_id"])]
+    if not cv.get("corporation_id"): cv["corporation_id"] = []
+    for i in cv.get("corporation_id", []):
+        cv["baike_flt"] = max(corporations.baike(i), cv["baike_flt"] if "baike_flt" in cv else 0)
+
+    if work_st_tm:
+        try:
+            if re.match(r"[0-9]{9,}", work_st_tm): work_st_tm = turnTm2Dt(work_st_tm)
+            y, m, d = getYMD(work_st_tm)
+            cv["work_exp_flt"] = min(int(str(datetime.date.today())[0:4]) - int(y), cv.get("work_exp_flt", 1000))
+        except Exception as e:
+            print("EXCEPTION: ", e, work_st_tm, cv.get("work_exp_flt"))
+
+    cv["job_num_int"] = 0
+    if duas:
+        cv["dua_flt"] = np.mean(duas)
+        cv["cur_dua_int"] = duas[0]
+        cv["job_num_int"] = len(duas)
+    if scales: cv["scale_flt"] = np.max(scales)
+    return cv
+
+
+def turnTm2Dt(b):
+    if not b: return
+    b = str(b).strip()
+    if re.match(r"[0-9]{10,}", b): b = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(int(b[:10])))
+    return b
+
+
+def getYMD(b):
+    y, m, d = "", "", "01"
+    if not b: return (y, m, d)
+    b = turnTm2Dt(b)
+    if re.match(r"[0-9]{4}", b): y = int(b[:4])
+    r = re.search(r"[0-9]{4}.?([0-9]{1,2})", b)
+    if r: m = r.group(1)
+    r = re.search(r"[0-9]{4}.?[0-9]{,2}.?([0-9]{1,2})", b)
+    if r: d = r.group(1)
+    if not d or int(d) == 0 or int(d) > 31: d = "1"
+    if not m or int(m) > 12 or int(m) < 1: m = "1"
+    return (y, m, d)
+
+
+def birth(cv):
+    if not cv.get("birth"):
+        cv["integerity_flt"] *= 0.9
+        return cv
+    y, m, d = getYMD(cv["birth"])
+    if not m or not y: return cv
+    b = "%s-%02d-%02d" % (y, int(m), int(d))
+    cv["birth_dt"] = b
+    cv["birthday_kwd"] = "%02d%02d" % (int(m), int(d))
+
+    cv["age_int"] = datetime.datetime.now().year - int(y)
+    return cv
+
+
+def parse(cv):
+    for k in cv.keys():
+        if cv[k] == '\\N': cv[k] = ''
+    # cv = cv.asDict()
+    tks_fld = ["address", "corporation_name", "discipline_name", "email", "expect_city_names",
+               "expect_industry_name", "expect_position_name", "industry_name", "industry_names", "name",
+               "position_name", "school_name", "self_remark", "title_name"]
+    small_tks_fld = ["corporation_name", "expect_position_name", "position_name", "school_name", "title_name"]
+    kwd_fld = ["address", "city", "corporation_type", "degree", "discipline_name", "expect_city_names", "email",
+               "expect_industry_name", "expect_position_name", "expect_type", "gender", "industry_name",
+               "industry_names", "political_status", "position_name", "scale", "school_name", "phone", "tel"]
+    num_fld = ["annual_salary", "annual_salary_from", "annual_salary_to", "expect_annual_salary", "expect_salary_from",
+               "expect_salary_to", "salary_month"]
+
+    is_fld = [
+        ("is_fertility", "已育", "未育"),
+        ("is_house", "有房", "没房"),
+        ("is_management_experience", "有管理经验", "无管理经验"),
+        ("is_marital", "已婚", "未婚"),
+        ("is_oversea", "有海外经验", "无海外经验")
+    ]
+
+    rmkeys = []
+    for k in cv.keys():
+        if cv[k] is None: rmkeys.append(k)
+        if (type(cv[k]) == type([]) or type(cv[k]) == type("")) and len(cv[k]) == 0: rmkeys.append(k)
+    for k in rmkeys: del cv[k]
+
+    integerity = 0.
+    flds_num = 0.
+
+    def hasValues(flds):
+        nonlocal integerity, flds_num
+        flds_num += len(flds)
+        for f in flds:
+            v = str(cv.get(f, ""))
+            if len(v) > 0 and v != '0' and v != '[]': integerity += 1
+
+    hasValues(tks_fld)
+    hasValues(small_tks_fld)
+    hasValues(kwd_fld)
+    hasValues(num_fld)
+    cv["integerity_flt"] = integerity / flds_num
+
+    if cv.get("corporation_type"):
+        for p, r in [(r"(公司|企业|其它|其他|Others*|\n|未填写|Enterprises|Company|companies)", ""),
+                     (r"[／/．·　<\(（]+.*", ""),
+                     (r".*(合资|民企|股份制|中外|私营|个体|Private|创业|Owned|投资).*", "民营"),
+                     (r".*(机关|事业).*", "机关"),
+                     (r".*(非盈利|Non-profit).*", "非盈利"),
+                     (r".*(外企|外商|欧美|foreign|Institution|Australia|港资).*", "外企"),
+                     (r".*国有.*", "国企"),
+                     (r"[ （）\(\)人/·0-9-]+", ""),
+                     (r".*(元|规模|于|=|北京|上海|至今|中国|工资|州|shanghai|强|餐饮|融资|职).*", "")]:
+            cv["corporation_type"] = re.sub(p, r, cv["corporation_type"], 1000, re.IGNORECASE)
+        if len(cv["corporation_type"]) < 2: del cv["corporation_type"]
+
+    if cv.get("political_status"):
+        for p, r in [
+            (r".*党员.*", "党员"),
+            (r".*(无党派|公民).*", "群众"),
+            (r".*团员.*", "团员")]:
+            cv["political_status"] = re.sub(p, r, cv["political_status"])
+        if not re.search(r"[党团群]", cv["political_status"]): del cv["political_status"]
+
+    if cv.get("phone"): cv["phone"] = re.sub(r"^0*86([0-9]{11})", r"\1", re.sub(r"[^0-9]+", "", cv["phone"]))
+
+    keys = list(cv.keys())
+    for k in keys:
+        # deal with json objects
+        if k.find("_obj") > 0:
+            try:
+                cv[k] = json_loads(cv[k])
+                cv[k] = [a for _, a in cv[k].items()]
+                nms = []
+                for n in cv[k]:
+                    if type(n) != type({}) or "name" not in n or not n.get("name"): continue
+                    n["name"] = re.sub(r"(（442）|\t )", "", n["name"]).strip().lower()
+                    if not n["name"]: continue
+                    nms.append(n["name"])
+                if nms:
+                    t = k[:-4]
+                    cv[f"{t}_kwd"] = nms
+                    cv[f"{t}_tks"] = rag_tokenizer.tokenize(" ".join(nms))
+            except Exception as e:
+                print("【EXCEPTION】:", str(traceback.format_exc()), cv[k])
+                cv[k] = []
+
+        # tokenize fields
+        if k in tks_fld:
+            cv[f"{k}_tks"] = rag_tokenizer.tokenize(cv[k])
+            if k in small_tks_fld: cv[f"{k}_sm_tks"] = rag_tokenizer.tokenize(cv[f"{k}_tks"])
+
+        # keyword fields
+        if k in kwd_fld: cv[f"{k}_kwd"] = [n.lower()
+                                           for n in re.split(r"[\t,，；;. ]",
+                                                             re.sub(r"([^a-zA-Z])[ ]+([^a-zA-Z ])", r"\1，\2", cv[k])
+                                                             ) if n]
+
+        if k in num_fld and cv.get(k): cv[f"{k}_int"] = cv[k]
+
+    cv["email_kwd"] = cv.get("email_tks", "").replace(" ", "")
+    # for name field
+    if cv.get("name"):
+        nm = re.sub(r"[\n——\-\(（\+].*", "", cv["name"].strip())
+        nm = re.sub(r"[ \t　]+", " ", nm)
+        if re.match(r"[a-zA-Z ]+$", nm):
+            if len(nm.split(" ")) > 1:
+                cv["name"] = nm
+            else:
+                nm = ""
+        elif nm and (surname.isit(nm[0]) or surname.isit(nm[:2])):
+            nm = re.sub(r"[a-zA-Z]+.*", "", nm[:5])
+        else:
+            nm = ""
+        cv["name"] = nm.strip()
+        name = cv["name"]
+
+        # name pingyin and its prefix
+        cv["name_py_tks"] = " ".join(PY.get_pinyins(nm[:20], '')) + " " + " ".join(PY.get_pinyins(nm[:20], ' '))
+        cv["name_py_pref0_tks"] = ""
+        cv["name_py_pref_tks"] = ""
+        for py in PY.get_pinyins(nm[:20], ''):
+            for i in range(2, len(py) + 1): cv["name_py_pref_tks"] += " " + py[:i]
+        for py in PY.get_pinyins(nm[:20], ' '):
+            py = py.split(" ")
+            for i in range(1, len(py) + 1): cv["name_py_pref0_tks"] += " " + "".join(py[:i])
+
+        cv["name_kwd"] = name
+        cv["name_pinyin_kwd"] = PY.get_pinyins(nm[:20], ' ')[:3]
+        cv["name_tks"] = (
+                rag_tokenizer.tokenize(name) + " " + (" ".join(list(name)) if not re.match(r"[a-zA-Z ]+$", name) else "")
+        ) if name else ""
+    else:
+        cv["integerity_flt"] /= 2.
+
+    if cv.get("phone"):
+        r = re.search(r"(1[3456789][0-9]{9})", cv["phone"])
+        if not r:
+            cv["phone"] = ""
+        else:
+            cv["phone"] = r.group(1)
+
+    # deal with date  fields
+    if cv.get("updated_at") and isinstance(cv["updated_at"], datetime.datetime):
+        cv["updated_at_dt"] = cv["updated_at"].strftime('%Y-%m-%d %H:%M:%S')
+    else:
+        y, m, d = getYMD(str(cv.get("updated_at", "")))
+        if not y: y = "2012"
+        if not m: m = "01"
+        if not d: d = "01"
+        cv["updated_at_dt"] = f"%s-%02d-%02d 00:00:00" % (y, int(m), int(d))
+        # long text tokenize
+
+    if cv.get("responsibilities"): cv["responsibilities_ltks"] = rag_tokenizer.tokenize(rmHtmlTag(cv["responsibilities"]))
+
+    # for yes or no field
+    fea = []
+    for f, y, n in is_fld:
+        if f not in cv: continue
+        if cv[f] == '是': fea.append(y)
+        if cv[f] == '否': fea.append(n)
+
+    if fea: cv["tag_kwd"] = fea
+
+    cv = forEdu(cv)
+    cv = forProj(cv)
+    cv = forWork(cv)
+    cv = birth(cv)
+
+    cv["corp_proj_sch_deg_kwd"] = [c for c in cv.get("corp_tag_kwd", [])]
+    for i in range(len(cv["corp_proj_sch_deg_kwd"])):
+        for j in cv.get("sch_rank_kwd", []): cv["corp_proj_sch_deg_kwd"][i] += "+" + j
+    for i in range(len(cv["corp_proj_sch_deg_kwd"])):
+        if cv.get("highest_degree_kwd"): cv["corp_proj_sch_deg_kwd"][i] += "+" + cv["highest_degree_kwd"]
+
+    try:
+        if not cv.get("work_exp_flt") and cv.get("work_start_time"):
+            if re.match(r"[0-9]{9,}", str(cv["work_start_time"])):
+                cv["work_start_dt"] = turnTm2Dt(cv["work_start_time"])
+                cv["work_exp_flt"] = (time.time() - int(int(cv["work_start_time"]) / 1000)) / 3600. / 24. / 365.
+            elif re.match(r"[0-9]{4}[^0-9]", str(cv["work_start_time"])):
+                y, m, d = getYMD(str(cv["work_start_time"]))
+                cv["work_start_dt"] = f"%s-%02d-%02d 00:00:00" % (y, int(m), int(d))
+                cv["work_exp_flt"] = int(str(datetime.date.today())[0:4]) - int(y)
+    except Exception as e:
+        print("【EXCEPTION】", e, "==>", cv.get("work_start_time"))
+    if "work_exp_flt" not in cv and cv.get("work_experience", 0): cv["work_exp_flt"] = int(cv["work_experience"]) / 12.
+
+    keys = list(cv.keys())
+    for k in keys:
+        if not re.search(r"_(fea|tks|nst|dt|int|flt|ltks|kwd|id)$", k): del cv[k]
+    for k in cv.keys():
+        if not re.search("_(kwd|id)$", k) or type(cv[k]) != type([]): continue
+        cv[k] = list(set([re.sub("(市)$", "", str(n)) for n in cv[k] if n not in ['中国', '0']]))
+    keys = [k for k in cv.keys() if re.search(r"_feas*$", k)]
+    for k in keys:
+        if cv[k] <= 0: del cv[k]
+
+    cv["tob_resume_id"] = str(cv["tob_resume_id"])
+    cv["id"] = cv["tob_resume_id"]
+    print("CCCCCCCCCCCCCCC")
+
+    return dealWithInt64(cv)
+
+
+def dealWithInt64(d):
+    if isinstance(d, dict):
+        for n, v in d.items():
+            d[n] = dealWithInt64(v)
+
+    if isinstance(d, list):
+        d = [dealWithInt64(t) for t in d]
+
+    if isinstance(d, np.integer): d = int(d)
+    return d
+

deepdoc/utils/file_utils.py

@@ -0,0 +1,216 @@
+#
+#  Copyright 2024 The InfiniFlow Authors. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+import base64
+import json
+import os
+import re
+from io import BytesIO
+
+import pdfplumber
+from PIL import Image
+from cachetools import LRUCache, cached
+from ruamel.yaml import YAML
+from strenum import StrEnum
+
+class FileType(StrEnum):
+    PDF = 'pdf'
+    DOC = 'doc'
+    VISUAL = 'visual'
+    AURAL = 'aural'
+    VIRTUAL = 'virtual'
+    FOLDER = 'folder'
+    OTHER = "other"
+
+
+PROJECT_BASE = os.getenv("RAG_PROJECT_BASE") or os.getenv("RAG_DEPLOY_BASE")
+RAG_BASE = os.getenv("RAG_BASE")
+
+
+def get_project_base_directory(*args):
+    global PROJECT_BASE
+    if PROJECT_BASE is None:
+        PROJECT_BASE = os.path.abspath(
+            os.path.join(
+                os.path.dirname(os.path.realpath(__file__)),
+                os.pardir,
+                os.pardir,
+            )
+        )
+
+    if args:
+        return os.path.join(PROJECT_BASE, *args)
+    return PROJECT_BASE
+
+
+def get_rag_directory(*args):
+    global RAG_BASE
+    if RAG_BASE is None:
+        RAG_BASE = os.path.abspath(
+            os.path.join(
+                os.path.dirname(os.path.realpath(__file__)),
+                os.pardir,
+                os.pardir,
+                os.pardir,
+            )
+        )
+    if args:
+        return os.path.join(RAG_BASE, *args)
+    return RAG_BASE
+
+
+def get_rag_python_directory(*args):
+    return get_rag_directory("python", *args)
+
+
+def get_home_cache_dir():
+    dir = os.path.join(os.path.expanduser('~'), ".ragflow")
+    try:
+        os.mkdir(dir)
+    except OSError as error:
+        pass
+    return dir
+
+
+@cached(cache=LRUCache(maxsize=10))
+def load_json_conf(conf_path):
+    if os.path.isabs(conf_path):
+        json_conf_path = conf_path
+    else:
+        json_conf_path = os.path.join(get_project_base_directory(), conf_path)
+    try:
+        with open(json_conf_path) as f:
+            return json.load(f)
+    except BaseException:
+        raise EnvironmentError(
+            "loading json file config from '{}' failed!".format(json_conf_path)
+        )
+
+
+def dump_json_conf(config_data, conf_path):
+    if os.path.isabs(conf_path):
+        json_conf_path = conf_path
+    else:
+        json_conf_path = os.path.join(get_project_base_directory(), conf_path)
+    try:
+        with open(json_conf_path, "w") as f:
+            json.dump(config_data, f, indent=4)
+    except BaseException:
+        raise EnvironmentError(
+            "loading json file config from '{}' failed!".format(json_conf_path)
+        )
+
+
+def load_json_conf_real_time(conf_path):
+    if os.path.isabs(conf_path):
+        json_conf_path = conf_path
+    else:
+        json_conf_path = os.path.join(get_project_base_directory(), conf_path)
+    try:
+        with open(json_conf_path) as f:
+            return json.load(f)
+    except BaseException:
+        raise EnvironmentError(
+            "loading json file config from '{}' failed!".format(json_conf_path)
+        )
+
+
+def load_yaml_conf(conf_path):
+    if not os.path.isabs(conf_path):
+        conf_path = os.path.join(get_project_base_directory(), conf_path)
+    try:
+        with open(conf_path) as f:
+            yaml = YAML(typ='safe', pure=True)
+            return yaml.load(f)
+    except Exception as e:
+        raise EnvironmentError(
+            "loading yaml file config from {} failed:".format(conf_path), e
+        )
+
+
+def rewrite_yaml_conf(conf_path, config):
+    if not os.path.isabs(conf_path):
+        conf_path = os.path.join(get_project_base_directory(), conf_path)
+    try:
+        with open(conf_path, "w") as f:
+            yaml = YAML(typ="safe")
+            yaml.dump(config, f)
+    except Exception as e:
+        raise EnvironmentError(
+            "rewrite yaml file config {} failed:".format(conf_path), e
+        )
+
+
+def rewrite_json_file(filepath, json_data):
+    with open(filepath, "w") as f:
+        json.dump(json_data, f, indent=4, separators=(",", ": "))
+    f.close()
+
+
+def filename_type(filename):
+    filename = filename.lower()
+    if re.match(r".*\.pdf$", filename):
+        return FileType.PDF.value
+
+    if re.match(
+            r".*\.(doc|docx|ppt|pptx|yml|xml|htm|json|csv|txt|ini|xls|xlsx|wps|rtf|hlp|pages|numbers|key|md|py|js|java|c|cpp|h|php|go|ts|sh|cs|kt)$", filename):
+        return FileType.DOC.value
+
+    if re.match(
+            r".*\.(wav|flac|ape|alac|wavpack|wv|mp3|aac|ogg|vorbis|opus|mp3)$", filename):
+        return FileType.AURAL.value
+
+    if re.match(r".*\.(jpg|jpeg|png|tif|gif|pcx|tga|exif|fpx|svg|psd|cdr|pcd|dxf|ufo|eps|ai|raw|WMF|webp|avif|apng|icon|ico|mpg|mpeg|avi|rm|rmvb|mov|wmv|asf|dat|asx|wvx|mpe|mpa|mp4)$", filename):
+        return FileType.VISUAL.value
+
+    return FileType.OTHER.value
+
+
+def thumbnail(filename, blob):
+    filename = filename.lower()
+    if re.match(r".*\.pdf$", filename):
+        pdf = pdfplumber.open(BytesIO(blob))
+        buffered = BytesIO()
+        pdf.pages[0].to_image(resolution=32).annotated.save(buffered, format="png")
+        return "data:image/png;base64," + \
+            base64.b64encode(buffered.getvalue()).decode("utf-8")
+
+    if re.match(r".*\.(jpg|jpeg|png|tif|gif|icon|ico|webp)$", filename):
+        image = Image.open(BytesIO(blob))
+        image.thumbnail((30, 30))
+        buffered = BytesIO()
+        image.save(buffered, format="png")
+        return "data:image/png;base64," + \
+            base64.b64encode(buffered.getvalue()).decode("utf-8")
+
+    if re.match(r".*\.(ppt|pptx)$", filename):
+        import aspose.slides as slides
+        import aspose.pydrawing as drawing
+        try:
+            with slides.Presentation(BytesIO(blob)) as presentation:
+                buffered = BytesIO()
+                presentation.slides[0].get_thumbnail(0.03, 0.03).save(
+                    buffered, drawing.imaging.ImageFormat.png)
+                return "data:image/png;base64," + \
+                    base64.b64encode(buffered.getvalue()).decode("utf-8")
+        except Exception as e:
+            pass
+
+
+def traversal_files(base):
+    for root, ds, fs in os.walk(base):
+        for f in fs:
+            fullname = os.path.join(root, f)
+            yield fullname

deepdoc/utils/log_utils.py

@@ -0,0 +1,313 @@
+#
+#  Copyright 2024 The InfiniFlow Authors. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+import os
+import typing
+import traceback
+import logging
+import inspect
+from logging.handlers import TimedRotatingFileHandler
+from threading import RLock
+
+from deepdoc.utils import file_utils
+
+
+class LoggerFactory(object):
+    TYPE = "FILE"
+    LOG_FORMAT = "[%(levelname)s] [%(asctime)s] [%(module)s.%(funcName)s] [line:%(lineno)d]: %(message)s"
+    logging.basicConfig(format=LOG_FORMAT)
+    LEVEL = logging.DEBUG
+    logger_dict = {}
+    global_handler_dict = {}
+
+    LOG_DIR = None
+    PARENT_LOG_DIR = None
+    log_share = True
+
+    append_to_parent_log = None
+
+    lock = RLock()
+    # CRITICAL = 50
+    # FATAL = CRITICAL
+    # ERROR = 40
+    # WARNING = 30
+    # WARN = WARNING
+    # INFO = 20
+    # DEBUG = 10
+    # NOTSET = 0
+    levels = (10, 20, 30, 40)
+    schedule_logger_dict = {}
+
+    @staticmethod
+    def set_directory(directory=None, parent_log_dir=None,
+                      append_to_parent_log=None, force=False):
+        if parent_log_dir:
+            LoggerFactory.PARENT_LOG_DIR = parent_log_dir
+        if append_to_parent_log:
+            LoggerFactory.append_to_parent_log = append_to_parent_log
+        with LoggerFactory.lock:
+            if not directory:
+                directory = file_utils.get_project_base_directory("logs")
+            if not LoggerFactory.LOG_DIR or force:
+                LoggerFactory.LOG_DIR = directory
+            if LoggerFactory.log_share:
+                oldmask = os.umask(000)
+                os.makedirs(LoggerFactory.LOG_DIR, exist_ok=True)
+                os.umask(oldmask)
+            else:
+                os.makedirs(LoggerFactory.LOG_DIR, exist_ok=True)
+            for loggerName, ghandler in LoggerFactory.global_handler_dict.items():
+                for className, (logger,
+                                handler) in LoggerFactory.logger_dict.items():
+                    logger.removeHandler(ghandler)
+                ghandler.close()
+            LoggerFactory.global_handler_dict = {}
+            for className, (logger,
+                            handler) in LoggerFactory.logger_dict.items():
+                logger.removeHandler(handler)
+                _handler = None
+                if handler:
+                    handler.close()
+                if className != "default":
+                    _handler = LoggerFactory.get_handler(className)
+                    logger.addHandler(_handler)
+                LoggerFactory.assemble_global_handler(logger)
+                LoggerFactory.logger_dict[className] = logger, _handler
+
+    @staticmethod
+    def new_logger(name):
+        logger = logging.getLogger(name)
+        logger.propagate = False
+        logger.setLevel(LoggerFactory.LEVEL)
+        return logger
+
+    @staticmethod
+    def get_logger(class_name=None):
+        with LoggerFactory.lock:
+            if class_name in LoggerFactory.logger_dict.keys():
+                logger, handler = LoggerFactory.logger_dict[class_name]
+                if not logger:
+                    logger, handler = LoggerFactory.init_logger(class_name)
+            else:
+                logger, handler = LoggerFactory.init_logger(class_name)
+            return logger
+
+    @staticmethod
+    def get_global_handler(logger_name, level=None, log_dir=None):
+        if not LoggerFactory.LOG_DIR:
+            return logging.StreamHandler()
+        if log_dir:
+            logger_name_key = logger_name + "_" + log_dir
+        else:
+            logger_name_key = logger_name + "_" + LoggerFactory.LOG_DIR
+        # if loggerName not in LoggerFactory.globalHandlerDict:
+        if logger_name_key not in LoggerFactory.global_handler_dict:
+            with LoggerFactory.lock:
+                if logger_name_key not in LoggerFactory.global_handler_dict:
+                    handler = LoggerFactory.get_handler(
+                        logger_name, level, log_dir)
+                    LoggerFactory.global_handler_dict[logger_name_key] = handler
+        return LoggerFactory.global_handler_dict[logger_name_key]
+
+    @staticmethod
+    def get_handler(class_name, level=None, log_dir=None,
+                    log_type=None, job_id=None):
+        if not log_type:
+            if not LoggerFactory.LOG_DIR or not class_name:
+                return logging.StreamHandler()
+                # return Diy_StreamHandler()
+
+            if not log_dir:
+                log_file = os.path.join(
+                    LoggerFactory.LOG_DIR,
+                    "{}.log".format(class_name))
+            else:
+                log_file = os.path.join(log_dir, "{}.log".format(class_name))
+        else:
+            log_file = os.path.join(log_dir, "rag_flow_{}.log".format(
+                log_type) if level == LoggerFactory.LEVEL else 'rag_flow_{}_error.log'.format(log_type))
+
+        os.makedirs(os.path.dirname(log_file), exist_ok=True)
+        if LoggerFactory.log_share:
+            handler = ROpenHandler(log_file,
+                                   when='D',
+                                   interval=1,
+                                   backupCount=14,
+                                   delay=True)
+        else:
+            handler = TimedRotatingFileHandler(log_file,
+                                               when='D',
+                                               interval=1,
+                                               backupCount=14,
+                                               delay=True)
+        if level:
+            handler.level = level
+
+        return handler
+
+    @staticmethod
+    def init_logger(class_name):
+        with LoggerFactory.lock:
+            logger = LoggerFactory.new_logger(class_name)
+            handler = None
+            if class_name:
+                handler = LoggerFactory.get_handler(class_name)
+                logger.addHandler(handler)
+                LoggerFactory.logger_dict[class_name] = logger, handler
+
+            else:
+                LoggerFactory.logger_dict["default"] = logger, handler
+
+            LoggerFactory.assemble_global_handler(logger)
+            return logger, handler
+
+    @staticmethod
+    def assemble_global_handler(logger):
+        if LoggerFactory.LOG_DIR:
+            for level in LoggerFactory.levels:
+                if level >= LoggerFactory.LEVEL:
+                    level_logger_name = logging._levelToName[level]
+                    logger.addHandler(
+                        LoggerFactory.get_global_handler(
+                            level_logger_name, level))
+        if LoggerFactory.append_to_parent_log and LoggerFactory.PARENT_LOG_DIR:
+            for level in LoggerFactory.levels:
+                if level >= LoggerFactory.LEVEL:
+                    level_logger_name = logging._levelToName[level]
+                    logger.addHandler(
+                        LoggerFactory.get_global_handler(level_logger_name, level, LoggerFactory.PARENT_LOG_DIR))
+
+
+def setDirectory(directory=None):
+    LoggerFactory.set_directory(directory)
+
+
+def setLevel(level):
+    LoggerFactory.LEVEL = level
+
+
+def getLogger(className=None, useLevelFile=False):
+    if className is None:
+        frame = inspect.stack()[1]
+        module = inspect.getmodule(frame[0])
+        className = 'stat'
+    return LoggerFactory.get_logger(className)
+
+
+def exception_to_trace_string(ex):
+    return "".join(traceback.TracebackException.from_exception(ex).format())
+
+
+class ROpenHandler(TimedRotatingFileHandler):
+    def _open(self):
+        prevumask = os.umask(000)
+        rtv = TimedRotatingFileHandler._open(self)
+        os.umask(prevumask)
+        return rtv
+
+
+def sql_logger(job_id='', log_type='sql'):
+    key = job_id + log_type
+    if key in LoggerFactory.schedule_logger_dict.keys():
+        return LoggerFactory.schedule_logger_dict[key]
+    return get_job_logger(job_id=job_id, log_type=log_type)
+
+
+def ready_log(msg, job=None, task=None, role=None, party_id=None, detail=None):
+    prefix, suffix = base_msg(job, task, role, party_id, detail)
+    return f"{prefix}{msg} ready{suffix}"
+
+
+def start_log(msg, job=None, task=None, role=None, party_id=None, detail=None):
+    prefix, suffix = base_msg(job, task, role, party_id, detail)
+    return f"{prefix}start to {msg}{suffix}"
+
+
+def successful_log(msg, job=None, task=None, role=None,
+                   party_id=None, detail=None):
+    prefix, suffix = base_msg(job, task, role, party_id, detail)
+    return f"{prefix}{msg} successfully{suffix}"
+
+
+def warning_log(msg, job=None, task=None, role=None,
+                party_id=None, detail=None):
+    prefix, suffix = base_msg(job, task, role, party_id, detail)
+    return f"{prefix}{msg} is not effective{suffix}"
+
+
+def failed_log(msg, job=None, task=None, role=None,
+               party_id=None, detail=None):
+    prefix, suffix = base_msg(job, task, role, party_id, detail)
+    return f"{prefix}failed to {msg}{suffix}"
+
+
+def base_msg(job=None, task=None, role: str = None,
+             party_id: typing.Union[str, int] = None, detail=None):
+    if detail:
+        detail_msg = f" detail: \n{detail}"
+    else:
+        detail_msg = ""
+    if task is not None:
+        return f"task {task.f_task_id} {task.f_task_version} ", f" on {task.f_role} {task.f_party_id}{detail_msg}"
+    elif job is not None:
+        return "", f" on {job.f_role} {job.f_party_id}{detail_msg}"
+    elif role and party_id:
+        return "", f" on {role} {party_id}{detail_msg}"
+    else:
+        return "", f"{detail_msg}"
+
+
+def exception_to_trace_string(ex):
+    return "".join(traceback.TracebackException.from_exception(ex).format())
+
+
+def get_logger_base_dir():
+    job_log_dir = file_utils.get_rag_flow_directory('logs')
+    return job_log_dir
+
+
+def get_job_logger(job_id, log_type):
+    rag_flow_log_dir = file_utils.get_rag_flow_directory('logs', 'rag_flow')
+    job_log_dir = file_utils.get_rag_flow_directory('logs', job_id)
+    if not job_id:
+        log_dirs = [rag_flow_log_dir]
+    else:
+        if log_type == 'audit':
+            log_dirs = [job_log_dir, rag_flow_log_dir]
+        else:
+            log_dirs = [job_log_dir]
+    if LoggerFactory.log_share:
+        oldmask = os.umask(000)
+        os.makedirs(job_log_dir, exist_ok=True)
+        os.makedirs(rag_flow_log_dir, exist_ok=True)
+        os.umask(oldmask)
+    else:
+        os.makedirs(job_log_dir, exist_ok=True)
+        os.makedirs(rag_flow_log_dir, exist_ok=True)
+    logger = LoggerFactory.new_logger(f"{job_id}_{log_type}")
+    for job_log_dir in log_dirs:
+        handler = LoggerFactory.get_handler(class_name=None, level=LoggerFactory.LEVEL,
+                                            log_dir=job_log_dir, log_type=log_type, job_id=job_id)
+        error_handler = LoggerFactory.get_handler(
+            class_name=None,
+            level=logging.ERROR,
+            log_dir=job_log_dir,
+            log_type=log_type,
+            job_id=job_id)
+        logger.addHandler(handler)
+        logger.addHandler(error_handler)
+    with LoggerFactory.lock:
+        LoggerFactory.schedule_logger_dict[job_id + log_type] = logger
+    return logger

deepdoc/utils/rag_tokenizer.py

@@ -0,0 +1,423 @@
+# -*- coding: utf-8 -*-
+
+import copy
+import datrie
+import math
+import os
+import re
+import string
+import sys
+from hanziconv import HanziConv
+from huggingface_hub import snapshot_download
+from nltk import word_tokenize
+from nltk.stem import PorterStemmer, WordNetLemmatizer
+from deepdoc.utils.file_utils import get_project_base_directory
+
+
+class RagTokenizer:
+    def key_(self, line):
+        return str(line.lower().encode("utf-8"))[2:-1]
+
+    def rkey_(self, line):
+        return str(("DD" + (line[::-1].lower())).encode("utf-8"))[2:-1]
+
+    def loadDict_(self, fnm):
+        print("[HUQIE]:Build trie", fnm, file=sys.stderr)
+        try:
+            of = open(fnm, "r")
+            while True:
+                line = of.readline()
+                if not line:
+                    break
+                line = re.sub(r"[\r\n]+", "", line)
+                line = re.split(r"[ \t]", line)
+                k = self.key_(line[0])
+                F = int(math.log(float(line[1]) / self.DENOMINATOR) + .5)
+                if k not in self.trie_ or self.trie_[k][0] < F:
+                    self.trie_[self.key_(line[0])] = (F, line[2])
+                self.trie_[self.rkey_(line[0])] = 1
+            self.trie_.save(fnm + ".trie")
+            of.close()
+        except Exception as e:
+            print("[HUQIE]:Faild to build trie, ", fnm, e, file=sys.stderr)
+
+    def __init__(self, debug=False):
+        self.DEBUG = debug
+        self.DENOMINATOR = 1000000
+        self.trie_ = datrie.Trie(string.printable)
+        self.DIR_ = os.path.join(get_project_base_directory(), "rag/res", "huqie")
+
+        self.stemmer = PorterStemmer()
+        self.lemmatizer = WordNetLemmatizer()
+
+        self.SPLIT_CHAR = r"([ ,\.<>/?;'\[\]\\`!@#$%^&*\(\)\{\}\|_+=《》，。？、；‘’：“”【】~！￥%……（）——-]+|[a-z\.-]+|[0-9,\.-]+)"
+        try:
+            self.trie_ = datrie.Trie.load(self.DIR_ + ".txt.trie")
+            return
+        except Exception as e:
+            print("[HUQIE]:Build default trie", file=sys.stderr)
+            self.trie_ = datrie.Trie(string.printable)
+
+        self.loadDict_(self.DIR_ + ".txt")
+
+    def loadUserDict(self, fnm):
+        try:
+            self.trie_ = datrie.Trie.load(fnm + ".trie")
+            return
+        except Exception as e:
+            self.trie_ = datrie.Trie(string.printable)
+        self.loadDict_(fnm)
+
+    def addUserDict(self, fnm):
+        self.loadDict_(fnm)
+
+    def _strQ2B(self, ustring):
+        """把字符串全角转半角"""
+        rstring = ""
+        for uchar in ustring:
+            inside_code = ord(uchar)
+            if inside_code == 0x3000:
+                inside_code = 0x0020
+            else:
+                inside_code -= 0xfee0
+            if inside_code < 0x0020 or inside_code > 0x7e:  # 转完之后不是半角字符返回原来的字符
+                rstring += uchar
+            else:
+                rstring += chr(inside_code)
+        return rstring
+
+    def _tradi2simp(self, line):
+        return HanziConv.toSimplified(line)
+
+    def dfs_(self, chars, s, preTks, tkslist):
+        MAX_L = 10
+        res = s
+        # if s > MAX_L or s>= len(chars):
+        if s >= len(chars):
+            tkslist.append(preTks)
+            return res
+
+        # pruning
+        S = s + 1
+        if s + 2 <= len(chars):
+            t1, t2 = "".join(chars[s:s + 1]), "".join(chars[s:s + 2])
+            if self.trie_.has_keys_with_prefix(self.key_(t1)) and not self.trie_.has_keys_with_prefix(
+                    self.key_(t2)):
+                S = s + 2
+        if len(preTks) > 2 and len(
+                preTks[-1][0]) == 1 and len(preTks[-2][0]) == 1 and len(preTks[-3][0]) == 1:
+            t1 = preTks[-1][0] + "".join(chars[s:s + 1])
+            if self.trie_.has_keys_with_prefix(self.key_(t1)):
+                S = s + 2
+
+        ################
+        for e in range(S, len(chars) + 1):
+            t = "".join(chars[s:e])
+            k = self.key_(t)
+
+            if e > s + 1 and not self.trie_.has_keys_with_prefix(k):
+                break
+
+            if k in self.trie_:
+                pretks = copy.deepcopy(preTks)
+                if k in self.trie_:
+                    pretks.append((t, self.trie_[k]))
+                else:
+                    pretks.append((t, (-12, '')))
+                res = max(res, self.dfs_(chars, e, pretks, tkslist))
+
+        if res > s:
+            return res
+
+        t = "".join(chars[s:s + 1])
+        k = self.key_(t)
+        if k in self.trie_:
+            preTks.append((t, self.trie_[k]))
+        else:
+            preTks.append((t, (-12, '')))
+
+        return self.dfs_(chars, s + 1, preTks, tkslist)
+
+    def freq(self, tk):
+        k = self.key_(tk)
+        if k not in self.trie_:
+            return 0
+        return int(math.exp(self.trie_[k][0]) * self.DENOMINATOR + 0.5)
+
+    def tag(self, tk):
+        k = self.key_(tk)
+        if k not in self.trie_:
+            return ""
+        return self.trie_[k][1]
+
+    def score_(self, tfts):
+        B = 30
+        F, L, tks = 0, 0, []
+        for tk, (freq, tag) in tfts:
+            F += freq
+            L += 0 if len(tk) < 2 else 1
+            tks.append(tk)
+        F /= len(tks)
+        L /= len(tks)
+        if self.DEBUG:
+            print("[SC]", tks, len(tks), L, F, B / len(tks) + L + F)
+        return tks, B / len(tks) + L + F
+
+    def sortTks_(self, tkslist):
+        res = []
+        for tfts in tkslist:
+            tks, s = self.score_(tfts)
+            res.append((tks, s))
+        return sorted(res, key=lambda x: x[1], reverse=True)
+
+    def merge_(self, tks):
+        patts = [
+            (r"[ ]+", " "),
+            (r"([0-9\+\.,%\*=-]) ([0-9\+\.,%\*=-])", r"\1\2"),
+        ]
+        # for p,s in patts: tks = re.sub(p, s, tks)
+
+        # if split chars is part of token
+        res = []
+        tks = re.sub(r"[ ]+", " ", tks).split(" ")
+        s = 0
+        while True:
+            if s >= len(tks):
+                break
+            E = s + 1
+            for e in range(s + 2, min(len(tks) + 2, s + 6)):
+                tk = "".join(tks[s:e])
+                if re.search(self.SPLIT_CHAR, tk) and self.freq(tk):
+                    E = e
+            res.append("".join(tks[s:E]))
+            s = E
+
+        return " ".join(res)
+
+    def maxForward_(self, line):
+        res = []
+        s = 0
+        while s < len(line):
+            e = s + 1
+            t = line[s:e]
+            while e < len(line) and self.trie_.has_keys_with_prefix(
+                    self.key_(t)):
+                e += 1
+                t = line[s:e]
+
+            while e - 1 > s and self.key_(t) not in self.trie_:
+                e -= 1
+                t = line[s:e]
+
+            if self.key_(t) in self.trie_:
+                res.append((t, self.trie_[self.key_(t)]))
+            else:
+                res.append((t, (0, '')))
+
+            s = e
+
+        return self.score_(res)
+
+    def maxBackward_(self, line):
+        res = []
+        s = len(line) - 1
+        while s >= 0:
+            e = s + 1
+            t = line[s:e]
+            while s > 0 and self.trie_.has_keys_with_prefix(self.rkey_(t)):
+                s -= 1
+                t = line[s:e]
+
+            while s + 1 < e and self.key_(t) not in self.trie_:
+                s += 1
+                t = line[s:e]
+
+            if self.key_(t) in self.trie_:
+                res.append((t, self.trie_[self.key_(t)]))
+            else:
+                res.append((t, (0, '')))
+
+            s -= 1
+
+        return self.score_(res[::-1])
+
+    def tokenize(self, line):
+        line = self._strQ2B(line).lower()
+        line = self._tradi2simp(line)
+        zh_num = len([1 for c in line if is_chinese(c)])
+        if zh_num < len(line) * 0.2:
+            return " ".join([self.stemmer.stem(self.lemmatizer.lemmatize(t)) for t in word_tokenize(line)])
+
+        arr = re.split(self.SPLIT_CHAR, line)
+        res = []
+        for L in arr:
+            if len(L) < 2 or re.match(
+                    r"[a-z\.-]+$", L) or re.match(r"[0-9\.-]+$", L):
+                res.append(L)
+                continue
+            # print(L)
+
+            # use maxforward for the first time
+            tks, s = self.maxForward_(L)
+            tks1, s1 = self.maxBackward_(L)
+            if self.DEBUG:
+                print("[FW]", tks, s)
+                print("[BW]", tks1, s1)
+
+            diff = [0 for _ in range(max(len(tks1), len(tks)))]
+            for i in range(min(len(tks1), len(tks))):
+                if tks[i] != tks1[i]:
+                    diff[i] = 1
+
+            if s1 > s:
+                tks = tks1
+
+            i = 0
+            while i < len(tks):
+                s = i
+                while s < len(tks) and diff[s] == 0:
+                    s += 1
+                if s == len(tks):
+                    res.append(" ".join(tks[i:]))
+                    break
+                if s > i:
+                    res.append(" ".join(tks[i:s]))
+
+                e = s
+                while e < len(tks) and e - s < 5 and diff[e] == 1:
+                    e += 1
+
+                tkslist = []
+                self.dfs_("".join(tks[s:e + 1]), 0, [], tkslist)
+                res.append(" ".join(self.sortTks_(tkslist)[0][0]))
+
+                i = e + 1
+
+        res = " ".join(res)
+        if self.DEBUG:
+            print("[TKS]", self.merge_(res))
+        return self.merge_(res)
+
+    def fine_grained_tokenize(self, tks):
+        tks = tks.split(" ")
+        zh_num = len([1 for c in tks if c and is_chinese(c[0])])
+        if zh_num < len(tks) * 0.2:
+            res = []
+            for tk in tks:
+                res.extend(tk.split("/"))
+            return " ".join(res)
+
+        res = []
+        for tk in tks:
+            if len(tk) < 3 or re.match(r"[0-9,\.-]+$", tk):
+                res.append(tk)
+                continue
+            tkslist = []
+            if len(tk) > 10:
+                tkslist.append(tk)
+            else:
+                self.dfs_(tk, 0, [], tkslist)
+            if len(tkslist) < 2:
+                res.append(tk)
+                continue
+            stk = self.sortTks_(tkslist)[1][0]
+            if len(stk) == len(tk):
+                stk = tk
+            else:
+                if re.match(r"[a-z\.-]+$", tk):
+                    for t in stk:
+                        if len(t) < 3:
+                            stk = tk
+                            break
+                    else:
+                        stk = " ".join(stk)
+                else:
+                    stk = " ".join(stk)
+
+            res.append(stk)
+
+        return " ".join(res)
+
+
+def is_chinese(s):
+    if s >= u'\u4e00' and s <= u'\u9fa5':
+        return True
+    else:
+        return False
+
+
+def is_number(s):
+    if s >= u'\u0030' and s <= u'\u0039':
+        return True
+    else:
+        return False
+
+
+def is_alphabet(s):
+    if (s >= u'\u0041' and s <= u'\u005a') or (
+            s >= u'\u0061' and s <= u'\u007a'):
+        return True
+    else:
+        return False
+
+
+def naiveQie(txt):
+    tks = []
+    for t in txt.split(" "):
+        if tks and re.match(r".*[a-zA-Z]$", tks[-1]
+                            ) and re.match(r".*[a-zA-Z]$", t):
+            tks.append(" ")
+        tks.append(t)
+    return tks
+
+
+tokenizer = RagTokenizer()
+tokenize = tokenizer.tokenize
+fine_grained_tokenize = tokenizer.fine_grained_tokenize
+tag = tokenizer.tag
+freq = tokenizer.freq
+loadUserDict = tokenizer.loadUserDict
+addUserDict = tokenizer.addUserDict
+tradi2simp = tokenizer._tradi2simp
+strQ2B = tokenizer._strQ2B
+
+if __name__ == '__main__':
+    tknzr = RagTokenizer(debug=True)
+    # huqie.addUserDict("/tmp/tmp.new.tks.dict")
+    tks = tknzr.tokenize(
+        "哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize(
+        "公开征求意见稿提出，境外投资者可使用自有人民币或外汇投资。使用外汇投资的，可通过债券持有人在香港人民币业务清算行及香港地区经批准可进入境内银行间外汇市场进行交易的境外人民币业务参加行（以下统称香港结算行）办理外汇资金兑换。香港结算行由此所产生的头寸可到境内银行间外汇市场平盘。使用外汇投资的，在其投资的债券到期或卖出后，原则上应兑换回外汇。")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize(
+        "多校划片就是一个小区对应多个小学初中，让买了学区房的家庭也不确定到底能上哪个学校。目的是通过这种方式为学区房降温，把就近入学落到实处。南京市长江大桥")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize(
+        "实际上当时他们已经将业务中心偏移到安全部门和针对政府企业的部门 Scripts are compiled and cached aaaaaaaaa")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize("虽然我不怎么玩")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize("蓝月亮如何在外资夹击中生存,那是全宇宙最有意思的")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize(
+        "涡轮增压发动机num最大功率,不像别的共享买车锁电子化的手段,我们接过来是否有意义,黄黄爱美食,不过，今天阿奇要讲到的这家农贸市场，说实话，还真蛮有特色的！不仅环境好，还打出了")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize("这周日你去吗？这周日你有空吗？")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize("Unity3D开发经验 测试开发工程师 c++双11双11 985 211 ")
+    print(tknzr.fine_grained_tokenize(tks))
+    tks = tknzr.tokenize(
+        "数据分析项目经理|数据分析挖掘|数据分析方向|商品数据分析|搜索数据分析 sql python hive tableau Cocos2d-")
+    print(tknzr.fine_grained_tokenize(tks))
+    if len(sys.argv) < 2:
+        sys.exit()
+    tknzr.DEBUG = False
+    tknzr.loadUserDict(sys.argv[1])
+    of = open(sys.argv[2], "r")
+    while True:
+        line = of.readline()
+        if not line:
+            break
+        print(tknzr.tokenize(line))
+    of.close()

deepdoc/vision/__init__.py

@@ -0,0 +1,48 @@
+import pdfplumber
+
+from .ocr import OCR
+from .recognizer import Recognizer
+from .layout_recognizer import LayoutRecognizer
+from .table_structure_recognizer import TableStructureRecognizer
+
+
+def init_in_out(args):
+    from PIL import Image
+    import os
+    import traceback
+    from deepdoc.utils.file_utils import traversal_files
+    images = []
+    outputs = []
+
+    if not os.path.exists(args.output_dir):
+        os.mkdir(args.output_dir)
+
+    def pdf_pages(fnm, zoomin=3):
+        nonlocal outputs, images
+        pdf = pdfplumber.open(fnm)
+        images = [p.to_image(resolution=72 * zoomin).annotated for i, p in
+                            enumerate(pdf.pages)]
+
+        for i, page in enumerate(images):
+            outputs.append(os.path.split(fnm)[-1] + f"_{i}.jpg")
+
+    def images_and_outputs(fnm):
+        nonlocal outputs, images
+        if fnm.split(".")[-1].lower() == "pdf":
+            pdf_pages(fnm)
+            return
+        try:
+            images.append(Image.open(fnm))
+            outputs.append(os.path.split(fnm)[-1])
+        except Exception as e:
+            traceback.print_exc()
+
+    if os.path.isdir(args.inputs):
+        for fnm in traversal_files(args.inputs):
+            images_and_outputs(fnm)
+    else:
+        images_and_outputs(args.inputs)
+
+    for i in range(len(outputs)): outputs[i] = os.path.join(args.output_dir, outputs[i])
+
+    return images, outputs