chore: 删除未使用的 bge-base/bge-large/punct 模型

- bge-base-zh-v1.5、bge-large-zh-v1.5:只有 tokenizer/config 元数据,
  onnx 权重从未提交(README 一直写"需自行下载”),代码里也完全没有
  任何地方引用这两个目录,纯占位死文件。
- punct(标点恢复模型,75M onnx):electron-builder.config.ts 的
  extraResources 从未把它纳入打包,装出来的应用里从来没有过这份模型,
  语音识别的标点恢复功能实际一直是禁用状态。用户确认现阶段不需要,
  直接删除源文件;代码逻辑保留(punctuationModelDir 为空时优雅跳过),
  以后要用可以自行准备模型再在设置里指定目录。

models/ 从 ~475M 降到 399M。README 目录说明与模型文件章节同步更新。

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ssurui 2026-08-23 00:36:17 +08:00
parent 33ce4ec09c
commit d686e600be
13 changed files with 5 additions and 315520 deletions

View File

@ -42,11 +42,8 @@ elec_rag_ui/
│ │ └── types/ # TypeScript 类型定义 │ │ └── types/ # TypeScript 类型定义
│ └── batch_questions.txt # 批量测试问题示例 │ └── batch_questions.txt # 批量测试问题示例
├── models/ # 本地模型(onnx 由 Git LFS 管理) ├── models/ # 本地模型(onnx 由 Git LFS 管理)
│ ├── bge-small-zh-v1.5/ # 嵌入模型(512 维,~24MB) │ ├── bge-small-zh-v1.5/ # 嵌入模型(512 维,~24MB),本地 CPU 向量化回退用
│ ├── bge-base-zh-v1.5/ # 嵌入模型(768 维,~98MB,onnx 需自行下载) │ └── asr/ # Paraformer 流式语音识别模型(~376MB)
│ ├── bge-large-zh-v1.5/ # 嵌入模型(1024 维,~312MB,onnx 需自行下载)
│ ├── asr/ # Paraformer 流式语音识别模型(~376MB)
│ └── punct/ # 标点恢复模型(~72MB)
├── docs/ # 示例文档(用于测试索引与问答) ├── docs/ # 示例文档(用于测试索引与问答)
├── resources/ # 应用图标、TTS/ASR worker ├── resources/ # 应用图标、TTS/ASR worker
├── stubs/sharp/ # sharp 原生模块 stub ├── stubs/sharp/ # sharp 原生模块 stub
@ -79,9 +76,9 @@ git lfs install
GIT_SSL_NO_VERIFY=true git lfs pull GIT_SSL_NO_VERIFY=true git lfs pull
``` ```
**bge-base / bge-large** 的 onnx 文件需自行从 HuggingFace 下载: > 语音识别默认不带自动加标点(标点恢复模型未随包分发);如需要可自行准备
- [bge-base-zh-v1.5](https://huggingface.co/BAAI/bge-base-zh-v1.5)(下载 `onnx/model_quantized.onnx`,约 98MB) > [sherpa-onnx punct 模型](https://k2-fsa.github.io/sherpa/onnx/punctuation/models.html),
- [bge-large-zh-v1.5](https://huggingface.co/BAAI/bge-large-zh-v1.5)(下载 `onnx/model_quantized.onnx`,约 312MB) > 在设置里把"标点恢复模型目录"指过去即可,代码本身已支持该功能。
## 开发模式 ## 开发模式

View File

@ -1,39 +0,0 @@
{
"_name_or_path": "BAAI/bge-base-zh-v1.5",
"architectures": [
"BertModel"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": 0,
"classifier_dropout": null,
"directionality": "bidi",
"eos_token_id": 2,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"id2label": {
"0": "LABEL_0"
},
"initializer_range": 0.02,
"intermediate_size": 3072,
"label2id": {
"LABEL_0": 0
},
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"output_past": true,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"position_embedding_type": "absolute",
"transformers_version": "4.34.0.dev0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 21128
}

View File

@ -1,7 +0,0 @@
{
"cls_token": "[CLS]",
"mask_token": "[MASK]",
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"unk_token": "[UNK]"
}

File diff suppressed because it is too large Load Diff

View File

@ -1,15 +0,0 @@
{
"clean_up_tokenization_spaces": true,
"cls_token": "[CLS]",
"do_basic_tokenize": true,
"do_lower_case": true,
"mask_token": "[MASK]",
"model_max_length": 512,
"never_split": null,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

View File

@ -1,39 +0,0 @@
{
"_name_or_path": "BAAI/bge-large-zh-v1.5",
"architectures": [
"BertModel"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": 0,
"classifier_dropout": null,
"directionality": "bidi",
"eos_token_id": 2,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 1024,
"id2label": {
"0": "LABEL_0"
},
"initializer_range": 0.02,
"intermediate_size": 4096,
"label2id": {
"LABEL_0": 0
},
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 16,
"num_hidden_layers": 24,
"output_past": true,
"pad_token_id": 0,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"position_embedding_type": "absolute",
"transformers_version": "4.34.0.dev0",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 21128
}

View File

@ -1,7 +0,0 @@
{
"cls_token": "[CLS]",
"mask_token": "[MASK]",
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"unk_token": "[UNK]"
}

File diff suppressed because it is too large Load Diff

View File

@ -1,15 +0,0 @@
{
"clean_up_tokenization_spaces": true,
"cls_token": "[CLS]",
"do_basic_tokenize": true,
"do_lower_case": true,
"mask_token": "[MASK]",
"model_max_length": 1000000000000000019884624838656,
"never_split": null,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

View File

@ -1,56 +0,0 @@
# Introduction
This model is converted from
https://modelscope.cn/models/iic/punc_ct-transformer_zh-cn-common-vocab272727-pytorch/summary
The commands to generate the onnx model are given below:
```
pip install funasr modelscope
pip install kaldi-native-fbank torchaudio onnx onnxruntime
mkdir -p /tmp/models
cd /tmp/models
git clone https://www.modelscope.cn/iic/punc_ct-transformer_zh-cn-common-vocab272727-pytorch.git
cd punc_ct-transformer_zh-cn-common-vocab272727-pytorch
git lfs pull --include model.pt
cd /tmp
git clone https://github.com/alibaba-damo-academy/FunASR
cd FunASR/runtime/python/onnxruntime
cat >export-onnx.py <<EOF
from funasr_onnx import CT_Transformer
model_dir = "/tmp/punc_ct-transformer_zh-cn-common-vocab272727-pytorch" # model = CT_Transformer(model_dir, quantize=True) model = CT_Transformer(model_dir)
EOF
chmod +x export-onnx.py
./export-onnx.py
```
You will find the exported
`model.onnx` file inside
`/tmp/models/punc_ct-transformer_zh-cn-common-vocab272727-pytorch`.
Now you can use ./add-model-metadata.py in this repo to add metadata to the generated
`model.onnx`.
You can use
```
from onnxruntime.quantization import QuantType, quantize_dynamic
quantize_dynamic(
model_input="./model.onnx",
model_output="./model.int8.onnx",
weight_type=QuantType.QUInt8,
)
```
to get the `int8` quantized model
```
-rw-r--r-- 1 fangjun staff 72M Jun 18 10:33 model.int8.onnx
-rw-r--r-- 1 fangjun staff 281M Apr 12 2024 model.onnx
```

View File

@ -1,46 +0,0 @@
model: CTTransformer
model_conf:
ignore_id: 0
embed_unit: 256
att_unit: 256
dropout_rate: 0.1
punc_list:
- <unk>
- _
- ,
- 。
- ?
- 、
punc_weight:
- 1.0
- 1.0
- 1.0
- 1.0
- 1.0
- 1.0
sentence_end_id: 3
encoder: SANMEncoder
encoder_conf:
input_size: 256
output_size: 256
attention_heads: 8
linear_units: 1024
num_blocks: 4
dropout_rate: 0.1
positional_dropout_rate: 0.1
attention_dropout_rate: 0.0
input_layer: pe
pos_enc_class: SinusoidalPositionEncoder
normalize_before: true
kernel_size: 11
sanm_shfit: 0
selfattention_layer_type: sanm
padding_idx: 0
tokenizer: CharTokenizer
tokenizer_conf:
unk_symbol: <unk>