semvec/pypi-packages huggingface dataset:
KenkoAdvance 1.0.0
自用的一些功能
License: MIT
Requires Python: >=3.9

# KenkoAdvance

一些方便自己的自定义小功能

作者：AkagiYui

Work In Progress:
- [ ] AyBase64
- [ ] AyAdb

## `类` GocqConnection

描述：go-cqhttp连接类

需要的包：websocket-client requests

`from KenkoAdvance.GocqConnection import GocqConnection`

## `类` TextToMidi

描述：文本转Midi

作者：Scarlett凛子 `QQ: 1587524`

需要的包：mido

`from KenkoAdvance.TextToMidi import TextToMidi`


## `类` HiddenPrints

描述：暂时全局关闭标准输出

收录自：https://zhuanlan.zhihu.com/p/360137954


## `类` AyStr

描述：自定义字符串类

> 自定义字符串类 
> 
> 重写in支持判断列表或元组成员是否出现在字符串里 
> 
> 重写find()支持查找子串列表或元组


## `类` AyDict

描述：自定义字典类

> 键用半角实心点.做下级入口
> 
> 例 dict['web']['port'] == AyDict['web.port']


## `函数` get_file_format

描述：用文件头判断文件格式

收录自：https://blog.csdn.net/privateobject/article/details/78069500


## `函数` int_to_chinese

描述：数字转汉字

收录自：https://blog.csdn.net/PlusChang/article/details/72991191


## `函数` get_self_uptime

描述：获取自身运行时长

需要的包：psutil

## `函数` get_machine_uptime

描述：获取开机时长

需要的包：psutil

## `函数` get_self_ip

描述：获取自身IP

收录自：https://www.zhihu.com/question/49036683/answer/1243217025

## `函数` chinese_to_int

描述：汉字转数字

收录自：https://www.iplaypy.com/code/base/b2600.html

## `函数` chinese_to_int_e

描述：汉字转数字

收录自：https://www.jb51.net/article/114789.htm
corpus-statistics 0.1.1
A spaCy pipeline component for counting tokens a pipeline has seen.
License: MIT
Requires Python: >=3.8,<3.11

# Corpus Statistics: A Very Basic (for now) spaCy Pipeline Component

If you want to know what tokens your spaCy pipeline has seen, this is the component for you.

```bash
pip install corpus-statistics
# OR for latest
pip install git+https://github.com/pmbaumgartner/corpus_statistics
```

⚡️ **Example**

```python
from spacy.lang.en import English

# Use some example data
from datasets import load_dataset
dataset = load_dataset("imdb")
texts = dataset["train"]["text"]

nlp = English()  # or spacy.load('a_model')
nlp.add_pipe("simple_corpus_stats")  

# ✨ start the magic 
for doc in nlp.pipe(texts):
    # ➡️ do your pipeline stuff! ➡️
    pass

corpus_stats = nlp.get_pipe("simple_corpus_stats")

# check if a token has been processed through this pipeline
token = "apple"
if token in corpus_stats:
    token_count = corpus_stats[token]
    print(f"'{token}' mentioned {token_count} times")

# 'apple' mentioned 24 times
```

It's got all your favorite [legomena](https://en.wikipedia.org/wiki/Hapax_legomenon) like `hapax` and `dis`.

```python
only_seen_once = len(corpus_stats.hapax_legomena)
percent_of_vocab = only_seen_once / corpus_stats.vocab_size
print(f"{percent_of_vocab*100:.1f}% tokens only occurred once.")
# 47.6% tokens only occurred once.

only_seen_twice = len(corpus_stats.dis_legomena)
percent_of_vocab_2x = only_seen_twice / corpus_stats.vocab_size
print(f"{percent_of_vocab_2x*100:.1f}% tokens occurred twice.")
# 12.3% tokens occurred twice.
```

We counted some things too:

```python
# corpus_stats.vocabulary is a collections.Counter 🔢
print(*corpus_stats.vocabulary.most_common(5), sep="\n")
# ('the', 289838)
# (',', 275296)
# ('.', 236702)
# ('and', 156484)
# ('a', 156282)

mean_doc_length = sum(corpus_stats.doc_lengths) / corpus_stats.corpus_length
print(f"Mean doc length: {mean_doc_length:.1f}")
# Mean doc length: 272.5
```

# Use in Model Training and Config Files

This can be quite helpful if you wanted to know what tokens were seen in your training data. You can include this component in your training config as follows.

```
...
[nlp]
lang = "en"
pipeline = ["simple_corpus_stats", ...]
...

[components]

[components.simple_corpus_stats]
factory = "simple_corpus_stats"
n_train = 1000  # This is important! See below
```

⚠️ 🔁 If you use this component in a training config, your pipeline will see the same docs multiple times, due to the number of training epochs and evaluation steps, so the vocab counter will be incorrect. To correct for this, you need to specify the number of examples in your training dataset as the `n_train` config parameter. 

```python
import spacy

nlp = spacy.load("your_trained_model")
corpus_stats = nlp.get_pipe("simple_corpus_stats")

assert min(corpus_stats.vocabulary.values()) == 1

# value from config
assert len(corpus_stats.doc_lengths) == 1000
```