Spaces:
Sleeping
Sleeping
Commit
•
21cb44b
1
Parent(s):
8e2190e
description example
Browse files
app.py
CHANGED
|
@@ -89,6 +89,9 @@ def update_dataset_card(
|
|
| 89 |
description = """
|
| 90 |
Corpus Creator is a tool designed to help you easily convert a collection of text files into a dataset suitable for various natural language processing (NLP) tasks.
|
| 91 |
In particular the app is focused on splitting texts into chunks of a specified size and overlap. This can be useful for preparing data for synthetic data generation, pipelines or annotation tasks.
|
|
|
|
|
|
|
|
|
|
| 92 |
The resulting text chunks are stored in a dataset that can be previewed and uploaded to the Hugging Face Hub for easy sharing and access by the community.
|
| 93 |
The chunking is done using `Llama-index`'s [`SentenceSplitter`](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/?h=sentencesplitter#sentencesplitter) classes.
|
| 94 |
|
|
|
|
| 89 |
description = """
|
| 90 |
Corpus Creator is a tool designed to help you easily convert a collection of text files into a dataset suitable for various natural language processing (NLP) tasks.
|
| 91 |
In particular the app is focused on splitting texts into chunks of a specified size and overlap. This can be useful for preparing data for synthetic data generation, pipelines or annotation tasks.
|
| 92 |
+
|
| 93 |
+
See an [example dataset](davanstrien/MOH-Bethnal-Green) created using this tool starting from a collection of plain text files.
|
| 94 |
+
|
| 95 |
The resulting text chunks are stored in a dataset that can be previewed and uploaded to the Hugging Face Hub for easy sharing and access by the community.
|
| 96 |
The chunking is done using `Llama-index`'s [`SentenceSplitter`](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/?h=sentencesplitter#sentencesplitter) classes.
|
| 97 |
|