mhenrichsen
commited on
Commit
•
218eab5
1
Parent(s):
f8946de
Update README.md
Browse files
README.md
CHANGED
@@ -6,6 +6,10 @@ tags:
|
|
6 |
model-index:
|
7 |
- name: storage/context
|
8 |
results: []
|
|
|
|
|
|
|
|
|
9 |
---
|
10 |
|
11 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
@@ -19,18 +23,28 @@ It achieves the following results on the evaluation set:
|
|
19 |
- Loss: 0.0253
|
20 |
|
21 |
## Model description
|
|
|
|
|
|
|
22 |
|
23 |
-
|
|
|
24 |
|
25 |
-
|
|
|
26 |
|
27 |
-
|
|
|
28 |
|
29 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
30 |
|
31 |
-
More information needed
|
32 |
|
33 |
-
## Training procedure
|
34 |
|
35 |
### Training hyperparameters
|
36 |
|
@@ -77,4 +91,4 @@ The following hyperparameters were used during training:
|
|
77 |
- Transformers 4.35.2
|
78 |
- Pytorch 2.0.1+cu118
|
79 |
- Datasets 2.15.0
|
80 |
-
- Tokenizers 0.15.0
|
|
|
6 |
model-index:
|
7 |
- name: storage/context
|
8 |
results: []
|
9 |
+
datasets:
|
10 |
+
- mhenrichsen/context-aware-splits-english
|
11 |
+
language:
|
12 |
+
- en
|
13 |
---
|
14 |
|
15 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
|
|
23 |
- Loss: 0.0253
|
24 |
|
25 |
## Model description
|
26 |
+
- This model is used to split texts in a context aware way. Used for RAG applications.
|
27 |
+
- This model is an adapter for Mistral 7b.
|
28 |
+
It uses the Alpaca format:
|
29 |
|
30 |
+
```
|
31 |
+
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
|
32 |
|
33 |
+
### Instruction:
|
34 |
+
Your task is to segment text into smaller blocks. Split the text where it makes sense and be vary of the context. The ideal split should be close to {WORD_COUNT} words.
|
35 |
|
36 |
+
### Input:
|
37 |
+
Q: Information/File Manager I'm looking for a file manager application which helps to organize a large amount of movies, pictures, music, text documents, databases, audio-books and ebooks. Right now I only use the Finder which doesn't work well, because I really need a function to put single files into multiple categories. Simply using the file system for this creates a confusing nesting of files. A: Depending on the number of categories you require to handle, you could always use a combination of the finder with the built in label functionality, thus a movie can be held in one area (movies directory, for example), but "tagged" as something else. Using smart directories and saved searches you can view your files by a combination of the attributes (location, label, media type) to create custom views. All without purchasing software. Cheap and cheerful, but may be suitable to your needs. A: Maybe use a file manager that supports Open Meta. Or use symbolic links for organizing all your media files. Or even use hardlinked files if you dare.
|
38 |
|
39 |
+
### Response:
|
40 |
+
```
|
41 |
+
|
42 |
+
Response:
|
43 |
+
```
|
44 |
+
{'splits': ["Q: Information/File Manager I'm looking for a file manager application which helps to organize a large amount of movies, pictures, music, text documents, databases, audio-books and ebooks. Right now I only use the Finder which doesn't work well, because I really need a function to put single files into multiple categories. Simply using the file system for this creates a confusing nesting of files.", 'A: Depending on the number of categories you require to handle, you could always use a combination of the finder with the built in label functionality, thus a movie can be held in one area (movies directory, for example), but "tagged" as something else. Using smart directories and saved searches you can view your files by a combination of the attributes (location, label, media type) to create custom views. All without purchasing software. Cheap and cheerful, but may be suitable to your needs.', 'A: Maybe use a file manager that supports Open Meta. Or use symbolic links for organizing all your media files. Or even use hardlinked files if you dare.'], 'topic': 'Discussion on file manager applications for organizing large amount of media files.'}
|
45 |
+
```
|
46 |
|
|
|
47 |
|
|
|
48 |
|
49 |
### Training hyperparameters
|
50 |
|
|
|
91 |
- Transformers 4.35.2
|
92 |
- Pytorch 2.0.1+cu118
|
93 |
- Datasets 2.15.0
|
94 |
+
- Tokenizers 0.15.0
|