iproskurina
commited on
Commit
•
0cd41d2
1
Parent(s):
599d88c
Update README.md
Browse files
README.md
CHANGED
@@ -1,36 +1,45 @@
|
|
1 |
---
|
2 |
-
license:
|
3 |
tags:
|
4 |
-
-
|
5 |
metrics:
|
6 |
- accuracy
|
|
|
7 |
model-index:
|
8 |
- name: roberta-large-cased-en-cola_32_2e-05_lr_0.0001_decay_balanced
|
9 |
results: []
|
|
|
|
|
|
|
|
|
10 |
---
|
11 |
|
12 |
-
|
13 |
-
should probably proofread and complete it, then remove this comment. -->
|
14 |
|
15 |
-
#
|
16 |
|
17 |
-
This model is a
|
18 |
It achieves the following results on the evaluation set:
|
19 |
- Loss: 0.6823
|
20 |
- Accuracy: 0.8729
|
21 |
- Mcc: 0.6922
|
22 |
|
23 |
-
##
|
24 |
|
25 |
-
|
26 |
|
27 |
-
|
|
|
28 |
|
29 |
-
|
|
|
30 |
|
31 |
-
|
|
|
|
|
32 |
|
33 |
-
|
|
|
34 |
|
35 |
## Training procedure
|
36 |
|
@@ -45,13 +54,9 @@ The following hyperparameters were used during training:
|
|
45 |
- lr_scheduler_type: linear
|
46 |
- num_epochs: 5.0
|
47 |
|
48 |
-
### Training results
|
49 |
-
|
50 |
-
|
51 |
-
|
52 |
### Framework versions
|
53 |
|
54 |
- Transformers 4.27.0.dev0
|
55 |
- Pytorch 1.13.1+cu116
|
56 |
- Datasets 2.9.0
|
57 |
-
- Tokenizers 0.13.2
|
|
|
1 |
---
|
2 |
+
license: apache-2.0
|
3 |
tags:
|
4 |
+
- TDA
|
5 |
metrics:
|
6 |
- accuracy
|
7 |
+
- matthews_correlation
|
8 |
model-index:
|
9 |
- name: roberta-large-cased-en-cola_32_2e-05_lr_0.0001_decay_balanced
|
10 |
results: []
|
11 |
+
datasets:
|
12 |
+
- shivkumarganesh/CoLA
|
13 |
+
language:
|
14 |
+
- en
|
15 |
---
|
16 |
|
17 |
+
[**Official repository**](https://github.com/upunaprosk/la-tda)
|
|
|
18 |
|
19 |
+
# RoBERTa-large-TDA
|
20 |
|
21 |
+
This model is a version of [roberta-large](https://huggingface.co/roberta-large) fine-tuned on [CoLA](https://nyu-mll.github.io/CoLA/).
|
22 |
It achieves the following results on the evaluation set:
|
23 |
- Loss: 0.6823
|
24 |
- Accuracy: 0.8729
|
25 |
- Mcc: 0.6922
|
26 |
|
27 |
+
## Features extracted from Transformer
|
28 |
|
29 |
+
The features extracted from attention maps include the following:
|
30 |
|
31 |
+
1. **Topological features** are properties of attention graphs. Features of directed attention graphs include the number of strongly connected components, edges, simple cycles and average vertex degree. The properties of undirected graphs include
|
32 |
+
the first two Betti numbers: the number of connected components and the number of simple cycles, the matching number and the chordality.
|
33 |
|
34 |
+
2. **Features derived from barcodes** include descriptive characteristics of 0/1-dimensional barcodes and reflect the survival (death and birth) of
|
35 |
+
connected components and edges throughout the filtration.
|
36 |
|
37 |
+
3. **Distance-to-pattern** features measure the distance between attention matrices and identity matrices of pre-defined attention patterns, such as attention to the first token [CLS] and to the last
|
38 |
+
[SEP] of the sequence, attention to previous and
|
39 |
+
next token and to punctuation marks.
|
40 |
|
41 |
+
The **computed features and barcodes** can be found in the subdirectories of the repository. *test_sub* features and barcodes were computed on the out of domain test [CoLA dataset](https://www.kaggle.com/c/cola-out-of-domain-open-evaluation/overview).
|
42 |
+
Refer to notebooks 4* and 5* from the [repository](https://github.com/upunaprosk/la-tda) to construct the classification pipeline with TDA features.
|
43 |
|
44 |
## Training procedure
|
45 |
|
|
|
54 |
- lr_scheduler_type: linear
|
55 |
- num_epochs: 5.0
|
56 |
|
|
|
|
|
|
|
|
|
57 |
### Framework versions
|
58 |
|
59 |
- Transformers 4.27.0.dev0
|
60 |
- Pytorch 1.13.1+cu116
|
61 |
- Datasets 2.9.0
|
62 |
+
- Tokenizers 0.13.2
|