iproskurina commited on
Commit
290c499
1 Parent(s): 90e0d60

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +21 -16
README.md CHANGED
@@ -1,36 +1,45 @@
1
  ---
2
  license: apache-2.0
3
  tags:
4
- - generated_from_trainer
5
  metrics:
6
  - accuracy
 
7
  model-index:
8
  - name: bert-base-cased-en-cola_32_3e-05_lr_0.01_decay_balanced
9
  results: []
 
 
 
 
10
  ---
11
 
12
- <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
- should probably proofread and complete it, then remove this comment. -->
14
 
15
- # bert-base-cased-en-cola_32_3e-05_lr_0.01_decay_balanced
16
 
17
- This model is a fine-tuned version of [bert-base-cased](https://huggingface.co/bert-base-cased) on an unknown dataset.
18
  It achieves the following results on the evaluation set:
19
  - Loss: 0.6809
20
  - Accuracy: 0.8501
21
  - Mcc: 0.6337
22
 
23
- ## Model description
24
 
25
- More information needed
26
 
27
- ## Intended uses & limitations
 
28
 
29
- More information needed
 
30
 
31
- ## Training and evaluation data
 
 
32
 
33
- More information needed
 
34
 
35
  ## Training procedure
36
 
@@ -45,13 +54,9 @@ The following hyperparameters were used during training:
45
  - lr_scheduler_type: linear
46
  - num_epochs: 3.0
47
 
48
- ### Training results
49
-
50
-
51
-
52
  ### Framework versions
53
 
54
  - Transformers 4.23.0.dev0
55
  - Pytorch 1.12.1+cu113
56
  - Datasets 2.5.1
57
- - Tokenizers 0.13.0
 
1
  ---
2
  license: apache-2.0
3
  tags:
4
+ - TDA
5
  metrics:
6
  - accuracy
7
+ - matthews_correlation
8
  model-index:
9
  - name: bert-base-cased-en-cola_32_3e-05_lr_0.01_decay_balanced
10
  results: []
11
+ datasets:
12
+ - shivkumarganesh/CoLA
13
+ language:
14
+ - en
15
  ---
16
 
17
+ [**Official repository**](https://github.com/upunaprosk/la-tda)
 
18
 
19
+ # BERT-TDA
20
 
21
+ This model is a version of [bert-base-cased](https://huggingface.co/bert-base-cased) fine-tuned on [CoLA](https://nyu-mll.github.io/CoLA/).
22
  It achieves the following results on the evaluation set:
23
  - Loss: 0.6809
24
  - Accuracy: 0.8501
25
  - Mcc: 0.6337
26
 
27
+ ## Features extracted from Transformer
28
 
29
+ The features extracted from attention maps include the following:
30
 
31
+ 1. **Topological features** are properties of attention graphs. Features of directed attention graphs include the number of strongly connected components, edges, simple cycles and average vertex degree. The properties of undirected graphs include
32
+ the first two Betti numbers: the number of connected components and the number of simple cycles, the matching number and the chordality.
33
 
34
+ 2. **Features derived from barcodes** include descriptive characteristics of 0/1-dimensional barcodes and reflect the survival (death and birth) of
35
+ connected components and edges throughout the filtration.
36
 
37
+ 3. **Distance-to-pattern** features measure the distance between attention matrices and identity matrices of pre-defined attention patterns, such as attention to the first token [CLS] and to the last
38
+ [SEP] of the sequence, attention to previous and
39
+ next token and to punctuation marks.
40
 
41
+ The computed features and barcodes can be found in the subdirectories of the repository. *test_sub* features and barcodes were computed on the out of domain test [CoLA dataset](https://www.kaggle.com/c/cola-out-of-domain-open-evaluation/overview).
42
+ Refer to notebooks 4* and 5* from the [repository](https://github.com/upunaprosk/la-tda) to construct the classification pipeline with TDA features.
43
 
44
  ## Training procedure
45
 
 
54
  - lr_scheduler_type: linear
55
  - num_epochs: 3.0
56
 
 
 
 
 
57
  ### Framework versions
58
 
59
  - Transformers 4.23.0.dev0
60
  - Pytorch 1.12.1+cu113
61
  - Datasets 2.5.1
62
+ - Tokenizers 0.13.0