hamzagorgulu's picture
Upload tokenizer
4a49b27
raw
history blame
1 kB
#version: 0.2 - Trained by `huggingface/tokenizers`
Ġ 4
0 0
Ġ4 7
A N
AN N
A L
Ġ4 8
T I
P AL
00 7
00 3
0 3
L I
T AL
F LI
A B
00 2
03 2
1 5
0 1
15 03
A I
7 0
8 70
01 4
U A
03 9
P I
01 0
C D
0 8
6 08
S I
0 2
9 00
8 02
U SI
LI C
F I
F S
FS L
01 9
15 1
00 1
151 2
4 7
FI C
X L
1 8
X A
SI O
Ġ SIO
03 4
4 8
15 0
15 2
5 0
8 50
15 5
155 5
AB C
0 4
01 3
PI C
15 3
0 15
153 5
1 7
152 7
A H
H A
TI C
150 5
01 6
AI C
1 001
D I
P DI
152 2
02 7
04 3
F D
03 7
00 5
A AH
01 2
150 7
00 9
T AH
17 1
150 6
04 2
4 18
00 4
17 6
AAH H
171 0
15 02
4 1
150 4
03 6
152 0
176 9
C E
M S
S O
CE MS
A M
C C
M CC
P S
P MCC
01 1
AM PS
2 003
151 8
H H
02 2
04 1
155 1
15 15
41 4
F FIC
150 9
41 6
15 01
153 3
17 01
B AL
02 1
C I
H CI
00 6
AMPS A
152 1
AMPS C
B A
2 021
F AL
151 0
151 3
AMPS B
FAL L
1 003
1 70
1 005
T X
15 6
15 9
170 5
TX I
151 6
153 4
176 0
15 08
152 3
152 9
F F
151 1
151 4
156 7
159 4
FF X
03 5
02 0
1 2
4 0
15 40
00 8
02 4
171 1
2 4
6 18
8 08
152 5
04 6
17 24
156 3
159 2
1 6
2 02
3 1
3 4
17 34
16 31
202 5
03 8