Spaces:

p1atdev
/

danbooru-tags-transformer-v2

Running on Zero

p1atdev commited on May 1

Commit

48cea3d

•

1 Parent(s): b421feb

chore: strip before tokenize

Files changed (1) hide show

v2.py CHANGED Viewed

@@ -55,7 +55,7 @@ def normalize_tags(tokenizer: PreTrainedTokenizerBase, tags: str):
             [
                 token
                 for token in tokenizer.encode_plus(
-                    tags,
                     return_tensors="pt",
                 ).input_ids[0]
                 if int(token) != tokenizer.unk_token_id

             [
                 token
                 for token in tokenizer.encode_plus(
+                    tags.strip(),
                     return_tensors="pt",
                 ).input_ids[0]
                 if int(token) != tokenizer.unk_token_id