nancyH/supp_tokenizer_bench

Supplementary: tokenizer characterization benchmark

Code, data, and results behind Supplementary Figure 2:

• (a) motif preservation — perfect match rate and motif fragmentation
• (b) Jensen–Shannon distance between functional-region token distributions and a matched random genomic background

Tokenizers compared: DNABERT-2 BPE, Grover BPE, cCRE-guided BPE, motif-guided BPE, DNAMotifTokenizer.

Layout

Shared

• tokenizers/ — all five tokenizers used in both panels
  • DNAbert2_tokenizer.json, Grover_tokenizer.json — original DNABERT-2 / Grover tokenizers
  • cCRE_region_BPE_tokenizer.json, motif_region_BPE_tokenizer.json
  • DNAMotifTokenizer/ — trie, motif list, vocab (loaded via code/dna_motif_tokenizer.py, seed = 42)
• code/dna_motif_tokenizer.py — wrapper exposing DNAMotifTokenizer through a tokenizers-like interface

Panel a — motif preservation

• data/motifs.txt — 904 unique motifs; wildcard (*) motifs are expanded to all A/C/G/T variants (≤5 wildcards), giving 3,737 test sequences of length 5–12
• code/motif_coverage_eval.py — tokenizes each variant (no special tokens) and computes per-tokenizer metrics
• results/summary_by_tokenizer.csv — per-tokenizer metrics plotted in the figure
• results/motif_variant_results.csv, results/consistency_by_motif.csv — per-variant and per-motif detail
• code/plot_supp_fig2a_motif.py → figures/supp_fig2a_motif.{pdf,png}

Metrics:

• Perfect match rate — % of motif variants encoded as exactly one token
• Avg. tokens per motif — mean number of tokens per motif variant (lower = less fragmentation)

Panel b — region vs random-background JS distance

• data/regions/ — promoter (2 kb), enhancer, and exon interval sets (merged BED)
• data/background/ — chrom.sizes (canonical hg38 chromosomes), region_union.bed (union of the three region sets, used as exclusion), random.bed (exon intervals shuffled into the non-region genome with bedtools shuffle -seed 42, so total bp matches exon)
• data/fastas/ — hg38 sequences for promoter, enhancer, exon, and random (bedtools getfasta)
• code/js_vs_random_heatmap.py — builds the background, tokenizes each FASTA, and computes the Jensen–Shannon distance (base 2) between each region's token-frequency distribution and the random background
• results/js_vs_random_matrix.csv — 5 tokenizers × 3 regions matrix plotted in the figure
• results/js_vs_random.log — run log with token counts per region and tokenizer
• figures/js_vs_random_heatmap.{pdf,png} — heatmap from js_vs_random_heatmap.py
• code/plot_combined_panel_a_b.py → figures/combined_panel_a_b.{pdf,png} — combined draft of panels a and b

Reproduce

# Panel a
python code/motif_coverage_eval.py --motif-file data/motifs.txt --output-dir results
python code/plot_supp_fig2a_motif.py

# Panel b (requires bedtools and hg38.fa from UCSC)
python code/js_vs_random_heatmap.py              # full run
python code/js_vs_random_heatmap.py --plot-only  # redraw from results/js_vs_random_matrix.csv

motif_coverage_eval.py, js_vs_random_heatmap.py, and plot_combined_panel_a_b.py are uploaded as run and resolve paths relative to the original project layout (DEFAULT_TOKENIZER_PATHS, ROOT, REGION_BEDS, etc.); point them at tokenizers/ and data/ before running. The hg38 reference genome is not included (public, ~3 GB); the extracted FASTAs in data/fastas/ are what the analysis tokenized.