metadata

title: Tokenizer
emoji: 📚
colorFrom: green
colorTo: yellow
sdk: docker
pinned: false

Tokenizer API

A simple API for tokenizing text using spaCy.

Usage Examples

Using curl

curl -X POST "https://theirstory-tokenizer.hf.space/tokenize" \
     -H "Content-Type: application/json" \
     -H "accept: application/json" \
     -d '{"text": "Hello world"}'

Using Python

import requests

response = requests.post(
    "https://theirstory-tokenizer.hf.space/tokenize",
    json={"text": "Hello world"},
    headers={"accept": "application/json"}
)
tokens = response.json()
print(tokens)

Using JavaScript

const response = await fetch('https://theirstory-tokenizer.hf.space/tokenize', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
        'accept': 'application/json'
    },
    body: JSON.stringify({
        text: 'Hello world'
    })
});
const tokens = await response.json();
console.log(tokens);

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference