Tokenize text with the loaded model's tokenizer
One step with Hugging Face. We can help you get access.
Look something up
Tokenize text with the loaded model's tokenizer
with Hugging Face
Use Hugging Face: its Hugging Face Text Generation Inference Info API lets you tokenize text with the loaded model's tokenizer in a single step (POST /tokenize). capabilities.work can help you get signed up and connected.
- How many tokens does a piece of text become with the model loaded on TGI?
- Can I see token IDs and their string pieces, with or without special tokens?
Say it to your AI assistant
- Tokenize {inputs} with the loaded model.
- Split {inputs} into tokens, setting add special tokens to {add_special_tokens}.
For your developer
POST /tokenize in Hugging Face Text Generation Inference Info API. Full spec on APIs.io.