GPT-3.5-turbo and GPT-4 tokenizer

Описание: Tokenizer: https://www.typeblock.co/resources/to...
Logit Bias: https://help.openai.com/en/articles/5...

For the longest time I haven't been able to find a tokenizer online similar to the one for GPT-3 for other models. So I made one and put it up online for free for anyone.

It's built on top of the tiktoken library and is basically just a lambda function in the backend.

You could just download the tiktoken library directly but I hope this saves time if you need the token ids for logit bias.