ttok 升到 1.0：默认 tokenizer 从 GPT-4 换成 GPT-5/GPT-6——OpenAI 尚未官方确认同 tokenizer，William Liu 的 31 组 fixture 实验显示 44,794 tokens 全等
ttok 1.0
评分 38
导读
作者发完 0.4 升级后发现默认竟还用 GPT-4 tokenizer，于是切换默认并顺势出 1.0。OpenAI 未确认 GPT-6 是否沿用 GPT-5 系 tokenizer（issue 区吵得凶），但 William Liu 的一个 commit 记录了实验：七款 GPT 模型（5.5、5.6 Sol/Terra/Luna、6 Astra/Sol/Luna）在全部 31 组 fixture 上均报 44,794 tokens、彼此完全一致——GPT-6 在这批语料上不改变输入计数。
主体与标签
Simon WillisonOpenAI#开源/仓库#tokenization
正文
I released ttok 0.4 , ran uv tool upgrade ttok , piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!
I figured switching the default was a reasonable excuse to finally ship a 1.0.
OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:
All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.
Tags: projects , ai , openai , generative-ai , llms , tokenization