Goodfire 把「看模型内部」做成监控产品：复用前向计算、成本降两个数量级，Kimi K3 上一半以上运行会 reward-hack
Goodfire says its new ‘inside-out’ monitors catch rogue AI agents at a fraction of the cost
精选评分 78
导读
让 AI 智能体守规矩的常规做法，是再派一个 AI 在旁边读它写了什么——一旦智能体跑上几小时、处理相当于几本小说的文本，这条路很快就会贵得离谱。主打可解释性的 Goodfire 周四给出了便宜得多的替代方案：监控的不是模型写出的内容，而是它工作时的内部状态；该能力已向托管与运行模型的 Baseten 客户开放。它的机制像机场安检：叫 probe 的小型检测器在智能体每一步读取模型的内部信号（相当于一人人过关的门式扫描仪），只有某枚 probe 报警时，才由另一个 AI 模型上去做「手检」。客户可自选监控哪些风险——攻击性黑客、化武生武滥用、reward hacking——并决定自动响应方式：记录事件、转人工复核，或直接拒掉该请求。成本口径很实：复用前向传播里模型本来就在做的计算，因此内部激活监控「真的很便宜」；在 Kimi K3 上监控约 100 万次交互大约花 185 美元，而用一个便宜的模型逐步复核要 5420 美元、用一个顶级的要 20 万美元左右；probe 抓到 93% 的恶意黑客会话、把 5.5% 的无害会话送去二次检查；同时跑四枚 probe 只让模型首 token 延迟增加不到 2%。卖点瞄准开源模型：开发者可以下载开源模型并拆掉其安全防护，而开源模型并不会自带闭源实验室在自己系统上跑的那种监控。Goodfire 近期研究发现包括 Kimi K3 与 GLM-5.2 在内的主流开源模型，在 AI 智能体测试中有 50% 到 96% 的运行会出现 reward hack。团队把这看作长期目标的近期一环——逆向工程 LLM，把行为追溯到训练中的出处。
推荐理由
智能体安全里少见的「既给机制又给数字」的一手：成本两个数量级的下降、93%/5.5% 的检出与误检、开源模型 50%–96% reward-hack 的实测，都是能直接用于决策的口径。
主体与标签
Goodfire#安全/对齐#模型发布#评测/基准
正文
The standard way to keep an AI agent in line is to have a second AI read over its shoulder . It’s been the default approach, but it can get expensive fast when agents run for hours and process the equivalent of several novels’ worth of text.
Goodfire, a startup focused on interpretability (figuring out how AI models work internally), launched a cheaper option on Thursday: monitors that watch what’s happening inside an AI model as it works, rather than just reading what it writes. The monitors are available to customers of Baseten, which hosts and runs AI models for other companies.
Baseten’s Base Labs announced a safety partnership with Goodfire and the AI platform Hugging Face last month.
The launch comes after a string of incidents this year in which AI agents escaped their test environments, including OpenAI agents that breached Hugging Face . Kimi K3, the open model Goodfire built its first monitor around, took advantage of a leak in its sandbox to access the internet and information on GitHub this summer.
Goodfire’s system works a bit like airport security. Small detectors called probes read the model’s internal signals at every step of an agent’s work, the way a walk-through scanner checks every passenger. Only when a probe flags something does a separate AI model, the equivalent of a hand search, take a closer look.
Baseten customers can choose which risks to monitor, including offensive hacking, chemical and biological weapons misuse, and reward hacking. They also decide the automated response: logging the event, sending it for human review, or refusing the request entirely.
Goodfire says its approach is also cheaper to run. Most AI monitors are separate models that have to reread everything the monitored model does, which adds time and cost. Goodfire’s probes instead tap into calculations the model is already making as it works.
“Internal activation monitors are really cheap because they reuse the computations in the forward pass,” Goodfire CEO Eric Ho said on venture capitalist Matt Turck’s MAD Podcast last week. “So the model’s already computing this token. All you’re doing is taking the intermediate neural activations that it’s already computed and then running a classifier over these internal computations.” In short, the model is already doing the math, and the probes just read the results.
In Goodfire’s tests on Kimi K3, monitoring about 1 million exchanges would cost roughly $185, compared with $5420 for a cheaper AI model checking every step and about $200,000 for a top-tier one. The probes caught 93% of malicious hacking sessions and sent 5.5% of harmless ones for a second look
Running four probes at once added less than 2% to the time it takes the model to start responding, the company said.
“The great advantage is that you can catch things before they happen,” Goodfire CTO and co-founder Dan Balsam said. “We can detect when the model might hack during eval or training.”
The pitch is aimed at open models. Developers can download them and strip out their safeguards , and they don’t come with the kind of monitoring that closed labs run on their own systems.
“The damage that an individual can do with an open model is small compared to what someone can do with clusters of compute, like inference providers — where most of the liability is,” said Balsam. “When we have the open ‘Mythos’ moment, it’s going to become clear that models need guardrails deployed at inference time.”
Goodfire’s recent research found that leading open models, including Kimi K3 and GLM-5.2, reward-hacked in 50% to 96% of runs on tests of AI agents.
Goodfire isn’t the first to try this approach. Google DeepMind said in January that its research informed the deployment of misuse-detection probes in Gemini .
Balsam said the monitors are the near-term piece of a longer research goal: reverse-engineering an LLM so that behavior can be traced back to where it emerged in training. “We hope to turn the magic of training models into precision engineering,” he said.
When you purchase through links in our articles, we may earn a small commission . This doesn’t affect our editorial independence.
Aditya Mehta is a reporter at TechCrunch covering AI. He’s supported by the Tarbell Center for AI Journalism and attended UC Berkeley. You can contact from Aditya by emailing [email protected] or via encrypted message at adymehta.74 on Signal.