Choose Rich Live
Tech

OpenAI embeds outside safety labs to evaluate internal models and safeguards

OpenAI has signed agreements with Apollo Research, METR, Transluce, and Goodfire to grant independent researchers deep access to its internal systems for alignment testing and safety assessments.

Portrait of Sam Altman speaking on stage with a headset microphone.
Image: @Cointelegraph

OpenAI announced agreements with four independent evaluation groups—Apollo Research, METR, Transluce, and Goodfire—for the initial phase of its embedded assessments program. The initiative grants external researchers deep access to OpenAI's internal systems and models to scrutinize safeguards and investigate alignment risks.[5][2][1][4]

Each organization has been assigned a specific testing focus. Six Apollo Research evaluators will embed to examine whether OpenAI models exhibit subversive behavior or secretly work against developers, while evaluating whether monitoring mechanisms detect it. METR will stress-test monitoring tools and look for misaligned behavior in OpenAI's largest reinforcement learning training run to date. Transluce is set to audit post-training alignment evaluations with an extended follow-up planned for early 2027, and Goodfire will embed interpretability researchers to inspect internal model activity during the training process.[3][6]

The partner labs will be permitted to publish their results, though OpenAI retains the authority to redact confidential information. TechCrunch reported that limits on evaluator access and disclosures have prompted questions regarding their overall independence. Additionally, OpenAI reportedly explored acquiring one of the outside safety organizations for more than $5 billion.[3][6]

Key facts

  • OpenAI signed agreements with Apollo Research, METR, Transluce, and Goodfire for the first phase of an embedded safety assessment program.
  • Six Apollo Research evaluators will embed to assess whether OpenAI models could engage in subversive behavior or work against developers.
  • METR will search for misaligned behavior during OpenAI's largest reinforcement learning run to date and stress-test its monitoring systems.
  • Transluce will audit OpenAI's post-training alignment evaluations, with a follow-up assessment scheduled for early 2027.
  • Goodfire will place interpretability researchers inside OpenAI to analyze model activity during training.
  • The evaluation labs are permitted to publish their findings, but OpenAI holds the right to redact confidential information.
  • Restrictions on evaluator access and disclosure have raised concerns about independence, according to TechCrunch.
  • OpenAI reportedly explored acquiring one of the outside safety labs for more than $5 billion.

Sources · 6 sources

  1. CO

    Cointelegraph@CointelegraphPost on X ·

    🤖 JUST IN: OpenAI has signed agreements with four independent evaluators, Apollo Research, METR, Transluce and Goodfire, to assess AI model safety and safeguards. https://t.co/F1CJushR6H

    Open source
  2. TE

    TechmemeArticle ·

    OpenAI says it signed agreements with independent assessors Apollo Research, METR, Transluce, and Goodfire for its first phase of AI model embedded assessments (OpenAI) OpenAI : OpenAI says it signed agreements with independent assessors Apollo Research, METR, Transluce, and Goodfire for its first phase of AI model embedded assessments — We have signed agreements with four independent assessors—Apollo Research, METR, Transluce, and Goodfire—for our first phase of embedded assessments.

    Open source
  3. TE

    TechCrunch@TechCrunchPost on X ·

    OpenAI will embed six Apollo Research evaluators to investigate whether its AI models engage in subversive behavior, but restrictions on access and disclosure raise questions about their independence. https://t.co/BLCoJ1AtHY

    Open source
  4. CN

    CNBC@CNBCPost on X ·

    OpenAI inks agreements with four independent evaluators to boost company's safety work https://t.co/Bwe4ATOKX8

    Open source
  5. OP

    OpenAI@OpenAIPost on X ·

    We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. https://t.co/SNAAG0uHmp

    Open source
  6. MT

    MTS@MTSlivePost on X ·

    SITUATION EXPLAINED: OpenAI is letting 4 outside safety labs inside to test its models, and reportedly explored buying one of them for $5B+. • Apollo will test whether OpenAI's models could secretly work against their developers, and whether OpenAI's safeguards would catch it • METR will look for misaligned behavior in OpenAI's largest RL training run to date and stress-test its monitoring systems • Transluce will audit OpenAI's own post-training alignment evals, with a longer follow-up planned for early 2027 • Goodfire will embed interpretability researchers to analyze what's happening inside OpenAI's models during training • The labs can publish their findings, but OpenAI can redact confidential information @theojaffee: "OpenAI will be working with some embedded evaluators, which is very exciting. I think it's an optimistic sign for the future of alignment."

    Open source