OpenAI o1式思维链，开源模型也可以有，成功案例来了

北京时间 9 月 13 日午夜，OpenAI 发布了推理性能强大的 ο1 系列模型。之后，各路研究者一直在尝试挖掘 ο1 卓越性能背后的技术并尝试复现它。当然，OpenAI 也想了一些方法来抑制窥探，比如有多名用户声称曾试图诱导 ο1 模型公布其思维过程，然后收到了 OpenAI 的封号威胁。

尽管如此，不过三四天时间，就已经有研究者宣称已经成功复现/开发出了与 ο1 性能差不多的推理技术，并且还不止一个！

Llamaberry：教会 AI 像聪明人一样思考

Llamaberry 的提出者是 Martin Bowling。他开发的项目包括 RAGMiner.dev 和 Replit；其中前者可以帮助用户毫不费力地将网站转换成 Markdown、XML 或 JSON 等格式以便 RAG 和 LLM 应用使用，而后者则是一个使用 AI 将想法变成代码的项目。

Llamaberry 的核心思路是使用思维链（CoT）来实现推理。这个名字自然源自代表 o1 模型的 Strawberry（草莓）。

HuggingFace地址：https://huggingface.co/spaces/martinbowling/Llamaberry

什么是思维链？Bowling 在博客中打了个比方：「思维链推理就像是给 AI 一个笔记本来展示其工作过程。其中不仅仅是简单地给出答案，而是会带领我们经历其思维过程。」

Llamaberry 能教会 AI 透彻地思考，就像是一位人类专家攻克难题时那样。

具体来说，Llamaberry 是一个多轮思维链推理系统的实现，其基于运行在 Groq 上的 Llama 3.1 70B 模型。

多轮推理是关键

多轮推理，顾名思义，就是让模型在给出答案之前进行多步思考，而不是一步给出答案。打个比方，这就像是看一位大厨从备菜到完成摆盘一步步地完成一道精美菜肴，而不是直接微波加热预制菜。

举个示例：

第 1 轮：AI 先尝试解决当前问题。

第 2 轮：AI 回顾第一次尝试并尽力改进或优化其思维过程。

第 3 轮：再进行一轮反思和改进。

综合结果：最后，将所有这些思考综合到一起，得到一个连贯且合理的答案。

下面展示了这个多轮过程的示意图：

可以看到，前一轮的输出会成为后一轮的输入，从而让 AI 可在每个阶段不断完善其思维。最后，所有这些思考会凝练成一个合理的最终答案。就像看着一枚莓果逐渐成熟！

如何实现

下面将深入 Llamaberry 的实现细节。

1.设置舞台

首先，我们需要为 AI 助手设置一些基本规则，代码如下：

initial_system_prompt = “””You are an AI assistant capable of detailed, step-by-step thinking. When presented with a question or problem, break down your thought process into clear, logical steps. For each step, explain your reasoning. Conclude with a final answer. Use the following markdown structure:

## Reasoning

1. [First step]

  **Explanation:** [Detailed explanation of this step]

2. [Second step]

  **Explanation:** [Detailed explanation of this step]

…

## Answer

[Final answer]

Be comprehensive and show your reasoning clearly.”””

这就是提供给 AI 大厨的菜谱。它知道需要逐步分解其思考过程并解释每个步骤，并且以 Markdown 格式将它们显示出来。

2.思考过程

在每一轮推理中，都需要让 AI 对问题进行思考。但在第一轮结束后，还需要求它思考之前已经思考过的东西。这就像问朋友，「嘿，还记得你之前说过什么吗？让我们再想一想。」

下面是每一轮的生成方式：

async def generate_turn(query: str, previous_turns: list = None) -> str:

  is_first_turn = previous_turns is None or len(previous_turns) == 0

if is_first_turn:

    messages = [{

      “role”: “system”,

      “content”: initial_system_prompt

    }, {

      “role”: “user”,

      “content”: query

    }]

  else:

    previous_content = “\n\n”.join(previous_turns)

    messages = [{

      “role”: “system”,

      “content”: followup_system_prompt

    }, {

      “role”:

      “user”,

      “content”:

      f”Original Query: {query}\n\nPrevious Turns:\n{previous_content}\n\nProvide the next turn of reasoning.”

    }]

  return await call_llm(messages)

该函数是多轮推理的核心。它决定我们是在第一轮还是在后续轮，并以此构建提示词。对于后续轮，它会将之前所有推理轮都包含在上下文中，让 AI 可以基于之前的思考进行构建。

3.综合结果

经过三轮思考之后，再让 AI 检视其所有思考并得出一个合理的最终答案。这就像是让专家讨论问题，然后编写最终报告。

async def synthesize_turns(query: str, turns: list) -> str:

  turns_text = “\n\n”.join(

    [f”Turn {i+1}:\n{turn}” for i, turn in enumerate(turns)])

  messages = [{

    “role”: “system”,

    “content”: synthesis_prompt

  }, {

    “role”:

    “user”,

    “content”:

    f”Original Query: {query}\n\nTurns of Reasoning:\n{turns_text}”

  }]

  return await call_llm(messages)

这个综合步骤至关重要，是 AI 综合所有思考内容凝练出最终答案的一步。

单轮推理 vs 多轮推理

Martin Bowling 在其博客中简单比较了单轮和多轮推理。

这里他采用了一个简单示例，即询问 AI：「人工智能对就业市场有什么潜在影响？」

单轮推理：

AI: Artificial intelligence will likely have significant impacts on the job market, including:

1. Automation of routine tasks

2. Creation of new job roles in AI development and maintenance

3. Potential job displacement in certain industries

4. Increased demand for skills in data analysis and machine learning

While AI may eliminate some jobs, it will also create new opportunities and increase productivity in many fields.

使用 Llamaberry 执行多轮推理：

第 1 轮：

AI: Let’s approach this step-by-step:

1. Identify current AI capabilities

  Explanation: AI can perform data analysis, pattern recognition, and automate routine tasks.

2. Analyze vulnerable job sectors

  Explanation: Manufacturing, customer service, and data entry are likely to be affected.

3. Consider new job opportunities

  Explanation: AI development, data science, and AI ethics consultation may see growth.

4. Examine potential economic effects

  Explanation: Increased productivity might lead to economic growth, but wealth distribution could be a challenge.

Initial Answer: AI will likely cause significant shifts in the job market, with both job losses in some sectors and new opportunities in others.

第 2 轮：

AI: Let’s refine our thinking:

1. Reassess the timeline of AI impact

  Explanation: The effects won’t be immediate but will unfold over years or decades.

2. Consider the adaptability of the workforce

  Explanation: Retraining and education will be crucial for workers to transition to new roles.

3. Analyze potential societal changes

  Explanation: The nature of work itself might change, with more emphasis on creativity and emotional intelligence.

4. Examine global differences

  Explanation: AI’s impact will vary across different countries and economies.

Refined Answer: AI’s impact on the job market will be complex and gradual, requiring proactive measures in education and economic policy to ensure a smooth transition.

第 3 轮：

AI: Let’s delve deeper:

1. Explore the concept of human-AI collaboration

  Explanation: Many jobs may evolve to involve working alongside AI rather than being replaced by it.

2. Consider the ethical implications

  Explanation: Issues like algorithmic bias in hiring processes need to be addressed.

3. Analyze the potential for new economic models

  Explanation: Universal Basic Income or reduced working hours might become more prominent discussions.

4. Examine historical parallels

  Explanation: Previous technological revolutions, like the Industrial Revolution, can offer insights into potential outcomes.

Further Refined Answer: While AI will undoubtedly transform the job market, its impact will likely be nuanced, involving collaboration between humans and AI, ethical considerations, and potentially new economic paradigms.

综合答案：

AI: After careful consideration across multiple turns of reasoning, we can synthesize our thoughts on the potential impacts of artificial intelligence on the job market:

1. Job Transformation: AI will lead to significant changes in the job market, but these changes will be gradual and complex. While some jobs may be automated, many others will be transformed to involve collaboration between humans and AI. Industries like manufacturing, customer service, and data entry are likely to see the most immediate impacts.

2. New Opportunities: The rise of AI will create new job categories, particularly in fields like AI development, data science, AI ethics consultation, and human-AI interaction design. There will likely be an increased demand for skills in creativity, emotional intelligence, and complex problem-solving that AI cannot easily replicate.

3. Economic and Social Implications: The integration of AI into the workforce has the potential to increase productivity and economic growth. However, it also raises concerns about wealth distribution and may necessitate discussions about new economic models, such as Universal Basic Income or reduced working hours.

4. Education and Adaptation: The changing job market will require a strong emphasis on continuous learning and adaptation. Educational systems and workforce training programs will need to evolve to prepare people for the new reality of working alongside AI.

5. Ethical Considerations: As AI becomes more prevalent in hiring and work processes, addressing issues like algorithmic bias and ensuring fair opportunities for all workers will be crucial.

6. Global Variations: The impact of AI on the job market will not be uniform across the globe. Different countries and economies will experience varying effects based on their economic structures, technological readiness, and policy responses.

7. Historical Context: While AI represents a significant shift, it’s important to view it in the context of previous technological revolutions. Like the Industrial Revolution, the AI revolution will likely lead to short-term disruptions but potentially long-term benefits if managed properly.

In conclusion, while AI will undoubtedly have profound impacts on the job market, these impacts will be multifaceted and evolve over time. The key to navigating this transition successfully lies in proactive measures in education, economic policy, and ethical governance, ensuring that the benefits of AI are broadly shared while mitigating potential negative effects.

可以看到，多轮方法能实现更加精细、全面的分析。这能从多个角度考虑问题，优化其思考，最后得到更加深思熟虑和全面详细的答案。

Bowling 表示，这样的多轮推理在思考深度、自我提升、透明度、灵活性等方面都有巨大优势，同时其还采用了 Markdown 格式，使其结果更容易阅读和用于后续流程。

因此，这种技术可让 AI 进入更多实际应用场景，比如医疗诊断、法律分析、科学研究、商业策略、教育。当然，在使用 Llamaberry 等多轮推理系统时，也需要注意它们的局限和潜在的道德伦理问题，比如偏见、计算成本、过度依赖和隐私问题。

如何使用 Llamaberry?

Llamaberry 也很容易使用，点击几下就能拥有你自己的多轮推理系统。步骤如下：

1. 前往 Replit，点击该链接获取 Llamaberry 模板：https://replit.com/@MartinBowling/Llamaberry-Powered-By-Groq?v=1
2. 创建模板分支：点击 Fork 按钮创建你自己的 Llamaberry 项目副本。
3. 获取你的 Groq API Key：注册 Groq 账户，获取 API Key。
4. 设置环境：在你的分支 Replit 项目中，找到「Secrets」选项卡。添加一个新密钥，密钥为 GROQ_API_KEY，值是你的 Groq API 密钥。
5. 运行项目：单击 Replit 界面顶部的 Run 按钮。这将启动 Llamaberry 应用。
6. 开始实验：应用运行起来后，你将看到一个 Gradio 界面。你可以在其中输入问题并查看 Llamaberry 多轮推理的实际效果！并且输出是简洁漂亮的 Markdown 格式！

了解了 Llamaberry，下面来看另一个号称实现了类 o1 推理链的项目：g1。

g1：实现类似 ο1 的推理链

g1 这个项目来自 Benjamin Klieger，他是 Groq 的一位研究者。也因此，g1 同样基于 Groq，并且其也使用了 Llama 3.1 70b 模型。

不同于 Llamaberry 使用的多轮思维链推理，g1 的策略是角色扮演、思维链提示 、格式化以及另一些提示技巧。并且，g1 开源了。

项目地址：https://github.com/bklieger-groq/g1

开发者宣称 g1 有 70% 的时间能成功数出 Strawberry 中有多少个 R，同时无需任何微调或少样本技术。下面是其一次执行过程： 

开发者 Klieger 表示，g1 和 ο1 一样能让 LLM 有能力「思考」和解决之前的领先模型难以应对的逻辑问题。但不同之处在于，g1 会大方地展示所有推理 token。同时，他也强调了 g1 和 ο1 在技术上的差异，其中后者使用了大规模强化学习来执行思维链推理。而 g1 则是通过发掘提示词工程的潜力来帮助 LLM 解决简单的逻辑问题，让现有的开源模型也能受益于动态推理链和优化般的探索界面。

g1 的工作方式

由 Llama 3.1 70b 支持的 g1 会创建一种动态的思维链。

在每个步骤中，LLM 可以选择是继续进行另一个推理步骤，还是提供最终答案。每个步骤都有标题，并且对用户可见。

系统提示词中还会包含给 LLM 的提示。其提示策略如下：

You are an expert AI assistant that explains your reasoning step by step. For each step, provide a title that describes what you’re doing in that step, along with the content. Decide if you need another step or if you’re ready to give the final answer. Respond in JSON format with ‘title’, ‘content’, and ‘next_action’ (either ‘continue’ or ‘final_answer’) keys. USE AS MANY REASONING STEPS AS POSSIBLE. AT LEAST 3. BE AWARE OF YOUR LIMITATIONS AS AN LLM AND WHAT YOU CAN AND CANNOT DO. IN YOUR REASONING, INCLUDE EXPLORATION OF ALTERNATIVE ANSWERS. CONSIDER YOU MAY BE WRONG, AND IF YOU ARE WRONG IN YOUR REASONING, WHERE IT WOULD BE. FULLY TEST ALL OTHER POSSIBILITIES. YOU CAN BE WRONG. WHEN YOU SAY YOU ARE RE-EXAMINING, ACTUALLY RE-EXAMINE, AND USE ANOTHER APPROACH TO DO SO. DO NOT JUST SAY YOU ARE RE-EXAMINING. USE AT LEAST 3 METHODS TO DERIVE THE ANSWER. USE BEST PRACTICES.

Example of a valid JSON response:

json

{

  “title”: “Identifying Key Information”,

  “content”: “To begin solving this problem, we need to carefully examine the given information and identify the crucial elements that will guide our solution process. This involves…”,

  “next_action”: “continue”

}

对这些提示词的详细解释请参阅原项目的 Prompt Breakdown 一节。这里就不赘述了，仅给出几个示例，比如可以在提示词中加入「include exploration of alternative answers」（探索其它答案）和「use at least 3 methods to derive the answer」（使用至少三种方法来得出答案）。

这样一来，通过组合思维链以及尝试多种方法、探索其它答案、质疑之前草拟的解答、考虑 LLM 的局限性等策略，就能显著提升 LLM 的推理能力。

在数 Strawberry 中有多少个 R 这个经典问题上，无需任何训练，g1 就能帮助 Llama 3.1 70b 达到约 70% 的准确度（n=10, How many Rs are in strawberry?）。而如果不使用提示技术，Llama 3.1 70b 的准确率为 0%，ChatGPT-4o 的也只有 30%。

下面展示了另一个示例：0.9 和 0.11 哪个更大？

详细的安装过程和代码请参阅原项目。

最后，顺便一提，另有开发者发布了 g1 的分支版 Mult1，该版本的一大改进是可使用多个 AI 提供商来创建类似 o1 的推理链，感兴趣的读者可访问：https://github.com/tcsenpai/multi1

文章来自于“机器之心”，作者“Panda”。