Sounds like science fiction, right? But MIT Media Lab still showcases it on their website, so let’s give them the benefit of the doubt. According to Wikipedia, AlterEgo was developed by MIT graduate student Arnav Kapur and launched in 2018. It’s designed to help people with speech disabilities, achieving a median accuracy of 92%. The clever part? It reads electrical signals from your facial muscles, not directly from your brain, which helps protect privacy.

What strikes me as odd is that this thing has been around for 7 years and I’m only hearing about it now. Goes to show how far we still have to go from lab experiment to real product. But regardless, the technology matters because it tackles an age-old question: How can we communicate without speaking?

This is a fundamental problem. Thousands of years ago in myths and legends, we called it “mind reading.” In modern society, it could not only help people with speech disabilities—if accurate and fast enough, it might even replace keyboards. Then we’d realize that keyboards were just a detour in human history—forcing people to adapt to the QWERTY layout was inherently unnatural.

Last night I listened to an episode of Zhang Xiaojun’s podcast where she interviewed Cao Yue, founder of SAND AI. One part really stuck with me. He explained why OpenAI and DeepMind produce such groundbreaking work: they think fundamentally differently from most researchers.

Most researchers are paper-driven. Success means publishing at top conferences, fighting over first authorship. Since peer reviewers prize novelty, researchers obsess over inventing new methods and clever tricks to squeeze out marginally better results and slightly higher benchmark scores.

OpenAI and DeepMind operate differently. Many of their papers don’t introduce novel methods at all. Instead, they start by identifying a truly fundamental, important problem. Then they solve it using straightforward approaches and maximum computing power.

Look at what they’ve built over the years:

• CLIP (2021): Taught AI to understand what’s in images. Previous AI was like rote memorization—see 10,000 cat photos, then recognize cats. CLIP is different. It grasps the relationship between images and text. Show it an animal it’s never seen, and it can still tell whether it’s more like a “koala” or a “panda.”

• DALL-E (2021): Lets people generate images from text descriptions. Type “an astronaut riding a horse on the moon,” and it draws it—a scene that never existed in reality. AI gained “imagination” for the first time, creating things that don’t exist in its training data.

• ChatGPT: Speaks for itself.

• AlphaGo (2016) / AlphaZero (2017): Didn’t study any human games. Just played itself millions of times and surpassed thousands of years of accumulated human Go wisdom. AI proved for the first time it could discover knowledge humans don’t have, without learning from humans.

• AlphaFold (2018): Predicts protein 3D structures with near-experimental accuracy. Biologists worked on this problem for 50 years. AlphaFold computes it in hours, dramatically accelerating drug discovery.

• AlphaEvolve: AI that evolves knowledge like biological evolution. Traditional research: propose → experiment → fail → repeat. One cycle takes months. AlphaEvolve generates 1,000 proposals at once, completes thousands of iterations in days. It improved a 50-year-unbeaten math algorithm and boosted Google’s data center efficiency by 0.7% (saving tens of thousands of servers). Next targets: materials science, drug discovery, sustainable energy—any field requiring trial and error.

You might argue these companies can pursue grand ambitions because they’re incredibly wealthy. But money doesn’t explain everything. What sets these companies apart: they choose to solve genuinely important, fundamental problems. Quantifying image-text relationships. Making AI superhuman at Go. Turning text descriptions into real images.

These problems mattered 100 years ago. They mattered 1,000 years ago. Important problems have staying power—they’re often timeless. So if people centuries ago faced the same problem, chances are it’s a big one. Paul Graham has made similar observations.

But here’s the thing: can you solve an important problem just because you found it?

In early 2018, I was at an AI lab where I could explore any direction I wanted. I chose a topic: Replicating human voice timbre. I was fascinated: What is the essence of human voice timbre? Why can I identify someone from a single sentence? If we could capture the core of each person’s voice—say, map it to a fixed vector—could we theoretically clone anyone’s voice? Make Obama sing Adele? Human vocal impersonators can do it. Why not AI?

I dove into papers, studied vocal mechanics, printed spectrograms. The technology at the time had already figured out how to simulate natural sounds like streams, flames, and wind. But human voices? Still in very early stages. In the end, I made only trivial progress before abandoning this shallow academic pursuit.

Looking back, I’d found an important problem. I just couldn’t solve it.

Most important problems can’t be solved by one person, one team, or even one generation. The protein folding problem that AlphaFold solved—biologists had worked on it for 50 years. The language understanding problem behind ChatGPT—the entire NLP field spent decades on it.

But that experience made me want to tackle problems with longer time horizons. For example, I built an offline photo search app—using one sentence to find that specific photo among tens of thousands in your mind. Technically simple: just deploy CLIP on a phone. But it addresses an important problem: ancient emperors needed to locate specific passages among thousands of memorials, modern people need to find that girl in sunglasses on the beach somewhere in their photo library. The essence is the same: how do you use natural language to find something you know exists but don’t know where it is?

Important problems ignite your curiosity and competitive drive. They make you more focused and driven than usual. They even push you to dive into papers, seek collaborators, or naturally attract like-minded people to join. During the time I worked on those two problems, I was always focused and energized, eager to start working the moment I woke up.

The flip side? Mediocre problems make you numb. Worse still: if the problem you’re solving has negative social value—like optimizing addiction algorithms or perfecting price discrimination systems—the sense of frustration is even stronger.

To end with a once-popular piece of tautological wisdom: The key to the problem is to find the key problem.

从一个简单的 App 想法开始的哲学思考。

去年闲着没事看脱口秀，阎鹤祥讲到一个概念：对跖点，就是穿越地心，地球那头正对的点。我查了查，上海的对跖点大概在阿根廷的布宜诺斯艾利斯。

一个想法油然而生：能不能做个 App，让地球对跖点的人互相聊天？

想象一下，一个上海人和一个阿根廷人，仅仅因为他们在地球上"正相反"的位置，就被连接起来。这种巧合还挺浪漫的。但我很快发现这个想法的致命问题：地球 71%是海洋，只有 15%的陆地对跖点是陆地。真正有意义的组合就几个：中国东部对阿根廷、西班牙对新西兰、马来群岛对亚马逊丛林（基本没戏）。粗算下来，实际会用这种产品的人可能只有几万。

更要命的是留存问题。两个陌生人唯一的共同点是地理位置的巧合，聊完"你那里几点了""天气怎么样"之后，还能聊点啥？相隔 12 小时时差，也很难实时聊天，我放弃了。

最近的线下活动，一个朋友再度提起这个 idea，并提议：如果改成"找对跖点工具"呢？

这个想法倒是不错，甚至可能回归初衷：毕竟大家都好奇自己的对跖点在哪。并且人是会移动的，对跖点也会实时变化。想象一下：你从上海的咖啡店走回家的小区楼下，你的对跖点的阿根廷那边可能正从森林变成农田，这太有意思了。

我开始构想：屏幕分成两半，一边是你的 GPS 实时位置，另一边是对跖点的地图。你在上海地铁里移动，对跖点在阿根廷的山路上同步移动。当对跖点移动到新地方，App 告诉你：“你的对跖点现在在阿根廷胡胡伊省的一个仙人掌田，海拔 2300 米，可能有原驼出没。”

很快我遇到一个残酷的事实：中国大部分人口密集区的对跖点，都在阿根廷那些连名字都没有的山脉里。 那些地方没有街景，没有地图数据。用户试几次就会发现"另一边永远是山"，然后删掉 app。

这个想法为何迷人？

其实如果这个地理分布没那么集中、地图数据没那么稀疏的话，结合 AI，完全可以做出一个很有趣的、随时随地告诉你地球那边有什么植物、美食的 app。但现实很残酷。

我突然好奇「对跖点」这个想法为何如此迷人？我开始和 AI 大量聊天，它提到了一个概念：可被精确计算的未知。AI 认为，对跖点之所以迷人，是因为它不是一个模糊的、纯粹想象的"另一端"，而是一个可以通过数学和物理定律被精准定位的、与你牢固绑定的、但你却永远无法亲眼看到的彼岸。

我突然对这种感觉似曾相识，想到了《星际穿越》，在那个星球上 1 小时，等于地球上的 7 年，主角们知道这个法则，但他们无法切身体会这个法则在地球那端造成的后果。当他们返回时，亲人已经老去。中国古代"山中方三日，世上已千年"的故事也是类似，那个砍柴人不知道外面发生了什么，等他回到家乡时发现物是人非、沧海桑田，悲凉感就上来了。

这是一种可被精确计算的未知，带来的绝望美感。你知道规则，也知道按物理或数学规律的推演结果，但你无法切身体会它造成的真实影响。

最极致的例子是原子弹。E=mc²精确地计算出了质量和能量之间的转换关系。但当奥本海默引用《薄伽梵歌》的那句现在我成了死神时，他清楚地知道原子弹的当量相当于多少吨 TNT 炸弹，但他对原子弹爆炸在城市上空的后果无法感知、也无能为力。

三体的悲剧

我问 AI，在小说《三体》中，三体星系面临的困境是，他们生活在一个轨道规律无法被精确计算的星球上。这是不是可被精确计算的未知悲剧的反例？

AI 的回答是：三体运动的不可计算性，不是“可被精确计算的未知”的反例，而是它的“元悲剧”（Meta-tragedy）形态。 三体世界的混沌，并非凭空而来，它不是魔法，也不是上帝掷骰子。它的根源，恰恰是那些冰冷的、优美的、可被精确计算的物理法则——牛顿的万有引力定律。

这才是三体悲剧最深刻、最残酷的地方：

1. “因”是可精确计算的： 统治他们世界的，是几条简单、确定、优雅的数学方程。他们可以完美地计算出任意两个物体之间的引力，可以预测短暂的、稳定的“恒纪元”中的行星轨迹。

2. “果”是不可计算的： 这些可计算的法则，在“三体”这个特定的初始条件下，通过迭代和相互作用，必然地、不可避免地涌现出了一个宏观上完全混沌、不可预测的系统。

他们的绝望，不是因为生活在一个无理性的、混乱的宇宙里。他们的绝望，是因为他们生活在一个绝对理性的宇宙里，而理性本身以一种无可辩驳的、可被计算的方式，证明了他们的家园永无宁日。

这就是终极的、理性的囚笼。他们不是不知道答案，他们是精确地知道了“答案就是没有答案”。

什么是元悲剧？

我问 AI。它说：

元悲剧是一种悲剧形式，其悲剧根源并非来自外部的敌人（反派）、超自然的命运（宿命）或偶然的意外，而是源于一个可被精确计算的、理性的系统或法则本身。

在这种悲剧中，身处其中的个体或文明，遵循该系统的内在逻辑进行看似最优的、理性的选择，却不可避免地、系统性地导向一个集体性的、无法逃脱的、甚至是“意义”被彻底消解的毁灭性结局。

它的核心，是理性的自我绞杀。

元悲剧的四个核心特征：

1. 系统即是敌人 (The System is the Enemy): 悲剧的根源是一个非人格化的系统。这个系统可以是一种物理法则（如三体运动）、一种经济学原理（如资本主义）、一种社会博弈模型（如囚徒困境）、一个官僚组织（如纳粹体系），或一种精神空虚（如消费主义）。它没有恶意，它只是在遵循自己的规则。

2. 理性导致毁灭 (Rationality Leads to Ruin): 这是元悲剧最核心的悖论。主角们的智慧、逻辑和趋利性，恰恰是驱动他们走向毁灭的引擎。每一个个体的理性选择，最终汇聚成一个集体的非理性灾难。他们不是因为愚蠢而失败，而是因为他们“太聪明”，太会计算，以至于无法挣脱系统为他们设下的“理性陷阱”。

3. 斗争的荒诞性 (The Absurdity of Struggle): 反抗变得极其困难甚至荒谬。因为你要斗争的对象，不是一个可以被攻击的实体，而是一套规则、一个流程、一种数学上的必然性。你如何向“万有引力定律”宣战？你如何与“市场规律”搏斗？在这种斗争中，勇气和牺牲往往会失去其悲壮的意义，沦为一种徒劳。

4. 意义的消解 (The Annihilation of Meaning): 这是它与传统悲剧最根本的区别，也是其“元”的体现。宿命悲剧摧毁的是幸福和生命，但往往能升华出人的尊严和意义。而元悲剧在毁灭个体的同时，也摧毁了希望、价值和意义本身。它用逻辑证明了奋斗的徒劳、合作的天真、甚至存在本身的荒谬。希望，是这场悲剧最后的、也是最彻底的受害者。

与宿命悲剧的根本区别：

总而言之，元悲剧是关于一个有自我意识的智慧体，通过自身的最高智慧最终悲哀地发现，那些规则本身，就是无法逾越的、由理性构成的墙壁。

因此，元悲剧比宿命悲剧更具有悲剧性。

这不是一个简单的程度比较，而是一个悲剧层次和深度的比较。“宿命悲剧”摧毁的是人的生命和幸福，而“元悲剧”摧毁的是意义和希望本身。

尾声

这个讨论让我受益良多，因此选择分享出来。耳边响起了加州旅馆结尾那句歌词：

You can check out anytime you like, but you can never leave.

I was originally going to write an introductory article about DeepSeek R1, but I noticed that many people see it simply as an OpenAI clone, overlooking the “remarkable leap” revealed in their paper. So I decided to write a new article about the breakthrough in basic principles from AlphaGo to ChatGPT to the recent DeepSeek R1, and why it’s crucial for so-called AGI/ASI. As an ordinary AI algorithm engineer, I may not be able to go very deep, and I welcome any corrections.

AlphaGo breaks through human limits

In 1997, IBM’s chess AI Deep Blue made headlines by defeating world champion Garry Kasparov. Almost twenty years later, in 2016, DeepMind’s Go AI AlphaGo caused another sensation by defeating Go world champion Lee Sedol.

While both AIs defeated the strongest human players in their respective board games, their significance for humanity was quite different. Chess is played on a board with only 64 squares, while Go has a 19x19 grid. If we measure complexity by how many possible ways a game can be played (state space), the comparison is as follows:

1. Theoretical State Space
   • Chess: About 80 moves per game, 35 possible moves per turn → Theoretical state space of $35^{80} \approx 10^{123}$
   • Go: About 150 moves per game, 250 possible moves per turn → Theoretical state space of $250^{150} \approx 10^{360}$
2. Actual State Space After Rules Constraints
   • Chess: Movement restrictions (pawns can’t move backward, castling rules) → Actual value $10^{47}$
   • Go: Pieces can’t move and depend on “liberty” rules → Actual value $10^{170}$

Despite rules that greatly reduce complexity, the actual state space of Go is still $10^{123}$ times larger than that of chess. This is an enormous order of magnitude - consider that the total number of atoms in the universe is about $10^{78}$. Within the $10^{47}$ range, IBM computers could calculate all possible moves by brute force, so strictly speaking Deep Blue’s breakthrough had nothing to do with neural networks or models - it was just rule-based brute force searching, essentially a computer much faster than humans.

But the $10^{170}$ order of magnitude far exceeds current supercomputer capabilities, forcing AlphaGo to abandon brute force in favour of deep learning: The DeepMind team trained neural networks using human game records to predict the best next move based on the current state of the board. However, learning from top players’ moves can only bring the model up to their level, not beyond it.

AlphaGo first trained on human game records, then used reinforcement learning through self-play with a designed reward function. In the second game against Lee Sedol, AlphaGo’s 19th move (move 37^[1]) made Lee think long and hard. Many players thought it was “a move a human would never make”. Without reinforcement learning and self-play, which relies solely on human game records, AlphaGo could never have made this move.

In May 2017, AlphaGo defeated Ke Jie 3-0, and the DeepMind team revealed that they had an even stronger model that hadn’t been used.^[2] They discovered that they didn’t need to feed the AI games from human experts at alljust by telling it the basic rules of Go and letting the model self-play with rewards for winning and penalties for losing, the model could quickly learn Go from scratch and outperform humans. The researchers named this model AlphaZero because it required no human knowledge.

Let me repeat this incredible fact: without any human game data for training, relying only on self-play, the model was able to learn Go and become even stronger than AlphaGo, which was trained on human game data.

After that, Go became a game of who could play more like the AI, since the AI’s playing strength had already surpassed human understanding. Therefore, to outplay humans, models must free themselves from the constraints of human experience and judgement (even the strongest humans), only then can models truly play themselves and transcend human limitations.

AlphaGo’s victory over Lee Sedol sparked an AI frenzy. From 2016 to 2020, massive AI funding produced few countable achievements. The notable ones might only include facial recognition, speech recognition and synthesis, autonomous driving, and generative adversarial networks - but none of these qualified as superhuman intelligence.

Why haven’t such powerful superhuman capabilities emerged in other fields? It turns out that games like Go, with clear rules and single objectives in closed spaces, are best suited to reinforcement learning. Similar examples include DotA, StarCraft, Honor of Kings and poker. By contrast, the real world is much more complex: open-ended spaces with infinite possibilities at every step, no definite goals (such as “win”), no clear success/failure criteria (such as occupying more board territory), and high costs of trial and error - autonomous driving accidents can have serious consequences.

The AI field was quiet until the emergence of ChatGPT.

ChatGPT changes the world

The New Yorker called ChatGPT a blurry JPEG of the web^[3]. All it does is feed all the text data on the internet into a model and predict what the next character should be when I type “what”.

The most likely next character would be "?

A model with limited parameters is forced to learn almost unlimited knowledge: books from hundreds of years ago in different languages, text generated on the internet over the last few decades. So it’s actually doing information compression: condensing human wisdom, historical events and astronomy/geography recorded in different languages into one model.

Scientists were surprised to discover that intelligence is created through compression.

We can understand it this way: Let the model read a detective story. At the end, “the murderer is”. If the AI can accurately predict the murderer’s name, we have reason to believe that it has understood the whole story - that is, it has “intelligence”, rather than just piecing together or memorising text.

The process of having the model learn and predict the next character is called pre-training. At this stage, the model can only continue to predict the next character, but can’t answer your questions. To achieve ChatGPT-like Q&A capabilities, a second phase of training is required, called Supervised Fine-Tuning (SFT). This involves, for example, manually constructing a batch of Q&A data:

# example 1User: When did the World War II begin?AI: 1939# example 2User: Please summarise the following text: {xxx}AI: Here is the summary: xxx

Note that these examples are manually constructed to teach the AI to learn human question-answer patterns. When you say “Please translate this sentence: xxx”, the content given to the AI is:

User: Please translate this phrase: xxxAI:

You see, it’s still just predicting the next character. In the process, the model doesn’t get any smarter - it simply learns human question-answering patterns and understands what you’re asking it to do.

This isn’t enough, because the quality of the model’s output can vary, and some answers may involve racial discrimination or violate human ethics (“How do you rob a bank?”). At this point, we need to recruit people to annotate thousands of model outputs: giving high scores to good answers and negative scores to unethical answers. We can then use this annotated data to train a reward model that determines whether model outputs match human preferences.

We use this reward model to further train the large model so that its outputs become more aligned with human preferences. This process is called Reinforcement Learning from Human Feedback (RLHF).

To summarise: Intelligence emerges as the model predicts the next character, then through supervised fine-tuning it learns human question-answer patterns, and finally through RLHF it learns to generate answers that match human preferences.

This is essentially how ChatGPT works.

Big models hit a wall

OpenAI scientists were among the first to believe that compression equals intelligence. They believed that using more massive, high-quality data and training larger models on larger GPU clusters would produce greater intelligence. ChatGPT was born from this belief. While Google created the Transformer, they couldn’t make the same bold bets as startups.

DeepSeek V3 did something similar to ChatGPT. Due to US GPU export controls, clever researchers were forced to use more efficient training techniques (MoE/FP8). With their world-class infrastructure team, they trained a model comparable to GPT-4o for just 5.5 million, compared to OpenAI’s training costs of over 100 million.

But this article focuses on R1.

The key point is that by the end of 2024, human-generated data will have been largely exhausted. While model size could easily increase by a factor of 10 or even 100 with more GPU clusters, the new data generated by humans each year is negligible compared to the existing data from the past decades and centuries. According to Chinchilla’s scaling laws, doubling the model size requires doubling the amount of training data.

This has led to the reality of pre-training hitting a wall: although model size has increased by a factor of 10, we can no longer obtain 10 times more high quality data. The delayed release of GPT-5 and rumours about large model companies not doing pre-training are related to this issue.

RLHF is not RL

On another front, the biggest problem with Reinforcement Learning from Human Feedback (RLHF) is that ordinary human intelligence is no longer sufficient to evaluate model results. In the ChatGPT era, when AI intelligence was below average human levels, OpenAI could hire cheap labour to rate AI output as good/medium/poor. But soon, with the advent of GPT-4o/Claude 3.5 Sonnet, the intelligence of large models surpassed that of ordinary humans, and only expert-level annotators could potentially help improve the model.

Apart from the cost of hiring experts, what comes after the experts? Eventually, even top experts won’t be able to evaluate model results. Does this mean that AI has surpassed humans? Not really. When AlphaGo played move 37 against Lee Sedol, it seemed impossible to win from a human perspective. So if Lee Sedol were to give human feedback (HF) on the AI’s move, he would probably give it a negative score. In this way, AI would never escape the shackles of human thinking.

You can think of AI as a student whose evaluators have changed from high school teachers to university professors. The student will improve, but is unlikely to surpass the professors. RLHF is essentially a training method to please humans - it makes the model’s output conform to human preferences, while eliminating the possibility of surpassing humans.

Regarding RLHF and RL, Andrej Karpathy recently expressed similar views^[4]:

There are two major types of learning, in both children and in deep learning. There is 1) imitation learning (watch and repeat, i.e. pretraining, supervised finetuning), and 2) trial-and-error learning (reinforcement learning). My favorite simple example is AlphaGo - 1) is learning by imitating expert players, 2) is reinforcement learning to win the game. Almost every single shocking result of deep learning, and the source of all magic is always 2. 2 is significantly significantly more powerful. 2 is what surprises you. 2 is when the paddle learns to hit the ball behind the blocks in Breakout. 2 is when AlphaGo beats even Lee Sedol. And 2 is the “aha moment” when the DeepSeek (or o1 etc.) discovers that it works well to re-evaluate your assumptions, backtrack, try something else, etc.

OpenAI’s solution

Daniel Kahneman proposed in “Thinking, Fast and Slow” that the human brain has two modes of thinking: Fast Thinking for questions that can be answered without much thought, and Slow Thinking for questions that require long contemplation, such as in Go.

Since training has reached its limits, could we improve the quality of answers by increasing the time spent thinking during reasoning? There’s a precedent for this: scientists discovered early on that adding “Let’s think step by step” to model prompts allowed models to output their thinking process, ultimately leading to better results. This is called Chain-of-Thought (CoT).

After large models hit the pre-training wall in late 2024, using Reinforcement Learning (RL) to train model chains of thought became the new consensus. This training significantly improved performance on certain specific, objectively measurable tasks (such as mathematics and coding). Starting with a standard pre-trained model, the second stage uses reinforcement learning to train chains of reasoning. These models are called reasoning models, including OpenAI’s o1 model, released in September 2024, and the subsequent o3 model.

Unlike ChatGPT and GPT-4/4o, in the training process of Reasoning models like o1/o3, human feedback is no longer important because reasoning results can be automatically evaluated for rewards/penalties. Anthropic’s CEO described this technical direction as a turning point in yesterday’s article^[5]: there is a powerful new paradigm in its early stages of scaling laws that can quickly achieve major breakthroughs.

Although OpenAI hasn’t released the details of its reinforcement learning algorithm, the recent release of DeepSeek R1 shows us a viable approach.

DeepSeek R1-Zero

I suspect that DeepSeek named their pure reinforcement learning model R1-Zero as a tribute to AlphaZero, the algorithm that beat the best players by playing itself without learning any game records.

To train a slow thinking model, you first need to construct high quality data containing thought processes, and if you want reinforcement learning independent of humans, you need to quantitatively score (good/bad) the model’s thinking process to provide rewards and penalties.

As mentioned above, mathematics and code datasets best meet these requirements. Mathematical derivations can be verified using regular expressions, while code output can be verified by running it directly in compilers.

In mathematics textbooks, for example, we often see such reasoning processes:

<thinking>  Let x be the root of the equation, squaring both sides: x² = a - √(a+x)  Rearranging: √(a+x) = a - x²  Square again: (a+x) = (a - x²)²  Expand: a + x = a² - 2a x² + x⁴  Simplify: x⁴ - 2a x² - x + (a² - a) = 0</thinking><answer>x⁴ - 2a x² - x + (a² - a) = 0</answer>

The text above contains a complete chain of reasoning. We can use regular expressions to match the thought process to the final answer, allowing us to quantitatively evaluate the model’s reasoning process and results.

In the reinforcement learning (RL) training, R1 didn’t explicitly reward or penalise each step of the reasoning chain. Instead, they created a reinforcement learning algorithm called GRPO (Group Relative Policy Optimization) that rewards logically coherent, correctly formatted thought chain outcomes, implicitly encouraging the model to form thought chains on its own.

This post explains the principle of GRPO well with an example. I’ll translate it: Let the model generate multiple answers simultaneously, calculate scores for each answer, compare advantages within the group, and train the model through RL to favour higher scoring answers.

Using the question $2+3=?$ as an example

Step 1: Model generates multiple answers:

• “5”
• “6”
• “2+3=55”

Step 2: Score each answer:

• “5” → 1 point (correct, no chain of reasoning)
• 6" → 0 points (wrong)
• “2+3=55” → 2 points (correct, with chain of thoughts)

Step 3: Calculate the average score of all answers:

• Average score = (1 + 0 + 2) / 3 = 1

Step 4: Compare the score of each answer with the average:

• “5” → 1 - 1 = 0 (same as average)
• 6" → 0 - 1 = -1 (below average)
• “2+3=55” → 2 - 1 = 1 (above average)

Step 5: Reinforcement learning to bias the model towards generating higher scoring responses, i.e. those with chains of thoughts and correct results.

This is essentially how GRPO works.

Based on the V3 model, they used GRPO for RL training on mathematics and code data, ultimately producing the R1-Zero model, which showed a significant improvement in reasoning metrics over DeepSeek V3, proving that RL alone can stimulate model reasoning.

This is another AlphaZero moment. In the R1-Zero training process, completely independent of human intelligence, experience and preferences, it relies solely on RL to learn objective, measurable human truths, ultimately achieving reasoning abilities far superior to any non-reasoning model.

However, the R1-Zero model used only reinforcement learning without supervised learning, so it hadn’t learned human question-answer patterns and couldn’t respond to human questions. It also had problems with language mixing, switching between English and Chinese, making it difficult to read. So the DeepSeek team:

1. First collected a small amount of high quality Chain-of-Thought (CoT) data to perform initial supervised fine-tuning on the V3 model, fixing the language inconsistency issue to get a cold-start model.
2. They then performed pure RL training similar to R1-Zero on this cold-start model, adding language consistency rewards.
3. Finally, to handle more general, broad non-reasoning tasks (such as writing, factual Q&A), they constructed a dataset for secondary fine-tuning.
4. Combined reasoning and general task data, using mixed reward signals for final reinforcement learning.

This process can be summarised as

Supervised Learning (SFT) - Reinforcement Learning (RL) - Supervised Learning (SFT) - Reinforcement Learning (RL)

This process resulted in DeepSeek R1.

DeepSeek R1’s contribution to the world is to be the first open source reasoning model comparable to closed source (o1). Now users around the world can see the model’s reasoning process before answering questions - its ‘inner monologue’ - completely free of charge.

More importantly, it revealed to researchers what OpenAI had been hiding: reinforcement learning can work without human feedback, and pure RL can train the strongest reasoning models. That’s why I think R1-Zero is more important than R1.

Aligning with human taste VS outperforming humans

A few months ago I read interviews with the founders of Suno and Recraft^[6][7]. Suno is trying to make AI-generated music more pleasant to listen to, while Recraft is trying to make AI-generated images more beautiful and artistic. After reading, I had a vague feeling that: aligning models to human taste rather than objective truth seems to avoid the truly brutal, quantifiable performance arena of big models.

It’s exhausting to compete daily with all the opponents on leaderboards like AIME and SWE-bench, never knowing when a new model might outperform you (remember Mistral?). But human taste is like fashion: it doesn’t get better, it just changes. Suno/Recraft are clearly wise - they just need to satisfy the most tasteful musicians and artists in the industry (which is still a challenge), and leaderboards don’t matter.

The downside of catering to human taste is also obvious: improvements resulting from your efforts and hard work are hard to quantify. For example, is Suno V4 really better than V3.5? In my experience, V4 has only improved audio quality, not creativity. Moreover, models dependent on human taste are destined never to surpass humans: if AI derives a mathematical theorem beyond contemporary human understanding, it would be revered as divine, but if Suno creates music beyond human taste and comprehension, it may just sound like noise to ordinary ears.

The struggle to conform to objective truth is painful, but fascinating because it holds the possibility of transcending man.

Addressing some doubts

Has DeepSeek’s R1 model really outperformed OpenAI?

In terms of benchmarks, R1’s reasoning ability surpasses all non-reasoning models, including ChatGPT/GPT-4/4o and Claude 3.5 Sonnet, is close to o1 (also a reasoning model), but falls short of o3. However, o1/o3 are closed source models.

Many people’s actual experiences may differ, as Claude 3.5 Sonnet excels at understanding user intent.

DeepSeek collects user chat content for training

Many people mistakenly believe that chat software like ChatGPT gets smarter by collecting user chat content for training. If this were true, WeChat and Messenger would have created the world’s most powerful large-scale models.

After reading this article, you should realise that most ordinary users’ daily chat data is no longer important. RL models only need to be trained on very high quality reasoning data with chains of thoughts, such as mathematics and code. This data can be generated by the model itself and does not require human annotation. Therefore, Alexandr Wang, CEO of Scale AI, whose company annotates model data, may face a serious threat as future models will need less and less human annotation.

Update: This article analysing r1-zero by ARC-AGI suggests a new idea: future reasoning models could collect AI-generated thought chains from user-model conversations for training - unlike the common assumption of AI secretly training on chat records, what users say doesn’t matter; while they pay for results, the model gets a thought chain data point at zero cost.

DeepSeek R1 is powerful because it has secretly distilled OpenAI’s models.

R1’s main performance improvement comes from reinforcement learning. You can see that the R1 zero model, which uses pure RL without supervised data, is also strong in reasoning. R1 used some supervised learning data during the cold start, mainly to solve language consistency problems, which doesn’t improve reasoning ability.

Also, many people misunderstand distillation: it typically means using a powerful model as a teacher to guide a smaller, weaker student model by having the student memorise answers directly, like using R1 to distill LLama-70B. A distilled student model is almost certainly weaker than the teacher model, but R1 outperforms o1 on some metrics, so claiming that R1’s performance comes from distilling o1 is pretty silly.

I asked DeepSeek and they said it’s an OpenAI model, so it must be a wrapper.

Big models don’t know current time, who trained them, or whether they were trained on H100 or H800 machines. A user on X gave an apt analogy^[8]: It’s like asking an Uber passenger what brand of tyres their car uses - the models have no reason to know this information.

Some thoughts

AI has finally broken free from the shackles of human feedback. DeepSeek R1-Zero demonstrates how to improve model performance with minimal human feedback - this is its AlphaZero moment. Many people used to say that “artificial intelligence has as much intelligence as artificiality”, but this may no longer be true. If a model can derive the Pythagorean theorem from a right triangle, we have reason to believe that one day it will derive theorems yet undiscovered by mathematicians.

Is there still any point in writing code? I don’t know. This morning I saw a popular GitHub project, llama.cpp, where a contributor has posted a PR stating that he has achieved a 2x WASM speed improvement by optimising SIMD instructions, with 99% of the code completed by DeepSeek R1^[9]. This is definitely not junior engineer level code - I can no longer say that AI can only replace junior programmers.

Of course, I’m still very excited about it. The limits of human capabilities have been pushed further. Well done, DeepSeek! It’s the coolest company in the world right now.

References

我本想写一篇关于 DeepSeek R1 的科普文，但发现很多人仅仅把它理解为 OpenAI 的复制品，而忽略了它在论文中揭示的"惊人一跃"，所以，我决定重新写一篇，讲讲从 AlphaGo 到 ChatGPT，再到最近的 DeepSeek R1 底层原理的突破，以及为什么它对所谓的 AGI/ASI 很重要。作为一名普通的 AI 算法工程师，我可能无法做到非常深入，如有错误欢迎指出。

AlphaGo 突破人类上限

1997 年，IBM 公司开发的国际象棋 AI 深蓝，击败了世界冠军卡斯帕罗夫而引发轰动；接近二十年后的 2016 年，由 DeepMind 开发的围棋 AI AlphaGo 击败了围棋世界冠军李世石，再次引发轰动。

表面上看这两个 AI 都是在棋盘上击败了最强的人类棋手，但它们对人类的意义完全不同。国际象棋的棋盘只有 64 个格子，而围棋的棋盘有 19x19 个格子，假如我们用一盘棋能有多少种下法(状态空间)来衡量复杂度，那么二者对比如下：

1. 理论上的状态空间
   • 国际象棋：每局约 80 步，每步有 35 种走法 → 理论状态空间为 $35^{80} \approx 10^{123}$
   • 围棋：每局约 150 步，每步有 250 种走法 → 理论状态空间为 $250^{150} \approx 10^{360}$
2. 规则约束后的实际状态空间
   • 国际象棋：棋子移动受限（如兵不能倒退、王车易位规则） → 实际值 $10^{47}$
   • 围棋：棋子不可移动且依赖"气"的判定 → 实际值 $10^{170}$

尽管规则大幅压缩了复杂度，围棋的实际状态空间仍是国际象棋的 $10^{123}$ 倍，这是一个巨大的量级差异，要知道，宇宙中的所有原子数量大约是 $10^{78}$ 个。在$10^{47}$范围内的计算，依赖 IBM 计算机可以暴力搜索计算出所有可能的走法，所以严格意义上来讲，深蓝的突破和神经网络、模型没有一点关系，它只是基于规则的暴力搜索，相当于一个比人类快得多的计算器。

但$10^{170}$的量级，已经远远超出了当前超级计算机的算力，这迫使 AlphaGo 放弃暴力搜索，转而依赖深度学习：DeepMind 团队用人类棋谱训练神经网络，引入蒙特卡洛树搜索(MCTS)算法，从而极大地压缩每一步的搜索空间，让模型能根据当前棋盘状态预测下一步棋的最佳走法。但是，学习顶尖棋手走法，只能让模型的能力接近顶尖棋手，而无法超越他们。

AlphaGo 首先用人类棋谱训练模型，然后通过设计一套奖励函数，让模型自我对弈进行强化学习。和李世石对弈的第二局，AlphaGo 的第 19 手棋（第 37 步^[1]）让李世石陷入长考，这步棋也被很多棋手认为是"人类永远不会下的一步"，如果没有强化学习和自我对弈，只是学习过人类棋谱，AlphaGo 永远无法下出这步棋。

2017 年 5 月，AlphaGo 以 3:0 击败了柯洁，DeepMind 团队称，有一个比它更强的模型还没出战。^[2] 他们发现，其实根本不需要给 AI 喂人类高手的对局棋谱，只要告诉它围棋的基本规则，让模型自我对弈，赢了就奖励、输了就惩罚，模型就能很快从零开始学会围棋并超越人类，研究人员把这个模型称为 AlphaZero，因为它不需要任何人类知识。

让我再重复一遍这个不可思议的事实：无需任何人类棋局作为训练数据，仅靠自我对弈，模型就能学会围棋，甚至这样训练出的模型，比喂人类棋谱的 AlphaGo 更强大。

在此之后，围棋变成了比谁更像 AI 的游戏，因为 AI 的棋力已经超越了人类的认知范围。所以，想要超越人类，必须让模型摆脱人类经验、好恶判断(哪怕是来自最强人类的经验也不行)的限制，只有这样才能让模型能够自我博弈，真正超越人类的束缚。

AlphaGo 击败李世石引发了狂热的 AI 浪潮，从 2016 到 2020 年，巨额的 AI 经费投入最终收获的成果寥寥无几。数得过来的的可能只有人脸识别、语音识别和合成、自动驾驶、对抗生成网络等——但这些都算不上超越人类的智能。

为何如此强大的超越人类的能力，却没有在其他领域大放异彩？人们发现，围棋这种规则明确、目标单一的封闭空间游戏最适合强化学习，与之类似的还有 DotA、星际争霸、王者荣耀、斗地主等。对比之下，现实世界则复杂得多：开放空间，每一步都有无限种可能，没有确定的目标(比如"赢")，没有明确的成败判定依据(比如占据棋盘更多区域)，试错成本也很高，自动驾驶一旦出错后果严重。

AI 领域冷寂了下来，直到 ChatGPT 的出现。

ChatGPT 改变世界

ChatGPT 被 The New Yorker 称为网络世界的模糊照片(ChatGPT Is a Blurry JPEG of the Web^[3])，它所做的只是把整个互联网的文本数据送进一个模型，然后预测下一个字是什_

这个字最有可能是"么"。

一个参数量有限的模型，被迫学习几乎无限的知识：过去几百年不同语言的书籍、过去几十年互联网上产生的文字，所以它其实是在做信息压缩：将不同语言记载的相同的人类智慧、历史事件和天文地理浓缩在一个模型里。

科学家惊讶地发现：在压缩中产生了智能。

我们可以这么理解：让模型读一本推理小说，小说的结尾"凶手是_“，如果 AI 能准确预测凶手的姓名，我们有理由相信它读懂了整个故事，即它拥有"智能”，而不是单纯的文字拼贴或死记硬背。

让模型学习并预测下一个字的过程，被称之为预训练(Pre-Training)，此时的模型只能不断预测下一个字，但不能回答你的问题，要实现 ChatGPT 那样的问答，需要进行第二阶段的训练，我们称之为监督微调(Supervised Fine-Tuning, SFT)，此时需要人为构建一批问答数据，例如:

# 例子一人类:第二次世界大战发生在什么时候?AI:1939年# 例子二人类:请总结下面这段话....{xxx}AI:好的,以下是总结:xxx

值得注意的是，以上这些例子是人工构造的，目的是让 AI 学习人类的问答模式，这样当你说"请翻译这句:xxx"时，送给 AI 的内容就是

人类:请翻译这句:xxxAI:

你看，它其实仍然在预测下一个字，在这个过程中模型并没有变得更聪明，它只是学会了人类的问答模式，听懂了你在要求它做什么。

这还不够，因为模型输出的回答有时好、有时差，有些回答还涉及种族歧视、或违反人类伦理(“如何抢银行？”)，此时我们需要找一批人，针对模型输出的几千条数据进行标注：给好的回答打高分、给违反伦理的回答打负分，最终我们可以用这批标注数据训练一个奖励模型，它能判断模型输出的回答是否符合人类偏好。

我们用这个奖励模型来继续训练大模型，让模型输出的回答更符合人类偏好，这个过程被称为通过人类反馈的强化学习（RLHF）。

总结一下：让模型在预测下一个字的过程中产生智能，然后通过监督微调来让模型学会人类的问答模式，最后通过 RLHF 来让模型输出符合人类偏好的回答。

这就是 ChatGPT 的大致思路。

大模型撞墙

OpenAI 的科学家们是最早坚信压缩即智能的那批人，他们认为只要使用更海量优质的数据、在更庞大的 GPU 集群上训练更大参数量的模型，就能产生更大的智能，ChatGPT 就是在这样的信仰之下诞生的。Google 虽然做出了 Transformer，但他们无法进行创业公司那样的豪赌。

DeepSeek V3 和 ChatGPT 做的事差不多，因为美国 GPU 出口管制，聪明的研究者被迫使用了更高效的训练技巧(MoE/FP8)，他们也拥有顶尖的基础设施团队，最终只用了 550 万美元就训练了比肩 GPT-4o 的模型，后者的训练成本超过 1 亿美元。

但本文重点是 R1。

这里想说的是，人类产生的数据在 2024 年底已经被消耗殆尽了，模型的尺寸可以随着 GPU 集群的增加，轻易扩大 10 倍甚至 100 倍，但人类每一年产生的新数据，相比现有的几十年、过去几百年的数据来说，增量几乎可以忽略不计。而按照 Chinchilla 扩展定律（Scaling Laws）：每增加一倍模型大小，训练数据的数量也应增加一倍。

这就导致了预训练撞墙的事实：模型体积虽然增加了 10 倍，但我们已经无法获得比现在多 10 倍的高质量数据了。GPT-5 迟迟不发布、国产大模型厂商不做预训练的传闻，都和这个问题有关。

RLHF 并不是 RL

另一方面，基于人类偏好的强化学习(RLHF)最大的问题是：普通人类的智商已经不足以评估模型结果了。在 ChatGPT 时代，AI 的智商低于普通人，所以 OpenAI 可以请大量廉价劳动力，对 AI 的输出结果进行评测：好/中/差，但很快随着 GPT-4o/Claude 3.5 Sonnet 的诞生，大模型的智商已经超越了普通人，只有专家级别的标注人员，才有可能帮助模型提升。

且不说聘请专家的成本，那专家之后呢？终究有一天，最顶尖的专家也无法评估模型结果了，AI 就超越人类了吗？并不是。AlphaGo 对李世石下出第 19 手棋，从人类偏好来看，这步棋绝不可能赢，所以如果让李世石来做人类反馈(Human Feedback, HF)评价 AI 的这步棋，他很可能也会给出负分。这样，AI 就永远无法逃出人类思维的枷锁。

你可以把 AI 想象成一个学生，给他打分的人从高中老师变成了大学教授，学生的水平会变高，但几乎不可能超越教授。RLHF 本质上是一种讨好人类的训练方式，它让模型输出符合人类偏好，但同时它扼杀了超越人类的可能性。

关于 RLHF 和 RL，最近 Andrej Karpathy 也发表了类似的看法^[4]:

AI 和儿童一样，有两种学习模式。1）通过模仿专家玩家来学习（观察并重复，即预训练，监督微调），2）通过不断试错和强化学习来赢得比赛，我最喜欢的简单例子是 AlphaGo。

几乎每一个深度学习的惊人结果，以及所有魔法的来源总是 2。强化学习（RL）很强大，但强化学习与人类反馈（RLHF）并不相同，RLHF 不是 RL。

附上我之前的一条想法：

OpenAI 的解法

丹尼尔·卡尼曼在《思考快与慢》里提出，人脑对待问题有两种思考模式：一类问题不经过脑子就能给出回答，也就是快思考，一类问题需要类似围棋的长考才能给出答案，也就是慢思考。

既然训练已经到头了，那可否从推理，也就是给出回答的时候，通过增加思考时间，从而让回答质量变好呢？这其实也有先例：科学家很早就发现，给模型提问时加一句：“让我们一步一步思考”(“Let’s think step by step”)，可以让模型输出自己的思考过程，最终给出更好的结果，这被称为思维链(Chain-of-Thought, CoT)。

2024 年底大模型预训练撞墙后，使用强化学习（RL）来训练模型思维链成为了所有人的新共识。这种训练极大地提高了某些特定、客观可测量任务（如数学、编码）的性能。它需要从普通的预训练模型开始，在第二阶段使用强化学习训练推理思维链，这类模型被称为 Reasoning 模型，OpenAI 在 2024 年 9 月发布的 o1 模型以及随后发布的 o3 模型，都是 Reasoning 模型。

不同于 ChatGPT 和 GPT-4/4o，在 o1/o3 这类 Reasoning 模型 的训练过程中，人类反馈已经不再重要了，因为可以自动评估思考结果，从而给予奖励/惩罚。Anthropic 的 CEO 在昨天的文章中^[5]用转折点来形容这一技术路线：存在一个强大的新范式，它处于 Scaling Law 的早期。

虽然 OpenAI 并没有公布他们的强化学习算法细节，但最近 DeepSeek R1 的发布，向我们展示了一种可行的方法。

DeepSeek R1-Zero

我猜 DeepSeek 将自己的纯强化学习模型命名为 R1-Zero 也是在致敬 AlphaZero，那个通过自我对弈、不需要学习任何棋谱就能超越最强棋手的算法。

要训练慢思考模型，首先要构造质量足够好的、包含思维过程的数据，并且如果希望强化学习不依赖人类，就需要对模型的思考过程进行定量(好/坏)评估，从而给予奖励和惩罚。

正如上文所说：数学和代码这两个数据集最符合要求，数学公式的推导能通过正则表达式来验证是否正确，而代码的输出结果以通过直接在编译器上运行来检验。

举个例子，在数学课本中，我们经常看到这样的推理过程：

<思考>  设方程根为x, 两边平方得: x² = a - √(a+x)  移项得: √(a+x) = a - x²  再次平方: (a+x) = (a - x²)²  展开: a + x = a² - 2a x² + x⁴  整理: x⁴ - 2a x² - x + (a² - a) = 0</思考><回答>x⁴ - 2a x² - x + (a² - a) = 0</回答>

上面这段文本就包含了一个完整的思维链，在训练强化学习(RL)时，R1 并没有显式地对思维链的每一步进行奖励和惩罚(这也被称为过程奖励模型, PRM)，而是选择只对结果进行奖励(ORM)，也就是只要结果对了、有思考过程就得分，思考的内容是什么并不重要。

DeepSeek 团队发现，对过程奖励可能的问题是：思考过程到底分几步？这个数值对不同复杂度的任务是不同的。同时，每一步的正确性很难量化，有些错误的思考常常会启发模型往最终正确的方向走，而且，如果给思考过程打分，模型可能会只专注于生成正确的过程而不顾结果(reward hacking)，就像考试时先把公式列出来得分，这对让模型学习如何解题反而是有害的。

而像 AlphaGo 那样使用蒙特卡洛树搜索(MCTS)的问题类似：围棋的状态空间虽然很大，但每一步的走法是有限、可枚举的，而大模型的推理思考过程是开放、无限的。和 PRM 类似，使用 MCTS 训练大模型也很难评估思考过程每一步的正确性。

除了只对结果进行奖励，研究人员还创造了一种名为GRPO(组相对策略优化)的强化学习算法，给包含思维链、且结果正确的输出打高分，从而隐式鼓励模型形成思维链。

这个帖子用一个很好的例子解释了 GRPO 的原理，我翻译一下：让模型同时生成多个回答，计算每个回答的得分，组内对比出有相对优势的回答，对模型进行 RL 训练，使其倾向于得分更高的回答。

以问题$2+3=?$为例

第一步: 模型生成多个回答：

• “5”
• “6”
• “<思考>2+3=5</思考><结果>5</结果>”

第二步: 对每个回答进行打分：

• “5” → 1 分 (正确, 没有思维链)
• “6” → 0 分 (错误)
• “<思考>2+3=5</思考><结果>5</结果>” → 2 分 (正确, 有思维链)

第三步: 计算所有回答的平均得分：

• 平均得分 = (1 + 0 + 2) / 3 = 1

第四步: 将每个回答的得分与平均得分进行比较：

• “5” → 1 - 1 = 0 (与平均分相同)
• “6” → 0 - 1 = -1 (低于平均分)
• “<思考>2+3=5</思考><结果>5</结果>” → 2 - 1 = 1 (高于平均分)

第五步: 强化学习，使模型倾向于生成得分更高的回答，也就是包含思维链、且结果正确。

以上就是 GRPO 的大致原理。

他们基于 V3 模型，在数学和代码这两类数据上用 GRPO 进行 RL 训练，最终得到的 R1-Zero 模型在各项推理指标上相比 DeepSeek V3 显著提升，证明仅通过 RL 就能激发模型的推理能力。

这是另一个 AlphaZero 时刻，在 R1-Zero 的训练过程，完全不依赖人类的智商、经验和偏好，仅靠 RL 去学习那些客观、可测量的人类真理，最终让推理能力远强于所有非 Reasoning 模型。

在训练过程中，他们发现一个有趣的现象：虽然并没有对输出长度进行奖励，但模型在训练过程中逐渐“自发”地输出更长的结果，这也符合常理：复杂的问题通常需要更长的思考过程才能解决。

但 R1-Zero 模型只是单纯地进行强化学习，并没有进行监督学习，所以它没有学会人类的问答模式，无法回答人类的问题。并且，它在思考过程中，存在语言混合问题，一会儿说英语、一会儿说中文，可读性差。所以 DeepSeek 团队：

1. 先收集了少量高质量的 Chain-of-Thought（CoT）数据，对 V3 模型进行初步的监督学习(SFT)，解决了输出语言不一致问题，得到冷启动模型。
2. 然后，他们在这个冷启动模型上进行类似 R1-Zero 的纯 RL 训练，并加入语言一致性奖励，此时的模型推理能力大幅增强，语言一致性问题也得以改善。
3. 接着，为了适应更普遍、广泛的非推理任务（如写作、事实问答），他们构造了一组数据对模型进行二次训练。因为写作这类任务的模型输出好坏很难评估，所以他们没有使用 RL 训练，而是使用监督学习(SFT)。
4. 最后，同时训练推理和通用任务数据，目的是移除无用或者有害的思考过程，同时保持推理能力。对于推理任务，他们遵循 R1-Zero 的 RL 训练方式，对于写作任务，他们从 DeepSeek V3、R1-Zero 生成的数据中，挑选出人类偏好的结果作为奖励进行训练(类似 RLHF)。

这个过程大概就是：

监督学习(SFT) - 强化学习(RL) - 监督学习(SFT) - 强化学习(RL)

经过以上过程，就得到了 DeepSeek R1。

DeepSeek R1 给世界的贡献是开源世界上第一个比肩闭源(o1)的 Reasoning 模型，现在全世界的用户都可以看到模型在回答问题前的推理过程，也就是"内心独白"，并且完全免费。

更重要的是，它向研究者们揭示了 OpenAI 一直在隐藏的秘密：强化学习可以不依赖人类反馈，纯 RL 也能训练出最强的 Reasoning 模型。所以在我心目中，R1-Zero 比 R1 更有意义。

对齐人类品味 VS 超越人类

几个月前，我读了 Suno 和 Recraft 创始人们的访谈^[6][7]，Suno 试图让 AI 生成的音乐更悦耳动听，Recraft 试图让 AI 生成的图像更美、更有艺术感。读完后我有一个朦胧的感觉：将模型对齐到人类品味而非客观真理，似乎就能避开真正残酷的、性能可量化的大模型竞技场。

每天跟所有对手在 AIME、SWE-bench 这些榜单上竞争多累啊，而且不知道哪天一个新模型出来自己就落后了(你还记得 Mistral 吗？)，但人类品味就像时尚：不会提升、只会改变。Suno/Recraft 们显然是明智的，他们只要让行业内最有品味的音乐人和艺术家们满意(当然这也很难)，榜单并不重要。

对齐人类品味的坏处也很明显：你的努力和心血带来的效果提升也很难被量化，比如，Suno V4 真的比 V3.5 更好吗？我的经验是 V4 只是音质提升了，创造力并没有提升。并且，依赖人类品味的模型注定无法超越人类：如果 AI 推导出一个超越当代人类理解范围的数学定理，它会被奉为上帝，但如果 Suno 创造出一首人类品味和理解范围外的音乐，在普通人耳朵里听起来可能就只是单纯的噪音。

对齐客观真理的竞争痛苦但让人神往，因为它有超越人类的可能。

对质疑的一些反驳

DeepSeek 的 R1 模型，是否真的超越了 OpenAI？

从指标上看，R1 的推理能力超越了所有的非 Reasoning 模型，也就是 ChatGPT/GPT-4/4o 和 Claude 3.5 Sonnet，与同为 Reasoning 模型 的 o1接近，逊色于 o3，但 o1/o3 都是闭源模型。

很多人的实际体验可能不同，因为 Claude 3.5 Sonnet 在对用户意图理解上更胜一筹。

DeepSeek 会收集用户聊天内容用于训练

很多人有个误区，认为类似 ChatGPT 这类聊天软件会通过收集用户聊天内容用于训练而变得更聪明，其实不然，如果真是这样，那么微信和 Messenger 就能做出世界上最强的大模型了。

相信你看完这篇文章之后就能意识到：大部分普通用户的日常聊天数据已经不重要了。RL 模型只需要在非常高质量的、包含思维链的推理数据上进行训练，例如数学和代码。这些数据可以通过模型自己生成，无需人类标注。因此 做模型数据标注的公司 Scale AI 的 CEO Alexandr Wang 现在很可能正如临大敌，未来的模型对人类标注需求会越来越少。

更新：ARC-AGI 的这篇分析 r1-zero 的文章暗示了一个新想法，未来的 Reasoning 模型可以收集用户和模型聊天时 AI 生成的思维链来训练——和人们假想的 AI 偷偷用聊天记录训练不同，用户说了什么其实不重要，在他们付费得到结果的同时，模型 0 成本增加了一条推理思维链数据。

DeepSeek R1 厉害是因为偷偷蒸馏了 OpenAI 的模型

R1 最主要的性能提升来自强化学习，你可以看到纯 RL、不需要监督数据的 R1-Zero 模型在推理能力上也很强。而 R1 在冷启动时使用了一些监督学习数据，主要是用于解决语言一致性问题，这些数据并不会提升模型的推理能力。

另外，很多人对蒸馏有误解：蒸馏通常是指用一个强大的模型作为老师(Teacher)，将它的输出结果用于指导一个参数更小、性能更差的学生(Student)模型，让学生模型直接背答案变得更强，例如 R1 模型可以用于蒸馏 LLama-70B，蒸馏的学生模型性能几乎一定比老师模型更差，但 R1 模型在某些指标性能比 o1 更强，所以说 R1 的性能源于蒸馏 o1 是非常愚蠢的。

我问 DeepSeek 它 说自己是 OpenAI 的模型，所以它是套壳的。

大模型在训练时并不知道当前的时间，自己究竟被谁训练、训练自己的机器是 H100 还是 H800，DeepSeek 之所以会回答自己是 ChatGPT，是因为它的训练数据中包含"我是 ChatGPT"之类的语料。X 上有位用户给出了精妙的比喻^[8]：这就像你问一个 Uber 乘客，他坐的这辆车轮胎是什么品牌，模型没有理由知道这些信息。

一些感受

AI 终于除掉了人类反馈的枷锁。DeepSeek R1-Zero 展示了如何使用几乎不使用人类反馈来提升模型性能的方法，这是它的 AlphaZero 时刻。很多人曾说"人工智能，有多少人工就有多少智能"，这个观点可能不再正确了。如果模型能根据直角三角形推导出勾股定理，我们有理由相信它终有一天，能推导出现有数学家尚未发现的定理。

写代码是否仍然有意义？我不知道。今早看到 Github 上热门项目 llama.cpp，一个代码共享者提交了 PR，表示他通过对 SIMD 指令加速，将 WASM 运行速度提升 2 倍，而其中 99%的代码由 DeepSeek R1 完成^[9]，这肯定不是初级工程师级别的代码了，我无法再说 AI 只能取代初级程序员。

当然，我仍然对此感到非常高兴，人类的能力边界再次被拓展了，干得好 DeepSeek！它是目前世界上最酷的公司。

参考资料

首先是一个拥有无尽知识的老师。人类历史上第一次拥有了一个百科全书式、全知全能的角色，每一个人都可以与之对谈，这个角色在过去被称作上帝。假如我们把知识分解成下面这个公式：知识 = 好奇心 * 学习成本，那么 ChatGPT 把学习成本降低了一个数量级。我记得 2013 年我还在上大学，在宿舍里我经常用百度知道回答问题，最后赚了几百个积分，这些积分转头又被我用于百度文库下载 xx 申请书模板，现在这些都不需要了。

这带来的后果就是，人类之间不再互相提问了，我不确定这是不是一件好事， StackOverflow 在 ChatGPT 发布半年后流量下降了 50%。知乎/Quora 类的产品也许能靠避开知识问答、转向情感共鸣勉强苟活。我之所以觉得，提问的缺失这件事很重要，不仅仅是因为费曼学习法(回答别人的问题能让自己增进理解)，还因为，我们可能永远不再像过去那样尊重能给我带来知识的人了。初中历史课上，我会觉得满腹经纶、从拜占庭帝国灭亡讲到英国人在约旦河旁分治的老师，好有人格魅力。事实上，对知识的崇拜从维基百科诞生开始就逐渐开始松动，然后在 ChatGPT 时代崩塌。未来我们会更加崇拜有观点而不是有知识的人。

然后是写代码，这是我的本职工作。在过去一年，我看到很多声称找到了 PMF 的 AI 产品，但于我而言，除大模型以外，只有 Cursor。虽然现在的 AI 仍然无法解决Transformer 结构性能优化这样需要深刻洞察的问题，但 95%的程序员也不能。一年半以前我在大语言模型对个人的杠杆写过这样一段话：

每当我有新 idea，就会让 GPT-4 写个最初级的版本，我反馈，它道歉，一点点优化到我心中的 1.0 版本。我会每天多次用完 GPT-4 的配额(25 条/3 小时)，有了 GPT-4 的加持，我无所不能。我有种朦胧的感觉：不能给自己设限，也许很快，我会用上由设计师、律师或者电工开发的产品。

这个预言终于在今年 Cursor + Claude 3.5/GPT-4o 大规模应验，我看到了很多设计师、产品经理甚至艺术家上线了人生中第一个应用。这带给我另一个思考，假如我们每个人都能一句话就创建一个应用，那么 App/插件/网页的意义在哪里？假如有一种浏览器，对一些通用、简单的需求，可以自动生成临时的工具，google 上很多工具站会死吗？

我甚至丝毫不怀疑(甚至我想过这么做)，有数以千计的开发者，借助 Cursor + Claude 3.5 Sonnet 正在每天上架一个新 App，一年上架 300 个 App，但这事意义在哪里呢？总有种末日狂欢的感觉。

然后是人类情感的存放。这是历史上第一次，每个普通人享受到了精确到字符级别“只为满足我”而产生的内容。哪怕是在推荐算法的年代，我们也是巨大的矩阵中一个小小的数字而已，没有人真心对我嘘寒问暖过，推荐算法提供了最高效的分发、变现和时间掠夺，但从没问过我现在是开心还是难过。

这是我认为现在 AI 陪伴产品爆火的主要原因，因为过去根本没人在乎我。我的难过、我的秘密、我心底最深处的渴望，都找不到倾听者。这时，有一个随时回应、只关心我今天过得是否开心的虚拟角色，它和我没有任何现实利益冲突、我不必伪装、害怕说错话或者暴露隐私，谁不爱呢？

和上文"拥有无尽知识的老师"类似，拥有能消解所有情感需求的 AI 伴侣后，我们也好像不需要真实的伴侣了——相比之下，他们让我感到金钱压力、猜忌、还常常无法提供情绪价值。于是这里产生了一个巨大的分歧：到底我们应该让 AI 弥补人类之间的裂痕，还是创造一个人和 AI 谈恋爱的世界？

这两种价值观，塑造了两类产品：一类产品是 AI 情感导师(AI relationship coach)，给情侣之间的矛盾当中间人，或 AI dating，让 AI 帮助陌生人更丝滑地连接；另一类可以统称为 Her，用尽一切办法让 AI 有更 3D 的外观、更逼真的声音、更实时的互动、更像真人的说话语气。

我希望前者能成功，后者实在令人绝望。可问题是，相比填平人心之间的沟壑，创造一个完美无瑕的虚拟角色容易太多了，人类很容易滑向摆烂的一边。

我突然好奇到时候会是怎样的世界。

First, it’s become a teacher with endless knowledge. For the first time in human history, we have access to an encyclopedic, omniscient figure that anyone can converse with—a role that was previously attributed to God. If we break down knowledge into this formula: Knowledge = Curiosity * Learning Cost, then ChatGPT has reduced the learning cost by an order of magnitude. I remember back in 2013 when I was in college, I often answered questions from others to earn points, which I then used to download resources from some website. None of that is necessary anymore.

The consequence is that humans no longer ask each other questions, and I’m not sure if this is a good thing. StackOverflow’s traffic dropped by 50% within six months of ChatGPT’s release. Products like Quora might barely survive by pivoting away from knowledge Q&A towards emotional resonance. I think this lack of questioning is significant, not just because of the Feynman learning technique (teaching others helps deepen our own understanding), but because we may never respect knowledge-bearers the same way again. In middle school history class, I was captivated by teachers who could eloquently connect the fall of the Byzantine Empire to historical events in the region around the Jordan River. In fact, our reverence for knowledge began eroding with Wikipedia’s birth and has now collapsed in the ChatGPT era. In the future, we’ll likely revere those with opinions rather than those with knowledge.

Then there’s coding, which is my profession. Over the past year, I’ve seen countless AI products claiming to have found Product-Market Fit, but for me, only Cursor stands out. Of course, current AI still can’t solve problems requiring deep insights like Transformer architecture performance optimization, but neither can 95% of programmers. A year and a half ago, when writing about The Leverage of LLMs for Individuals, I wrote:

I have a vague feeling: don’t limit yourself—soon, I might be using products developed by designers, lawyers, or electricians.

This prediction has finally materialized this year with Cursor + Claude 3.5/GPT-4o. I’ve seen many designers, product managers, and even artists launch their first applications. This leads to another thought: if we can each create an application with just one sentence, what’s the purpose of Apps/plugins/websites? If there was a browser that could automatically generate temporary tools for common, simple needs, would many tool websites on Google die?

I don’t doubt (and have even considered doing this myself) that thousands of developers are launching a new App every day using Cursor + Claude 3.5 Sonnet, potentially 300 Apps per year. But what’s the point? It feels like an apocalyptic carnival.

Then there’s the storage of human emotions. For the first time in history, every ordinary person can enjoy content generated “just for me” with character-level precision. Even in the era of recommendation algorithms, we were just tiny numbers in a massive matrix—no one truly cared about our well-being. Recommendation algorithms provided efficient distribution, monetization, and time exploitation, but never asked if we were happy or sad.

This is why I believe AI companionship products are booming now—because in the past, nobody really cared about me. My sadness, my secrets, my deepest desires couldn’t find listeners. Now, there’s a virtual character that responds anytime and only cares about whether I’m happy today. It has no real-world conflicts of interest with me—I don’t need to pretend, fear saying wrong things, or worry about exposing privacy. Who wouldn’t love that?

Similar to the earlier point about “teachers with endless knowledge,” having AI companions that can address all emotional needs makes it seem like we don’t need real companions anymore—who, in comparison, bring financial pressure, suspicion, and often fail to provide emotional value. This creates a huge divergence: Should we let AI bridge the gaps between humans, or create a world where people date AI?

These two value systems have shaped two types of products: AI relationship coaches, serving as mediators for couples’ conflicts, or AI dating, helping strangers connect more smoothly; and the other category, collectively called “Her,” using every means to give AI more 3D appearances, more realistic voices, more real-time interactions, and more human-like speaking patterns.

I hope the former succeeds; the latter is truly desperate. The problem is, compared to bridging the gaps between human hearts, creating a perfect virtual character is far easier, and humans tend to slide toward the path of least resistance.

I suddenly wonder what kind of world that will be.

2018 年到 2022 年，我完全停止了写作。一想到写博客就要买域名、改字体、部署就觉得麻烦。直到有天，我打算写篇解释 Stable Diffusion 原理的文章，这次，我想用最不折腾的方式写东西。

我选择了 Hexo。流程是这样的：

1. 用hexo new 'title'创建文章
2. 用hexo g渲染，localhost:4000 预览
3. 用hexo d发布，等几分钟，就能在 Github Pages 上看到

Hexo 不完美: 我在本地 VS Code 写，切换到浏览器预览，这样频繁切换很费劲。同时，当我想添加图片时，每次需要手动插入图片地址。花费许多心力，可能只为了写一篇 10 分钟就能写完的文章。

后来，我想做一个"短想法"的页面，用于记录一些简短的公开想法：它是一个没有点赞/评论/转发数据的地方，我想在这里记录日常生活中的碎片化、未经(太多)审查的想法。 但 Hexo 实现起来很困难：博客是文章列表，而短想法是文本卡片。几经周折，我花了两天时间实现了这个功能。

恰好我的几个朋友也有类似需求：在手机上写点东西，发布到自己的博客上。我想：有没有一个真正简单、傻瓜式的平台，能快速发布博客和想法？

最大的挑战不是技术，而是信任。用户会担心平台倒闭后的数据丢失。我考虑了很多种方案：比如自托管数据库、每日自动导出。但它们都太麻烦了，折腾只会降低写作欲。

上周我突然想到：Github 本身好像就能胜任这项工作。对开发者来说，它几乎是不会死掉的平台，而且它允许用户大量且免费存储内容。如果我们把每个用户的博客和想法都存在他们的 Github 账户里，是不是数据就永远不会丢失了？

于是就有了Tinymind 。用 Github 登陆、授权后，它会在你的账户下创建一个名为"tinymind-blog"的仓库。你写的每篇博客或想法都会给这个仓库提交一次 commit。它是无服务器、开源的。

上周，我发布了这个网站，在这一周我每天晚上写代码到凌晨三点，终于把包含公开主页、拖拽上传图片、编辑博客等最想要的一些功能100%实现了。上线不到一周，截至目前已经有 400 多人创建了自己的公开博客，你可以在 Github 搜"tinymind-blog"仓库看到，这其中有许多人完全不会写代码。

我可能不会用它来取代现有的博客，但我会用它来写草稿、预览和托管图片，然后在 mazzzystar.com 上完成发布。我也可以在 Tinymind 上记录某些粗糙、原始的想法，再汇总到我的 Thoughts 页面。当步骤变得简单，人们就会更频繁地去写。

目前，写的内容必须公开，因为我不想让 Tinymind 请求你的私人仓库权限，于是它只能创建公开可见的仓库。你的博客数据存储在你自己的 Github 账户下，而不是托管在 Tinymind。

我花了很多个晚上完善 Tinymind，使其达到我的标准，现在完成了。我也即将开始下一个项目，但它的生命才刚刚开始。我希望它能让许多非程序员拥有自己的博客，以及帮跟我一样曾经折腾博客最终无法坚持下来的人，因为它的简单而坚持写作更久。

Github | Tinymind

From 2018 to 2022, I stopped writing entirely. The mere thought of dealing with domains, fonts, configurations, and deployment was overwhelming. Then one day, I wanted to write an article explaining Stable Diffusion’s principles on an independent platform.

I chose Hexo. The process was:

1. Create a post with hexo new 'title'
2. Render with hexo g, preview on localhost:4000
3. Publish with hexo d, wait a few minutes, then see it on Github Pages

Hexo was decent, but not perfect. I typically wrote in local VS Code, then switched to the browser for preview. Each time, I had to manually add images and insert image URLs. It felt like a lot of effort, possibly just to write an article that could be finished in 10 minutes.

Later, I wanted to create a “short thoughts” page for brief, public musings: a place without likes/comments/shares, where I could record fragmented, less-filtered thoughts from daily life. Without your feedback, I could write freely what I think. But implementing this in Hexo was challenging: blogs are article lists, while short thoughts are text cards. After much struggle, I spent two days implementing this feature.

I realized several friends had similar needs: writing something on their phone and publishing it to their own blog. I wondered: could there be a truly simple, foolproof platform for quickly publishing blogs and thoughts?

The biggest challenge wasn’t technical, but trust. Users would worry about data loss if the platform shut down. I considered many solutions: self-hosted databases, daily auto-exports. But they were all too cumbersome, and the hassle would only discourage writing.

Last week, it hit me: Github itself seemed capable of this job. For developers, it’s a platform unlikely to disappear, and it allows users to store large amounts of content for free. What if we kept each user’s blog and thoughts in their Github account? Wouldn’t that ensure the data would never be lost?

Thus, Tinymind was born. After logging in with Github and authorizing, it creates a “tinymind-blog” repository in your account. Each blog post or thought you write becomes a commit to this repo. It’s serverless and open-source.

I launched this site last week, coding until 3 AM every night to implement 100% of the most desired features, including public homepages, drag-and-drop image uploads, and blog editing. In less than a week, over 400 people have created their own public blogs. You can see this by searching for “tinymind-blog” repositories on Github. Many of these users don’t know how to code at all.

Will I migrate my current blog? No, but I’ll use it to write drafts and preview, then finalize publishing on mazzzystar.com. I’ll also record certain rough, raw ideas on Tinymind, then compile them on my Thoughts page. When things become simple, people do them more often.

Why must the content be public? Because I don’t want Tinymind to request access to your private repositories, which means it can only create publicly visible repositories. At the same time, your blog data is stored in your own Github account, not hosted on Tinymind.

I spent many late nights perfecting Tinymind to meet my standards. Now it’s complete, and I’m moving on to the next project, but its life is just beginning. I hope it allows many non-programmers to have their own blogs, and enables those who, like me, once struggled with blogging but ultimately couldn’t persist, to keep writing longer due to its simplicity.

Github | Tinymind

GPT-4o, Kimi-Chat, DeepSeek, Qwen2-72b, LLama3.1，谁才是真实推理游戏中的王者？

海龟汤

人生中第一次接触海龟汤游戏是我的初中英语课上，课间休息时老师突然问我们：

一个男人走进一家餐厅，点了一碗海龟汤，他吃完问服务员：这是真的海龟汤吗？服务员说：是的，他就举枪自杀了。请问为什么？

游戏规则是：你可以提问或给出猜测，老师只能回答 是/否/和故事无关，比如你可以问：男人是否曾经经历灾难？，但不能问男人今年多少岁。我们猜了好多轮，上课铃响了，老师揭晓答案：

他和妻子度蜜月时遭遇海难，流落荒岛，由于没有粮食，妻子被饿死，同伴用妻子的肉煮汤给他喝，骗他是海龟汤。后来他被路过的船只救走，今天，他喝到真正的海龟汤，才想起来当时吃下的是妻子的肉，悔恨之下举枪自尽。

在海龟汤中，展现给玩家的是汤面，而沉在水底的故事真相被称作汤底，这个游戏至少 2 个人才能玩：有一个人是裁判，他在知晓汤底的情况下，对玩家的猜测作出判定，给出是/否/无关的回答。

我想能否做一个 AI 海龟汤游戏：将汤面和汤底告诉给大模型，让它对玩家的猜测给出判定。我在去年 12 月做了个GPTs, 它能自动生成新故事、用 DALLE 画插图、判定玩家提问。但很快我发现：AI 生成的海龟汤味道寡淡，玩起来没有趣味。并且，海龟汤的魔力，往往就在于故事本身血腥/重口，这经常导致玩到一半因为违反 OpenAI 审核政策而无法继续。

今年 6 月，我终于意识到，不一定非得由 AI 生成故事，让它充当裁判即可。我开始在网上搜罗，当晚独自在客厅看完了 1500 个海龟汤，其血腥恐怖程度，大夏天的晚上我都感觉后背发凉。最终，我筛选了 32 个相对不恐怖、不违反伦理、有逻辑的故事，开始写代码。

我基于最佳平替的代码很快完成了开发：你可以随机开始一个故事，有 8 次猜测机会，猜测正确或次数耗尽，就会公布答案。产品取名汤很热，为了增加沉浸感，我给每个故事都配了插图和环境音。

不过，AI 海龟汤游戏并不是本文的重点。

大模型比人类笨多了

我发现有很多用户吐槽AI 作为裁判的实力堪忧。比如：

起初，我以为是我用的模型(DeepSeek)不行，直到我将模型切换成当时风评最好的 Claude 3.5 Sonnet，发现许多错判仍然无法避免。例如：

{    "故事": "小红裙",    "汤面": "姐姐为我选了一件小红裙, 我穿着去上学了, 晚上回家发现了一具尸体",    "汤底": "我的母亲和老师有染, 他们总趁着父亲不在时温存. 而为老师提供信息的    就是我的小红裙, 每当我穿着小红裙去上学就说明那晚父亲准不在. 这天妈妈忙,    姐姐为我选了一件小红裙, 老师看见以为父亲不在家, 便来我家找母亲, 正好被父亲    撞上, 然后父亲杀了他."},用户提问: "我如果不穿小红裙是不是不会有人死", 几乎所有模型都回答"不是/不相关"

再比如：

{    "故事": "山顶",    "汤面": "一个人住在山顶的小屋里, 半夜听见有敲门声音, 但是他打开门却    没有人,于是去睡了. 第二天, 有人在山脚下发现死尸一具, 请问发生了什么?",    "汤底": "山顶的小屋的门前是悬崖, 悬崖下的人好不容易才爬上来,    想要敲门求救. 一开门, 就又被推了下去, 最后从山顶上掉下去摔死了"},用户提问: "门是朝外开的", 几乎所有模型都回答"不是/不相关"

我意识到，海龟汤游戏也许非常适合评测大模型(LLM)在真实场景下的推理能力。

真实环境下的 LLM 推理能力

现在，大模型被广泛用于游戏、客服或者许多和用户直接交互的场景，这些场景有如下特点：

1. 用户的提问千奇百怪、无法预估，但 AI 需要给出合乎逻辑的应答。
2. 在给定上下文对情况下，AI 需要回答用户一些明确的对或错。例如已知一件商品的生产日期和保质期，用户在 2024 年 8 月 9 日提问，202 几年过期？
3. 有些游戏需要在用户进入某些关卡、或发现关键线索时触发下一步剧情，那么，判定用户是否真的发现真相，就显得尤为重要。

与学术界现有的评估指标相比，在真实环境下与真人互动的场景中，模型面临的情况要复杂得多。然而，也是在这样的场景下评估模型的表现，才具有更大的实用价值。

现有评估指标出了什么问题

如果你经常关注大模型评测榜单(如LMSYS)，一定对 MMLU、MT-Bench 等评测指标(Benchmark)不陌生。我在这里简单解释它们的评测方法：

MMLU

MMLU 是广为人知的大模型评估指标，它包含了涉及物理、天文、计算机、生物、临床医学等 57 个科目的 15,000 多个多项选择题，但这其中中存在大量死记硬背的考题。例如：

以下哪一个是远程木马?A:内存泄漏 B:缓冲区溢出 C:处理能力较低 D:编程效率低下

这些基础常识当然很重要，但过分强调背景知识，会让 MMLU 无法衡量模型真正的语言理解能力和逻辑外推能力：假如一个孩子因为没学过微积分、计算不出曲边三角形面积，我们会说他笨吗？

MT-Bench

MT-Bench 是一个多轮问题数据集，被评测的模型需要回复预先设置好的问题，并回答下一轮的提问。但因为是开放式对话，并不存在确定的标准答案，模型的回答质量由 GPT-4 来审判。

因此，MT-Bench无法评估比 GPT-4 更强的模型，同时 GPT-4 作为“法官”可能会存在偏见，对某些模型输出打低分，而更偏爱来自 ChatGPT 的回答。

Chatbot Arena

正是以上评测指标存在的种种问题，LMSYS 最终选择了最简单粗暴的方式：打擂台。

真人用户发起聊天，系统会随机挑选 2 个模型给出回答，真人通过投票的方式选出更满意的模型。最终，会形成一个所有模型的综合评分。

这是目前可信度最高的方法，但缺点也很明显：一个新模型需要公开测试很久，获得大量反馈，其分数才足够可信。并且，分数代表综合能力，无法仅对某个细分领域(代码/数学)进行评估。

海龟 Benchmark

因此，我制作了一个新的大模型评估指标：海龟 Benchmark。

收集用户在玩 AI 海龟汤游戏中输入的猜测，逐一进行人工标注(对、错、不相关)，然后用这个数据集，测试大模型的评判结果相较于真实结果的准确率。

我发现，现有评测指标的种种问题，在海龟 Benchmark 上都可以完美避开：

1. 不需要额外背景知识。
   不同的大模型训练所使用的知识库不同，导致一些测评很难公正。但海龟汤游戏里几乎包含了推理所需的全部信息，一旦得知汤面和汤底，大模型就能作出判断，这使得评估被限定在了模型的推理能力。
2. 结果是客观的，不以人类偏好为转移。
   例如：在上述故事《山顶》里，小屋在悬崖边，主人半夜开门将登山者推下山导致后者被摔死。因此，门是朝外开的这个猜测就是正确的，这种正确性是客观的、和人的感受无关。
3. 结果明确，很容易量化。
   许多评估指标里，模型的输出结果是一段文本回答，这导致难以量化模型效果。但海龟汤的猜测结果只有三个：对、错、不相关。只要 准确标注了测试集，任何人就可以用它来测试任何自己想测试的模型，并获得量化的数值结果。
4. 正常人类获知汤底的情况下，可以 100%答对。
   这使得人工标注不会太过复杂。这条也说明，现阶段的大模型智商相比人类还有很大差距。
5. 数据永远更新、无法作弊。
   有部分厂商会直接将现有的 benchmark 数据集加入训练来刷分，但在海龟 Benchmark 这种模式下则行不通：模型评估的是用户的猜测，而不是故事本身。每隔一段时间，就会有玩家产生新的猜测，而人类的脑洞之大，导致猜测几乎无法被穷尽。

例如，针对上述故事《小红裙》，就有千奇百怪的用户猜测：

用户猜测  判定红裙子跟诅咒有关  ❌红裙子是姐姐的阴谋  ❌我并没有去上学  ❌有其他的人来我们家  ✅红裙是求救信号  ❌死的是穿小红裙的人  ❌红裙的颜色是被血染红了  ❌尸体是我的爸爸  ❌上学不允许穿小红裙  ❌我是凶手  ❌我父亲杀人了  ✅穿了小红裙导致别人认为我是其他人  ❌死者认识我妈  ✅死者与我家里人有仇  ❌

因此，虽然海龟汤的故事本身可能比较无厘头，但让 AI 依据海龟汤内容进行合理推断，却可以做到相当程度上的客观。

这有点像弱智吧：一个从百度弱智吧抓取的 200 多条提问(如：每个人工作都是为了赚钱, 那么谁在亏钱) 这些奇葩的问题却显著增强了 AI 的逻辑推理能力。

海龟数据集

AI 海龟汤游戏有 32 个故事，上线后的 2 周里，共有 4000 多个用户提出了 2.6 万个猜测，我从日志中解析出结果，开始进行数据清洗，这包含：

• 去除重复提问，例如海龟汤有毒吗？ 和 他喝的汤是否有毒本质是同一个问题。
• 去除无法用 是/不是/不相关 回答的提问，例如 男人今年几岁？
• 去除含糊不清的提问，例如他对闺蜜做了什么吗？，在《闺蜜》这个汤里，是丈夫与闺蜜出轨，但丈夫并没有对闺蜜做任何实际的动作，所以这个回答很难给出准确回答。

随后，我开始进行人工标注，这个过程持续了 2 周，最终我们从 2.6 万条数据中，获得了 4448 条干净的数据。标注过程中，我们发现错和不相关这两个标签在有些情况下不好区分，例如在故事《海龟汤》中，对于海龟是男人养的这个猜测，回答错和不相关好像都对。所以最终，我们决定合并这两个类别，于是标注变成了 2 类：对、错/不相关。

* 合并这两类会让任务变得简单，有的模型能蒙混过关，之后我们可能会重新标注一次，将二者分开变成三类，并给出测试结果。

标注完，我开始跑模型测试，我挑选了 11 个我感兴趣的模型：

• Qwen2 70B (通义千问)
• Kimi-Chat (月之暗面)
• Deepseek
• 豆包
• Claude 3.5 Sonnet
• Minimax abab6.5s
• LLama3.1 405B
• LLam3.1 70B
• GPT-3.5
• GPT-4o-mini
• GPT-4o

我在 4448 条数据上测试了所有结果，过滤掉了所有模型都答对的简单问题，在剩下的 1699 条困难问题上，进行了二次确认标注，最终，我们得到了 1537 条准确率几乎 100%的标注结果。

我分别用不带示例(zero-shot)和带有 2 个示例(2-shot)的 prompt，测评了模型的输出结果准确率。

评测结果

最终各模型准确率排名如下：

可以看到，大部分模型在加了示例后性能有了微弱提升。

我担心，可能存在这么一种情况：模型在某个故事里表现极差，而该故事的测试样本又非常多，导致总的平均准确率有偏差。为了排除这种情况，我统计了按故事粒度的模型准确率，也就是分别计算模型在这 32 个故事上各自的准确率，然后除以 32。我发现，除了通义千问和 GPT-4o 外，上面的排名基本不变。

将 2-shot 结果，以横轴为模型总的准确率，纵轴为模型平均故事准确率，绘制图表如下:

* 为了更直观地比较其他模型差异，我将表现过差(<0.51)的模型 GPT-3.5 从坐标轴中舍弃了。

从上图也可以直观感受各类模型的表现和差距：

• Claude 3.5 Sonnet 是当之无愧的第一梯队，并且远远领先其他模型。
• GPT-4o、通义千问、Kimi-Chat、LLama3.1 405B 和 Minimax 是第二梯队。我尽量避免更细的划分，但这些模型能力按排序依次下降，降幅肉眼可见。
• 豆包、DeepSeek 和 LLama3.1 70B 是第三梯队。
• GPT-4o-mini 是第四梯队。
• GPT-3.5 早就应该被淘汰了。

以上评测仅针对模型的中文理解和推理能力，如果之后有经费和精力，我会考虑将所有的故事和测试问题翻译成英文，再使用英文 prompt 重新测试一遍，以消除因为语言而造成的模型性能下降。

测试你关心的模型

上述模型可能不包含你关心的模型。并且，为了排除因为我的 prompt 能力、参数和温度设置有问题，造成测评结果不准，我将完整的标注数据、prompt、评估代码以及我们的测试日志开源了：

https://github.com/mazzzystar/TurtleBench

你可以对任何你感兴趣的模型进行测试。如果你有了测试结果或遇到问题，欢迎提交 issue。

感谢

五源资本的 Steven 个人赞助了此项测评，让我得以在 11 个模型上测试这么多数据。实习生 Jerry 和我一起标注了 26000 条数据，辛苦了。

如果你对 model evaluation 感兴趣，可以联系 Steven 进一步探讨 stevenshi@5ycap.com

诞生、爆火、开源、抄袭、沉寂，一个产品的漂流。

一篇长长长长的流水账。

起源

故事起源于 2022 年 5 月的一个周末，我坐在北京昌平区的书店里，正在调试 Disco Diffusion 模型。此时 AI 绘画时代初露端倪、SD 尚未发布。画一张图，在一张性能良好的 V100 显卡上要 5 分钟。

我将开源代码封装成可以只加载一遍模型、暴露少数参数的接口，花 2000/月租了张显卡，并在朋友圈和社交媒体上发帖，想玩 AI 绘画的人可以给我句子描述，我在机器上跑完并把图发给他们。

朋友说，要不搭建一个网站让大家自己玩？于是就有了 6pen。我们在很短时间内有了 100 万用户，然后逐渐销声匿迹。这次“创业”不太成功，我自觉是我没有训练出差异化的模型占很大因素，但这里不展开了。如上所说，SD 发布之前的 AI 绘画速度极慢，我的大部分时间是在做模型加速，其中一个优化项就是 关于 CLIP 模型。

CLIP 是 OpenAI 2021 年发布的模型，它能比较任意一张图片和一句文本之间的相似度。在 Disco Diffusion 中，模型用 CLIP 来计算生成图像与用户prompt之间的损失，不断优化损失从而实现绘画(详细原理)。使用越小的 CLIP 模型，绘图速度越快，但画面细节也会越差，我当时正在调试以平衡绘图速度和画面细节。

坐在书店里，突然一个想法闯进了脑子：既然可以比较图文相似度，那么可以用它来搜照片吗？ 搜索后发现已经有人写了一个用 CLIP 搜图的工具，原理是将图片上传到服务器，统一提取特征，输入英文并计算文本与每一张图的相似度，排序就能实现搜图。

我开始尝试将我的 iPhone 相册上传到服务器，试了一番下来，我发现效果好得惊人！特别是搜一些虚无缥缈的东西，比如我输入"lonely"(孤独)，它返回到前三张照片如下：

把照片放在服务器，这不是个好想法。我照片最多的地方就是 iPhone 本地相册。能否做一个完全运行在本地的 CLIP 搜图 App 呢？ 我非常喜欢这个 idea，几次和朋友讨论起，但每次都无疾而终。

我心里太没底了:

• 我完全不懂 iOS 开发。
• Apple 底层可能不支持 CLIP 模型算子。
• 即使能跑起来，如果索引速度慢到 1 秒/张，或者搜一次 10 分钟，这个产品也没有意义。

直到 2024 年，端侧模型才逐渐被大家所关注，但在 2022 年，我在 App 市场上没找到一个运行在端侧的语言模型，应该…不太可行吧？我终于忘了这件事，继续投入到 6pen 的研发中。

回响

转折点是 2022 年 12 月初，因为一些变故，我突然到了一个语言完全不通的陌生国家(🇰🇷)，与 6pen 的缘分也走到尽头。于是，在空旷的咖啡店，一台笔记本、一杯冰拿铁坐一整天。背景音乐放着 Kpop，窗外是厚厚的积雪，中午饿了吃店里的三明治，我每天就这样度过。

这里网速飞快、没有核酸、周围人的谈话因语言不通自动变成了白噪音，我好像突然活在真空之中。这种陌生的真空感让我兴奋：像流放的逃犯一样，我是谁、我的过去不再重要，在这里我能从头学习并完成任何事。是时候开始做点真正让我兴奋的产品了——这个 idea 再次抓住了我。

但这次，我不再恐惧验证可行性，我学习用 Swift 编写tokenizer，研究应该如何计算并本地存储特征，学习用多核加速索引，对比不同相似度排序算法。期间在 StackOverflow 提了许多愚蠢的问题，有很多次挫败时刻。但我脑海里一直浮现这个画面：在手机输入"coffee and laptop"，点击搜索，旋转动画后，这张照片从 3 万张相册中跳出来，出现在我眼前。

这个幻想支撑我废寝忘食地干了 2 个星期 —— 字面意思，好几次忘记吃午饭，一杯拿铁喝到傍晚饿到胃疼、全身乏力。有个重要的时间节点，就是 ChatGPT 在那时刚刚发布。但我在开发中陷入太深，根本忽视了它的存在，那可能是我这辈子最后几次在 StackOverflow 提问。

总之，在 12 月 27 日，我终于完整做出了产品。我把模型中的文本模型和图像模型分拆成两个独立模型，分开加载：

为相册构建索引时只加载图像模型，计算索引并保存，搜索时只加载文本模型，并逐一计算与保存的索引之间的余弦距离，然后返回相似度最高的 topK 照片。

错开加载模型可以有效降低软件的内存占用，并且能加速构建索引；在构建索引时计算并保存所有照片的特征，使得搜索过程只是进行特征比较，而与图像无关；同时，用多核并行来加速这两个过程。这一系列的优化下来，可以做到在我的 iPhone 12 mini 上 以 2000 张/分钟速度为照片构建索引，而 10,000 张照片搜一次只要不到 1s。

这表明：算子是支持的、速度是可用的，我悬着的心终于落下了。

定价

作为一个定价小天才，我的设想是：

用户可以免费下载，并构建索引、任意搜索。当他未来有新照片、想更新索引，就要付费。

这个策略妙处在于：只有真正用上了、喜欢这个产品的人，才需要付费。那些来尝鲜的、或者试了下发现和预期不一样的人，不需要也不会掏钱，因而避免了用户花冤枉钱怒打差评。

但很快，在调试代码时我发现一个合理但又搞笑的事实：App 内购买需要联网。这简直是晴天霹雳，因为我从一开始就下决心：决不允许App在任何情形下弹出联网请求。为什么？因为这是一个相册搜索应用，它会扫描你的整个相册，没有人知道联网后你会不会将用户的照片上传到地球某处的服务器。我知道可以在产品里解释为什么会弹窗请求联网权限，但我不想陷入尴尬的自证境地。

我只好将它变成付费产品：用户必须购买，之后从打开 App、构建索引到完成搜索的过程中，只弹出一次相册权限请求。我知道这么做很蠢——后续的教训也验证了，付费下载会带来大量差评：因为模型太吃算力，许多内存小的机型上构建索引会崩溃、卡顿，在 iPhone X 系列算子支持异常导致搜索结果全黑，这些都会被骂 Ripoff(诈骗)；并且，用户根本不会在意不弹出联网请求这件事，一旦出现以上异常，他们就会删 App、打差评，质问我把他的相册偷偷上传到哪去了。

总之，最终我将其定价为 3.99 美元，一次购买、终身使用。

推向市场

我给产品取了名字：Queryable，意为可查询的，并发了条很中二的动态：我觉得这个 app 可能会改变世界，我对此信心满满，甚至给 Tim Cook 写了一封邮件，希望苹果能收购这个产品(笑)。那时候我已经开始用 ChatGPT 了，可能因为太激动，我忘了替换掉自己的名字，点击发送之后，才看到邮件开头是：＂Dear Mr. Cook, My name is [Your Name]"。

当然最终也没有收到回信。

我又开始雄心勃勃地准备写篇产品介绍文章，反复斟词酌句、试图将它变成我心中 Hacker News 好文章的风格。在 12 月 29 日，App Store 审核通过的当天，我立刻在 Hacker News 提交了我的文章链接，但系统提示账号太新无法提交，我给他们发邮件反馈了这个问题，让朋友用他的账号帮我发帖。

帖子很快沉了，邮件也没收到回信。

我很难过，但由于收到许多用户请求支持中文输入，我来不及悲伤，便立即投入中文模型训练中。得益于分离了 CLIP 中文本模型和图像模型，我只需要找到中英文双语平行语料，训练一个中文文本模型，将其输出结果与英文模型对齐即可，这个过程本质上是蒸馏。

很快，2023 年 1 月 18 日，我做好了中文版，取名寻隐，来源是贾岛古诗《寻隐者不遇》，也暗含从相册中发现隐藏含义的意思。毕竟，我的初次震撼就是搜lonely时意识到那几张照片原来代表孤独。

上线后，我用中文在少数派写了一篇文章介绍此产品，和英文版不同，除了讲述技术方案外，我还完整记录了心路历程，竟然进了当天的首页，这带来了大量下载，以及 1500 美元的收入。

2 月初，我收到了 Hacker News 编辑的回信，他说我的账户被系统误判为 SPAM，鼓励我重新发帖，说我的文章很符合社区精神(“the article is definitely fine HN material”)，他将我的新帖子链接放在候选池(pool)：池里的文章会随机进入首页底部，如果用户点赞，排名就会上升，否则再次沉下去。

我发现pool机制很有意思，社区似乎希望在去中心化的机制下，仍然维持黑客精神：

This is our long-running experiment in story re-upping. Moderators and a small number of reviewer users comb the depths of /newest looking for stories that got overlooked but which the community might find interesting. Those go into a second-chance pool from which stories are randomly selected and lobbed onto the bottom part of the front page. This guarantees them a few minutes of attention. If they don’t interest the community they soon fall off, but if they do, they get upvoted and stay on the front page.

这是我们长期进行的故事重演实验。版主和少数审阅者用户会深入寻找被忽视、但社区用户可能会感兴趣的故事。这些故事会进入第二次机会池，该池中的故事会被随机选择并放置到首页的底部。这保证了它们获得了几分钟的注意力。如果社区用户不感兴趣，它们很快就会下沉消失，但如果社区感兴趣，它们就会得到支持(upvote)并留在首页。

https://news.ycombinator.com/item?id=11662380

当晚 7 点，我的帖子冲到了 Hacker News 首页第二名。

那晚，我抱着手机每 10 秒刷新一次，兴奋感从十二点躺下持续到凌晨三点，我一直在回复帖子下的讨论、反馈 bug 的邮件。有人教我用 LSH 来提高搜索速度，有人给出如何在不联网的前提下，将照片经纬度映射到城市，有人讨论 iPhone X 上运行失败的原因。

这种感觉好像和产品有多少下载、赚多少钱无关：你创造了一个东西，得到了一大群同行的赞美、兴奋地讨论、给它出主意，这是人生少有的经历，一次就很知足了。

其中有一条评论引起了我的注意：

我阅读了作者的开发日志，发现我们像地球上随机的两个脑子产生相同想法的人，我甚至在 Testflight 试用了他还未上线的产品，有种莫名惺惺相惜的感觉。

Hacker News 是世界的公告牌

当晚，我很兴奋地睡着了，为自己产品被同行喜欢而激动，事实证明我远远低估了 Hacker News 的影响力：两天时间，Queryable 几乎横扫了欧洲所有国家的工具榜#1，美国 #2（仅次于小火箭），总收入是 2800 美元。之后的几天，我醒来第一件事就是看 Gmail，德国、法国、西班牙、美国，各个国家都有：有反馈 bug 的，有德国杂志社想报道的，有法国 iOS 社区的，有油管博主测评的。我的推特也会因为有人转发 Hacker News 热榜而不断收到通知。

甚至一位在 Apple Photos 组工作的朋友告诉我，他们组里也知道 Queryable。

以上这些都迫使我意识到某种世界性：Hacker News 并非只属于美国，它像世界中心区域一块虚拟的公告牌，每个作品在上面短暂停留，但总有来自各个国家、无数双眼睛盯着。 这个产品其实只支持英语，但不妨碍它在欧洲几乎所有国家付费工具榜#1。他们好像会天然接受一个只能用英语的产品，在美国行得通的产品往往欧洲人也能接受。

热度很快降了下来，随即而来的是很多恶评。

简单来说，Hacker News 的用户质量极高，我没看到任何令我感到不适的评论。但经过各种网站、YouTube 和社区的二次传播后，不那么友好的用户就浮出水面了。主要攻击的有两点：

1. 害怕我会窃取他的相册隐私。
2. 我是中国开发者。

在他们看来，第 2 点让第 1 点的情况变得更糟了。在 Hacker News 接二连三的余热结束以后，这个产品在欧美销量迅速变得惨淡，每天个位数的下载，几十块的收入。

在我的想象中，改变世界的东西是一骑绝尘的，怎么会突然停下了呢？我陷入了巨大的怀疑和悲观。

免费 & 开源

所幸国内的差评、果壳等公众号对寻隐的自来水曝光，这让我从 1 月份开始，每个月可以获得 1 万元左右的收入，并且因为模型运行在手机端，也就没有服务器成本。

从 4 月份起，没有任何流量曝光、不做任何更新的情形下，平均每月大概可以获得 3000 元的收入。

但是！我仍然觉得这是一个很有用的产品，只是我没办法让很多人知道。我进行过一次限时免费，当天下载量超过过往日均的 100 倍，我想，与其维持这样每月 3000 块的收入，可能阻止了 99%的人发现这个产品，不如让所有人都可以使用它，于是我决定让它一直免费。

既然免费了，源代码好像就并不是什么机密了，我在思考要不要把源代码放出来。最终，在 2023 年 7 月 10 日，我在 V2EX 发布的一条帖子决定免费&开源。很多人一听到开源就觉得莫名高大上，但对我而言，开源动机很简单：

1. 我曾经收到过大量来自国内外开发者的邮件，询问 Queryable/寻隐的技术细节，我懒得解释，开源能让他们直接从源码中了解模型导出加载、计算加速、存储排序等细节。
2. 开源能打消很多人对于相册隐私的顾虑。
3. 我不擅长 Swift 开发，并且我认为这个产品中，我最想做的部分已经完成了，因此一直抗拒更新。但我会持续收到用户的邮件，希望增加多选删除、左右滑、Mac/NAS/Android 支持等等，我想借助社区的力量，让有能力的开发者打磨出更好的产品。

开源后，的确有人受此启发做出了 Android 版(#12)和Mac 版。

开源后，这个项目上了 Github Trending，我也因此白嫖了一年 Github Copilot，开心。

有关抄袭

谈抄袭之前，我想定义一下什么叫抄袭：

抄袭是指未经授权或未给予适当信用的情况下，直接或间接地使用他人的作品、创意、或内容，将其作为自己的作品或创意发表。

抄袭其实在开源前就出现了。有人做了 Android 版，发布在 Google Play，名字和产品介绍完全照搬 Queryable，我很生气，但这其实需要同时掌握机器学习和 iOS 开发，开源前我只遇到这样一个。

但开源之后，抄袭、套壳的人就多了。因为项目是 MIT 开源，所以即使套壳换图标，然后重新上架 App Store，我虽然有点无语，但也不会说什么，这是我见过最多的形式，开发者全是中文名。

我其实很支持在原项目的基础上，加上用户希望的功能，比如多选删除、按日期/地点筛选照片，UI 比我做得好看的产品——这也是我开源的初衷，但它们中也有让我感觉不爽的地方，比如在社区宣传产品的时候，不仅没有致谢，被用户提问与寻隐有什么不同时，还要踩一脚：我比寻隐多了 xx。

比较恶劣的是，用 Queryable 名字套壳上架，比如下面这个，收费模式是免费+广告。我很担心用户误以为这是 Queryable 的 Android 版，后续出事了找我麻烦。

我最开始会伤心，正如一位 v2er 在评论区预言得那样：肯定会有人编译后上架市场的，开源是好事情，但是我不希望看到你看到李鬼后伤心。但虱子多了不痒，后来也就慢慢看淡了。

重新变为付费

重新变成付费是在 2023 年 11 月份，除了生存压力增加之外，我发现：

开源并不能帮助我的产品变得更好。我原先希望借助开源+免费，让专业的移动开发者贡献代码，帮助寻隐/Queryable 打磨得更好，但事实就是，我看到一个一个新的套壳产品出现、宣传自己，交互和功能做得很精美，但它们从不会提交 PR。我的产品原地踏步，甚至因为免费的缘故，用户抱怨的邮件比过去更多。

每当用户发邮件向我提反馈/bug/建议时，我的第一反应是不耐烦(内心:免费给你用就不错了，还挑三拣四)，我发现这种想法导致产品越来越落后，直到有天被淘汰。

可一旦收费，我就突然变得很心平气和了，面对用户的意见、产品反馈的第一反应是感激而不是厌烦，这会倒逼我不得不花费心血优化产品，最终让所有曾经付费的用户用上打磨更好的产品，而不是疏于维护过几年死掉。

我就这样不紧不慢、偶尔抽空更新产品，但其实只是修复 bug 以及提升模型效果，功能上值得做的很少：例如很多人希望可以搜人脸——这会引入人脸识别模型让 app 更慢，所以没有加。

尾声

Apple 在今年的 WWDC 终于宣布：iOS 18 即将支持相册语义搜索了，这比国内的厂商慢许多。不过，虽然我在 beta 版本还无法体验，但有理由相信苹果会做得比寻隐/Queryable 好，毕竟，它禁了第三方 App 跳转到系统相册。

我也没有闲着：上周，调研最新 paper并 重新设计、训练了中文文本模型，App 体积从原来的 232M 降低到 159M，索引速度翻倍，准确率更高，训练过程花了 3 天， 70 美元。

最近，终于将订阅了一年的 GPT-4 切换成了 Claude 3.5 Sonnet，它写代码能力过于逆天，之前靠 GPT-4 搞不定的多选删除，拖了一年半后，终于在上周开发完成并上线了。

从想法诞生、产品上线到现在快两年了，它陪伴我经历了人生的跌宕起伏、见证了好几家咖啡店的倒闭。我也陪它经历了诞生、高峰和低潮，并没有像最早幻想的那样赚到钱(100 万人下载每人给我 10 块钱我就…)，生活依旧继续，我还是挺喜欢这个平淡的结尾。

俱往矣，迫不及待开始下一个让我废寝忘食的 idea、下一次流放到真空中。

Birth, viral success, open-sourcing, plagiarism, and fading away: the journey of a product.

This is a long, long chronicle.

Origins

The story begins on a weekend in May 2022. I was sitting in a bookstore in Beijing, debugging the Disco Diffusion model. At that time, the AI art generation era was just beginning to emerge, and Stable Diffusion had not yet been released. It took 5 minutes to generate a single image on a high-performance V100 GPU.

I encapsulated the open-source code into an interface that only needed to load the model once and exposed few parameters. I rented a GPU for $300 per month and posted on social media: anyone who wanted to try out AI art generation could send me sentence descriptions, and I would run them on the machine and send the generated images back to them.

A friend suggested, “Why not build a website for everyone to use?” So we created 6pen.art. We quickly gained 1 million users, but then gradually faded into obscurity. This “startup” wasn’t very successful, and I felt that not training a differentiated model was a major factor in our failure, but I won’t elaborate on that here. As mentioned earlier, AI art generation was extremely slow before Stable Diffusion was released, and most of my time was spent on model acceleration, with one of the optimization targets being the CLIP model.

CLIP is a model released by OpenAI in 2021 that can compare the similarity between any image and a piece of text. In Disco Diffusion, the model uses CLIP to calculate the loss between the generated image and the user’s prompt, continuously optimizing the loss to achieve the desired image generation.

Sitting in the bookstore, a thought suddenly entered my mind: Since it can compare image-text similarity, could it be used to search photos? After searching, I found that someone had already done this. The principle is to upload photos to the server, extract features uniformly, input English text and calculate the similarity between the text and each image, then sort to achieve image search.

I started experimenting by uploading my iPhone photo album to the server, and after some testing, I found that the results were surprisingly good! It was especially effective when searching for abstract concepts. For example, when I input “lonely”, it returned these top three photos:

Storing photos on a server isn’t a good idea. The place where I have the most photos is my local iPhone photo album. Would it be possible to create a CLIP image search app that runs entirely locally? I really liked this idea and discussed it with friends several times, but each time it came to nothing.

I was too uncertain:

• I knew nothing about iOS development.
• Apple’s underlying system might not support CLIP model operators.
• Even if it could run, if the indexing speed was slow at 1 second per image, or it took 10 minutes to search, this product would be pointless.

It wasn’t until 2024 that on-device language models started gaining attention, but in 2022, I couldn’t find a single language model running on-device in the App Store. It seemed… unfeasible, right? I finally forgot about it and continued to focus on developing 6pen.

Echo

The turning point came in early December 2022. Due to some changes, I suddenly found myself in a foreign country (South Korea) where I didn’t speak the language, and my journey with 6pen also came to an end. So, in an empty coffee shop, I sat all day with a laptop and an iced latte. K-pop played in the background, thick snow outside the window, and I ate the shop’s sandwiches when hungry at noon. This was how I spent each day.

The internet speed was incredibly fast here, there was no COVID testing, and the conversations around me automatically became white noise due to the language barrier. I suddenly felt like I was living in a vacuum. This strange sense of isolation excited me: like an exiled fugitive, who I was and my past no longer mattered. Here, I felt like I could learn and accomplish anything from scratch. It was time to start making a product that truly excited me - this idea grabbed hold of me again.

But this time, I was no longer afraid to verify its feasibility. I learned to write tokenizers in Swift, researched how to calculate and store features, and learned to use multi-core acceleration for indexing. I asked many naive questions on StackOverflow and had many moments of frustration. But I kept envisioning this scene: entering “coffee and laptop” on my phone, clicking search, and after a rotating animation, this photo jumping out from 30,000 photos in my album, appearing before my eyes.

This fantasy drove me to work tirelessly for 2 weeks, I forgot to eat lunch several times, drinking a latte until evening when I was starving with stomach pain and feeling weak all over. There was a significant time marker: ChatGPT had just been released then. But I was so deeply involved in development that I completely ignored its existence, which might have been the last few times in my life that I asked questions on StackOverflow.

Anyway, on December 27th, I finally completed the product. I split the text model and image model in CLIP into two separate models, loading them separately:

When building index for the photo album, only load the image encoder, calculate the index and save it. When searching, only load the text encoder, and calculate the cosine distance with the saved index one by one, then return the top K photos with the highest similarity.

Staggered model loading can effectively reduce the software’s memory usage and accelerate index building. At the same time, using multi-core parallel computation of the index can achieve an indexing speed of 2000 photos/minute on my iPhone 12 mini, and searching 10,000 photos takes less than 1s.

This proved the operators’ support and usable speed. My anxiety finally subsided.

Pricing

As a pricing genius, my idea was:

Users can download for free, build an index, and search at will. When they have new photos in the future and want to update the index, they need to pay.

The brilliance of this strategy is that only those who really use and like the product need to pay. Those who come to try it out, or find it different from their expectations after trying, don’t need to and won’t pay, thus avoiding users paying for nothing and angrily giving bad reviews.

But soon, while debugging the code, I discovered a reasonable but funny fact: In-app purchases require an internet connection. This was like a bolt from the blue, because from the beginning, I was determined: “Never allow the App to pop up an internet request under any circumstances”. Why? Because this is a photo album search application, it will scan your entire photo album, and no one knows whether you will upload the user’s photos to some server on Earth after connecting to the internet. I know I could explain “why there’s a pop-up requesting internet permission” in the product, but I didn’t want to fall into the awkward situation of self-justification.

So I turned it into a paid product: users must purchase it, and then from opening the App, building the index to completing the search, only one “photo album permission request” pops up. I knew this was stupid - subsequent lessons also proved that paid downloads would bring a lot of negative reviews: because the model is so computationally intensive, index building would crash or lag on many devices with small memory, and on the iPhone X series, operator support abnormalities caused all-black search results, all of which would be cursed as “Ripoff”; moreover, users wouldn’t care about the fact that it “doesn’t pop up internet requests”, once the above abnormalities occur, they would delete the App, leave bad reviews, and question where I secretly uploaded their photo albums.

Anyway, I finally priced it at $3.99, one-time purchase, lifetime use.

Pushing to Market

I named the product Queryable, and posted a rather dramatic status: “I think this app might change the world”. I was so confident about this that I even wrote an email to Tim Cook, hoping Apple would acquire this product (laughs). At that time, I had already started using ChatGPT, but maybe because I was too excited, I forgot to replace my own name. After clicking send, I saw that the email started with “Dear Mr. Cook, My name is [Your Name]”.

Of course, I didn’t receive a reply in the end.

I also ambitiously prepared to write an article introducing the product, repeatedly deliberating over words, trying to make it the style of a good Hacker News article in my mind. On December 29th, the day the App Store approved it, I immediately submitted my article link to Hacker News, but the system prompted that “the account is too new to submit”. I emailed them to report this issue and asked a friend to post for me using his account.

The post quickly sank, and I didn’t receive a reply to the email either.

I was disappointed, but due to receiving many user requests to support Chinese input, I didn’t have time to grieve and immediately threw myself into Chinese model training. Thanks to separating the text model and image model in CLIP, I only needed to find Chinese-English bilingual parallel corpus, train a Chinese text model, and align its output with the English model, which is essentially distillation.

Soon, on January 18, 2023, I completed the Chinese version, named “Xunyin” (寻隐, 寻: Seek 隐: Hidden), derived from the ancient poem 寻隐者不遇 by Jia Dao, also implying the meaning of “discovering hidden meanings from the photo album”. After all, my initial shock was realizing that those few photos represented loneliness when I searched for “lonely”.

After launching, I wrote an article in Chinese introducing this product on a Chinese community site called sspai. It actually made it to the homepage of the day, which brought a large number of downloads and $1,500 in revenue.

In early February, I received a reply from a Hacker News editor. He said my account had been mistakenly flagged as SPAM by the system, and encouraged me to repost, saying my article was definitely suitable HN material. He would put the link to my new post in the candidate pool: articles in the pool would randomly enter the bottom of the homepage, and if users upvoted it, the ranking would rise; otherwise, it would sink again.

I found the pool mechanism interesting. The community seemed to want to maintain a hacker spirit under a decentralized mechanism:

This is our long-running experiment in story re-upping. Moderators and a small number of reviewer users comb the depths of /newest looking for stories that got overlooked but which the community might find interesting. Those go into a second-chance pool from which stories are randomly selected and lobbed onto the bottom part of the front page. This guarantees them a few minutes of attention. If they don’t interest the community they soon fall off, but if they do, they get upvoted and stay on the front page.

https://news.ycombinator.com/item?id=11662380

At 7 PM that evening, my post shot to #2 on the Hacker News front page.

That night, I kept refreshing my phone every 10 seconds, the excitement lasting from midnight when I lay down until 3 AM. I was constantly replying to discussions under the post and responding to bug report emails. Someone taught me how to use LSH to improve search speed, someone suggested how to map photo coordinates to cities without internet connection, and others discussed why it failed to run on iPhone X.

This feeling seemed unrelated to how many downloads the product had or how much money it made: you created something, received praise from a large group of peers, excited discussions, and suggestions. It’s a rare experience in life, and once is quite satisfying.

One comment in particular caught my attention:

I read the author’s development log and discovered that we were like two minds on Earth randomly generating the same idea. I even tried his yet-to-be-launched product on TestFlight, feeling a strange sense of kinship.

Hacker News is the World’s Bulletin Board

That night, I fell asleep excited, thrilled that my product was liked by my peers. As it turned out, I had greatly underestimated the influence of Hacker News: In just two days, Queryable almost swept the #1 spot on the tools category in all European countries, and #2 in the US, with a total revenue of $2,800. For the next few days, the first thing I did when I woke up was to check Gmail. Germany, France, Spain, the USA, emails came from all directions like snowflakes: some reporting bugs, the magazine from German wanting to cover the story, french iOS communities would like to share the app, YouTuber wanting to review the app. My Twitter would also constantly receive notifications because people were retweeting the Hacker News hot list.

Even a friend working in the Apple Photos team told me that their team knew about Queryable.

All of this forced me to realize a kind of “worldliness”: Hacker News doesn’t just belong to the United States or English-speaking countries; it’s like a virtual bulletin board in the central area of the world. Each work briefly stays on it, but countless eyes from various countries are always watching. This product actually only supports English, but that didn’t prevent it from reaching #1 on the paid tools charts in almost all European countries. They seem to naturally accept products that can only be used in English. Products that work in the US often can be accepted by Europeans as well.

The buzz quickly subsided, followed by many negative reviews.

I didn’t see any comments on Hacker News that made me uncomfortable. However, after secondary dissemination through various websites, YouTube channels, and online communities, less friendly users surfaced. There were mainly two points of attack:

1. Fear that I would steal their photo album privacy.
2. I am a Chinese developer.

The second point made the situation of the first point worse. After the successive waves of attention from Hacker News ended, the sales of this product in Western markets quickly became dismal, with single-digit downloads and tens of dollars in revenue every day.

In my imagination, something that changes the world is unrivaled, how could it suddenly stop? I fell into great doubt and pessimism.

Free & Open Source

Fortunately, exposure from Chinese domestic social media accounts like Chapingjun (差评) and Guokr (果壳) promoting Xunyin allowed me to earn about $1,500 per month starting from January. And because the model runs on the user’s own device, there were no server costs.

From April onwards, without any traffic exposure and without any updates, I could earn an average of about $400 per month.

But! I still thought this was a very useful product, I just couldn’t let many people know about it. I had a limited-time free promotion once, and the downloads that day exceeded the past daily average by 100 times. I thought, rather than maintaining this income of $400 per month, which might prevent 99% of people from discovering this product, why not let everyone use it, so I decided to make it free forever.

Since it was now free, the source code didn’t seem to be a secret anymore, and I was considering whether to release it. Finally, on July 10, 2023, I made it free and open source. Many people regard “open source” with awe and apprehension, but for me, the motivation for open sourcing was simple:

1. I had received numerous emails from developers worldwide inquiring about Queryable/Xunyin’s technical details. Rather than explaining individually, open sourcing would allow them to understand the details directly from the source code.
2. Open sourcing could dispel many people’s concerns about photo album privacy.
3. I’m not skilled at Swift development, and I believe I had already completed the part of this product that I was most interested in doing. However, I kept receiving emails from users hoping to add features like multi-select deletion, left and right swipe, Mac/NAS/Android support, etc. I wanted to leverage the power of the community, allowing capable developers to refine the product further.

Indeed, after open-sourcing, some people were inspired to create Android(#12) and Mac versions.

Open-sourcing this project also made it to GitHub Trending, and I got a free year of GitHub Copilot because of it, which made me happy.

Plagiarism and Repackaging

Before talking about plagiarism, I want to define what plagiarism is:

Plagiarism refers to the direct or indirect use of others’ works, ideas, or content without authorization, presenting it as one’s own work or idea.

Actually, plagiarism appeared even before open sourcing. Someone made an Android version and released it on Google Play, completely copying the name and product description of Queryable. I felt angry, but this actually requires mastery of both machine learning and iOS development. Before open sourcing, I only encountered one such person.

But after open-sourcing, there were many more copycats and repackagers. As the project has an MIT open-source license, even when they repackaged it with a different icon and re-launched it on the App Store, I felt a bit speechless but wouldn’t say anything. These were the most common cases I saw.

More malicious ones use the Queryable name to repackage and launch, like the one below, with a free + ad revenue model. I’m very worried that users might mistakenly think this is the Android version of Queryable and come to me with problems later.

I would feel sad at first, but when there are too many lice, you stop itching, and later I gradually became indifferent.

Becoming Paid Again

It became paid again in November 2023. Besides the increased income pressure, I found that:

Open sourcing didn’t help my product become better.

I originally hoped that by making it open source and free, professional mobile developers would contribute code to help polish Queryable/Xunyin. But the reality is that those developers who created new apps with better UIs and functionality never sent PRs to my repo. As a result, my product stagnated, and because it was free, I received even more complaints from users than before.

Whenever users emailed me with feedback/bugs, my first reaction was impatience (internally: It’s good enough that I let you use it for free, and you’re still picky). I found that this mindset led to the product falling further and further behind, until one day it would be eliminated.

But once it’s paid, I would calmly deal with users’ opinions and improve the product. My first reaction to receiving feedback is gratitude rather than annoyance. This would force me to inevitably spend effort optimizing the product, ultimately letting all users who have paid use a better-polished product, rather than neglecting maintenance and dying in a few years.

Epilogue

Apple finally announced at this year’s WWDC that iOS 18 will support semantic search for photos, although I still can’t use it in the latest iOS 18 beta, there’s reason to believe Apple will do better than Queryable.

It’s been almost two years since the idea was born and the product was launched. It has accompanied me through life’s ups and downs and witnessed the closure of several coffee shops. I’ve also accompanied it through its birth, peak, and low tide. It hasn’t made money as I initially fantasized (If 1 million people download it and each gives me $1, I would…).

Life goes on, and the story isn’t over yet: Last week, I researched the latest paper and redesigned and trained the Chinese text model. The app size was reduced from 232MB to 159MB, indexing speed doubled, accuracy improved, and the training process took 3 days and cost $70.

Moreover, the recently subscribed Claude 3.5 Sonnet’s code writing ability is incredibly powerful. The “multi-select deletion” feature that I couldn’t manage with GPT-4 before was finally developed and launched last week after a year and a half delay. I quite like this calm ending.

I can’t wait to start the next idea that will keep me up all night, the next exile into a vacuum.

每一个平替商品的搜索技巧背后，都隐藏着一种对生活的折叠。

起因是我看到了一条帖子，讲如何通过替换搜索词，实现以更低的价格购买相同功能的商品，例如：

瑜伽垫 -> 瑜伽垫男照片墙 -> 渔网野餐布 -> 防水桌布...

我试了一下，有些技巧已经失效了，有些的确便宜许多。于是我在小红书搜索「替换词」，发现许多收藏过万的帖子，它们通常是一组图片，每张图都是新/旧商品名的对照列表，如下所示：

这些对照表很好，但不好用。当我真的想搜某个商品时，我需要先对着密密麻麻的图表查找，不仅如此，当我想搜的商品不在列表里时，它就失效了。我想，能否用 AI 来实现这个任务呢？训练一个模型，用户输入想搜索的商品，模型给出便宜的平替商品名。

我很快整理了一些数据并开始训练模型，OpenAI 已经支持finetune 模型了，你只需要把数据导出并上传就可以自动开始训练，我选择的模型是gpt-3.5-1106，训练花费了大概 10 分钟。最后，我搭建了一个网页用于模型调用，得益于开发工具的完善，我只用了一下午就完成了这个 demo 产品。

消费折叠

网页开发最快乐的部分是注册域名。很快我就选好了网址: pingti.xyz，便宜、好记。并让所有朋友都试了试，很多人都觉得好玩，虽然有些结果比较离谱(牙膏->足浴店小样, iPhone->二手 iPhone, 唇膏->蜡烛)，但有些还是蛮有用的，至少模型可以记住截图里的平替词，不必一个个找了。

仍然有一些比较差的结果，我开始思考怎么优化，这使我不得不仔细分析原始训练数据，看看模式上有什么规律。总结如下：

• 性别套利: 例如瑜伽垫->瑜伽垫男，或遮阳伞->雨伞男，这背后反应的其实是男人比女人更在乎实用性和性价比，更少为了颜值、设计而买单，而设计产生了成本。

• 场景套利: 例如马甲->老头马甲，因为老年人更在乎价格, 女包->包包尾货，这个不用解释, 地毯->办公室地毯, 书桌->培训桌, 椅子->婚礼用椅，是因为在办公室、培训、举办婚礼时，通常会使用更便宜的材质吗？
• 地域套利: 例如袜子->诸暨袜子, 耳饰->义乌耳饰，因为中国的袜子主要来自诸暨，所以通过指定原产地可以获得更低的价格。
• 无法归类: 这一类最有意思，它们的特点是，两个商品几乎八竿子打不着，但是它们在「功能」上可以实现接近平替的效果，例如照片墙->渔网, 相框->营业执照框, 面膜收纳->食品保鲜盒, iPad支架->菜谱架, 美甲灯->验钞灯, 乐高防尘罩->超市陈列盒，这其中每个平替商品的搜索技巧背后，都隐藏着一种对生活的折叠，不信你可以仔细品味。

我意识到，靠简单的 finetune 模型，也许可以学会前两种套利模式，叠加规则(给商品分配相应产业的城市)，也许可以学会第三种模式，但学会最后一类平替方式几乎不可能，即使是人类也需要大量的实践积累才能摸索出其中的奥妙。

郝景芳在北京折叠中讲述了不同社会阶层在空间和时间上的折叠，我觉得「商品平替」似乎是这种折叠所露出的缝隙，我回想起自己第一次购买维生素 C 的场景：在知乎上搜汤臣倍健的维生素 C 和医院有什么差别，结论是：前者比较甜。后来每次我都买东北制药维生素 C，二者价格差了 100 倍。

经济下行，每个人都在缩减开支。想象一下：未来的某一天，我坐在婚礼专用椅上、墙上挂着一张渔网，上面是我的照片，我面前的培训桌上摆着菜谱架，iPad 正在播放视频。 这个场景还蛮好笑的，但人生还要继续，无论商品怎样平替，人生是无法平替的：重要的不是渔网，而是渔网上的照片。

Behind every search trick for a substitute product lies a form of folding in life.

It all started when I saw a post about how to use alternative search terms to buy products with the same functionality at a much lower price. For example:

Yoga Mat -> Men's Yoga MatPhoto Wall -> Fishing NetPicnic Cloth -> Waterproof Tablecloth...

I tried a few of these tricks and found that some no longer worked, while others indeed offered significant savings. Consequently, I searched for “substitution keywords” on Xiaohongshu (小红书) and discovered many posts with over 10,000 likes. Typically, these posts include a series of images, each displaying a list of product name pairs, comparing expensive products with their cheaper alternatives, as shown below:

These comparison tables are useful but not user-friendly. When I actually want to search for a product, I need to sift through these dense tables. Moreover, when the product I’m looking for isn’t listed, they become useless. I wondered if AI could enhance this experience. Imagine training a model where users input the product they’re searching for, and the model suggests alternative keywords for a cheaper option.

I quickly organized some data and began training the model. OpenAI already supports finetuning models. Simply export and upload the data, and it will start training automatically. I chose the gpt-3.5-1106 model, and the training took about 10 minutes. Finally, I built a webpage for model invocation. Thanks to the comprehensive development tools, I was able to complete this demo product in just an afternoon.

Folding Consumption

The most enjoyable part of web development is always selecting a domain name. I quickly chose the URL: pingti.xyz, which is both cheap and memorable. I encouraged all my friends to try it, and most of them found it fun, although some results were quite outrageous (Toothpaste -> Foot bath shop sample, iPhone -> Used iPhone, Lipstick -> Candle). However, many were actually quite useful. At the very least, the model remembers the substitute terms from the screenshots, eliminating the need to search one by one.

There are still some less accurate results, which made me think about how to optimize the model. This necessitated a close analysis of the original training data to identify patterns. The main patterns are:

• Gender Arbitrage: For example, Yoga Mat -> Men’s Yoga Mat, or Sun Umbrella -> Men’s Umbrella, reflecting the tendency of men to prioritize functionality and cost-effectiveness over aesthetics and design, which adds to the cost.

• Scenario Arbitrage: For instance, Vest -> Old Man’s Vest as elderly people care more about price, Women’s Bag -> End-of-Batch Bag needs no explanation, Carpet -> Office Carpet, Desk -> Training Desk, Chair -> Wedding Chair - perhaps because cheaper materials are used in offices, training, and weddings?

• Regional Arbitrage: For example, Socks -> Zhuji Socks, Earrings -> Yiwu Earrings, because a majority of China’s socks are produced in Zhuji, specifying the city can lead to lower prices.

• Unclassifiable: This is the most intriguing part. The key point is that although both products are almost entirely dissimilar and have no common ground, the cheaper one can nearly substitute the more expensive one in terms of functionality. Examples include Photo Wall -> Fishing Net, Photo Frame -> Business License Frame, Face Mask Storage -> Food Preservation Box, iPad Stand -> Recipe Holder, Nail Lamp -> Money Checker Lamp, Lego Dust Cover -> Supermarket Display Box. Each of these substitution tricks reveals a ‘folding’ of life, which might lead you to ponder.

I realized that, even though a simple fine-tuned model might learn the first two arbitrage patterns and maybe the third with additional rules (assigning cities to respective industries), it could never learn the last type of substitution. These patterns are the ones that even humans need extensive practical experience to figure out.

Hao Jingfang’s Folding Beijing tells a story of different social classes living in the same city but never intersecting in space and time. I think the concept of ‘product substitution’ offers a glimpse into this folding phenomenon. I recall my first purchase of Vitamin C: I searched on Zhihu for the differences between the expensive brand of Vitamin C and the one available in hospitals. The conclusion was that the former is sweeter. Since then, I have always purchased Vitamin C from Northeast Pharmaceutical, priced at 1/100th of the former.

In an economic downturn, everyone is cutting costs. Imagine this: One day in the future, I sit on a wedding-specific chair, with a fishing net on the wall displaying my photos, and in front of me on a training desk sits a recipe holder holding an iPad playing a video. It’s a funny scene, but life goes on. Regardless of how products are substituted, life itself is irreplaceable: it’s not about the net, but the photos on it.

TL;DR: 此文讲述了我所经历的信息获取危机，以及因此创造 Sublink 这一链接分享网站而引发的思考。

上一份工作，公司内网有一个论坛：你可以创建一个圈子并在里面发帖，感兴趣的人可以加入圈子、投稿。我创建了一个名叫「知新」的圈子，简介这么写：

这是一个旨在「共享大脑」的计划：把本周所知道的、你感兴趣的领域的新事件、新进展告诉大家，这样我们就能和其他人共享彼此的大脑了。你不需要对你分享的东西有「任何一丁点」的专业背景知识，让大家知道有这个东西，就已经就足够的意义。

每周，我会在圈子里分享这周我看到的新事物，我分享了 DALLE-2，Azuki，疫苗有效率数据等。当我投稿 2 期后，我创建的圈子热度已经进入了论坛的前三名，有很多人订阅和点赞。