trentmkelly/LessWrong-43k huggingface dataset:
Two interviews with the founder of DeepSeek
by Cosmia_Nebula — 2024-11-29T03:18:47.246Z
Link: https://rentry.co/DeepSeek-interview/

# The Madness of High-Flyer: The Approach to LLM by an AI Giant that Few See
暗涌Waves (2023-05-23 22:50)

Written by 于丽丽
Edited by 刘旌
Translated by Cosmia Nebula

High-Flyer is probably the most exotic among the swarming multitude of competitors in the battle of large models.

This is a game destined for the few, and while many startups are adjusting their direction or even retreating after the big players enter the game, this quantitative fund is alone in its march.

In May, High-Flyer named its new independent organization for making large models DeepSeek (深度求索) and emphasized that it would focus on making real human-level artificial intelligence. Their goal is not only to replicate ChatGPT, but also to research and discover mysteries of artificial general intelligence (AGI).

Not only that, in a track that is considered to be extraordinarily dependent on scarce talent, High-Flyer is also assembling a group of obsessive people and offer up what they consider to be their greatest weapon: a group of people's curiosity.

Among quants, High-Flyer is a “top fund” that has reached over 100 billion yuan in size, but its focus on this new wave of AI is actually quite dramatic and unexpected.

When the shortage of high-performance GPU chips from domestic cloud vendors became the most direct factor limiting generative AI in China, according to Caijing Eleven (财经十一人), there are no more than five companies with more than 10,000 GPUs in China. Other than a few headline big corps [like Baidu], it also included a quantitative fund called High-Flyer. (It is commonly believed that 10,000 NVIDIA A100 chips are the bare-minimum for training large models from stratch.)

In fact, this company has long been a hidden AI giant: Founded in 2015, High-Flyer founded an AI company in 2019. It built out the Supercomputer "Firefly I" (萤火一号) with a total investment of ~200 million yuan, carrying 1,100 GPUs. 2 years later, “Firefly II” (萤火二号) supercomputer investment increased to 1 billion yuan, equipped with about 10,000 NVIDIA A100 graphics cards.

This means that from the point of view of computing power alone, High-Flyer had a ticket to self-train ChatGPT earlier than those big corps.

Large models strongly depend on compute, algorithms and data, so it takes 50 million yuan to start, and ~10 million yuan to train once, and it's hard for small (< 1 billion yuan market-cap) companies to keep up. Depsite al...
Gratitudes: Rational Thanks Giving
by Seth Herd — 2024-11-29T03:09:47.410Z

Epistemic status: I wish I'd thought of writing this before the day rolled around. Brief and unpolished, although this is something I've thought about a lot on both personal and computational neuroscience levels. There are no strong conclusions, just some thoughts on gratitude you may find interesting or even useful.

Hopefully you've had a fun Thanksgiving celebration, including feasting and appreciating family and friends. You may have even spent a little time thinking of the things you're grateful for. This seems like a pretty useful thing to do.

Gratitude is one of the best ways we know of so far to increase happiness, particularly in time/effect tradeoff. Keeping a journal of things you're grateful for, and writing in it daily is the most common intervention; it seems to work pretty well for some people, but I've never tried it.

Not only does gratitude seem to work empirically to increase happiness, it should work, for pretty deep theoretical reasons.

Minds like ours, a gamush of neural networks using predictive and reinforcement learning, need to keep track of what's good and bad. They need to learn to do things that make their outcomes better. That requires judging not only what's good, but what's better-than-expected. That in turn almost inevitably creates a comparative basis for happiness/joy/satisfaction. Evolution does not want us sitting around being happy with outcomes just as good as last time; it wants us to strive for better!

But evolution is not entirely the boss of us.

The world is too complex for our monkey brains to make a fully accurate objective judgment about what's a better-than-expected situation. Attention is going to play a huge role, by selecting some part of the complex world on which to make that judgement. And in our hyper-complex social world, we've got a lot of leeway in using attention strategically to get the results we want.

Danger! you say. This sounds like distorting our epistemics!

Well, sort of, but just in a specific way that seems like it should be thoroughly harmless if done correctly.

When I think "wow I'm grateful for my comfortable warm clothing", the fact that I have comfortable warm clothing is an accurate fact about the world. Whether or not I'm lucky to have them can be an epistemic question, but only with further framing: lucky compared to whom or what hypothetical? It's that framing that makes a fact about reality in...
Depression and Creativity 
by Bill Benzon — 2024-11-29T00:27:41.072Z

This is cross-posted from New Savanna.

NOTE: I’ve posted this interaction with Claude, not so much to present the ideas Claude offered about possible relationships between depression and creativity, but as an example of the kind of conversational interaction one can have with it. I was particularly impressed with the way it advanced the conversation by asking questions.
______________________________________________________________________

I’m working on an article about depression and decided to ask about the connection between depression and creativity. Claude makes a few general remarks and asks for my thoughts. So I mention some remarks that my teacher, David Hays, had made years ago, mentioning that he was a computational linguist. After some general remarks, Claude asks:

... did Hays develop this idea through his work in computational linguistics? It seems like there might be interesting parallels between how computer systems need to reorganize data structures and how human minds might need to reorganize mental frameworks.

Interesting, most interesting. Our ensuing interaction felt a bit like a real conversation. I was surprised and pleased. This felt a bit like, you know, real intelligence.

I’ve appended its remarks below. I’m using my standard convention, setting my words in bold while Claude’s is plain.

Version: Claude 3.5 Sonnet

* * * * *

Is there evidence for a relationship between depression and creativity?

The relationship between depression and creativity is complex and has been studied extensively. The research shows several interesting patterns:

Historical observations have noted many renowned artists, writers, and musicians who experienced depression, like Virginia Woolf, Vincent van Gogh, and Sylvia Plath. However, we should be careful not to romanticize mental illness or assume correlation equals causation.

Scientific studies have found some evidence for links between depression and certain aspects of creativity:

Some research suggests that people with mood disorders may experience enhanced divergent thinking during certain phases, allowing them to generate more unique associations and ideas. The changes in emotional processing and perspective that can come with depression might contribute to seeing things in new ways or having deeper emotional insights.

However, severe depression typically impairs cognitive function and motivation, which can significan...
Estimates of GPU or equivalent resources of large AI players for 2024/5
by CharlesD — 2024-11-28T23:01:58.522Z

AI infrastructure numbers are hard to find with any precision. There are many reported numbers of “[company] spending Xbn on infrastructure this quarter” and “[company] has bought 100k H100s or “has a cluster of 100k H100s” but when I went looking for an estimate of how much compute a given company had access to, I could not find consistent numbers available. Here I’ve tried to pull together information from a variety of sources to get ballpark estimates of (i) as of EOY 2024, who do we expect to have how much compute? and (ii) how do we expect that to change in 2025? I then spend a little time talking about what that might mean for training compute availability at the main frontier labs. Before going into this, I want to lay out a few caveats:

These numbers are all estimates I’ve made from publicly available data, in limited time, and are likely to contain errors and miss some important information somewhere.
There are very likely much better estimates available from paywalled vendors, who can spend more time going into detail of how many fabs there are, what each fab is likely producing, where the data centers are and how many chips are in each one, and other detailed minutiae and come to much more accurate numbers. This is not meant to be a good substitute for that, and if you need very accurate estimates I suggest you go pay one of several vendors for that data.

With that said, let’s get started.

Nvidia chip production

The first place to start is by looking at the producers of the most important data center GPUs, Nvidia. As of November 21st, after Nvidia reported 2025 Q3 earnings[1] calendar year Data Center revenues for Nvidia look to be around $110bn. This is up from $42bn in 2023, and is projected to be $173bn in 2025 (based on this estimate of $177bn for fiscal 2026).[2]

Data Center revenues are overwhelmingly based on chip sales. 2025 chip sales are estimated to be 6.5-7m GPUs, which will almost entirely be Hopper and Blackwell models. I have estimated 2m Hopper models and 5m Blackwell models based on the proportion of each expected from the CoWoS-S and CoWoS-L manufacturing processes and the expected pace of Blackwell ramp up.

2024 production

Sources for 2024 production numbers were thin and often conflicting, but estimates of 1.5m Hopper GPUs for Q4 2024 (though this will include some H20 chips, a significantly inferior chip,...
Automated monitoring systems
by hiki_t — 2024-11-28T18:54:29.886Z

Initial draft on 28th Nov, 2024

After participating in the last (18th Nov, 2024) and this week (25th Nov, 2024)'s group discussion about topics of the “𝘀𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗼𝘃𝗲𝗿𝘀𝗶𝗴𝗵𝘁” and AI model's “𝗿𝗼𝗯𝘂𝘀𝘁𝗻𝗲𝘀𝘀” at the

AI Safety Fundamentals

program, I have been stuck with thinking what are effective use cases of those techniques, such as the "𝗗𝗲𝗯𝗮𝘁𝗲" technique, to expand AI capabilities beyond human intelligence/abilities, while maintaining AI systems safe to humans.

After an emergence of the state of the art (SOTA) Large Language Models (LLMs) like GPT3 (2020~), acting almost human-like ability to converse and perform well on a variety of standardized tests (e.g. SAT/GRE), many of us might have raised substantial attentions to two future directions,

(1) how to train AI beyond human intelligence? (Scalable oversight) and

(2) how to prevent SOTA AI from performing unintended behaviors (e.g. deceptive behaviors), which will soon or later become out of control by humans? (AI’s robustness)

These two motives have boomed to invent a variety of new techniques, such as RLHF(human feedback-guided training) on LLMs, CAI(constitutional rule-guided training), Debate(letting AI speak out loud their internal logic), Task-decomposition(decomposing complex task into simpler sub-tasks), Scope(limiting AI abilities on undesired tasks), Adversarial training(training to become insensitive to adversarial attacks).

However, many of these techniques still heavily rely on human-based values and logic, capping AI within human capabilities. My understanding of human-beyond AI is not restricted to human perspectives, but it can autonomously grow on its own by experiencing the external world by itself and experimenting to figure out sensible rules of the world (as well as inspecting positive and negative consequences of operating own behaviors) like humans or other biological agents do on the Earth.

In order to do the self-experiment in a case of the debating technique, I feel more and more of requirement of implementing automated monitoring systems (e.g. monitoring models) observing what is currently happening in the belonging world, which are independent from action systems (e.g. debating models). Then, I think of a specific use case, such that if debating models can improve Cybersecurity by gaining rewards through protecting a deployed webpage ...
AI #92: Behind the Curve
by Zvi — 2024-11-28T14:40:05.448Z

People don’t give thanks enough, and it’s actual Thanksgiving, so here goes.

Thank you for continuing to take this journey with me every week.

It’s a lot of words. Even if you pick and choose, and you probably should, it’s a lot of words. You don’t have many slots to spend on things like this. I appreciate it.

Thanks in particular for those who are actually thinking about all this, and taking it seriously, and forming their own opinions. It is the only way. To everyone who is standing up, peacefully and honestly, for whatever they truly think will make the world better, even if I disagree with you.

Thanks to all those working to ensure we all don’t die, and also those working to make the world a little richer, a little more full of joy and fun and health and wonder, in the meantime. Thanks for all the super cool toys, for they truly are super cool.

And thanks to all the parts of reality that work to so often keep it light and interesting along the way, and for not losing touch with the rest of the world. This is heavy stuff. You cannot let it fully take over your head. One must imagine Buffy at the prom.

Thanks of course to my health, my kids, all my family and friends, and all the friends I have that I don’t even know about yet. The world is really cool like that.

Special thanks to those who help make my writing possible and sustainable.

From this past week, I’ll also give thanks for those who organized The Curve, a conference I was able to attend last weekend, and those who help run Lighthaven, and all the really cool people I met there.

Thanks to the universe, for allowing us to live in interesting times, and plausibly giving us paths to victory. What a time to be alive, huh?

Oh, and I am so thankful I managed to actually stay out of the damn election, and that we are finally past it, that we’re mostly on a break from legislative sessions where I need to keep reading bills, and for the new College Football Playoff.

Life can be good, ya know?

Table of Contents

Language Models Offer Mundane Utility. Finding a better way to code.

It’s a Poet Whether or Not You Know It. No one said it was a good one.

Huh, Upgrades. The small new LLM features are coming fast and furious.

Thanks for the Memories. Memory is a great feature, if you can pull it off.

Curve Ball. Reports from a conference.

ASI: A Scenario. Playing Daniel Kokotajlo’s tabletop wargame exercise.

Deep

...
Is the mind a program?
by EuanMcLean — 2024-11-28T09:42:02.892Z

This is the second in a sequence of posts scrutinizing computational functionalism (CF). In my last post, I defined a concrete claim that computational functionalists tend to make:

Practical CF: A simulation of a human brain on a classical computer, capturing the dynamics of the brain on some coarse-grained level of abstraction, that can run on a computer small and light enough to fit on the surface of Earth, with the simulation running at the same speed as base reality[1], would cause the same conscious experience as that brain.

I contrasted this with “theoretical CF”, the claim that an arbitrarily high-fidelity simulation of a brain would be conscious. In this post, I’ll scrutinize the practical CF claim.

My assumptions

I assume realism about phenomenal consciousness: Given some physical process, there is an objective fact of the matter whether or not that process is having a phenomenal experience, and what that phenomenal experience is. I am in camp #2 of Rafael’s two camps.
I assume a materialist position: that there exists a correct theory of phenomenal consciousness that specifies a map between the third-person properties of a physical system and whether or not it has phenomenal consciousness (and if so, the nature of that phenomenal experience).
I assume that phenomenal consciousness is a sub-component of the mind.

The search for a software/hardware separation in the brain

Practical CF requires that there exists some program that can be run on a digital computer that brings about a conscious human experience. This program is the “software of the brain”. The program must simulate some abstraction of the human brain that:

Is simple enough that it can be simulated on a classical computer on the surface of the Earth,
Is causally closed from lower-level details of the brain, and
Encodes all details of the brain’s conscious experience.

Condition (a), simplicity, is required to satisfy the definition of practical CF. This requires an abstraction that excludes the vast majority of the physical degrees of freedom of the brain. Running the numbers[2], an Earth-bound classical computer can only run simulations at a level of abstraction well above atoms, molecules, and biophysics.

Condition (b), causal closure, is required to satisfy the definition of a simulation (given in the appendix). We require the simulation to correctly predict some future (abstract...
LLMs Do Not Think Step-by-step In Implicit Reasoning
by Bogdan Ionut Cirstea — 2024-11-28T09:16:57.463Z
Link: https://arxiv.org/abs/2411.15862

Author: Yijiong Yu.

Abstract:

It has been well-known that Chain-of-Thought can remarkably enhance LLMs’ performance on complex tasks. However, because it also introduces slower inference speeds and higher computational costs, many researches have attempted to use implicit CoT, which does not need LLMs to explicitly generate the intermediate steps. But there is still gap between their efficacy and typical explicit CoT methods. This leaves us a doubt that, does implicit CoT really equal to explicit CoT? Therefore, in this study, we address this question through experiments. We probe the information of intermediate steps from the model’s hidden states when it is performing implicit CoT. The results surprisingly indicate that LLMs hardly think about intermediate steps, suggesting they may just rely on experience rather than strict step-by-step reasoning. Moreover, we find LLMs’ implicit reasoning capabilities are susceptible and unstable, reaffirming the necessity of explicit CoT to effectively support complex tasks.

They probe for representations of intermediate steps in simple multi-step arithmetic problems, and aren't able to recover such information robustly for e.g. the 3rd step in 5-step problems. They also show that using CoT is much more robust to prompt variations.

Relevant with respect to opaque reasoning and out of context reasoning (OOCR) in Transformer architectures.
A Meritocracy of Taste
by Daniele De Nuntiis — 2024-11-28T09:10:10.598Z

Epistemic status: the idea it’s not fully fleshed out (there are a bunch of problems that I’m skipping over, the post is meant more as: “has anyone been seriously thinking about this?” and throwing out a starting point, than: “here is a proposal that would actually work”) and I wouldn’t be surprised if either it’s unfeasible or it has already been implemented.

I was reading Neutrality and the part about social media struck me: “Or, they use ‘impartializing tactics’ chosen in the early-Web-2.0 days when people were more naive and utopian, like ‘allow everyone to trivially make a user account, give all user accounts initially identical affordances, prioritize user-upvoted content.’ […] — with LLMs, zero-knowledge proofs, formal verification, prediction markets, and the like — can make a better stab at these supposedly-subjective virtues than ‘one guy’s opinion’ or ‘a committee’s report’ or ‘upvotes’?”. I thought “Seriously, why aren’t social media companies doing this already?” (and they might be). The thing that stood out to me was that we could try to use prediction markets, or just markets, to improve the “ALGORITHM”.

The easiest possible way to do it (and that almost definitely wouldn’t work) is to use likes (or upvotes, etc.) as currency, we could give people a certain amount of likes and treat them as “bids” one could place on a given post, if the post does well, you earn more likes. Another approach, that solves the obvious problem of “But what if I run out of likes?”, is to use a reputational system: likes are weighted by the reputation you have, infinite likes finite reputation. There are a couple other problems that need to be solved. First, how do we measure a successful prediction? Second, how can we avoid Goodhart without overcomplicating the system? (Another issue is setting up the right incentives, a paid model probably works better than one based on ads but I’m not sure what would be best here).

Let’s start from Goodhart: the solution is just don’t tell the users what you’re doing. Don’t put the reputation anywhere on the site and you should mostly be fine.

Now, how do you measure a good prediction? First we need to figure out what good is. Personally, I think avoiding to promote all the memetic slop, “like to see the animation” or the random celebrities' news or “funny” videos, would be a decent goal. We want to promote good posts for the ri...
Importing Bluesky Comments
by jefftk — 2024-11-28T03:50:06.635Z

I decided years ago that instead of hosting a comment section I'd pull
in comments from elsewhere: first

Facebook
(no longer working because of anti-scraping), then

Google
Plus (which means I didn't lose my g+ discussions when they turned
it down), then LessWrong, Reddit, HN, the EA Forum, and

Mastodon.
Prompted by

Daniel, I've now added
Bluesky support as well.

It went pretty quickly: there's a public URL you can hit to get the
comments on a post as JSON, and fitting that into my existing comment
setup was not bad (commit).
New o1-like model (QwQ) beats Claude 3.5 Sonnet with only 32B parameters
by Jesse Hoogland — 2024-11-27T22:06:12.914Z
Link: https://qwenlm.github.io/blog/qwq-32b-preview/

A new o1-like model based on Qwen-2.5-32B reportedly beats Claude 3.5 Sonnet[1] on a bunch of difficult reasoning benchmarks. A new regime dawns.

The blog post reveals nothing but the most inane slop ever sampled:

What does it mean to think, to question, to understand? These are the deep waters that QwQ (Qwen with Questions) wades into. Like an eternal student of wisdom, it approaches every problem - be it mathematics, code, or knowledge of our world - with genuine wonder and doubt. QwQ embodies that ancient philosophical spirit: it knows that it knows nothing, and that’s precisely what drives its curiosity. Before settling on any answer, it turns inward, questioning its own assumptions, exploring different paths of thought, always seeking deeper truth. Yet, like all seekers of wisdom, QwQ has its limitations. This version is but an early step on a longer journey - a student still learning to walk the path of reasoning. Its thoughts sometimes wander, its answers aren’t always complete, and its wisdom is still growing. But isn’t that the beauty of true learning? To be both capable and humble, knowledgeable yet always questioning? We invite you to explore alongside QwQ, embracing both its insights and its imperfections as part of the endless quest for understanding.

The model is available on HuggingFace. It's not yet clear when we'll hear more about the training details. EDIT: We can expect an official announcement tomorrow.

^

Not clear if this is Sonnet 3.5 new or old. For this, I blame Anthropic.