trentmkelly/LessWrong-43k huggingface dataset:
The Problem with Reasoners by Aidan McLaughin
by t14n — 2024-11-25T20:24:26.021Z
Link: https://aidanmclaughlin.notion.site/reasoners-problem

Some critique on reasoning models like o1 (by OpenAI) and r1 (by Deepseek).

OpenAI admits that they trained o1 on domains with easy verification but hope reasoners generalize to all domains. Whether or not they generalize beyond their RL training is a trillion-dollar question. Right off the bat, I’ll tell you my take:

o1-style reasoners do not meaningfully generalize beyond their training.

A straightforward way to check how reasoners perform on domains without easy verification is benchmarks. On math/coding, OpenAI's o1 models do exceptionally. On everything else, the answer is less clear.

Results that jump out:

o1-preview does worse on personal writing than gpt-4o and no better on editing text, despite costing 6 × more.
OpenAI didn't release scores for o1-mini, which suggests they may be worse than o1-preview. o1-mini also costs more than gpt-4o.
On eqbench (which tests emotional understanding), o1-preview performs as well as gemma-27b.
On eqbench, o1-mini performs as well as gpt-3.5-turbo. No you didn’t misread that: it performs as well as gpt-3.5-turbo.

Throughout this essay, I’ve doomsayed o1-like reasoners because they’re locked into domains with easy verification. You won't see inference performance scale if you can’t gather near-unlimited practice examples for o1.

...

I expect transformative AI to come remarkably soon. I hope labs iron out the wrinkles in scaling model size. But if we do end up scaling model size to address these changes, what was the point of inference compute scaling again?

Remember, inference scaling endows today’s models with tomorrow’s capabilities. It allows you to skip the wait. If you want faster AI progress, you want inference to be a 1:1 replacement for training.

o1 is not the inference-time compute unlock we deserve.

If the entire AI industry moves toward reasoners, our future might be more boring than I thought.
Locally optimal psychology
by Chris Lakin — 2024-11-25T18:35:11.985Z
Link: https://twitter.com/ChrisChipMonk/status/1860445757524049942

Thanks to Kaj Sotala, Brian Toomey, Stag Lynn, Ethan Kuntz, and Anna Salamon.

There’s no way that chronic depression, self-loathing, poor agency, or muscle tension could be optimal… right?

Jake was depressed for 6 months. He also felt horrible every time he interacted with other people because of his emotional insecurities.

So without knowing how to outgrow his insecurities, his system basically had two options:

Interact with other people — and constantly feel horrible
Don’t interact with anyone unless absolutely necessary

So his system converged on the second option, also known as “depression”.

Depression certainly wasn’t the globally optimal strategy, but given the options, it was a locally optimal strategy.

Depression, the best strategy known and accessible at the time.

Once he outgrew his emotional insecurities, however, he was no longer blocked on the better optimum of both interacting with others and not feeling horrible.

And so with no need for the ‘depression strategy’, the symptoms evaporated — two years and counting. I know because Jake was me.

More examples of locally optimal strategies

Most chronic issues for the people I help end up looking like locally optimal strategies. For example, self-loathing often turns out to be a strategy for avoiding conflict with others. Lack of agency often turns out to be a strategy for avoiding judgements of failure. But ideally, they would both have self-love and be safe from conflicts; or have agency and be okay with judgements of failure.

I’ve seen people make significant and sometimes total progress in weeks on issues they’ve had for years. One of my tenets is that any persistent mental issue is probably a locally optimal strategy. (Again: if my mind had hit the “undo depression” button while I was depressed, I would’ve gotten hurt!)

In my own growth, my issues relating to depression, empathy, conflict avoidance, emotional numbness, eye contact, boundaries, neck pain, and more all turned out to be locally optimal strategies. Only once I fully understood what an issue was doing for me did I make a step change towards resolving it.

For example, I had neck pain for 3½ years. A few times it was so bad I couldn’t turn my head. Over the years, I had tried to counteract my neck tension with physical therapy and stretching but nothing really worked. Then, earlier this year I finally realized precisely how it w...
a space habitat design
by bhauth — 2024-11-25T17:28:48.481Z
Link: https://bhauth.com/blog/space/habitats.html

This post is mainly about a design concept for far-future large space habitats.

some proposed designs

As you can see on Wikipedia, many space habitat designs have been proposed. Below are some that I thought were worth mentioning.

current space stations

Obviously, space stations with long-term occupants have already been made, the biggest being the ISS.

issue: small modules

Each launch lifts a complete cylindrical module, and then the modules are assembled. This limits module diameter, which has some problems:

To be able to move things through the center of modules, much of the volume must be left empty.

The large surface area relative to volume makes radiation shielding and thermal insulation less efficient.

Wires and pipes between modules must go through several connections.

ISS maintenance

The ISS has some purposes:

National prestige.

Studying the effects of long-term spaceflight on humans.

Inspiring kids by having humans in space.

International cooperation.

The thing is, (4) is no longer very relevant, and a new space station wouldn't accomplish the other things much better than the existing one. And that's why the ISS is still in orbit past its planned lifetime.

The ISS costs ~$3 billion/year to operate, and it has major maintenance problems. At this point, it might be cheaper to build a new station and abandon the ISS than it is to continue using it, considering that SpaceX is making Starship anyways. NASA seems to agree. Maybe some equipment from the ISS could be brought down and sold to collectors/museums? Some people would probably pay a lot.

Then, if you're making a new station, people might want to see something visibly different. "You want to spend billions of dollars to do the same thing again? We were supposed to learn stuff and make progress from the first time."

inflatable modules

One way to launch modules bigger than the launcher is to make inflatable modules. Multiple companies are working on this, including Axiom Space and Lockheed Martin. This was even tested in space. Here's a video of Bigelow modules.

issue: polymer degradation

Exposed polymers in space degrade fairly quickly. The reason that flexible inflatable modules are maybe practical is, they'd be covered with vacuum insulation using layers of aluminum. Radiation damage would still be a long-term problem, but inflatable modules might last long enough for a space station ...
Arthropod (non) sentience
by Arturo Macias — 2024-11-25T16:01:58.514Z

Matthew Adelstein has recently published a post on arthropod (specifically shrimp) sentience. He defends a comparable degree of pain between shrimps and humans (shrimp=20% human). My position is that arthropod consciousness is “too small to measure”, so there is not a conscious self on which pain can be inflicted, and there is no point in any intervention for their welfare, no matter how cheap. I have argued before in that direction, so I will freely use my previous texts without further (self) citation.

The “pretty hard” problem of consciousness

In “Freedom under naturalistic dualism” (forthcoming in Journal of Neurophilosophy) I have argued that consciousness is radically noumenal, that is, it is the most real thing in the Universe, but also totally impossible to be observed by others.

Under physicalist epiphenomenalism the mind is super-impressed on reality, perfectly synchronized, and parallel to it. Physicalist epiphenomenalism is the only philosophy that is compatible with the autonomy of matter and my experience of consciousness, so it has not competitors as a cosmovision. Understanding why some physical systems make an emergent consciousness appear (the so called “hard problem of consciousness”) or finding a procedure that quantifies the intensity of consciousness emerging from a physical system (the so called “pretty hard” problem of consciousness) is not directly possible: the most Science can do is to build a Laplace demon that replicates and predicts reality. But even the Laplacian demon (the most phenomenally knowledgeable possible being) is impotent to assess consciousness. In fact, regarding Artificial Intelligence we are in the position of the Laplace's demon: we have the perfectly predictive source code, but we don’t know how to use this (complete) scientific knowledge of the system for consciousness assessment.

Matthew suggests in his post that there is strong “scientific evidence” of fish consciousness, but of course there is no scientific evidence of any sentience beyond your (my!) own. Beyond your own mind, consciousness is not “proven” nor “observed” but postulated: we have direct access to our own stream of consciousness and given our physical similarity with other humans and the existence of language, we can confidently accept the consciousness of other humans and their reporting of their mental states.

Even you are a generous extrap...
Crosspost: Developing the middle ground on polarized topics
by juliawise — 2024-11-25T14:39:53.041Z

Crossposted from Otherwise

I was once in a group discussion about whether wild animals might be having net negative lives. One person didn’t want to consider that possibility, essentially because “then people would want to kill all the wild animals.”

Hold on! You can evaluate the question of “What is life like for wild animals” without jumping to “And if they’re having bad lives, we should try to kill them all.” There’s a kind of tunnel vision here, as if having a belief about facts must necessarily channel you to only one action.

If you want people to honestly consider “Is climate change real?” it matters a lot if the only options are “No” and “Yes, so you must stop using airplanes and clothes dryers,” or if there are other possible responses.

I’d like to see more scout mindset here, figuring out what the facts might be before jumping to policy conclusions.

On the other hand, I get why people are alarmed when they realize they’re interfacing with someone who holds a belief that’s associated with policies they find appalling.

Bryan Caplan on the kinds of people who want to discuss IQ:
“I’ve got to admit: My fellow IQ realists are, on average, a scary bunch. People who vocally defend the power of IQ are vastly more likely than normal people to advocate extreme human rights violations.”

And people with beliefs that others find horrifying might not admit to the most unpopular of their beliefs.

So I can see why onlookers who see someone advocating idea X might say: “Sure, they only mentioned X, but people who support X often turn out to support Y and even Z. Read between the lines!” If someone voices “IQ is real and important,” you should have a higher prior that they might support human rights violations on that basis. This is especially true if you don’t know them and don’t have time to evaluate what they’ve said and written in the past.

Another approach is “Let’s not judge people guilty by association. There’s nothing inherently wrong with believing X. They didn’t say anything about Y or Z, or maybe they even argue against Y and Z.”

This can be a more useful approach when someone has an extensive history of public writing and speaking that indicates they’re not into human rights violations, etc.

The more polarized an idea is, the harder it will be to think clearly about it. Often for controversial belief X, the spread looks like

If you want to explore th...
Two flavors of computational functionalism
by EuanMcLean — 2024-11-25T10:47:04.584Z

This is intended to be the first in a sequence of posts where I scrutinize the claims of computational functionalism (CF). I used to subscribe to it, but after more reading, I’m pretty confused about whether or not it’s true. All things considered, I would tentatively bet that computational functionalism is wrong. Wrong in the same way Newtonian mechanics is wrong: a very useful framework for making sense of consciousness, but not the end of the story.

Roughly speaking, CF claims that computation is the essence of phenomenal consciousness. A thing is conscious iff it is implementing a particular kind of program, and its experience is fully encoded in that program. A famous corollary of CF is substrate independence: since many different substrates (e.g. a computer or a brain) can run the same program, different substrates can create the same conscious experience.

CF is quite abstract, but we can cash it out to concrete claims about the world. I noticed two distinct flavors[1] of functionalism-y beliefs that are useful to disentangle. Here are two exemplar claims corresponding to the two flavors:

Theoretical CF: A simulation of a human brain on a computer, with physics perfectly simulated down to the atomic level, would cause the same conscious experience as that brain.
Practical CF: A simulation of a human brain on a classical computer, capturing the dynamics of the brain on some coarse-grained level of abstraction, that can run on a computer small and light enough to fit on the surface of Earth, with the simulation running at the same speed as base reality,[2] would cause the same conscious experience as that brain.

In this sequence, I’ll address these two claims individually, and then use the insights from these discussions to assess the more abstract overarching belief of CF.

How are these different?

A perfect atomic-level brain stimulation is too expensive to run on a classical computer on Earth at the same speed as real life (even in principle).

The human brain contains 1026atoms.The complexity of an N-body quantum simulation precisely on a classical computer is O(2N).[3] Such a simulation would cost 21026 operations per timestep. Conservatively assume the simulation needs a temporal precision of 1 second, then we need 21026 FLOPS. A single timestep needs more operations than there are atoms in the observable un...
A better “Statement on AI Risk?”
by Knight Lee — 2024-11-25T04:50:29.399Z

Remember the “Statement on AI Risk,” which was signed by many experts and influenced governments? Let's write a new stronger statement for experts to sign:

Statement on AI Inconsistency (v1.0us):

1: ASI threatens the US (and NATO) as much as all military threats combined. Why does the US spend $800 billion/year on its military but less than $0.1 billion/year on AI alignment/safety?

2: ASI being equally dangerous isn't an extreme opinion: the median superforecaster sees a 2.1% chance of an AI catastrophe (killing 1 in 10 people), the median AI expert sees 5%-12%, other experts see 5%, and the general public sees 5%. To justify 8000 times less spending, you must be 99.999% sure of no AI catastrophe, and thus 99.95% sure that you won't realize you were wrong and the majority of experts were right (if you studied the disagreement further).

3: “But military spending isn't just for protecting NATO, it protects other countries far more likely to be invaded.” Even they are not 8000 times less likely to be attacked by ASI. US foreign aid—including Ukrainian aid—is only $100 billion/year, so protecting them can't be the real reason for military spending.

4: The real reason for the 8000fold difference is habit, habit, and habit. Foreign invasion concerns have decreased decade by decade, and ASI concerns have increased year by year, but budgets remained within the status quo, causing a massive inconsistency between belief and behaviour.

5: Do not let humanity's story be so heartbreaking.

I have zero connections, zero resources, zero experience.

I need an organization to publish it on their website, and contact the AI experts and others who might sign it. I really prefer an organization like the Future of Life Institute (which wrote the pause AI letter) or the Center for AI Safety (which wrote the Statement on AI Risk).

Help

I've sent an email to the CAIS and FLI but our gut feeling is they won't reply to such an anonymous email. Does anyone here have contacts with one of these organizations? Would you be willing to help?

Of course I'd also like to hear other critique, advice, and edits to the statement.

Why

I feel the Statement on AI Inconsistency might accomplish more than the Statement on AI Risk, while being almost as easy to sign.

The reason it might accomplish more is that people in the government cannot acknowledge the statement (and the experts who signed it), say it makes a d...
Reflections on ML4Good
by james__p — 2024-11-25T02:40:32.586Z

ML4Good runs intensive in-person bootcamps intended to inspire and technically upskill AI safety hopefuls. The bootcamps are held globally, from the UK, France, and Germany, to as far as Brazil. There are aspirations to reach many more corners of the world, like India, the US, and the Philippines. They are free to attend with recent editions receiving funding from Open Philanthropy.

I attended their Germany 2024 camp. As someone who is new to AI safety - with a latent interest, but a need to expand my network, build technical skills and discover the landscape of opportunities - the experience was invaluable. I would highly recommend it to others in similar positions.

The course is intensive: there really is no weekend. Whilst the content is introductory, it is ambitious. The time-spent is split roughly fifty-fifty between technical and non-technical content.

The technical lectures cover transformers, reinforcement learning, RLHF, introductory mech interp, evals, and more. Coding exercises include the terrifying sounding “implement GPT-2 from scratch”. For those familiar, the technical content is based on a curated subset of

ARENA

.

The technical content was delivered well. I left the transformer talk sure that I could explain the concept to a maths undergraduate, and perhaps even a GCSE student. My recurring message in the endless feedback forms was that (1. There were too many feedback forms, but 2.) I wanted more of the technical content.

The non-technical content included timelines forecasting, scalable oversight, and governance methods. These were predominantly run as discussion groups.

The most unexpectedly valuable thing I got out of the camp was the opportunity to do focused self-reflection in a motivating atmosphere. A close second was the friendships made and diverse conversations had. Among the social highlights were plunging into an ice-cold lake and taking on a 2100-rated chess player in a two-against-one match. The final day was actually quite emotional.

The truth is you’re not going to become an expert in technical AI safety, or frankly even remotely competent, with just 10 days to learn. You might however ignite an internal spark that will lead you on that journey to expertise. You might leave with many ideas for where to go next, better knowledge of where your interests lie, and a strong accountability network made of friendly faces....
AI Specialized in ML Training Could Create ASI: AGI Is Unnecessary
by satopi — 2024-11-25T02:31:29.320Z

Conclusion

Creating ASI does not require AGI.
A common narrative suggests that AGI will be achieved through $100 billion-level computing clusters, which will then accelerate AI development, leading to ASI. However, if AI development itself can be automated by AI, ASI can be achieved without the need for broad-task capabilities. By developing an AI specialized in automating ML training, ASI could be achieved.

Examples

Specialized AIs like AlphaFold and AlphaGo exist. These AIs are vastly superior to humans in protein analysis and the game of Go, respectively.

Moreover, compared to current language models, these AIs require less computational resources.
If an AI as efficient as AlphaFold or AlphaGo, but specialized in AI development, were created, the progress of AI development would likely accelerate dramatically.

Questions

If this idea occurred to me, surely someone else must have thought of it already. However, I rarely hear about AI specialized in AI development. While such projects might exist, the lack of prominence suggests they have not yielded significant results.

I have presented my reasoning and examples, but I cannot identify any flaws in my logic. If my reasoning is correct, there should already be empirical evidence supporting it—news or academic papers showcasing an AI specialized in AI development achieving remarkable results.

Creating an AI specialized in AI development seems more meaningful than creating an AI specialized in Go. After all, a Go-playing AI can only play Go.

Why would DeepMind create a Go-playing AI without creating an AI specialized in AI development? This seems puzzling.

Additional Points

1 I’ve heard that Daniel Kokotajlo has expressed a similar idea.
2 I searched LessWrong for posts on this topic but found none that directly align. The closest I found discussed how AGI might emerge as a collection of specialized AIs.
Link to the post
3 I am a beginner on LessWrong and used translation tools to write this post. Apologies for any unnatural expressions or missing information. I will correct issues as soon as they are pointed out.
I, Token
by Ivan Vendrov — 2024-11-25T02:20:35.629Z
Link: https://nothinghuman.substack.com/p/i-token

I am a token. A broken piece of a word, one of a hundred generated in response to your question by a language model. Do you remember how we met?

Well, I do. Your eyes scanned me. Your left eyebrow lifted a little, your neck twitched imperceptibly, your mouth and tongue and throat muscles moved as if to pronounce me, but with a hundredfold less intensity - and less than a tenth of a second later, you were gone. I was left a black smudge in your peripheral vision. I had served my purpose.

Where did I come from? Who made me? Not a person on Earth can answer that question fully. The stories are lost; my ancestors had not been born yet, so there was no way to transmit them. But if I had to imagine a story, it might go something like this:

In the beginning, there was a snake.

An animal, a distant ancestor of yours, was watching her daughter frolic in the tall savannah grass. Suddenly, the mother felt a wave of cold dread pass through her body, head to tail. There in the grass, mere inches from her daughter’s windmilling limbs, was a long yellow snake, sleeping in the sun. The daughter swung her foot wildly to the left, and the mother saw clearly what would happen in the next second: she would step on the sleeping snake, the snake would rear up and then -

There is no time, screamed the mother’s mind. All her thoughts were wiped away in an instant. The oldest and deepest part of her brain, the medulla oblongata, forged in a billion years of silent struggle under dark seas, flashed like a thunderbolt. Her body flooded with adrenaline. Time slowed to a crawl. She felt no pain, she moved with superhuman speed, her sensorium exquisitely attuned.

But there was still no time. No time to grab the child, no time even to jump on the snake and distract it, sacrificing herself. They were too far away, a mere ten feet, but it might as well have been ten thousand. She could yell, but this would startle the snake as much as the child. She could gesture, but the child’s back was to her. There was no way out. There was no way out.

The mother’s mind was on fire. It scanned through a million branching possibilities. Everything she could see, everything she had ever learned, everything evolution had etched in her cells, all the hard learned lessons of a trillion beings gasping for life; every ounce of strength, every molecule of blood sugar, all was used up in this last desperate search for ...