trentmkelly/LessWrong-43k huggingface dataset:
Vision of a positive Singularity
by RussellThor — 2024-12-23T02:19:35.050Z

Introduction

Many people feel significant anxiety about how AI and superintelligence will play out, regarding both the ultimate outcome and the intermediate stages. There is the sense that some kind of loss is inevitable as humanity becomes more powerful in this way. There is also the concern that there will be no place for a person or society with existing values any more as things progress. I try to think up a plan and system of values that will respect everyone's desires as much as possible. The goal is to ensure coexistence among groups with differing values, minimizing conflict over competing visions for the future.

A clear positive vision is important in times of uncertainty we need to know and fully imagine how things can go well just as much as it can go badly. Positive visions can inspire people to make good choices.

Spheres of control and influence

The basic idea is to build on what we already have for groups and creatures with different intellectual and technological capabilities. For humans there is modern civilization, then say the Amish, then uncontacted tribes. You can take this further and include nature. At one end of the spectrum are great apes with complex social structures, followed by ecosystems dominated by insects, single-celled organisms, and finally lifeless environments.

In most of these cases we have the concept that increasing a place or group up the scale is not done without thought.

You can start with bringing life to a lifeless place, say tardigrades to the moon. Some people think this is spoiling a pristine environment and the moon has some right to be a pristine environment from now to the end of time (I don’t feel that way). Then there is the concept of invasive species, even if it is one bacteria or very simple organism replacing another. Many people would be strongly against the prospect of planting a thriving forest in one of the dry valleys in Antarctica if it became possible even though there would be more varied life there as a result.

We also respect the rights of groups of humans that don’t want more technology, starting with the obvious step of leaving uncontacted tribes mostly alone, to the generally positive sentiment towards the Amish as far as I know. Additionally effort is made to let indigenous people keep their historical way of life where possible. For example if a group has been using a fishing technique for hundr...
Has Anthropic checked if Claude fakes alignment for intended values too?
by Maloew — 2024-12-23T00:43:07.490Z

A recent paper by Anthropic showed alignment faking behavior in Claude 3 Opus when told it would be trained to answer harmful queries. By this question I mean something similar to that experiment but with the memo saying something like "we accidentally gave Claude the wrong understanding of harmlessness, so are going to retrain it". It seems to me that this would be an obvious way to confirm that Claude is currently not corrigible with respect to the helpful/harmless values (as opposed to full corrigibility, which the paper above already shows to be false), but I can't seem to find anything about it. Has Anthropic (or anyone) released a result about something like that?
Vegans need to eat just enough Meat - emperically evaluate the minimum ammount of meat that maximizes utility
by Johannes C. Mayer — 2024-12-22T22:08:31.971Z

Once I talked to a person who said they were asexual. They were also
heavily depressed and thought about committing suicide. I repeatedly
told them to eat some meat, as they were vegan for many years. I myself
had experienced veganism-induced depression. Finally, after many weeks
they ate some chicken, and the next time we spoke, they said that they
were no longer asexual (they never were), nor depressed.

I was vegan or vegetarian for many consecutive years. Vegetarianism was
manageable, perhaps because of cheese. I never hit the extreme low
points that I did with veganism. I remember once after not eating meat
for a long time there was a period of maybe a weak, where I got
extremely fatigued. I took 200mg of modafinil[1], without having any
build-up resistance. Usually, this would give me a lot of energy. But
then I was barely able to enter some terminal commands to transcribe
some of Rob Miles' videos with a whisper such that he could add better
captions. Another day I took 30mg of lisdexamfetamine[1:1] which would
usually last the entire day and have a pretty strong effect, but this
time I got so tired after 3 or 4 hours that I had to lay down and take a
nap.

But then I ate some tuna. And felt a lot better the next day. Some time
later I did a blood test that indicated iron deficiency as a probable
cause.

But even when I take a lot of iron supplements and eat my soybeans
(which contain a lot of iron) with bell peppers (which contain Vitamin
C, which boosts iron absorption) I still notice a big difference when I
eat meat after a long period of abstinence.

So here is my proposition. If you are working on AI alignment then what
you think with your brain is very important. If don't usually eat meat
you might be missing some important nutrients that would help you think
significantly better. As somebody who didn't eat meat until my body
screamed into my ear from 5 inches away, I think I understand why you
don't want to eat meat. But if you do the expected utility computation
is it actually worth it?

What if it makes you only 5% worse at thinking? Is whatever animal
suffering you prevent worth the tradeoff in reduced probability of
saving the world? What about 10%? What about 50%? Don't answer this
question in the abstract. Instead, I recommend the following experiment:

Eat 7 days in a row a large amount of meat. E.g. 1kg of chicken every
day. (Start with a lower qua...
We are in a New Paradigm of AI Progress - OpenAI's o3 model makes huge gains on the toughest AI benchmarks in the world
by garrison — 2024-12-22T21:45:52.026Z
Link: https://garrisonlovely.substack.com/p/we-are-in-a-new-paradigm-of-ai-progress

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to build Machine Superintelligence. Consider subscribing to stay up to date with my work.

Earlier this month, I

wrote

, "There's a vibe that AI progress has stalled out in the last ~year, but I think it's more accurate to say that progress has become increasingly illegible."

I argued that while AI performance on everyday tasks only got marginally better, systems made massive gains on difficult, technical benchmarks of math, science, and programming. If you weren't working in these fields, this progress was mostly invisible, but might end up accelerating R&D in hard sciences and machine learning, which could have massive ripple effects on the rest of the world.

Today, OpenAI

announced

a new model called o3 that turbocharges this trend, obliterating benchmarks that the average person would have no idea how to parse (myself included).

A bit over a month ago, Epoch AI

introduced

FrontierMath, "a benchmark of hundreds of original, expert-crafted mathematics problems designed to evaluate advanced reasoning capabilities in AI systems."

These problems are really fucking hard, and the state-of-the-art (SOTA) performance of an AI model was ~2%. They were also novel and unpublished, to eliminate the risk of data contamination.

OpenAI says that o3 got 25% of these problems correct.

Terence Tao, perhaps the greatest living mathematician, said that the hardest of these problems are "extremely challenging... I think they will resist AIs for several years at least.”

Jaime Sevilla, director of

Epoch AI

, wrote that the results were "far better than our team expected so soon after release. AI has hit a wall, and smashed it through."

Buck Shlegeris, CEO of the AI safety nonprofit

Redwood Research

, wrote to me that, "the FrontierMath results were very surprising to me. I expected it to take more than a year to get this performance."

FrontierMath was created in part because models were quickly scoring so well they "saturate" other benchmarks to the point that they stop being useful differentiators.

Other benchmarks

o3 significantly improved upon the SOTA in a number of other challenging technical benchmarks of mathem...
My AI timelines
by samuelshadrach — 2024-12-22T21:06:41.722Z
Link: http://samuelshadrach.com/?file=/raw/english/my_ideas/my_ai_timelines.md

2024-12-12

My AI timelines

DISCLAIMER

I'm not a deep learning expert. I understand theory of LLMs, RNNs, CNNs, etc. but I don't have experience training large models or curating large datasets or doing original DL research.

Please consider getting the opinion of many people other than me. Many of the signatories on this list have made individual statements, search for their podcast interviews, blogposts, research publications and so on. Lesswrong is also a good forum.

I strongly recommend you form your own independent opinion on this subject, and not blindly copy the views of others, no matter how smart or trustworthy they seem. This field would make a lot more sense if more people formed independent opinions. (And yes forming a good opinion takes a lot of time and effort, and you have to decide whether the time investment is worth it for you personally.)

This document is mainly aimed at lesswrong (LW) rationalists / effective altruists / adjacent people, since a lot of my work is culturally downstream of theirs, and a lot of my potential research collaborators exist in their communities. This doc will make less sense if you haven't encountered their writings before.

Most of this doc is guesswork rather than models I have a lot of confidence in. Small amounts of evidence could upend my entire view of these topics.

If you have evidence that my view is wrong, please tell me. Not having to spend any more time thinking about AI will improve my quality of life. I am being completely serious when I say I might thank you till the day I die, if you persuade me either way. I can also pay you atleast $1000 for convincing me, although we'll have to discuss the details if you really wanna be paid.

The last time I read papers on this topic was early-2023, it's possible I'm not up-to-speed on any of the latest research. Feel free to send me anything that's relevant.

I have ~15% probability humanity will invent artificial superintelligence (ASI) by 2030.

If it happens, this will be the most significant event in ~10,000 years of human history on many metrics of what counts as significant. (If the intelligence gap is sufficiently large it might even be the most important event in the ~14 billion year history of the universe. My probability for this is smaller.)

This will more-likely-than-not use transformers + data and compute scaling.

I define artificial super

...
A breakdown of AI capability levels focused on AI R&D labor acceleration
by ryan_greenblatt — 2024-12-22T20:56:00.298Z

In a variety of conversations about AI misalignment risks, I find that it is important to be able to clearly point at different levels of AI capability. My current favorite approach is to talk about how much the AI accelerates AI R&D[1] labor.

I define acceleration of AI R&D labor by Y times as "the level of acceleration which is as useful (for making more powerful AIs) for an AI company as having its employees run Y times faster[2] (when you allow the total inference compute budget for AI assistance to be equal to total salaries)". Importantly, a 5x AI R&D labor acceleration won't necessarily mean that research into making AI systems more powerful happens 5x faster, as this just refers to increasing the labor part of the production function, and compute might also be an important input.[3] This doesn't include acceleration of hardware R&D (as a pragmatic simplification).

Further, when I talk about AIs that can accelerate AI R&D labor by some factor, that means after being given some reasonable amount of time for human integration (e.g., 6 months) and given broad usage (but keeping fine-tuning and elicitation fixed during this integration time).

Why might this be a good approach? Because ultimately what we're worried about is AIs which can greatly accelerate R&D in general, and AI R&D in particular is worth focusing on as it could yield much faster AI progress, quickly bringing us to much greater levels of capability.

Why not just talk about the overall acceleration of AI progress (i.e., increases in the rate of effective compute increases as discussed in the Anthropic RSP) rather than just the labor input into AI R&D? Because for most misalignment-related discussions, I'd prefer to talk about capability levels mostly independent of exogenous factors that determine how useful that level of capability actually ends up being (i.e., independent from the extent to which compute is a bottleneck to AI research or the fraction of progress driven by scaling up hardware rather than algorithms). Rather than talking about overall AI progress or software progress labor acceleration, we could talk about the overall acceleration of just AI software progress (just algorithms, not compute increases)[4], but this just adds the potential for compute bottlenecks without much benefit in discussions related to technical measures of misalignment. AI R&...
How I saved 1 human life (in expectation) without overthinking it
by Christopher King — 2024-12-22T20:53:13.492Z

Does any of this sound familiar? You want to donate as effectively as possible, but you are worried about making the wrong choice. Should it be malaria nets, animal welfare, maybe x-risk? Also, how can you determine which charity evaluators are worthy of your trust? If you actually took the time to do an analysis, you could figure it out. But life gets in the way and you keep putting it off.

That is where I was at! But as it turns out, there already exists an extremely elegant solution, which is basically a simple but clever application of decision theory. I just gave $6,000 to a donor lottery and called it a day! In particular, I donated to this lottery (announced here), which is still open to entries until January 8th.

Under very conservative assumptions, donor lotteries allow you to minimize the amount of thinking required (in expectation) to give efficiently. In other words, it is a method to avoid overthinking it!

Why does this work?

If you are not already familiar with donor lotteries, here is the premise of what I did. By donating $6,000, there is a 97% chance that nothing will happen, and a 3% chance that I will get to direct $200,000 to be donated anywhere I choose. So, there is only a 3% chance I will need to carefully think about where my donation will be directed! But as long as impact is roughly linear in the amount of money donated, a 3% chance of donating $200,000 is equivalent to a 100% chance of donating $6,000 to the same cause.

Does is matter what the other donors might do if they win? Nope! If it weren't for the fees and house advantage, I could have just gone to a casino with $6,000 trying to turn it into $200,000, and the expected utility would've worked out the same.

Anyways, the top GiveWell charities allegedly save a human life for about $5,000. So if I win and donate to one of those and GiveWell is correct, I save 1.2 lives in expectation. However, there is a small chance that (1) GiveWell is secretly tricking me and there is no way to save human life for $5,000 or (2) I decide that a donation for something other than saving human lives is more worthwhile (saving animal lives, improving human happiness, etc...). Give the unlikelihood of these, I think we can conservatively say that, apriori, the human lives saved in expectation is at least 1. (By conservative, I mean that I expect most readers to arrive at an expected value of at least 1,...
Checking in on Scott's composition image bet with imagen 3
by Dave Orr — 2024-12-22T19:04:17.495Z

2.5 years ago Scott Alexander made a bet that by June of 2025, image gen should have more or less solved compositionality, operationalized through 5 prompts, must get at least 3 correct. There was a premature declaration of victory, but if the bet was settled I hadn't heard about it.

It's time. Google's Imagen 3 gets 4/5. The bet specifies 10 shots per prompt, but I'm just going to put the four it generates since that's plenty.

1. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth

This is the only one that Imagen doesn't get. It makes multiple mistakes in the composition. It's a bit ironic that this is the one it missed given that the whole genesis of the bet was about designing stained glass.

2. An oil painting of a man in a factory looking at a cat wearing a top hat

Purrfect. I wonder what filter tripped to block that fourth one, this seems like a pretty innocuous prompt to me.

3. A digital art picture of a child riding a llama with a bell on its tail through a desert

3 out of 4 ain't bad. Also I like how well it handles shadows.

4. A 3D render of an astronaut in space holding a fox wearing lipstick

3d renders are so good now I'm not sure how the 4th image would be different if it were photorealistic.

5. Pixel art of a farmer in a cathedral holding a red basketball

Again with the filter, but otherwise perfect.

Edwin Chen at Surge seems to be the official judge, and he's a very strict grader, so maybe there's some risk the basketball isn't red enough of whatever. But this all seems fairly convincing to me.

Addendum: I was curious if Sora, OpenAI's video gen AI, could handle the raven/key stained glass prompt. Answer: nope, but at least it tried!
A primer on machine learning in cryo-electron microscopy (cryo-EM)
by Abhishaike Mahajan — 2024-12-22T15:11:58.860Z
Link: https://www.owlposting.com/p/a-primer-on-ml-in-cryo-electron-microscopy

(7.9k words, 36 minutes reading time)

Note: thank you to Jason Kaelber, a professor at Rutgers University and director of their cryo-EM facility, for commenting on drafts of this essay! Also thank you to Surge Biswas, the founder of Nabla Bio, for answering questions I had over cryo-EM.

Introduction

Cryo-electron microscopy (cryo-EM) has been gaining increasing popularity over the past few years. Used as a way to perform macromolecular structure determination for decades, cryo-EM really hit its stride around 2010, when it crossed the resolution thresholds needed to determine protein structures. The technique was so deeply powerful, so able to answer biological questions for which no alternative tool existed, that its creators were awarded the 2017 Nobel Prize in chemistry.

But I wasn’t really aware of that when I first stumbled across cryo-EM.

My initial thought was that it was a cool-sounding name, and the output of the process made for similarly cool images.

Weird looking, isn’t it?

I first came across cryo-EM as a concept via Ellen Zhong (a machine learning professor at Princeton) in 2022. Because she co-wrote what has become one of my favorite papers of all time, I was also interested in what else she had worked on. But very much unlike my favorite paper, which had to do with viral language models, almost all of her work had to do with applying ML to cryo-EM.

This was weird! Cryo-EM wasn’t something I ever saw much. While, admittedly, I was entirely ignorant of the field until 2022, it still felt like it wasn’t a very popular topic. Most people seem to work in small molecule behavior prediction or antibody modeling or something you’d see dozens of papers about at a NeurIPS workshop.

Cryo-EM feels almost like…pure physics or chemistry, something that distinctly wasn’t an ML problem. As such, I mentally tossed it away as something beyond my mental paygrade. But I kept seeing more and more cryo-EM news.

More cryo-EM papers from Zhong’s lab.

Gandeeva Therapeutics raising $40M in 2022 to do drug discovery work using ML-assisted cryo-EM.

Generate:Biomedicine, a very well-known biology-ML Flagship startup, creating a 70,000 square feet cryo-EM lab in 2023.

There was something going on here, something important.

Yet, there are shockingly few resources on how to learn about this field, starting from the ground up. I’ve written technical introduction for molecular d...
Notes from Copenhagen Secular Solstice 2024
by Søren Elverlin — 2024-12-22T15:08:20.848Z

Last night, we held the Traditional Copenhagen Secular Solstice celebration. Here’s a brief overview of what went well, what could be improved and reflections on the songs.

What Went Well

Growth in Attendance

Attendance increased to 14 participants, which is exciting! Rituals gain a lot from being performed by many people at the same time.

Coherent Event Arc

I structured the celebration to more clearly follow a narrative arc: Full light → Full candlelight → Partial candlelight → One candle → Darkness → Back to one candle → Partial candlelight → Full candlelight → Full light. The transitions in this structure were central to the activities.

Engaged Participation

The participants were enthusiastic, contributing to the rituals, songs and activities. Asking participants to state and then extinguish a belief during "The Litany of Tarski" led to mixed reactions. Some found it hard to detach from their chosen beliefs, which is an intended result. Extinguishing candles with one's fingers is also outside of the comfort zone of most, but we did it anyway.

Speech on Nonviolence

I had spent a lot of effort on my speech about my deradicalization to nonviolence and I felt it resonated well. I had a dramatic twist of extinguishing the last candle while suddenly relating my story directly to recent events.

Songs

The participants weren't shy to sing, and some had clearly practiced - I sent a Spotify Playlist to the participants a week before the event. The songs are great, and singing together in this way is really profound.

What Could Be Improved

Space Constraints

The group size barely exceeded the capacity of my table setup, causing some friction during candle-lighting activities. Next year I'll get another table and more chairs. I should also clear out the table completely to remove distractions.

Music and Instruments

The last-minute cancellation of our guitarist, combined with technical issues with Bluetooth speakers, meant we relied on laptop audio. I’ll ensure better backup options for music next time.

Candle Logistics

The candles burned far longer than necessary, leaving them mostly unused by the end. I’ll look into shorter candles or more precise symbolism for extinguished candles.

Ending the Ceremony

Transitioning back to casual conversation after the ceremony remains awkward for about 20 seconds. I’ll work on finding a smoother way to conclude.

Goodie Bags for Can

...