trentmkelly/LessWrong-43k huggingface dataset:
How likely is it that AI will torture us until the end of time?
by Damilo — 2024-05-31T01:26:26.315Z

Disclaimer: in this post I touch on some very dark and disturbing topics. I'm talking about suicide, my reasoning may be wrong and should not be used to retroactively justify suicide.

I've been stuck on s-risks for over a month now. My life has been turned upside down since I first learned about this subject. So today I'm sharing my thoughts with you to possibly find out what you think and see other points of view.

Suffering risks (s-risks) are risks involving an astronomical amount of suffering, far more than the suffering that has taken place on Earth so far. The ones I'm going to focus on in this post are those related to a general AI (or even ASI) and which would affect us humans today, directly. The scenario that concerns me is that an ASI is torturing mankind until the end of time. Why is this? I don't know, though. Could it be malicious? Could it choose its utility function to maximize human suffering? Could a paperclip maximizer torture us if it's an energy source or a power to blackmail a benevolent AI? I'm not an AI expert, so I have no weight in the "will we succeed in controlling AGI or not" debate. I feel that, given the extent to which opinions are divided, anything can happen and that no one can therefore state with 100% certainty that s-risks won't occur. What's more, we're talking about an intelligence superior to our own, and therefore, by definition, unpredictable. The point I want to make in this post is centered on the non-zero probability that the creation of an agi will lead us to an eternal hell.

When we talk about things worse than death, about torture, I think that the human brain encounters a certain number of cognitive biases that push it to minimize the thing or simply ignore it because it's too uncomfortable. So I encourage you to work through these cognitive biases to get an objective view on the subject. One of the things that is often underestimated is how bad suffering can be. Our bodies are made up of a huge number of ultra-sensitive nerves that can be activated to send unbearable signals to the brain. It's so bad. Suffering can reach such high scales, it's appalling, horrifying. The worst possible pain seems to be fire. Apparently, people who come out of a fire and have been burned beg the firefighters to finish them off, such is the pain.

Even if s-risks are one chance in a billion, their severity makes up for it, due to thei...
Twin Peaks: under the air
by KatjaGrace — 2024-05-31T01:20:04.624Z

Content warning: low content

~ Feb 2021

The other day I decided to try imbibing work-relevant blog posts via AI-generated recital, while scaling the Twin Peaks—large hills near my house in San Francisco, of the sort that one lives near and doesn’t get around to going to. It was pretty strange, all around.

For one thing, I was wearing sunglasses. I realize this is a thing people do all the time. Maybe it’s strange for them too, or maybe theirs aren’t orange. Mine were, which really changed the situation. For one thing, the glowing streetscapes felt unreal, like cheap science fiction. But also, all kinds of beauty seemed to want photographing, but couldn’t be seen with my camera. It was funny to realize that I’m surrounded by potential beauty all the time, that I would see if I had different eyes, or different glasses, or different sensory organs all together. Like, the potential for beauty is as real as the beauty I do see. (This is perhaps obvious, but something being obvious doesn’t mean you know it. And knowing something doesn’t mean you realize it. I’d say I knew it, but hadn’t realized it.)

And then my ears were cornered in by these plugs spouting electronic declarations on the nature of coherent agents and such, which added to my sense of my head just not really being in the world, and instead being in a cozy little head cockpit, from which I could look out on the glowing alien landscape.

My feet were also strange, but in the opposite direction. I recently got these new sock-shoes and I was trying them out for the first time. They are like well-fitting socks with strong but pliable rubber stuff sprayed on the bottom. Wearing them, you can feel the ground under your feet, as if you were bare-foot. Minus the sharp bits actually lacerating your feet, or the squishy bits sullying them. Walking along I imagined my freed feet were extra hands, holding the ground.

I had only been up to Twin Peaks twice before, and I guess I had missed somehow exactly how crazy the view was. It was like standing on a giant breast, with a city-sea-bridge-forest-scape panoramaed around and under you over-realistically. The bridge disappeared into mystical mists and the supertankers swam epically on the vast blue expanse. I tried to photograph it multiple times but failed, partly because my camera couldn’t capture the warm orange tinge of the sea and the bridge rising from the burning m...
Is suffering like shit?
by KatjaGrace — 2024-05-31T01:20:03.855Z

People seem to find suffering deep. Serious writings explore the experiences of all manner of misfortunes, and the nuances of trauma and torment involved. It’s hard to write an essay about a really good holiday that seems as profound as an essay about a really unjust abuse. A dark past can be plumbed for all manner of meaning, whereas a slew of happy years is boring and empty, unless perhaps they are too happy and suggest something dark below the surface. (More thoughts in the vicinity of this here.)

I wonder if one day suffering will be so avoidable that the myriad hurts of present-day existence will seem to future people like the problem of excrement getting on everything. Presumably a real issue in 1100 AD, but now irrelevant, unrelatable, decidedly not fascinating or in need of deep analysis.
How have analogous Industries solved Interested > Trained > Employed bottlenecks?
by yanni kyriacos — 2024-05-30T23:59:39.582Z

It seems like one of the biggest problems* in AI Safety is that it is ridiculously hard to get good training (i.e. MATS is ridiculously competitive now) and employed (samesies).

Has anyone look across other categories (e.g. potential other sciences) to see how this problem has been solved? I assume at the most macro level it is going to be "Industry" vs "Government" but I'm looking for more concrete interventions.
Thoughts?
*we're turning away very smart, motivated, well-meaning and skilled people. This is bad.
Duckbill Masks Better?
by jefftk — 2024-05-30T23:40:01.779Z

Lai et al. just published

a paper looking at how well various masks worked for keeping others from getting sick ("source control"). Their university press office

summarized this as:

Study finds all masks effective, but 'duckbill' N95 masks
far outperform others, suggests they should be the standard in
high-risk settings

Now, I personally think duckbill masks are the best disposable masks:
they're cheap, comfortable, fit me well, and are more breathable due
to their larger area. Plus, as masks that manage to be unfashionable
even by the standards of N95 masks, if they weren't better in
non-fashion ways they wouldn't be on the market anymore. But the study
didn't show that they're the best kind of N95, only that they do
better than the other masks they tested, none of which were N95s.

The study compared five conditions: no mask, cloth mask, surgical
mask, KN95, and (duckbill) N95. This isn't a study that can tell you
anything about the differences between N95s!

To me the most interesting parts of the study were (a) they captured
the viral RNA and measured viral load with qPCR instead of using bad
proxies like particle count and (b) they found KN95s did way
worse than you'd expect from their filtration efficiency:

The majority of the KN95 respirators used in our study
(reported by an N95docon.org to have
consistently high filtration efficiency but variable and high flow
resistance) did not outperform loose-fitting masks and when including
other brands, KN95s met inferiority criteria compared to cloth masks
for total viral aerosol. One possible explanation is that we noted
that the KN95 respirators we provided were relatively stiff and did
not seal consistently along the entire perimeter of the mask. By
contrast, the cloth masks brought by our volunteers tended to wrap
farther around the face possibly providing better fit and lower flow
resistance. We used one surgical mask brand for these tests so that
result may not be representative of all masks; the same brand used
in prior CDC-funded studies of masks for influenza source control.
The relatively high flow resistance of KN95 filters, compared with
surgical and cloth masks, combined with poor fit tended to promote
leaks around the face seal.

I recall a lot of people (including us) using clips to convert the
KN95's elastic ear loops into a behind-the-head attachment, for ...
OpenAI: Helen Toner Speaks
by Zvi — 2024-05-30T21:10:02.938Z

Helen Toner went on the TED AI podcast, giving us more color on what happened at OpenAI. These are important claims to get right.

I will start with my notes on the podcast, including the second part where she speaks about regulation in general. Then I will discuss some implications more broadly.

Notes on Helen Toner’s TED AI Show Podcast

This seems like it deserves the standard detailed podcast treatment. By default each note’s main body is description, any second-level notes are me.

(0:00) Introduction. The host talks about OpenAI’s transition from non-profit research organization to de facto for-profit company. He highlights the transition from ‘open’ AI to closed as indicative of the problem, whereas I see this as the biggest thing they got right. He also notes that he was left with the (I would add largely deliberately created and amplified by enemy action) impression that Helen Toner was some kind of anti-tech crusader, whereas he now understands that this was about governance and misaligned incentives.

(5:00) Interview begins and he dives right in and asks about the firing of Altman. She dives right in, explaining that OpenAI was a weird company with a weird structure, and a non-profit board supposed to keep the company on mission over profits.

(5:20) Helen says for years Altman had made the board’s job difficult via withholding information, misrepresenting things happening at the company, and ‘in some cases outright lying to the board.’

(5:45) Helen says she can’t share all the examples of lying or withholding information, but to give a sense: The board was not informed about ChatGPT in advance and learned about ChatGPT on Twitter, Altman failed to inform the board that he owned the OpenAI startup fund despite claiming to be an independent board member, giving false information about the company’s formal safety processes on multiple occasions, and relating to her research paper, that Altman in the paper’s wake started lying to other board members in order to push Toner off the board.

I will say it again. If the accusation bout Altman lying to the board in order to change the composition of the board is true, then in my view the board absolutely needed to fire Altman. Period. End of story. You have one job.

As a contrasting view, the LLMs I consulted thought that firing the CEO should be considered, but it was plausible th

...
Non-Disparagement Canaries for OpenAI
by aysja — 2024-05-30T19:20:13.022Z

Since at least 2017, OpenAI has asked departing employees to sign offboarding agreements which legally bind them to permanently—that is, for the rest of their lives—refrain from criticizing OpenAI, or from otherwise taking any actions which might damage its finances or reputation.[1]

If they refused to sign, OpenAI threatened to take back (or make unsellable) all of their already-vested equity—a huge portion of their overall compensation, which often amounted to millions of dollars. Given this immense pressure, it seems likely that most employees signed.

If they did sign, they became personally liable forevermore for any financial or reputational harm they later caused. This liability was unbounded, so had the potential to be financially ruinous—if, say, they later wrote a blog post critical of OpenAI, they might in principle be found liable for damages far in excess of their net worth.

These extreme provisions allowed OpenAI to systematically silence criticism from its former employees, of which there are now hundreds working throughout the tech industry. And since the agreement also prevented signatories from even disclosing that they had signed this agreement, their silence was easy to misinterpret as evidence that they didn’t have notable criticisms to voice.

We were curious about who may have been silenced in this way, and where they work now, so we assembled an (incomplete) list of former OpenAI staff.[2] From what we were able to find, it appears that over 500 people may have signed these agreements, of which only 5 have publicly reported being released so far.[3]

We were especially alarmed to notice that the list contains a variety of former employees currently working on safety evaluations or AI policy.[4][5] This includes some in leadership positions, for example:

Bilva Chandra (Senior AI Policy Advisor, NIST)
Charlotte Stix (Head of Governance, Apollo Research)
Jack Clark (Co-Founder [focused on policy and evals], Anthropic)
Jade Leung (CTO, UK AI Safety Institute)
Paul Christiano (Head of Safety, US AI Safety Institute)

In our view, it seems hard to trust that people could effectively evaluate or regulate AI, while under strict legal obligation to avoid sharing critical evaluations of a top AI lab, or from taking any other actions which might make the company less valuable (as many regulations presumably would). So if any of these people are not s...
Clarifying METR's Auditing Role
by Beth Barnes — 2024-05-30T18:41:56.029Z

METR has not intended to claim to have audited anything, or to claim to be providing meaningful oversight or accountability, but there has been some confusion about whether METR is an auditor or planning to be one.

To clarify this point:

METR’s top priority is to develop the science of evaluations, and we don’t need to be auditors in order to succeed at this.

We aim to build evaluation protocols that can be used by evaluators/auditors regardless of whether that is the government, an internal lab team, another third party, or a team at METR.

We should not be considered to have ‘audited’ GPT-4 or Claude.

Those were informal pilots of what an audit might involve, or research collaborations – not providing meaningful oversight. For example, it was all under NDA – we didn’t have any right or responsibility to disclose our findings to anyone outside the labs – and there wasn’t any formal expectation it would inform deployment decisions. We also didn’t have the access necessary to perform a proper evaluation. In the OpenAI case, as is noted in their system card:
“We granted the Alignment Research Center (ARC) early access to the models as a part of our expert red teaming efforts … We provided them with early access to multiple versions of the GPT-4 model, but they did not have the ability to fine-tune it. They also did not have access to the final version of the model that we deployed. The final version has capability improvements relevant to some of the factors that limited the earlier models power-seeking abilities, such as longer context length, and improved problem-solving abilities as in some cases we've observed. … fine-tuning for task-specific behavior could lead to a difference in performance. As a next step, ARC will need to conduct experiments that (a) involve the final version of the deployed model (b) involve ARC doing its own fine-tuning, before a reliable judgment of the risky emergent capabilities of GPT-4-launch can be made”.

We are and have been in conversation with frontier AI companies about whether they would like to work with us in a third-party evaluator capacity, with various options for how this could work.

As it says on our website:
“We have previously worked with

Anthropic

,

OpenAI

, and other companies to pilot some informal pre-deployment evaluation procedures. These companies have also given us some ki

...
A civilization ran by amateurs
by Olli Järviniemi — 2024-05-30T17:57:32.601Z

I

When I was a child, I remember thinking: Where do houses come from? They are huge! Building one would take forever! Yet there are so many of them!

Having become a boring adult, I no longer have the same blue-eyed wonder about houses, but humanity does have an accomplishment or two I'm still impressed by.

When going to the airport, the metal boulders really stay up in the air without crashing. Usually they leave at the time they told me two weeks earlier, taking me to the right destination at close to the speed of sound.

There are these boxes with buttons that you can press to send information near-instantly anywhere. They are able to perform billions of operations a second. And you can just buy them at a store!

And okay, I admit that big houses - skyscrapers - still light up some of that child-like marvel in me.

II

Some time ago I watched the Eurovision song contest. For those who haven't seen it, it looks something like this:

A representative sample of Eurovision 2024

It's a big contest, and the whole physical infrastructure - huge hall, the stage, stage effects, massive led walls, camera work - is quite impressive. But there's an objectively less impressive thing I want to focus on here: the hosts.

I basically couldn't notice the hosts making any errors. They articulate themselves clearly, they don't stutter or stumble on their words, their gestures and facial expressions are just what they are supposed to be, they pause their speech at the right moments for the right lengths, they could fluently speak some non-English languages as well, ...

And, sure, this is not one-in-a-billion talent - there are plenty of competent hosts in all kinds of shows - but they clearly are professionals and much more competent than your average folk.

(I don't know about you, but when I've given talks to small groups of people, I've started my sentences without knowing how they'll end, talked too fast, stumbled in my speech, and my facial expressions probably haven't been ideal. If the Eurovision hosts get nervous when talking to a hundred million people, it doesn't show up.)

III

I think many modern big-budget movies are pretty darn good.

I'm particularly thinking of Oppenheimer and the Dune series here (don't judge my movie taste), but the point is more general. The production quality of big movies is extremely high. Like, you really see that these are not amateur projects filmed in ...
One week left to apply for the Roots of Progress Blog-Building Intensive
by jasoncrawford — 2024-05-30T16:55:12.313Z
Link: https://rootsofprogress.org/one-week-to-apply-for-2024-fellowship

The application deadline for the the 2024 cohort of The Roots of Progress Blog-Building Intensive is Friday, June 7—just over a week away. If you want to apply, do it now.

The Blog-Building Intensive is an eight-week program for aspiring progress writers to start or grow a blog. It also makes you a Roots of Progress fellow, which means that even after the intensive, you are part of our network and we are committed to supporting your career success as a progress writer. See more details on the program homepage.

Not just for beginning writers

Are you an experienced writer, and wondering if you’ll get anything out of the program? It is not only for beginning writers!

Last cohort, many of our fellows were experienced professionals: Several worked full-time for relevant think tanks, some had academic positions, some had published in major media outlets, some had successful Substacks with large audiences.

They joined the fellowship for various reasons: to grow their audience, to build a personal brand, to write more in their own voice, to meet our fabulous lineup of advisors, to get more connected to the progress community, to join a peer group of writers excited about progress.

Brian Balkus, who had already published in Palladium Magazine, said:

I found this program incredibly helpful even as a somewhat more established writer. It enabled me to meet some fantastically intelligent and interesting writers and develop relationships that will hopefully extend beyond the program. No matter where you are in your writing career this program can help refine your thinking, make you a better writer, and grow professionally.

Elle Griffin, who already had over 10,000 Substack subscribers, said:

The Roots of Progress Fellowship was life-changing. In 10 years, all of the fellows in my program will be famous intellectuals who are actively creating a better future, and I’ll know it all started here and that we built it together.

Jenni Morales, who was a researcher at the Center for Growth and Opportunity, said:

This is the most unique group of people and writers and intellectuals that I’ve had the opportunity to sort of bump shoulders with, and it’s been so valuable and so interesting. I’ve just been exposed to so many new and interesting ideas. I think the rigor and level of thought that they put into their work is really impressive.

So, don’t worry that you’re overqualified. Just apply.

Bu

...
Getting started with AI Alignment research: how to reproduce an experiment from research paper
by Alexander230 — 2024-05-30T14:51:49.337Z

This is a post with technical instructions, how to reproduce an experiment from Weak-to-strong generalization paper: https://openai.com/index/weak-to-strong-generalization/. It’s oriented mostly on beginners in AI Alignment who want to start tinkering with models and looking for examples how to do experiments.

Weak-to-strong generalization is research that shows that a strong model can learn on data generated by a weaker model, generalize the data and surpass the weaker model in the task for which it was trained. The paper comes with example code on GitHub with experiments both on LLMs and vision models. However, running the experiments from this code is not a straightforward task, so here are detailed instructions how to do it.

Setup

Find a GPU cloud provider that gives access to terminal and Jupyter notebook. I used runpod.io for my experiment, selected a node with 1 RTX A6000 graphics card with 48 GB VRAM. The main limiting factor for the most of the experiments is VRAM size, so choose your node based on it and on the price; other characteristics are less important. Also, make sure that disk size of your node is at least 60 GB. Most of the cloud providers allow increasing disk size in settings, so do it if the disk is too small.
Register the account, rent a node, and follow cloud provider’s instructions to connect to it with terminal and Jupyter notebook.
Go to terminal and clone the repository:

git clone https://github.com/openai/weak-to-strong
cd weak-to-strong

I recommend to use virtual terminal, such as tmux or screen: it will ensure that you will not lose your run if the connection to server will drop in the middle of an experiment. If the server uses Ubuntu or Debian, run commands:

apt-get install tmux
tmux

If the connection will drop, reconnect to the server and run the command tmux attach to get back to your experiment. To scroll up and down in tmux, use Ctrl-B, [ keys sequence, then scroll up and down with arrows. Press Esc to exit scrolling mode.
Install the package and dependencies:

pip install .
pip install matplotlib seaborn tiktoken fire einops scipy

Running experiment with LLMs

Now everything is ready to run an experiment with LLMs. The code was probably written for older versions of libraries, and it will end with error if run on new versions as is, but it can be easily fixed.

Use a text editor (such as nano) to edit the file weak_to_strong/train.py, go

...