trentmkelly/LessWrong-43k huggingface dataset:
Extended Navel-Gazing On My 2023 Donations
by jenn — 2023-12-31T18:10:29.311Z
Link: https://jenn.site/2023/12/donations-the-third-year/

Previously: Donations, The First Year

Here's an update on what my household donated to this year, and why. Please be warned that there is some upsetting content related to the ongoing Israel-Hamas conflict in this post, in the first section.

The Against Malaria Foundation

Around 90% of our donations ($15,000 of $16,500 total, all amounts in CAD) went to the Against Malaria Foundation (AMF). I remain a very old school EA mostly committed to global health and poverty reduction interventions for humans.

If I was a US citizen I'd donate a portion of this to GiveWell's Unrestricted Fund for reasons I'll touch on below, but as a Canadian the key consideration for me was which GiveWell-recommended charities and funds had a Canadian entity, and unfortunately (or fortunately for eliminating analysis paralysis?) the AMF was the only recommended charity registered in Canada. This meant I could donate tax-deductibly, which meant I can donate ~20% more.

(Or so I thought at the time. I've now discovered CAFCanada, but that's a problem for my 2024 donations.)

The AMF almost didn't get my donation this year.

According to Givewell's 2021 analysis, the AMF saves in expectation one life for every $7300 CAD donated. In the days after the onset of the Israel-Palestinian conflict, I began researching nonprofits offering medical aid to Palestinians, thinking that there's a chance their impact might surpass that benchmark[1].

I read many annual reports for many charities, focusing extra on their work in previous years of conflict. In the end none of them were anywhere close to how effective the AMF is (like at least an order of magnitude off), with one exception.

Glia Gaza is a small team of Canadian doctors who are providing emergency care and 3D printed tourniquets to wounded Palestinians. The tourniquets came in different sizes for women and children in addition to men (most suppliers only supply tourniquets in adult male sizes).

I researched the efficacy of tourniquets in saving lives. If you are dealing with bullet wounds, they help a lot when you use them to staunch bleeding and prolong the time you have to get to a hospital. They help, too, if there are no hospitals, just by significantly reducing the chance that you bleed out and die right there.

Tying a tourniquet is challenging; it's easy to make mistakes that could worsen the situation or fail to apply them tightly enough. Glia cre...
aisafety.info, the Table of Content
by Charbel-Raphaël — 2023-12-31T13:57:15.916Z

Here is a list of Q&A from https://aisafety.info/ . When I discovered the site, I was impressed by the volume of material produced. However, the interface is optimized for beginners. The following table of contents is for individuals who wish to navigate the various sections more freely. It was constructed by clustering the Q&A into subtopics. I'm not involved with aisafety.info, I just want to increase the visibility of the content they produced by presenting it in a different way. They are also working on a new interface. This table can also be found https://aisafety.info/toc/.

🆕 New to AI safety? Start here.

📘 Introduction to AI Safety

What is AI safety?

What is AI alignment?
What is AI governance?
What is the general nature of the concern about AI alignment?
What is the "control problem"?
What is the difference between AI safety, AI alignment, AI control, friendly AI, AI ethics, AI existential safety, and AGI safety?

Why would an AI do bad things?
How powerful will a mature superintelligence be?
Why is safety important for smarter-than-human AI?
How likely is extinction from superintelligent AI?
What are some introductions to AI safety?

🧠 Introduction to ML

What are large language models?

What is GPT-3?
What are OpenAI Codex and GitHub Copilot?
How does "chain-of-thought" prompting work?
How can progress in GPT-style non-agentic AI lead to capable AI agents?

What is compute?

What are scaling laws?
What are the "no free lunch" theorems?
What is the "Bitter Lesson"?

What is reinforcement learning (RL)?

🤖 Types of AI

What is artificial intelligence (AI)?
What is "narrow AI"? & What is artificial general intelligence (AGI)?
What is tool AI? & What is an agent?
What is "transformative AI"?
What are the differences between AGI, transformative AI, and superintelligence?
What is "superintelligence"?
What is a shoggoth?
What is "whole brain emulation"?
What are brain-computer interfaces?

🚀 Takeoff & Intelligence explosion

Takeoff

What is "AI takeoff"?
Why does AI takeoff speed matter?
What are the different possible AI takeoff speeds?
What is a singleton?

Intelligence explosion

What is an intelligence explosion?
How likely is an intelligence explosion?
How could an intelligence explosion be useful?

What are the differences between a singularity, an intelligence explosion, and a hard takeoff?

📅 Timelines

Expert surveys

What evidence do experts usually base their ti

...
AIOS
by samhealy — 2023-12-31T13:23:56.552Z

Summary

A reductio ad absurdum of the position that deep neural networks, given enough time, can do literally anything. In playful Socratic dialogue form.

Welcome to AIOS!

Snay/Coyle Systems is proud to announce AIOS.

Its notional 'clock speed' is 60Hz, representing a typical screen's refresh rate and a typical human interface device's polling rate. But it's no slouch!

Its outputs are just over 1.67 billion bits, representing up to 200MB of arbitrary data to be sent across a 100Gbps Ethernet connection per tick, a 4K video frame buffer/monitor output, and 0.0166 seconds of multi-channel high-definition audio.

Its inputs are its own previous output state and a 0.0166-second buffer of data from the Ethernet connection and standard human interface devices: keyboard, mouse/trackpad/touchscreen, and in future versions a microphone and webcam.

Its hidden layers contain 10^x neurons where x is very, very big. It was trained on 10^y sessions from consenting human users of Snay/Coyle's 'traditional' (and soon to be obsolete) human-coded operating system BSOS, where y is also very big.

The model was fine-tuned by RLHF.

And now it runs a usable facsimile of BSOS, without a single line of code!

That's the dumbest thing I ever heard.

Can you say why?

I mean, where to begin? First of all, of course there's code involved. How does the model run inference?

Fair point, that was a bit of a liberty. Obviously the model is running on standard CPU/GPU architecture. But after training, the weights of the model itself do not contain a trace of BSOS source code. Also gone are concepts like registers, instruction sets, CPU caching, discrete graphics processing, kernel space, user space, all those pesky details.

Fine, whatever. So you're saying that AIOS will feel exactly like BSOS to a human user?

As you know, DNNs are curve-fitting approximators, of necessarily finite precision, and even when running in discretised form on digital hardware, their outputs' least significant bits will stray somewhat from any precise, analytical ground truth. But the universal approximation theorem tells us we just need to keep increasing the approximation's precision to stay within an arbitrarily small error range.

So, will AIOS's outputs exactly match those of BSOS? No. Will they appear the same to human users? Yes! And if an outlying human savant can somehow tell the difference, then ...
AI Alignment Metastrategy
by Vanessa Kosoy — 2023-12-31T12:06:11.433Z

I call "alignment strategy" the high-level approach to solving the technical problem[1]. For example, value learning is one strategy, while delegating alignment research to AI is another. I call "alignment metastrategy" the high-level approach to converging on solving the technical problem in a manner which is timely and effective. (Examples will follow.)

In a previous article, I summarized my criticism of prosaic alignment. However, my analysis of the associated metastrategy was too sloppy. I will attempt to somewhat remedy that here, and also briefly discuss other metastrategies, to serve as points of contrast and comparison.

Conservative Metastrategy

The conservative metastrategy follows the following algorithm:

As much as possible, stop all work on AI capability outside of this process.
Develop the mathematical theory of intelligent agents to a level where we can propose adequate alignment protocols with high confidence. Ideally, the theoretical problems should be solved in such order that results with direct capability applications emerge as late as possible.
Design and implement empirical tests of the theory that incur minimal risk in worlds in which the theory contains errors or the assumptions of the theory are violated in practice.
If the tests show problems, go back to step 2.
Proceed with incrementally more ambitious tests in the same manner, until you're ready to deploy an AI defense system.

This is my own favorite metastrategy. The main reason it can fail is if the unconservative research we failed to stop creates unaligned TAI before we can deploy an AI defense system (currently, we have a long way to go to complete step 2).

I think that it's pretty clear that a competent civilization would follow this path, since it seems like the only one which leads to a good long-term outcome without taking unnecessary risks[2]. Of course, in itself that is an insufficient argument to prove that, in our actual civilization, the conservative metastrategy is the best for those concerned with AI risk. But, it is suggestive.

Beyond that, I won't lay out the case for the conservative metastrategy here. The interested reader can turn to 1 2 3 4 5.

Incrementalist Metastrategy

The incrementalist metastrategy follows the following algorithm:

Find an advance in AI capability (by any means, including trial and error).
Find a way to align the new AI design (prioritizing s

...
Does the hardness of AI alignment undermine FOOM?
by TruePath — 2023-12-31T11:05:49.846Z

Since the arguments that AI alignment is hard don't depend on any specifics about our level of intelligence shouldn't those same arguments convince a future AI to refrain from engaging in self-improvement?

More specifically, if the argument that we should expect a more intelligent AI we build to have a simple global utility function that isn't aligned with our own goals is valid then why won't the very same argument convince a future AI that it can't trust an even more intelligent AI it generates will share it's goals?

Note that the standard AI x-risk arguments also assume that a highly intelligent agent will be extremely likely to optimize some simple global utility function so this implies the AI will care about alignment for future versions of itself [1] implying it won't pursue improvement for the same reasons it's claimed we should hesitate to build AGI.

I'm not saying this argument can't be countered, but I think doing so at the very least requires clarifying the assumptions and reasoning claiming to show that alignment will be hard to achieve in useful ways.

For instance, do these arguments implicitly assume the AI we create is very different from our own brains so don't apply to AI self-improvement (tho maybe the improvement requires major changes too)? If so, doesn't that suggest that AGI that really closely tracks our own brain operation is safe?

--

1: except in the super unlikely case it happens to have the one exact utility function that says always maximize local increases in intelligence regardless of it's long term effect.
Speed of Failing
by nano_brasca — 2023-12-31T10:39:01.134Z

Introduction

Last night, I was watching Anatomy of a Fall.

During the film’s first minutes, we hear someone playing the piano entirely out of tempo.

As soon as the movie develops, the piano gets brighter, funnier, and even with a message.

The player is learning, and the piano becomes an instrument where this character can talk with the audience.

He hit the wrong notes, of course, but also he was dragging the keys to be an utterly unarranged set of notes.

In the end, as Debussy said, music is the space between notes or “silence.”

I

When I started playing the piano (or at least learning to play the piano), each key had a meaning, and after playing the wrong notes for a while, I started to hit the right ones. However, after a few minutes of trying to find the correct ones while reading the music sheet, I realized it didn’t sound like music, or at least it didn’t sound like the piece I was looking for. It was a John Cage piece, at most.

After hours and hours in front of the piano, increasing each movement’s speed and the keys’ overall correctness, I started to hit the notes I wanted correctly.

I played for a few hours more, and then, suddenly, the music began to appear. Note after note, Erik Satie’s Gymnopédie No. 1 at the correct phase started to appear.

Before practicing and reaching the correct tempo, the same notes I was playing didn’t sound like the piece at all.

Now, the exact same exercise made me think that speed is as important as being correct.

II

It’s been a few years since I stopped making music, producing, and playing altogether, although a good discovery gave me pause for thought last night.

It’s not correctness but speed that makes your work outstanding.

Furthermore, it doesn’t make sense if I now replicate what Darwin did, researching and getting those conclusions in the Galápagos, even if the DIY culture started to gain so much traction in the past few years.

At most, it will help you at a personal level, but it won’t reach any boundary or frontier; it will be a more (and totally valid) personal journey.

III

Now, to contribute ideas and create something, you need to combine two characteristics: you become good at the Pareto frontier if you are knowledgeable at something, but you also need the speed to keep it up.

Ideas come from the same family of thoughts: you can be correct in what you think you are saying, but you require speed to practice all that you claim.

Spe...
Estimating Returns to Intelligence vs Numbers, Strength and Looks
by TruePath — 2023-12-31T10:03:38.780Z

A key assumption in most x-risk arguments for AI is that the ability of an agent to exert control over the world increases rapidly with intelligence. After all, AI safety would be easy if all it required was ensuring that people remain far more numerous and physically capable than the AI or even ensuring that the total computational power available to AI agents is small compared to that available to humanity.

What these arguments require is that a single highly (but not infinitely) intelligent agent will have be able to overwhelm the advantages humans might retain in terms of numbers, looks and computational power either by manipulating people to do it's bidding or hacking other systems. However, I've yet to see any attempt to quantify the relationship between intelligence and control assumed in these arguments.

It occurs to me that we have information about these relationships that can inform such assumptions. For instance, if we wish to estimate the returns to intelligence in hacking we look at how the number of exploits discovered by researchers varies with their intelligence.

To estimate the returns to intelligence in terms of manipulation we could at the distribution of intelligence in highly effective politicians/media personalities and compare it to other traits like height or looks. Or even, if we assume that evolution largely selects for ability to influence others, look at the distribution of these traits in the population.

I realize that doing this would probably require a number of substantial assumptions but I'm curious if anyone has tried. And yes I realize this entirely ignores the issue of defining intelligence beyond human capability (though if the notion has any validity we could probably use something like the rate at which unknown theorems, weighted by importance, can be proved).
Planning to build a cryptographic box with perfect secrecy
by Lysandre Terrisse — 2023-12-31T09:31:47.941Z

Summary

Since September 2023, I started learning a lot of math and programming skills in order to develop the safest cryptographic box in the world (and yes, I am aiming high). In these four months, I learned important things you may want to know:

Fully Homomorphic Encryption (FHE) schemes with perfect secrecy do exist.
These FHE schemes do not need any computational assumption.
These FHE schemes are tractable (in the worst case, encrypting a program before running it makes it three times slower).
We can therefore run infinitely dangerous programs without obtaining any information about them or their outputs. This may be useful in order to run a superintelligence without destroying the world.
However, these schemes work only on quantum computers.

In this post, I will firstly talk about how I learned about this FHE scheme, then I will explain my plan for making this cryptographic box, and finally, I will mention some ethical concerns about this cryptographic box.

Before reading this post, I recommend you to read this post by Paul Christiano, and the comments that go with it. These are very informative, and they sharpened my views for this project. Paul Christiano presents a way to extract a friendly AI from an unfriendly one. This being only one example of what can be done with a cryptographic box, I will mostly consider cryptographic boxes as a solution to a problem that I call the malign computation problem.

Introduction

In August 2022, I started reading AGI Safety Literature Review. At one point, the authors tell this:

One way to box an AGI is to homomorphically encrypt it. Trask (2017) shows how to train homomorphically encrypted neural networks. By homomorphically encrypting an AGI, its predictions and actions also come out encrypted. A human operator with the secret key can choose to decrypt them only when he wants to.

When I have read this for the first time, I told myself that I should check this work because it seemed important.

And then I completely forgot about it.

Then, in April 2023, during a PHP lesson, I realized that the problem of processing a request made by a malevolent user is similar to the problem of boxing a superintelligence. After the lesson, I asked the teacher how to prevent code injections, and he gave me two answers:

Do not show your code to the public. This answer didn't convince me, because even current hackers know how to go around this

...
Does ChatGPT know what a tragedy is?
by Bill Benzon — 2023-12-31T07:10:54.254Z

This is cross-posted from New Savanna.

Tragedy is one of those things where one says, “I can’t define it, but I know it when I see it.” Consequently philosophers and literary scholars have been haggling over it for centuries. But I’m not interested in all the nuances.

Neither was Brian Phillips. Back in the Jurassic Era, by which I mean the 1970s, he was a student of David Hays at the same time I was. Hays had come up with the idea that abstract ideas can be defined over stories. Any story that meets the requirment can be considered an instance of that particular abstract idea.

Brian decided to investigate the idea of tragedy, which he defined as “Someone does a good act that: results in his death.” That won’t get you Macbeth or Othello, but it will do for crude computational purposes.

Here’s how Brian defined tragedy using a classical symbolic notation, a cognitive network.

The concept being defined, tragedy, is that node at the upper right. The graph to its left is the pattern that defines it. At the left we see the first clause of the definition, some agent (AGT) acts in a good way. That action is connected by a CAUSE arc to the second clause, which simply asserts that some person does. The bottom part of the diagram specifies that the agent in the first clause must be the experiencer (EXP) in the second. Both the agent and the experiencer are manifestations (MAN) of the same person.

Brian then collected a bunch of simple stories from newspapers which he could use to test the model. Some were instances of tragedy as he defined it and some where not. The system worked.

I decided to give ChatGPT a little test. Although I’m sure it could give me a reasonable explanation of what a tragedy is, I wanted to constrain it to the definition that Brian Phillips used. So I created six different two-part prompts. The first part of each prompt had the short definition the Brian Phillips used along with a request to determine whether or not the following story is a tragedy. The second part of the prompt consisted of a short story that I wrote. I presented each two-part prompt in a separate session. ChatGPT the right answer in each case.

Tragedy, First Trial

Let us say that tragedy is when someone does a good act that results in their death. Is the following story an example of tragedy?

John was along when he saw that a car had gone off the road an into a small river. He jumped in...
Taking responsibility and partial derivatives
by Ruby — 2023-12-31T04:33:51.419Z

A common pattern for myself over the years is to get into some kind of interpersonal ~"conflict", feel mildly to extremely indignant about how the other person is at fault, then later either through confrontation or reflection, realize that I actually held substantial responsibility. I then feel very guilty.

(When I say "conflict" I mean something broader, e.g. I mean to include cases where you're mad at your boss even if you never actually confront them.)

I noticed this pattern some years ago such I did become skeptical of my indignation even when I couldn't yet see where I was responsible. Yet this led me to a feeling of frustration. How is it that I'm always at fault? Why can I never be justifiably indignant at someone else?

I believe the answer to this can be explained via partial derivatives. It doesn't have to be explained via partial derivatives, but I think partial derivatives are this super great concept that's helpful all over the place[1], so I'm going to invoke it. See this footnote for a quick explanation[2].

Suppose we have a Situation in which there is a Problem. In the real world, any Situation is composed of a large number of parameters. The amount of Problem there is is a function of the parameters. And for any interpersonal situation, different parameters are controlled by the different parties involved in the situation.

The needlessly mathematical Partial Derivative Model of Interpersonal Conflict says that for any nontrivial situation, likely both partners control parameters that have non-negligible impact on how much of a Problem there is.

In other words, if you want to blame the other person, you'll succeed. And if you want to blame yourself, you'll succeed.

I have been good at doing those serially, but might be a better model to them in parallel: see all the ways in which each of you are contributing to the amount of Problem.

This isn't to say that always everyone is equally to blame. If someone runs a red light and hits your car, they're at fault even if you could have chosen to work from home that day. In many cases, it's less clear cut and I think it's worth tracking how each person is contributing.

The asymmetry in the situation is that by definition you control the parameters you're in control of, so it's worthwhile paying attention them. If you can get over being Right and instead focus on the outcomes you want, you might be able to atta...