trentmkelly/LessWrong-43k huggingface dataset:
Have I Solved the Two Envelopes Problem Once and For All?
by JackOfAllTrades — 2024-03-19T19:57:20.682Z

I was about today years old when I learned of the two envelopes problem during one of my not-so-unusual attempts to do a breadth-first-search of the entirety of Wikipedia. Below is a summary of the relevant parts of the relevant article. (For your convenience, I omitted some irrelevant details in the "switching argument".)

Problem

A person is given two indistinguishable envelopes, each of which contains a sum of money. One envelope contains twice as much as the other. The person may pick one envelope and keep whatever amount it contains. They pick one envelope at random but before they open it they are given the chance to take the other envelope instead.

The switching argument

Now suppose the person reasons as follows:

Denote by A the amount in the player's selected envelope.
The probability that A is the smaller amount is 1/2, and that it is the larger amount is also 1/2.
The other envelope may contain either 2A or A/2.
...

The puzzle

The puzzle is to find the flaw in the line of reasoning in the switching argument. This includes determining exactly why and under what conditions that step is not correct, to be sure not to make this mistake in a situation where the misstep may not be so obvious. In short, the problem is to solve the paradox. The puzzle is not solved by finding another way to calculate the probabilities that does not lead to a contradiction.

The Wikipedia article currently has 37 citations and describes a number of proposed solutions to the problem. I would assume that there are some insights in each of the described proposals, but I felt that reading them would most likely distract me with false leads. So, I'm not entirely sure what others have said about the problem. However, my solution is 101 words long. So, if I have to read 1,001 words in order to figure out whether someone else has solved it, then, in a certain sense, they haven't "solved" it. Maybe I haven't either. Maybe there is a 17-syllable haiku that could serve as a solution. But the compression algorithm underlying my uses of language is not, I suspect, haiku-complete.

Preface

The apparent existence of a problem here is a consequence of the way in which we are attempting to explain the apparent existence of a problem. Therefore, we could call it an illusory problem. However, such a label is misleading because even illusory problems can rather quickly turn into problems that are very real wh...
How can one be less wrong, if their conversation partner loses the interest on discussing the topic with them?
by Ooker — 2024-03-19T18:11:29.401Z

I must admit that I'm a fan of this video:

I understand that there are topics that you are interested, and there are not. I, for example, am currently interested with this topic and ignore the rest of others in the homepage.

If you decide to join this conversation, I guess it's safe to say that the topic stimulates the interest of yours than other topics. So we share a mutual interest. Let's say we start talking back and forth, and then you lose the interest of it while we are still not completely agree in all aspect, then how can the goal of reasoning be achieved? I understand that our conversations can be long, and we all need rest, but I suppose we our minds should keep our interest on it until we share an agreement. That's how I take on the theory of reasoning in the video. I guess if one of use lose the interest before an agreement comes, then the purpose of reasoning fails, and that's bad for both of us.

I can end this question by asking what you think about this, and I will. What do you think about this? But I guess the most valuable answers are not from the ones whose interest are still fresh, but the ones who have lost the interest and about to abandon it. What makes your interest lost? Do you think the reasoning goal has been fulfilled?
Carlo: uncertainty analysis in Google Sheets
by ProbabilityEnjoyer — 2024-03-19T17:59:48.612Z
Link: https://carlo.app/

I've been working on Carlo, a tool that lets you do uncertainty and sensitivity analysis with Google Sheets spreadsheets.

Please note Carlo is an (expensive) commercial product. The pricing is aimed at professionals making important decisions.

There's a lot more detail at the link, but in brief, some key features of Carlo are:

Works with your existing Google Sheets calculations

Variance-based sensitivity analysis

Our sensitivity analysis offers a true metric of variable importance: it can tell you what fraction of the output variance is due to each of the inputs and their interactions.

Flexible input

Inputs can be given using novel, convenient probability distributions that flexibly match your beliefs.
AI Safety Evaluations: A Regulatory Review
by Elliot Mckernon — 2024-03-19T15:05:23.769Z

This article is the second in a series of ~10 posts comprising a 2024 State of the AI Regulatory Landscape Review, conducted by the

Governance Recommendations Research Program

at

Convergence Analysis

. Each post will cover a specific domain of AI governance (e.g. incident reporting, safety evals, model registries, etc.). We’ll provide an overview of existing regulations, focusing on the US, EU, and China as the leading governmental bodies currently developing AI legislation. Additionally, we’ll discuss the relevant context behind each domain and conduct a short analysis.

This series is intended to be a primer for policymakers, researchers, and individuals seeking to develop a high-level overview of the current AI governance space. We’ll publish individual posts on our website and release a comprehensive report at the end of this series.

Let us know in the comments if this format is useful, if there are any topics you'd like us to cover, or if you spot any errors or omissions!

Context

Governments and researchers are eager to develop tools and techniques to evaluate AI. These include risk assessments that are common in industry regulation, but also techniques that are more unique to advanced AI, such as capability evaluations and alignment evaluations.

In this section, we’ll define some terms and introduce some recent research on evaluating AI. Most existing AI regulation is yet to incorporate these new techniques, but many experts believe they’ll be a critical component of long-term safety (such as

responsible scaling policies

), and

many regulatory proposals

from experts include calls for specific assessment systems and requirements, which we’ll discuss shortly.

There are three main features of AI models that people are interested in evaluating:

Safety: How likely is this model to cause harm? Assessing the safety of AI models is crucial but difficult due to their enormous flexibility. Safety assessors often use techniques from other industries, such as

red-teaming

, where trained users deliberately, actively try to prompt dangerous behavior or unintended behavior, a technique derived from airport and cyber-security.
Capability: How powerful is this model? AI developers often like to use benchmarks to boast about their models, publishing demonstrations or tests of computational power or novel behaviors and features. Capabil

...
Mechanism for feature learning in neural networks and backpropagation-free machine learning models
by Matt Goldenberg — 2024-03-19T14:55:59.296Z
Link: https://www.science.org/doi/10.1126/science.adi5639

Understanding how neural networks learn features, or relevant patterns in data, for prediction is necessary for their reliable use in technological and scientific applications. In this work, we presented a unifying mathematical mechanism, known as Average Gradient Outer Product (AGOP), that characterized feature learning in neural networks. We provided empirical evidence that AGOP captured features learned by various neural network architectures, including transformer-based language models, convolutional networks, multi-layer perceptrons, and recurrent neural networks. Moreover, we demonstrated that AGOP, which is backpropagation-free, enabled feature learning in machine learning models, such as kernel machines, that apriori could not identify task-specific features. Overall, we established a fundamental mechanism that captured feature learning in neural networks and enabled feature learning in general machine learning models.
Monthly Roundup #16: March 2024
by Zvi — 2024-03-19T13:10:05.529Z

AI developments have picked up the pace. That does not mean that everything else stopped to get out of the way. The world continues.

Do I have the power?

Emmett Shear speaking truth: Wielding power is of course potentially dangerous and it should be done with due care, but there is no virtue in refusing the call.

There is also an art to avoiding power, and some key places to exercise it. Be keenly aware of when having power in a given context would ruin everything.

Natural General Lack of Intelligence in Tech

Eliezer Yudkowsky reverses course, admits aliens are among us and we have proof.

Eliezer Yudkowsky: To understand the user interfaces on microwave ovens, you need to understand that microwave UI designers are aliens. As in, literal nonhuman aliens who infiltrated Earth, who believe that humans desperately want to hear piercingly loud beeps whenever they press a button.

One junior engineer who hadn’t been taken over and was still actually human, suggested placing a visible on-off switch for turning the sound off — for example, in case your spouse or children were sleeping, and you didn’t want to wake them up. That junior engineer was immediately laughed off the team by senior aliens who were very sure that humans wanted to hear loud screaming beeps every time they pressed buttons. And furthermore sure that, even if anyone didn’t want their microwave emitting piercingly loud beeps at 4am, they would be perfectly happy to look up a complicated set of directions for how to turn the sound on or off, rather than needing a visible on-off switch. And even if any humans had trouble remembering that, they’d be much rarer than humans who couldn’t figure out how to set the timer for popcorn without a clearly labeled “Popcorn” button, which does a different random thing in every brand of microwave oven. There’s only so much real estate in a microwave control panel; it’s much more important to have an inscrutable button that says “Potato”, than a physical switch that turns the sound off (and which stays in the same position after power cycles, and which can be inspected to see if the sound is currently off or on).

This is the same species of aliens that thinks humans want piercing blue lights to shine from any household appliance that might go in somebody’s bedroom at night, like a humidifier. They are genuinely aghast at the thought that anyone mi

...
Experimentation (Part 7 of "The Sense Of Physical Necessity")
by LoganStrohl — 2024-03-18T21:25:56.527Z

This is the seventh post in a sequence that demonstrates a complete naturalist study, specifically a study of query hugging (sort of), as described in The Nuts and Bolts of Naturalism. This one demos phase four: Experimentation. For context on this sequence, see the intro post. Reminder that this is meant as reference material.

Wait, there’s more to this study? But we’ve just discussed the main insight that came out of it, and how it illustrates the point of naturalism. Why is there more?

There is more because by this point I was interested not only in insights, but in mastery. There is more to mastery than reconceptualization.

However, I would like to point out that everything I’d done so far preceded experimentation. I had not even begun to try to change anything—yet I had learned quite a lot, through mere observation without interference. This is why many naturalist studies are complete before experimentation even begins. Often, this level of understanding is all that’s needed.

(From the end of “

Naturalist Collection

”)

But sometimes one further step is necessary. You can tell that you should move on to “Experimentation” if you feel grounded about your study topic, if you think you've really trained yourself to notice and directly observe what’s there in whatever realm you’ve focused on—but you still have an unsatisfied curiosity about how to behave around your topic.

In this case, when I arrived at the end of Collection, I found that I wanted to know what was possible. I wanted to move freely around this chest luster, this sense of physical necessity; to explore its boundaries and the actions available to me in the presence of that experience (and its antecedents). So, I chose to continue my study.

The goal of

experimentation in naturalism

is to create space from alternative action.

If you’re constantly observing in response to a stimulus, rather than immediately taking whatever action you ordinarily would by default, then you have already taken the most crucial step toward breaking a default stimulus-response pattern. You have already created a space between the stimulus and your original default response.

In the Experimentation phase of naturalist study, you’ll use actions that are larger than “observation” to stretch that space. You’ll experiment with saying this, thinking that, or moving your body in such and such a way, until the link be

...
INTERVIEW: Round 2 - StakeOut.AI w/ Dr. Peter Park
by jacobhaimes — 2024-03-18T21:21:41.293Z
Link: https://into-ai-safety.github.io/episode/interview/episode-16/

Hi again, I'm back with the second episode covering my interview with Dr. Peter Park, an AI Existential Safety Postdoctoral Fellow working with Dr. Max Tegmark at MIT. Dr. Park was a cofounder of StakeOut.AI, a non-profit focused on making AI go well for humans, along with Harry Luk and one other individual, whose name has been removed due to requirements of her current position.

Unfortunately, due to funding pressures, the organization recently had to dissolve, but the founders continue to contribute positively towards society in their respective roles.

Specifically, this episode focuses on what Dr. Park calls the battlegrounds in the effort for making AI go well. In addition we talk about Rich Sutton, the OpenAI drama from November 2023, and I unfortunately make the podcast's first mention of Elon Musk or related products.

This interview was made possible through the 2024 Winter AI Safety Camp.

Note that the interview will be broken up into 3 episodes, and this is only the second; the third will be released next week.

As I have mentioned previously, any feedback, advice, comments, etc. is greatly appreciated.

Spotify
Apple Podcasts
Amazon Music
YouTube Podcasts
Neuroscience and Alignment
by Garrett Baker — 2024-03-18T21:09:52.004Z

I've been in many conversations where I've mentioned the idea of using neuroscience for outer alignment, and the people who I'm talking to usually seem pretty confused about why I would want to do that. Well, I'm confused about why one wouldn't want to do that, and in this post I explain why.

As far as I see it, there are three main strategies people have for trying to deal with AI alignment in worlds where AI alignment is hard.

Value alignment
Corrigibility
Control/scalable alignment

In my opinion, these are all great efforts, but I personally like the idea of working on value alignment directly. Why? First some negatives of the others:

Corrigibility requires moderate to extreme levels of philosophical deconfusion, an effort worth doing for some, but a very small set not including myself. Another negative of this approach is that by-default the robust solutions to the problems won't be easily implementable in deep learning.
Control/scalable alignment requires understanding the capabilities & behaviors of inherently unpredictable systems. Sounds hard![1]

Why is value alignment different from these? Because we have working example of a value-aligned system right in front of us: The human brain. This permits an entirely scientific approach, requiring minimal philosophical deconfusion. And in contrast to corrigibility solutions, biological and artificial neural-networks are based upon the same fundamental principles, so there's a much greater chance that insights from the one easily work in the other.

In the most perfect world, we would never touch corrigibility or control with a 10-foot stick, and instead once we realized the vast benefits and potential pitfalls of AGI, we'd get to work on decoding human values (or more likely the generators of human values) directly from the source.

Indeed, in worlds where control or scalable alignment go well, I expect the research area our AI minions will most prioritize is neuroscience. The AIs will likely be too dumb or have the wrong inductive biases to hold an entire human morality in their head, and even if they do, we don't know whether they do, so we need them to demonstrate that their values are the same as our values in a way which can't be gamed by exploiting our many biases or philosophical inadequacies. The best way to do that is through empiricism, directly studying & making predictions about the thing you'...
GPT, the magical collaboration zone, Lex Fridman and Sam Altman
by Bill Benzon — 2024-03-18T20:04:17.551Z

Cross-posted from New Savanna.

I was making one more run around the web before I buckled down and got back to a major writing task, when I came across the brand-spanking-new conversation between Lex Fridman and Sam Altman. Lex is Lex, and an interesting guy, and Sam is, well, he's interesting to me, but – there was a hint of megalomania at the end of that NYTimes story from Mar. 31, 2023, that rubbed me the wrong way, and all the AI hype – he IS the CEO of OpenAI. So it seemed to me that I just had to listen in, not necessarily to the whole thing – and I could legit play solitaire while listening – and so I did, skipping over stuff.

But then the conversation hit an interesting patch. So – and I'm not going to try to re-create the context – they're talking about GPT-4 at roughly 46:03:

Altman: what are the best things it can do

Fridman: what are the best things it can do and the the limits of those best things that allow you to say it sucks therefore gives you an inspiration and hope for the future

Altman: you know one thing I've been using it for more recently is sort of a like a brainstorming partner
and for that there's a glimmer of something amazing in there
I don't think it gets you know
when people talk about it
it what it does they're like
ah it helps me code more productively
it helps me write more faster and better
it helps me you know translate from this language to another
all these like amazing things
but there's something about the like kind of creative brainstorming partner
I need to come up with a name for this thing
I need to like think about this problem in a different way
I'm not sure what to do here
uh that I think like gives a glimpse of something I hope to see more of
um one of the other things that you can see like a very small glimpse of is
when it can help on longer Horizon tasks
you know break down some multiple steps
maybe like execute some of those steps
search the internet
write code whatever put that together uh
when that works which is not very often
it's like very magical

At about 52:54:

Fridman: I use it as a reading partner for reading books
it helps me think
help me think through ideas especially when the books are classic
so it's really well written about and it actually is is I
I find it often to be significantly

...
Measuring Coherence of Policies in Toy Environments
by dx26 — 2024-03-18T17:59:08.118Z

This post was produced as part of the Astra Fellowship under the Winter 2024 Cohort, mentored by Richard Ngo. Thanks to Martín Soto, Jeremy Gillen, Daniel Kokotajlo, and Lukas Berglund for feedback.

Summary

Discussions around the likelihood and threat models of AI existential risk (x-risk) often hinge on some informal concept of a “coherent”, goal-directed AGI in the future maximizing some utility function unaligned with human values. Whether and how coherence may develop in future AI systems, especially in the era of LLMs, has been a subject of considerable debate. In this post, we provide a preliminary mathematical definition of the coherence of a policy as how likely it is to have been sampled via uniform reward sampling (URS), or uniformly sampling a reward function and then sampling from the set of policies optimal for that reward function, versus uniform policy sampling (UPS). We provide extensions of the model for sub-optimality and for “simple” reward functions via uniform sparsity sampling (USS). We then build a classifier for the coherence of policies in small deterministic MDPs, and find that properties of the MDP and policy, like the number of self-loops that the policy takes, are predictive of coherence when used as features for the classifier. Moreover, coherent policies tend to preserve optionality, navigate toward high-reward areas of the MDP, and have other “agentic” properties. We hope that our metric can be iterated upon to achieve better definitions of coherence and a better understanding of what properties dangerous AIs will have.

Introduction

Much of the current discussion about AI x-risk centers around “agentic”, goal-directed AIs having misaligned goals. For instance, one of the most dangerous possibilities being discussed is of mesa-optimizers developing within superhuman models, leading to scheming behavior and deceptive alignment. A significant proportion of current alignment work focuses on detecting, analyzing (e.g. via analogous case studies of model organisms), and possibly preventing deception. Some researchers in the field believe that intelligence and capabilities are inherently tied with “coherence”, and thus any sufficiently capable AI will approximately be a coherent utility function maximizer.

In their paper “Risks From Learned Optimization” formally introducing mesa-optimization and dec...
AtP*: An efficient and scalable method for localizing LLM behaviour to components
by Neel Nanda — 2024-03-18T17:28:37.513Z
Link: https://arxiv.org/abs/2403.00745

Authors: János Kramár, Tom Lieberum, Rohin Shah, Neel Nanda

A new paper from the Google DeepMind mechanistic interpretability team, from core contributors János Kramár and Tom Lieberum

Tweet thread summary, paper

Abstract:

Activation Patching is a method of directly computing causal attributions of behavior to model components. However, applying it exhaustively requires a sweep with cost scaling linearly in the number of model components, which can be prohibitively expensive for SoTA Large Language Models (LLMs). We investigate Attribution Patching (AtP), a fast gradient-based approximation to Activation Patching and find two classes of failure modes of AtP which lead to significant false negatives. We propose a variant of AtP called AtP*, with two changes to address these failure modes while retaining scalability. We present the first systematic study of AtP and alternative methods for faster activation patching and show that AtP significantly outperforms all other investigated methods, with AtP* providing further significant improvement. Finally, we provide a method to bound the probability of remaining false negatives of AtP* estimates.