trentmkelly/LessWrong-43k huggingface dataset:
AI alignment as a translation problem
by Roman Leventov — 2024-02-05T14:14:15.060Z

Yet another way to think about the alignment problem

Consider two learning agents (humans or AIs) that have made different measurements of some system and have different interests (concerns) regarding how the system should be evolved or managed (controlled). Let’s set aside the discussion of bargaining power and the wider game both agents play and focus on how the agents can agree about a specific way of controlling the system, assuming the agents have to respect each other’s interests.

For such an agreement to happen, both agents must see the plan for controlling the system of interest as beneficial from the perspective of their models and decision theories[1]. This means that they can find a shared model that they both see as a generalisation of their respective models, at least in everything that pertains to describing the system of interest, their respective interests regarding the system, and control of the system.

Gödel’s theorems prevent agents from completely “knowing” their own generalisation method[2], so the only way for agents to arrive at such a shared understanding is to present each other some symbols (i.e., classical information) about the system of interest, learn from it and incorporate this knowledge into their model (i.e., generalise from the previous version of their model)[3], and check if they can come up with decisions (plans) regarding the system of interest that they both estimate as net positive.

Note that without the loss of generality, the above process could be interleaved with actual control according to some decisions and plans deemed “good enough” or sufficiently low-risk after some initial alignment and collective deliberation episode. After the agents collect new observations, they could have another alignment episode, then more action, and so on.

Scalable oversight, weak-to-strong generalisation, and interpretability

To me, the above description of the alignment problem demonstrates that “

scalable oversight and weak-to-strong generalisation

” are largely misconceptions of this problem, except insofar as oversight is the implementation of the power balance between humans and AIs[4] or the prevention of deception (I factored out both these aspects from the above picture).

Yes, there will always be something that humans perceive about their systems of interest (including themselves) that AIs won’t perceive, but this looks to be ...
Safe Stasis Fallacy
by Davidmanheim — 2024-02-05T10:54:44.061Z

A critical failure mode in many discussions of technological risk is the assumption that maintaining status quo for technology would lead to maintaining the status quo for society. Lewis Anslow suggests that this "propensity to treat technological stagnation as safer than technological acceleration" is a fallacy. I agree that it is an important failure of reasoning among some EAs, and want to call it out clearly.

One obvious example of this, flagged by Anslow, is the anti-nuclear movement. It was not an explicitly pro-coal position, but because there was continued pressure for economic growth, the result of delaying nuclear technology wasn't less power usage, it was more coal. To the extent that they succeeded narrowly, they damaged the environment.

The risk from artificial intelligence systems today is very arguably significant, but stopping future progress won't reduce the impact of extant innovations. Stopping where we are today would still lead to continued problems with mass disinformation assisted by generative AI, and we'll see continued progress towards automation of huge parts of modern work even without more capable systems as the systems which exist are deployed in new ways. It also seems vanishingly unlikely that the pressures on middle class jobs, artists, and writers will decrease even if we rolled back the last 5 years of progress in AI - but we wouldn't have the accompanying productivity gains which could be used to pay for UBI or other programs.

All of that said, the perils of stasis aren't necessarily greater than those of progress. This is not a question of safety versus risk, it is a risk-risk tradeoff. And the balance of the tradeoff is debatable - it is entirely possible to agree that there are significant risks to technological stasis, and even that AI would solve problems, and still debate what strategy to promote - whether it is safer to accelerate through the time of perils, promote differential technological development, or to shut it all down.
How has internalising a post-AGI world affected your current choices?
by yanni kyriacos — 2024-02-05T05:43:14.082Z

You're probably in one of the three groups below.

Group 1: You don't think AGI is coming anytime soon, so it is business as usual for you.

Group 2: You do think it is coming soon, but you haven't made any life changes beyond career choice and donations. I'm not that interested in hearing about career choices and donations (examples below for what I'd prefer to hear about).

Group 3: You do think it is coming soon, and you have made noteworthy life changes.

I would mostly like to hear from Group 3.

A quick brain dump of the kinds of things I am pointing at:

You aren't saving for retirement. Heck, you aren't saving for 2030.
You've decided not to buy property and rent instead. Or you have, with mortgage conditions that you'd normally wouldn't have.
You've changed stock investment strategies.
You're going to have kids / you're not going to have kids.
You smoke cigarettes now, or don't wear sunscreen. You don't worry about cardiovascular health, or the long-term impact of drugs (I'm kind of tempted to take supplements for weight lifting I normally wouldn't TBH).
You're spending much more time with family and friends, because they might be gone in X years.
You've become more spiritual on purpose (maybe as a coping mechanism).
You don't invest in learning about X topic or building skill Y, because they take ages to pay off.

Thanks!
Yanni
A thought experiment for comparing "biological" vs "digital" intelligence increase/explosion
by Super AGI — 2024-02-05T04:57:18.211Z

Intro

The idea of an "intelligence explosion", where artificial intelligence (AI) rapidly surpasses Human intelligence, might be mind-bending for many. Especially for those not deeply engaged with AI. But what if we could explore this concept through a different lens, one that doesn't involve machines or digital systems at all? This approach could allow us to examine the intelligence explosion in a more approachable and relatable way, potentially making it accessible to a wider audience.

This thought experiment delves into a hypothetical "intelligence explosion", but uses a biological species instead of a "digital intelligence", offering a fresh perspective on intelligence advancement and its potential impacts. By examining how Humans could interact with and react to this species' intelligence growth, we could gain valuable insights into the broader questions surrounding an intelligence explosion, even without directly addressing AI.

The Scenario

Suppose Humans shared the world with an equally intelligent species, let's call them "Fluvins".
Humans are their normal selves as we see them in the present day. Fluvins are considered generally as intelligent, but they are much smaller than Humans, say 1cm tall.
Humans and Fluvins are able to get along and share the resources of the planet. They coexist stably.
The Humans find that the Fluvins can reproduce. But, they don't seem to age, at least on Human time scales. They will breed up to a certain population density and then stop at that level.
Then, one day, the Humans discover that the Fluvins have found a way to increase their brain's intelligence and capabilities (at least) exponentially.
To simplify the discussion, let's say the Humans also discover that they could slow or stop the Fluvins from increasing their intelligence by including some additives into their water supply.

Questions for Consideration

Would/should Humans take actions to try to stop the Fluvins from increasing their intelligence and capabilities?
Could the discovery benefit both groups, if cooperatively managed?
Does replacing current "digital intelligence" with a similar "biological intelligence" in this thought experiment provide useful parallels, or just distracting differences?

Alternates/Scenarios options

What if the Fluvins were able to speak English, post messages and chat online at a Huma

...
Noticing Panic
by Cole Wyeth — 2024-02-05T03:45:51.794Z

In Competent Elites @Eliezer Yudkowsky discusses "executive nature," which he describes as the ability to "function without recourse," to make decisions without some higher decision maker to fall back on.

I do not have executive nature.

I like to be well prepared. I like to read every word of every chapter of a textbook before a problem set, take a look, give the last chapter another skim, and then start working.

Being thrust into a new job and forced to explore a confusing repository and complete an open ended project is... stressful. The idea of founding a startup terrifies me (how could I settle on one business idea to throw years of my life at? And just thinking about the paperwork and legal codes that I would have to learn fills me with dread).

There's a particular feeling I get just before freezing up in situations like this. It involves a sinking suspicion that failure is inevitable, a loss of self confidence, and a sort of physical awkwardness or even claustrophobia, like trying to claw my way up the sheer wall of a narrow shaft. In a word, it is panic.

I believe that this is, for me, the feeling of insufficient executive nature.

It may sound a little discouraging, but the very consistency of this feeling is, perhaps, a key to overcoming my limitations.

The rationalist technique that I've found most useful is probably noticing confusion. It is useful because it is a trigger to re-evaluate my beliefs. It is a clue that an active effort must be made to pursue epistemic rigor before instinctively brushing over important evidence. In a way, "noticing confusion" is useful because it links my abstract understanding of Bayes with the physical and emotional experience of being a human. It is not, by itself, a recipe for correct epistemic conduct, but it provides a precious opportunity to take hold of the mind and steer it down a wiser path.

Perhaps noticing panic is for planning and acting what noticing confusion is for belief.

So how should one act when noticing panic?

I do not know yet. But I do have a guess.

I think that I panic when there are too many levels of planning between me and an objective.

For instance, a simple task like performing a calculation or following a recipe has zero levels of planning. Solving a more difficult problem, for instance a routine proof, might have one level of planning: I do not know how to wr...
Personal predictions
by Daniele De Nuntiis — 2024-02-04T03:59:28.537Z

Crossposted from Substack.

Epistemic status: my experience making predictions in the past year.

Roughly a year ago, as I was reading the Sequences, I decided it was time for me to start making and tracking predictions to test how calibrated I was. The only thing I figured I needed was some tool to automate the process as much as possible, and the only option I remember considering was Prediction Book but I didn’t like the user experience that much (now Prediction Book is shutting down, Fatebook seems a good alternative), so I decided to make my own tracker in Notion, and started making predictions.

Personal predictions

One of the first questions I found myself asking was: “What about personal stuff?”. Can I predict "How long is it gonna take me to write this post?". It felt somewhat different from the examples I saw online where one has close to zero influence on the outcome (Which song will win Best Original Song at the 96th Academy Awards?). On one side it seemed the kind of thing that could bias my calibration, on the other, planning was something I wanted to get better at.

I didn’t want to be wrong, so if I made a positive prediction about an outcome, e.g. "I’ll finish this book by the end of the week”, I would exert more effort toward that goal than I would have otherwise. It’s a kind of placebo effect or self fulfilling prophecy (for this reason I’m also a bit weary of predicting negative outcomes).

Trying to choose a confidence interval for a personal prediction generated a kind of loop in my mind “Predicting I'll do X will make me more incentivized to do it, so I should be more confident, more confidence means more incentive, and so on” It’s obvious that I can’t just predict myself into doing things with certainty and in fact it converges pretty quickly, but it still felt slightly odd. And also what if I deceived myself into making over or underconfident claims just to fix my score? I decided to keep making personal predictions and to tag them in my database so I could remove them later just in case (I tried removing them now, all my predictions N:165 Brier: 0.196, personal filtered out N:52 Brier: 0.225, I guess I should have predicted more nonpersonal stuff but it seems to be ok).

Also by having to generate an estimate for something it was harder to bullshit myself. It made me think more thoroughly about how likely I actually was of doing the thing in que...
A sketch of acausal trade in practice
by Richard_Ngo — 2024-02-04T00:32:54.622Z

One implication of functional decision theory is that it might be possible to coordinate across causally-disconnected regions of the universe. This possibility has been variously known as

acausal trade

,

multiverse-wide cooperation via superrationality

(MSR), or

evidential cooperation across large worlds

(ECL). I’ll use the MSR abbreviation (since I don’t want to assume that the cooperation needs to be motivated by evidential decision theory).

I broadly believe that the assumptions behind MSR make sense. But I think the standard presentations of it don’t give very good intuitions for how it might actually work in practice. In this post I present an alternative framing of MSR centered around the concept of “psychohistory”.

Background

Assume that humanity colonizes the universe at a large scale (e.g. controlling many galaxies), and also discovers that we live in some kind of much-larger multiverse (e.g. any of Tegmark's four types of multiverse).

Assume that humanity becomes a highly coordinated entity capable of making large-scale commitments, and also converges towards believing that some version of superrationality or functional decision theory is normatively correct.

Reasoning about psychohistory

In order to make accurate predictions about other causally-disconnected civilizations, we would need to develop a scientific understanding of the dynamics of civilization development, and in particular the values and governance structures that other civilizations are likely to end up with. Call this science “psychohistory”.

Why should we think that psychohistory is even possible? If we build intergalactic civilizations of astronomical complexity, then they might become more difficult to understand in proportion to our increased collective intelligence. But in many domains that we study, there are a few core principles which provide a great deal of insight (e.g. atoms, DNA, relativity). Surprisingly, this also occurs in domains which involve complex interactions between many different agents (e.g. evolution, economics). Economics is still a long way from being able to reliably predict novel macro-level phenomena, but we can at least imagine how a deep understanding of economics might allow us to do so. (For example, Malthus was very prescient in predicting “general gluts”, aka depressions.) The relationship between microeconomics and macroeconomics provides ...
Brute Force Manufactured Consensus is Hiding the Crime of the Century
by Roko — 2024-02-03T20:36:59.806Z

People often parse information through an epistemic consensus filter. They do not ask "is this true", they ask "will others be OK with me thinking this is true". This makes them very malleable to brute force manufactured consensus; if every screen they look at says the same thing they will adopt that position because their brain interprets it as everyone in the tribe believing it.

- Anon, 4Chan, slightly edited

Ordinary people who haven't spent years of their lives thinking about rationality and epistemology don't form beliefs by impartially tallying up evidence like a Bayesian reasoner. Whilst there is a lot of variation, my impression is that the majority of humans we share this Earth with use a completely different algorithm for vetting potential beliefs: they just believe some average of what everyone and everything around them believes, especially what they see on screens, newspapers and "respectable", "mainstream" websites.

This is a great algorithm from the point of view of the individual human. If the mainstream is wrong, well, "nobody got fired for buying IBM", as they say - you won't be personally singled out for being wrong if everyone else is also wrong. If the mainstream is right, you're also right. Win-win.

The problem with the "copy other people's beliefs" algorithm is that it is vulnerable to false information cascades. And when a small but powerful adversarial group controls the seed point for many people's beliefs (such as being able to control the scientific process to output chosen falsehoods), you can end up with an entire society believing an absurd falsehood that happens to be very convenient for that small, powerful adversarial subgroup.

DEFUSING your concerns

This is not a theoretical concern; I believe that brute-force manufactured consensus by the perpetrators is the cause of a lack of action to properly investigate and prosecute what I believe is the crime of the century: a group of scientists who I believe committed the equivalent of a modern holocaust (either deliberately or accidentally) are going to get away with it. For those who are not aware, the death toll of Covid-19 is estimated at between 19 million and 35 million.

Covid-19 likely came from a known lab (Wuhan Institute of Virology), was likely created by a known group of people (Peter Daszak & friends) acting against best practices and willing to lie about their safety sta...
My thoughts on the Beff Jezos - Connor Leahy debate
by kwiat.dev — 2024-02-03T19:47:08.326Z

Link:

Personal note: I'm somewhat in between safetyism and e/acc in terms of their general ideologies/philosophies. I don't really consider myself a part of either group. My view on AI x-risk is that AI can be potentially an existential threat, but we're nowhere near that point right now, so safety research is valuable, but not urgent. For this reason, in practical terms, I'm somewhat closer to e/acc, because I think there's a lot of value to be found in technological progress, so we should keep developing useful AI.

I'm hoping this debate will contain solid arguments as to why we shouldn't keep developing AI at full speed, ideally ones that I haven't heard before. I will write this post as a series of notes throughout the video.

One hour in

This is insufferable. Connor started with fairly direct questions, Beff bounces around them for no good reason, but eventually reaches a simple answer - yes, it's possible that some technologies should be banned. So far this seems to be the only concrete thing that was said?

At some point they start building their respective cases - what if you had a false vacuum device? Would we be fucked? Should we hide it? What should we do? And on Beff's side - what if there are dangerous aliens?

For the love of god, please talk about the actual topic.

About 50 minutes in, Connor goes on an offensive in a way that, to me is an extremely blatant slippery slope reasoning. The main point is that if you care about growth, you cannot care about anything else, because of course everyone's views are the extremist parodies of themselves. Embarrassing tbh. Ostensibly, Connor avoids making any concrete statements about his own values, because any such statements could be treated the same way. "You like puppies and friendship? Well I guess nobody will grow food anymore because they will be busy cuddling puppies".

He also points out, many many times, that "is" != "ought", which felt like virtue signalling? Throwing around shibboleths? Not quite sure. But not once was it a good argument as far as I can tell. Example exchange (my interpretation, the conversation was chaotic so hopefully I'm not misunderstanding)

B: Your values are not growth? How so?

C: Because I like puppies and happines and friendship [...]

B: Why do you like friendship? Because evolution hard-coded this in humans

C: You're mixing "is" and "ought"

&n...
Attitudes about Applied Rationality
by Camille Berger  — 2024-02-03T14:42:22.770Z

tl;dr: within the LW community, there are many clusters of strategies to achieve rationality: doing basic exercices, using jargon, reading, partaking workshops, privileging object-level activities, and several other opinions like putting an accent on feedback loops, difficult conversations or altered states of consciousness.
Epistemic status: This is a vague model to help me understand other rationalists and why some of them keep doing things I think are wrong, or suggest me to do things I think are wrong. This is not based on real data. I will update according to possible discussions in the comments. Please be critical.
[Update : The term "Theory" is replaced with the term "Attitude" in the following paragraphs]
Spending time in the rationalist community made me realize that there were several endeavors at reaching rationality that seemed to exist, some of which conflicted with others. This made me quite frustrated as I thought that my interpretation was the only one.

The following list is an attempt at distinguishing the several approaches I've noticed, which I will call Attitudes (in lack of a better name). Of course, any rationalist will probably have elements of all attitudes at the same time. See each attitude as the claim that a particular set of elements prevails above others. Referring to one attitude usually goes on par with being fairly suspicious of others.

Finally, remember that these categories are an attempt to distinguish what people are doing, not a guide about what side you should pick (if the sides exist at all). I suspect that most people end up settling on one attitude for practical reasons, more than because they have deeply thought about it at all.

Basics Attitude

Proponents of the Basics Attitude put a high emphasis on activities such as calibration, forecasting, lifehacks, and other fairly standard practices of epistemic and instrumental rationality. They don't see any real value in reading extensively LessWrong or going to workshops. They first and foremost believe in real-life, readily available practice.

For them, spending too much time in the rationalist community, as opposed to doing simple exercises, is the main failure mode to avoid.

Speaking Attitude

Proponents of the Speaking Attitude, although often relying on basics, usually put a high emphasis on using concepts met on LessWrong in daily parlance, although...