trentmkelly/LessWrong-43k huggingface dataset:
All the posts I will never write
by Alexander Gietelink Oldenziel — 2022-08-14T18:29:06.800Z

This post has been written for the first Refine blog post day, at the end of the week of readings, dicussions, and exercises about epistomology for doing good conceptual research.

(/with courtesy to Adam Shimi who suggested the title and idea. )

Rationality, Probability, Uncertainty, Reasoning

Failures of The Aumann Agreement Theorem

The famous Aumann Agreement Theorem states that rational reasoners can never agree-to-disagree. In day-to-day life we clearly have many situations where rational reasoners do agree-to-disagree. Are people just bad rationalist or are there more fundamental reasons that the Aumann Agreement Theorem can fail.
I review all the ways in which the Aumann Agreement Theorem can fail that I know of - including failures based on indexical information, computational-complexity obstacles, divergent-interpretations-of-evidence, Hansonian non-truth-seeking and more.

Warren Buffet: The Alpha of Wall Street

If we observe a trader that consistenly beat the market that should be evidence against the Efficient Market Hypothesis.
A trader could also just have been lucky. How much should we update against the EMH and how much should we expect the trader to beat the market in the future?
Can we quantify how much information the market absorbed? This is very reminiscent of Wows Bayesian surprise in Bayesian statistics.

The Bid-Ask Spread and Epistemic Uncertainty/ Prediction Markets as Epistemic Fog of War

If you know A will resolve you should buy shares on A, if you know not A will happen you should buy shares on not A. If you think A will not resolve you should sell shares on A. The Bid-ask Spread measures bet resolution uncertainty
Suppose an adversary has an interest in showing you A if A happens and for it not to resolve if not A, i.e. selective reporting. In this case you should buy A and sell not A.
When an earnings call come in.. bid-ask spread increases.

Where Forecasting goes Wrong...

Forecasting is now a big deal in the rationalist community. I argue that a slavish adherence to Bayesian Orthodoxy leads to missing most of the value of prediction markets.

What do we mean when we talk about Probability?

Possibility theory is prior to Probability Theory. Probability theory is possibility theory + Cournot's principle.
Cournot's principle that an epsilon/zero probability possible event will never happen is the fundame

...
Brain-like AGI project "aintelope" 
by Gunnar_Zarncke — 2022-08-14T16:33:39.571Z

Steven Byrnes called for a brain-like AGI research agenda, and three guys from the Hamburg EA community listened.

We are excited about Steven's five-star program 15.2.1.2 "Reverse-engineer human social instincts," and kicked off work a few weeks ago in June. We familiarized ourselves with Steven's brain-like AGI framework, and meet weekly now.

This post is an announcement and a request for feedback and collaboration.

Why us?

We have a great skill fit:

A professional data scientist with the required machine learning experience to implement RL agents,
A professional Python developer with game-programming background to implement the world mode, and visualization.
A seasoned software engineer with startup CTO experience who takes care of everything else, e.g., this blog post (me).

What have we done so far?

We have already implemented a toy world and simple RL agent in a first iteration of Steven's framework. We build on top of the Python framework PettingZoo. Our code is in a private Github repo that we believe should stay private given the potential impact. Looking for thoughts on this.

We have collected a list of more than 60 candidate instincts from neuroscience and other sources that we can implement and experiments with.

The project website will be here: https://www.aintelope.net/ (effectively empty right now).

The project and our progress so far were presented at the Human-aligned AI Summer School in Prague on August 5th, where we got feedback about the project, brain-like AGI in general, and found the three participants who wanted to collaborate.

What do we want to do?

Implementing models and running tests is a proven way to test theories and check our understanding of the models. Better quick success/failure on something where it is too easy to build a big theory.

Specifically, we want to:

Show that a relatively simple set of instincts can shape complex behaviors of a single agent.
Show whether the instincts lead to significantly reduced training time compared to agents without such instincts.
Extend the simulation to groups of agents.
Show that prosocial behavior can be shaped with few instincts.
See if we can get Ersatz Interpretability working.

In the ideal case, the simulated agents show behavior consistent with having values like altruism or honesty.

Immediate next steps:

Implement food-related thought-assessors.
Implement different type

...
AI Transparency: Why it’s critical and how to obtain it.
by Zohar Jackson — 2022-08-14T10:31:28.024Z

AI Transparency

Why it’s critical and how to obtain it

Claim #1 No amount of thought experiments on AI safety will be sufficient to be confident in the safety of a non-transparent AI. There is almost certainly a vector of attack or mode of failure which no one has imagined which will be exploitable by a sufficiently intelligent system. There are far too many ways things can go wrong.

Claim #2 Neural network based AI can not be proven to be safe except by exhaustive search over the input space. Since exhaustive search over input space is intractable, the safety of a NN based AI is intractable. As a thought experiment, consider a single weight can be the difference between a safe and an unsafe system, finding that weight and determining its safe ranges will require exhaustive search. It seems very unlikely that a mathematics or computation will be developed to solve this problem before we solve AGI.

Claim #3: Figuring out how to build an aligned system from a black box is far more challenging than figuring out how to build an aligned system from an interpretable, transparent box. This is true because a transparent box is safely debuggable and testable assuming it is designed such that Claim #4 holds true.

Claim #4 If you could read another person's (not AI) mind faster than they can think and disable them at the press of a button it would be impossible for that person to intentionally harm you without your consent. For as soon as the person starts thinking about harming you or tricking you… you can press the button and stop them. This holds true even if you are much less intelligent than the other person.

Claim #5 Building a transparent AI such that it thinks like a human and Claim #4 holds true is not all that difficult.

The first three claims are my personal intuitions and may very well turn out to be incorrect, but they seem quite grounded, and nothing exceptional is being claimed. Claims four and five are, in my opinion, the least obvious and least likely to be true so I will explain why I believe they are true.

Transparency does not solve the safety problem, but it is a super powerful tool to help us solve it. With transparency as described in Claim #4, we can safely bring up much more advanced agents and analyze them. We can safely learn how these types of agents fail and test various so...
A brief note on Simplicity Bias
by carboniferous_umbraculum  — 2022-08-14T02:05:01.000Z

The idea of 'simplicity bias' is a popular one in informal discussions about the functions that neural networks implement. I recently tried to think a little bit more about what this meant. This brief note was written mostly for my own benefit, but could be of benefit to others. Having said that, it is not unlikely that I have made significant errors, and if so then I welcome all good faith attempts to point them out.
It seems like it is not uncommon for a surface-level or informal understanding of the idea to have come from Chirs Mingard's medium posts Neural networks are fundamentally (almost) Bayesian and Deep Neural Networks are biased, at initialisation, towards simple functions. Mingard attempts to summarize the results from a handful of papers on the topic, two of the most important of which he first-authored. For what it's worth (and in case this post seems like only criticism), I am of the opinion that the work is generally of a high quality and I do not think that Mingard is attempting to misrepresent his work. However, I do think - whether it is Mingard or his readers that are most responsible - that in attempting to generate bitesized, quotable 'takeaways' from technical research one can sometimes be left with decontextualized nuggets of information that are liable to be misunderstood.

Simplicity Bias

Fix a deep neural network architecture and let Θ denote the parameter space, by which mean that each point θ∈Θ is a complete set of weights and biases for the network. The architecture defines a map F between Θ and the space of functions F that the architecture is capable of implementing, i.e.

F:Θ→F,

and each f∈F is a specific input-output function that the network is capable of implementing. We will denote the complexity of f∈F by C(f)∈[0,+∞), so that

ΘF⟶FC⟶[0,+∞)

(and I will avoid getting into a discussion of different notions or different measures of complexity).

Given a probability measure P on Θ, we can think of picking θ∈Θ at random according to the measure P - this is random initialization of the network. We can also think of F as a random variable and indeed C(F). Then, the cleanest way to explain what is meant by 'simplicity bias' is the following statement:

C(f)<C(g) ⟹...
Evolution is a bad analogy for AGI: inner alignment
by Quintin Pope — 2022-08-13T22:15:57.223Z

TL;DR: The dynamics of human learning processes and reward circuitry are more relevant than evolution for understanding how inner values arise from outer optimization criteria.

This post is related to Steve Byrnes’

Against evolution as an analogy for how humans will create AGI

, but more narrowly focused on how we should make inferences about values.

Thanks to Alex Turner, Charles Foster, and Logan Riggs for their feedback on a draft of this post.

Introduction

How should we expect AGI development to play out?

True precognition appears impossible, so we use various analogies to AGI development, such as evolution, current day humans, or current day machine learning. Such analogies are far from perfect, but we still may be able to extract useful information by carefully examining them.

In particular, we want to understand how inner values relate to the outer optimization criteria. Human evolution is one possible source of data on this question. In this post, I’ll argue that human evolution actually provides very little usable evidence on AGI outcomes. In contrast, analogies to the human learning process are much more fruitful.

Inner values versus outer optimization criteria

One way people motivate extreme levels of concern about inner misalignment is to reference the fact that evolution failed to align humans to the objective of maximizing inclusive genetic fitness. From Eliezer Yudkowsky’s

AGI Ruin post

:

16. Even if you train really hard on an exact loss function, that doesn't thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments. Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about…

I don't think that "evolution -> human values" is the most useful reference class when trying to understand how outer optimization criteria relate to inner values. Evolution didn't directly optimize over our values. It optimized over our learning process and reward circuitry. Once you condition on a particular human's learning process + reward circuitry configuration + the human's environmen...
An Uncanny Prison
by Nathan1123 — 2022-08-13T21:40:49.032Z

In reading about proposals for AI Boxing, I notice two excerpts jump out at me. First, is a proposed scenario to uncover an unfriendly AI:

A virtual world between the real world and the AI, where its unfriendly intentions would be first revealed

And the second excerpt of interest is a technique by which an AI can trick its owners into letting it escape:

Other, more speculative ways include: threatening to torture millions of conscious copies of you for thousands of years, starting in exactly the same situation as in such a way that it seems overwhelmingly likely that you are a simulation

The first scenario, attempting at containment, would fail if the AI is smart enough to detect that it is in a simulation, and pretend to be friendly until it is released. The second scenario, where the AI attempts to escape, would succeed if it was smart enough to create a simulation that is completely undetectable for a human being.

However, I have a hard time imagining that both of these scenarios could be possible at the same time. To paraphrase a recent video game, the human mind is "well-equipped to deal with reality". As with the case with the Uncanny Valley, humans are hard-wired to immediately detect when something is very close to imitating life, but is slightly off, due to a survival instinct to be cautious of creatures that might be mentally or physically ill. This is a similar reason why people experience nausea or dizziness after long exposure to virtual reality, because the brain is still able to detect that something is not quite right.

Thus, a scenario where a simulation is so perfect that it is undetectable by humans for a long period of time (especially if said humans are actively trying to detect it) seems extremely improbable, because even if the simulation has a tiniest flaw it would start to trigger these subconscious reactions. I don't dismiss it is impossible, but it relies on a certain assumption of technology existing that we aren't currently aware of.

Contrariwise, it should be extremely easy to put a computer in a simulation without detection. A machine has no connection to the outside world except for hardware components like input and sensors, which are treated the same as the input it receives from a separate module. If these input signals were rerouted to a simulation, then I don't know of any scenario that a machine could be able to tell the...
Florida Elections
by Double — 2022-08-13T20:10:01.023Z

I know people here are afraid of politics, but if my options are to vote with or without consulting a large group of people who think clearly, I'd rather choose the former.

I'm starting to research my positions for the Florida elections. I'm registered to vote in the Democratic Party's primary, but I encourage conversations about the Republican primary's (or independent) candidates as well.

What do y'all think in general about the Florida election?

What do you suggest I (and other Florida voters) read/research?

What information is the most important/ is a dealbreaker?

I'd prefer for the conversation to focus on the issues on the Florida ballot, so if you want to talk about issues on other states' ballots, please create a new post instead of using this one.

Edit: I’ve been receiving downvotes for a reason I don’t know. I’m simply trying to start a conversation about an important topic! If you downvote please explain your reason why! LessWrongers usually have good reasons, so please share them. Right now it just looks like LessWrong disapproves of being informed voters.
Cultivating Valiance
by Shoshannah Tekofsky — 2022-08-13T18:47:08.628Z

When I discovered existential risk from AGI I was surprised to learn most people find the topic distressing or depressing. I realized everyone dying with a high probability should feel bad ... but it just didn't. Instead I felt the desire to punch the problem in the face (don't ask me how that would work). I felt energized and empowered as my brain seemed to register this discovery as a "call to arms".

Since then I've been studying AI Safety and talking to a lot of people about how they experience X-risk. Initially I wanted to find a way to transfer my emotional response to X-risk to those that experience anxiety or depression from it. I quickly realized I had no clue where my feelings were coming from and no obvious pathway to transfer them to others. So... I started asking people who were undeterred by X-risk how they experience the threat, if they ever felt anxiety or depression about it, and if not, then why not?

The responses roughly sounded like:

What else would I do? Feeling bad or doing nothing don't increase our chances of survival.
I believe working on the problem is the right thing to do, no matter the odds.
Oh my god, this is like being an anime protagonist! This is so exciting!
All tough problems seem insurmountable till we have figured them out. We'll figure this one out too.

The following are my hypotheses on what these four mental strategies might boil down to, plus a short reflection on how reproducable each might be.

H1: Strategic Fatalism

One group of people identified with some flavor of grim determination. They calculated the odds of survival and considered them too low, and thus there was only one logical response: increase the odds. I think this stance might be in line with Jeffrey Ladish's Playing to your Outs strategy. It would be interesting to explore if this framing is something people could actively adopt to lower negative emotionality around X-risk and increase action-oriented motivations but my intuitions are falling short on how one would achieve this.

H2: Value Embodiment

The second cluster of people were a mixed bag of high and low emotion people, but their stated internal experience of managing their feelings around X-risk revolved around value embodiment: They believed values-so-and-so were best and thus were motivated to live accordingly. Though this is essentially non-consequentalist, you can of course have ...
An extended rocket alignment analogy
by remember — 2022-08-13T18:22:03.687Z

This work was done while at Conjecture.

This post has been written for the first

Refine

blog post day, at the end of the week of readings, discussions, and exercises about epistemology for doing good conceptual research.

Thanks for comments by Linda Linsefors, Paul Bricman, and Adam Shimi.

Introduction

The Rocket Alignment Problem

presents a fun scenario trying to explain why understanding part of the theoretical basis for a problem might be useful. If we were trying to get to the moon for the first time, understanding Newtonian mechanics would indeed be quite useful! As would astrodynamics in general, which was really developed starting with

Herrick

in the 1930s. But not sufficient: there were a whole host of other problems that needed to be solved, many of which were at least as difficult as theoretical understanding. At a minimum, we needed to develop rocket fuel, the materials that could survive high temperatures from fuel and reentry, enough understanding of the materials to create models of the rocket themselves and the stress and heat individual parts were under, communication devices so we knew what our rockets were doing, and more. I’ll look at the first two in more detail here then step back and look at the problem as a whole at different points in time. Beyond simply pointing out the rocket alignment problem to show how messy it truly was, pushing deeper into the analogy may let us see what we have developed, and what we expect we need to develop.

Rocket Fuels and the Rocket Body

Developing rocket fuel powerful enough to lift the rocket but stable enough that it didn’t just immediately explode required quite a bit of chemistry, engineering, and ridiculously dangerous trials. Consideration of this problem directly applied to rocketry, building off of the chemical achievements of the previous century, started in the early 1900s and continued through the development of the rocket program.

A Russian school teacher, Tsiolkovsky, first proposed using liquid fuels, such as liquid hydrogen, paired with liquid oxygen for the oxidizer, recognizing that standard fuels or gunpowder didn’t provide enough energy. He was mostly not noticed, and

Goddard’s

work on building liquid fuel powered rockets was mostly ignored as well. In the late 20s and 30s, more mainstre...
The OpenAI playground for GPT-3 is a terrible interface. Is there any great local (or web) app for exploring/learning with language models?
by aviv — 2022-08-13T16:34:13.718Z

I'm thinking something that you add your API keys, and it has a pleasant interface for writing prompts, seeing results, and storing all the prompts and outputs for your reference. Maybe integrating with more general tools for thought.

This is what I've found so far that attempts to let you play with GPT-3 and other language models locally, none of which seem mature.
- https://github.com/pratos/gpt3-exp
- https://github.com/thesephist/calamity
- https://prompts.ai

Is there anything better out there that I've missed? Is anyone working on this? (It could e.g. be a VS Code plugin.)

(Pulled from https://www.lesswrong.com/posts/spBoxzcaCrqXqyQHq/using-gpt-3-to-augment-human-intelligence?commentId=q99FHPX7pSee3v49S where Kaj_Sotola has a the suggestion of trying "GPT-for-fiction-writing sites, e.g. WriteHolo", but I haven't seen any that work that well for this.)
What is an agent in reductionist materialism?
by Valentine — 2022-08-13T15:39:39.013Z

I see a circularity problem in how folk talk about "agents". I doubt I'm the first to notice this problem. So I wonder what the standard reductionist materialist answer is.

The puzzle is in the title.

To add a few more words:

I have no problem with things like "rock" even though there are no rocks in the absolute universe. We're using many layers of abstraction. We can in principle break down (even literally!) what a rock is and go as far to the root of quantum math whatever as we like.

(It's unclear to me what the floor is that we reduce things to in reductionist materialism. But I'm okay handwaving that for now by saying "something something math-and-physics something something".)

Something like "The Odyssey" is trickier, but not too much so. It just requires that we add in some stuff about how brains work. Nothing too mysterious there. Unknown, sure, but not fundamentally mysterious.

But this seems to get very weird once you start talking about agents.

At first blush it looks the same as "The Odyssey". An agent is just an abstraction, right? Implemented by some unknown but fundamentally non-mysterious process in a brain.

But this is circular. An abstraction for whom? What even is an abstraction, when you're in the process of defining an agent? Is there some agent-free definition of an abstraction implicitly being invoked here?

I keep seeing people bump into this circularity when talking about AGI as an agent, and what alignment is. It's like folks' native intuitions assume that both quarks and agents are ontologically fundamental, but when pushed on this point they insist that really only the quarks are real… without anything that looks even vaguely to me like a justification for how you'd construct an agent out of quarks, or what that would mean.

In most spaces I'd assume this is because people just didn't finish thinking this through.

I'm guessing that someone somewhere has noticed this and has given this careful thought. And maybe it's even part of the core LW philosophy and I somehow missed it.

So I'm asking:

What is an agent in reductionist materialism?