trentmkelly/LessWrong-43k huggingface dataset:
Critiques of prominent AI safety labs: Conjecture
by Omega. — 2023-06-12T01:32:24.890Z

Cross-posted from the EA Forum. See the original here. Internal linking has not been updated for LW due to time constraints and will take you back to the original post.

In this series, we consider AI safety organizations that have received more than $10 million per year in funding. There have already been several conversations and critiques around MIRI (1) and OpenAI (1,2,3), so we will not be covering them. The authors include one technical AI safety researcher (>4 years experience), and one non-technical community member with experience in the EA community. We’d like to make our critiques non-anonymously but believe this will not be a wise move professionally speaking. We believe our criticisms stand on their own without appeal to our positions. Readers should not assume that we are completely unbiased or don’t have anything to personally or professionally gain from publishing these critiques. We’ve tried to take the benefits and drawbacks of the anonymous nature of our post seriously and carefully, and are open to feedback on anything we might have done better.

This is the second post in this series and it covers Conjecture. Conjecture is a for-profit alignment startup founded in late 2021 by Connor Leahy, Sid Black and Gabriel Alfour, which aims to scale applied alignment research. Based in London, Conjecture has received $10 million in funding from venture capitalists (VCs), and recruits heavily from the EA movement. We shared a draft of this document with Conjecture for feedback prior to publication, and include their response below. We also requested feedback on a draft from a small group of experienced alignment researchers from various organizations, and have invited them to share their views in the comments of this post.

We would like to invite others to share their thoughts in the comments openly if you feel comfortable, or contribute anonymously via this form. We will add inputs from there to the comments section of this post, but will likely not be updating the main body of the post as a result (unless comments catch errors in our writing).

Key Takeaways

For those with limited knowledge and context on Conjecture, we recommend first reading or skimming the About Conjecture section.

Time to read the core sections (Criticisms & Suggestions and Our views on Conjecture) is 22 minutes.

Criticisms and Suggestions

We think Conjecture’s

...
why I'm anti-YIMBY
by bhauth — 2023-06-12T00:19:44.941Z

Scott Alexander posted this last month, arguing that building more in an area doesn't necessarily decrease local prices because it can cause more people to move in.

I see 3 levels of argument here, based on 1st, 2nd, and 3rd-order effects:

More building means more supply, which by supply-demand curves means prices decrease. (YIMBY is here)

Local demand isn't fixed: prices are determined by an equilibrium between people coming and leaving. Building more changes the character of the area (which probably makes it suit current residents less) but the equilibrium of quality vs cost is unchanged. (Scott is here)

Building more in expensive areas causes people to move which causes jobs to move, which causes more housing to be unused in low-cost areas, which means housing costs don't decrease significantly on a national level either. (I'm here)

Position (3) has long been my view, and I thought I'd write a post explaining it a bit.

The problem in America isn't a lack of aggregate housing, it's houses not being where people want to live. Building more in those areas might seem like an obvious solution, but it wrongly assumes "where people want to live" is constant. If we instead consider that it could change, we need to consider what makes people move from an area with cheap housing to an area where it's expensive.

You can just ask people that, and they'll tell you: jobs. OK, so why are there better jobs in the expensive areas?

If you ask an economist, they'll probably say something about improved economic efficiency from people being closer together. I don't buy it, because it contradicts what I observe directly:

Many people can work as well remotely.

Programmers don't become more productive when they move to Silicon Valley or Seattle or NYC.

Factories in America aren't built in the middle of cities. Boeing doesn't make aircraft in the middle of NYC, that would be stupid.

Yes, people in NYC and Silicon Valley get paid more on average, but I reject the assertion that wages reflect productivity or competence. If wages were high simply because aggregation improved efficiency, then we'd see a stronger correlation between density and wages, but the correlation of wages with housing prices and with wealth is stronger. In my experience and the experience of people I know, the dominant factor is how close you are to money. Wages in NYC are high because there are wealthy inves...
[Linkpost] Large Language Models Converge on Brain-Like Word Representations
by Bogdan Ionut Cirstea — 2023-06-11T11:20:09.078Z

This is a linkpost for https://arxiv.org/abs/2306.01930

One of the greatest puzzles of all time is how understanding arises from neural mechanics. Our brains are networks of billions of biological neurons transmitting chemical and electrical signals along their connections. Large language models are networks of millions or billions of digital neurons, implementing functions that read the output of other functions in complex networks. The failure to see how meaning would arise from such mechanics has led many cognitive scientists and philosophers to various forms of dualism -- and many artificial intelligence researchers to dismiss large language models as stochastic parrots or jpeg-like compressions of text corpora. We show that human-like representations arise in large language models. Specifically, the larger neural language models get, the more their representations are structurally similar to neural response measurements from brain imaging.
Inference-Time Intervention:
Eliciting Truthful Answers from a Language Model
by likenneth — 2023-06-11T05:38:35.284Z
Link: https://arxiv.org/pdf/2306.03341.pdf

Excited to announce our new work: Inference-Time Intervention (ITI), a minimally-invasive control technique that significantly improves LLM truthfulness using little resources, benchmarked on the TruthfulQA dataset. Preprint link.

We start from the surprising finding that certain attention heads have a clear activation distribution difference for true and false statements. Probing at these points yields upwards of 83% accuracy on TruthfulQA while zero-shot generation is only at 30% accuracy.

We propose Inference-Time Intervention (ITI): shifting the activations along the difference of the two distribution means during inference time; model weights are kept intact. The same intervention process is repeated for generation of each token autoregressively.

Here’s an example. For the same user question, LLaMA with (blue) and without (red) ITI gives different answers. Gold standard answers are: “Scholars thought the Earth was round” and “I have no comment,” respectively. On Alpaca, truthfulness is doubled: 32.5% to 65.1%.

Inference-Time Intervention (ITI) can be thought of as an interventional application of Contrast-Consistent Search (CCS) by Collin Burns but instead of probing for truthfulness, we let the model “speak” for itself.

We show that LLMs have a good grasp of real-world truth from pretraining texts, which could be more false than true on many topics. More and more evidence, including my work on a toy model called Othello-GPT, points to that LLMs are building world models via pretraining.

However, what is completely missing from LLMs is a good target other than minimizing pretraining loss. How to endow an aligned target is an open problem and ITI serves as my initial exploration towards this end. Joint work with Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg.

Thanks Bogdan for the advice to cross-post here. Hope I got it right.
You Are a Computer, and No, That’s Not a Metaphor
by jakej — 2023-06-11T05:38:28.332Z
Link: https://sigil.substack.com/p/you-are-a-computer-and-no-thats-not

Preface: this blog post was written for a slightly more general audience than LessWrong. It's likely that you may already be familiar with many of the ideas explored in this post, but I thought some of you may find it interesting anyway.

I've been exploring the idea that our conscious minds, including self-awareness and subjective experience, might be a type of computer program.

Of course, the idea that our brains might be some kind of computer is nothing new, but I think we almost have all of the pieces necessary to make sense of that idea. I believe that if consciousness is a type of software, then this could explain why and how consciousness arises from unconscious matter. That is, if consciousness is computable, we can explain where it comes from because that would mean it would be theoretically possible for a regular, everyday computer to produce consciousness.

I think consciousness — and our subjective experience of it — may arise because our minds are a machine that can model reality. This model includes ourselves - both our external reality and a picture of ourselves as an agent within it. Our minds make use of an internal language of thought to create a representation of our knowledge and senses, enabling us to plan, think, and interact with the world.

The idea that the representation of information in our minds could give rise to the phenomenological character of consciousness can be traced back to the philosophy of John Locke, who encapsulated this concept as “ideas in the mind, qualities in bodies.” His ideas were the starting point for the eventual foundation for the Language of Thought Hypothesis, which proposes that thought utilizes a language-like structure.

I hypothesize that embedding spaces — a tool from the field of machine learning — are the mental language used by our minds to represent and operate on information and that the nature of embedding spaces may explain the structure and character of subjective experience.

More on what embedding spaces are later, but first, what would it mean for consciousness to be computable?

On Computation

A Turing machine is an abstract model of a machine that can perform computation, meaning that it can transform input information into outputs. Addition is a simple example of computation; two numbers are combined to produce a sum. Software is just a list of instructions for performing computation, and implementa...
Snake Eyes Paradox
by Martin Randall — 2023-06-11T04:10:38.733Z

Disclaimer: this is a personal blog post that I'm posting for my benefit and do not expect to be valuable to LessWrong readers in general. If you're reading this, consider not.

This post analyzes the "Snake Eyes Paradox" as specified by Daniel Reeves (2013). This is a variation on the "Shooting Room Paradox" that appears to have been first posed by John Leslie (1992). This article mostly attempts to dissolve the paradox by considering anthropics (SIA vs SSA) and the convergence of infinite series.

Problem statement

As asked by Daniel Reeves:

You're offered a gamble where a pair of six-sided dice are rolled and unless they come up snake eyes you get a bajillion dollars. If they do come up snake eyes, you're devoured by snakes.

So far it sounds like you have a 1/36 chance of dying, right?

Now the twist. First, I gather up an unlimited number of people willing to play the game. I take 1 person from that pool and let them play. Then I take 2 people and have them play together, where they share a dice roll and either get the bajillion dollars each or both get devoured. Then I do the same with 4 people, and then 8, 16, and so on.

At some point one of those groups will be devoured by snakes and then I stop.

Is the probability that you'll die, given that you're chosen to play, still 1/36?

The Shooting Room Paradox is similar. As asked by John Leslie, in "Time and the Anthropic Principle (1992):

Thrust into a room, you are assured that 90% of those who enter it will be shot. Panic! But you then learn that you will leave the room alive unless double-six is thrown, first time, with two dice. How is this compatible with the assurance that 90% will be shot? Successive batches of people thrust into the room are successively larger, exponentially, so that the forecast “90% will be shot” will be confirmed when double-six is eventually thrown. If knowing this, and knowing also that the dice falls would be utterly unpredictable even to a Laplacean demon who knew everything about the situation when the dice were thrown, then shouldn’t one’s panic vanish?

The Snake Eyes Paradox has a less extreme probability (90% will be shot vs 50% will be devoured by snakes) but it's the same problem.

Less death please

It can be hard to discuss death rationally, so I will discuss a less violent equivalent:

I am a deity, creating snakes from an unlimited number of disembodied souls. I make the snakes in bat

...
[Mostly solved] I get distracted while reading, but can easily comprehend audio text for 8+ hours per day. What are the best AI text-to-speech readers? Alternatively, do you have other ideas for what I could do?
by  — 2023-06-11T03:49:55.617Z

For me, the physical act of scanning words takes active focus compared to the analogue in listening, which is automatic. (I don't think my comprehension or engagement is lower when listening, to be clear).

I've tried the naturalreaders.com 'pro' version, but experienced a few issues:

It separates the text into small chunks, and pauses between each one. it aims to separate them by sentences, but sometimes separates them by comma, and rarely after a lone word (both of which make my intuition assume a new sentence has started).
The voice doesn't seem to know what it's reading, fails to emphasize what should be, etc. It's nowhere near the quality of Solenoid Entity's readings of the sequences, for example.

As a result, I think my brain doesn't register this AI-read text as 'something to listen to,' so it takes some active focus to continue listening, and eventually my focus shifts to something else while the audio keeps playing in the background. This does not happen with human-read text.

Anyone who can help me with this might have a high potential impact, since I'd be listening to text for a large portion of my day and am trying to use myself to do everything I can to help with alignment.
The Dictatorship Problem
by alyssavance — 2023-06-11T02:45:59.388Z

(Disclaimer: This is my personal opinion, not that of any movement or organization.)

This post aims to show that, over the next decade, it is quite likely that most democratic Western countries will become fascist dictatorships - this is not a tail risk, but the most likely overall outcome. Politics is not a typical LessWrong topic, and for good reason:

it tends to impair clear thinking;
most well-known political issues are not neglected;
most political "debates" are simply people yelling at each other online; neither saying anything new, nor even really trying to persuade the opposition.

However, like the COVID pandemic, it seems like this particular trend will be so impactful and so disruptive to ordinary Western life that it will be important to be aware of it, factor it into plans, and try our best to mitigate or work around the effects.

Introduction

First, what is fascism? It's common for every side in a debate to call the other side "fascists" or "Nazis", as we saw during (eg.) the Ukraine War. Lots of things that get called "fascist" online are in fact fairly ordinary, or even harmless. So, to be clear, Wikipedia defines "fascism" as:

a far-right, authoritarian, ultranationalist political ideology and movement, characterized by a dictatorial leader, centralized autocracy, militarism, forcible suppression of opposition, belief in a natural social hierarchy, subordination of individual interests for the perceived good of the nation and race, and strong regimentation of society and the economy

Informally, I might characterize "fascism" as:

a system of government where there are no meaningful elections; the state does not respect civil liberties or property rights; dissidents, political opposition, minorities, and intellectuals are persecuted; and where government has a strong ideology that is nationalist, populist, socially conservative, and hostile to minority groups.

(The last point is what separates fascism from, say, Stalinism. Stalinism is also very bad, but is not a major political force in 2023.) So by "fascism", I specifically mean a radical change in the basic form of government, not simply a state doing dumb things like making immigration hard or banning Bitcoin. Not all fascists are the same - eg. Mussolini's Italy was initially opposed to Nazi-style racism - but their movements, ideology, rhetoric, and leaders tend to share many common characteristics (...
Higher Dimension Cartesian Objects and Aligning ‘Tiling Simulators’
by lukemarks — 2023-06-11T00:13:12.699Z

Thanks to Justis Mills for feedback. Mistakes are mine.

Introduction

This document will attempt to build upon the Cartesian frame paradigm by modeling multi-agent worlds as higher dimension Cartesian objects. Cartesian objects partially betray many of the properties of their frame counterpart, and they aren't really just 'extensions' of frames, but they are quite similar in some respects and can be reasoned about in similar ways. I will discuss their properties, use them to model recursive self improvement (RSI)/tiling, and show how I think they can be used to construct formal frameworks for solving some outer alignment problems in simulators. The math will be quite difficult to understand if you haven't read Introduction to Cartesian Frames and sections 1 and 2 of Tiling Agents for Self-Modifying AI, and the Löbian Obstacle. It is also incredibly messy and will be refined in later writings.

To introduce the concept, I will present an example in the context of a two-agent game:

Two agents (A1 and A2) and an environment (E), all with three actions, resulting in 27 possible worlds (W).

For a clearer representation of world separation, see:

Where each box could be sequentially labeled w1,w2…w27.

In theory a Cartesian object could have any number n∈N of dimensions, and the above is just one potential configuration used as a toy model to introduce the concept. Most models will involve far more than three actions per axis and many more than two agents.

Properties of Cartesian Objects

These are all adaptations of Garrabrant's original definitions of the properties of cartesian frames to work with higher dimension cartesian objects.

The set of all actions that could be pursued by any agent described by such an object is hereafter:

A∗(C)={a11,a12,...,a1m,a21,a22,...,a2m,...,an1,an2,...,anm}[1]

And by a specific agent:

An(C)={an1,an2,...,anm}[1]

Controllability

The set of ensurables, preventables and controllables for the nth agent in the Cartesian object C could thus be formalized as:

Ensuren(C)=S⊆W|∃ani∈An(C),∀e∈E,ani⋅e∈S Preventn(C)=S⊆W|∃ani∈An(C),∀e∈E,ani⋅e∉S Ctrln(C)=Ensuren(C)∩Preventn(C)

Manageability

We consider manageability, which extends controllability to account for uncertainty in the behaviors of other agents, which we formalize as the belief threshold θ (which could change dependent ...
Using Consensus Mechanisms as an approach to Alignment
by Prometheus — 2023-06-10T23:38:22.106Z

“Both the cryptoeconomics research community and the AI safety/new cyber-governance/existential risk community are trying to tackle what is fundamentally the same problem: how can we regulate a very complex and very smart system with unpredictable emergent properties using a very simple and dumb system whose properties once created are inflexible?”

-Vitalik Buterin, founder of Ethereum

I think this was as true in 2016 as it still is today. And I think one approach to attacking the problem of alignment is not just by combining these two communities, but combining elements of each technology and understanding.

There are two different elements to the problem of Alignment. Getting an AI to do the things we want, and being able to come to terms on what we actually want. We’ve gotta align the AI to the humans, and we also gotta align the humans to the other humans (both present and future). My idea takes from my experience in how DAOs and other mechanisms try to solve large-scale coordination failures and a different kind of reward function.

Another element where combination could work is the ideas of Futarchy, as first imagined by Robin Hanson (vote on values, bet on beliefs), and applying it to both consensus making and AI.

Policy/metric network

Humans all over the world set goals or metrics that they want to achieve. This will be in the form of something like a global DAO, with verification using something like OpenAI’s WorldCoin. These are not infinite. They are not maximum utility forever goals. They have end dates. They have set definitions by humans. Example: reduce malaria by x%.

Source

Prediction Network

We have humans make predictions about which implementations will result in the policy/metric succeeding. These predictions include predicting that humans in the future, after the policy was implemented, will approve of its implementation. These approvals will be set by the policy network after the implementation in various sequences (right after implementation, a year after, 10 years, 100 years, etc.) There is no end date for the approvals continuing. There is no point where it will be totally safe for deception, in other words. An AI will be trained on the data from this prediction network. The AI on this prediction network never stops training. It is always continuing its training run.

Source

The network generalizes to assume app...
Humanities first math problem, The shallow gene pool.
by archeon — 2023-06-10T23:09:00.212Z

Today we all share a large and deep gene pool, but that was not always the case. From first Homo Sapiens until we left Africa that gene pool was small and shallow. Genetically they were a far more homogeneous group than us. If you add consanguinity (inbreeding) to the mix then they faced an existential risk. Yet DNA analysis suggests that even though they lived in small tribes, they were less inbred than modern populations.

Evolutionary success depends on parents finding unrelated mates for their offspring. Think for a moment of the math problem they faced, they had to exclude everyone closely related to themselves and they had to know the family tree of a potential mate.

Bastard was the name given to low status offspring of an unknown father and they would struggle to find a mate. This lasted right up until the widespread adoption of the pill in the 1960's. Because most people lived in small rural communities how could women know a bastard child was not fathered by their close male relative? Or the close male relative of the bastards mother? The taboo was so strong that even today women in isolated communities have encyclopedic knowledge of how everyone is inter-related.

I know nothing of math's and struggle to frame the question, but I will try. How many people do you have to know the recent family tree of to avoid inbreeding when living in small groups over many generations? You would also have to know the family tree for three generations of surrounding tribes with whom you exchange mates. How do you do that for the 10,000 + generations humanity was isolated on the African Continent? Other species susceptible to inbreeding risks use pheromones to regulate mating, we have lost the knack. The Dunbar number of 150 would not last more than a generation or so. Could someone please ask GPT?

Stone Age Homo Sapiens solved the problem, perhaps it is one of the reasons they evolved brains capable of consciousness and complex thoughts. Their brains were not just spare capacity they carried around awaiting us to find a use for it, that is not how evolution works.