trentmkelly/LessWrong-43k huggingface dataset:
Are AIs conscious?  It might depend
by Logan Zoellner — 2024-03-15T23:09:26.621Z

As AI progresses rapidly, humanity is going to have to solve a large number of problems in a short period of time. The most pressing of these right now is the AI Alignment problem. After all, hardly anything else matters if we are all dead. A problem that will soon be equally pressing, however, is the Hard Problem of Consciousness. We have already begun to create millions of minds worth of AI Intelligence. And soon the number of AIs will absolutely dwarf the number of humans on Earth. This creates an enormous S-Risk. If AI are conscious beings capable of suffering, we might be creating trillions of lives not worth living.

For the moment, we are probably in the clear. Any decent theory of consciousness should require that conscious beings have a persistent model of the world which includes themselves in the model. LLMs fail this test immediately. They have no persistent model of the world. Indeed they have no persistent model of themselves. Instead, they seem to draw from a pool of billions of potential selves at sampling time and when they are done sampling these selves aren't killed, they simply return to the infinite sea of possibilities.

With robots, however, it is a different story. Who among us hasn't seen a video of a robot falling over and felt a tinge of empathy. Robots have a persistent self (the robot's body) and a model of the world (which they must in order to navigate it). It would be strange, therefore, if robots were much less conscious than say fruit-flies. But how conscious? And does this give them moral standing?

Two theories of Consciousness

One class of theories about consciousness holds that beings are conscious by virtue of possessing a set of properties. Folk consciousness, for example, holds that humans are conscious by virtue of possessing souls. Pan-psychism holds that everything is conscious by virtue of existing. IIT holds that beings are conscious by virtue of being able to hold large sums of information in their head.

As a mathematical realist, however, I find these theories difficult to accept. Within the digits of pi, there exist infinite copies of the information that describes me. Yet I do not consider my consciousness to exist in these copies, but rather in the real world. If I were to die tomorrow, I wou...
Beyond Maxipok — good reflective governance as a target for action
by owencb — 2024-03-15T22:22:03.704Z

Summary

Avoiding existential risk in the foreseeable future is a necessary but not sufficient condition to get good long-term outcomes. If we do get great outcomes, that almost certainly means there was a good reflective process which helped to steer us there.

Positioning the world to enter into such a reflective process is a proper target for action today. This complements the “Maxipok” principle of seeking to minimize existential risk. However, since we don’t yet understand the requirements of a good reflective process, we should not yet plan to commit to a particular version.

Meta

This post expresses some of my thinking. The thinking is in many ways not complete, but it’s been in that state for years; I've found myself several times linking people to private docs; and at some point I think it’s helpful to have something public, so I’m tidying this up and putting it out. Credit for having informed my thinking goes to a lot of people.

The post also stays at a relatively high level of abstraction in analysing existential risk and related areas. I think that in practice it's very important how these ideas intersect with what we're expecting with AI. But it's cleaner to keep the parts of the analysis which don't rely on particular empirical forecasts separate, so I'll leave discussion of AI for another post.

Distribution of futures

How good will the future be? Of course it’s extremely uncertain, and may depend a lot on the perspective it’s being asked from. But that doesn’t mean we can’t say anything. If I condition on some broadly-consequentialist axiology that I might come to endorse on reflection, here’s a cartoon showing very roughly what my probability distribution might look like, with all possible worlds arranged best-to-worst as you go top-to-bottom:

(Probability mass on this diagram is represented by “vertical space”. If you want an nth centile world, you can draw a line n% of the way up and read off its goodness in the horizontal dimension.)

This isn’t supposed to be in the vicinity of precise; especially the goodness scale is not necessarily linear. But the key high-level features that I do (tentatively) stand behind, and want to draw attention to, are:

There’s a reasonable probability mass on “non-futures”; things which are close to zero total value
There’s a reasonable probability mass on “great futures”, which are all in-the-same-ballpark as good as ea

...
Middle Child Phenomenon
by PhilosophicalSoul — 2024-03-15T20:47:25.233Z

Since I've seen no one talk about this, I'm coining the phrase 'Middle Child Phenomenon'.

A law student who entered university four (4) years ago is faced with a curriculum that became completely outdated two (2) years in. Let's take a cohort of 1000 law students from 2020, and explain from there.

100 students drop out by 2022 for a myriad of reasons; sickness, family, poverty, academic difficulty, etc.
80%/720 students are the nobodies who cruise by, haven't really become a lifer yet.

What happens to the remaining 180 students?

50%/90 students are the all-rounders, committed enough to do well academically, have time for extracurriculars and do an apprentice at a Big 5 law firm.
The other 50%/90 students are split into two (2) groups: (i) the CV-Chasers, these win competitions, and can recite the constitutional precedents of the last decade, and (ii) the Golden Children who do everything, and they do it damn well.

The Golden Children of around forty-five (45) students talk here and there about artificial intelligence, maybe host a few workshops. They point out a giant, GIANT, GIANT problem with alignment, and it goes something like:

Richard: "Hey, why doesn't our curriculum have any artificial intelligence electives? Why aren't there any professors including homework problems, or real-life examples for us to deal with?"

Sally: "As they say: '...born too late to sell textbooks, born too early to fix the justice system but born just in time to read the gajillion discovery files Saxmon just gave me. Let's-s-s-s go-o-o--'

Jake: "Hang on for sec, this is an actual problem. Either we beg our supervisors to let us work with Dr Bellows on that first AI case he's cooking up, or we have to do our own reading."

Richard: "And if we don't do something now--since these Professors who've never entered a courtroom in their lives--we're going to graduate in a completely different world."

Sally: "Then get replaced by the young hotshots who are blessed with a curriculum that actually gives a damn...I'm getting it now."

Richard: "It's a shitfest in the shitfactory on planet shitstorm indeed."

The point is, forty-five (45) students is 4.5% of every yearly cohort. Even less than that will be good at helping the war effort with AI alignment, and that's assuming they get hired in a firm with the raw ammunition necessary to institute motion applications that materially change legislation an...
Capability or Alignment? Respect the LLM Base Model’s Capability During Alignment
by Jingfeng Yang — 2024-03-15T17:56:18.591Z

Last year saw a boom of LLMs research. Based on the research, one important lesson would be that we should devote most of our efforts to training a general-purpose LLM base model, and leverage it as much as possible after all. I might be opinionated, but I always believe that one general principle is that we need to respect the base model’s capability during alignment. This argument might be common sense among many people, but it is still controversial among others, especially when it comes to the boundary between capability and alignment. Feel free to correct me if you have more solid empirical evidence.

In this post, I will first define model capability and alignment respectively. Then I will discuss capability and alignment boundaries. I will also show some evidence on LLM capabilities coming from the base model and explain why. Based on this, I will introduce some principles to respect base model capability during each method of alignment. Finally, I emphasize the importance of evaluation used to show the effectiveness of our principle.

All the arguments are based on the goal that base model construction and alignment is to get a general purpose model, chatbot and A(G)I, or at least a specialist that behaves properly in real-world cases, instead of optimizing performance on any specific tasks and domains or performance on benchmarks.

What’s Alignment?

The alignment problem is defined as “How can we create agents that behave in accordance with the user’s intentions” [25]. In the context of LLM, “agent” could be the language model itself, or the LLM augmented with tools, memories and planning/reasoning abilities as a whole. The “intentions” could be explicit or implicit. Explicit intentions could be requirements expressed in the natural language instructions, while implicit intentions are numerous and hard to be captured by limited objectives or natural language, like “do not be toxic” etc. [26] Those implicit intents could also be ambiguous and diverse, or even inconsistent among different people. Many people currently classify them into three main alignment directions: Helpfulness (Implicit Intent and Explicit Instruction Following), Harmlessness (Safety) and Honesty (Truthfulness and less Hallucinations). In this sense, alignment is more like a goal instead of a method. To achieve the goal of alignment, there could be many methods. The most effectiv...
Rational Animations offers animation production and writing services!
by Writer — 2024-03-15T17:26:07.976Z

Rational Animations is now open to take on external work! We offer several services related to writing and animation production. In particular:

Production management
Storyboarding
Visual development
Animation
Editing and compositing
Writing, such as distilling research and creating explainers, stories, or screenplays

We can take on animation projects from start to finish or help you with any phase of the animation production process.

You can look at our most recent work and animation showreel by visiting our new website, rationalanimations.com.

If you'd like to hire us, you can contact us at business@rationalanimations.com
Improving SAE's by Sqrt()-ing L1 & Removing Lowest Activating Features 
by Logan Riggs — 2024-03-15T16:30:00.744Z

TL;DR

We achieve better SAE performance by:

Removing the lowest activating features
Replacing the L1(feature_activations) penalty function with L1(sqrt(feature_activations))

with 'better' meaning: we can reconstruct the original LLM activations w/ lower MSE & with fewer features/datapoint.

As a sneak peak (the graph should make more sense as we build up to it, don't worry!):

The L1(sqrt()) graph (ie dotted one) is farther to the lower-left corner (this is good!) of low Cross Entropy loss with low features/datapoint (ie L0 Norm).

Now in more details:

Sparse Autoencoders (SAEs) reconstruct each datapoint in [layer 3's residual stream activations of Pythia-70M-deduped] using a certain amount of features (this is the L0-norm of the hidden activation in the SAE). Typically the higher activations are interpretable & the lowest of activations non-interpretable.

This is a histogram of the specific tokens that activated a specific feature. If you run 1M tokens in an LLM, 1000 of those activate this feature. Of those 1000, there are ~30 that activate between 5.53 & 6.15 which are all the token apostrophe (aka ').

Here is a feature that activates mostly on apostrophe (removing it also makes it worse at predicting "s"). The lower activations are conceptually similar, but then we have a huge amount of tokens that are something else.

From a datapoint viewpoint, there's a similar story: given a specific datapoint, the top activation features make a lot of sense, but the lowest ones don't (ie if 20 features activate that reconstruct a specific datapoint, the top ~5 features make a decent amount of sense & the lower 15 make less and less sense)

Are these low-activating features actually important for downstream performance (eg CE)? Or are they modeling noise in the underlying LLM (which is why we see conceptually similar datapoints in lower activation points)?

Ablating Lowest Features

There are a few different ways to remove the "lowest" feature activations.

Dataset View:

Lowest k-features per datapoint

Feature View: Features have different activation values. Some are an OOM larger than others on average, so we can set feature specific thresholds.

Percentage of max activation - remove all feature activations that are < [10%] of max activation for that feature
Quantile - Remove all features in the [10th] percentile activations for each feat

...
Controlling AGI Risk
by TeaSea — 2024-03-15T04:56:09.284Z

A theory of AGI safety based on constraints and affordances.

I've got this proto-idea of what's missing in much public discussion and action on AI safety. I'm hoping that by sharing it here, the hive-mind might come together and turn it into something useful.

Effective control of AI risk requires a broader approach than those taken so far. Efforts to-date have largely gravitated into the two camps of value alignment and governance. Value alignment aims to design AI systems that reliably act in the best interest of humans. Governance efforts aim to constrain people who develop, deploy or use AI to do so in ways that ensure the AI doesn't cause unacceptable harm.

These two camps are each and together necessary but insufficient to adequately control AI risk.

Firstly, AI capabilities emulate human cognitive capabilities. Their potential applications are so broad that their scope for application transcends all previous technologies. Most of the thinking and action-to-date on controlling AI risk has been based on how we've controlled the risks of previous technologies such as electricity, mechanized transport, and nuclear weapons. So far, we've mostly thought of AI as a technology to be used by humans, not as itself a user of technology.

Secondly, the acceleration of AI evolution is unlikely to decrease; the converse looks more likely, that increasingly capable and powerful AI will further accelerate the ongoing development of AI capability (including via self-improvement). Traditional governance mechanisms can't keep pace with this and any value alignment of systems could be transcended by the next emergent system. Just as AI is likely to impact, interact with, and become embedded in the whole of society, whole-of-society risk control practices must evolve.

AI systems are already becoming embedded in large swathes of society.

It is likely that AGI will soon be here.

Control of risk from AGI needs to be as ubiquitous as control of risk from people.

Definitions:

Risk
The potential for unintended or undesirable loss or harm

AGI
Artificial General Intelligence: AI that can perform any intellectual task that a human can

Sociotechnical system/s (STS)
A system in which agents (traditionally, people) interact with objects (including technologies) to achieve aims and fulfil purposes

Agent
An entity in an STS that makes decisions and initiates actions

Sensor
A mechanism via which an agent ac...
Constructive Cauchy sequences vs. Dedekind cuts
by jessicata — 2024-03-14T23:04:07.300Z
Link: https://unstableontology.com/2024/03/14/constructive-cauchy-sequences-vs-dedekind-cuts/

In classical ZF and ZFC, there are two standard ways of defining reals: as Cauchy sequences and as Dedekind cuts. Classically, these are equivalent, but are inequivalent constructively. This makes a difference as to which real numbers are definable in type theory.

Cauchy sequences and Dedekind cuts in classical ZF

Classically, a Cauchy sequence is a sequence of reals x1,x2,…, such that for any ϵ>0, there is a natural N such that for any m,n>N, |xm−xn|<ϵ. Such a sequence must have a real limit, and the sequence represents this real number. Representing reals using a construction that depends on reals is unsatisfactory, so we define a Cauchy sequence of rationals (CSR) to be a Cauchy sequence in which each xi is rational.

A Cauchy sequence lets us approximate the represented real to any positive degree of precision. If we want to approximate the real by a rational within ϵ, we find N corresponding to this ϵ and use xN+1 as the approximation. We are assured that this approximation must be within ϵ of any future xi in the sequence; therefore, the approximation error (that is, |xN+1−limi→∞xi|) will not exceed ϵ.

A Dedekind cut, on the other hand, is a partition of the rationals into two sets A,B such that:

A and B are non-empty.

For rationals x < y, if y∈A, then x∈A (A is downward closed).

For x∈A, there is also y∈A with x<y (A has no greatest element).

It represents the real number supA. As with Cauchy sequences, we can approximate this number to within some arbitrary ϵ; we do this by doing a binary search to find rationals x<y with x∈A,y∈B,|x−y|<ϵ, at which point x approximates supA to within ϵ. (Note that we need to find rational bounds on supA before commencing a straightforward binary search, but this is possible by listing the integers sorted by absolute value until finding at least one in A and one in B.)

Translating a Dedekind cut to a CSR is straightforward. We set the terms of the sequence to be successive binary search approximations of supA, each of which are rational. Since the binary search converges, the sequence is Cauchy.

To translate a CSR to a Dedekind cut, we will want to set A to be the set of rational numbers strictly less than the sequence’s limit; this is correct regardless if the limit is rational (check both cases). These constitute the set of rationals y for which there exists some rational ϵ>0 and some na...
Toward a Broader Conception of Adverse Selection
by Ricki Heicklen — 2024-03-14T22:40:57.920Z
Link: https://bayesshammai.substack.com/p/conditional-on-getting-to-trade-your

“I refuse to join any club that would have me as a member” -Marx[1]

Adverse Selection is the phenomenon in which information asymmetries in non-cooperative environments make trading dangerous. It has traditionally been understood to describe financial markets in which buyers and sellers systematically differ, such as

a market for used cars

in which sellers have the information advantage, where resulting feedback loops can lead to market collapses.

In this post, I make the case that adverse selection effects appear in many everyday contexts beyond specialized markets or strictly financial exchanges. I argue that modeling many of our decisions as taking place in competitive environments analogous to financial markets will help us notice instances of adverse selection that we otherwise wouldn’t.

The strong version of my central thesis is that conditional on getting to trade[2], your trade wasn’t all that great. Any time you make a trade, you should be asking yourself “what do others know that I don’t?” This does not mean that your trade is necessarily net bad, just that it is worse than it would have naively seemed before conditioning on having gotten to do the trade. The opportunity to trade is evidence that somebody else—in some cases, everybody else—passed over the decision to take that trade, or actively chose to take the other side of it.

This post is the first in a sequence on adverse selection, laying out my definition of the concept via a list of examples. The second post will discuss what factors determine the degree to which adverse selection is present in a given environment, and heuristics to detect them. The third will explore steps we can take to protect ourselves against adverse selection, as individuals and as a collective. The fourth will compare competitive and cooperative environments, the fifth will discuss a related claim that ambiguity obscures defection, and the sixth will connect to the concept of Goodharting and what levers of control we have over it.

Beware availability

When some options are taken and others are not, there may be a good reason for the difference in availability, even if that reason is not immediately obvious to you.

Alice’s Restaurant vs Bob’s Burgers

You’re on a road trip and passing through the exotic city of Teaneck, New Jersey, and decide to grab dinner at 7pm. The town has two di

...
More people getting into AI safety should do a PhD
by AdamGleave — 2024-03-14T22:14:48.855Z
Link: https://gleave.me/post/why-do-phd/

Doing a PhD is a strong option to get great at developing and evaluating research ideas. These skills are necessary to become an AI safety research lead, one of the key talent bottlenecks in AI safety, and are helpful in a variety of other roles. By contrast, my impression is that currently many individuals with the goal of being a research lead pursue options like independent research or engineering-focused positions instead of doing a PhD. This post details the reasons I believe these alternatives are usually much worse at training people to be research leads.

I think many early-career researchers in AI safety are undervaluing PhDs. Anecdotally, I think it’s noteworthy that people in the AI safety community were often surprised to find out I was doing a PhD, and positively shocked when I told them I was having a great experience. In addition, I expect many of the negatives attributed to PhDs are really negatives on any pathway involving open-ended, exploratory research that is key to growing to become a research lead.

I am not arguing that most people contributing to AI safety should do PhDs. In fact, a PhD is not the best preparation for the majority of roles. If you want to become a really strong empirical research contributor, then start working as a research engineer on a great team: you will learn how to execute and implement faster than in a PhD. There are also a variety of key roles in communications, project management, field building and operations where a PhD is of limited use. But I believe a PhD is excellent preparation for becoming a research lead with your own distinctive research direction that you can clearly communicate and ultimately supervise junior researchers to work on.

However, career paths are highly individual and involve myriad trade-offs. Doing a PhD may or may not be the right path for any individual person: I simply think it has a better track record than most alternatives, and so should be the default for most people. In the post I’ll also consider counter-arguments to a PhD, as well as reasons why particular people might be better fits for alternative options. I also discuss how to make the most of a PhD if you do decide to pursue this route.

Author Contributions: This post primarily reflects the opinion of Adam Gleave so is written using an “I” personal pronoun. Alejandro Ortega and Sean McGowa...