trentmkelly/LessWrong-43k huggingface dataset:
What's the deal with AI consciousness?
by TW123 — 2023-01-11T16:37:02.200Z
Link: https://aiwatchtower.substack.com/p/whats-the-deal-with-ai-consciousness

Personal blog; written for a general audience. Thanks to Mantas Mazeika for feedback, and to Daniel Greco for teaching a class I took on philosophy of mind. Disclaimer: I am nowhere close to an expert on consciousness.

On November 28th, 2022, NYU Professor David Chalmers gave a talk at one of the top AI conferences, Neural Information Processing Systems (NeurIPS -- an earlier public version of the talk is

here

). But he wasn’t going to talk about his research in machine learning. In fact, Chalmers is a philosopher, not an AI researcher. He had instead been invited to speak about something that had until recently been shunned by most serious AI researchers: consciousness. Today, AI consciousness is no longer a fringe discussion: The New York Times

recently

called AI consciousness “the last word.”

For hundreds of years, people have been imagining what it would be like to create artificial beings that think and feel just like we do. AI, in some ways, can be seen as a continuation of that trend. But is consciousness a legitimate question or a red herring? Should it really inform how we think about AI?

The hard problem of consciousness

Before we talk about what consciousness is, we have to talk about what it’s not. Consciousness is not, contrary to the popular imagination, the same thing as intelligence. And, contrary to the implication of one researcher recently quoted in the New York Times, consciousness does not have any obvious relation to “resilience.”

Consciousness, in the most precise use of the term, means subjective experience. It means feelings. When you see red, you have a slightly different internal experience from seeing blue, which is quite different from the internal experience you get when you give yourself a papercut. To test for consciousness, Thomas Nagel asks whether there is something it is like to be that thing. For example, there’s something it’s like to be me, and if you are reading this text there’s presumably something it’s like to be you. I think there’s probably something it’s like to be my cat. But there’s nothing that it’s like to be a rock, and when Google software indexes this post after I publish it, there’s nothing it’s like to be that software.

The problem with consciousness is that it is very difficult, if not impossible, to test for, at least given our current understanding. Chalmers asks us to imagine a zombie: a being that acts jus...
Any significant updates on long covid risk analysis?
by Randomized, Controlled — 2023-01-11T14:31:17.445Z

[ETA: This question is specifically about risk analysis of long covid, given current knowledge. Discussion should be about things such as:

base rates

priors and posteriors for LC risk

evidence that would make one update in favor of certain new policies with regard to covid

other decision-relevant considerations.

Moderation guidelines have been added to reflect this.]

Much of the attention on here to the long 'rona [LC] seems to have fizzled out about a year ago. Zvi continued to occasionally report on it, but often in the form of roundly mocking poorly designed studies or sloppy conclusions. This study, also from last year (h/t Zvi), found 50 - 80% reduction in LC in fully vaxx'd folks.

A Q largely asking the same thing from last year.

This recent Zvi 'rona report cites a promising study on LC symptoms controlled against other viral infections. I do worry about Zvi having a sampling or reporting bias, I feel that a systematic review of these LC results would be a great service to the community and perhaps even humanity.

Having said that, this quote from Zvi's December is choice:

Long Infection is a thing across infections. Getting sick is not good for you.
We do not pay enough attention to Long Infection, but Long Covid is not special.

There have also been a handful of LC doomers around these parts, eg, Siebe has a gloomy estimate of 1 - 15% chance of LC even among the vax'd.

I know talking about [long] covid after labor day is gauche, but it is still a thing, and naive waife that I am, I would have hoped a year later our civ's understanding of the risk profile would be "a year better." Whatever the heck that means.
internal in nonstandard analysis
by Alok Singh — 2023-01-11T09:58:49.301Z

Internal set theory is a theory introduced by Edward Nelson, notably an ultrafinitist. it introduces a new function std: * -> Bool. You have to just know if something is standard or not. This may sound bad, but is actually a brilliant way of venting complexity. Figuring out if something is standard is obvious.

1 is standard.

pi is standard.

1 + epsilon is nonstandard if epsilon is infinitesimal.

a hyperfinite number is nonstandard.

Because this is a syntactic theory, it can’t really distinguish infinitesimal and unlimited elements from regular ones. But we can provide the semantics and just know if something is standard or not.

as long as we stick to internal formulas, this extension transfers to the original axioms.

a function is internal if it treats standard and nonstandard elements the same.

example:

1 + x is internal. no distinction is made on what x is.

(1 if standard else 0) is external aka not internal.

no recursive calls can lead to anything with the term ‘standard‘ in it. in practice, this is also fine.

this trick of interpreting syntax in a larger domain of elements (sometimes called abstract interpretation) is also how dual numbers in autodiff work too. the hyperreals are the dual numbers’ bigger sister.

amazingly, griewank’s autodiff book only mentions nonstd analysis once, for like 2 sentences.

James Bradbury if you’re reading this learn some stuff about the hyperreals and make an autodiff system that can differentiate functions that aren’t even continuous.
Compounding Resource X
by Raemon — 2023-01-11T03:14:08.565Z

A concept I got from Anna Salamon, and maybe also from my coworker Jacob Lagerros, although I haven't run this by either of them and they may not endorse my phrasing. I think I probably run with it in a different direction than Anna would endorse.

Epistemic effort: thought about it for a few hours. Basic concept makes sense to me. If it doesn't make sense to you, let me and maybe we can talk-cruxes.

Say you have a problem you don't know how to solve, and seems computationally intractable to strategize about. There are a few ways to go about solving it anyway. But one idea is to follow a heuristic where you look for Resource X, where X has two properties:

X compounds over time
If you have a lot of X, you win.

Say you're playing the game "Chess", or you're playing the game "The Stock Market."

In the stock market, money compounds fairly straightforwardly. You invest money, it results in you getting more money. If you're ultimate goal is either to have a lot of money, or spend a lot of money on a thing, then you win. Hurray. In Chess, you can't calculate all the moves in advance, but, you can try to gain control of the center of the board. “Control over the center” tends to help you gain more control over the center, and if you have enough of it, you win.

Why does it matter that it "compound?". In this scenario you've decided you're trying to win by applying a lot of resource X. If you're starting out without much X, you need some kind of story for how you're going to get enough. If you need 1,000,000 metaphorical units of X within 10 years, and you only have 10 (or, zero).... well, maybe you can linearly gain X at a rate of 100,000 per year. Maybe you could find a strategy that doesn't get any X at all and then suddenly gets all 1,000,000 at the last second. But in practice, if you're starting with a little bit of something, getting a lot of it tends to involve some compounding mechanism.

If you're creating a startup, users can help you get more users. They spread awareness via worth of mouth which grows over time, and in some cases by creating network effects that make other users find your product more valuable. They also, of course, can get you more money, which you can invest in more employees, infrastructure or advertising.

My coworker Jacob used to ask a similar question, re: strategizing at Lightcone Infrastructure. We're trying to provide a lot...
Running With a Backpack
by jefftk — 2023-01-11T03:00:03.454Z

In the comments on my

running by
default post,

one

of

the

more

common
responses was that people often have backpacks and a bouncing backpack
is somewhere between "annoying" and "painful, risking bruising".

There was some
discussion on how to set up your backpack to reduce bouncing.
Tips:

Some backpacks have straps you can buckle across your front, or
you can get add-on ones. Wear them super snug.

Put your water bottle inside where it's more central and less
likely to fall out.

Figure out if there are things you don't need to be
transporting so you can have a lighter pack.

Use a fancy backpack with good padding and a rigid back.

Hold the shoulder straps as you run to absorb jostling.

These all seem like fine ideas, but I don't do any of them. My
backpack is relatively low-end ($27) and
non-rigid, without any sternum or waist straps. I keep my water
bottle in the outside pocket, I have many little things in it that I could
probably carry fewer of, and I don't hold the shoulder straps. And
yet I probably do the majority of my running with a backpack on: I'm
commuting and I have things I need to bring with me. Even when I was
trying to run fast and repeatedly beat my personal best
times I had my pack.

What works for me instead is that I've learned to run with a gait that
doesn't bounce my backpack. I take a lot (~190
bpm) of short strides which means I have less time to fall between
steps, and I generally stay pretty level. I think this is probably
also beneficial from a perspective of minimizing the impact forces on
your joints (ex: a paper I
skimmed). Whether my backpack is heavy or light, it mostly stays
put on my back.

It's still less pleasant to run with the extra weight of a backpack
since it's more work, but if I think of it as intentional exercise (a
bit like a weighted vest you already have with you!) I don't mind it
too much.
A simple thought experiment showing why recessions are an unnecessary bug in our economic system
by skogsnisse — 2023-01-11T00:43:47.111Z

On Monday we all go to the offices and factories we work at and produce goods and services. We exchange these goods and services among each other for mutual benefit.

On Tuesday it's declared there is a recession. The same offices, desks, factories, machines are still sitting in the same buildings, but now for some reason we stop going to work, and we stop exchanging goods and services.

Why is that? We could decide to just go to work anyway and do the exact same thing we did on Monday - the desks and machines are still there exactly the same as yesterday for us to use, and we can still exchange goods and services at will.
We don’t trade with ants
by KatjaGrace — 2023-01-10T23:50:11.476Z

When discussing advanced AI, sometimes the following exchanges happens:

“Perhaps advanced AI won’t kill us. Perhaps it will trade with us”

“We don’t trade with ants”

I think it’s interesting to get clear on exactly why we don’t trade with ants, and whether it is relevant to the AI situation.

When a person says “we don’t trade with ants”, I think the implicit explanation is that humans are so big, powerful and smart compared to ants that we don’t need to trade with them because they have nothing of value and if they did we could just take it; anything they can do we can do better, and we can just walk all over them. Why negotiate when you can steal?

I think this is broadly wrong, and that it is also an interesting case of the classic cognitive error of imagining that trade is about swapping fixed-value objects, rather than creating new value from a confluence of one’s needs and the other’s affordances. It’s only in the imaginary zero-sum world that you can generally replace trade with stealing the other party’s stuff, if the other party is weak enough.

Ants, with their skills, could do a lot that we would plausibly find worth paying for. Some ideas:

Cleaning things that are hard for humans to reach (crevices, buildup in pipes, outsides of tall buildings)

Chasing away other insects, including in agriculture

Surveillance and spying

Building, sculpting, moving, and mending things in hard to reach places and at small scales (e.g. dig tunnels, deliver adhesives to cracks)

Getting out of our houses before we are driven to expend effort killing them, and similarly for all the other places ants conflict with humans (stinging, eating crops, ..)

(For an extended list, see ‘Appendix: potentially valuable things things ants can do’)

We can’t take almost any of this by force, we can at best kill them and take their dirt and the minuscule mouthfuls of our foods they were eating.

Could we pay them for all this?

A single ant eats about 2mg per day according to a random website, so you could support a colony of a million ants with 2kg of food per day. Supposing they accepted pay in sugar, or something similarly expensive, 2kg costs around $3. Perhaps you would need to pay them more than subsistence to attract them away from foraging freely, since apparently food-gathering ants usually collect more than they eat, to support others in their colony. So let’s guess $5.

My ...
Who are the people who are currently profiting from inflation?
by skogsnisse — 2023-01-10T21:39:10.962Z

When prices increase we pay more for goods and services. Cost of living is higher because of inflation - we have to work more to earn enough money to buy the same goods and services. Who is at the other end of that transaction? Who is it that gets paid more for the same goods and services? Until everyones gets a raise so they can work the same number of hours to afford life as before inflation, someone is profiting. Who is that?
Is Progress Real? 
by rogersbacon — 2023-01-10T17:42:08.188Z
Link: https://www.secretorum.life/p/is-progress-real

...

I couldn't find the essay “Is Progress Real?” by the historians Will and Ariel Durant (

The Lessons of History

, 1968) anywhere on the internet so here I am now, posting it on the internet. I think it's a classic, short but filled with provoking ideas and beautiful prose and I don’t think it’s an exaggeration to say that no one writes like this anymore.

Afterwards, I offer some commentary on the essay and make a devastating criticism of the idea of progress (a death blow, really). Enjoy.

Is Progress Real?

Against this panorama of nations, morals, and religions rising and falling, the idea of progress finds itself in dubious shape. Is it only the vain and traditional boast of each "modern'" generation? Since I have admitted no substantial change in man's nature during historic times, all technological advances will have to be written off as merely new means of achieving old ends—the acquisition of goods, the pursuit of one sex by the other (or by the same), the overcoming of competition, the fighting of wars. One of the discouraging discoveries of our disillusioning century is that science is neutral: it will kill for us as readily as it will heal, and will destroy for us more readily than it can build. How inadequate now seems the proud motto of Francis Bacon, "Knowledge is power"! Sometimes we feel that the Middle Ages and the Renaissance, which stressed mythology and art rather than science and power, may have been wiser than we, who repeatedly enlarge our instrumentalities without improving our purposes.

Our progress in science and technique has involved some tincture of evil with good. Our comforts and conveniences may have weakened our physical stamina and our moral fiber. We have immensely developed our means of locomotion, but some of us use them to facilitate crime and to kill our fellow men or ourselves. We double, triple, centuple our speed, but we shatter our nerves in the process, and are the same trousered apes at two thousand miles an hour as when we had legs. We applaud the cures and incisions of modern medicine if they bring no side effects worse than the malady; we appreciate the assiduity of our physicians in their mad race with the resilience of microbes and the inventiveness of disease; we are grateful for the added years that medical science gives us if they are not a burden—some prolongation of illness, disability, and gloom. We have...
200 COP in MI: Interpreting Reinforcement Learning
by Neel Nanda — 2023-01-10T17:37:44.941Z

This is the ninth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. Look up jargon in

my Mechanistic Interpretability Explainer

Motivating papers: Acquisition of Chess Knowledge in AlphaZero, Understanding RL Vision

Disclaimer: My area of expertise is language model interpretability, not reinforcement learning. These are fairly uninformed takes and I’m sure I’m missing a lot of context and relevant prior work, but this is hopefully more useful than nothing!

Motivation

Reinforcement learning is the study of how to create agents - models that can act in an environment and form strategies to get high reward. I think there are a lot of deep confusions we have about how RL systems work and how they learn, and that trying to reverse engineer these systems could teach us a lot! Some high-level questions about RL that I’d love to get clarity on:

How much do RL agents learn to actually model their environment vs just being a big bundle of learned heuristics?

In particular, how strong a difference is there between model-free and model-based systems here?

Do models explicitly reason about their future actions? What would a circuit for this even look like?
What are the training dynamics of an RL system like? What does its path to an eventual solution look like? A core hard part about training RL systems is that the best strategies tend to require multi-step plans, but taking the first step only makes sense if you know that you’ll do the next few steps correctly.

What does it look like on a circuit level when a model successfully explores and finds a new and better solution?

How much do the lessons from mechanistic interpretability transfer? Can we even reverse engineer RL systems at all?

How well do our conceptual frameworks for thinking about networks transfer to RL? Are there any things that clearly break, or new confusions that need to be grappled with?
I expect training dynamics to be a particularly thorny issue - rather than learning a fixed data distribution, whether an action is a good idea depends on which actions the model will take in future.

Further, I think that it’s fairly likely that, whatever AGI looks like, it involves a significant component ...
AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years
by basil.halperin — 2023-01-10T16:06:52.329Z

by

Trevor Chow

,

Basil Halperin

, and

J. Zachary Mazlish

In this post, we point out that short AI timelines would cause real interest rates to be high, and would do so under expectations of either unaligned or aligned AI. However,

30- to 50-year real interest rates are low

. We argue that this suggests one of two possibilities:

Long(er) timelines. Financial markets are often highly effective information aggregators (the “efficient market hypothesis”), and therefore real interest rates accurately reflect that transformative AI is unlikely to be developed in the next 30-50 years.
Market inefficiency. Markets are radically underestimating how soon advanced AI technology will be developed, and real interest rates are therefore too low. There is thus an opportunity for philanthropists to borrow while real rates are low to cheaply do good today; and/or an opportunity for anyone to earn excess returns by betting that real rates will rise.

In the rest of this post we flesh out this argument.

Both intuitively and under every mainstream economic model, the “

explosive growth

” caused by aligned AI would cause high real interest rates.
Both intuitively and under every mainstream economic model, the existential risk caused by unaligned AI would cause high real interest rates.
We show that in the historical data, indeed, real interest rates have been correlated with future growth.
Plugging the

Cotra probabilities

for AI timelines into the baseline workhorse model of economic growth implies substantially higher real interest rates today.
In particular, we argue that markets are decisively rejecting the shortest possible timelines of 0-10 years.
We argue that

the efficient market hypothesis (EMH) is a reasonable prior

, and therefore one reasonable interpretation of low real rates is that since markets are simply not forecasting short timelines, neither should we be forecasting short timelines.
Alternatively, if you believe that financial markets are wrong, then you have the opportunity to (1) borrow cheaply today and use that money to e.g. fund AI safety work; and/or (2) earn alpha by betting that real rates will rise.

An order-of-magnitude estimate is that,

if markets are getting this wrong, then there is easily $1 trillion lying on the table in the US treasury bond market alone

– setting aside the enormous implications for...
The Alignment Problem from a Deep Learning Perspective  (major rewrite)
by SoerenMind — 2023-01-10T16:06:05.057Z
Link: https://arxiv.org/abs/2209.00626

We've recently uploaded a major rewrite of Richard Ngo's:

The Alignment Problem from a Deep Learning Perspective

We hope it can reach ML researchers by being more grounded in the deep learning literature and empirical findings, and being more rigorous than typical introductions to the alignment problem.

There are many changes. Most obviously, the paper was previously structured into three phases of training: planning towards a mix of desirable and undesirable internally-represented goals, situational awareness, and goals that generalize OOD. Now it is structured around three emergent phenomena: deceptive reward hacking, planning towards internally represented goals, and power-seeking.

Feedback request

Since we're submitting this to ICML in two weeks, we're looking for feedback, including feedback on presentation and feedback you'd give as a critical ML reviewer. It can be nitpicky. If your feedback isn't that interesting to forum readers, you may want to email it to us (find our emails in the Arxiv PDF). It's most useful by 17 January. Many thanks in advance!

Full text copy

Richard Ngo - OpenAI, Lawrence Chan - UC Berkeley (EECS), Sören Mindermann - University of Oxford (CS)

Abstract

Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide range of important tasks. We outline a case for expecting that, without substantial effort to prevent it, AGIs could learn to pursue goals which are very undesirable (in other words, misaligned) from a human perspective. We argue that AGIs trained in similar ways as today's most capable models could learn to act deceptively to receive higher reward; learn internally-represented goals which generalize beyond their training distributions; and pursue those goals using power-seeking strategies. We outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and briefly review research directions aimed at preventing these problems.

Contents

(Page numbers from the PDF version)

1 Introduction 2

2 Deceptive reward hacking 3

2.1 Reward misspecification and reward hacking …………………. 3

2.2 Defining situational awareness ………………. . . . . . . . . . . 4

2.3 Situational awareness enables deceptive reward hacking ……………… 4

3 Internally-represented goals 5

4 Power-seeking behavior 7

4.1 Many broadly-scope...