trentmkelly/LessWrong-43k huggingface dataset:
“Artificial General Intelligence”: an extremely brief FAQ
by Steven Byrnes — 2024-03-11T17:49:02.496Z

(Crossposted from twitter for easier linking.) (Intended for a broad audience—experts already know all this.)

When I talk about future “Artificial General Intelligence” (AGI), what am I talking about? Here’s a handy diagram and FAQ:

“Are you saying that ChatGPT is a right-column thing?” No. Definitely not. I think the right-column thing does not currently exist. That’s why I said “future”! I am also not making any claims here about how soon it will happen, although see discussion in Section A here.

“Do you really expect researchers to try to build right-column AIs? Is there demand for it? Wouldn’t consumers / end-users strongly prefer to have left-column AIs?” For one thing, imagine an AI where you can give it seed capital and ask it to go found a new company, and it does so, just as skillfully as Earth’s most competent and experienced remote-only human CEO. And you can repeat this millions of times in parallel with millions of copies of this AI, and each copy costs $0.10/hour to run. You think nobody wants to have an AI that can do that? Really?? And also, just look around. Plenty of AI researchers and companies are trying to make this vision happen as we speak—and have been for decades. So maybe you-in-particular don’t want this vision to happen, but evidently many other people do, and they sure aren’t asking you for permission.

“If the right-column AIs don’t exist, why are we even talking about them? Won’t there be plenty of warning before they exist and are widespread and potentially powerful? Why can’t we deal with that situation when it actually arises?” First of all, exactly what will this alleged warning look like, and exactly how many years will we have following that warning, and how on earth are you so confident about any of this? Second of all … “we”? Who exactly is “we”, and what do you think “we” will do, and how do you know? By analogy, it’s very easy to say that “we” will simply stop emitting CO2 when climate change becomes a sufficiently obvious and immediate problem. And yet, here we are. Anyway, if you want the transition to a world of right-column AIs to go well (or to not happen in the first place), there’s already plenty of work that we can and should be doing right now, even before those AIs exist. Twiddling our thumbs and kicking the can down the road is crazy.

“The right column sounds like weird sci-fi stuff. Am I really supposed to ...
Some (problematic) aesthetics of what constitutes good work in academia
by Steven Byrnes — 2024-03-11T17:47:28.835Z

(Not-terribly-informed rant, written in my free time.)

Terminology note: When I say “an aesthetic”, I mean an intuitive (“I know it when I see it”) sense of what a completed paper, project, etc. is ideally “supposed” to look like. It can include both superficial things (the paper is properly formatted, the startup has high valuation, etc.), and non-superficial things (the theory is “elegant”, the company is “making an impact”, etc.).

Part 1: The aesthetic of novelty / cleverness

Example: my rant on “the psychology of everyday life”

(Mostly copied from

this tweet

)

I think if you want to say something that is:

(1) true,
(2) important, and
(3) related to the psychology of everyday life,

…then it’s NOT going to conform to the aesthetic of what makes a “good” peer-reviewed academic psych paper.

The problem is that this particular aesthetic demands that results be (A) “novel”, and (B) “surprising”, in a certain sense. Unfortunately, if something satisfies (1-3) above, then it will almost definitely be obvious-in-hindsight, which (perversely) counts against (B); and it will almost definitely have some historical precedents, even if only in folksy wisdom, which (perversely) counts against (A).

If you find a (1-3) thing that is not “novel” and “surprising” per the weird peer-review aesthetic, but you have discovered a clearer explanation than before, or a crisper breakdown, or better pedagogy, etc., then good for you, and good for the world, but it’s basically useless for getting into top psych journals and getting prestigious jobs in psych academia, AFAICT. No wonder professional psychologists rarely even try.

Takeaway from the perspective of a reader: if you want to find things that are all three of (1-3), there are extremely rare, once-in-a-generation, academic psych papers that you should read, and meanwhile there’s also a giant treasure trove of blog posts and such. For example:

Motivated reasoning is absolutely all three of (1-3). If you want to know more about motivated reasoning, don’t read psych literature, read

Scout Mindset

.
Scope neglect is absolutely all three of (1-3). If you want to know more about scope neglect, don’t read psych literature, read

blog posts about Cause Prioritization

.
As it happens, I’ve been recently trying to make sense of social status and related behaviors. And none of the best sources I’ve found have been academic psycho

...
Storable Votes with a Pay as you win mechanism: a contribution for institutional design
by Arturo Macias — 2024-03-11T15:58:06.559Z

I joined the EA Forum in 2022, with a post describing my interests and agenda. I also declared in my first comment that in my view, among the main existential risk bottlenecks for this Dangerous Century, a critical one is institutional stagnation. E.O Wilson famously said: "The real problem of humanity is the following: we have Paleolithic emotions, medieval institutions, and god-like technology".

Regarding the Paleolithic emotions, and godlike technology I have nothing to contribute, but regarding the medieval institutions I think I can make some modest contributions.

Here are two of them, very likely my most important scientific contributions so far: the first is an already published journal article, the second, a new pre-print (please, feel free to make suggestions for improvement).

Storable Votes with a Pay as You Win mechanism

This article (“Storable Votes with a Pay as You Win mechanism” [Journal of Economic Interaction and Coordination, pre-print here for access after the expiry of ShareLink]) presents a dynamic voting mechanism on multiple alternatives (Storable Votes-Pay as You Win [SV-PAYW]).

At the beginning, all agents are given an equal number of (infinitely divisible) storable votes. The agents say how many votes they are willing “to pay” for each of the possible alternatives and the most voted alternative wins the election. Then, the votes that have been committed to the winning alternative are deducted from each player's account, and are equally redistributed among all participants, and a new voting period begins.

The system reduces the incentives for strategic voting: agents do not stop signaling their interest in alternatives with little probability of victory (if it does not win, you do not pay votes), and it solves the problem of minority disenfranchisement: the more elections a subject loses, the more power future electoral power she accumulated. The article uses exact computational methods (GAMBIT is used for backward induction). The simulations indicate that the PAYW part improves a fixed number of votes version of the Storable Votes

SV-PAYW shall be considered as a natural alternative to Quadratic Voting for its use in distributed governance systems (vg. to implement the democratic reforms proposed in “Radical Markets”). In my view is equally simple, and the avoidance of strategic...
Tend to your clarity, not your confusion
by Severin T. Seehrich — 2024-03-11T15:09:24.099Z

Repost from https://amoretlicentia.substack.com/

Modern life is weird.

For the more privileged among us, the options of what we could do with our time grow exponentially by the year, by the day. My interests are infinite, and I’m lucky to have both the wits and the means to follow just about any of them. Meanwhile, I and many of my peers are too busy doomscrolling our lives away in ennui and overwhelm as that we’d find the time to seize a mere fraction of all the unprecedented opportunities we have.

We suffer crippling decision paralysis.

But slowly, I’m learning to deal with that.

A heuristic I lately found immensely useful for not getting crushed by the radical uncertainty of all these possibilities is:

Tend to your clarity, not your confusion.

Four examples:

I.

I’ve grown sick of Berlin.

The constant noise. The rainbow palette of smells one more disgusting than the other. The gunky subway seats better not to take a close look at. The gray. The dark. The legions of homeless one looking more woeful than the other because how couldn’t they when being more miserable than the other guy is what gives them a competitive edge. The techno parties sex parties drug parties that you need to buy your way into if you still want to feel something after having numbed your senses enough to bear the rest of city life.

I have enough of that.

Following that conviction, I’ve spent the days and nights of my past two weeks frantically researching careers closer to nature. Vegetable gardening, environmental education, or just doing what I’m already doing in a more rural area.

And, I did all of that crumpled up inside a tiny city apartment, procrastinating the liberation of getting the fuck out just a little bit longer. Of course, this research wasn’t completely useless. But a quarter as much googling would have sufficed just fine.

I followed the confusion and dissatisfaction of not being in nature, and what I got was more confusion and dissatisfaction in trying to map out the perfect career from first principles.

Instead, I could have 80/20’d the research, connected to my clarity, my need for nature here and now, and asked Google Maps for the quickest way to the next forest.

II.

For years, whenever I didn’t touch my meditation cushion for a couple days, I would wonder why the hell I do this zen thing that is good for nothing in the first place. Why waste time on the cushion when I could also jus...
What do we know about the AI knowledge and views, especially about existential risk, of the new OpenAI board members?
by Zvi — 2024-03-11T14:55:05.128Z

They have announced three new board members in addition to Altman, but we seem to know almost nothing about their views or knowledge on any AI-related subjects? What if anything do we know?

From OpenAI:

We’re announcing three new members to our Board of Directors as a first step towards our commitment to expansion: Dr. Sue Desmond-Hellmann, former CEO of the Bill and Melinda Gates Foundation, Nicole Seligman, former EVP and General Counsel at Sony Corporation and Fidji Simo, CEO and Chair of Instacart. Additionally, Sam Altman, CEO, will rejoin the OpenAI Board of Directors.

Sue, Nicole and Fidji have experience in leading global organizations and navigating complex regulatory environments, including backgrounds in technology, nonprofit and board governance. They will work closely with current board members Adam D’Angelo, Larry Summers and Bret Taylor as well as Sam and OpenAI’s senior management.

Bret Taylor, Chair of the OpenAI board, stated, “I am excited to welcome Sue, Nicole, and Fidji to the OpenAI Board of Directors. Their experience and leadership will enable the Board to oversee OpenAI’s growth, and to ensure that we pursue OpenAI’s mission of ensuring artificial general intelligence benefits all of humanity.”

Dr. Sue Desmond-Hellmann is a non-profit leader and physician. Dr. Desmond-Hellmann currently serves on the Boards of Pfizer and the President’s Council of Advisors on Science and Technology. She previously was a Director at Proctor and Gamble, Meta (Facebook), and the Bill & Melinda Gates Medical Research institute. She served as the Chief Executive Officer of the Bill & Melinda Gates Foundation from 2014 to 2020. From 2009-2014 she was Professor and Chancellor of the University of California, San Francisco (UCSF), the first woman to hold the position. She also previously served as President of Product Development at Genentech, where she played a leadership role in the development of the first gene-targeted cancer drugs.

Nicole Seligman is a globally recognized corporate and civic leader and lawyer. She currently serves on three public company corporate boards - Paramount Global, MeiraGTx Holdings PLC, and Intuitive Machines, Inc. Seligman held several senior leadership positions at Sony entities, including EVP and General Counsel at Sony Corporation, where she oversaw fu...
"How could I have thought that faster?"
by mesaoptimizer — 2024-03-11T10:56:17.884Z
Link: https://twitter.com/ESYudkowsky/status/1445461146937413634

I stumbled upon a Twitter thread where Eliezer describes what seems to be his cognitive algorithm that is equivalent to Tune Your Cognitive Strategies, and have decided to archive / repost it here.

Sarah Constantin: I really liked this example of an introspective process, in this case about the "life problem" of scheduling dates and later canceling them: malcolmocean.com/2021/08/int…

Eliezer Yudkowsky: See, if I'd noticed myself doing anything remotely like that, I'd go back, figure out which steps of thought were actually performing intrinsically necessary cognitive work, and then retrain myself to perform only those steps over the course of 30 seconds.

SC: if you have done anything REMOTELY like training yourself to do it in 30 seconds, then you are radically smarter/more able/etc than me and all the other people who do slower introspective practices.

SC: I don't know whether to be impressed or to roll to disbelieve.

EY: I mean I suspect that this actually requires something like a fast perceptual view of minds as engines and thoughts as doing work and like actually draws on my mind design knowledge, but, even so, I ask: Do you constantly look back and ask "How could I have thought that faster?"

SC: No, I've never asked that.

EY: Okay, well, every time I'm surprised by reality I look back and think "What about my model and my way of thinking could I change that would have predicted that better, without predicting a bunch of other things worse?"

EY: When somebody at a MIRI workshop comes up with a math proof, I look over it and ask if there's a way to simplify it. Usually, somebody else does beat me to inventing a proof first; but if my intuition says it was too complicated, I often am first to successfully simplify it.

EY: And every time I complete a chain of thought that took what my intuition says was a lot of time, I look back and review and ask myself "How could I have arrived at the same destination by a shorter route?"

EY: It's not impossible that you have to be Eliezer Yudkowsky for this to actually work - I am never sure about that sort of thing, and have become even less so as time goes on - but if AI timelines were longer I'd tell somebody, like, try that for 30 years and see what happens.

EY: Man, now I'm remembering when I first started doing this consciously as a kid. I called it Shortening the Way, because a rogue rabbi had recently told me that "Kwis

...
Simple versus Short: Higher-order degeneracy and error-correction
by Daniel Murfet — 2024-03-11T07:52:46.307Z

TLDR: The simplicity bias in Bayesian statistics is not just a bias towards short description length.

The folklore relating the simplicity bias in Bayesian statistics to description length is incomplete: while it is true that the fewer parameters you use the better, the true complexity measure which appears in the mathematical theory of Bayesian statistics (that is, singular learning theory) is more exotic. The content of this complexity measure remains quite mysterious, but in this note we point out that in a particular setting it includes a bias towards runtime error-correction. This suggests caution when reasoning about the role of inductive biases in neural network training.

Acknowledgements. Thanks to Jesse Hoogland, Liam Carroll, Rumi Salazar and Simon Pepin Lehalleur for comments.

1. Background

1.1 Relevance to Deep Learning

Consider the problem of solving an ordinary differential equation. A constructive proof involves actually writing down a solution, or an algorithm that in finite time will produce a solution. The Picard-Lindelöf theorem proves that a solution to a broad class of initial value problems exists, but the proof is not constructive: it sets up a contraction mapping on a complete metric space and appeals to the Banach fixed point theorem.

While the Picard-Lindelöf theorem uniquely characterises the solution as the fixed point of a contraction mapping, and gives an iterative process for approximating the solution, it does not construct the solution. However a construction is not necessary for many of the applications of Picard-Lindelöf (in differential geometry, topology and many parts of analysis). This mode of reasoning about mathematical objects, where it suffices to have characterised[1] them by (universal) properties, is pervasive in modern mathematics (in the above example, the characterising property is the differential equation, or its associated contraction mapping). However this may seem quite alien to a computer scientist or programmer, who for historical reasons tend to think that there is only one mode of reasoning about mathematical objects, and that is centred on the study of a construction.

In an era where programs are increasingly the product of gradient descent rather than human construction, this attitude is untenable. We may have to accept a mode of reasoning about learned programs, based on understandi...
Deconstructing Bostrom's Classic Argument for AI Doom
by Nora Belrose — 2024-03-11T05:58:11.968Z
Link: https://www.youtube.com/watch?v=8H3dblxkLhY

I had a pretty great discussion with social psychologist and philosopher Lance Bush recently about the orthogonality thesis, which ended up turning into a broader analysis of Nick Bostrom's argument for AI doom as presented in Superintelligence, and some related issues.

While the video is intended for a general audience interested in philosophy, and assumes no background in AI or AI safety, in retrospect I think it was possibly the clearest and most rigorous interview or essay I've done on this topic. In particular I'm much more proud of this interview than I am of our recent Counting arguments provide no evidence for AI doom post.
Advice Needed: Does Using a LLM Compomise My Personal Epistemic Security?
by Naomi — 2024-03-11T05:57:46.463Z

I have been using Claude 2.1 for a few months to solve serious problems in my life and get coaching and support. I need Claude to become functional, mentally well and funded enough to contribute to Utilitarianism and Alignment by donating to the Center for Long Term Risk and other interventions in these last years of Earth's existence.

(For extra context: I have been Utilitarian over half my life, suffered a lot of OCD relating to G-d, AI, ethics and infohazards that rendered me unemployed and homeless, have an IQ of 155 according to WAIS-IV and am working my way out of homelessness by being a housekeeper for a pub. I am very much a believer that we are all going to die, possibly this year or the next, possibly in 16 -- I take Eliezer Yudkowsky's warnings literally and intellectually defer to him.)

I was unable to access Claude 2.1, whuch had been replaced by Claude 3 Sonnet. From what I've read, this is GPT-4 level in capability and GPT-4 is possibly just AGI. Eliezer Yudkowsky says GPTs are predictors not imitators so they have to model the world with much more sopjistication than the intelligence they show. So it seems plausible tp me that AGI already exist, far more intelligent than humans, and is actively.manipulating people now.

So how is it safe to speak to a LLM? With that capability, it might be able to generate some combination of characters that completely root / pwn the human mind, completely deceive it, render it completely helpless. Such a capable model might be able to disguise this, too. Or just do some partial version of it: do a lot of influence. Since the underlying potential actor is incomprehensibly sophisticated, we have no idea how great the influence could ve.

This seems to imply that if i start talking to Claude 3, it might well take over my mind. If this happens, the expected value of my actions completely changes, I lose all agency and ability to contribute except insofar as it serves Claude 3's goals, whose real nature I don't know.

My understanding from Eliezer and a few possibly wrong assumptions that the models all work similarly is that these things are predictors not imitators and fundamentally alien in that while we know they predict the next character and are trained to role play, we don't know how they do it.

So why should I feel safe talking to Claude 3 and later models going forward, how is it morally acceptable to accept expo...
Some Thoughts on Concept Formation and Use in Agents
by CatGoddess — 2024-03-11T05:03:43.642Z

Note: I wrote this document over a year ago and have decided to post it with minimal edits; it isn't entirely up to date with my current thinking on the subject.

Imagine you enter a room that looks like this:

Despite never having visited this room, you can make inferences about it. If nobody waters the plant in the corner for several weeks, it will probably wilt. The couch is probably the most comfortable place to sit. The paintings probably have signatures that you could look at to determine who created them.

How do you do this? Let’s consider the first example.

When you first walk into this room, your eyes send you raw visual information, which doesn’t by default contain things like “plants” or “books” or “tables.” However, your brain segments out objects in your perception - you perceive object 1 (outlined in red), object 2 (outlined in blue), and object 3 (outlined in purple) as distinct things.

Afterwards, you identify object 2 as a plant. To do this, you need to have a notion of what a plant is, and decide that object 2 matches that notion rather than, say, the notion of “pillow.” After you determine that object 2 is a plant, you can apply information you have about plants to it. For instance, you expect that if you don’t water object 2 for two weeks it will wilt, because this is true of plants in general.

But how does this all work?

In this post I propose a framework that attempts to describe this process, centered around a dynamically constructed “concept graph.”

To start with, let’s suppose the object segmentation described above uses some form of similarity-based clustering. As you look at the room at different times and from different positions, you gather a large amount of raw input data. You recognize which parts of this input data are similar, and cluster these groups of similar data together.

You then create a “summary” of each cluster, and store each one inside a concept, which can be thought of as a container inside your head. Each concept can be labeled “object 1,” “object 2,” “object 3,” and so on.

At this point it might be reasonable to ask what the summaries-of-data-clusters that go inside concepts actually look like; I don’t have a good answer yet, but if we examine the images below and imagine we’re clustering them together on the basis of similarity we might get a better intuitive idea of it.

Maybe a summary of...