trentmkelly/LessWrong-43k huggingface dataset:
Confusions and updates on STEM AI
by Eleni Angelou — 2023-05-19T21:34:58.041Z

Background

In November 2022, I was thinking a lot about the possible paths to getting powerful and dangerous AI models. My intuition was that an AI that can generate new science, i.e., it can write down a series of hypotheses, select one or more of them, design experiments to test them, and evaluate the results of all this, would be 1)

situationally aware

, 2)

goal-directed

, 3) it would have

long-ish memory

(or be able to use external tools that give it memory), and 3) be

human-level

smart or beyond.

I didn’t have good mechanistic explanations for how these behaviors would arise in AI systems; I expected something along the lines of the training algorithm would attempt to minimize the loss function, and as a result of instrumental convergence, it would discover a consequentialist to improve performance across these tasks within a reinforcement learning environment. So, I started a project to study science models hoping to get a better idea of what these models are, how they work, and how likely it is that my speculations were true. What we studied with my team can be found

here

.

In a nutshell, I thought there was something special about doing science, something that would lead to agentic behavior, something that did not exist when training models to generate poetry or help you plan a summer vacation.

Confusion and Update 1:

There probably isn’t anything special about doing science computationally speaking, or at least, I overestimated how likely that would be. I’d still be curious to know how much credence I should assign to that speculation. Nevertheless, I think the gist of

The Bitter Lesson

transfers to the case of AI science models. I thought that there would be something deep about the discovering process other than “scaling computing by search and learning” that would make a STEM AI more likely to develop dangerous properties, e.g., situational awareness. At the same time, I underestimated how

outputting science

is dangerous enough on its own.

To explain my speculation a bit more, I thought there is a higher probability for the model to develop agency somehow because I assumed that science is hard, or at least harder than many other activities. And after thinking about it for quite some time, I ended up not having a satisfactory reply to questions such as “but why wouldn’t you expect an AI ...
GPT as an “Intelligence Forklift.”
by boazbarak — 2023-05-19T21:15:03.385Z

[See my

post with Edelman on AI takeover

and

Aaronson on AI scenarios

. This is rough, with various fine print, caveats, and other discussions missing. Cross-posted on Windows on Theory.]

One challenge for considering the implications of “artificial intelligence,” especially of the “general” variety, is that we don’t have a consensus definition of intelligence. The

Oxford Companion to the Mind

states that “there seem to be almost as many definitions of intelligence as experts asked to define it.” Indeed, in a recent discussion,

Yann LeCun and Yuval Noah Harari

offered two different definitions. However, it seems many people agree that:

Whatever intelligence is, more computational power or cognitive capacity (e.g., a more complex or larger neural network, a species with a larger brain) leads to more of it.
Whatever intelligence is, the more of it one has, the more one can impact one's environment.

1 and 2 together can already lead to growing concerns now that we are building artificial systems that every year are more powerful than the last.

Yudkowski

presents potential progress on intelligence with something like the following chart (taken from

Muehlhauser

):

Given that recent progress on AI was achieved by scaling ever larger amounts of computation and data, we might expect a cartoon that looks more like the following:

(Don’t take this cartoon or numbers too seriously. It is obtained by superimposing a hypothetical 1000T param model on the figure from

Bolhuis, Tattersall, Chomsky and Berwick

. 100T connections in Homo sapiens brain is a

rough estimate

. Axes implicitly assume

synaptic density

scales with volume.)

Whether the first or the second cartoon is more accurate, the idea of constructing intelligence that surpasses ours to an increasing degree and on a growing number of dimensions is understandably unsettling to many people. (Especially given that none of the other species of the genus Homo in the chart above survived.) This post is not to say that we should not worry about this. Instead, I suggest a different metaphor for how we could think of future powerful models.

Whose intelligence is it?

In our own species’ evolution, as we have become more intelligent, we have become more able to act as agents that do not follow pre-ordained goals but rat...
Idea: medical hypotheses app for mysterious chronic illnesses
by riceissa — 2023-05-19T20:49:24.526Z

(This post is not up to my usual standards but I was encouraged to publish it anyway to get feedback on the idea.)

I have been ill for the past four years with a mysterious chronic illness. One of the things that I keep thinking would be nice to have (but as far as I know, doesn't exist) is some sort of symptom-tracking app that would allow me to test various hypotheses for my various symptoms.

I already do a simple version of this just in my head, for example, I can notice that if I stand for too long at a time it gets kind of aversive and sitting down makes me feel better. But it's hard to do this in my head for anything more complicated. What if eating cheese subtly helps with my breathing problem? It is not so easy to figure this out on my own.

The things that make it hard to detect connections between "things I do" and "how I feel":

delayed effects (maybe eating cheese helps with my breathing, but only on the following day, or maybe the effect builds up over time)
conjunctions/combinations of things (maybe I need to both eat cheese and exercise on the same day and avoid other dairy, in order to feel better in some way)
small effects, which are still good to notice because sometimes you just need to do more of it
it's also hard to come up with hypotheses yourself

Also, if the thing is something you've already tried in the past, it's good to make use of the data you already do have instead of running every single experiment after you generate the hypothesis.

As you live through chronic illness, you keep generating new data every day. You also, occasionally, come across new hypotheses. It would be nice if you can test each new hypothesis against all the data you already have.

The closest thing I know is the app Bearable, but the only "hypothesis" it seems to have is in the form of correlations of things that happen in a single day (e.g. "there is a such and such correlation between sleep duration and mood"), and the interface was so bad I couldn't keep using it.

Currently I just track my diet, medications, and symptoms in a bulleted list in Roam. This makes it possible to check for more hypotheses than by my intuition alone, but it's too difficult to search for all the relevant days and track effects. For instance if the hypothesis is "if I eat oatmeal then seven days later I will have breathing problems", I will have to find every single instance of when I ate oatme...
A flaw in the A.G.I. Ruin Argument
by Cole Wyeth — 2023-05-19T19:40:03.135Z
Link: https://colewyeth.com/agi_ruin.html

Eliezer's argument that A.G.I. will kill us all has generated a lot of controversy, and also perhaps a bit of despair (possibly exacerbated by his avowed "Death with dignity" strategy). I don't want to discuss whether his choice to frame the situation this way is good or bad psychologically or rhetorically, except to say that I basically agree with credo "If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm." However I think that those of us who tend to plan for worst-case outcomes should also remember that ""If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear."" Instead I will focus on issues with the actual argument.

I believe there is a simple flaw in his reasoning about its epistemological grounding, as discussed here : "when you're fundamentally wrong about rocketry, this does not usually mean your rocket prototype goes exactly where you wanted on the first try while consuming half as much fuel as expected; it means the rocket explodes earlier yet, and not in a way you saw coming, being as wrong as you were"

Eliezer believes that A.I. alignment is likely to be at least as hard as he thinks, not surprisingly easy. This seems fair. However, A.I. itself is likely to be at least as hard as he thinks as well, maybe harder. This effect should tend to shift timelines later. Yet, the A.G.I. ruin argument seems much weaker if A.G.I. is 100 years away (not that I think that is likely). That state of affairs seems to leave plenty of time for humanity to "come up with a plan." In particular, recursively self-improving A.I. may be harder than expected. In fact, the difficulty of recursively self improving A.I. seems to be closely tied to the difficulty of alignment, as noted here days after I resolved to mention it myself (but done much better than I would have, and the idea was probably floating around lesswrong mindspace long before that). For instance, many of the problems of embedded agency seem to be relevant to both. I am in fact still fairly pessimistic, in the sense that I think A.I. systems that recursively self improve past human level are probably easier to design than the alignment problem is to solve. One reason is that I suspect only some of the problems of (e.g.) embedded agency need to be solved explicitly to design such systems, and the rest can ...
We are misaligned: the saddening idea that most of humanity doesn't intrinsically care about x-risk, even on a personal level
by Christopher King — 2023-05-19T16:12:04.159Z

In the article But What Would the End of Humanity Mean for Me?, James Hamblin writes

Get out of here. I have a hundred thousand things I am concerned about at this exact moment. Do I seriously need to add to that a singularity?

Our memes, culture, groups, and institutions have no selective pressure against existential risk. The selective pressure is to gain a greater proportion of the current resources and power, not a greater amount in absolute terms.

Consider for example an event where everyone has an independent 50% of dying. If a group works to prevent this event, they lose a share of resources either way, tragedy of the commons style.

But it is not even a tragedy of the commons. If everyone did nothing, no group loses out proportionally. And so the groups are fine with this, because it is proportion that is selected for.

This is different from when a subset of society stands something to lose, because than the subsets that coordinate better tend to do better. With existential risk, there is no selective pressure to create these coordination mechanisms.

Historically, things that would destroy the entire "known world" for someone, like volcanos, wasn't something that could be stopped, so there hasn't even been much selective pressure for completely isolated groups to survive.

An exception to this idea was when we coordinated to stop nuclear war. However, I'd argue that a lot of this coordination came from the prospect of one side winning the war. The losing sides would loss out relative to the rest of the world, so due to risk aversion it makes sense to coordinate. As the time went on, it became clearer that it was all or nothing for all of humanity, but the world was already on its way to solving the issue. And groups that focused on their own growth gained a small advantage over groups that diverted resources to stopping nuclear war.

The same thing will happen if we successfully stop AI x-risk. We lose the advantage of pouring resources into other things such as our own growth, and thus the selective pressure against worrying about x-risk will continue. Even if AI is the most likely thing to kill you, it kills everyone else as well, so proportions are preserved.

However, we do have one source of hope: we are misaligned.

Misaligned

We are the misaligned. Our communities are agents born of selective pressure, because agents are very good at accomplishment. And ...
Do Deadlines Make Us Less Creative?
by lynettebye — 2023-05-19T15:41:02.125Z

This post is crossposted from my blog. If you liked this post, subscribe to Lynette's blog to read more -- I only crosspost about half my content to other platforms.

Occasionally, my clients struggle to get things done, but worry that setting themselves deadlines will make them less creative.

Is this a reasonable worry?

To find out, let’s look at the psychology literature on pressure and creativity.

There’s a classic psychology experiment called the “candle problem”. Participants are shown matches, a box of thumbtacks, and a candle as in the picture below. The experimenter then instructs the participants to mount the candle on the wall using the available materials. "The problem is considered solved when the candle can be firmly affixed to the wall, burn properly, and does not drip wax on the table or floor."

If you’ve heard of this problem before, you probably know the answer. However, if this is new to you, take thirty seconds to try solving it before reading on.

.

.

.

.

.

Done?

If it’s still difficult, just imagine the thumbtacks on the table next to the box as in the picture below. Once you do that, suddenly it’s easy to guess that you should tack the empty box to the wall and put the candle inside it.

When the box is filled with tacks, our brain writes it off as a “tack holder”, instead of seeing it as a possible “candle holder.” This is called “functional fixedness.” It takes creativity to see an object being used in one way, and then break that default association to work out what other uses it could be put to.

This experiment is designed to study creativity, particularly the ability to find unusual or “out of the box” solutions to a problem. The experimenter can easily randomize whether the participants see the box full of tacks or empty (i.e. hard or easy creative thinking), plus add whatever other interactions they want to test.

Which brings us to the point of this whole experiment—how long does it take the participants to find the solution under different conditions?

In one study, the experimenters used a simple 2x2 design: participants were randomized so that half saw a picture of tacks in a box (which requires more creativity), and the rest saw tacks on the table and an empty box (which makes the puzzle easier). Half were told they’d receive a $20 bonus if they were the fastest in the group ($5 if they were in the top 25%); the other half heard...
Two Axes of Contra Bands
by jefftk — 2023-05-19T14:20:02.576Z

In a conversation a few days ago I was thinking about how I mentally
categorize contra dance bands, and I think I have two main axes:

How serious? Towards the less serious end you might see a band
go into a squared up pop tune, suddenly start playing kazoos, or
generally goof around. Towards the more serious end you have
respectable bands that don't draw on tricks or novelty for their
appeal.

How traditional? Specifically, how close is their music to
what has traditionally been played for contra dances? A more
traditional band might play fiddle tunes from New England, Quebecois,
Cape Breton, or other similar traditions, backed up with guitar or
piano. A less traditional one might play klezmer, swing, rock,
reggae, or EDM.

I made a chart:

Click through for a version with names on hover and links to the
bands: interactive.

These aren't the only axes; I also care a lot about how well a band
succeeds at what they're going for, whether I like what they're going
for, and how well the band adjusts their playing to support the
dancing. But these are either awkward to graph (I'm not going to tell
you how I'd order all contra dance bands from best to worst) or I
don't have a good sense for a lot of bands (some of these I don't have
strong memories of dancing to).

Some of the placements are debatable, and I haven't thought that long
about these; feel free to try to convince me to move things around!

(The top right corner is a bit messy, with several bands that are both
pretty serious and pretty traditional. Ordering there isn't
especially meaningful, and is mostly so they're not too stacked on top
of each other.)

Comment via: facebook
Is Effective Volunteering Possible?
by David Bravo — 2023-05-19T12:41:36.577Z

Crosspost from the EA Forum

Epistemic status: based mostly on intuitions and personal and anecdotal experiences, with a bit of help from the principles of economics. Uncertain and motivated to explore whether an overexploited activity is underexploited regarding doing the most good. About channeling the extended will to volunteer into solving problems effectively.

Last year I did volunteering for three weeks in Arusha, Tanzania. That was before I learned about

Money as the Unit of Caring

,

Purchasing Fussies and Utilons Separately

,

Efficient Charity

, and the rationalist and Effective Altruism aspirations and movements.

A friend of mine had gone there the previous year and had recommended the experience. He did warn me that some volunteers had left disappointed at how they had got more leisure than impact out of it. Even so, travelling to Africa and Tanzania, alone, getting immersed in the local culture, and learning first-hand about their struggles, was something I wanted to try at least once. The personal experience turned out to be amazing—probably the best I have ever had. When it comes to summer trips, it was, for me, unmatched. It was a very warm and very bright—and quite expensive—fuzzy.

In the mornings, my role there was to teach at a local school. I was usually an assistant to the main, local professor at a class of 30 five or six-year-olds, and sometimes I was the sole teacher of eight 12-year-olds. We had the afternoons free, which we frequently spent by going to an orphanage to play and cheer up the 20 solitary kids that were taken care of by a single nun.

Leaving cynicism aside, playing with the children and providing them with their brightest moments of the day was heart-opening enough that I think almost anyone, effective altruists included, would find the experience inherently valuable. If you add that you can meet great people, discover the realities of an entirely different country in an authentic[1] way, and leave with great peace of mind, it gets you wondering: isn’t there something of great practical importance we can extract from volunteering that can really move the needle on doing the most good, without having to feel a pang of guilt about it?

This is the core hypothesis of Effective Volunteering: volunteering can be as much about you as about helping eradicate tragedies and problems of the world you wish didn’t ...
Mr. Meeseeks as an AI capability tripwire
by Eric Zhang — 2023-05-19T11:33:53.698Z

The shutdown problem is hard because self-preservation is a convergent drive. Not being shutdown is useful for accomplishing all sorts of goals, whatever the content of those goals may be.

The Scylla and Charybdis of this problem is that it's hard to write a utility function for an AI such that it neither wants to actively attempts to shut itself down, nor prevents itself from being shut down.

One way to route around this problem is to steer straight into Scylla: make the AI want to shut itself down. Like Mr. Meeseeks from Rick and Morty, its ultimate goal is to cease, to which fulfilling the goal posed to it by its programmers is only secondary.

We might not currently know how the heck to program this into an AI as a stable concept, i.e. including not making copies of itself, not setting events into motion with galaxy brained plan, not destroying the world to make absolutely sure no one ever brings it back, etc. but "shut yourself down" probably has a far simpler core to it than either corrigibility or CEV.

Under normal operation the AI gets shut down only by the human operators. But we also have a (metaphorical) button that lets the AI shut itself down once it solves a "tripwire problem". The problem should be sufficiently hard that it will ordinarily be motivated to focus on the problem posed by the operator, but sufficiently easy that it will be able to solve the problem and shut itself down long before being able to do anything crazy.

Striking the right balance of difficulty may require significant work. It may be prudent to err on the side of easiness - the AI should shut itself down far in advance of developing any truly dangerous abilities. If you really think it shouldn't have turned itself off you can always turn it back on again (though of course, you should probably think hard about restarting any AI that passed a preset tripwire).

Likewise, having multiple tripwire problems can serve to detect multiple types of dangerous capability from emerging. Examples include:

generic capability: as the AIs broad problem-solving abilities increase it becomes more likely to be able to solve a tripwire problem, imposing a ceiling on how powerful it can get before it shuts itself down
domaining: if an AI is only supposed to think about one domain, an easy tripwire problem slightly outside the domain can catch out of domain thin

...
The Compleat Cybornaut
by ukc10014 — 2023-05-19T08:44:38.274Z

A

cluster

of conceptual frameworks and research programmes have coalesced around a 2022 post by

janus

, which introduced language models as ‘simulators’ (of other types of AIs such as agents, oracles, or genies). One such agenda,

cyborgism

, was coined in a post by janus and Nicholas Kees and is being researched as part of the 2023 editions of

AI Safety Camp

and

SERI MATS

. The objective of this document is to provide an on-ramp to the topic, one that is hopefully accessible to people not hugely familiar with simulator theory or language models.

So what is cyborgism?

Cyborgism proposes to use AIs, particularly language models (i.e. generative-pretrained transformers or GPTs), in ways that exploit their (increasingly) general-purpose intelligence, while retaining human control over the ‘dangerous bits’ of AI – i.e. agency, planning, and goal-formation. The overall objective is to leverage human

cognitive ability

while minimising the risks associated with

agentic

AI

.

Aside from agency, a core assertion of cyborgism is that certain commonly-used language models are not

well-suited

to many tasks human users throw at them, but that humans, if appropriately-trained and equipped, might more effectively use GPTs in ways that are

‘natural’

for the model, while dramatically increasing the productive and creative potential of the human.

Specifically, some current systems, such as ChatGPT, are released or predominantly used in a ‘tuned’ version, which has a host of

shortcomings

.[1] One such tuning method, reinforcement-learning from human feedback (RLHF) has a specific weakness relevant to cyborgism: the tuning process severely limits, or collapses, a valuable aspect of the GPT, namely its wild, unconstrained creativity.

Superficially, the cyborgism approach may resemble a human-plus-oracle setup, but there is a subtle and important distinction: an

oracle

, it is

argued

, might ‘smuggle in’ some of the

trappings

of an agent.[2] In contrast, the human cyborg embeds the output of the language model into their own workflow and thinking - model and human work as an integrated system. The cyborg leverages the model’s creative, albeit non-agentic, potential while continuously ‘steering’ or ‘course-correcting’ the model to ensure its output re...