trentmkelly/LessWrong-43k huggingface dataset:
Is this the beginning of the end for LLMS [as the royal road to AGI, whatever that is]?
by Bill Benzon — 2023-08-24T14:50:19.312Z

It’s hard to tell, but it sure is...shall we say...interesting.

Back in the summer of 2020 when GPT-3 was unveiled I wrote a working paper, GPT-3: Waterloo or Rubicon? Here be Dragons. My objective was to convince myself that the underlying technology wasn’t just some weird statistical fluke, that there was in fact something going on of substantial interest and value. To my mind, I succeeded in that. But I was skeptical as well.

Here's what I put on the first page of that working paper, even before the abstract:

GPT-3 is a significant achievement.

But I fear the community that has created it may, like other communities have done before – machine translation in the mid-1960s, symbolic computing in the mid-1980s, triumphantly walk over the edge of a cliff and find itself standing proudly in mid-air.

This is not necessary and certainly not inevitable.

A great deal has been written about GPTs and transformers more generally, both in the technical literature and in commentary of various levels of sophistication. I have read only a small portion of this. But nothing I have read indicates any interest in the nature of language or mind. Interest seems relegated to the GPT engine itself. And yet the product of that engine, a language model, is opaque. I believe that, if we are to move to a level of accomplishment beyond what has been exhibited to date, we must understand what that engine is doing so that we may gain control over it. We must think about the nature of language and of the mind.

I didn’t expect that anyone with any influence in these matters would pay any attention to me – though one can always hope – but that’s no reason not to write.

That was 2020 and GPT-3. Two years later ChatGPT was launched to great acclaim, and justly so. I certainly spent a great deal of time playing with, investigating it, and writing about it. But I didn’t forget my cautionary remarks from 2020.

Now we’re hearing rumblings that things aren’t working out so well. Back on August 12 the ever skeptical Gary Marcus posted, What if Generative AI turned out to be a Dud? Some possible economic and geopolitical implications. His first two paragraphs:

With the possible exception of the quick to rise and quick to fall alleged room-temperature superconductor LK-99, few things I have ever seen have been more hyped than generative AI. Valuations for many companies are in the billions, coverage in the

...
AI Safety Bounties
by PatrickL — 2023-08-24T14:29:59.103Z
Link: https://rethinkpriorities.org/publications/ai-safety-bounties

Earlier this year, Vaniver recommended Bug Bounties for Advanced ML Systems.
I spent a while at Rethink Priorities considering and expanding on this idea, suggesting potential program models, and assessing the benefits and risks of programs like this, which I've called 'AI Safety Bounties':

Short summary

AI safety bounties are programs where public participants or approved security researchers receive rewards for identifying issues within powerful ML systems (analogous to bug bounties in cybersecurity). Safety bounties could be valuable for legitimizing examples of AI risks, bringing more talent to stress-test systems, and identifying common attack vectors.

I expect safety bounties to be worth trialing for organizations working on reducing catastrophic AI risks. Traditional bug bounties seem fairly successful: they attract roughly one participant per $50 of prize money, and have become increasingly popular with software firms over time. The most analogous program for AI systems led to relatively few useful examples compared to other stress-testing methods, but one knowledgeable interviewee suggested that future programs could be significantly improved.

However, I am not confident that bounties will continue to be net-positive as AI capabilities advance. At some point, I think the accident risk and harmful knowledge proliferation from open sourcing stress-testing may outweigh the benefits of bounties

In my view, the most promising structure for such a program is a third party defining dangerous capability thresholds (“evals”) and providing rewards for hunters who expose behaviors which cross these thresholds. I expect trialing such a program to cost up to $500k if well-resourced, and to take four months of operational and researcher time from safety-focused people.

I also suggest two formats for lab-run bounties: open contests with subjective prize criteria decided on by a panel of judges, and private invitations for trusted bug hunters to test their internal systems.

Author's note: This report was written between January and June 2023. Since then, safety bounties have become a more well-established part of the AI ecosystem, which I'm excited to see. Beyond defining and proposing safety bounties as a general intervention, I hope this report can provide useful analyses and design suggestions for readers already interested in implementing safety bounties, or in better ...
AI Regulation May Be More Important Than AI Alignment For Existential Safety
by otto.barten — 2023-08-24T11:41:54.690Z

Summary: Aligning a single powerful AI is not enough: we're only safe if no-one, ever, can build an unaligned powerful AI. Yudkowsky tried to solve this with the pivotal act: the first aligned AI does something (such as melting all GPUs) which makes sure no unaligned AIs can ever get built, by anyone. However, the labs are currently apparently not aiming to implement a pivotal act. That means that aligning an AGI, while creating lots of value, would not reduce existential risk. Instead, global hardware/data regulation is what's needed to reduce existential risk. Therefore, those aiming to reduce AI existential risk should focus on AI Regulation, rather than on AI Alignment.

Epistemic status: I’ve been thinking about this for a few years, while working professionally on x-risk reduction. I think I know most literature on the topic. I have also discussed the topic with a fair number of experts (who in some cases seemed to agree, and in other cases did not seem to agree).

Thanks to David Krueger, Matthijs Maas, Roman Yampolskiy, Tim Bakker, Ruben Dieleman, and Alex van der Meer for helpful conversations, comments, and/or feedback. These people do not necessarily share the views expressed in this post.

This post is mostly about AI x-risk caused by a take-over. It may or may not be valid for

other types of AI x-risks

. This post is mostly about the ‘end game’ of AI existential risk, not about intermediate states.

AI existential risk is an evolutionary problem. As Eliezer Yudkowsky and others have pointed out: even if there are safe AIs, those are irrelevant, since they will not prevent others from building dangerous AIs. Examples of safe AIs could be oracles or satisficers,

insofar

as

it turns out to be possible to combine these AI types with high intelligence. But, as Yudkowsky would

put it

: “if all you need is an object that doesn't do dangerous things, you could try a sponge”. Even if a limited AI would be a safe AI, it would not reduce AI existential risk. This is because at some point, someone would create an AI with an unbounded goal (create as many paperclips as possible, predict the next word in the sentence with unlimited accuracy, etc.). This is the AI that would kill us, not the safe one.

This is the evolutionary nature of the AI existential risk problem. It is described excellently by Anthony Berglas in his underrated

book

, and more...
AI Probability Trees - Katja Grace
by Nathan Young — 2023-08-24T09:45:47.487Z

I am interviewing AI experts on what they think will happen with AI. Here is Katja Grace and her thoughts. AI risk scares me but often I feel pretty disconnected from it. This has helped me think about it.

Here are Katja’s thoughts in brief:

Will we get AI tools that can run a civil service department with 100x fewer people by 2040? 50%
In what fraction of worlds are the majority of AI tools controlled by agentic AI? 70%
In what fraction of worlds will agentic AI achieve their goals? 90%
In what fraction of agentic worlds will the majority of AI controlled resources be implemented for bad goals? 50%
Will policy slow AGI down by 10 years? 25%, How much reduction of badness does 10 years buy us? 30%

You can look at an interactive graph here:

or see all the graphs I’ve done so far

https://estimaker.app/ai

. There is an image at the bottom of this page.

You can watch the full video here:

Longer explanations

Please remember this is by 2040. This was originally a table

Question

Katja's Commentary
My commentary

Will we get AI tools that can run a civil service department with 100x fewer people by 2040? (50%)

Katja's commentary: Katja says she’s unsure and gives 50% accordingly.
My commentary:

A common failure mode in my discussions about AGI is how vague the capabilities that make “AGI” are. Some people are envisaging tools capable of flawlessly running worldwide conspiracies while others seem to think of top human level coding ability.
Katja and I settled on this example because it involves a varied set of tasks at scale and replacing a lot of human labour.
The tasks include:

Large scale engineering projects
Cash transfers
Real world disagreements (how to set tax rates, etc)
Complex system problems (traffic flows, healthcare takeup)

Civil Services act on a scale of millions of people.

In what fraction of those worlds does agentic ever control the majority of resources (even via AI tools) (70%) If you add up the resources controlled by agentic AI and the AI tools they control, is that more than half?

Katja's commentary:

Katja’s intuition is that for extinction, agency is probably necessary.
“The reason I think that some of it is probably agentic is that it seems like agents or something roughly agent-like are economically valuable. People want to make things like agents. They are already trying to make things like agents.”
“I

...
What wiki-editing features would make you use the LessWrong wiki more?
by Nathan Young — 2023-08-24T09:22:01.300Z

The LessWrong wiki does not seem as well used as it should be. I guess this is a lack of editors.

@Vladimir_Nesov made a good point that many standard wiki-editing features are missing which makes the prospect unappealing.

The point is that absence of the feature makes engagement with the wiki less promising, as it becomes inconvenient and hence infeasible in practice to protect it in detail, and so less appealing to invest effort in it. I mentioned that as a hypothesis for explaining currently near-absent editor engagement

So in the spirit of this, what features would cause you personally to edit a lot more than you currently do?

I may try and pay some devs to write the pull requests if they seem feasible.
On Lucidity
by Leber — 2023-08-24T08:45:40.129Z
Link: https://leber.substack.com/p/on-lucidity

I wrote a post about the trait of Lucidity, which is something I've been thinking about for a while now. I think it will be of particular interest to this crowd, and would love to hear your thoughts on the concept and how it might relate to other LW aspects.

Specifically, I would love to hear:

1. Whether you concur that such a distinct trait exists?

2. How does Lucidity overlaps and interact with with other aspects like rationalism or post-rat?

3. What are some other hallmarks of Lucidity?

I'd also really appreciate any general tips on writing as I begin my foray into publicly writing! Thanks in advance.
The God of Humanity, and the God of the Robot Utilitarians
by Raemon — 2023-08-24T08:27:57.396Z

My personal religion involves two gods – the god of humanity (who I sometimes call "Humo") and the god of the robot utilitarians (who I sometimes call "Robutil").

When I'm facing a moral crisis, I query my shoulder-Humo and my shoulder-Robutil for their thoughts. Sometimes they say the same thing, and there's no real crisis. For example, some naive young EAs try to be utility monks, donate all their money, never take breaks, only do productive things... but Robutil and Humo both agree that quality intellectual world requires slack and psychological health. (Both to handle crises and to notice subtle things, which you might need, even in emergencies)

If you're an aspiring effective altruist, you should definitely at least be doing all the things that Humo and Robutil agree on. (i.e. get to to the middle point of Tyler Alterman's story here).

But Humo and Robutil in fact disagree on some things, and disagree on emphasis.

They disagree on how much effort you should spend to avoid accidentally recruiting people you don't have much use for.

They disagree on how many people it's acceptable to accidentally fuck up psychologically, while you experiment with new programs to empower and/or recruit them.

They disagree on how hard to push yourself to grow better/stronger/wiser/faster, and how much you should sacrifice to do so.

Humo and Robutil each struggle to understand things differently. Robutil eventually acknowledges you need Slack, but it didn't occur to him initially. His understanding was born in the burnout and tunnel-vision of thousands of young idealists, and Humo eventually (patiently, kindly) saying "I told you so." (Robutil responds "but you didn't provide any arguments about how that maximized utility!". Humo responds "but I said it was obviously unhealthy!" Robutil says "wtf does 'unhealthy' even mean? taboo unhealthy!")

It took Robutil longer still to consider that perhaps humans (with their current self-awareness) not only need to prioritize their own wellbeing and your friendships, but that it can be valuable to prioritize them for their own sake, not just as part of a utilitarian calculus, because trying to justify them in utilitarian terms may be a subtly wrong step in the dance that leaves them hollow, burned out for years

(Though Robutil notes that this is likely a temporary state of affairs. A human with sufficiently nuanced self-knowl...
I measure Google's MusicLM over 3 months as it appears to go from jaw-dropping to embarrassingly repeating itself
by AttentionResearcher — 2023-08-24T04:20:46.553Z

Google Research submitted a paper on January 26 2023 for MusicLM, a mind-bogglingly incredibly powerful AI model that converts user text prompts into music.

https://google-research.github.io/seanet/musiclm/examples/

On May 10th, Google Research released to the public a waitlist that allows applicants to try it out. In about 6 seconds, it returns to the user 2 songs each 20 seconds long.

https://blog.google/technology/ai/musiclm-google-ai-test-kitchen/

The music I have gotten it to make is beyond and dreamy. It is often human level to me. It is incredibly fast to make it release my dreams. I could easily make a 20 minute long track get incredibly complex and advanced if they let us extend it from X seconds in.

I have been testing its limits and abilities since May 10th up to today just over 3 months later. After about 1 month in, I noticed the AI was losing many abilities, and the same prompts were making noticeably different music, which made it clear to me something might be changing for the worse. By 2 months in or so the outputs simply were no longer intelligent and advanced and jaw dropping, as if the AI was an old-school model which weren't very good back in the day. 3 months in now I can see this truly appears to be happening, because most songs just repeat a short 2 or 4 second beat now, without doing much at all. It's embarrassingly horrible right now. Except for a few prompts, a few still somewhat work, some feel like 2 months in quality, some 1 month in. I'd say very few if any are 0 months in level. I feel like a lost my pet dog, or a really good dream I once had. I wish they never changed the model. I saved all my dreamy tests, though there is a few harder prompts I just came up with that I want to document but now can't. It says come back in 30 days (oddly, just when MusicLM would get noticeably worse) after some 100 prompts, but this can be bypassed by coming back the next day.

My early tests. The rest on the 1st SoundCloud are good too:

https://www.reddit.com/r/singularity/comments/13h0zyy/i_really_crank_out_music_tracks_with_musiclm_this/

My 2nd SoundCloud has both the 3-months-in tests and more early tests I had.

https://soundcloud.com/steven-aiello-992033649/sets

The problem I believe starts when MusicLM asks the user to choose between the 2 generated songs which they like more by giving it a trophy to improve the model....
Enhancing Corrigibility in AI Systems through Robust Feedback Loops
by Justausername — 2023-08-24T03:53:41.469Z

Abstract

This article explores the concept of corrigibility in artificial intelligence and proposes a detailed framework for a robust feedback loop to enhance corrigibility. The ability to continuously learn and correct errors is critical for safe and beneficial AI, but developing corrigible systems comes with significant technical and ethical challenges. The feedback loop outlined involves gathering user input, interpreting feedback contextually, enabling AI actions and learning, confirming changes, and iterative improvement. The article analyzes potential limitations of this approach and provides detailed examples of implementation methods using advanced natural language processing, reinforcement learning, and adversarial training techniques. It emphasizes the need for thoughtful design and testing to mitigate risks and biases. Fostering corrigibility is essential for aligning AI systems with human values.

Introduction

As AI systems continue to permeate every sector of society, their potential to shape our lives and influence decision-making processes has received intense scrutiny. One crucial aspect of AI safety and ethics is corrigibility — the ability of an AI system to accept and respond to corrections from its users or operators. This article proposes a robust feedback loop as a promising strategy to enhance AI corrigibility.

Implementing a Robust Feedback Loop

The idea behind implementing a robust feedback loop is to create a dynamic and interactive environment where users and the AI system can learn from each other. This enables the system to align more closely with the user’s values, expectations, and instructions. The proposed feedback loop consists of five steps:

1. User Feedback

Allowing users to provide feedback on the AI’s outputs is the foundation of this model. The feedback could be regarding factual inaccuracies, misconceptions of context, or any actions that violate the user’s values or expectations. The system should be equipped with a simple, intuitive interface to facilitate this.

2. Feedback Interpretation

The AI system should interpret the feedback considering the appropriate context. It should not limit its understanding to immediate corrections but also infer the larger implications for similar future situations. Advanced natural language processing techniques and contextual understanding algorithms can be used to achieve this.

3. Action and

...
The lost millennium
by Ege Erdil — 2023-08-24T03:48:40.035Z

Note: This post is speculative, and you should take the claims in it with a grain of salt.

Throughout human history, we know that economic growth, for which population growth was a good proxy under Malthusian conditions, has accelerated many times. Indeed, we can see this just by extrapolating current growth rates backward: given that there are only so many people in the world and they are only at a finite multiple of subsistence income per person, it would not be possible for economic growth rates on the order of 3% per year to have lasted for more than a thousand years or so.

Though it's difficult to do so, people have also come up with estimates for exactly how this growth happened in the past. One such estimate is from the HYDE dataset, also available at Our World In Data; and a whole collection of other estimates can be found here.

Let's examine the HYDE dataset first. The data before 1 AD is given per millennium, and looking at the growth rates gives us the table below:

Millennium
Mean annual population growth rate (percent)

10000 BC - 9000 BC
0.0236%

9000 BC - 8000 BC
0.0254%

8000 BC - 7000 BC
0.0279%

7000 BC - 6000 BC
0.0320%

6000 BC - 5000 BC
0.0368%

5000 BC - 4000 BC
0.0411%

4000 BC - 3000 BC
0.0435%

3000 BC - 2000 BC
0.0489%

2000 BC - 1000 BC
0.0415%

1000 BC - 1 AD
0.0746%

Aside from a slight reversal in the second millennium BC[1], the estimates suggest a gradual acceleration in the growth rate of population. This would be consistent with e.g. a hyperbolic model for population growth, but here I don't want to make any such assumption and just look at the raw data.

What is surprising, then, is the next entry that should be in this table: in the first millennium, 1 AD - 1000 AD, population growth is estimated to have averaged a mere 0.033%/year. In other words, according to this dataset, even the fifth millennium BC had faster population growth than the first millennium!

Something like this result turns out to be robust to changing the dataset we're using. McEvedy and Jones report a mean growth rate of 0.044%/year for the first millennium, compared to 0.137%/year from 1000 BC to 200 BC and 0.062%/year from 200 BC to 1 AD. This more granular data in the first millennium BC, not available in the HYDE dataset, suggests that the slowdown in population growth likely started before the first millennium. To find a millennium with sl...
Regreasing a KitchenAid Mixer
by jefftk — 2023-08-24T02:30:04.121Z

I recently got an old KitchenAid mixer from a friend. It's a k45ss,
from maybe the early 80s, and the general advice I see is that with
typical home use they should be regreased every ~10y. I ordered 4oz of
food-safe synthetic

grease and followed
Mr. Mixer's YouTube guide (parts

1,

2,

3). With my
phone playing the videos I followed along: opening the mixer up,
removing the old grease, adding the new grease, and putting it back
together. There were slight differences (my strain relief was
annoying, and I didn't have a

pin
inside the gear assembly and needed to remove the worm assembly first
instead) but it was pretty close.

The planetary gear assembly was pretty low on grease, and the grease
there was a bit dry:

The upper area had more grease, though the grease was a harder
than I expect it was supposed to be:

The gears were in good shape, and were nice and solid feeling.

Degreasing, regreasing, and assembly went fine. Probably 1.5hr all
together? When I was done, however, I realized I had a part left over:
one of three washers.

I looked back over the video, and it talked about installing two of
these washers, which I remembered doing, but I was stumped by how I
ended up with a third. There wasn't any logical place which should
have a washer but didn't.

I talked about this at dinner, and one of my housemates said they also
had an old KitchenAid that had probably never been regreased (k45,
cord isn't grounded, maybe late 60s). We decided to do theirs
together, and maybe we'd see where the washer went.

The combination of having a second pair of hands and having just done
one the previous evening made it go a lot faster: less than an
hour. Very similar to mine, except the strain relief was even more
annoying. We did figure out the washer mystery, though: there are two
washers in the planetary mechanism, stacked on top of each other.

I'm not sure whether it's worth opening the bottom of mine back up to
install the extra washer. I'm leaning towards yes (they wouldn't put
in two washers for no reason, doesn't require taking the whole thing
apart) but possibly I don't really need to?