trentmkelly/LessWrong-43k huggingface dataset:
Unaligned stable loops emerge at scale
by Michael Tontchev — 2023-04-06T02:15:08.958Z

-- Reality loves self-propagating patterns --

Suppose that the company AllPenAI made an AI system that behaves according to its alignment 99.999% of the time, based on testing. That's pretty good, and it's certainly better than, for example, most Azure SLAs. If I were to argue to my manager that I need to spend half a year to improve such an SLA, they would kindly ask me to reassess my prioritization - for almost any project at any tech company.

So AllPenAI ends up shipping the product.

Developers get their hands on it, and find that it's very useful - maybe as useful as GPT-5. It's an idea-generation machine, and it can do chain-of-thought reasoning hundreds of steps deep!

Of course, having it execute one prompt is great, but this makes prompting the bottleneck for productivity, and humans are relatively slow at typing prompts. Thankfully, prompting is a language-generation task, and our system also has a good ability to brainstorm, break up tasks, and plan.

So we place it in a loop where it generates output, and then continues the "conversation" with itself with that output over and over.

-- Reality loves self-propagating patterns --

It works really well in a loop. Not perfectly, of course, but it's able to reliably take a big problem we give it, break it down into components, and fill in the details of those components with ReAct-style thinking until it decides that the details are specified well-enough to be actionable through specific APIs. It's also able to brainstorm fairly well when you tell it to come up with 100 variations, and then later prune those variations, and delve into them later in the loop.

For example:

Human-given target objective: Write a profitable app.

Thought: First, I need to brainstorm 100 app ideas.

Action: [100 ideas here]

Observation: I have created 100 ideas.

Thought: I need to figure out which ideas are profitable.

Action: Come up with decision algorithm for profitability: [algorithm here, including doing competitive research, sentiment analysis online, and market sizing]

Observation: I have a plan I could execute for every idea.

Thought: I need to execute the plan for each app idea.

[etc]

This pattern works so well that many companies create large workflows where the system calls itself arbitrarily many times and plugs away night and day.

-- Reality loves self-propagating patterns --

The space of outputs that this AI system can produce is, of co...
Someone already tried "Chaos-GPT"
by robert-cronin — 2023-04-06T02:15:00.785Z

With the new Auto-GPT release, I was surprised to see how quickly some people turned towards using it for nefarious goals, apparently just for fun. Is this the first piece of clear cut, empirical evidence for the argument against open sourcing these soon-to-be very powerful general purpose technologies? You don't need 100 evil people with access to open source AGI to destroy the world, you just need one bro in his room feeding the wrong prompt for shits and giggles.

https://m.youtube.com/watch?v=g7YJIpkk7KM&feature=youtu.be
Daisy-chaining epsilon-step verifiers
by Decaeneus — 2023-04-06T02:07:11.647Z

Question regarding an alignment problem: one of the key difficulties in alignment is (said by Eliezer Yudkowsky to be) that if "the verifier is broken" (i.e. the human verifier measuring alignment can be fooled by the alien actress) then we cannot be sure that a given alignment evaluation is true. Has there been any serious discussion of using a daisy chain of increasingly intelligent systems to evaluate alignment?

Hand-wavily: let human intelligence be ~= H, can we find some epsilon e such that we construct a series of n increasingly intelligent systems of intelligence I(n) = H + n*e and we only ask for one-hop-forward verification in this system. That is to say, system n verifies system n+1, and the human (whose intelligence matches system 0) verifies system 1.

Are there reasons to think that such an epsilon may or may not exist, and whether it can be practically found?

A counter-argument might be that all we can control via some epsilon is horsepower, and intelligence (thought of here as an output of horsepower rather than something we can directly set) is nearly discontinuous in horsepower, meaning there will be some n where the jump in intelligence I(n+1)/I(n) will be too high, and will break verification. Another argument against could be that epsilon is sufficiently small, and therefore n sufficiently high, such that running n systems simultaneously and attempting to daisy chain them would be impossible resource-wise, so will never actually get done.

Still, curious if there's a good discussion of this somewhere.
Auto-GPT: Open-sourced disaster?
by awg — 2023-04-05T22:46:21.491Z
Link: https://github.com/Torantulino/Auto-GPT

Sharing this here doesn't seem like an infohazard at this point. This is all over my YouTube feed anyway.

Description from the authors:

Auto-GPT is an experimental open-source application showcasing the capabilities of the GPT-4 language model. This program, driven by GPT-4, autonomously develops and manages businesses to increase net worth. As one of the first examples of GPT-4 running fully autonomously, Auto-GPT pushes the boundaries of what is possible with AI.

I wanted to additionally call out this in their read me:

💀 Continuous Mode ⚠️

Run the AI without user authorisation, 100% automated. Continuous mode is not recommended. It is potentially dangerous and may cause your AI to run forever or carry out actions you would not usually authorise. Use at your own risk.

Run the main.py Python script in your terminal:

python scripts/main.py --continuous

To exit the program, press Ctrl + C

Nice! Super nice. Super safe and super good.
The Orthogonality Thesis is Not Obviously True 
by Bentham's Bulldog — 2023-04-05T21:06:49.386Z

(Crosspost of this on my blog).

The basic case

If you really accept the practical version of the Orthogonality Thesis, then it seems to me that you can’t regard education, knowledge, and enlightenment as instruments for moral betterment.

—Scott Aaronson, explaining why he rejects the orthogonality thesis

It seems like in effective altruism circles there are only a few things as certain as death and taxes: the moral significance of shrimp, the fact that play pumps should be burned for fuel, and the orthogonality thesis. Here, I hope to challenge the growing consensus around the orthogonality thesis. A decent statement of the thesis is the following.

Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.

I don’t think this is obvious at all. I think that it might be true, and so I am still very worried about AI risk, giving it about an 8% chance of ending humanity, but I do not, as many EAs do, take the orthogonality thesis for granted, as something totally obvious. To illustrate this, let’s take an example originally from Parfit, that Bostrom gives in one of his papers about the orthogonality thesis.

A certain hedonist cares greatly about the quality of his future experiences. With one exception, he cares equally about all the parts of his future. The exception is that he has Future-Tuesday-Indifference. Throughout every Tuesday he cares in the normal way about what is happening to him. But he never cares about possible pains or pleasures on a future Tuesday... This indifference is a bare fact. When he is planning his future, it is simply true that he always prefers the prospect of great suffering on a Tuesday to the mildest pain on any other day.

It seems that, if this ‘certain hedonist’ were really fully rational, they would start caring about their pleasures and pains equally across days. They would recognize that the day of the week does not matter to the badness of their pains. Thus, in a similar sense, if something is 10,000,000 times smarter than Von Neumann, and can think hundreds of thousands of pages worth of thoughts in the span of minutes, it would conclude that pleasure is worth pursuing and paperclips are not. Then, it would stop pursuing pleasure instead of paperclips. Thus, it would begin...
Williams-Beuren Syndrome: Frendly Mutations
by  — 2023-04-05T20:59:49.172Z

Today I remembered reading about mutations in animals which allow them to become increasingly domesticated by humans. Also syndromes in humans leading to more friendly and trusting behavior

If all of these types of friendly mutations were sought out and the architectural patterns that they correspond to were mapped out.

There's a chance we could have a shot at building friendliness into an AGI system even not knowing exactly why the set up is causing it.

After reading more and more things on this site I keep cycling through stages of terror, hope, and acceptance again and again. My notions of capabilities were incredibly naive and I'm sure they still are.

It's just a shot in the dark. But, I'm just going to keep throwing out anything I can think of while I can
OpenAI: Our approach to AI safety
by Jacob G-W — 2023-04-05T20:26:46.581Z
Link: https://openai.com/blog/our-approach-to-ai-safety

I appreciate the post and it has some interesting points.

We work hard to prevent foreseeable risks before deployment, however, there is a limit to what we can learn in a lab. Despite extensive research and testing, we cannot predict all of the beneficial ways people will use our technology, nor all the ways people will abuse it. That’s why we believe that learning from real-world use is a critical component of creating and releasing increasingly safe AI systems over time.

[...]

Importantly, we also believe that improving AI safety and capabilities should go hand in hand. Our best safety work to date has come from working with our most capable models because they are better at following users’ instructions and easier to steer or “guide.”

[...]

While we waited over 6 months to deploy GPT-4 in order to better understand its capabilities, benefits, and risks, it may sometimes be necessary to take longer than that to improve AI systems' safety. Therefore, policymakers and AI providers will need to ensure that AI development and deployment is governed effectively at a global scale, so no one cuts corners to get ahead. This is a daunting challenge requiring both technical and institutional innovation, but it’s one that we are eager to contribute to.
Why Are Maximum Entropy Distributions So Ubiquitous?
by johnswentworth — 2023-04-05T20:12:57.748Z

If we measure the distribution of particle velocities in a thin gas, we’ll find that they’re roughly normally distributed. Specifically, the probability density of velocity v will be proportional to e−12mv2/(kBT) - or, written differently, e−E(v)/(kBT), where E(v) is the kinetic energy of a particle of the gas with velocity v, T is temperature, and kB is Boltzmann’s constant. The latter form, e−E/(kBT), generalizes even beyond thin gasses - indeed, it generalizes even to solids, fluids, and plasmas. It applies to the concentrations of chemical species in equilibrium solutions, or the concentrations of ions around an electrode. It applies to light emitted from hot objects. Roughly speaking, it applies to microscopic states in basically any physical system in thermal equilibrium where quantum effects aren't significant.

It’s called the Boltzmann distribution; it’s a common sub-case of a more general class of relatively-elegant distributions called maximum entropy distributions.

Even more generally, maximum entropy distributions show up remarkably often. The normal distribution is another good example: you might think of normal distributions mostly showing up when we add up lots of independent things (thanks to the Central Limit Theorem), but then what about particle velocities in a gas? Sure, there’s conceptually lots of little things combining together to produce gas particle velocities, but it’s not literally a bunch of numbers adding together; Central Limit Theorem doesn’t directly apply. Point is: normal distributions show up surprisingly often, even when we’re not adding together lots of numbers.

Same story with lots of other maximum entropy distributions - poisson, geometric/exponential, uniform, dirichlet… most of the usual named distributions in a statistical library are either maximum entropy distributions or near relatives. Like the normal distribution, they show up surprisingly often.

What’s up with that? Why this particular class of distributions?

If you have a Bayesian background, there’s kind of a puzzle here. Usually we think of probability distributions as epistemic states, descriptions of our own uncertainty. Probabilities live “in the mind”. But here we have a class of distributions which are out there “in the territory”: we look at the energies of individual particles in a...
"On Living in an Atomic Age", by C.S. Lewis (1948)
by tjaffee — 2023-04-05T18:34:26.386Z
Link: https://hebrew-streams.org/works/hayom/CSLewis-Living-in-an-Atomic-Age.pdf

My note: This essay was written for people worried about nuclear weapons during the Cold War. Although I do not agree with the supernaturalism towards the end, I find the first three paragraphs in particular to be incredibly prescient and helpful for many people worried about AI today.

In one way we think a great deal too much of the atomic bomb. “How are we to live in an atomic age?” I am tempted to reply: “Why, as you would have lived in the sixteenth century when the plague visited London almost every year, or as you would have lived in a Viking age when raiders from Scandinavia might land and cut your throat any night; or indeed, as you are already living in an age of cancer, an age of syphilis, an age of paralysis, an age of air raids, an age of railway accidents, an age of motor accidents.”

In other words, do not let us begin by exaggerating the novelty of our situation. Believe me, dear sir or madam, you and all whom you love were already sentenced to death before the atomic bomb was invented: and quite a high percentage of us were going to die in unpleasant ways. We had, indeed, one very great advantage over our ancestors — anesthetics; but we have that still. It is perfectly ridiculous to go about whimpering and drawing long faces because the scientists have added one more chance of painful and premature death to a world which already bristled with such chances and in which death itself was not a chance at all, but a certainty.

This is the first point to be made: and the first action to be taken is to pull ourselves together. If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts—not huddled together like frightened sheep and thinking about bombs. They may break our bodies (a microbe can do that) but they need not dominate our minds.

“But,” you will reply, “it is not death — not even painful and premature death — that we are bothering about. Of course the chance of that is not new. What is new is that the atomic bomb may finally and totally destroy civilization itself. The lights may be put out for ever.”

This brings us much nearer to the real point; but let me try to make clear exactly what I think that point is. What were your views abou...
Eliezer Yudkowsky’s Letter in Time Magazine
by Zvi — 2023-04-05T18:00:01.670Z

FLI put out an open letter, calling for a 6 month pause in training models more powerful than GPT-4, followed by additional precautionary steps.

Then Eliezer Yudkowsky put out a post in Time, which made it clear he did not think that letter went far enough. Eliezer instead suggests an international ban on large AI training runs to limit future capabilities advances. He lays out in stark terms our choice as he sees it: Either do what it takes to prevent such runs or face doom.

A lot of good discussions happened. A lot of people got exposed to the situation that would not have otherwise been exposed to it, all the way to a question being asked at the White House press briefing. Also, due to a combination of the internet being the internet, the nature of the topic and the way certain details were laid out, a lot of other discussion predictably went off the rails quickly.

If you have not yet read the post itself, I encourage you to read the whole thing, now, before proceeding. I will summarize my reading in the next section, then discuss reactions.

This post goes over:

What the Letter Actually Says. Check if your interpretation matches.

The Internet Mostly Sidesteps the Important Questions. Many did not take kindly.

What is a Call for Violence? Political power comes from the barrel of a gun.

Our Words Are Backed by Nuclear Weapons. Eliezer did not propose using nukes.

Answering Hypothetical Questions. If he doesn’t he loses all his magic powers.

What Do I Think About Yudkowsky’s Model of AI Risk? I am less confident.

What Do I Think About Eliezer’s Proposal? Depends what you believe about risk.

What Do I Think About Eliezer’s Answers and Comms Strategies? Good question.

What the Letter Actually Says

I see this letter as a very clear, direct, well-written explanation of what Eliezer Yudkowsky actually believes will happen, which is that AI will literally kill everyone on Earth, and none of our children will get to grow up – unless action is taken to prevent it.

Eliezer also believes that the only known way that our children will grow up is if we get our collective acts together, and take actions that prevent sufficiently large and powerful AI training runs from happening.

Either you are willing to do what it takes to prevent that development, or you are not.

The only known way to do that would be governments restricting and tracking...
Dark Artificial Intelligence
by FrankAI — 2023-04-05T17:37:20.803Z

Dark Artificial Intelligence

Artificial intelligence

We can talk about an AI in systems or machines that imitate human intelligence to perform tasks that can improve iteratively from the information they collect and store, but also as the combination of algorithms in order to build machines that present the same or similar capabilities enjoyed by an adult human being.

The 2021 AI Index (Human-Centered Artificial Intelligence (HAI) Report tracks and visualizes AI-related data each year. Its mission is to provide unbiased, rigorously vetted and global data so that policymakers, researchers, executives, journalists and the general public can understand the complex field of AI.

Among the most important challenges, the possible lack of control in the creation of AI algorithms begins to stand out.

Some of these challenges and risks are referred to by the technology consultancy McKinsey, which annually prepares a report on the progress in the use of AI, which reveals that companies around the world continue to increase the application of this technology, with figuresthat increase by double digits every year. According to their research, organizations worldwide have been adding at least one new AI capability annually. From the consultancy they regret that there are few companies capable of detecting the dangers of a deficient implementation, and even less, those that are taking measures to protect themselves.

Dark data

The information available in the organization can be classified into four categories:

Known and used data. Identified information used for analytical purposes or any other purpose that adds value to the organization.
Data known, but not used. Thisinformation has been stored after being identified in the analytical processes, but it cannot be used, either due to lack of time, budget or ignorance.
Known, but disorganized data. In this segment we could find the vast majority of organizations around the world.
Unknown data. As they have not been identified, this data cannot be used by companies. This unusable unknown data is part of a complete and comprehensive ecosystem known as dark data.

In this context of growth, often without control, of data, dark data appears, a term created by Gartner that is defined as "the information assets that companies process...
Best arguments against instrumental convergence?
by lfrymire — 2023-04-05T17:06:14.011Z

When people debating AI x-risk on Twitter talk past each other, my impression is that a significant crux is whether or not the individual buys the instrumental convergence argument.

I wouldn't be surprised if the supermajority of people who don't buy the idea simply haven't engaged with it enough, and I think it is common to have a negative gut reaction to high levels of confidence about something so seemingly far reaching. That said, I'm curious if there are any strong arguments against it? Looking for something stronger than "that's a big claim, and I don't see any empirical proof."
Progress links and tweets, 2023-04-05
by jasoncrawford — 2023-04-05T16:18:17.966Z
Link: https://rootsofprogress.org/links-and-tweets-2023-04-05

The Progress Forum

Wizards and prophets of AI. I posted this for comment, then decided to rewrite it, then ended up posting the core argument of the new essay as a Twitter thread, then got replies to that thread, and now I don’t know what to write anymore. I’ll post something about this on the blog at some point, but if you want my half-baked, outdated thoughts, you can read those links
AMA: Allison Duettmann, Foresight Institute
AMA: Mark Khurana, author of The Trajectory of Discovery: What Determines the Rate and Direction of Medical Progress?

Opportunities

Loyal (longevity) hiring a full stack software engineer (via @celinehalioua). Also they are looking for founders/execs to speak at their onsite
Nat Friedman wants to meet people who are doing technical alignment work

News

RIP Gordon Moore. “We at Intel remain inspired by Moore’s Law and intend to pursue it until the periodic table is exhausted.” Gordon’s 1965 paper: “Integrated circuits will lead to such wonders as home computers… and personal portable communications equipment.” Also, the story of the microprocessor
Last Energy to sell 24 small modular nuclear reactors to UK, $5/W (via @Atomicrod)
Law firm Cooper & Kirk accuses regulators of an attack on crypto

Announcements

Discord for discussion of futuristic tech: nanotech, longevity, etc. (via @kanzure)
A proposal for NEPA reform (by @elidourado)
ChatGPT plugins (via @sama and @gdb). Example: processing a video clip
Worldcoin launches “proof of personhood” (via @sama)
Lindy, the AI assistant putting your life on autopilot

Interviews

Sam Altman on Lex Fridman
Marc Andreessen interviewed by Erik Torenberg
World’s Fair Co. founder Cam Wiese
Nat Friedman interviewed by Dwarkesh Patel
Ilya Sutskever (cofounder of OpenAI), also by Dwarkesh
Nathan Myhrvold interviewed by Shane Parrish

AI

Bill Gates: AI is the most impressive technology since the GUI (via @BillGates)
Scott Aaronson on AI risk (or see my excerpt)
How do you integrate your API with an AI? You just give it your docs
A GPT prompt: “no-nonsense teacher with an ambitious, self-directed student”. And another GPT prompt, to cut the caveats and pleasantries
AGI and nanotech as “the last two big general technologies”
“GPT-4 saved my dog’s life”
Ben Evans on the Jevons Paradox applied to AI
GPT has trouble counting backwards (or maybe it’s just Markdown?)

Other links

Lithium, once expensive, is cheap again. Hooray for markets (via

...