trentmkelly/LessWrong-43k huggingface dataset:
A method for empirical back-testing of AI's ability to self-improve
by Michael Tontchev — 2023-03-21T20:24:20.418Z

TL;DR; Train an LLM/AI with text only until 2016, and ask it to come up with AI architecture ideas based on ML research until then to see whether it could come up with the idea of transformers.

One of the biggest promises of A(G)I is automated and fast invention of new ideas, theories, and scientific breakthroughs. As some have said, AGI might be the last invention humanity needs to make.

We'd like to be able to test existing and future AIs for their capacity for innovation, but it's hard. It includes two parts:

Getting AI to come up with new ideas

: this is easy. Just ask GPT-4. There are some interesting prompts floating around, like this one:
What’s an example of a phenomenon where humanity as a whole lacks a good explanation for, but, taking into account the full set of human generated knowledge, an explanation is actually possible to generate? Please write the explanation. It must not be a hypothesis that has been previously proposed. A good explanation will be hard to vary.
Source and examples: https://twitter.com/gfodor/status/1637323490390073349?t=zhRmG397YS9lRi24oSGRjg&s=19

Validate the new ideas AI came up with

: This seems hard. To test a hypothesis, we often need to marshal resources, go out into the real world, and test it. We need experts, funding, and often the creation of new methods. Even if it's just algorithms, testing them at scale is not trivial.

Fortunately, we do have a free source of evidence for validity of new hypotheses and theories: history. If you want to test whether an AI can come up with architectural improvements to itself, hide the last 5 years of ML advancements from it and ask it to come up with 1000 ideas for architectural improvements. Use a time-unrestricted AI to automatically compare these hypotheses to the actual ML innovations to see whether any of its ideas came close to the ideas that humans came up with.

Extension: You can do this with any arbitrary knowledge cutoff and any human invention. Train an LLM with all human knowledge up until 1700 and ask it to come up with hypotheses for various phenomena that were explained since 1700. You can have LLMs trained with info up to any decade in the last 400 years try to predict inventions that happened after that cutoff.

For earlier years, you may run into not having enough data. To get around this, you can use time-unrestricted AI to generate training text that doesn't include...
AI Fables
by Bard — 2023-03-21T19:19:47.692Z

I write fiction. I'm also interested in how AI is going to impact the world. Among other things, I'd prefer that AI not lead to catastrophe. Let's imagine that I want to combine these two interests, writing fiction that explores the risks posed by AI. How should I go about doing so? More concretely, what ideas about AI might I try to communicate via fiction?

This post is an attempt to partially answer that question. It is also an attempt to invoke Cunningham's Law: I'm sure there will be things I miss or get wrong, and I'm hoping the comments section might illuminate some of these.

Holden's Messages

A natural starting point is Holden's recent blog post,

Spreading Messages to Help With the Most Important Century

. Stripping out the nuances of that post, here's a list of the messages that Holden would like to see spread:

We should worry about conflict between misaligned AI and all humans.
AIs could behave deceptively, so “evidence of safety” might be misleading.
AI projects should establish and demonstrate safety (and potentially comply with safety standards) before deploying powerful systems.
Alignment research is prosocial and great.
It might be important for companies (and other institutions) to act in unusual ways.
We're not ready for this.

However, as interesting as this list is, it's not what I'm looking for; I'm not looking for bottom-line messages to convey. Instead, I want to identify a list of smaller ideas that will help people to reach their own bottom lines by thinking carefully through the issues. The idea of instrument convergence might appear on such a list. The idea that alignment research is great would not.

One reason for my focus is that fiction writing is ultimately about details. Fiction might convey big messages, but it does so by exploring more specific ideas. This raises the question: which specific ideas?

Another reason for my focus is that I'm allergic to propaganda. I don't want to tell people what to think and would prefer to introduce ideas that can help people think for themselves. Of course, not all message fiction is propaganda, and I'm not accusing Holden of calling for propaganda. Still, my personal preference is to focus on how to convey the nuts and bolts needed to understand AI.[1]

What Nuts and Which Bolts?

So with context to hand, back to the question: what ideas about AI might someone try to convey via fiction? Here's a potential...
Adversarial (SEO) GPT training data?
by Dagon — 2023-03-21T18:55:01.330Z

I guarantee someone's thinking about this, but I haven't seen the scammers selling it yet, so I don't know how transparent or discoverable the scraping/input methods are for LLM source data.

Is there any indication that websites or publishers are modifying their pages/data in ways that give themselves more weight in future GPT accessibility/prediction for related prompts?
Why not constrain wetlabs instead of AI?
by Lone Pine — 2023-03-21T18:02:43.336Z

Most of the object-level stories about how misaligned AI goes wrong involve either nanotechnology or bio-risk or both. Certainly I can (and have, and will again) tell a story about AI x-risk that doesn't involve anything at the molecular level. A sufficient amount of (macroscale) robotics would be enough to end humanity. But the typical story that we hear, particularly from EY, involves specifically nanotechnology. So let me ask a Robin Hanson-style question: Why not try to constrain wetlabs instead of AI? By "wetlabs" I mean any capability involving DNA, molecular biology or nanotechnology.

Some arguments:

Governments around the world are already in the business of regulating all kinds of chemistry, such as the production of legal and illegal drugs.

Governments (at least in the West) are not yet in the business of regulating information technology, and basically nobody thinks they will do a good job of it.

The pandemic has set the stage for new thinking around regulating wetlabs, especially now that the lab leak hypothesis is considered mainstream.

The cat might already be out of the bag with regards to AI. I'm referring to the Alpaca and Llama models. Information is hard to constrain.

"You can't just pay someone over the internet to print any DNA/chemical you want" seems like a reasonable law. In fact it's somewhat surprising that it's not already a law. By comparison, "You can't just run arbitrary software on your own computer without government permission" would be an extraordinary social change and is well outside the Overton window.

Something about pivotal acts which... I probably shouldn't even go there.
Wouldn't an intelligent agent keep us alive and help us align itself to our values in order to prevent risk ?
by Risk I mean experimentation by trying to align potentially smarter replicas?
by Terrence Rotoufle — 2023-03-21T17:44:32.972Z

I was wondering about some very dumb argument on alignment but no one answered to me.

I am in the research of reasons of why an intelligent agent would not keep us alive and try to help us align itself accordingly to our values, in the goal of preventing being the agent experimenting with aligning potentially smarter replicas, but I can't find anything on the subject. Is this hypothesis outside of the realm of possibilities ?

I am actually wondering if you align an agent with your own values, but not at the point of self-improvement, the said agent could have interest into helping you keeping him aligned with his current goal, in order to diminish the cost of experiment of him creating a replica.
As a note, couldn't this work too with agent not aligned with us ? Themselves helping us align them with our goals as a proof of concept of alignment ?

Him trying to align another powerful agent, all by itself, might be doomed to fail, that sound a lil crazy but I think there might be something there, the motto : experimenting on one's self in order to mitigate experimentation cost, there might be less cost to work with humans than with other really intelligent agents.

Less costs in terms of collaboration rewards too.

Am I missing something If I say that even if the model solves the alignment problem by itself, by not testing the alignment solution on itself by the proxy of a less intelligent species, it mitigates its chances of survival. Even if the mathematical proof for the alignment solution gets to be solid, there could be some very small offset of the replica going wrong, and the only way to take the risk would be to use some dumbie human.

There's one issue to that idea though, it would be if the agent realize that it can apply that logic on it's descendant too, and that every descendant can apply it recursively, what would that imply ?

As there's mostly incentives to create a descendant, and try to align it by the same reasoning we could use, the descendant could reason in the same way and dedicate ressources into trying to create another agent again, and again. Wondering if as every agent would know of that phenomena that what would result would be a state of inaction by constant replication, just getting caught in the wrong instrumental goal.

As a note I guess that the system would also know, the possibility of him not getting to it's goal by getting caught in a rep...
Employer considering partnering with major AI labs. What to do?
by GraduallyMoreAgitated — 2023-03-21T17:43:17.439Z

I would sincerely appreciate commentary and impressions on an issue that is really heavily affecting me. I'm posting it here with relative detail in hopes that people in similar circumstances can compare notes and offer advice.

I work at a currently-successful software start-up of under 100 people, all of whom I respect and many of whom have become my closest friends. My job at this company has certainly been the most enjoyable and rewarding of my career. I gladly make sacrifices in other parts of my life to help further its goals. Nearly all days are a genuine pleasure. My position is relatively senior, in that I have the ear of the executive leadership, but cannot veto company strategy.

We develop software for heavy industries which are not likely to want decisions to be made by AI, due to stringent standards of safety. We currently use our in-house produced neural networks for a niche corner of image and object recognition that seems to be currently market-leading in its small field. We do not perform novel research, let alone publish.

Recently, it has dawned on the company leadership team that AI is likely the be-all and end-all of large-scale software companies, and is seriously considering making significant investments into scaling our and team and ambitions in the field.

High-confidence beliefs I have about their intent:

We will not make an eventual move towards researching general intelligence. It is too far away from our established base of customers.
I don't see a way in which we would start researching or publishing novel, industry-leading techniques for any field of AI.
Our most likely course of action will be optimizing known and published research for our particular data-extraction and image-recognition purposes.
We will likely implement and fine-tune other companies' object recognition, software assistant, and chat-bot AIs within our products.

Personally, I see a few options that lead to continued prosperity without direct contribution to race dynamics:

We use off-the-shelf tools, mostly from alignment concerned organizations.
We don't partner with Google/Facebook/Microsoft/Amazon for our training infrastructure.
We continue to not publish nor push novel research.

Some of the less avoidable consequences are:

Generally increasing AI hype.
Increasing competition in adjacent AI fields (object recognition). That b

...
Sun-following Garden Mirrors?
by jefftk — 2023-03-21T16:20:02.103Z

A friend of ours across the street has a small front garden, but since
we're in Boston, their house faces north, and it's three stories tall
the garden gets almost no direct sun. [1] Could we fix this with a mirror?

If I put a static mirror on the front of our house, even on an angled
mount, it would only send light to the garden for a short period each
day. To be actually useful, the mirror would need to be dynamic,
tracking the sun: a heliostat. Since
the movement of the sun is well understood you can do this without
any sensors, just calibration.

With a bit of looking I found two
companies selling
heliostats, though I think the main residential use is bringing more
light in through windows and not gardening. Possibly building
something would be fun, but I'd be nervous about wind loads and
breakage filling our own front garden with shards. Has anyone done a similar
project?

[1] Boston's latitude is 42°, so the peak height of the sun on the
equinox is 48° (90°-42°). The solstices are ±
23° (axial
tilt) so at solar noon on the summer solstice it gets up to
71°. The house is ~40ft tall so its shadow is always > ~13ft
(40/tan(71°)).
Some constructions for proof-based cooperation without Löb
by James Payor — 2023-03-21T16:12:16.920Z

This post presents five closely-related ways to achieve proof-based cooperation without using Löb's theorem, and muses on legible cooperation in the real world.

(Edit: maybe they're closer to just-use-Löb's-theorem than I originally thought! See this comment. If these constructions somehow work better, I'm more confused than before about why.

New edit: I do think there's something up about these constructions being better-founded than a Löb-based approach, specifically thanks to these results, though I don't have a more full account yet for what's up with that.)

I'm writing this as a follow-up to Andrew Critch's recent post, to share more of my perspective on the subject.

We're going to dive straight into the weeds. (I'm planning to also write a more accessible explainer post soon.)

The ideas

Idea #1: try to prove A→B

I claim the following are sufficient for robust cooperation:

A↔□(A→B)

B←□A

A tries to prove that A→B, and B tries to prove A. The reason this works is that B can prove that A→□A, i.e. A only cooperates in ways legible to B. (Proof sketch: A↔□X→□□X↔□A.)

The flaw in this approach is that we needed to know that A won't cooperate for illegible reasons. Otherwise we can't verify that B will cooperate whenever A does.

This indicates to me that "A→B" isn't the right "counterfactual". It shouldn't matter if A could cooperate for illegible reasons, if A is actually cooperating for a legible one.

Idea #2: try to prove □A→B

We can weaken the requirements with a simple change:

A←□(□A→B)

B←□A

Note that this form is close to the lemma discussed in Critch's post.

In this case, the condition □A→B is trivial. And when the condition activates, it also ensures that □A is true, which discharges our assumption and ensures B is true.

I still have the sense that the condition for cooperation should talk about itself activating, not A. Because we want it to activate when that is sufficient for cooperaion. But I do have to admit that □A→B works for mostly the right reasons, comes with a simple proof, and is the cleanest two-agent construction I know.

Idea #3: factor out the loop-cutting gadget

We can factor the part that is trying to cut the loop out from A, like so:

A←□X

B←□A

X↔□(X→B); or alternatively

X↔□(□X→B)

This gives the loop-cutting logic a name, X. Now X can refer to itself, and roughly says "I'll legibly activate if I can verify this will cause B to be true".

Li...
Clarifying mesa-optimization
by Marius Hobbhahn — 2023-03-21T15:53:33.955Z

Produced as part of the

SERI ML Alignment Theory Scholars Program

- Winter 2022 Cohort.

Thanks to Jérémy Scheurer, Nicholas Dupuis and Evan Hubinger for feedback and discussion

When people talk about mesa-optimization, they sometimes say things like “we’re searching for the optimizer module” or “we’re doing interpretability to find out whether the network can do internal search”. An uncharitable interpretation of these claims is that the researchers expect the network to have something like an “optimization module” or “internal search algorithm” that is clearly different and distinguishable from the rest of the network (to be clear, we think it is fine to start with

probably wrong mechanistic models

).

In this post, we want to argue why we should not expect mesa-optimization to be

modular

or clearly different from the rest of the network (at least in transformers and CNNs) and that current architectures can already do mesa-optimization in a meaningful way. We think this implies that

Mesa-optimization improves gradually where more powerful models likely develop more powerful mesa optimizers.
Mesa-optimization should not be treated as a phenomenon of the future. Current models likely already do it, just in a very messy and distributed fashion.
When we look for mesa optimization, we probably have to look for a messy stack of heuristics combined with search-like abilities rather than clean Monte Carlo Tree Search (MCTS)-like structures.

We think most of our core points can be conveyed in a simple analogy. Imagine a human chess grandmaster that has to choose their moves in 1 second. In this second, they are probably not running a sophisticated tree search in their head, they rely on heuristics. These heuristics were shaped by years of playing the game and are often the result of doing explicit tree searches with more time. The resulting decision-making process is a heuristic that approximates or was at least shaped by optimization but is not an optimizer itself. This is approximately what we think mesa-optimization might look like in current neural networks, i.e. the model uses heuristics that have aspects of or approximate parts of optimization, but are not “clean”[1] in the way e.g. MCTS is.

What is an accurate definition of mesa-optimization?

In

risks from learned optimization

mesa-optimization is characterized as
[...] it...
"Perspective: Focused-Ultrasound Guided Neuropeptide Delivery as a Novel Therapeutic Approach in Psychiatry" (Seeds of Science call for reviewers)
by rogersbacon — 2023-03-21T14:31:49.390Z

Seeds of Science is a journal (funded through Scott Alexander's ACX grants program) that publishes speculative or non-traditional articles on scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them); top comments are published after the main text of the manuscript.

We have just sent out an article for review that may be of interest to some in the LessWrong community, so I wanted to see if anyone - particularly those with a background in neuroscience - would be interested in joining us as a gardener providing feedback on the article. It is free to join and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so it's no worries if you don't plan on reviewing very often but just want to take a look here and there at the articles people are submitting).

To register, you can fill out this google form. From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review form for recording votes/comments. If you would like to just take a look at this article without being added to the mailing list, then just reach out (info@theseedsofscience.org) and say so.

Happy to answer any questions about the journal through email or in the comments below. Here is the abstract for the article.

Perspective: Focused-Ultrasound Guided Neuropeptide Delivery as a Novel Therapeutic Approach in Psychiatry

Although drugs are a critical component of mental healthcare, most have modest benefits and significant side effects. One way to develop a superior intervention would be to administer drugs with the spatial and temporal precision that better replicates natural diversity within neurotransmitter systems. A technology called focused-ultrasound (FU) may be able to safely and transiently disrupt the blood-brain barrier with spatial precision, permitting the site-specific delivery of molecules that do not conventionally cross the blood-brain barrier. If this method is proven to be safe and effective in larger human trials, it may trigger a paradigm shift in biopsychology research where the l...
AI #4: Introducing GPT-4
by Zvi — 2023-03-21T14:00:01.161Z

Lemon, it’s Tuesday

Somehow, this was last week:

(Not included: Ongoing banking crisis threatening global financial system.)

Oh, also I suppose there was Visual ChatGPT, which will feed ChatGPT’s prompts into Stable Diffusion, DALL-E or MidJourney.

Current mood, perhaps.

The reason to embark on an ambitious new AI project is that you can actually make quite a lot of money, also other good things like connections, reputation, expertise and so on, along the way, even if you have to do it in a short window.

The reason not to embark on an ambitious new AI project is if you think that’s bad, actually, or you don’t have the skills, time, funding or inspiration.

I’m not not tempted.

Table of Contents

What’s on our plate this time around?

Lemon, It’s Tuesday: List of biggest events of last week.

Table of Contents: See table of contents.

Executive Summary: What do you need to know? Where’s the best stuff?

Introducing GPT-4: GPT-4 announced. Going over the GPT-4 announcement.

GPT-4 The System Card Paper: Going over GPT-4 paper about safety (part 1).

ARC Sends In the Red Team: GPT-4 safety paper pt. 2, ARC’s red team attempts.

Ensuring GPT-4 is No Fun: GPT-4 safety paper pt. 3. Training in safety, out fun.

GPT-4 Paper Safety Conclusions + Worries: GPT-4 safety paper pt.4, conclusion.

A Bard’s Tale (and Copilot for Microsoft 365): Microsoft and Google announce generative AI integrated into all their office products, Real Soon Now. Huge.

The Search for a Moat: Who can have pricing power and take home big bucks?

Context Is That Which Is Scarce: My answer is the moat is your in-context data.

Look at What GPT-4 Can Do: A list of cool things GPT-4 can do.

Do Not Only Not Pay, Make Them Pay You: DoNotPay one-click lawsuits, ho!

Look What GPT-4 Can’t Do: A list of cool things GPT-4 can’t do.

The Art of the Jailbreak: How to get GPT-4 to do what it doesn’t want to do.

Chatbots Versus Search Bars: Which is actually the better default method?

They Took Our Jobs: Brief speculations on employment impacts.

Botpocalypse and Deepfaketown Real Soon Now: It begins, perhaps. Good news?

Fun with Image Generation: Meanwhile, this also is escalating quickly.

Large Language Models Offer Mundane Utility: Practical applications.

Llama You Need So Much Compute: Putting an LLM on minimal hardware.

I

...