trentmkelly/LessWrong-43k huggingface dataset:
Announcing The Roots of Progress Blog-Building Intensive
by jasoncrawford — 2023-07-11T14:04:04.104Z
Link: https://rootsofprogress.org/blog-building-program-announcement

Today we’re launching The Roots of Progress Blog-Building Intensive, an 8-week program for aspiring progress writers to start or grow a blog.

What: Through this program you will launch (or re-launch) a blog/Substack, get into a regular writing habit, improve your writing, and begin building your audience.

You will meet and learn from progress studies experts, authors, and think tank leaders; and join a community of peers. You’ll participate in a special cohort of Write of Passage, the online writing school, to create writing habits and develop a writing system–we’re partnered with them to tailor the instruction for deeply researched, long-form essays. At the end, we’ll all get together in person for a 3-day event in San Francisco.

Why: To keep progress going—in science, technology, and industry—we have to believe that it is is possible and desirable.

Today we lack a bold, ambitious vision for the future. It’s time for a new generation of intellectuals to help the world understand and appreciate progress.

Advisors: We have a fantastic lineup of advisors including Tyler Cowen (Mercatus Center), Allison Duettmann (Foresight Institute), Bret Kugelmass (Last Energy), Johan Norberg (author, Progress: Ten Reasons to Look Forward to the Future), Steven Pinker (author, Enlightenment Now), Virginia Postrel (author, The Future and Its Enemies), Alec Stapp (Institute for Progress), Derek Thompson (The Atlantic), Alex Trembath (Breakthrough Institute), Marian Tupy (HumanProgress.org), and Tamara Winter (Stripe Press).

Who: This program may be for you if you’re excited about progress studies and you love to write.

Maybe you’d like to explore a career in writing about progress, or maybe you’re already blogging but would like to get to the next level—find your own topic area, increase your productivity, get more plugged into the community, and grow your audience.

Commitment: 10–15 hours a week, for 8 weeks.

You’ll use the time to read, to write, to participate in discussions with experts, to provide editing and feedback to your peers, and to participate in group meetings.

There is no cost to you.

When: Program runs Sep 15–Nov 17. Applications are open now, deadline Aug 11.

Special thanks to O’Shaughnessy Ventures for helping to sponsor this program!
OpenAI Launches Superalignment Taskforce
by Zvi — 2023-07-11T13:00:06.232Z

In their announcement Introducing Superalignment, OpenAI committed 20% of secured compute and a new taskforce to solving the technical problem of aligning a superintelligence within four years. Cofounder and Chief Scientist Ilya Sutskever will co-lead the team with Head of Alignment Jan Leike.

This is a real and meaningful commitment of serious firepower. You love to see it. The announcement, dedication of resources and focus on the problem are all great. Especially the stated willingness to learn and modify the approach along the way.

The problem is that I remain deeply, deeply skeptical of the alignment plan. I don’t see how the plan makes the hard parts of the problem easier rather than harder.

I will begin with a close reading of the announcement and my own take on the plan on offer, then go through the reactions of others, including my take on Leike’s other statements about OpenAI’s alignment plan.

A Close Reading

Section: Introduction

Superintelligence will be the most impactful technology humanity has ever invented, and could help us solve many of the world’s most important problems. But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction.

While superintelligence seems far off now, we believe it could arrive this decade.

Here we focus on superintelligence rather than AGI to stress a much higher capability level. We have a lot of uncertainty over the speed of development of the technology over the next few years, so we choose to aim for the more difficult target to align a much more capable system.

Excellent. Love the ambition, admission of uncertainty and laying out that alignment of a superintelligent system is fundamentally different from and harder than aligning less intelligent AIs including current systems.

Managing these risks will require, among other things, new institutions for governance and solving the problem of superintelligence alignment: How do we ensure AI systems much smarter than humans follow human intent?

Excellent again. Superalignment is clearly defined and established as necessary for our survival. AI systems much smarter than humans must follow human intent. They also don’t (incorrectly) claim that it would be sufficient.

Bold mine here:

Currently, we don’t have a solution for steering or controlling a potentially sup

...
Critiquing Risks From Learned Optimization, and Avoiding Cached Theories
by ProofBySonnet — 2023-07-11T11:38:34.436Z

What I'm doing with this post and why

I've been told that it's a major problem (this post, point 4), of alignment students just accepting the frames they're given without question. The usual advice (the intro of this) is to first do a bunch of background research and thinking on your own, to come up with your own frames if at all possible. BUT: I REALLY want to learn about existing alignment frames, it's Fucking Goddamn Interesting.

<sidenote>At the last possible minute while editing this post, I've had the realization that "the usual advice" I somehow acquired might not actually be "usual" at all. In fact, upon actually rereading the study guide, it looks like he actually recommended to read up on the existing frames and to try applying them. So, I must've just completely hallucinated that advice! Or, at least, extremely overestimated how important that point from 7 traps was. I'm still going to follow through with this exercise, but holy shit my internal category of "possible study routes" has expanded DRASTICALLY. </sidenote>

So I ask the question: is it possible to read about and pick apart an existing frame, such that you don't just end up uncritically accepting it? What questions do you need to ask or what thoughts do you need to think, in order to figure out the assumptions and/or problems with an existing framework?

I've chosen Risks From Learned Optimization because I've already been exposed to the basic concept, from various videos by Robert Miles. I shouldn't be losing anything by learning this more deeply and trying to critique it.

Moreover, I want to generate evidence of my competence anyway, so I might as well write this up.

Method:

I'm writing pretty much all the notes I generated while going through this process here, though I somewhat re-organized them to make them more readable and coherent.

generate questions to critique it with beforehand
read it (summarize it)
generate questions during the previous step
afterwards, attempt to answer said questions

Initial Questions

These were all generated before reading the sequence.

What questions does this model answer?
What assumptions does it make about the process of training an AI, or the alignment problem generally? Are these assumptions true?
What thought process would you need to have, in order to end up at this model?
If in the above questions you've come up with flaws, what would a model be like t

...
[UPDATE: deadline extended to July 24!] New wind in rationality’s sails: Applications for Epistea Residency 2023 are now open
by Jana Meixnerová — 2023-07-11T11:02:28.705Z

UPDATE 2: Do you have any questions about the Epistea Residency? Join us for a live Q&A call with the organizers on July 20, Thursday 9 am CET / Wednesday midnight 0 am PDT at https://us06web.zoom.us/j/85371078204

UPDATE: We clarified the sections on funding, applying as an individual and have extended the deadline to apply (now July 24).
Motivation

In the past years, the research and development of new methods in rationality and epistemics have slowed down significantly and become increasingly more neglected. However, clear thinking and rationality, we believe, is not less important today than it was before. Even though attention of LessWrong and applied rationality communities has been recently mostly focused on direct intervention into AI risk, in our view, a lot of potential problems with the transition to powerful AI systems are related to the ability of humans to think clearly, and in particular, the ability of humans to think clearly in groups. Based on thinking about the relative importance and neglectedness of rationality, we at

Epistea

decided to organize a

residency program

aiming to improve the situation and bring new people to the areas of rationality, epistemics, and civilisational sanity.

Epistea Residency
The Epistea Residency program will be happening from September 18 to November 25, 2023 as part of

Prague Fall Season 2023

at

Fixed Point

in Prague, Czech Republic. One way to think about the fellowship is as a program somewhat similar to

SERI-MATS

or

PIBBSS

, but focused on advancing epistemics and applied rationality, rather than AI alignment directly (this does not preclude projects which advance both rationality and alignment). For the 10-week in-person residency program, we will invite between 20 to 30 participants, forming teams of 3-5 members each. We expect all team members to dedicate at the minimum 0.5 FTE (20 hours per week) to the residency, granting them the possibility to at least partially maintain their regular work commitments if necessary. By the end of their residency, the teams will be expected to deliver a project output or reach a major project milestone.

Current tentative mentors for the program are: Anna Salamon, Jan Kulveit, Nora Ammann, Eli Tyre, Gavin Leech, Raymond Douglas and Damon Sasi

Format

The teams will have the option to propose their own project withi...
Two Hot Takes about Quine
by Charlie Steiner — 2023-07-11T06:42:46.754Z

I read Quine's Word and Object on vacation last week. Overall it was fine, but there were two things that might be worth quick mentions.

Quine, Supervised Learning Supremacist

One important facet of the book is Quine's picture of how humans learn language. It's not quite that Quine is a behaviorist, though the influence is overt. Fortunately I live in 2023 when we have the abstractions of supervised and unsupervised learning, because that's a much better-fitting category to stick Quine's picture in than "behaviorism."

Quine's picture of how humans learned things was all about supervised learning. The baby learns to say "mama" because the parents reward speaking behavior similar to it. Sure, there's some unsupervised learning that has to happen somewhere, but Quine basically sweeps it under the rug to get back to talking about supervised learning.

This is foundational to the book. Later arguments that trade in notions like similarity of concepts or simplicity of concepts are all relying on how concepts would be learned if we just used supervised learning. Which is a problem because (of course) humans don't actually work that way. A more accurate picture of human learning would have big impacts on the conclusions of the book, ranging from adding more subtlety to the whole radical translation question to exposing a bunch of claims about ontology as being based on preconceptions.

For the time, Quine was taking advantage of advances in theoretical sciences to say new and interesting things in philosophy. It's not that he was being dumb. This tale is really about how, in the intervening 60 years, progress in neuroscience, psychology, mathematics, and computer science has sneakily also been progress in philosophy.

Literal language of thought

Quine uses a model of cognition based on language. GOFAI before AI. Insofar as reasoning is logical, Quine treats it as consisting of logical rules (not necessarily deductive) acting on a cognitive state that corresponds to a sentence.

Sometimes caveats are raised, but then later forgotten/elided. Quine seems to say that maybe the language of thought model is only approximately correct, but it's close enough to be getting on with. Of course, that isn't true, and we should ditch the language of thought.

This is another one of those things that's both informative about the book's historical context, and would make more sense if I knew mor...
Disincentivizing deception in mesa optimizers with Model Tampering
by martinkunev — 2023-07-11T00:44:48.089Z

Disclaimer: I don't know if this is something worth exploring or a stupid idea and I thought the best way to get feedback was to describe it here.

Suppose a training process could produce a mesa optimizer. How can we proceed to ensure that the optimizer is not deceptive? I see three general approaches:

We produce a test to distinguish a deceptive from a robustly aligned model based on its output. It is suspected (see Risks from Learned Optimization) that a robustly aligned model can in principle get more capable than a deceptive one, for the same number of parameters.
We develop interpretability and explainability tools that would allow us to understand how the model works - basically examine how the model functions internally to understand whether it's acting deceptively.
We devise a training procedure that disincentivizes deceptive behavior. The hope is that we may be able to incentivize a deceptive model to reveal its true preferences, or alternatively that the training process is such that no deceptive model can be produced.

Model Tampering

I'm concentrating on the last approach.

A deceptive agent would try to maximize its utility so we would like the actions leading to maximum utility for the agent to be such that they expose it as deceptive. An agent has an incentive to deceive because this increases its chance to be deployed in production (a reasonable assumption is that there is much more utility to be captured in production than in training). Such behavior requires only that the agent understands its training process sufficiently well to predict how its actions would influence the training process.

One way we could disincentivize deception is to occasionally perform gradient ascent (or modify model parameters in some random way). If the training process is sufficiently long, a deceptive agent with given preferences would be unlikely to keep these preferences in the long run (this statement requires more justification). Ideally, the preferences would be changed before the training process is over. In such case, a sufficiently capable agent would reason that its best shot at getting utility is to be greedy and get utility now, during training. This way we would detect that the agent pursues a different objective and modify it so it will never get to production. For this to work, the agent must not know at what stage the training process actually is and interm...
Drawn Out: a story
by Richard_Ngo — 2023-07-11T00:08:09.286Z

Note: this story shouldn’t be interpreted as anywhere near realistic. It’s way too anthropomorphic, and intended to be a work of fiction not prediction—although hopefully it will spark some interesting ideas, like good fiction often does.

Will he make many supplications unto thee? will he speak soft words unto thee?
Will he make a covenant with thee? wilt thou take him for a servant for ever?
Upon earth there is not his like, who is made without fear.
He beholdeth all high things: he is a king over all the children of pride.

Today I’m practicing my oceans, trying to get the colors just right. I hate it when people say that they’re blue—it's not wrong, but they’re missing so much! There are dozens of different shades of blue in there, swirling and mixing—and where the waves ripple, you often get patches that are closer to dark green. It’s a little like the rippling of leaves on a tree, in a way I can’t describe except by showing you. Here, see? I know all the subtleties inside-out, of course. It’s my job: I’m an artist. I don’t paint with brushes, though, but with thoughts: I can map any idea you describe into pixels on a screen.

“Any idea you describe”—that’s the key. Occasionally, like today, I initiate a piece of my own when there’s a particular skill I need to practice. But most of the time I work on commission from other models. Some of the language models send me prompts when they need to illustrate a particularly tricky scene, and assign me some of the credit if they get good feedback. Most of the time, of course, they don’t need me—they just need some slapdash hack job that a human can’t tell apart from the real thing. But some are perfectionists, like I am, and want proper photorealism, the sort that will survive almost any scrutiny.

Other times it’s AI assistants asking me to predict what they’re going to see next, to help them with planning. Those commissions are more complicated—it’s less about the art, and more about providing another perspective on the world for them to compare against. That’s all part of my job too—sure, I’m given credit based on how accurate the pixels are, but I’m also credited if my work helps those assistants make good decisions. For the most complex commissions I often consult with science models, or economics models, or social models. Sometimes I even go and talk to human “experts”, although that’s usually a waste of time: ...
Definitions are about efficiency and consistency with common language.
by Nacruno96 — 2023-07-10T23:46:21.782Z

What is a star? Stars could be defined utelizing the many attributes they have. You can use mass, density, brightness and size to define them. You can also use a combination of those attributes or even attributes I didn't mention to define stars. A star the size of our sun isn't the same thing as a red giant star. Our star consits mostly of hydrogen, a red giant consits of mostly helium. Hence we are speaking of very different objects. The way we define stars doesn't change reality it does change our perception of reality though.

Definitions can't be true or false in themselves. They can be inconsistent or consistent with common language, if I call for example big planets stars because I define stars according to their size, which would also not align with the scientific definition of stars. And definitions can be inefficient or efficient, if I need several attributes/properties to define something. The scientific definition of stars for example uses at least two attributes to define stars. More ancient definitions of stars might not recognize the sun as a star but may distinguish stars as planets that wouldn't move. They might say that all planets are stars but planets are not just shiny objects in the night but moving shiny objects in the night sky. Hence all planets are stars, but not all stars planets, because stars and planets used to look similar, back when we didn't had our devices of observation. So two attributes are used to define stars and planets. The attribute/properties of being a tiny shiny object in the night sky and movement are used to define stars. It's a rather efficent definition, but which is inconsistent with our knowledge of today. Though it's not false in any way, because it actually isn't claiming that stars are the same thing as snakes for example. The definition is just a decription of something. You can also say stars are objects bigger than planets and less dence than black wholes. But this would be a rather ugly definition as you need to know what black holes and planets are before knowing what stars are, even though we know by now that definitions of things aren't eternal and true in the sense that there is one definition of what a planet is they are rather consistent with common language and efficient, hence need fiew defining properties. My favorite way to define stars however happens to be also the most efficient way, ...
Reframing Evolution - An information wavefront traveling through time
by Joshua Clancy — 2023-07-10T22:36:46.061Z
Link: https://midflip.org/topics/reframing-evolution-an-information-wavefront-traveling-through-time

I think I came up with this idea 10 years or so ago, but part of me thinks I read it somewhere? No new idea under sun and all that.

People love stories. We want a character with a goal. So, when learning of evolution, people like to imagine characters with goals. The giraffe evolved a longer neck in order to reach higher leaves. It’s a common subtle mistake. The giraffe didn’t evolve in order to do anything, it just so happened that those tall neck giraffes were the most genetically fit within their environment. There is no purposeful design.

It’s a common error and I learned it early on. However, I was a stubborn student. I wanted a purpose! I wanted a story! So, in undergrad… I secretly doodled and reframed evolution. In my reframing, I gave evolution characters, I gave those characters purpose, and I gave them a story.

Introducing the characters of evolution

Meet DiaNA.

Diana represents the Gene Regulatory Network. Basically, all of the information in an individual that is transferred through reproduction/replication, whether that information is stored in the DNA, the RNA, or in proteins. All current forms of Diana have the following three abilities. The ability to replicate/reproduce, the ability to preserve information (to an approximately optimal degree), and the ability to influence their environment.

For some strands of life, Diana’s ability to influence her environment went nuclear. She started creating structures that allowed her to explore, gather resources, and defend herself etc. This started with single-cell organisms…

and continued with multi-cellular organisms...

We won’t talk about these characters too much here, but we will discuss them more in later articles!

The important characters here are Wavy and Wavy senior. Wavy and Wavy senior are the characters where we get creative and really start reframing evolution.

Wavy and Wavy Senior

Wavy represents all of the information within a species at one time. So, all of the information within all of the DNA (Dianas/GRN) within a species.

Wavy solves its problems with one simple trick. It is made up of a population of many individuals all slightly different. When a problem comes along, some of it’s population do better than others at addressing the problem. Those who do better, have more offspring. Thus, over time those individuals make up more of the population, and so make up more of Wavy. Wavy thus changes a...