trentmkelly/LessWrong-43k huggingface dataset:
Walking and Navigating
by qazzquimby — 2021-10-25T01:54:16.421Z

This is the first time I've written something in public, so a portion of my brain is shouting that I'm embarrassing myself.

This post is intended to take some common-sense ideas about foresight during project work, and make them more concretely actionable. It could definitely be improved.

Effort on a task is useful to to the extent that it either brings the task towards completion, or gives you information about the best path forward. I'll call these making progress, and gaining assurance (the best word I've found for "justified confidence"). I like to think of them as walking and navigating.

Some effort gets lost doing neither.

Carefully proofreading a paragraph that you later cut.
Making a tool for someone that, by misunderstanding, doesn't do what they want.
Creating a big exact plan that falls apart when the requirements change.
Giving up on a project before releasing it.

The effort is lost because they worked without enough assurance. They risked their work without enough evidence to back up the gamble.
The overplanning example was trying to avoid against exactly this, with a comprehensive plan, but they didn't have enough evidence to *plan* so far.

Gaining assurance is like improving form, making further efforts more efficient. Having more assurance improves the expected value of the work you produce.

Gaining Assurance:

For your project, identify the most important potential failures, looking first for large scale risks (no one wants this product, this idea is based on a flawed premise) and working down to small, local risks in whatever you're doing.

Try Pre-Postmortem. Think of the most important questions. What would you be concerned about if you were hearing someone else talk about this project?
Later in the project the risks should be much smaller, because the larger ones should have already been managed.
- Rush testing high-risk decisions to learn fast.
(Related tangent, delay low-risk decisions for as long as is practical. They provide little assurance, and later on you'll have more information to make a better decision.)

Find at least one way to decrease uncertainty around the risk, like experiments, prototypes/drafts/outlines, or contemplation.

Aim to maximize information, cutting the possibility space down aggressively.
Favor low fidelity. Avoid any polish that doesn't efficiently increase the information you receive. A rough experiment c

...
Self-Integrity and the Drowning Child
by Eliezer Yudkowsky — 2021-10-24T20:57:01.742Z

(Excerpted from "mad investor chaos and the woman of asmodeus", about an unusually selfish dath ilani, "Keltham", who dies in a plane accident and ends up in Cheliax, a country governed by D&D!Hell. Keltham is here remembering an incident from his childhood.)

And the Watcher told the class a parable, about an adult, coming across a child who'd somehow bypassed the various safeguards around a wilderness area, and fallen into a muddy pond, and seemed to be showing signs of drowning (for they'd already been told, then, what drowning looked like). The water, in this parable, didn't look like it would be over their own adult heads. But - in the parable - they'd just bought some incredibly-expensive clothing, costing dozens of their own labor-hours, and less resilient than usual, that would be ruined by the muddy water.

And the Watcher asked the class if they thought it was right to save the child, at the cost of ruining their clothing.

Everyone in there moved their hand to the 'yes' position, of course. Except Keltham, who by this point had already decided quite clearly who he was, and who simply closed his hand into a fist, otherwise saying neither 'yes' nor 'no' to the question, defying it entirely.

The Watcher asked him to explain, and Keltham said that it seemed to him that it was okay for an adult to take an extra fifteen seconds to strip off all their super-expensive clothing and then jump in to save the child.

The Watcher invited the other children to argue with Keltham about that, which they did, though Keltham's first defense, that his utility function was what it was, had not been a friendly one, or inviting of further argument. But they did eventually convince Keltham that, especially if you weren't sure you could call in other help or get attention or successfully drag the child's body towards help, if that child actually did drown - meaning the child's true life was at stake - then it would make sense to jump in right away, not take the extra risk of waiting another quarter-minute to strip off your clothes, and bill the child's parents' insurance for the cost. Or at least, that was where Keltham shifted his position, in the face of that argumentative pressure.

Some kids, at that point, questioned the Watcher about this actually being a pretty good point, and why wouldn't anyone just bill the child's parents' insuranc...
Phil Trammell on Economic Growth Under Transformative AI
by Michaël Trazzi — 2021-10-24T18:11:17.694Z
Link: https://youtu.be/2GCNmmDrRsk

This is a transcript with slides for the latest episode (audio, youtube) of "The Inside View", a podcast I host about the future of AI progress. I interview Phil Trammell, an Oxford PhD student in economics and research associate at the Global Priorities Institute. Phil was my roommate and last time I called him he casually said that he had written a literature review on the econ of transformative AI. A few weeks ago, I decided that I would read that report and translate what I learn along the way to diagrams.

As someone with a background in CS/AI who did not know much about econ, I found this conversation insightful to think about different takeoff scenarios and what could cause them.

Michael: thanks for being at the podcast, it's been a few months since we've wanted to do this. Today we're going to be talking about the paper you published on the GPI website a year ago called "Economic Growth under Transformative AI". That's a paper you co-wrote. Can you tell the audience a bit about what is this GPI, for those who don't know, and what you do there

Phil: right so GPI is an institute at Oxford doing research in philosophy and economics, mainly economic theory, at least at the moment, relevant to people who want to do the most good. So philanthropists in the EA community would be a sort of obvious audience. I'm an econ PhD student at Oxford and I'm sort of being sponsored by GPI and I work part-time at GPI... but mainly my work just overlaps with GPI's goals. So the econ research I do as a grad student is the sort of work that GPI wants done. I sometimes do kind of outreach and so on through GPI but mainly the two hats just sort of overlap. And one of the things I did in my capacity is an econ researcher, which is now on the GPI website as you just mentioned, is this literature review. It's not really an original paper but a synthesis of work that has been done over the past few years on AI and economic growth.

Michael: and that's perfect because the podcast is about AI, so people I think will be kind of familiar with AI or AGI, or those concepts, maybe not with transformative AI if they are not effective altruists, so could you maybe define what you mean by "transformative AI" in this paper?

Phil: people have used that term in slightly different ways. What I mean by it is AI that has a transformative effect, and I'll define that in a moment, on the economic gr...
P₂B: Plan to P₂B Better
by Ramana Kumar — 2021-10-24T15:21:09.904Z

tl;dr: Most good plans involve taking steps to make better plans. Making better plans is the convergent instrumental goal, of which all familiar convergent instrumental goals are an instance. This is key to understanding what agency is and why it is powerful.

Planning means using a world model to predict the consequences of various courses of actions one could take, and taking actions that have good predicted consequences. (We think of this with the handle “doing things for reasons,” though we acknowledge this may be an idiosyncratic use of “reasons.”)

We take “planning” to include things that are relevantly similar to this procedure, such as following a bag of heuristics that approximates it. We’re also including actually following the plans, in what might more clunkily be called “planning-acting.”

Planning, in this broad sense, seems essential to the kind of goal-directed, consequential, agent-like intelligence that we expect to be highly impactful. This sequence explains why.

One Convergent Instrumental Goal to Rule them All

Consider the maxim

“make there be more and/or better planning towards your goal.”

This section argues that all the classic convergent instrumental goals are special cases of this maxim.

To flesh this out a little, here are some categories of ways to follow the maxim. Remember that a planner is typically close (in terms of what it might affect via action) to at least one planner – itself – so these directions can typically be applied in the first case to the planner itself.

Make the planners with your goal better at planning. For example, get them new relevant data to work with¹, get them to run faster or more effective algorithms, build protections against value drift, etc.
Make the planners with your goal have better options. For example, move them to better locations, get them more resources, get them more power or a greater number of options to select from, have them take steps in an object-level plan towards the goal.
Make there be more planners with your goal. For example, keep yourself running and aligned with your goal, acquire delegates and subordinates, convince followers and converts, build successors.

Reviewing Omohundro's “

The Basic AI Drives

” and Bostrom’s “

The Superintelligent Will

,” we extract a list of convergent instrumental goals, and find that they are all instances of the maxim “make there be more/better planners for the...
Deleted comments archive?
by Said Achmiz — 2021-10-24T11:19:43.462Z

This is a question for the Less Wrong moderation team.

Suppose that I post a comment on a post, and the author of that post deletes that comment. Now, that comment isn’t lost, per se (as its contents are sent to me via private message), but it’s no longer publicly accessible. This is likely to be unfortunate, from my perspective, as I sometimes write comments with the intention of having something I can then link to (and often the comment-thread context is relevant to the comment’s meaning).

Would it be permissible to make a post (of the “personal” type) where I re-post any of my comments that are deleted in this fashion? (I am assuming here that the reason for the comment’s deletion wasn’t anything like “having this comment on the site is illegal”, “this comment reveals privileged information”, etc. Likewise, I am assuming that any such comment does not violate any general site rules—despite that it may violate the commenting/moderation guidelines specified by a given post author—and so its presence on the site is not inherently problematic.)
Nanosystems are Poorly Abstracted
by J Bostock — 2021-10-24T10:44:27.934Z

To do something, you need energy, and you need information. Energy to do the thing, and information to define what's being done. The mental image I have of a "robot", has inputs of both continuously. The input of energy comes as electricity, and the input of information comes in the form of digital instructions. They're both abstracted away from the sort of work being done.

A missile (some kinds at least) takes in digital information, and converts that to a location. It then flies there. But when it does it gets its energy from a specific source: fuel. It can't do anything with the fuel except fly forwards.

A motor takes in abstracted energy but not information. The energy input is electric, but the motor can be used for lots of different things. The information it takes in, however, is entirely related to energy flow.

A wrench does neither. The energy input isn't abstracted, and neither is the information input.

Direct/Specific Information	Abstracted/General Information
Direct/Specific Energy Input

Category 1:

Wrench

Category 2:

Missile

Abstracted/General Energy Input

Category 3:

Motor

Category 4:

Robot

To get to generality, we need category 4. We can also define a category 0: which are non-energy-transmitting devices. These are structural objects like bridges.

So how does this tie in to nanotech?

The ultimate goal of nanotechnology is atom-precise, scalable production. "programmable matter". This would be the second most influential bit of technology developed this century. I've often wondered what's up with nanotech, and what needs to be done to advance the field.

To do this, it needs to be general. To be general we need to get to category 4 in the bottom right part of that diagram.

Types of Nanosystems

There are lots of possible ways of making very small things do things.

Miniaturized Physical Systems

Here I will include various systems which are typically not atom-precise, such as nanometer-sized lipid droplets (like those used in RNA vaccines), and nanoscale materials like carbon nanotubes. These almost always fall into category 0. Some of these nanomaterials are interesting in that they can respond to stimuli (like changing volume upon pH change, or being part of a battery which charges and discharges) but this only moves them to category 1.

It might be worth mentioning the existence of systems like the ones which allow manipulation of single atoms. For var...
Frankfurt Declaration on the Cambridge Declaration on Consciousness
by Jan — 2021-10-24T09:54:13.127Z
Link: https://universalprior.substack.com/p/frankfurt-declaration-on-the-cambridge

TL;DR: I admit that I have no idea what consciousness is. I communicate this in the form of a declaration.

A fateful conference dinner…

#

IAN

says: The problem with being a "good person" is that you never have enough energy left over for doing really evil things.

Imagine the following: It is the year 2012 and you have been invited to a fancy conference dinner in Cambridge. There is champagne and you are feeling great. Stephen Hawking is here, good grief! You feel like you have made some real progress today, with so many interesting talks and discussions. You even managed to plug your latest paper to one of the conference chairs. All is well. Suddenly a sheet of paper drifts into your field of view. A declaration on the topic of the conference - animal consciousness. You eagerly sign (eleven other people already did). It's probably alright, you think. Science is about discourse. We are just stating our position...

A picture of a fancy conference dinner with champagne.

CLIP Guided Diffusion

.

In 2012, a group of neuroscientists signed the Cambridge Declaration on Consciousness, which "unequivocally" asserted that "humans are not unique in possessing the neurological substrates that generate consciousness."

Wikipedia

… and its predictable aftermath

The subsequent news cycle is as wild as you'd predict. Here’s

New Scientist

:

"Animals are conscious and should be treated as such".

and

LiveScience

:

"After 2,500 Studies, It's Time to Declare Animal Sentience Proven".

The declaration has also made it into the reports of some of our favorite

effective altruism

organizations. Here’s the

Cause Profile

on effectivealtruism.org:

"Many experts now believe that animals have conscious experiences"

and (in a footnote of) the Founder's Pledge

Animal Welfare Cause Report

:

Similarly, the Cambridge Declaration on Consciousness, an important collection of “unequivocal” statements made by a group of prominent neuroscientists in 2012, noted that “the weight of evidence indicates that humans are not unique in possessing the neurological substrates that generate consciousness”.

However, not everybody is as convinced by the declaration. Luke Muehlhauser is characteristically awesome by providing a nuanced take in the

Report on Consciousness and Moral Patienthood

, listing three caveats:

1. The document reads more like a political document than a scientific document. (See e.g.

this

commentary.)
2. As far a

...
Deliberate Play
by jsteinhardt — 2021-10-24T02:50:16.947Z

Deliberate practice is an important concept in sports and other high-performance endeavors. It is "practicing with a clear awareness of the specific components of a skill we’re aiming to improve and exactly how to improve them." If you haven't seen this idea before, stop and go read this article (where I got the definition from) or here for a shorter guide. No, really--go read about it. It's one of the most powerful ideas I've come across.

To recap, deliberate practice focuses on improving a specific skill with an awareness of how it translates to a game situation. This is commonly done by executing a drill while visualizing a game-like situation (e.g. imagining defenders you are avoiding) and otherwise replicating game conditions (e.g. practicing free throws after you just ran sprints, to simulate fatigue).

Beyond drills, you can also bring deliberate practice to games by setting a specific intention of what to work on ("practice playing good defense in situation X"), and focusing on that throughout the game. I play pick-up games with other club players a couple times a week, and most of us set some intention going into the game.

However, this post is not about deliberate practice but about two adjacent mental stances, the first of which I'll call max performance. To describe it, consider two contrasting mental stances one could have during a game:

Deliberate practice. Try to improve specific skills. It's okay to make suboptimal plays or otherwise mess up in the process of practicing those skills, but otherwise you want to play well and at high intensity to simulate realistic competition.

Max performance. Play as well as possible. You will likely be very focused (but ideally in a flow-like state) and somewhat pushing your physical boundaries. On the other hand, you are specifically not pushing the boundaries of your skill set and instead relying on what you can do consistently.

Max performance is what you do if you are trying to win that particular game. Many people's image of "trying really hard" is actually not max performance, because they are "playing outside themselves" and trying to do things that they didn't practice enough.

If your goal is long-term growth, you should primarily be doing deliberate practice and not max performance. But max performance is also important, to make sure that skills are translating and because there's no substitute for a f...
Towards Deconfusing Gradient Hacking
by leogao — 2021-10-24T00:43:32.916Z

[Epistemic status: brainstorming, less confused than last time since it seems to provide for a nice taxonomy of things to look into rather than just a bunch of random loose threads.]

This post would not exist without the EleutherAI discord and all the wonderful people there. Special thanks to Evan Hubinger, John Schulman, and Adam Shimi for providing feedback on drafts of this post.

The goal of this post is to identify a different way of framing the gradient hacking problem, connect it to our intuitions, and use it to create a rough taxonomy of different types of potential gradient hacking. Hopefully, using this taxonomy, we can tackle more well-defined subproblems to narrow down the space of things that could potentially gradient hack.

When we think about gradient hacking, the most intuitive framing is to consider some kind of agent embedded inside a larger network (like a GPT) that somehow intentionally modifies the loss landscape of the larger network with respect to the base loss, and that this modification makes it so that in optimizing for the base objective, the base optimizer also happens to optimize the mesaobjective. Here I consider the base objective to be a function Θ→R from the params of the network to the reals, that has all the training data baked in for simplicity, and the mesaobjective another function Θ→R, possibly with some constraint that both objectives have to be indifferent between models which behave the same on all inputs. The "somehow" is often considered to be some kind of perturbing or otherwise making the output of the larger network worse whenever the mesaobjective isn't met, therefore creating an incentive for gradient descent to improve the mesaobjective. One example of this line of thinking can be found in my last post about gradient hacking. Unfortunately, I think there are some confusions with this framing.

Full loss landscapes

First off, the base loss landscape of the entire model is a function Θ→R that's the same across all training steps, and the configuration of the weights selects somewhere on this loss landscape. Configuring the weights differently can put the model on a different spot on this landscape, but it can't change the shape of the landscape itself.

Note that this doesn't contradict the interpretation of the gradient hacker as having control over the loss landscape through subju...
T-Mobile: Spurious Account Takeover Warning
by jefftk — 2021-10-23T15:30:06.030Z

This morning, reading in bed, I got a very worrying notification:

This is the notification you would receive if someone was in the
process of taking control of your phone number, which could then give
them access to other accounts where you had used that phone number as
a backup or for two-factor authentication. So I was very concerned!

In case this was a different sort of scam, however, I wasn't about to
call the phone number (which could be anyone) but I visited the
website and talked to someone over chat. They confirmed that my pin
had been changed, but also said that since I have a prepaid account
they couldn't tell me more than that. They told me to call T-Mobile
customer support at 611.

When I called 611, they looked into it, and said that this was an
automatic message sent as part of migrating my account to a new
billing system. They confirmed no one had reset my pin other than
their automated system.

I'm disappointed in T-Mobile for either not realizing their migration
would trigger this message, or deciding to go ahead with it despite
the user impact.