trentmkelly/LessWrong-43k huggingface dataset:
Relying on Future Creativity
by J Bostock — 2021-11-30T20:12:43.468Z

I used to write songs for a rock band. Sometimes I'd have a song written, thinking it was my best work ever, then when it came time to rehearse, we'd realize it wasn't going to work out. Maybe we didn't have the instruments to do it justice (we were a three piece), or it was out of my comfortable vocal range, or just too technically tricky for us to get right.

Then I'd be left with this feeling like I'd wasted my best efforts, like I couldn't ever make something that good again. And I could prove it to myself! I didn't know what song I'd write that was better than the one we'd just had to ditch!

I get the same with research. Sometimes I'll come up with an idea of the next thing I want to test, or design. Then it doesn't work, or it's not practical, or its done before. And then it's tempting to feel exactly the same way: that I've bungled my one single shot.

But I've learnt to not feel this way, and it was my song writing which helped first. Now when my idea doesn't work, my first thought isn't "oh no my idea has failed", it's "OK I guess I'll have to just have another idea". I've learnt to rely on my future creativity.

This isn't easy, and sometimes it feels like walking on air. Coming up with an idea is different from other tasks, because by definition it's different every time. When I imagine playing a piece on the piano, I know exactly the form of what I'm going to do. When I imagine coming up with an idea it's a total mental blank. If I knew where I was going I'd already be there. Creativity is also a lot less reliable than physical skills, so I often have no idea how long it's going to take me to come up with something.

There's no magic bullet, but the most important thing is getting good at noticing when you're treating your future creativity as nonexistent. The biggest sign of improvement in this skill is that you don't feel like you're stepping off a cliff every time you abandon a creative endeavour, be it idea or project.
Reneging Prosocially
by Duncan Sabien (Inactive) — 2021-11-30T19:01:12.441Z

Author's note: this essay was originally published elsewhere in 2019. It's now being permanently rehosted at this link. I'll be rehosting a small number of other upper-quintile essays from my past writing over the coming weeks.

Quite often, I will make an agreement, and then find myself regretting it. I’ll commit to spending a certain amount of hours helping someone with their problem, or I’ll agree to take part in an outing or a party or a project, or I’ll trade some item for a certain amount of value in return, and then later find that my predictions about how I would feel were pretty far off, and I'm unhappy.

Sometimes, I just suck it up and stick it out. But sometimes I renege on the agreement. In this essay, I’d like to lay out a model for how to renege prosocially, i.e. how to go back on your agreements in a way that strengthens rather than damaging the social fabric (or at the very least leaves it no worse than it was).

Black boxes and APIs

Once, I was a passenger in my father’s car when another car cut us off on the freeway.

“Do you know what he did wrong?” my father asked, gesturing at the other car.

(I was maybe thirteen at the time, and would soon be going through drivers’ education.)

“Uh. He didn’t use his turn signal?”

“Close,” my father answered. “Why is it bad not to use your turn signal?”

I couldn’t come up with anything, and eventually my father continued. “It’s bad because it makes you less predictable,” he said. “Just like the fact that he raced up on us on the right, and then cut in front of us. You pass on the left and not the right because that’s what people are expecting. You signal your lane changes so that people know what you’re about to do. If you drive erratically, everybody else has to change the way they’re driving around you, and then nobody knows how to coordinate. It breaks the pattern, and we rely on the pattern to stay safe.”

(It’s no surprise that I talk the way I do, is it?)

In this essay, I’m going to generally avoid talking about things like people’s internal experiences, and people’s individual character traits, and so forth. I won’t talk about that stuff zero, but most of the time, it doesn’t bear directly on the question of what happens when people interact with each other, except when filtered through a layer of
[X internal states] → [Y external actions].

The guy in the other car could have been a jerk, or he could have been tir...
Tears Must Flow
by sapphire — 2021-11-30T18:37:57.408Z

This Thanksgiving I was with my girlfriend's family. Turkey was served. I held back tears for a little and finished my food. But I couldn't hold them back very long. I ended up going down to the basement to cry. I cried off and on for over an hour. My girlfriend held me. Eventually we went back upstairs but my mood never really improved. My girlfriend's mom was hurt. I just cannot do this anymore. I told myself I would get through thanksgving for my girlfriend's sake but I just couldn't stop crying.

In California, I met several birds who had been rescued from farms. These birds grew up in truly hellish conditions. Most chickens live in crowded amonia filled sheds. Here is a very short, very grisly video of the conditions on factory turkey farms. They never, or almost, never got to enjoy feeling safe in the sun. All of the birds I met were rescued because they were in especially terrible shape, even by the standard of birds trapped on factory farms. Despite what they endured they seemed to enjoy many parts of their lives. Several birds seemed actively consistently happy. People, including non-human people, can recover from a lot even if their scars never fully heal.

It is just so sad to see the body of a bird and think:

There was still time to say we were sorry. There was probably still time to give you a decent life. A life where you could have felt happy and safe at least some of the time. Maybe in a different world we could have sat in the sun together. We could have been friends. I had a miserable childhood but I eventually got to lead a real life. But instead of apologizing and making amends we killed you. Now you will never get to experience the life you deserved. I am so sorry...

Tears must flow.

Several activists I know have taken the liberation pledge. The pledge is pretty simple.

one | Publicly refuse to eat animals - live vegan.
two | Publicly refuse to sit where animals are being eaten.
three | Encourage others to take the pledge.

I don't really believe in taking pledges and try to avoid promises if I can. I trust my future self. But right now I am in compliance with the pledge and expect I will be in the future. Pledge takers commonly wear a bracelet made from a bent fork. Its time to bend a fork.
Machine Agents, Hybrid Superintelligences, and The Loss of Human Control (Chapter 1)
by Justin Bullock — 2021-11-30T17:35:32.607Z

“And everybody knows that the Plague is coming

Everybody knows that it's moving fast

Everybody knows that the naked man and woman

Are just a shining artifact of the past

Everybody knows the scene is dead

But there's gonna be a meter on your bed

That will disclose

What everybody knows

And everybody knows that you're in trouble

Everybody knows what you've been through

From the bloody cross on top of Calvary

To the beach of Malibu

Everybody knows it's coming apart

Take one last look at this Sacred Heart

Before it blows

And everybody knows”

-Leonard Cohen, Everybody Knows

Here, at the very beginning of our story, I need to highlight two seemingly boring and mundane points. These points are:

1. Large organizations are immensely powerful in modern society

2. They exhibit their own emergent behavior

These are my starting claims. I assume these as obvious points upon brief reflection. One so obvious as to be commonplace in our language. For example, these claims apply to the large organizations that constitute collective "Big Government", "Big Business", and "Big Science." We speak and write as if this is a generally accepted fact about our world. We say things like "The US did this for that reason," "Facebook (Meta?) did that to maximize its profit," "The National Science Foundation decided to fund research for the benefits it perceived over and above the potential costs." These two claims, I think, all on their own, should cause us pause for concern when we consider the sheer size and reach of our modern organizations (something we'll return to in a future post). Given their size, complexity, and emergent behavior, these increasingly powerful organizations are also demonstrably unaligned with the general interests of human freedom, human intelligence, and human love.

These two claims need to be taken together with one additional claim, a third important point for our story:

3. These large organizations systematically deploy machine agents to accomplish their goals

These three claims, I argue, point to the current existence and power of superintelligent hybrids. Superintelligences that are already beyond human control. These superintelligences are the large hybrid organizations of people and machines. These hybrid entities are currently more intelligent, they have more memory and more co...
Looking for reasoned discussion on Geert Vanden Bossche's ideas? +6 Months on...
by megabranflake — 2021-11-30T16:30:14.447Z

I'm again looking for reasoned discussion on Geert Vanden Bossche's ideas. This time we can compare to our discussion 6 months ago.

See: https://www.lesswrong.com/posts/mjgcFJYFFgjqod9gi/looking-for-reasoned-discussion-on-geert-vanden-bossche-s.

Has his previous theory's been proven right?

Geert has continued commenting on events.

See: https://www.voiceforscienceandsolidarity.org/scientific-blog/mass-vaccination-will-push-sars-cov-2-spike-protein-beyond-omicron

Will his current theory's been proven right?

Please upvote for visibility. It is very interesting that we can tract the performance of less wrongs communities inference over time. Many thanks :)
How do Bayesians tell what does and doesn't count as evidence (which, e.g., hypotheses may render more or less probable if true)? Is it possible for something to fuzzily-be evidence?
by LVSN — 2021-11-30T12:52:37.929Z

"The basic idea underlying most uses of Bayes’ theorem is that a hypothesis is supported by any evidence which is rendered (either sufficiently or simply) probable by the truth of that hypothesis."
— Bayesian Argumentation: The Practical Side of Probability
Self-studying to develop an inside-view model of AI alignment; co-studiers welcome!
by Vael Gates — 2021-11-30T09:25:05.146Z

tldr: It's hard for me to develop inside views of AI alignment, but I feel like I'm approximately ready for it now. So I'm developing a curriculum for myself, and I'd welcome people who want to join me in creating their own curricula and discussing progress regularly!

I’m one of those people who finds developing an “inside view” hard.

Sometimes someone will ask me about something I have expertise in, and I'll realize I don't actually understand the topic deeply enough to answer them.
Sometimes someone will ask me something, and I'll cite a bunch of perspectives from other people instead of directly knowing what I think.
Sometimes I don't know I have opinions, until I'm chatting with someone and some complicated model falls out of my mouth.
Related: When I'm talking with people with strong frames, I'll have a hard time seeing other perspectives. I get upset when non-expert people talk too confidently around me, because I feel like I can't screen their perspectives out and they'll influence the "average" perspective in my head.
Related: When I'm trying to learn something, I usually successfully learn enough to hit my goal and no more. I'm quite good at the goal-directed skill, but my intrinsic interests don't usually line up with technical content, so I don't acquire the type of deep understanding that comes from nerd-sniping.
Related: It's historically felt (socially) safer to me to defer to experts I respect, and academia probably also trained this for me.

In any case, after a lot of years of orienting around this, I now feel like I'm approximately ready to develop an inside view about AI alignment. I've consumed a lot of AI content by this point, and feel like it's about time, so I'm psyched that my psyche feels like it's finally in the right place for this.

So, because I'm one of those people who likes courses and structure: I'm developing a curriculum for myself, and I'd welcome anyone who wants to join me in creating their own and discussing progress regularly!

Vael's curriculum (optimized for me, by me)

STRUCTURE

(squints) This feels like it's probably going to be 3 months to a year.
2 weekly blog posts, due Sunday before bed.

Blog post #1: An opinion on an AI paper (fine if previously read)
Blog post #2: A personal piece on anything else (this isn't necessarily relevant to developing an inside-view, but it is relevant to keeping this sustainab

...
My take on higher-order game theory
by Nisan — 2021-11-30T05:56:00.990Z

This is how I currently think about higher-order game theory, the study of agents thinking about agents thinking about agents....

This post doesn't add any new big ideas beyond what was already in the post by Diffractor linked above. I just have a slightly different perspective that emphasizes the "metathreat" approach and the role of nondeterminism.

This is a work in progress. There's a bunch of technical work that must be done to make this rigorous. I'll save the details for the last section.

Multiple levels of strategic thinking

Suppose you're an agent with accurate beliefs about your opponents. It doesn't matter where your beliefs come from; perhaps you have experience with these opponents, or perhaps you read your opponents' source code and thought about it. Your beliefs are accurate, although for now we'll be vague about what exactly "accurate" means.

In a game of Chicken, you may want to play Swerve, since that's the maximin strategy. This is zeroth-order thinking because you don't need to predict what your opponent will do.

Or maybe you predict what your opponent will do and play the best response to that, Swerving if they'll go Straight and going Straight if they'll Swerve. This is first-order thinking.

Or maybe you know your opponent will use first-order thinking. So you resolve to go Straight, your opponent will predict this, and they will Swerve. This is second-order thinking.

Beyond second-order, Chicken turns into a commitment race.

In Prisoner's Dilemma, zeroth- and first-order thinking recommend playing Defect, as that's a dominant strategy. Second-order thinking says that it's good to Cooperate on the margin if by doing so you cause your opponent to also Cooperate on the margin. Let's say it's worth it to Cooperate with probability p if your opponent thereby Cooperates with probability more than p2 (although the exact numbers depend on the payoff matrix).

If you're a third-order agent, you might commit to Cooperating with probability equal to 51% times the probability that your opponent Cooperates. This causes your opponent to Cooperate with certainty, and thus you're bound to Cooperate with probability 51%. This is a Pareto-optimal outcome that favors you heavily.

Fixed points

So far we've talked about agents of order n+1 playing against agents of order n. What about games between agents of equal order?

If two first-order agents play the best respon...
AI Governance Fundamentals - Curriculum and Application
by Mau — 2021-11-30T02:19:59.104Z

[Edit: The course has received major updates since this post was made. See here for the most recent course information and curriculum. I've removed the largely outdated information from this post.]

While many people are interested in transformative AI governance, there is currently no scalable introduction to the field that offers substantial breadth, depth, context, accountability, and information about relevant career opportunities. Anecdotally, finding even a few of those things can be tough. Aiming to improve this state of affairs, I’m excited to introduce the AI Governance Fundamentals program. This program seeks to efficiently bring people up to speed on issues in transformative AI governance through an 11-week virtual course. It consists of 8 weeks of readings, facilitated group discussions, speaker sessions, and a 4-week capstone project, for a total of ~2-4 hours per week.
Seeking Truth Too Hard Can Keep You from Winning
by Gordon Seidoh Worley — 2021-11-30T02:16:58.695Z

LW-style rationality is about winning. Instrumentally, most rationalists think they can win, in part, by seeking truth. I frequently run into comments here where folks take truth as an effectively "sacred" concern in the sense that truth matters to them above all else. In this post, I hope to convince you that, because of the problem of the criterion, seeking truth too hard and holding it sacred work against learning the truth because doing so makes you insufficiently skilled at reasoning about non-truth-seeking agents.

Let's settle a couple things first.

What is truth? Rather than get into philosophical debates about this one, let's use a reasonable working definition that by truth we mean "accurate predictions about our experiences". This sort of truth is nice because it makes minimal metaphysical claims (e.g. no need to suppose something like an external reality required by a correspondence theory of truth) and it's compatible with Bayesianism (you strive to believe things that match your observations). Also, it seems to be the thing we actually care about when we seek truth: we want to know things that tell us what we'll find as we experience the world.

What's the problem of the criterion? Really, you still don't know? First time here? In short, the problem of the criterion is the problem that to know something means you know how to know it, but knowing how to know something is knowing something, so it creates an infinite loop (we call this loop epistemic circularity). The only way to break the loop is to ground knowledge (information you believe predicts your experiences, i.e. stuff you think is true) in something other than more knowledge. That something is purpose.

So, what's the problem with prioritizing truth? On first glance, nothing. Predicting your experiences accurately is quite useful for getting more of the kinds of experiences you want, which is to say, winning. The problems arise when you over optimize for truth.

The trouble is that not all humans, let alone all agent-like things in the universe, are rational or truth seeking (they have other purposes that ground their thinking). This means that you're going to need some skill at reasoning about non-truth-seeking agents. But the human brain is kinda bad at thinking about minds not like our own. A highly effective way to overcome this is to build cognitive empathy for others by learning to think l...
Visible Thoughts Project and Bounty Announcement
by So8res — 2021-11-30T00:19:08.408Z

(Update Jan. 12: We released an FAQ last month, with more details. Last updated Jan. 7.)

(Update Jan. 19: We now have an example of a successful partial run, which you can use to inform how you do your runs. Details.)

We at MIRI are soliciting help with an AI-alignment project centered around building a dataset, described below. We have $200,000 in prizes for building the first fragments of the dataset, plus an additional $1M prize/budget for anyone who demonstrates the ability to build a larger dataset at scale.

If this project goes well, then it may be the first of a series of prizes we offer for various projects.

Below, I’ll say more about the project, and about the payouts and interim support we’re offering.

The Project

Hypothesis: Language models can be made more understandable (and perhaps also more capable, though this is not the goal) by training them to produce visible thoughts.

We’d like to test this hypothesis by fine-tuning/retraining a language model using a dataset composed of thought-annotated dungeon runs. (In the manner of

AI dungeon

.)

A normal (un-annotated) dungeon run is a sequence of steps in which the player inputs text actions and the dungeon master responds with text describing what happened in the world as a result.

We’d like a collection of such runs, that are annotated with "visible thoughts" (visible to potential operators or programmers of the system, not to players) describing things like what just happened or is about to happen in the world, what sorts of things the player is probably paying attention to, where the current sources of plot tension are, and so on — the sorts of things a human author would think while acting as a dungeon master. (This is distinct from producing thoughts explaining what happened in the dungeon; “visible thoughts” are meant to play an active role in constructing the output.)

Once we have such a dataset, MIRI’s hope is that present or future technology will be able to train a model or models which iteratively produce visible thoughts along with storytelling, based on user actions plus previous history (including previous thoughts). The goal is to transition the state of AI dungeon technology from “An AI outputs story text in response to actions (and we have no idea how)” to “An AI produces thoughts as visible intermediates on the way to story text, allowing us to watch the AI think abou...
Soares, Tallinn, and Yudkowsky discuss AGI cognition
by So8res — 2021-11-29T19:26:33.232Z

This is a collection of follow-up discussions in the wake of Richard Ngo and Eliezer Yudkowsky's Sep. 5–8 and Sep. 14 conversations.

Color key:

Chat	Google Doc content	Inline comments

7. Follow-ups to the Ngo/Yudkowsky conversation

[Bensinger][1:50] (Nov. 23 follow-up comment)

A general background note: Readers who aren't already familiar with ethical injunctions or the unilateralist's curse should probably read Ends Don't Justify Means (Among Humans), along with an explanation of the unilateralist's curse.

7.1. Jaan Tallinn's commentary

[Tallinn][6:38] (Sep. 18)

thanks for the interesting debate! here are my comments so far: [GDocs link]

[Tallinn] (Sep. 18 Google Doc)

meta

a few meta notes first:

i’m happy with the below comments being shared further without explicit permission – just make sure you respect the sharing constraints of the discussion that they’re based on;
there’s a lot of content now in the debate that branches out in multiple directions – i suspect a strong distillation step is needed to make it coherent and publishable;
the main purpose of this document is to give a datapoint how the debate is coming across to a reader – it’s very probable that i’ve misunderstood some things, but that’s the point;
i’m also largely using my own terms/metaphors – for additional triangulation.

pit of generality

it feels to me like the main crux is about the topology of the space of cognitive systems in combination with what it implies about takeoff. here’s the way i understand eliezer’s position:

there’s a “pit of generality” attractor in cognitive systems space: once an AI system gets sufficiently close to the edge (“past the atmospheric turbulence layer”), it’s bound to improve in catastrophic manner;

[Yudkowsky][11:10] (Sep. 18 comment)

it’s bound to improve in catastrophic manner

I think this is true with quite high probability about an AI that gets high enough, if not otherwise corrigibilized, boosting up to strong superintelligence - this is what it means metaphorically to get "past the atmospheric turbulence layer".

"High enough" should not be very far above the human level and may be below it; John von Neumann with the ability to run some chains of thought at high serial speed, access to his own source code, and the ability to try branches of himself,

...