trentmkelly/LessWrong-43k huggingface dataset:
Psycho-cybernetics: experimental notes
by romeostevensit — 2018-09-18T19:21:03.601Z

This post is a series of missives and notes I took while reading a popularization of cybernetics concepts as applied to self-help that was hugely influential in the self help field when first published in 1960. I am unsure if these notes will be of any interest to others. This is not a book review or a summary, but rather my own impressions of the models that the author was trying to build up and the cross connections between those concepts and others.

In general, I wish more people would make posts about books without feeling the need to do boring parts they are uninterested in (summarizing and reviewing) and more just discussing the ideas they found valuable. I think this would lower the friction for such posts, resulting in more of them. I often wind up finding such thoughts and comments about non-fiction works by LWers pretty valuable. I have more of these if people are interested.

Why this book: If you wish to understand the box you live in, investigate records from the time it was being built. The social psychology and cognitive science results that much of the lesswrong memeplex hangs its hat on are subject to an incentive structure whereby surprising results are the ones that are promoted or made more visible. But surprising relative to what? Is there some generic folk psychology template that I am comparing to? This plays some role, but I think I have underestimated the degree to which the defaults are constructs. I wanted to get a sense of how they might have been built, which lead to an investigation of Alfred Korzybski, the first person to utter "the map is not the territory", and the inception of cybernetics as a field of discipline, which also heavily influenced the people and work that later went on at Bell Labs and thus shaped the emergence of the information age.

I found this book interesting in particular because it did not use the standard anecdote-concept format of most self-help (with perhaps 2-8 concepts in an entire book) but instead seemed much more concept dense. I do recommend reading the whole thing if these notes seem interesting.

Preface

You should do the thing that I am doing in this post with the book: generate notes from each chapter and tie them to your own concepts and experiences in order to get them to stick. The book contains blank sections for you to do this. (Interesting in that this isn't the first time I&#

...
Towards a New Impact Measure
by TurnTrout — 2018-09-18T17:21:34.114Z

In which I propose a closed-form solution to low impact, increasing corrigibility and seemingly taking major steps to neutralize basic AI drives 1 (self-improvement), 5 (self-protectiveness), and 6 (acquisition of resources).

Previously: Worrying about the Vase: Whitelisting, Overcoming Clinginess in Impact Measures, Impact Measure Desiderata

To be used inside an advanced agent, an impact measure... must capture so much variance that there is no clever strategy whereby an advanced agent can produce some special type of variance that evades the measure.

~ Safe Impact Measure

If we have a safe impact measure, we may have arbitrarily-intelligent unaligned agents which do small (bad) things instead of big (bad) things.

For the abridged experience, read up to "Notation", skip to "Experimental Results", and then to "Desiderata".

What is "Impact"?

One lazy Sunday afternoon, I worried that I had written myself out of a job. After all, Overcoming Clinginess in Impact Measures basically said, "Suppose an impact measure extracts 'effects on the world'. If the agent penalizes itself for these effects, it's incentivized to stop the environment (and any agents in it) from producing them. On the other hand, if it can somehow model other agents and avoid penalizing their effects, the agent is now incentivized to get the other agents to do its dirty work." This seemed to be strong evidence against the possibility of a simple conceptual core underlying "impact", and I didn't know what to do.

At this point, it sometimes makes sense to step back and try to say exactly what you don't know how to solve – try to crisply state what it is that you want an unbounded solution for. Sometimes you can't even do that much, and then you may actually have to spend some time thinking 'philosophically' – the sort of stage where you talk to yourself about some mysterious ideal quantity of [chess] move-goodness and you try to pin down what its properties might be.

~ Methodology of Unbounded Analysis

There's an interesting story here, but it can wait.

As you may have guessed, I now believe there is a such a simple core. Surprisingly, the problem comes from thinking about "effects on the world". Let's begin anew.

Rather than asking "What is goodness made out of?", we begin from

...
Defining by opposites
by Bucky — 2018-09-18T09:26:38.579Z

In Taboo your words Eliezer talks about how confusion occurs when people are using the same word to mean something different. Refusing to use the word in question can help with the discussion. He mainly focusses on philosophical debate but it’s a technique which works in everyday life too.

I came across an example of this. Alex claims that he makes decisions logically. Bob disagrees.

I’m fairly sure they were talking about two different things but I couldn’t think of a way of explaining this quickly enough. I realised afterwards I should have thought about defining their alternative perceptions of “logically” with their opposites.

I think for Alex the opposite would have been “haphazardly” and for Bob the opposite would have been “emotionally”.

“Logically” is loaded up with good karma and I think that this was the main cause of the argument. Alex feels Bob is unfairly robbing him of all the good karma. Bob thinks Alex is unfairly claiming all the good karma. In reality Bob is just trying to stop Alex claiming one bit of good karma (being able to make decisions unemotionally).

Now had Alex and Bob realised this I’m not sure it would have dissolved the disagreement but they would have been arguing about the same thing. If they had figured this out (or I had explained it to them) the argument may have got worse - use the technique with caution.

***

Tabooing your words takes effort. I submit a sub-tool to help if you’re in a hurry – try to think of the opposite of what you mean.

This won’t always help. In the classic tree falling in the woods argument I can’t think of opposites for the concept of “sound” which would help. However, in more day-to-day experiences I suspect the richness of language would be more likely to bear fruit.

It is probably most helpful when you have to think fast in conversations where the participants are less likely to be willing to take the time to use the full taboo your words technique.
Tips on talking about Polyamory
by ChaosHufflepuff — 2018-09-18T03:31:48.576Z

In a few weeks, I'm going to be running a 45 minute session for a small group of people on polyamory. Things to note before I get into the main body of this post:

1. I may possibly end up identifying as poly, but I have never been in a relationship and as an ace/aro person it is not something that I dwell on often. But as it stands now, I am not poly (as far as my personal experience indicates).

2. The group of people are generally quite socially progressive, and will probably be open-minded on the topic

So for a youth group seminar, I was asked to run a 45 minute session, and I thought it would be interesting to base it on polyamory. Originally I was thinking of using polyamory as a concept with which to introduce the idea of steelmanning, but this group is (for the most part) entirely unfamiliar with rationality or rationalist thinking methods, and the person coordinating the seminar thought that it would be better to leave out rationalist stuff for now, since it's not a topic of immediate interest to the group, and they will be choosing between my session and a few others.

I have a pile of reading to do before I plan this session (thanks to the Bayesian Conspiracy podcast's polyamory episode(s) link lists), but I thought it would be good to check in here and see if anyone had anything to suggest or share that would help with discussing a topic like this.

I haven't yet decided how I will format the session. It could be anything from me talking for a while to an interactive discussion.

I know I've left this quite general and open (which was intentional, since I'm looking for anything and everything people may have as input).

Thoughts?
The Case For Gods
by Justin Vriend — 2018-09-17T21:06:34.502Z

[this is a repost from my personal blog explanationing.wordpress.com. Look there for posts explaining my priors.]

I’ve been playing with belief in God since I was a child.

During my adolescent years, when I started really digging the catchphrases of pseudo-rational-types like Terry Goodkind, I became a staunch atheist. It wasn’t until after my last grandparent died and I really came face-to-face with the living faith of my family that I suspected there might something more to belief in God than had previously met my horny-for-evidence hormone-blinded eye.

So, in order to make sense of this glimmer of an intuition that there’s something to faith, I joined a Bible study in my first year at university. I talked with other young Christians about what they believed. I teased out the little glowing thread of my own experience of divinity, the divinity I saw and felt in the room at my Oma’s funeral service. By the end of my winter semester, I felt as though I had a fairly close relationship with God, and I had a good working understanding of my fellow Bible-studiers as well. However, as the semester came to a close, I gave up on the project of Christianity in favour of something else that caught my fancy, who can remember what.

Two years later, (ie., this past week), a friend of mine wondered in passing at my faith in God. The word “faith” rankled, because even when I was trying on the hat of Christianity, I never made use of faith. While I loved the people in my Bible study, and respected their personal relationships with God, my long-abiding sternly-atheistic dismay at faith persisted.

I believed then and I believe now that God can be explicitly modeled in a way that preserves both rationality and the essential sense in which spiritual people mean “God.”

Return to the idea that our brain is playing a game of “interpret reality-data into predictively useful systems of symbols.”

If that’s the game, what’s winning? If the game is as described above, winning is “correct prediction.” However, within conscious experience, our sense of winning-at-life and correct prediction are rarely equivalent.

For example, when I look at my girlfriend’s eyes while she looks into mine, and the euphoria of love bubbles up inside of me, prediction is the farthest thing from my mind. That’s not to say the predictive model of cognition can’t, at some level, capture why I love my girlfriend. Howe...
HLAI 2018 Talks
by Gordon Seidoh Worley — 2018-09-17T18:13:19.421Z
Link: https://slideslive.com/humanlevel-ai-2018

The keynote and general interest talks from HLAI 2018 are available online, along with a few others. Sadly the panel I participated in was not fully recorded, but there is a short video of an excerpt of it. If you want to watch just one video I think the most relevant one to folks here will be the panel discussion on beneficial AI.
Alignment Newsletter #24
by Rohin Shah — 2018-09-17T16:20:01.955Z

Starting from this week, Richard Ngo will join me in writing summaries. His summaries are marked as such; I'm reviewing some of them now but expect to review less over time.

Highlights

Introducing the Unrestricted Adversarial Examples Challenge (Tom B. Brown et al): There's a new adversarial examples contest, after the one from NIPS 2017. The goal of this contest is to figure out how to create a model that never confidently makes a mistake on a very simple task, even in the presence of a powerful adversary. This leads to many differences from the previous contest. The task is a lot simpler -- classifiers only need to distinguish between bicycles and birds, with an option of saying "ambiguous". Instead of using the L-infinity norm ball to define what an adversarial example is, attackers are allowed to supply any image whatsoever, as long as a team of human evaluators agrees unanimously on the classification of the image. The contest has no time bound, and will run until some defense survives for 90 days without being broken even once. A defense is not broken if it says "ambiguous" on an adversarial example. Any submitted defense will be published, which means that attackers can specialize their attacks to that specific model (i.e. it is white box).

My opinion: I really like this contest format, it seems like it's actually answering the question we care about, for a simple task. If I were designing a defense, the first thing I'd aim for would be to get a lot of training data, ideally from different distributions in the real world, but data augmentation techniques may also be necessary, especially for eg. images of a bicycle against an unrealistic textured background. The second thing would be to shrink the size of the model, to make it more likely that it generalizes better (in accordance with Occam's razor or the minimum description length principle). After that I'd think about the defenses proposed in the literature. I'm not sure how the verification-based approaches will work, since they are intrinsically tied to the L-infinity norm ball definition of adversarial examples, or something similar -- you can't include the human evaluators in your specification of what you want to verify.

The What-If Tool: Code-Free Probing of Machine Learning Models (James Wexler): When you train an ML model, it is often h...
Team Cohesion and Exclusionary Egalitarianism
by lionhearted (Sebastian Marshall) — 2018-09-17T04:48:33.894Z

I. Teamwork: that shouldn't be too hard, eh?

Sooner or later, most people who want to make a difference in the world start thinking about how to have highly effective teams, teamwork, and collaboration.

On the surface, it seems like it should be simple and straightforward — if you and someone else both believe in the same cause, you're both competent at your individual roles, and you get along well between the two of you, you should be able to be an effective team.

Finding a third member of the team would just mean getting someone else who cares about the cause, is competent, and gets along with the two of you. And so on. Building teams should be easy and straightforward, no?

No.

Ah, were it so simple! But anyone who actually attempts to do so, finds out that it's not the case.

I've been kicking around an idea about effective organizations for a year or two now, drawing on both personal experience in team-building and hiring and management, as well as historical case studies.

I was prompted to write this after reading the (excellent) A Dialogue on Rationalist Activism —

"Well, for one thing, people would automatically pattern-match pretty much any attempt at forming an organizational structure to a "religion" or a "cult" even though what's actually being attempted is the literal opposite of those things. When it comes to actual formal documents specifying the objectives and structure of the organization, people would get endlessly caught up in relatively inconsequential choices of language or focus, perpetually bickering over the last 1% of linguistic distinction that separates their aims. You would think people who prize rationality would be able to shield themselves from the narcissism of small differences, but I suspect not, in reality."

Now, if you'd never actually tried to build a team in the real world to work on a problem, you might think seems insane. "...perpetually bickering over the last 1% of linguistic distinction that separates their aims..." — that wouldn't actually happen, right? What a waste of time, no?

II. Looking at the problem from different angles

And yet, you set out into the world to build a team and get collaboration going at scale, and, well, this often happens.

There's a lot of ways you could analyze this problem and look to overcome it.

You might look at it as a p...
Berkeley REACH Supporters Update: September 2018
by stardust — 2018-09-16T23:56:30.973Z

Since our big fundraising push at the end of June (thanks!), REACH has been making progress on a number of fronts.

Finances & Structure

The June fundraising push let us commit to a 13-month lease (Aug 1, 2018 - Aug 31, 2019).

In June, we raised about $18k in one time donations, which allowed us to pay the $11k deposit for the space and have a bit left over for administrative expenses as well as saving some for the future.

Since late June, the Patreon monthly pledge total has fluctuated between $4,400 and $5,600 (with some people adjusting pledges up and some down), currently at $4,478. We have continued renting out rooms, bringing in another ~$1.5k/month. This allowed us to fund some improvements to the space on top of paying rent from July - September! (See below for specifics.) We also recently heard that we will likely be receiving a grant which will allow us to pay a part time manager for the space! (Details forthcoming once it is officially announced.)

REACH incorporated as a non-profit (Berkeley Rationality and Effective Altruism Community Hub Incorporated) in May with a three-member board: Sarah (Stardust) Spikes, Andrew Rettek, and Doe Dearr. We have received our Employer Identification Number (EIN) which has allowed us to open a bank account for REACH. The board began the application for 501(c)3 status on 8/2; we have heard that it takes anywhere from 3 to 9 months to be fully processed.

In August, Stardust started an external (mostly remote) 20hr/week job, but is still the Executive Director. We are working on setting up payroll for a part-time manager for the space.

Survey & Stakeholder Meeting

In July, we ran a survey (around 40 responses,

detailed analysis here

) and a stakeholder meeting (around 30 attendees). Here’s some of the feedback we heard:

You weren’t sure when events were happening, when it was okay to drop in, and whether you had to be quiet during coworking hours.

We started announcing and running intentional coworking days at least once a week, which include facilitated (quiet) pomodoros with announced breaks for chatting and sometimes a group lunch expedition. Room C is generally open for people who want to chat or take video calls during poms.

There is less-structured coworking with no quietness rules from 11am to 5pm on weekdays whenever there is not a structured day scheduled, though if everyone present decides to start doing

...
Fundamental Philosophical Problems Inherent in AI discourse
by AlexSadler — 2018-09-16T21:03:04.789Z

Historically AI had several winters, where hype bubbles rose and collapsed. It is also full of pronouncements or predictions such as

“We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.”

There has also been an important philosophical confusion that plagued AI in the 1970s. It has been discussed in depth in a great text “When Artificial Intelligence Meets Natural Stupidity”, as well as here. The confusion was around attempting to represent knowledge in diagrams like these:

STATE-OF-MIND
^
| IS-A
|
HAPPINESS

An even more silly example of “knowledge”

HITLER
^
| IS-A
|
NIXON

From the paper:

Ø the tendency to see in natural language a natural source of problems and solutions. Many researchers tend to talk as if an internal knowledge representation ought to be closely related to the "corresponding" English sentences; and that operations on the structure should resemble human conversation or "word problems".

This is a good example of how a serious philosophical confusion about how much knowledge can be “contained in” a statement drove a large amount of research into systems that later proved unworkable.

What philosophical assumptions is AI research fundamentally mistaken about today?

Right now, the dominance and usefulness of current deep learning neural networks makes it seem that they capture some key pieces of general reasoning and could lead us to a more general intelligence. Sarah Constantin disagrees. David Chapman isn’t sure either.

I wish to highlight several other politico-philosophical problems that are inherent in writing about AI and AI safety. If the AI researchers of the past believed that writing “happiness is-a state of mind” lead to good reasoning, there are likely similar issues that occur today.

The current big problems are:

1. Misunderstanding of “bias”

2. Contradictory treatment of human trustworthiness

3. Deep confusions about the nature of “values”

4. Overemphasized individuality and misunderstanding the nature of status

5. Overreliance on symbolic intelligence, especially in the moral realm

6. Punting the problems elsewhere

1. Human Bias

The idea that humans are “biased” doesn’t sound too controversial. That description usually arises when a person fails to fulfill a model of rationality or ethics th...
Realism about rationality
by Richard_Ngo — 2018-09-16T10:46:29.239Z
Link: http://thinkingcomplete.blogspot.com/2018/09/rational-and-real.html

Epistemic status: trying to vaguely gesture at vague intuitions. A similar idea was explored here under the heading "the intelligibility of intelligence", although I hadn't seen it before writing this post. As of 2020, I consider this follow-up comment to be a better summary of the thing I was trying to convey with this post than the post itself. The core disagreement is about how much we expect the limiting case of arbitrarily high intelligence to tell us about the AGIs whose behaviour we're worried about.

There’s a mindset which is common in the rationalist community, which I call “realism about rationality” (the name being intended as a parallel to moral realism). I feel like my skepticism about agent foundations research is closely tied to my skepticism about this mindset, and so in this essay I try to articulate what it is.

Humans ascribe properties to entities in the world in order to describe and predict them. Here are three such properties: "momentum", "evolutionary fitness", and "intelligence". These are all pretty useful properties for high-level reasoning in the fields of physics, biology and AI, respectively. There's a key difference between the first two, though. Momentum is very amenable to formalisation: we can describe it using precise equations, and even prove things about it. Evolutionary fitness is the opposite: although nothing in biology makes sense without it, no biologist can take an organism and write down a simple equation to define its fitness in terms of more basic traits. This isn't just because biologists haven't figured out that equation yet. Rather, we have excellent reasons to think that fitness is an incredibly complicated "function" which basically requires you to describe that organism's entire phenotype, genotype and environment.

In a nutshell, then, realism about rationality is a mindset in which reasoning and intelligence are more like momentum than like fitness. It's a mindset which makes the following ideas seem natural:

The idea that there is a simple yet powerful theoretical framework which describes human intelligence and/or intelligence in general. (I don't count brute force approaches like AIXI for the same reason I don't consider physics a simple yet powerful description of biology).
The idea that there is an “ideal” decision theory.
The idea that AGI will very likely be an “agent”.
The idea that Turing machines and Ko

...