trentmkelly/LessWrong-43k huggingface dataset:
Question about Test-sets and Bayesian machine learning 
by  — 2021-08-09T17:16:56.777Z

What does “test-set performance” represent in Bayesian machine learning? In Bayesian ML:

we have some data D={(y1,x1),…,(ym,xm)}
we assume a model M (this includes any assumptions we make about the prior densities)
and we compute the posterior predictive density p(~y|~x,D,M)

I have seen people argue that we need a test-set to compare between two models M1 and M2 as we do not know what “the one true model” is. I don’t fully understand how “evaluating performance” on “out-of-sample” data helps us with comparing two models but isn’t this what the quantity p(M|D) is for?
The Reductionist Trap
by J Bostock — 2021-08-09T17:00:56.699Z

This used to be called "Non-Reduction and Nonapples" but thanks to a comment by Slider I've realized this put the emphasis in the wrong place.

Selling nonapples is a great piece about how critiquing a certain technique, or process, or system can often sneak into proposing a non-existent alternative. This new system, as it is overly general, fails to address the problems that the first exists to solve.

I think reductionism vs non-reductionism is sometimes a similar debate, but sometimes not. First let's distinguish between philosophical reductionism, and practical reductionism.

Philosophical reductionism is (sort of) the belief that complex systems act the way they do because of the simpler actions of simpler systems that make them up. I think that this is pretty solidly true in our universe, and I won't discuss it further.
Practical reductionism is the belief that the best and most effective way to study complex systems is to study their components individually, then build up to a larger model of the whole system.

(As an aside I actually think a lot of the reductionism vs non-reductionism debate suffers from a motte-and-bailey issue around this)

Microbiological Ecosystems

An example of practical reductionism is the study of microbiological ecosystems: there are lots and lots of different microbes in any given area (such as the sea, the soil, your intestines) which all require different nutrients and produce different sorts of waste. Some photosynthesize, some turn atmospheric nitrogen into a form which is bioavailable, and some break down carbon-containing material. Clearly if we knew everything about every type of microbe present we could understand the ecosystem. Unfortunately, unlike in macroscopic ecosystems, it's difficult to study a single microbe's behaviour while it is inside the ecosystem. They're just too small! So the standard technique is to get a pure culture, containing just one sort of microbe, and study the behaviour of that culture.

Unfortunately, this often doesn't work very well. Microbes in culture can behave differently from how they normally behave in general systems. Lots of microbes are either impossible to form a pure culture of, or require some incredibly specific nutrient only produced by another microbe, that we don't know the identity of. This means the standard "practical reductionist" technique is doomed to failure.

Here people often li...
Goal-Directedness and Behavior, Redux
by adamShimi — 2021-08-09T14:26:25.748Z

Beyond past confusions

Over the last year, I wrote and thought many confused and confusing ideas on the relationship between goal-directedness of behavior. In the linked post for example, I defended a deconfusion of goal-directdness solely in terms of behavior; in doing so, I might pass for a behaviorist (someone thinking that mental constructs are not needed and so don't exist), or look like I imply that we should never use internal knowledge of our models to determine goal-directedness. Without even mentioning the factual errors.

So here is my attempt at a short and clear explanation of the link I see between goal-directedness and behavior. If you're confused by this take, or believe me to be confused, I would really appreciate a comment. My goal isn't to prove that I'm obviously right, just to get less confused and hopefully help lift the fog of confusion for everyone.

Thanks to Jack Koch for a recent discussion that reminded me of this issue, and to Richard Ngo for giving me food for thought on this subject with his comments.

Behavior in all its glory

What's my take? I think that when we talk about goal-directedness, what we really care about is a range of possible behaviors, some of which we worry about in the context of alignment and safety. We might for example think that goal-directed systems have convergent subgoals, which tells us how they could lack corrigibility and cause catastrophic outcomes. that such a goal-directed system could follow.

My entire point is that for deconfusing goal-directedness, we want a better understanding of this range of behaviors. At the moment, when thinking about a given behavior, I don't know whether that's the sort of thing a goal-directed system would do. And it seems problematic both for understanding the risks of goal-directed systems, and for detecting them.

Note that even a purely structural definition of goal-directedness would constrain the structure such that the system behave in a certain way. So even if we want a structural definition, clarifying the range of behaviors sounds like progress.

What I'm not saying

We shouldn't ascribe any cognition to the system, just find rules of association for its behavior (aka Behaviorism)

That's not even coherent with my favored approach to goal-directedness, the intentional stance. Dennett clearly ascribes beliefs and desires to beings and systems; his point is that the ascription

...
A new definition of "optimizer"
by Chantiel — 2021-08-09T13:42:39.705Z

Here, I provide a new definition of "optimizer", as well as better explain a previous one I gave. I had previously posted a definition, but I think the definition I gave was somewhat wrong. It also didn't help that I accidentally wrote my previous definition incorrectly, and only days later realized I had written out the definition mistakenly (and then the modified definition contained a new mistake.) The way I presented my definition in the last article was also overly confusing, so I'll try to explain it better.

First, I'll present a someone else's proposed definition of "optimizer" and explain how my definitions are intended to improve upon is. Then, I'll summarize the definition I (meant to) give in my previous post, point out some potential problems, and then outline changes that could fix them. However, these modification make the definition more complicated, so I then I provide a new, though informal, definition of optimizer that eloquently avoid the issues with the original definition. Then, I compare my definition to a previous proposed definition of "optimizer" and explain how my definition is intended to improve upon it. Afterwards, I explain why defining "optimizer" doesn't matter much, anyways.

Alex Flint's definition

The article, "The ground of optimization" defines "optimizer", saying:

An optimizing system is a system that has a tendency to evolve towards one of a set of configurations that we will call the target configuration set, when started from any configuration within a larger set of configurations, which we call the basin of attraction, and continues to exhibit this tendency with respect to the same target configuration set despite perturbations.

Though this definition has some value, it is extremely broad and includes things that wouldn't be normally considered optimizers. For example, it considers toasters to be optimizers. Specifically, consider a toaster left unattended toasting a piece of bread. No matter how you reconfigure the toast within the bread, it will still end up getting toasty or burnt. So the bread robustly evolves towards being toasty or burnt and is thus, by the above definition, an optimizer.

Similarly, consider a rocket traveling through space that's set the burn through all its fuel. Consider the target set of configurations to be "has no fuel left". The rocket robustly evolves towards that state. No matter where ...
Great Negotiation MOOC on Coursera
by eg — 2021-08-09T12:23:55.268Z

Course: https://www.coursera.org/learn/negotiation
Tldr: If you're young and haven't studied negotiation, it's probably worth spending ~10hr engaging with this course when you get a chance.
The MOOC is based on the first course in Yale's MBA program apparently. He claims it isn't watered down, but probably a smart 10yo could follow it.
Probably the best MOOC I've found in terms of usefulness X content quality X accessibility/engagingness.
It starts by establishing a simple model of negotiation as seeking equally beneficial positive-sum interactions and gradually addresses complications. It even explores some techniques that are a bit on the "dark arts" side.
What should rationalists call themselves?
by Nathan Young — 2021-08-09T08:50:07.161Z

It seems there is dissatisfaction with the term "rationalist". I've listed some below.
What word should rationalists use to identify themselves and have others identify them?
Note that rationalist will be an option so people can show a preference for the status quo.
Yudkowsky: https://twitter.com/esyudkowsky/status/1424491292189364224?s=21
Tyler Cowen: https://www.lesswrong.com/posts/nsSmscB5CuSQX9arn/responses-to-tyler-cowen-on-rationality
Against "blankfaces"
by philh — 2021-08-08T23:00:04.126Z

(Content note: minor spoilers for Harry Potter and the Order of the Phoenix.)

Scott Aaronson writes about blankfaces,

anyone who enjoys wielding the power entrusted in them to make others miserable by acting like a cog in a broken machine, rather than like a human being with courage, judgment, and responsibility for their actions. A blankface meets every appeal to facts, logic, and plain compassion with the same repetition of rules and regulations and the same blank stare—a blank stare that, more often than not, conceals a contemptuous smile.

I want to push back against this a bit.

First, one of the defining aspects of blankfacedness is their internal experience. It's someone who enjoys wielding their power. This is a very hard thing to judge from the outside.

I used to work in a cinema. One day a mother came in with her young child, perhaps fivish years old. She was late for a busy screening, and the person selling tickets warned they might not be able to sit together. She said that was fine, bought popcorn and went in. Soon afterwards she came back out, complaining that they couldn't sit together. She wanted a refund for the tickets (fine) and popcorn (not fine, but she insisted). The conversation between her and my manager escalated a bit. I don't remember who brought up the police, but she at least was very confident that she knew her rights and the police would back her up if they arrived. Eventually he gave her a refund.

If it had been up to me? And if I hadn't had to worry about things like PR and "someone yelling in the lobby would ruin the experience for people watching movies"? I think I would absolutely have used the "no, sorry, those are the rules" move. I would have been happy to do so. Does that make me a blankface? But it's not that I would have enjoyed wielding my power as such. Rather it's that I would have enjoyed punishing her, specifically, for acting in ways that I endorsedly think are bad to act in.

Does someone's internal experience matter, though? If they act a certain way, should we care how they feel? I think we should, and if we don't we shouldn't make claims about it.

That is, if what you care about is whether someone is acting a certain way, then don't mention enjoyment when you define a blankface. And if you really do care about the enjoyment part of it - well, how do you know what someone is feeling and why?

I predict that if th...
There are transparent monsters in the world - part 1
by Writer — 2021-08-08T19:47:18.999Z
Link: https://youtu.be/N_ayq66t77U

EA Forum cross-post

This is Rational Animations' script for the video linked above. Past feedback on the EA Forum has been helpful, so I encourage you to give your opinions in the comments. Between the video about longtermism and this one, we also published a video about Julia Galef's book "The Scout Mindset". Check it out on the channel if you are curious.

There are transparent monsters in the world. They are enormous, but they walk mostly unseen. You can only see them if you squint hard enough or if you recognize the distortions they produce on light. Monsters leave destruction in their wake, but not everyone can recognize the destruction as such. In any given era of humanity, only a few people know that transparent monsters exist.

The first few who recognize a monster often speak up. A few others join them, and the cause grows. The more people see the monster, the more visible it becomes.

During this process, and even after everyone can see the monster, some people will defend it. They'll say it's a necessary evil or actually beneficial. Maybe they'll say that it protects us from an even bigger monster.

Most of the time, these arguments don't last. In the end, most of humanity bands together to take the monster down. As a result, we become better and stronger.

Take it down!

But we only know of the victories. History doesn’t sing of those who weren’t believed. Those who died, broken and alienated. Those who are still alive, shouting without being heard. There are monsters in the world today. You might see some of them, but you don’t see all of them. Probably no one does.

Consider... slavery: it’s a monster that predates written records, and it took until the 19th century to become visible to most of the world. Some forms of slavery are still present, we have not eradicated it yet, but it is now widely recognized as a tragedy.

Philosophers in ancient Greece were preoccupied with understanding what a virtuous man should do. And yet, they believed slavery to be good, partly because of a different conception of virtue than today.

Here is what Aristotle writes in The Politics: “[...] But is there any one thus intended by nature to be a slave, and for whom such a condition is expedient and right, or rather is not all slavery a violation of nature? There is no difficulty in answering this question, on grounds both of reason and of fact. For that some should rul...
The Future: Where are the Colors and the Sports? 
by rogersbacon — 2021-08-08T18:55:53.414Z

One can imagine a variety of reasons for why “The Future” (as imagined in our collective consciousness) is blue and black, many of which are mundane and fairly obvious.

The screens on our phones, laptops, and TVs are black (the inspiration behind the name of the science fiction series Black Mirror) and it’s probably a good bet that the future will have even more screens (one day we will decide to replace the whole sky with a screen so that it’s never cloudy and every sunset is picturesque). In general, technology calls to mind darker and more metallic colors (silver and grey also seem abundant in our imagined future). The future also calls to mind space exploration which is a very black and blue type of affair - from our grey metallic spaceships we will all look back at our home planet, a pale blue dot hanging in the void.

There may be a less obvious and more interesting reason why we tend to envision the future with a certain aesthetic (darker color palette, sleek, smooth, shiny, etc.).

1

Construal-level theory describes how distance of all kinds (temporally, spatially, socially, emotionally, conceptually, etc.) form one general cognitive dimension and how this dimension (near vs. far) affects the abstraction of our thinking. To get a better sense of the theory, I’ll share an abstract from a review paper (slightly more technical) and a few sections from the wikipedia page (slightly less technical).

From Trope and Liberman (2010), “

Construal-Level Theory of Psychological Distance

”

People are capable of thinking about the future, the past, remote locations, another person's perspective, and counterfactual alternatives. Without denying the uniqueness of each process, it is proposed that they constitute different forms of traversing psychological distance. Psychological distance is egocentric: Its reference point is the self in the here and now, and the different ways in which an object might be removed from that point-in time, in space, in social distance, and in hypotheticality-constitute different distance dimensions. Transcending the self in the here and now entails mental construal, and the farther removed an object is from direct experience, the higher (more abstract) the level of construal of that object. Supporting this analysis, research shows (a) that the various distances are cognitively related to each other, (b) that they similarly influence and are inf

...
Book review: Knowledge and Decisions by Thomas Sowell
by victorsintnicolaas — 2021-08-08T17:05:09.432Z

Crossposted from equilibria.club

Link to the book on Goodreads

One of the most famous and insightful pieces of economic writing is Friedrich Hayek's The Use of Knowledge in Society. Most of you have probably read it, if you didn't I recommend it (only 5-7 pages)!

Economist Thomas Sowell wrote a book Knowledge and decisions, building further on this line of reasoning. Compared to the article, there is not that many new insights. But Sowell expands on Hayek's worldview and offers some interesting definitions and ways of looking at the world.

His book was written in 1980, and yet many insights about how the government operates ring true today. It is a due reminder that to understand the present and future, it is no wasted effort to understand the past.

The most foundational principle of the book: the core use of information is to make trade-offs. You can distinguish categorical from incremental decisions, Sowell prefers the latter. Or in other words, avoid step functions.

Core principles from the book

Sowell models the world as organizations of different sizes, operating under a diverse set of processes. Whereas Hayek's original essay is mainly focused on explaining the role of prices in conveying information, Sowell aims to look at other mechanisms which create and correct knowledge. For example, the importance of feedback:

"Given the crucial importance of feedback in using knowledge to make decisions, the transfer of decisions from one kind of institution to another raises serious and even grave questions as to which institution is inherently more open to feedback and which more thoroughly insulated from it. The nature of the feedback process is also important: Is it mere articulation, in which some may have great talents without a corresponding depth of understanding, and in which others may choose to listen to or ignore, or is it inarticulate but powerful mechanisms ranging from money to love?"

Information is not always true. The degree to which authentication of the information has taken place, separates pure beliefs from ever more verified pieces of knowledge:

"Authentication is as important as the raw information itself. [...] This means that the ratio of knowledge to belief may also vary enormously from one aspect of life to another. [...] In short, the degree of authentication of knowledge may be lower in the "higher" intellectual levels and mu

...
Predictions about the state of crypto in ten years
by mike_hawke — 2021-08-08T16:18:13.941Z

Recently, a couple friends of mine were blustering a lot about the future of crypto. I was inspired to write a prediction about the topic, to be evaluated in 10 years. My primary goal was to write something that would not be laughable in 2031, and my secondary goal was to be as specific as I could. It's not super rigorous, since I wanted to sketch my overall feelings, not cram them into narrow probabilistic claims.

(I will use the word crypto as a stand-in for the whole cluster of cryptocurrency, blockchain tech, smart contracts, and so on.)

Here are my claims:

To say that crypto tech has huge potential is an understatement. Crypto might revolutionize civilization as much or more than the invention of money itself.
When it comes to this topic, smart people routinely fail to synthesize the

Inside View and Outside Views

. You can tell that folks are also not applying the lessons of

Superforecasting

.
I hear that people tend to overestimate technological progress in the short term and underestimate it in the long term. I think this will turn out to be the case for crypto.
The technology is still being developed. Monthly price movements are a distraction. Predicting long-term value requires doing lots of effortful, interdisciplinary homework. Long-term evaluation is the diet & exercise of investment.
Maximalism of any kind seems silly to me, if only because of all the other uncertainties listed here.
I cannot take seriously any plan or prediction that does not consider things like Vitalik's list of hard problems.
I think crypto as it currently exists represents a threat to the status quo around government, not least of all with regard to taxation and money printing. There is no reason to think that governments will not use laws, zero-days, and physical force to preserve their powers. Blanket bans are unlikely; nationalization by force is a plausible scenario; and pressure on devs to hard code governmental privileges is all but guaranteed.
An investor friend of mine said, “I think ETH had tons of exciting tech potential a few years ago, but little enough is being done on it that I don’t really see how it will keep its valuation unless there’s a new building boom or it happens to slide into store-of-value territory with BTC.”

Sounds plausible to me. In 2015 I was really confident that we would have at least one good platform for prediction markets by now. Augur is bad and

...
Applications for Deconfusing Goal-Directedness
by adamShimi — 2021-08-08T13:05:25.674Z

Atonement for my sins towards deconfusion

I have

argued

that the deconfusion of goal-directedness should not follow what I dubbed “the backward approach”, that is starting from the applications for the concept and reverse-engineering its coherent existence (or contradiction) from there. I have also

argued

that deconfusion should always start and center around applications.

In summary, I was wrong. About the former

If deconfusion indeed starts at the applications, what about my arguments against the backward approach to goal-directedness? I wrote

The same can be said for all convergent instrumental subgoals in the paper, and as far as I know, every argument about AI risks using goal-directedness. In essence, the backward approach finds out that what is used in the argument is the fundamental property that the forward approach is trying to formalize. This means in turn that we should work on the deconfusion of goal-directedness from the philosophical intuitions instead of trying to focus on the arguments for AI risks, because these arguments depend completely on the intuitions themselves.

My best answer is this post: an exploration of applications of deconfusing goal-directedness, and how they actually inform and constrain the deconfusion itself. The gist is that in discarding a different approach than what felt natural to me, I failed to notice all the ways in which applications do constraint and direct deconfusion. In this specific case, the most fruitful and important applications I’ve found are convergent subgoals, i replacing optimal policies, formalizing inner alignment and separating approval-directed systems from pure maximizers.

Thanks to John S. Wentworth for pushing hard on the importance of starting at the applications.

Applications

Convergent subgoals

Convergent subgoals (self-preservation, resource acquisitions…) are often important ingredients in scenarios starting with misspecified objectives and ending with catastrophic consequences. Without them, even an AGI would let itself be shut down, greatly reducing the related risks. Convergent subgoals are also clearly linked with goal-directedness, since

the original argument

proposes that most goals lead to them.

As an application for deconfusion, what does this entail? Well, goal-directed systems should be the whole class of systems that could have convergent subgoals. It doesn’t necessarily mean that most g...