trentmkelly/LessWrong-43k huggingface dataset:
LW Update 2019-1-09 – Question Updates, UserProfile Sorting
by Raemon — 2019-01-09T22:34:31.338Z

First update of 2019!

Streamlined Mobile Above-the-Fold

On mobile, logged out users now see cleaner, slightly sleeker collections of Rationality: A-Z, Codex and HPMOR. Logged in users see only one recommended sequence instead of 3.

Improved Q&A Features

We have a few quality-of-life improvements to the new question system:

You can now move a comment into the answers section, if you later realize it was a pretty good answer.
Answers that appear on the frontpage are now styled differently from comments.
If you reply to an answer on the frontpage, on the corresponding Question page it now appears parented underneath the answer as intended.

UserProfile and All Posts Sorting

On User Profile pages, as well on the beta-testing All Posts page, there's now a reasonable method of sorting posts.

You can now sort posts by:

new
old
top (highest karma)
magic (our karma/time weighted algorithm)
recentComments (most recently commented posts sorted to the top, along with new posts)

You also can filter posts so you only see Curated, Frontpage, Questions, Meta and Event. (By default, User Profiles, All Posts and Daily will show all post types)

GreaterWrong now gets better spam protection

We had previously implemented a filter so that new user's posts didn't appear on the home page until approved by an admin. But this only applied to specific frontpage post views. Now, it applies to all views by default, so GreaterWrong should get less spam in their All Posts page.

Password Resetting

Resetting passwords should now work properly (previously it required an awkward workaround where you had to log out before you could reset it)
Open Thread January 2019
by Raemon — 2019-01-09T20:25:02.716Z

If it’s worth saying, but not worth its own post, you can put it here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, and seeing if there are any meetups in your area.

The Open Thread sequence is here.
No surjection onto function space for manifold X
by Stuart_Armstrong — 2019-01-09T18:07:26.157Z

[Note: highly technical. Skip if topology is not your thing]

In his post on formal open problems in decision theory, Scott asked whether there could exist a topological space X and a continuous surjection s from X to C(X,I). Here, I is the closed unit interval [0,1] and C(X,I) is the set of continuous functions from X to I.

I thought I had an argument for how R−N could be such an X. But that argument is wrong, as I'll demonstrate in this post. Instead I will show that:

Let X be a manifold (with or without boundary), or a union of finitely or countably many manifolds. Then there is no continuous surjective map from X to C(X,I).

By "union", imagine the manifolds lying inside Euclidean space (or, more generically, inside a metric space), not necessarily disjoint, and taking their unions there. Note that there are many examples of such Xs - for example, the rationals within the reals (being countable unions of points, which are trivial manifolds).

In fact, I will show the more general:

If X is Fréchet–Urysohn, and σ-compact, then there is no continuous surjective map from X to C(X,I).

To see that this more general result implies the one above, note that manifolds are σ-compact, and that if Xi is σ-compact, it can be covered by countably many compact sets, so ⋃i∈NXi can be covered by countably many sets of countably many compact sets, which is just countably many. Finally, all metric spaces are Fréchet–Urysohn.

Fréchet–Urysohn basically means "convergence of subsequences makes sense in the topology", and is not a strong restriction; indeed all first-countable spaces are Fréchet–Urysohn.

Proof

A note on topologies

Now C(X,I) is well-defined as a set, but it needs a topology to discuss issues of continuity. There are three natural topologies on it: the topology of uniform convergence, the compact-open topology (which, on this set, is equal to the topology of compact convergence), and the topology of pointwise convergence.

The one most people use, and that I'll be using, is the compact-open topology.

These are refinements of each other as so:

uniform convergence ⊃ open-compact ⊃ pointwise convergence.

Thus, if we could find a surjective s:X→C(X,I) in the uniform convergence topology, it would still be continuous in the compact-open topology. Conversely, if we could show that no such maps exist in the pointwise convergence topology, there would also be no maps in the

...
The 3 Books Technique for Learning a New Skilll
by Matt Goldenberg — 2019-01-09T12:45:19.294Z

When I'm learning a new skill, there's a technique I often use to quickly gain the basics of the new skill without getting drowned in the plethora of resources that exist. I've found that just 3 resources that cover the skill from 3 separate viewpoints(along with either daily practice or a project) is enough to quickly get all the pieces I need to learn the new skill.

I'm partial to books, so I've called this The 3 Books Technique, but feel free to substitute books for courses, mentors, or videos as needed.

The "What" Book

The "What" book is used as reference material. It should be a thorough resource that gives you a broad overview of your skill. If you run into a novel situation, you should be able to go to this book and get the information you need. It covers the "surface" section of the learning model from nature pictured above.

Positive reviews of this book should contain phrases like "Thorough" and "Got me out of a pinch more than once." Negative reviews of this book should talk about "overwhelming" and "didn't know where to start."

The "How" Book

The "How" Book explains the step-by-step, nuts and bolts of how to put the skill into practice. It often contains processes, tools, and steps. It covers the "deep" part of the learning model covered above.

Positive reviews of this book should talk about "Well structured" and "Clearly thought out." Negative reviews should mention it being "too rote" or "not enough theory."

The "Why" Book

The "WHY" book explains the mindset and intuitions behind the skill. It tries to get into the authors head and lets you understand what to do in novel situations. It should cover the "transfer" part of the learning model above.

Positive reviews of this book should talk about "gaining intuitions" or "really understanding". Negative reviews should contain phrases like "not practical" or "still don't know what steps to take."

The Project or Practice

Once I have these 3 resources, I'll choose a single project or a daily practice that allows me to practice the skills from the "How" book and the mindsets from the "Why" book. If I get stuck, I'll use the "What" book to help me.

Examples

Overcoming Procrastination

"What" Book: The Procrastination Equation by Piers Steel

"How" Book: The Now Habit by Neil Fiore

"Why" Book: The Replacing Guilt blog sequence by Nate Soares

Project or Practice: Five pomodoros every day where I deliberately use the tools fro...
What are questions?
by Pee Doom — 2019-01-09T07:37:30.246Z

BONUS CONTENT

(Meta: I have been paralyzed with indecisive perfectionism on this question for way too long the past day. This dynamic is no small part of why I have never posted an article or barely any comments on this site in eight years of reading it and being part of its culture. Every additional word I added to this post increased the surface area of potential errors, a trap where the more I corrected the more I saw a need for correction, the more I ran the farther the finish line seemed. It's weird since I noticed I could have just posted the question by itself and left it at that satisfactorily. I realized that I could get that easygoing feeling back and just route around this maladaptive mental pattern by labeling the post as "bonus content" to the question! This relaxes the demanding constraint of feeling like each additional word I write has to impress you. That means you read this at your own risk, sucka! No promises of formality and articulacy! I am free and can feel satisfied, yippee!) ☺️

Our quest consists of the simplest operations, each one worthy of examination. We cannot build towers of thought without a solid foundation. We cannot build better tools if we don't know how our current tools operate, and it's often good to bootstrap by using our tools on themselves.

It seems silly to ask. But I have a heuristic that that which is laughed at should be taken seriously. It is an action at the core of our epistemology. I wrote down at least a hundred questions I could ask here, and this struck me as the logical first step, the node before all others. What is the nature of the enterprise, before we chart a course to any particular destination?

So I ask: What the fuck are these question things anyway?

There's something about a conscious recognizing of what is unknown - the act of questioning being the movement from Unknown Unknown to Known Unknown (aka unconscious incompetence to conscious incompetence). What is the process of noticing ignorance?

But I want to leave this open-ended. More information on what correlates with questioning seems valuable, whatever it is. I don't know what I don't know about how I come to know what I know.

Here are generative subquestions to consider (some of them are their own can of gummy worms best left for a deeper answer another day)!

Can you Taboo "question"? What is happe...
Book Review: The Structure Of Scientific Revolutions
by Scott Alexander — 2019-01-09T07:10:02.152Z

When I hear scientists talk about Thomas Kuhn, he sounds very reasonable. Scientists have theories that guide their work. Sometimes they run into things their theories can’t explain. Then some genius develops a new theory, and scientists are guided by that one. So the cycle repeats, knowledge gained with every step.

When I hear philosophers talk about Thomas Kuhn, he sounds like a madman. There is no such thing as ground-level truth! Only theory! No objective sense-data! Only theory! No basis for accepting or rejecting any theory over any other! Only theory! No scientists! Only theories, wearing lab coats and fake beards, hoping nobody will notice the charade!

I decided to read Kuhn’s The Structure Of Scientific Revolutions in order to understand this better. Having finished, I have come to a conclusion: yup, I can see why this book causes so much confusion.

At first Kuhn’s thesis appears simple, maybe even obvious. I found myself worrying at times that he was knocking down a straw man, although of course we have to read the history of philosophy backwards and remember that Kuhn may already be in the water supply, so to speak. He argues against a simplistic view of science in which it is merely the gradual accumulation of facts. So Aristotle discovered a few true facts, Galileo added a few more on, then Newton discovered a few more, and now we have very many facts indeed.

In this model, good science cannot disagree with other good science. You’re either wrong – as various pseudoscientists and failed scientists have been throughout history, positing false ideas like “the brain is only there to cool the blood” or “the sun orbits the earth”. Or you’re right, your ideas are enshrined in the Sacristry Of Settled Science, and your facts join the accumulated store that passes through the ages.

Simple-version-of-Kuhn says this isn’t true. Science isn’t just facts. It’s paradigms – whole ways of looking at the world. Without a paradigm, scientists wouldn’t know what facts to gather, how to collect them, or what to do with them once they had them. With a paradigm, scientists gather and process facts in the ways the paradigm suggests (“normal science”). Eventually, this process runs into a hitch – apparent contradictions, or things that don’t quite fit predictions, or just a giant ugly mess of epicycles. Some genius develops a new paradigm (“paradigm shift” or “scientific r...
mindlevelup 3 Year Review
by  — 2019-01-09T06:36:01.090Z

[An overview of why I didn’t write as much in 2018. Aversions snowballed, and the blog format led to having expectations about content I found sometimes hard to satisfy. A look back at some of my favorite posts of last year, as well as what I plan to do in 2019.]

In 2018, not counting the small essays that found their way to my

short-form blog

, I wrote 16 full-length blog posts.

I am disappointed by this because one of my constant goals with blogging is to write new essays every week. Partially, it feels bad because I did mostly succeed in hitting my original goal of weekly essays in both 2016 and 2017, so I felt like going for another year would be doable. And partially it feels bad because I feel like this means I’ve grown less as a person because I’ve had less bloggable thoughts, or something like that.

This is a blog post to dive into what happened with my blog posts in 2018.

Near the end of 2017 I went off to college, and it was then that I realized my writing schedule had taken a hit. Aside from adjusting to the environment, rationality-esque things felt less important and useful. There weren’t a lot of rationalists at school, I had less free time, social incentives produced different focuses, etc. etc.. Typical stuff. Thus, it seems like the outside view might have predicted a drop in my writing output. Looking at my

2017 review post

, though, it seems like I was still enthusiastic and optimistic for 2018; I’d outlined some areas which I had wanted to explore.

Now, a year (and relatively few blog posts) later, it looks like things have gone contra my expectations, if not altogether awry.

Admitting that this has been a problem and addressing it has been difficult for me. For most of 2018, I had a large ball of guilt, wherein I kept reminding myself that I had this blog I was very much not attending to. In the rare moments where I would find the time and excitement to write a blog post, the feeling didn’t last. If I asked my gut what happened, I’d get an answer like:

“Well, it…just kinda snowballed, y’know? Like, I guess maybe I had some good thoughts, but then I wasn’t really interacting that much with the rationality community…then I had other things taking up time…and then I just…didn’t want to write all that much…and then I started to feel less like the sort of person who writes…and all the while I felt bad because I had this large archive of my previous ...
AlphaGo Zero and capability amplification
by paulfchristiano — 2019-01-09T00:40:13.391Z

AlphaGo Zero is an impressive demonstration of AI capabilities. It also happens to be a nice proof-of-concept of a promising alignment strategy.

How AlphaGo Zero works

AlphaGo Zero learns two functions (which take as input the current board):

A prior over moves p is trained to predict what AlphaGo will eventually decide to do.
A value function v is trained to predict which player will win (if AlphaGo plays both sides)

Both are trained with supervised learning. Once we have these two functions, AlphaGo actually picks it moves by using 1600 steps of Monte Carlo tree search (MCTS), using p and v to guide the search. It trains p to bypass this expensive search process and directly pick good moves. As p improves, the expensive search becomes more powerful, and p chases this moving target.

Iterated capability amplification

In the simplest form of iterated capability amplification, we train one function:

A “weak” policy A, which is trained to predict what the agent will eventually decide to do in a given situation.

Just like AlphaGo doesn’t use the prior p directly to pick moves, we don’t use the weak policy A directly to pick actions. Instead, we use a capability amplification scheme: we call A many times in order to produce more intelligent judgments. We train A to bypass this expensive amplification process and directly make intelligent decisions. As A improves, the amplified policy becomes more powerful, and A chases this moving target.

In the case of AlphaGo Zero, A is the prior over moves, and the amplification scheme is MCTS. (More precisely: A is the pair (p, v), and the amplification scheme is MCTS + using a rollout to see who wins.)

Outside of Go, A might be a question-answering system, which can be applied several times in order to first break a question down into pieces and then separately answer each component. Or it might be a policy that updates a cognitive workspace, which can be applied many times in order to “think longer” about an issue.

The significance

Reinforcement learners take a reward function and optimize it; unfortunately, it’s not clear where to get a reward function that faithfully tracks what we care about. That’s a key source of safety concerns.

By contrast, AlphaGo Zero takes a policy-improvement-operator (like MCTS) and converges towards a fixed point of that operator. If we can find a way to improve a policy while preserving its alignment, then ...
Predictors as Agents
by interstice — 2019-01-08T20:50:49.599Z

[UPDATE: I have concluded that the argument in this post is wrong. In particular, consider a generative model. Say the generative model has a choice between two possible 'fixed point' predictions A and B, and currently assigns A 70% probability and B 30% probability. Then the target distribution it is trying to match is A 70 % B 30%, so it will just stay like that forever(or drift, likely increasing the proportion of A). This is true even if B is easier to obtain good predictions for -- the model will shift from "70 % crappy model of A / 30% good model to B" --> "70% slightly better model of A / 30% good model of B". It won't increase the fraction of B.

In general this means that the model should converge to a distribution of fixed points corresponding to the learning bias of the model -- 'simpler' fixed points will have higher weight. This might end up looking kind of weird anyway, but it won't perform optimization in the sense I described below.]

In machine learning, we can make the distinction between predictive and agent-like systems. Predictive systems include classifiers or language models. Agent-like systems are the domain of reinforcement learning and include AlphaZero, OpenAI Five, etc. While predictive systems are passive, modelling a relationship between input and output, agent-like systems perform optimization and planning.
It's well-known around here that agent-like systems trained to optimize a given objective can exhibit behavior unexpected by the agent's creators. It is also known that systems trained to optimize one objective can end up optimizing for another, because optimization can spawn sub-agents with different objectives. Here I present another type of unexpected optimization: in realistic settings, systems that are trained purely for prediction can end up behaving like agents.
Here's how it works. Say we are training a predictive model on video input. Our system is connected to a video camera in some rich environment, such as an AI lab. It receives inputs from this camera, and outputs a probability distribution over future inputs(using something like a VAE, for instance). We train it to minimize the divergence between its predictions and the actual future inputs, exponentially decaying the loss for inputs farther in the future.
Because the predictor is embedded in the en...
Alignment Newsletter #40
by Rohin Shah — 2019-01-08T20:10:03.445Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

The Alignment Forum sequences have started again! As a reminder, treat them as though I had highlighted them.

Highlights

Reframing Superintelligence: Comprehensive AI Services as General Intelligence (Eric Drexler): This is a huge document; rather than summarize it all in this newsletter, I wrote up my summary in this post. For this newsletter, I've copied over the description of the model, but left out all of the implications and critiques.

The core idea is to look at the pathway by which we will develop general intelligence, rather than assuming that at some point we will get a superintelligent AGI agent. To predict how AI will progress in the future, we can look at how AI progresses currently -- through research and development (R&D) processes. AI researchers consider a problem, define a search space, formulate an objective, and use an optimization technique in order to obtain an AI system, called a service, that performs the task.

A service is an AI system that delivers bounded results for some task using bounded resources in bounded time. Superintelligent language translation would count as a service, even though it requires a very detailed understanding of the world, including engineering, history, science, etc. Episodic RL agents also count as services.

While each of the AI R&D subtasks is currently performed by a human, as AI progresses we should expect that we will automate these tasks as well. At that point, we will have automated R&D, leading to recursive technological improvement. This is not recursive self-improvement, because the improvement comes from R&D services creating improvements in basic AI building blocks, and those improvements feed back into the R&D services. All of this should happen before we get any powerful AGI agents that can do arbitrary general reasoning.

Rohin's opinion: I'm glad this has finally been published -- it's been informing my views for a long time now. I broadly buy the general view put forward here, with a few nitpicks that you can see in the post. I really do recommend you read at least the post -- that's just the summary of the report, so it's full of insights, and it should be interesting to technical saf...