trentmkelly/LessWrong-43k huggingface dataset:
Does GPT-2 Understand Anything?
by Douglas Summers-Stay — 2020-01-02T17:09:14.641Z

Some people have expressed that “GPT-2 doesn’t understand anything about language or reality. It’s just huge statistics.” In at least two senses, this is true.

First, GPT-2 has no sensory organs. So when it talks about how things look or sound or feel and gets it right, it is just because it read something similar on the web somewhere. The best understanding it could have is the kind of understanding one gets from reading, not from direct experiences. Nor does it have the kind of understanding that a person does when reading, where the words bring to mind memories of past direct experiences.

Second, GPT-2 has no qualia. This is related to the previous point, but distinct from it. One could imagine building a robotic body with cameras for eyes and microphones for ears that fed .png and .wav files to something like GPT-2 rather than .html files. Such a system would have what might be called experiences of the world. It would not, however, create an direct internal impression of redness or loudness, the ineffable conscious experience that accompanies sensation.

However, this is too high a bar to rule out understanding. Perhaps we should call the understanding that comes from direct personal experience “real understanding” and the kind that comes solely from reading with no connection to personal experience “abstract understanding.” Although I can’t “really understand” what it was like to fight in the Vietnam War (because I wasn’t there, man) I can still understand it in an abstract sense. With an abstract understanding, here are some things one can do:

•	answer questions about it in one’s own words
•	define it
•	use it appropriately in a sentence
•	provide details about it
•	summarize it

Professional teachers distinguish between tests of knowledge (which can be handled by mere memorization) and tests of understanding, with the latter being more difficult and useful (see Bloom’s Taxonomy). Understanding requires connecting a new idea to ideas a student is already familiar with.

GPT-2 is able to pass many such tests of understanding. With an appropriate prompt (such as giving examples of what form the answer to a question should take) it is able to answer questions, define terms, use words appropriately in a sentence, provide details,...
PlanAlyzer: assessing threats to the validity of online experiments
by Three-Monkey Mind — 2020-01-02T07:42:38.022Z
Link: https://blog.acolyer.org/2019/11/22/planalyzer/

It’s easy to make experimental design mistakes that invalidate your online controlled experiments. At an organisation like Facebook (who kindly supplied the corpus of experiments used in this study), the state of art is to have a pool of experts carefully review all experiments. PlanAlyzer acts a bit like a linter for online experiment designs, where those designs are specified in the PlanOut language.

As well as pointing out any bugs in the experiment design, PlanAlyzer will also output a set of contrasts — comparisons that you can safely make given the design of the experiment. Hopefully the comparison you wanted to make when you set up the experiment is in that set!

[…]

Regular readers of The Morning Paper will be well aware that there’s plenty that can go wrong in the design and interpretation of online controlled experiments (see e.g. ‘A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments’) . PlanAnalyzer is aimed at detecting threats to internal validity, the degree to which valid causal conclusions can (or cannot!) be drawn from a study.
Making decisions when both morally and empirically uncertain
by MichaelA — 2020-01-02T07:20:46.114Z

Cross-posted to the EA Forum. For an epistemic status statement and an outline of the purpose of this sequence of posts, please see the top of my prior post. There are also some explanations and caveats in that post which I won’t repeat - or will repeat only briefly - in this post.

Purpose of this post

In my prior post, I wrote:

We are often forced to make decisions under conditions of uncertainty. This uncertainty can be empirical (e.g., what is the likelihood that nuclear war would cause human extinction?) or moral (e.g., does the wellbeing of future generations matter morally?). The issue of making decisions under empirical uncertainty has been well-studied, and expected utility theory has emerged as the typical account of how a rational agent should proceed in these situations. The issue of making decisions under moral uncertainty appears to have received less attention (though see this list of relevant papers), despite also being of clear importance.

I then went on to describe three prominent approaches for dealing with moral uncertainty (based on Will MacAskill’s 2014 thesis):

Maximising Expected Choice-worthiness (MEC), if all theories under consideration by the decision-maker are cardinal and intertheoretically comparable.[1]

Variance Voting (VV), a form of what I’ll call “Normalised MEC”, if all theories under consideration are cardinal but not intertheoretically comparable.[2]

The Borda Rule (BR), if all theories under consideration are ordinal.

But I was surprised to discover that I couldn’t find any very explicit write-up of how to handle moral and empirical uncertainty at the same time. I assume this is because most people writing on relevant topics consider the approach I will propose in this post to be quite obvious (at least when using MEC with cardinal, intertheoretically comparable, consequentialist theories). Indeed, many existing models from EAs/rationalists (and likely from other communities) already effectively use something very much like the first approach I discuss here (“MEC-E”; explained below), just without explicitly noting that this is an integration of approaches for dealing with moral and empirical uncertainty.[3]

But it still seemed worth explicitly spelling out the approach I propose, which is, in a nutshell, using exactly the regular approaches to moral uncertainty mentioned above, but on outcomes rather than on actions,

...
Theories That Can Explain Everything
by Chris_Leong — 2020-01-02T02:12:28.772Z

It's generally accepted here that theories are valuable to the extent that they provide testable predictions. Being falsifiable means that incorrect theories can be discarded and replaced with theories that better model reality (see Making Beliefs Pay Rent). Unfortunately, reality doesn't play nice and we will sometimes possess excellent theoretical reasons for believing a theory, but that theory will possess far too many degrees of freedom to make it easily falsifiability.

The prototypical example are the kinds of hypotheses that are produced by evolutionary psychology. Clearly all aspects of humanity have been shaped by evolution and the idea that our behaviour is an exception would be truly astounding. In fact, I'd say that it is something of an anti-prediction.

But what use is a theory that doesn't make any solid predictions? Firstly, believing in such a theory will normally have a significant impact on your priors, even if no-one observation would provide strong evidence of its falsehood. But secondly, if the existing viable theories all claim A and you propose viable a theory that would be compatible with A or B, then that would make B viable again. And sometimes that can be a worthy contribution in and of itself. Indeed, you can have a funny situation arise where people nominally reject a theory for not sufficiently constraining expectations, while really opposing it because of how people's expectations would adjust if the theory was true.

See also: Building Intuitions on Non-Empirical Arguments in Science
Characterising utopia
by Richard_Ngo — 2020-01-02T00:00:01.268Z

When we consider how good the best possible future might be, it's tempting to focus on only a handful of dimensions of change. In transhumanist thinking, these dimensions tend to be characteristics of individuals: their happiness, longevity, intelligence, and so on. [1] This reflects the deeply individualistic nature of our modern societies overall, and transhumanists in particular. Yet when asked what makes their lives meaningful,

most people

prioritise their relationships with others. In contrast, there are strands of utopian literature which focus on social reorganisation (such as Huxley’s Island or Skinner’s Walden Two), but usually without acknowledging the potential of technology to radically improve the human condition. [2] Meanwhile, utilitarians conceive of the best future as whichever one maximises a given metric of individual welfare - but those metrics are often criticised for oversimplifying the range of goods that people actually care about. [3] In this essay I've tried to be as comprehensive as possible in cataloguing the ways that the future could be much better than the present, which I've divided into three categories: individual lives, relationships with others, and humanity overall. Each section consists of a series of bullet points, with nested elaborations and examples.
I hesitated for quite some time before making this essay public, though, because it feels a little naive. Partly that’s because the different claims don’t form a single coherent narrative. But on reflection I think I endorse that:

grand narratives are more seductive

but also more likely to totally miss the point. Additionally,

Holden Karnofsky has found

that “the mere act of describing [a utopia] makes it sound top-down and centralized” in a way which people dislike - another reason why discussing individual characteristics is probably more productive.
Another concern is that even though there’s a strong historical trend towards increases in material quality of life, the same is not necessarily true for social quality of life. Indeed, in many ways the former impedes the latter. In particular, the less time people need to spend obtaining necessities, the more individualistic they’re able to be, and the more time they can spend on negative-sum status games. I don’t know how to solve this problem, o...
Since figuring out human values is hard, what about, say, monkey values?
by Shmi — 2020-01-01T21:56:28.787Z

So, human values are fragile, vague and possibly not even a well defined concept, yet figuring it out seems essential for an aligned AI. It seems reasonable that, faced with a hard problem, one would start instead with a simpler one that has some connection to the original problem. For someone not working in the area of ML or AI alignment, it seems obvious that researching simpler-than-human values might be a way to make progress. But maybe this is one of those false obvious ideas that non-experts tend to push after a cursory learning about a complex research topic.

That said, assuming that the value complexity scales with intelligence, studying less intelligent agents and their version of values maybe something to pursue. Dolphin values. Monkey values. Dog values. Cat values. Fish values. Amoeba values. Sure, we lose the inside view in this case, but the trade-off seems at least being worthy of exploring. Is there any research going in that area?
What will quantum computers be used for?
by  — 2020-01-01T19:33:16.838Z

In the early days of electronic computers, machines like ENIAC were used for niche applications such as calculating ballistic trajectories and simulating nuclear explosions. These applications are very far removed from what computers are predominantly used for today.

It seems we have reached a similar development stage with regard to quantum computing. The applications researchers cite, cryptographic analysis and quantum systems simulation, are once again very far removed from everyday life.

Does anyone have a prediction on what quantum computers will be used for once they become affordable enough for regular people? Or will it forever remain just a research tool? Are quantum computers the key to cracking quantum chemistry and thereby molecular nanotechnology? (this is my guess as to what the actual impact of quantum computing will be)
Toy Organ
by jefftk — 2020-01-01T19:30:01.375Z

A few months ago I brought home an Emenee toy organ someone was
throwing out. It didn't work, and it sat in the basement for a while,
but today I had a go at fixing it. It's a very simple design: a fan
blows air through a hole in the bottom, setting up a pressure
difference between the inside and outside. When you press a key that
opens a corresponding hole and air can flow through, past a little
plastic reed, whose vibration makes the note.

With mine, the motor was running but I wasn't getting any sound. I
opened it up and nothing was obviously wrong. I figured it was
probably leaky, and put tape around base where the plastic sides meet
the chipboard bottom. This fixed it enough to get some sound:

(youtube)

The lower octave isn't working at all, and the higher notes get
progressively slower to sound and breathier until the very highest
don't sound at all, but this leaves an octave and a half of chromatic
range. Enough to play around with!

The left hand buttons play chords, and are arranged:

The circle of fifths arrangement makes a lot of sense, but the choice
to put major and minor chords adjacent does not. If you're playing in
a major key, you generally want the vi and ii minors, so Am and Dm in
the key of C. This means the vi minors they've included are used in
flatter keys than they've provided: F and C can use Dm and Am, but G
and D would like Em and Bm which are absent. Shifting the minors over
three notes would be much better. They layout would change from:

Bbm Fm Cm Gm Dm Am
Bb F C G D A

to

Gm Dm Am Em Bm F#m
Bb F C G D A

Now in each key you have:

ii vi
IV I V

instead of:

ii vi
IV I V

Playing folk music I also would have preferred that they center on G
or D, since I care much more about having E than Bb.

Another change that would be nice would be to offer a way to control
the way air gets into the organ. There's a hole on the bottom for the
fan, and if you cover it the organ stops working because air can't get
through. It has a screw-adjustable cover, which looks like it's
designed as a volume control:

You could build a simple adjustable cover connected to a foot pedal,
and get a real expression pedal. This would have added to the cost of
the instrument, so I understand why they wouldn't want to do this with
...
Don't Double-Crux With Suicide Rock
by Zack_M_Davis — 2020-01-01T19:02:55.707Z

Honest rational agents should never agree to disagree.

This idea is formalized in Aumann's agreement theorem and its various extensions (we can't foresee to disagree, uncommon priors require origin disputes, complexity bounds, &c.), but even without the sophisticated mathematics, a basic intuition should be clear: there's only one reality. Beliefs are for mapping reality, so if we're asking the same question and we're doing everything right, we should get the same answer. Crucially, even if we haven't seen the same evidence, the very fact that you believe something is itself evidence that I should take into account—and you should think the same way about my beliefs.

In "The Coin Guessing Game", Hal Finney gives a toy model illustrating what the process of convergence looks like in the context of a simple game about inferring the result of a coinflip. A coin is flipped, and two players get a "hint" about the result (Heads or Tails) along with an associated hint "quality" uniformly distributed between 0 and 1. Hints of quality 1 always match the actual result; hints of quality 0 are useless and might as well be another coinflip. Several "rounds" commence where players simultaneously reveal their current guess of the coinflip, incorporating both their own hint and its quality, and what they can infer about the other player's hint quality from their behavior in previous rounds. Eventually, agreement is reached. The process is somewhat alien from a human perspective (when's the last time you and an interlocutor switched sides in a debate multiple times before eventually agreeing?!), but not completely so: if someone whose rationality you trusted seemed visibly unmoved by your strongest arguments, you would infer that they had strong evidence or counterarguments of their own, even if there was some reason they couldn't tell you what they knew.

Honest rational agents should never agree to disagree.

In "Disagree With Suicide Rock", Robin Hanson discusses a scenario where disagreement seems clearly justified: if you encounter a rock with words painted on it claiming that you, personally, should commit suicide according to your own values, you should feel comfortable disagreeing with the words on the rock without fear of being in violation of the Aumann theorem. The rock is probably just a rock. The words are information from whoever painted them, and maybe that perso

...
[AN #79]: Recursive reward modeling as an alignment technique integrated with deep RL
by Rohin Shah — 2020-01-01T18:00:01.839Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Happy New Year!

Audio version

here

(may not be up yet).

Highlights

AI Alignment Podcast: On DeepMind, AI Safety, and Recursive Reward Modeling

(Lucas Perry and Jan Leike) (summarized by Rohin): While Jan originally worked on theory (specifically AIXI), DQN, AlphaZero and others demonstrated that deep RL was a plausible path to AGI, and so now Jan works on more empirical approaches. In particular, when selecting research directions, he looks for techniques that are deeply integrated with the current paradigm, that could scale to AGI and beyond. He also wants the technique to work for agents in general, rather than just question answering systems, since people will want to build agents that can act, at least in the digital world (e.g. composing emails). This has led him to work on

recursive reward modeling

(

AN #34

), which tries to solve the specification problem in the

SRA framework

(

AN #26

).

Reward functions are useful because they allow the AI to find novel solutions that we wouldn't think of (e.g. AlphaGo's move 37), but often are incorrectly specified, leading to reward hacking. This suggests that we should do reward modeling, where we learn a model of the reward function from human feedback. Of course, such a model is still likely to have errors leading to reward hacking, and so to avoid this, the reward model needs to be updated online. As long as it is easier to evaluate behavior than to produce behavior, reward modeling should allow AIs to find novel solutions that we wouldn't think of.

However, we would eventually like to apply reward modeling to tasks where evaluation is also hard. In this case, we can decompose the evaluation task into smaller tasks, and recursively apply reward modeling to train AI systems that can perform those small helper tasks. Then, assisted by these helpers, the human should be able to evaluate the original task. This is essentially forming a "tree" of reward modeling agents that are all building up to the reward model for the original, hard task. While currently the decomposition would be done by a human, you could in principle also use recursive reward ...
Circling as Cousin to Rationality
by Vaniver — 2020-01-01T01:16:42.727Z

Often, I talk to people who are highly skeptical, systematic thinkers who are frustrated with the level of inexplicable interest in Circling (previously discussed on LW) among some rationalists. “Sure,” they might say, “I can see how it might be a fun experience for some people, but why give it all this attention?” When people who are interested in Circling can’t give them a good response besides “try it, and perhaps then you’ll get why we like it,” there’s nothing in that response that distinguishes a contagious mind-virus from something useful for reasons not yet understood.

This post isn’t an attempt to fully explain what Circling is, nor do I think I’ll be able to capture everything that’s good about Circling. The hope is to clearly identify one way in which Circling is deeply principled in a way that rhymes with rationality, and potentially explains a substantial fraction of rationalist interest in Circling. As some context; I’m certified to lead Circles in the Circling Europe style after going through their training program, but I’ve done less Circling than Unreal had when she wrote

this post

, and I have minimal experience with the other styles.

Why am I interested in Circling?

Fundamentally, I think the thing that sets Circling apart is that it focuses on updating based on experience and strives to create a tight, high-bandwidth feedback loop to generate that experience. Add in some other principles and reflection, and you have a functioning culture of empiricism directed at human connection and psychology. I think they’d describe it a bit differently and put the emphasis in different places, while thinking that my characterization isn’t too unfair. This foundation of empiricism makes Circling seem to me like a ‘cousin of Rationality,’ though focused on people instead of systems.

I first noticed the way in which Circling was trying to implement empiricism early in my Circling experience, but it fully crystallized when a Circler said something that rhymes with P.C. Hodgell’s “That which can be destroyed by the truth should be.” I can’t remember the words precisely, but it was something like “in the practice, I have a deep level of trust that I should be open to the universe.” That is, he didn’t trust that authentic expression will predictably lead to success according to his current goals, but rather that a methodological commitment to putting himsel

...