trentmkelly/LessWrong-43k huggingface dataset:
Declarative Mathematics
by johnswentworth — 2019-03-21T19:05:08.688Z

Programmers generally distinguish between “imperative” languages in which you specify what to do (e.g. C) versus “declarative” languages in which you specify what you want, and let the computer figure out how to do it (e.g. SQL). Over time, we generally expect programming to become more declarative, as more of the details are left to the compiler/interpreter. Good examples include the transition to automated memory management and, more recently, high-level tools for concurrent/parallel programming.

It’s hard to say what programming languages will look like in twenty or fifty years, but it’s a pretty safe bet that they’ll be a lot more declarative.

I expect that applied mathematics will also become much more declarative, for largely the same reasons: as computers grow in power and software expands its reach, there will be less and less need for (most) humans to worry about the details of rote computation.

What does this look like? Well, let’s start with a few examples of “imperative” mathematics:

Most grade-school arithmetic: it’s explicitly focused on computation, and even spells out the exact steps to follow (e.g. long division).
Gaussian reduction, as typically taught in a first-semester linear algebra class. It’s the undergrads’ version of grade-school arithmetic.
Most of the computation performed by hand in physics, engineering and upper-level econ courses & research, i.e. algebra/DEs/PDEs.

Contrast to the declarative counterparts:

Figure out what arithmetic needs to be done (i.e. what numbers to plug in) and then use a calculator
Set up a system of linear equations, then have python or wolfram invert the matrix
Choose which phenomena to include in a model, set up the governing equations, then use either numerical simulation (for pretty graphs) or a computer algebra system (for asymptotics and scaling relations).

In the declarative case, most of the work is in formulating the problem, figuring out what questions to ask, and translating it all into a language which a computer can work with - numbers, or matrices, or systems of equations.

This is all pretty standard commentary at the level of mathematics education, but the real importance is in shaping the goals of applied mathematics. For the past century, the main objectives of mathematical research programs would be things like existence & uniqueness, or exhaustive classification of some objects, or algorit...
The Main Sources of AI Risk?
by Daniel Kokotajlo — 2019-03-21T18:28:33.068Z

There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?

Insufficient time/resources for AI safety (for example caused by intelligence explosion or AI race)

Insufficient global coordination, leading to the above

Misspecified or incorrectly learned goals/values

Inner optimizers

ML differentially accelerating easy to measure goals

Paul Christiano's "influence-seeking behavior" (a combination of 3 and 4 above?)

AI generally accelerating intellectual progress in a wrong direction (e.g., accelerating unsafe/risky technologies more than knowledge/wisdom about how to safely use those technologies)

Metaethical error

Metaphilosophical error

Other kinds of philosophical errors in AI design (e.g., giving AI a wrong prior or decision theory)

Other design/coding errors (e.g., accidentally putting a minus sign in front of utility function, supposedly corrigible AI not actually being corrigible)

Doing acausal reasoning in a wrong way (e.g., failing to make good acausal trades, being acausally extorted, failing to acausally influence others who can be so influenced)

Human-controlled AIs ending up with wrong values due to insufficient "metaphilosophical paternalism"

Human-controlled AIs causing ethical disasters (e.g., large scale suffering that can't be "balanced out" later) prior to reaching moral/philosophical maturity

Intentional corruption of human values

Unintentional corruption of human values

Mind crime (disvalue unintentionally incurred through morally relevant simulations in AIs' minds)

Premature value lock-in (i.e., freezing one's current conception of what's good into a utility function)

Extortion between AIs leading to vast disvalue

Distributional shifts causing apparently safe/aligned AIs to stop being safe/aligned

Value drift and other kinds of error as AIs self-modify, or AIs failing to solve value alignment for more advanced AIs

Treacherous turn / loss of property rights due to insufficient competitiveness of humans & human-aligned AIs

Gradual loss of influence due to insufficient competitiv

...
[Link] IDA 9/14: The Scheme
by RAISE — 2019-03-21T18:28:09.745Z

This is a linkpost for https://app.grasple.com/#/level/1669

Every Thursday for 4 weeks, we will be posting lessons about Iterated Distillation and Amplification. They're largely based on Paul Christiano's sequence here on LW. He graciously allowed us to use his work.

This is the third section, containing one lesson synthesizing the blog posts detailing the scheme and the first of our video series on the scheme, more of which will be coming in the next few weeks. The video can also be found on Youtube .

Note that access to the lessons requires creating an account here.

Have a nice day!
What should we expect from GPT-3?
by avturchin — 2019-03-21T14:28:37.702Z

When it will appear? (My guess is 2020).

Will it be created by OpenAI and will it be advertised? (My guess is that it will not be publicly known until 2021, but other companies may create open versions before it.)

How much data will be used for its training and what type of data? (My guess is 400 GB of text plus illustrating pictures, but not audio and video.)

What it will be able to do? (My guess: translation, picture generation based on text, text generation based on pictures – with 70 per cent of human performance.)

How many parameters will be in the model? (My guess is 100 billion to trillion.)

How much compute will be used for training? (No idea.)
[Question] Tracking accuracy of personal forecasts
by  — 2019-03-20T20:39:14.895Z

I've been thinking how I can improve my accuracy predicting events of personal interest (e.g., "Will my landlord get the washing machine fixed within the next two weeks", or "Will my parent die this year" for a more extreme example). Betting markets will not help me with that.

At first I thought about creating dedicated software that gathers such predictions, the final outcomes of predicted events, and presents their accuracy so that the user can spot bias. Then I realised a simple spreadsheet might suffice to gather data at first and assess how useful this is. And if the need arises in the future, it should be easy to import into dedicated software, provided that all the relevant data is already there.

Does anyone track their personal predictions? If so, what methodology do you use, and did it allow you to improve your accuracy?

As an RFC, here's the spreadsheet layout I have on mind:

Tags: (value 0 or 1):

Health
Finance
Interpersonal relations
...

Date of the forecast
Event (e.g., "My landlord will get the washing machine fixed within the next two weeks"). I'm planning to formulate them so that "yes" is always the desired outcome, so that it's easy to spot if I'm reliably too optimistic or pessimistic.
Estimated probability
Deadline of the forecast
Outcome (value 0 or 1, filled after the deadline of the forecast, or when the answer is known sooner)
Criticism catalyzes analytical thinking in groups
by rayraegah — 2019-03-20T16:27:29.826Z

Brainstorming, a technique long thought to enhance group creativity. Although the research shows little evidence of effectiveness, this technique (Alex Faickney Osborn, 1957) has persisted in practice. More specifically, brainstorming instructions increase the number of ideas in a group but are usually less than the total number of ideas generated by the same number of brainstorming individuals alone (Brown and Paulus, 1996).

The instructions for brainstorming are fairly precise:

Quantity: come up with as many ideas as you can.

Do not criticise others’ ideas.

Build on others’ ideas.

Freewheeling is welcome.

Not to criticise each others’ idea is an admonition as it may cause evaluation apprehension: people will be reluctant to express creative ideas; they won’t freewheel for fear of evaluation and risk.

We’re traditionally taught that the criticism inherent to analysis is a no-go in the process of ideation: the second rule of brainstorming is don’t criticise. The absence of criticism is supposed to set free creativity but as it happens, the human mind is not a creative volcano waiting for an excuse to erupt.

Charlan Nemeth, a professor of psychology at the University of California, demonstrated that people are not very creative when they are given complete freedom. As part of an experiment on word association, she asked people to associate freely on the word blue, the vast majority said first green, then sky or ocean. Her research shows that people are able to generate both a larger number of associations and more creative associations when faced with dissent and criticism.

As part of the aforementioned experiment, Nemeth placed her test subjects alongside actors posing as test subjects who would disagree on established truths like what colour an object is. It turned out that the test subjects confronted with the dissenting actors had much more creative associations than the test subjects in the group without actors. Rather than associating blue with green, sky, or ocean, they associated blue with words like berry pie. Instead of killing creativity, disagreement seems to foster it.

In dire straits, analysis becomes an essential tool to understand what is going on in the world and help evaluate existing ideas—it is also a catalyst of the creativity needed to come up with solutions.

Analysis injects dissent and criticism into the ideation process, and contrary to w

...
What's wrong with these analogies for understanding Informed Oversight and IDA?
by Wei Dai — 2019-03-20T09:11:33.613Z

In Can HCH epistemically dominate Ramanujan? Alex Zhu wrote:

If HCH is ascription universal, then it should be able to epistemically dominate an AI theorem-prover that reasons similarly to how Ramanujan reasoned. But I don’t currently have any intuitions as to why explicit verbal breakdowns of reasoning should be able to replicate the intuitions that generated Ramanujan’s results (or any style of reasoning employed by any mathematician since Ramanujan, for that matter).

And I answered:

My guess is that HCH has to reverse engineer the theorem prover, figure out how/why it works, and then reproduce the same kind of reasoning.

And then I followed up my own comment with:

It occurs to me that if the overseer understands everything that the ML model (that it’s training) is doing, and the training is via some kind of local optimization algorithm like gradient descent, the overseer is essentially manually programming the ML model by gradually nudging it from some initial (e.g., random) point in configuration space.

No one answered my comments with either a confirmation or denial, as to whether these guesses of how to understand Universality / Informed Oversight and IDA are correct. I'm surfacing this question as a top-level post because if "Informed Oversight = reverse engineering" and "IDA = programming by nudging" are good analogies for understanding Informed Oversight and IDA, it seems to have pretty significant implications.

In particular it seems to imply that there's not much hope for IDA to be competitive with ML-in-general, because if IDA is analogous to a highly constrained method of "manual" programming, that seems unlikely to be competitive with less constrained methods of "manual" programming (i.e., AIs designing and programming more advanced AIs in more general ways, similar to how humans do most programming today), which itself is presumably not competitive with general (unconstrained-by-safety) ML (otherwise ML would not be the competitive benchmark).

If these are not good ways to understand IO and IDA, can someone please point out why?
Alignment Newsletter #49
by Rohin Shah — 2019-03-20T04:20:01.333Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter.

Highlights

Exploring Neural Networks with Activation Atlases

(Shan Carter et al): Previous work by this group of people includes

The Building Blocks of Interpretability

and

Feature Visualization

, both of which apparently came out before this newsletter started so I don't have a summary to point to. Those were primarily about understanding what individual neurons in an image classifer were responding to, and the key idea was to "name" each neuron with the input that would maximally activate that neuron. This can give you a global view of what the network is doing.

However, such a global view makes it hard to understand the interaction between neurons. To understand these, we can look at a specific input image, and use techniques like attribution. Rather than attribute final classifications to the input, you could attribute classifications to neurons in the network, and then since individual neurons now had meanings (roughly: "fuzzy texture neuron", "tennis ball neuron", etc) you can gain insight to how the network is making decisions for that specific input.

However, ideally we would like to see how the network uses interactions between neurons to make decisions in general; not on a single image. This motivates activation atlases, which analyze the activations of a network on a large dataset of inputs. In particular, for each of a million images, they randomly choose a non-border patch from the image, and compute the activation vector at a particular layer of the network at that patch. This gives a dataset of a million activation vectors. They use standard dimensionality reduction techniques to map each activation vector into an (x, y) point on the 2D plane. They divide the 2D plane into a reasonably sized grid (e.g. 50x50), and for each grid cell they compute the average of all the activation vectors in the cell, visualize that activation vector using feature visualization, and put the resulting image into the grid cell. This gives a 50x50 grid of the "concepts" that the particular neural network layer we are analyzing can reason about. They also use attribution to show, for each grid cell, which class that grid cell most supports.

The paper then goes int...
Some thoughts after reading Artificial Intelligence: A Modern Approach
by swift_spiral — 2019-03-19T23:39:46.810Z

Intro

I have been trying to learn what I need to know to contribute to AI safety research. To that end, I recently finished working through Artificial Intelligence: A Modern Approach. Before reading this textbook, I knew a bit about machine learning, but did not know much about search algorithms, knowledge representation, Bayesian networks, or the many many other topic AIMA covers. I found this textbook a useful overview of its field. Below I will talk about a few concepts that I found particularly interesting.

Generating heuristics

If an algorithm is searching through an action space, it needs some way of deciding how close the possible next states it can reach are to its goal. One way of generating a useful heuristic is to relax some of the limitations on actions. Computing “how hard is it to get from state A to the goal?” takes a lot of work, so instead the algorithm could calculate “how hard would it be to get from state A to the goal if I could do things I cannot?” For example, if you are trying to plan how to coordinate planes, fuel, and cargo to fly various packages where they need to go, you could create a heuristic by ignoring fuel limitations of planes, forming a problem that is easier to solve but still gives information about the true problem. I would expect it to be hard to find relaxed problems that are still close enough to the full problem to give useful heuristics, but in the examples the book gives it seems to work pretty well.

Reasons for failure to learn

There are a few different reasons a learning algorithm can fail to find the right function: the correct hypothesis might not be in the hypothesis space, the data can be noisy so that perfect prediction from the information that the algorithm has access to is impossible, sample variance can makes the data the algorithm sees unrepresentative of the true distribution, and complexity of the hypothesis space can prevents the algorithm from finding the correct hypothesis. I found this a useful summary of possible barriers to learning.

Boosting

In boosting, you train an initial classifier, see which data it gets wrong, and then train another classifier by weighting the training samples that the first classifier got wrong more heavily. You do this a few times and then combine the classifiers to make your actual predictions. This gives surprisingly (to me, at least) good performance. If the first classif...
Rest Days vs Recovery Days
by Unreal — 2019-03-19T22:37:09.194Z

Based on a comment I made on this EA Forum Post on Burnout.

Related links: Sabbath hard and go home, Bring Back the Sabbath

That comment I made generated more positive feedback than usual (in that people seemed to find it helpful to read and found themselves thinking about it months after reading it), so I'm elevating it to a LW post of its own. Consider this an update to the original comment.

Like Ben Hoffman, I stumbled upon and rediscovered the Sabbath (although my implementation seems different from both Ben and Zvi). I was experiencing burnout at CFAR, and while I wasn't able to escape the effects entirely, I found some refuge in the following distinction between Rest Days and Recovery Days.

Recovery Days

A Recovery Day is where you're so tired or under-resourced that you can't do much of anything with yourself other than: stay in bed / sleep a lot, binge on Netflix or video games, stay in your room all day, play with your phone, use social media, and feel unmotivated to do much except easy, stimulating, and/or mind-numbing things. This is a Recovery Day and does not count as a Rest Day, but it is fine to take the time for them. However you aren't going to be refreshed from them. In order to really refresh, you need to take another day that counts as a Rest Day.

Another way a person might take time off is to do things that are like work but easier. Video games are a prime example. I play a lot of video games that involve optimizing systems, and I find these really motivating and fun. But I notice that this is a kind of "work"—my mind is trying to solve problems and implement solutions. The difference is that because it's easy and doable, I get addicted to them, and it's a way for me to escape the "real" problems at work, which tend to be harder to solve. This also doesn't count as Resting.

Rest Days

Rest Days are days where I have enough energy and resources that I feel motivated and able to get out and about. (One way I can tell I have energy is that sometimes I spontaneously feel like cooking, a rare occurrence.) On a Rest Day, your prime directive is to just "follow your gut" for the entire day and just do "what you feel like doing" in the moment.

There can be no obligations on a Rest Day. No scheduled calls or meetings. No promises to show up to a party. You can go to the par...
Partial preferences and models
by Stuart_Armstrong — 2019-03-19T16:29:23.162Z

Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.

EDIT: This model is currently obsolete, see here for the most current version.

I've talked about partial preferences and partial models before. I haven't been particularly consistent in terminology so far ("proto-preferences", "model fragments"), but from now on I'll stick with "partial".

Definitions

So what are partial models, and partial preferences?

Assume that every world is described by the values of N different variables, X={x1,x2,…xN}⊆RN.

A partial model is given by two sets, Y and Z, along with an addition map +:Y×Z→X. Thus for y∈Y and z∈Z, y+z is an element of X.

We'll want + to have 'reasonable' properties; for the moment I'm imagining Y and Z as manifolds and + as local homeomorphism. If you don't understand that terminology, it just means that + is well behaved and that as you move y and z around, you move y+z in every direction in X.

A partial preference given the partial model above are two values y+,y−∈Y, along with the value judgement that:

for all z∈Z, y++z describes a better world than y−+z.

We can generalise to non-linear subspaces, but this version works well for many circumstances.

Interpretations

The Y are the foreground variables that we care about in our partial model. The Z are the 'background variables' that are not relevant to the partial model at the moment.

So, for example, when I contemplate whether to walk or run back home, then the GDP of Sweden, the distance Voyager 2 is from Earth, the actual value of the cosmological constant, the number of deaths from malaria, and so on, are not actually relevant to that model. They are grouped under the (irrelevant) background variables category.

Notice that these variables are only irrelevant if they are in a 'reasonable range'. If the GDP of Sweden had suddenly hit zero, if Voyager 2 was about to crash into my head, if the cosmological constant suddenly jumped, or if malaria deaths reached 99.999% of the population, then this would affect my walking/running speed.

So the set Z also encodes background expectations about the world. Being able to say that certain values are in an 'irrelevant' range is a key part of symbol grounding and the frame problem: it allows us to separate Y and Z as being, in a sense, complementary or orthogonal to each other. Note that hum

...
IRL 3/8: Mitigating degeneracy: feature matching
by RAISE — 2019-03-18T20:15:47.657Z
Link: https://app.grasple.com/#/level/1501

Every Monday for 8 weeks, we will be posting lessons about Inverse Reinforcement Learning. This is lesson 3.

Note that access to the lessons requires creating an account here.

This lesson comes with the following supplementary material:

Featurization
Support Vector Machines

Have a nice day!
Is there a difference between uncertainty over your utility function and uncertainty over outcomes?
by Chris_Leong — 2019-03-18T18:41:38.246Z

I was discussing UDT yesterday and the question came up of how to treat uncertainty over your utility function. I suggested that this could be transformed into a question of uncertainty over outcomes. The intuition is that if you were to discover that apples were twice as valuable, you could simply pretend that you instead received twice as many apples. Is this approach correct? In particular, is it transformation compatible with UDT-style reasoning?