trentmkelly/LessWrong-43k huggingface dataset:
In the shadow of the Great War
by jasoncrawford — 2021-10-18T23:08:15.194Z
Link: https://rootsofprogress.org/in-the-shadow-of-the-great-war

The idea of progress fell out of favor in the course of the 20th century. But when exactly, and why?

In a recent essay I alluded to the pivotal role of the World Wars. Here’s a quote that adds weight to this—from Progress and Power, by historian Carl Becker, published in 1936:

For two centuries the Western world has been sustained by a profound belief in the doctrine of progress. Although God the Father had withdrawn into the places where Absolute Being dwells, it was still possible to maintain that the Idea or the Dialectic or Natural Law, functioning through the conscious purposes or the unconscious activities of men, could be counted on to safeguard mankind against future hazards. However formulated, with whatever apparatus of philosophic or scientific terminology defended, the doctrine was in essence an emotional conviction, a species of religion—a religion which, according to Professor [J. B.] Bury, served as a substitute for the declining faith in the Christian doctrine of salvation …

Since 1918 this hope has perceptibly faded. Standing within the deep shadow of the Great War, it is difficult to recover the nineteenth-century faith either in the fact or the doctrine of progress. The suggestion casually thrown out some years ago by Santayana, that “civilization is perhaps approaching one of those long winters which overtake it from time to time,” seems less perverse now than when it was made. Current events lend credit to the prophets of disaster who predict the collapse of a civilization that seemed but yesterday a permanent conquest of human reason …

At the present moment the world seems indeed out of joint, and it is difficult to believe with any conviction that a power not ourselves—the Idea or the Dialectic or Natural Law—will ever set it right. The present moment, therefore, when the fact of progress is disputed and the doctrine discredited, seems to me a proper time to raise the question: What, if anything, may be said on behalf of the human race? May we still, in whatever different fashion, believe in the progress of mankind?

(Emphasis added.)

I find it fascinating to see that the downfall of the idea of progress began as early as this, after World War I. World War II perhaps simply reinforced an existing trend.

I also find fascinating Becker’s idea that humanity required some sort of safeguard, a “power not ourselves” to “set it right.”

There is no pow...
Prioritization Research for Advancing Wisdom and Intelligence
by ozziegooen — 2021-10-18T22:28:48.730Z
Link: https://forum.effectivealtruism.org/posts/YpaQcARgLHFNBgyGa

LessWrong note: I wrote this more in a way slightly more optimized for the EA Forum than LessWrong, because the post seemed slightly more appropriate there.

Summary

I think it makes sense for Effective Altruists to pursue prioritization research to figure out how best to improve the wisdom and intelligence[1] of humanity. I describe endeavors that would optimize for longtermism, though similar research efforts could make sense for other worldviews.

The Basic Argument

For those interested in increasing humanity’s long-term wisdom and intelligence[1], several types of wildly different interventions are options on the table. For example, we could improve at teaching rationality, or we could make progress on online education. We could make forecasting systems and data platforms. We might even consider something more radical, like brain-computer interfaces or highly advanced pre-AGI AI systems.

These interventions share many of the same benefits. If we figure out ways to remove people’s cognitive biases, causing them to make better political decisions, that would be similar to the impact of forecasting systems on their political decisions. It seems natural to attempt to figure out how to compare these. We wouldn’t want to invest a lot of resources into one field, to realize 10 years later that we could have spent them better in another. This prioritization is pressing because Effective Altruists are currently scaling up work in several relevant areas (rationality, forecasting, institutional decision making) but mostly ignoring others (brain-computer interfaces, fundamental internet improvements).

The point of this diagram is that all of the various interventions on the left could contribute to helping humanity gain wisdom and intelligence. Different interventions produce other specific benefits as well, but these are more idiosyncratic in comparison. The benefits that come via the intermediate node of wisdom and intelligence can be directly compared between interventions.

In addition to caring about prioritization between cause areas, we should also care about estimating the importance of wisdom and intelligence work as a whole. Estimating the importance of wisdom and intelligence gains is crucial for multiple interventions, so it doesn’t make much sense to ask each intervention’s research base to independently tackle this question on their...
Beyond the human training distribution: would the AI CEO create almost-illegal teddies?
by Stuart_Armstrong — 2021-10-18T21:10:53.146Z

tl;dr: I showthat model splintering can be seen as going beyond the human training distribution (the distribution of real and imagined situations we have firm or vague preferences over), and argue why this is at the heart of AI alignment.

You are training an AI-CEO to maximise stock value, training it on examples of good/bad CEO decisions and corresponding stock-price increases/decreases.

There are some obvious failure modes. The AI could wirehead by hacking the stock-ticker, or it could do the usual optimise-the-universe-to-maximise-stock-price-for-now-dead-shareholders.

Let's assume that we've managed to avoid these degenerate solutions. Instead, the AI-CEO tries for something far weirder.

Beyond the human training distribution

The AI-CEO reorients the company towards the production of semi-sentient teddy bears that are generated in part from cloned human brain tissue. These teddies function as personal assistants and companions, and prototypes are distributed at the annual shareholder meeting.

However, the public reaction is negative, and the government bans the further production of these teddies. Consequently, the company shuts down for good. But the shareholders, who own the only existent versions of these teddies, get great kudos from possessing these rare entities, who also turn out to be great and supportive listeners - and excellent at managing their owners' digital media accounts, increasing their popularity and status.

And that, of course, was the AI-CEO's plan all along.

Hold off from judging this scenario, just for a second. And when you do judge it, observe your mental process as you do so. I've tried to build this scenario so that it is:

Outside the AI-CEO's training distribution.

Outside the human designer's implicit training distribution - few of us have thought deeply about the morality of semi-sentient teddy bears made with human tissue.

Possibly aligned with the AI-CEO's goals.

Neither clearly ideal nor clearly disastrous (depending on your moral views, you may need to adjust the scenario a bit to hit this).

If I've pitched it right, your reaction to the scenario should be similar to mine - "I need to think about this more, and I need more information". The AI-CEO is clearly providing some value to the shareholders; whether this value can be compared to the stock price is unclear. It's being manipulative, but not doing anything illegal. As f...
On The Risks of Emergent Behavior in Foundation Models
by jsteinhardt — 2021-10-18T20:00:15.896Z

This post first appeared as a commentary for the paper "On The Opportunities and Risks of Foundation Models".

Bommasani et al. (2021) discuss a trend in machine learning, whereby increasingly large-scale models are trained once and then adapted to many different tasks; they call such models "foundation models". I quite enjoyed their paper and the associated workshop, and felt they correctly identified the two most important themes in foundation models: emergence and homogenization. My main criticism is that despite identifying these themes, they did not carry them to their logical conclusions, so I hope to (partially) remedy that here.

In short, emergence implies that ML systems can quickly change to look different and "weird" compared to ML today, thus creating new risks that aren't currently apparent. Meanwhile, homogenization contributes to inertia, which could make us slow to adapt. This calls for thinking about these risks now, to provide the requisite lead time.

Emergent Behavior Creates Emergent Risks

Bommasani et al. (2021) use the following definition of emergence:

Emergence means that the behavior of a system is implicitly induced rather than explicitly constructed; it is both the source of scientific excitement and anxiety about unintended consequences.

This actually better matches the definition of a self-organizing system, which tends to produce emergent behavior. I will take emergence to be the idea that qualitative changes in behavior arise from varying a quantitative parameter ("More Is Different"). This is most common in self-organizing systems such as biology and economics (and machine learning), but can occur even for simple physical systems such as ice melting when temperature increases. In machine learning, Bommasani et al. highlight the emergence of "in-context" or "few-shot" learning; other examples include arithmetic and broad multitask capabilities.

The companion to emergence is phase transitions, exemplified in the melting ice example. While not always the case, emergent behavior often quickly manifests at some threshold. Radford et al. (2018) provided the first hint of emergent few-shot capabilities that are now ubiquitous three years later. More strikingly, arithmetic capabilities in GPT-3 emerge from only a 30x increase in model size (Brown et al., 2020; page 22), and Power et al. (2021) show that similar capabilities can emerge s...
Truthful AI: Developing and governing AI that does not lie
by Owain_Evans — 2021-10-18T18:37:38.325Z

This post contains the abstract and executive summary of a new 96-page paper from authors at the Future of Humanity Institute and OpenAI.

Update: The authors are doing an AMA about truthful AI during October 26-27.

Abstract

In many contexts, lying – the use of verbal falsehoods to deceive – is harmful. While lying has traditionally been a human affair, AI systems that make sophisticated verbal statements are becoming increasingly prevalent. This raises the question of how we should limit the harm caused by AI “lies” (i.e. falsehoods that are actively selected for). Human truthfulness is governed by social norms and by laws (against defamation, perjury, and fraud). Differences between AI and humans present an opportunity to have more precise standards of truthfulness for AI, and to have these standards rise over time. This could provide significant benefits to public epistemics and the economy, and mitigate risks of worst-case AI futures.

Establishing norms or laws of AI truthfulness will require significant work to:

identify clear truthfulness standards;
create institutions that can judge adherence to those standards; and
develop AI systems that are robustly truthful.

Our initial proposals for these areas include:

a standard of avoiding “negligent falsehoods” (a generalisation of lies that is easier to assess);
institutions to evaluate AI systems before and after real-world deployment;
explicitly training AI systems to be truthful via curated datasets and human interaction.

A concerning possibility is that evaluation mechanisms for eventual truthfulness standards could be captured by political interests, leading to harmful censorship and propaganda. Avoiding this might take careful attention. And since the scale of AI speech acts might grow dramatically over the coming decades, early truthfulness standards might be particularly important because of the precedents they set.

Executive Summary & Overview

The threat of automated, scalable, personalised lying

Today, lying is a human problem. AI-produced text or speech is relatively rare, and is not trusted to reliably convey crucial information. In today’s world, the idea of AI systems lying does not seem like a major concern.

Over the coming years and decades, however, we expect linguistically competent AI systems to be used much more widely. These would be the successors of language mode...
Rationalism for New EAs
by Zach Stein-Perlman — 2021-10-18T16:00:18.692Z

I organize a college EA group. As has become common for college EA groups, we run an "introductory fellowship" to introduce people to EA. This year, I'm making changes from the standard syllabus; among other changes, I want to add 1–2 hours of reading and 1 hour of discussion (and a few hours of optional-reading-if-you-want-more) on rationalism. What readings should I include? That is, what rationalist content would be most interesting and valuable to undergraduates who are interested in EA and have no experience with rationalism — and what content would make the potential aspiring rationalists among them realize that they want to learn more?

I have ideas, but I'll share them later, I think. I'm not very confident that this is a good idea, but I think it's at least a worthwhile experiment.

Recommendations for rationalist content directly relevant to EA are good too, but I'm really asking for rationalism content in its own right. I already use some miscellaneous rationalism(-ish) content because it's excellent for EA:

a couple short pieces like Scope Insensitivity

A Modest Proposal: Currency and Efficient Charity on opportunity costs

some AI stuff (I'm also open to recommendations on how to introduce AI to new EAs)
[MLSN #1]: ICLR Safety Paper Roundup
by Dan H — 2021-10-18T15:19:59.766Z

As part of a larger community building effort, I am writing a monthly safety newsletter which is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.

Welcome to the 1st issue of the ML Safety Newsletter. In this edition, we cover:

various safety papers submitted to ICLR
results showing that discrete representations can improve robustness
a benchmark which shows larger models are more likely to repeat misinformation
a benchmark for detecting when models are gaming proxies
... and much more.

Discrete Representations Strengthen Vision Transformer Robustness

Overview of the proposed Vision Transformer that uses discrete representations. The pixel embeddings (orange) are combined with discrete embedded tokens (pink) to create the input to the Vision Transformer.

There is much interest in the robustness of Vision Transformers, as they intrinsically

scale better

than ResNets in the face of unforeseen inputs and distribution shifts. This paper further enhances the robustness of Vision Transformers by augmenting the input with discrete tokens produced by a vector-quantized encoder. Why this works so well is unclear, but on datasets unlike the training distribution, their model achieves marked improvements. For example, when their model is trained on ImageNet and tested on

ImageNet-Rendition

(a dataset of cartoons, origami, paintings, toys, etc.), the model accuracy increases from 33.0% to 44.8%.

Paper

Other Recent Robustness Papers

Improving test-time adaptation to distribution shift using data augmentation.

Certifying robustness to adversarial patches.

Augmenting data by mixing discrete cosine transform image encodings.

Teaching models to reject adversarial examples when they are unsure of the correct class.

TruthfulQA: Measuring How Models Mimic Human Falsehoods

Models trained to predict the next token are incentivized to repeat common misconceptions.

A new benchmark shows that GPT-3 imitates human misconceptions. In fact, larger models more frequently repeat misconceptions, so simply training more capable models may make the problem worse. For example, GPT-J with 6 billion parameters is 17% worse on this benchmark than a model with 0.125 billion parameters. This demonstrates that simple objectives can inadvertently incentivize models to be m...
Causal vs Predictive Models, and the Causal Taboo
by Hazard — 2021-10-18T15:05:10.457Z

(I wrote this post in April 2020 for a non-LW audience)

Causation is pretty cool. Even cooler than causation, causal models! If you haven't heard the news, the past few decades have produced big leaps in understanding causality and how to reason about it. There's also been great descriptive work on how humans already intuitively deal with causality. Causality is so baked into the human mind that causal relationships can often be experienced at the perceptual level, even before any higher level reasoning systems can act. We're very good at spotting causal relationships when they're present, so good that we sometimes even detect them when they aren't there :)

To get an understanding of the difference between causal models and predictive models we're gonna use the slogan "Correlation does not equal causation" as our entry.

To be fair, that graph isn't really even a good example. That's more "just because lines sorta match up on a graph, doesn't mean you should expect them to continue to math up." I'd totally bet that these variables wouldn't even be correlated if you picked a different time span.

Here's a better example of correlation not equaling causation (based on a true story, but simplified to make a point):

Your new study finds that people who listened to Mozart as a kid have higher SAT scores. Mozart makes you smarter! The meaning inherent in that claims is "If I intervene by playing Mozart for a kid, they will become smarter than they otherwise would have."

What could go wrong? Well it turns out that the actual causal graph looks more like this:

Different story. If you know someone listened to Mozart as a kid, it is still 100% legit an accurate to predict higher SAT scores, but now it's also clear that intervening on the Mozart variable won't affect one's SAT. Mozart is useful for predicting SAT because they are caused by the same variable. Mozart is evidence that their family is wealthy, and that causally affects SAT scores. But if you control for wealth, and the Mozart effect goes away.

(Also for completion, the causal graph probs looks more like this):

Moral of the story: if you do stats well, you'll have a good model that you can use to make robust predictions, but it can't inform intervention unless you have a causal model relating the variables.

So that's one nugget of wisdom that can be taken from "Correlation does not equal causation". But there's some in...
Predictive Categories Make Bad Causal Variables
by Hazard — 2021-10-18T15:02:58.099Z

(this was written in April 2020 and I only just now realized I never posted it to LW)

This post is going to explore the consequences of different choices you can make when thinking about things causally. Shout out to johnswentworth for first seeding in my head this sort of investigation.

One mistake people are known to make is to vastly underestimate the causal factors behind a variable. Scott writes about this tendency in genetics:

What happens if your baby doesn’t have the gene for intelligence? Can they still succeed? [...] By the early 2000s, the American Psychological Association was a little more cautious, was saying intelligence might be linked to “dozens – if not hundreds” of genes. [...] The most recent estimate for how many genes are involved in complex traits like height or intelligence is approximately “all of them” – by the latest count, about twenty thousand.

Probably not too surprising. Everyone wants "The One Thing" that explains it all, but normally its the case that "These 35,000 Things" explain it all. The Folk Theory of Essences might be the most egregious example of people inferring a mono-causal relationship when reality is vastly poly-causal. George Lakoff (the metaphors and embodied cognition guy) explains:

The Folk Theory of Essences is commonplace, in this culture and other cultures around the world. According to that folk theory, everything has an essence that makes it the kind of thing it is. An essence is a collection of natural properties that inheres in whatever it is the essence of. Since natural properties are natural phenomena, natural properties (essences) can be seen as causes of the natural behavior of things. For example, it is a natural property of trees that they are made of wood. Trees have natural behaviors: They bend in the wind and they can burn. That natural property of trees-being made of wood (which is part of a tree's "essence")-is therefore conceptualized metaphorically as a cause of the bending and burning behavior of trees. Aristotle called this the material cause.

As a result, the Folk Theory of Essences has a part that is causal. We will state it as follows: Every thing has an essence that inheres in it and that makes it the kind of thing it is. The essence of each thing is the cause of that thing's natural behavior.

Thinking in terms of essences is very common. It seems to be how a lot of people ...
Epistemic Strategies of Selection Theorems
by adamShimi — 2021-10-18T08:57:23.109Z

Introduction: Epistemic Strategies Redux

This post examines the epistemic strategies of John Wentworth’s

selection

theorem

posts

.

(If you want to skim this post, just read the Summary subsection that display the different epistemic strategies as design patterns)

I introduced the concept in

a recent post

, but didn’t define them except as the “ways of producing” knowledge that are used in a piece of research. If we consider a post or paper as a computer program outputting (producing) knowledge about alignment, epistemic strategies are the underlying algorithm or, even more abstractly, the

design patterns

.

An example of epistemic strategy, common in natural sciences (and beyond), is

Look at the data
Find a good explanation
Predict new things with that explanation
Get new data for checking your prediction

More than just laying out some abstract recipe, analysis serves to understand how each step is done, whether that makes sense, and how each step (and the whole strategy) might fail. Just like a design pattern or an algorithm, it matters tremendously to know when to apply it and when to avoid it as well as subtleties to be aware of.

Laying this underlying structure bare matters in three ways:

It clarifies the research’s purpose and value for newcomers and researchers from other fields, with minimal assumptions of shared approaches.

Just like a programmer switching to a new domain of problems will get up to speed faster and more reliably if they get access to the patterns/algorithms/tricks used in their new domain.

It focuses feedback and criticism on the most important parts of the idea/proposition/argument.

Issues with an algorithm more often focus on the point of it instead of the details,whereas issues with an implementation of that algorithm can be as much about typos, optimization tricks and bad structure than about the actual core (the algorithm.

It builds a library of such strategies for alignment in particular, a cookbook newcomers and senior researchers alike can browse for inspiration or a take on some new post/paper they don’t grok.

Like the glorious

Game Programming Patterns

who does exactly that for game programming

Now, before starting, I need to point out that the selection theorems posts don’t present research results; they present epistemic strategies (the eponymous theorems). Does that mean my job has already been done? Not exactly: John’s posts do presen...