trentmkelly/LessWrong-43k huggingface dataset:
Clever arguers give weak evidence, not zero
by dkl9 — 2023-07-18T17:07:27.126Z
Link: https://dkl9.net/essays/clever_arguer.html

Followup to: The Bottom Line

We are warned that an argument made with a predecided conclusion does not evidentially entangle with the truth it claims to address, and thus is no evidence of that claim.

Recall that observation E is evidence for hypothesis H iff P(E|H)>P(E|¬H). What do the variables mean here?

E: there's a convincing argument for the predecided conclusion
H: the predecided conclusion is really true
P(E|H): probability that the clever arguer can make a clever argument, given that the conclusion is true
P(E|¬H): likewise, given that the conclusion is false

What the commentary from "The Bottom Line" leaves out is that making a convincing argument is a nontrivial task. For many false claims, a clever arguer with ordinary resources cannot make a convincing argument. If it's typically easy to make a convincing argument for something false, you're convinced by the wrong things.

Thus P(E|H), in this case, would usually be greater than P(E|¬H). An argument for a claim from a clever arguer only clearly proves that the arguer wanted us to believe it. A convincing argument for a claim — and if the argument isn't convincing, you'd ignore it — proves that the claim has convincing arguments for it accessible to that level of arguer, which is correlated with the claim being true.

But maybe I decided at the start that clever arguments for predecided conclusions are actual evidence, thereby breaking the entanglement of the rest of this essay. Well, are you convinced anyway?
Measuring and Improving the Faithfulness of Model-Generated Reasoning 
by Ansh Radhakrishnan — 2023-07-18T16:36:34.473Z

TL;DR: In two new papers from Anthropic, we propose metrics for evaluating how faithful chain-of-thought reasoning is to a language model's actual process for answering a question. Our metrics show that language models sometimes ignore their generated reasoning and other times don't, depending on the particular task + model size combination. Larger language models tend to ignore the generated reasoning more often than smaller models, a case of inverse scaling. We then show that an alternative to chain-of-thought prompting — answering questions by breaking them into subquestions — improves faithfulness while maintaining good task performance.

Paper Abstracts

Measuring Faithfulness in Chain-of-Thought Reasoning

Large language models (LLMs) perform better when they produce step-by-step, “Chain-of -Thought” (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model’s actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT(e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT’s performance boost does not seem to come from CoT’s added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.

Question Decomposition Improves the Faithfulness of Model-Generated Reasoning

As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks. However, this approach relies on the stated reasoning faithfully reflecting the model’s actual reasoning, which is not always the case. To improv

...
Least-problematic Resource for learning RL?
by Dalcy — 2023-07-18T16:30:48.535Z

Well, Sutton & Barto is the standard choice, but:

Superficial, not comprehensive, somewhat outdated circa 2018; a good chunk was focused on older techniques I never/rarely read about again, like SARSA and exponential feature decay for credit assignment. The closest I remember them getting to DRL was when they discussed the challenges faced by function approximators.

And also has some issues with eg claiming that the Reward is the optimization target. Other RL textbooks also seem similarly problematic - very outdated, with awkward language / conceptual confusions.

OpenAI's Spinning Up DRL seems better in the not-being-outdated front, but feels quite high-level, focusing mostly on practicality & implementation - while I'm looking also for a more theoretical discussion of RL.

I'm starting to think that there probably isn't such a resource fitting all my bills, so I'm considering the mix of (1) lightly reading textbooks for old-RL theory discussions and (2) instead covering modern surveys for catching up to the recent DRL stuff.

Are there any resources for learning RL that doesn't contain (any of) the problems I've mentioned above? Would like to know if I'm missing any.
Charter Cities: why they're exciting & how they might work
by Jackson Wagner — 2023-07-18T13:57:17.524Z

Hello! What follows is a work-in-progress script about the idea of Charter Cities, which the EA-adjacent youtube channel RationalAnimations plans to animate soon. I want to make sure I'm presenting the idea of charter cities properly and in a compelling, understandable way, so I thought it would be helpful to post it here and get feedback from this forum!

Introduction: charter cities, as economic growth, as neartermism

When you think about ways to help people in developing countries, you probably think of international aid -- providing medical supplies, or coordinating disaster relief, or even just giving cash, like we discussed in our recent video about global poverty. These things are great, and this kind of charitable aid saves lives every day. But there’s something a little curious here, because these interventions AREN’T what helped the world’s most prosperous countries succeed in the first place.

Up until the 1700s, essentially the entire population of the planet lived in poverty. [1] That started to change during the industrial revolution, when humanity developed more efficient, mechanized ways of producing goods and capturing energy. Hundreds of millions of people were able to live better lives thanks to economic growth -- learning to use technology to grow food, make clothes, and get from place to place more efficiently.

The overwhelming importance of economic growth remains true in modern times -- by far the greatest ongoing reductions in poverty and suffering are coming not from international aid projects, but from development, as low-income countries find new ways to do things more efficiently and climb the ladder of technological advancement.

the mouth of the Singapore River, in 1976 versus today

For example, in the year 1960, Singapore was a poor and undeveloped country, producing only $428 per citizen. Today, singapore’s economy has grown by many times, to around $73,000 per person.[2] That’s an incredible amount of progress within a single lifetime.

Almost every country on earth has made some economic progress over the past few decades, but unfortunately only a few have succeeded as spectacularly as economic-development superstars like Singapore, or South Korea. There are hundreds of millions of people still living on just a few dollars per day, like Singaporians in 1960. S...
Narrative Theory. Part 6. Artificial Neural Networks
by Eris — 2023-07-18T09:22:56.983Z

“Alan Turing started off by wanting to 'build the brain' and ended up with a computer”
- Henry Markram, The Blue Brain Project

Recently I’ve come to terms with the idea that I have to publish my research even if it feels unfinished or slightly controversial. The mind is too complex (who would have thought), each time you think you get something, the new bit comes up and crushes your model. Time after time after time. So, waiting for at least remotely good answers is not an option. I have to “fail fast” even though it’s not a widely accepted approach among scientists nowadays.
With that, Reinforcement learning and in-depth analysis of the mentioned models will be covered later. The goal of this part is to explain the reasoning behind the choice of the surface area.

Artificial Neural Networks are the face of modern artificial intelligence and the most successful branch of it too. But success unfortunately doesn’t mean biological plausibility. Even though most ML algorithms have been inspired by the aspects of biological neural networks final models end up pretty far from the source material. This makes their usefulness for the quest of reverse engineering the mind questionable. What I mean here is that almost no insights can be directly brought back to neuroscience to help with the research. I’ll explain why so in a bit. (note, this doesn’t mean that they can not serve as an inspiration. This is very much possible and, I’m sure, a good idea.)

There are three main show-stoppers:

(Reason #1) is the use of an implausible learning algorithm (read backpropagation). There were numerous attempts at finding something analogous to the backpropagation but all of them felt short as far as I know. The core objection to the biological plausibility of backpropagation is that weight updates in multi-layered networks require access to information that is non-local (i.e. error signals generated by units many layers downstream) In contrast, plasticity in biological synapses depends primarily on local information (i.e., pre- and post-synaptic neuronal activity)[1].

(Reason #2) is the fact that ANNs are being used to solve “synthetic” problems. The vast majority of ANNs originated from industry, designed to solve some practical real-world problem. For us, this means that the training data used for these models would have almost nothing in common with the human ontogenet...
Train for incorrigibility, then reverse it (Shutdown Problem Contest Submission)
by Daniel_Eth — 2023-07-18T08:26:20.687Z

The following was my submission to the AI Alignment Awards Shutdown Contest, for which it won an honorary mention prize:

My proposal is as follows:
1) pre-train a model for broad capabilities (though not to AGI)

2) create a list of different sub-characteristics that in aggregate yield incorrigibility, but where no single sub-characteristic implies incorrigibility (eg., situational awareness, power-seeking, social persuasiveness, etc)

3) for each sub-characteristic, construct a task vector[1] from the pre-trained model to a fine-tuned model possessing the sub-characteristic; ie., for each sub-characteristic, fine-tune the pre-trained model for that sub-characteristic (eg., for situational awareness, directly teach it about its physical vulnerabilities) and then subtract the weights of the pre-trained model from those of the fine-tuned model to create the vector

3) negate each task vector from the pre-trained model (ie., take the weights of the pre-trained model, and subtract each of the task vectors from it)

4) either train the resultant model to completion, or return to step 1 (iterating through the process, possibly many times).

Research indicates negation of task vectors from model weights can remove specific characteristics with minimal degradation to capabilities (eg., subtracting a task vector for toxicity from a pre-trained LLM leads to both lower perplexity and less toxicity than fine-tuning the LLM on nontoxic text).[1] Further, task vectors are additive – adding task vectors for two separate tasks to the same pre-trained model can yield a model capable of performing both tasks.[1] My hope is that by negating task vectors for various sub-characteristic of incorrigibility, the model adopts aversion to exhibiting each sub-characteristic. Also, by training simply for each sub-characteristic separately (and iterating over the process multiple times), we’d hopefully avoid ever approaching an AGI with all the incorrigibility sub-characteristics.

Compared to “direct attempts” of training for corrigibility, this proposal offers potential benefits. First, it may be easier to target inclusion of sub-characteristics than to target their absence (eg., training a model to not possess situational awareness seems more difficult than training for it). Second, “direct attempts” may yield models on the edge of incorrigibility, versus the hope here is that tas...
The shape of AGI: Cartoons and back of envelope
by boazbarak — 2023-07-17T20:57:30.371Z

​​

[Cross posted on

windowsontheory

; see here for my

prior writings

]

There have been several studies to estimate the timelines for artificial general intelligence (aka AGI). Ajeya Cotra wrote a

report in 2020

(see also

2022 update

) forecasting AGI based on comparisons with “biological anchors.” That is, comparing the total number of FLOPs invested in a training run or inference with estimates on the total number of FLOPs and bits of information that humans/evolution required to achieve intelligence.[1] In a recent

detailed report

, Davidson estimated timelines via a simulation (which you can play with

here

) involving 16 main parameters and 51 additional ones. Jacob Steinhardt

forecasted

how will large language models look in 2030.

In this blog post, I will not focus on the timelines or risks but rather on the shape we could expect AI systems to take and their economic impact. I will use no fancy models or math and keep everything on the level of cartoons or back-of-envelope calculations. My main contention is that even post-AGI, AI systems will be incomparable to humans and stay this way for an extended period of time, which may well be longer than the time to achieve AGI in the first place.[2]

The cartoon below illustrates the basic argument for why we expect AGI at some point. Namely, since capabilities grow with log compute, and compute (by Moore’s law) grows exponentially with time; eventually, AI systems should reach arbitrarily high capabilities.

Figure: Left: Cartoon of capabilities of AI systems as a function of computational resources. To a first approximation, historical improvement in capabilities scaled with the logarithm of commute invested. There are scenarios of either “hitting a wall” (e.g., running out of data) or “self improvement” (once we get near AGI, we can start using AIs to design better AIs) that could break this line in one direction or the other. Still, the baseline scenario is a straight line. Right: Cartoon of computational resources available per “moonshot investment” (e.g., hundreds of Billion dollars) as a function of time. We expect the computational resources per dollar to increase exponentially with time, eventually reaching the threshold needed for AGI.

Notes: There are established estimates for the slope of the graph on the right (comp...
Predictive history classes
by dkl9 — 2023-07-17T20:48:31.363Z
Link: https://dkl9.net/essays/predictive_history.html

Epistemic status: serious proposal with known difficulties and problems

Why do we study history? There are many potential reasons. More specifically, why do we teach history to everyone going to school? Many reasons become less relevant to those other than historians or history teachers.

The classic reason is "so that we don't repeat it". But most people will not end up in a position of power so as to decide whether we repeat or don't repeat history. A more honest reason, for most people, is that we learn history to generalise its lessons to the future or present. Assuming that purpose, we find current methods of history classes dreadfully ineffective. I propose an alternative paradigm of history classes which, I believe, will result in much more practical learning.

A cynical reader remarks that history education in school (in the form I'm targeting) is for indoctrination. From that view, take this not as a proposal for reform, but a suggestion on how to study well for your own sake.

Current history education focuses on teaching the students about past events. Students are then tested on those same past events. In higher history classes, they are also tested on analysing information about these events and arguing about their causes and consequences.

This is great for those who will be historians. Most students will not. This style of teaching does not guarantee any understanding of how to apply historical lessons to the present and future.

In schools where students just study to the test, make the test a good one. Instead of testing students on past events they studied, test them on events they didn't study. They are then forced to learn how to generalise history, applying its lessons to understand what they're going thru and what may come next. To get an objective answer by which to grade the students, test them on actual past events — just obscure ones that they wouldn't already know about. Sith they didn't study the material, they aren't expected to ever get a perfect score. This would make history classes more like maths or science classes, in that the students learn methods to creatively apply to problems they haven't exactly seen before.

This guiding principle should apply beyond the tests. We would replace history classes with predictive history classes. We teach about events and the patterns which underlie them, assign students to study and predict parts of...
Highlights from The Industrial Revolution, by T. S. Ashton
by jasoncrawford — 2023-07-17T19:02:21.245Z
Link: https://rootsofprogress.org/ashton-industrial-revolution-highlights

The Industrial Revolution, 1760-1830, by Thomas S. Ashton, is classic in the field, published in 1948. Here are some of my highlights from it. (Emphasis in bold added by me.)

The role of chance

What was the role of chance in the inventions of the Industrial Revolution?

It is true that there were inventors—men like Brindley and Murdoch—who were endowed with little learning, but with much native wit. It is true that there were others, such as Crompton and Cort, whose discoveries transformed whole industries, but left them to end their days in relative poverty. It is true that a few new products came into being as the result of accident. But such accounts have done harm by obscuring the fact that systematic thought lay behind most of the innovations in industrial practice, by making it appear that the distribution of awards and penalties in the economic system was wholly irrational, and, above all, by over-stressing the part played by chance in technical progress. “Chance,” as Pasteur said, “favours only the mind which is prepared”: most discoveries are achieved only after repeated trial and error.

The revolution of ideas

Ashton gives weight to both material and intellectual causes of the Industrial Revolution:

The conjuncture of growing supplies of land, labour, and capital made possible the expansion of industry; coal and steam provided the fuel and power for large-scale manufacture; low rates of interest, rising prices, and high expectations of profit offered the incentive. But behind and beyond these material and economic factors lay something more. Trade with foreign parts had widened men’s views of the world, and science their conception of the universe: the industrial revolution was also a revolution of ideas.

What kind of ideas? For example:

The Enquiry into the Nature and Causes of the Wealth of Nations, which appeared in 1776, was to serve as a court of appeal on matters of economics and politics for generations to come. Its judgements were the material from which men not given to the study of treatises framed their maxims of conduct for business and government alike. It was under its influence that the idea of a more or less fixed volume of trade and employment, directed and regulated by the State, gave way—gradually and with many setbacks—to thoughts of unlimited progress in a free and expanding economy.

More on Smith’s influence:

In 1776 Adam Smith turned hi

...
Existential Risk Persuasion Tournament
by PeterMcCluskey — 2023-07-17T18:04:02.794Z
Link: https://bayesianinvestor.com/blog/index.php/2023/07/17/existential-risk-persuasion-tournament/

I participated last summer in Tetlock's Existential Risk Persuasion
Tournament (755(!) page paper
here).

Superforecasters and "subject matter experts" engaged in a hybrid
between a prediction market and debates, to predict catastrophic and
existential risks this century.

I signed up as a superforecaster. My impression was that I knew as much
about AI risk as any of the subject matter experts with whom I
interacted (the tournament was divided up so that I was only aware of a
small fraction of the 169 participants).

I didn't notice anyone with substantial expertise in machine learning.
Experts were apparently chosen based on having some sort of respectable
publication related to AI, nuclear, climate, or biological catastrophic
risks. Those experts were more competent, in one of those fields, than
news media pundits or politicians. I.e. they're likely to be more
accurate than random guesses. But maybe not by a large margin.

That expertise leaves much to be desired. I'm unsure whether there was
a realistic way for the sponsors to attract better experts. There seems
to be not enough money or prestige to attract the very best experts.

Incentives

The success of the superforecasting approach depends heavily on
forecasters having decent incentives.

It's tricky to give people incentives to forecast events that will be
evaluated in 2100, or evaluated after humans go extinct.

The tournament provided a fairly standard scoring rule for questions
that resolve by 2030. That's a fairly safe way to get parts of the
tournament to work well.

The other questions were scored by how well the forecast matched the
median forecast of other participants (excluding participants that the
forecasters interacted with). It's hard to tell whether that incentive
helped or hurt the accuracy of the forecasts. It's easy to imagine that
it discouraged forecasters from relying on evidence that is hard to
articulate, or hard to verify. It provided an incentive for groupthink.
But the overall incentives were weak enough that altruistic pursuit of
accuracy might have prevailed. Or ideological dogmatism might have
prevailed. It will take time before we have even weak evidence as to
which was the case.

One incentive that occurred to me toward the end of the tournament was
the possibility of getting a verified longterm forecasting track record.
Suppose that in 2050 they redo the scores based on evidence availa...