trentmkelly/LessWrong-43k huggingface dataset:
Announcing Squiggle: Early Access
by ozziegooen — 2022-08-03T19:48:16.727Z
Link: https://forum.effectivealtruism.org/posts/TfPdb2aMKzgWXFvc3/announcing-squiggle-early-access

Update: We've announced a $1,000 prize for early experimentation with Squiggle.

Introduction

Squiggle

is a special-purpose programming language for probabilistic estimation[1]. Think: "

Guesstimate

as a programming language." Squiggle is free and open-source.

Our team has been using Squiggle for QURI estimations for the last few months and found it very helpful. The

Future Fund

recently began experimenting with Squiggle for estimating the value of different grants.

Now we're ready for others publicly to begin experimenting with it. The core API should be fairly stable; we plan to add functionality but intend to limit breaking changes.[2]

We'll do our best to summarize Squiggle for a diverse audience. If any of this seems intimidating, note that Squiggle can be used in ways not much more advanced than Guesstimate. If it looks too simple, feel free to skim or read the docs directly.

Work on Squiggle!

We're looking to hire people to work on Squiggle for the main tooling. We're also interested in volunteers or collaborators for the ecosystem (get in touch!).

Links

Public Website, Github Repo, Previous LessWrong Sequence

What Squiggle is and is not

What Squiggle Is

A simple programming language for doing math with probability distributions.
An embeddable language that can be used in Javascript applications. This means you can use Squiggle directly in other websites.
A tool to encode functions as forecasts that can be embedded in other applications.

What Squiggle Is Not

A complete replacement for enterprise Risk Analysis tools. (See

Crystal Ball

,

@Risk

,

Lumina Analytica

)
A

probabilistic programming language

. Squiggle does not support Bayesian inference. (Confusingly, "Probabilistic Programming Language" really refers to this specific class of language and is distinct from "languages that allow for using probability.")
A tool for substantial data analysis. (See programming languages like

Python

or

Julia

)
A programming language for anything other than estimation.
A visually-driven tool. (See

Guesstimate

and

Causal

)

Strengths

Simple and readable syntax, especially for dealing with probabilistic math.
Fast for relatively small models. Useful for rapid prototyping.
Optimized for using some numeric and symbolic approaches, not just Monte Carlo.
Embeddable in Javascript.
Free and open-source (MIT license).

W

...
Survey: What (de)motivates you about AI risk?
by Daniel_Friedrich — 2022-08-03T19:17:35.822Z
Link: https://forms.gle/LdA5gwLgwHFqpZgw7

Share your hot takes

on what leads people to see AI risk as important! I will test the most promising hypotheses in a future poll.

The time you take to think about them is up to you but careful introspection may take at least 8 minutes.
Externalized reasoning oversight: a research direction for language model alignment
by tamera — 2022-08-03T12:03:16.630Z

Summary

In this post, I’ll present a research direction that I’m interested in for alignment of pretrained language models.

TL;DR: Force a language model to think out loud, and use the reasoning itself as a channel for oversight. If this agenda is successful, it could defeat deception, power-seeking, and other forms of disapproved reasoning. This direction is broadly actionable now.

In recent publications, prompting pretrained language models to work through logical reasoning problems step-by-step has provided a boost to their capabilities. I claim that this externalized reasoning process may be used for alignment if three conditions are met:

We develop trustworthy tests to check if the reported reasoning is authentic (complete, straightforward, and causally responsible for the output).
We train models that pass the tests of reasoning authenticity.
We oversee the externalized reasoning to detect reasoning of which we would not approve, and either filter the resulting actions or fine-tune against reasoning approval.

If these conditions hold, we should be able to detect and avoid models that reason through convergent instrumental goals to be deceptive, power-seeking, non-myopic, or reason through other processes of which we don’t approve. Reasoning oversight should provide stronger guarantees of alignment than oversight on model outputs alone, since we would get insight into the causally responsible reasoning process that gave rise to a certain output.

I believe that productive research can be done today toward all three of these conditions.

I’m doing exploratory work into this strategy as part of my SERI MATS project in the large language model alignment stream.

Background

I’ll use “externalized reasoning” to refer to a prompting strategy for a pretrained language model that has it read in a question, produce a number of steps of reasoning, and then produce an answer to the question as its final output. Some examples of this include chain of thought prompting [1, 2], selection-inference [3], and scratchpad reasoning [4]. These types of strategies have recently led to state of the art results on a number of logical reasoning benchmark tasks, most notably 50.3% accuracy on the MATH dataset [5].

A few examples of chain of thought reasoning can be found here.

Future work may develop externalized reasoning approaches different from those that currently exist; ideally, th...
Open & Welcome Thread - Aug/Sep 2022
by Thomas — 2022-08-03T10:22:53.266Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the new Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
How does one recognize information and differentiate it from noise? 
by M. Y. Zuo — 2022-08-03T03:57:35.432Z

The normal methods of explanation, and the standard definitions, for 'information', such as the 'resolution of uncertainty' are especially difficult to put into practice.

As these presuppose having knowledge already comprised, and/or formed from, a large quantity of information. Such as the concepts of 'uncertainty' and 'resolution'.

How does one know they've truly learned these concepts, necessary for recognizing information, without already understanding the nature of information?

This seems to produce a recursive problem, a.k.a, a 'chicken and egg' problem.

Additionally, the capability to recognize information and differentiate it from random noise must already exist, in order to recognize and understand any definition of information, in fact to understand any sentence at all. So it's a multiply recursive problem.

Since, presumably, most members of this forum can understand sentences, how does this occur?

And since presumably no one could do so at birth, how does this capability arise in the intervening period from birth to adulthood?
Law-Following AI 4: Don't Rely on Vicarious Liability
by Cullen — 2022-08-02T23:26:00.426Z

This post is written in my personal capacity, and does not necessarily represent the views of OpenAI or any other organization. Cross-posted to the Effective Altruism Forum.

Image by OpenAI's DALL·E

If an agent A causes some harm while intending to benefit a principal P, what is P's liability? The answer to this question is important because any liability to P should affect A's calculus (insofar as A is trying to benefit and avoid harming P). Liability to P would help deter A from causing harm.

If A is a human, the law currently provides at least two mechanisms for discouraging :

Making A directly liable, and

Making P vicariously liable for A's actions.[1]

What if A is an AI? AIs are not (yet?) legal persons, and so cannot yet be held directly liable. Thus, the main legal deterrent would have to work on P or some other person in the causal chain, such as the developer of the AI agent (who may not be the principal). However, there are several reasons to worry about relying on this as a strategy to make A compliant (which, in this case, means not tortiously harming others) under the current state of law and AI.

First, the problem of evasion still remains. Because sufficiently intelligent agents can evade detection and attribution, A may often (perhaps usually) prefer evasion over compliance when compliance would hinder A's ability to benefit P.

Second, the applicability and appropriateness of various theories of vicarious liability to the actions of AI agents is heavily debated in legal scholarship.[2] These debates have cast some doubt on whether/which harms from AI "agents" can properly give rise to liability to human principals.[3] Other possible theories of human liability—such as products liability—also face doctrinal challenges.[4]

Third, note that relying on vicarious liability alone leaves A under fewer constraints than an analogous human would. Under most vicarious liability regimes, A would still be directly liable for her actions, even if P would also be vicariously liable. It seems unwise to legally constrain A less than we constrain humans in analogous circumstances.

Finally (and most decisively in my opinion) developing a theory that assigns liability to P based on A's actions (or actions + "mental" state) dramatically lowers the bar for creating an LFAI system in the first place. Once we have such a theory, A (if intent-aligned) should indeed incorpo...
Two-year update on my personal AI timelines
by Ajeya Cotra — 2022-08-02T23:07:48.698Z

I worked on my draft report on biological anchors for forecasting AI timelines mainly between ~May 2019 (three months after the release of GPT-2) and ~Jul 2020 (a month after the release of GPT-3), and posted it on LessWrong in Sep 2020 after an internal review process. At the time, my bottom line estimates from the bio anchors modeling exercise were:[1]

Roughly ~15% probability of transformative AI by 2036[2] (16 years from posting the report; 14 years from now).

A median of ~2050 for transformative AI (30 years from posting, 28 years from now).

These were roughly close to my all-things-considered probabilities at the time, as other salient analytical frames on timelines didn’t do much to push back on this view. (Though my subjective probabilities bounced around quite a lot around these values and if you’d asked me on different days and with different framings I’d have given meaningfully different numbers.)

It’s been about two years since the bulk of the work on that report was completed, during which I’ve mainly been thinking about AI. In that time it feels like very short timelines have become a lot more common and salient on LessWrong and in at least some parts of the ML community.

My personal timelines have also gotten considerably shorter over this period. I now expect something roughly like this:

~15% probability by 2030 (a decrease of ~6 years from 2036).

~35% probability by 2036 (a ~3x likelihood ratio[3] vs 15%).

This implies that each year in the 6 year period from 2030 to 2036 has an average of over 3% probability of TAI occurring in that particular year (smaller earlier and larger later).

A median of ~2040 (a decrease of ~10 years from 2050).

This implies that each year in the 4 year period from 2036 to 2040 has an average of almost 4% probability of TAI.

~60% probability by 2050 (a ~1.5x likelihood ratio vs 50%).

As a result, my timelines have also concentrated more around a somewhat narrower band of years. Previously, my probability increased from 10% to 60%[4] over the course of the ~32 years from ~2032 and ~2064; now this happens over the ~24 years between ~2026 and ~2050.

I expect these numbers to be pretty volatile too, and (as I did when writing bio anchors) I find it pretty fraught and stressful to decide on how to weigh various perspectives and considerations. I wouldn’t be surprised by significant movements.

I...
What are the Red Flags for Neural Network Suffering? - Seeds of Science call for reviewers
by rogersbacon — 2022-08-02T22:37:59.448Z

Seeds of Science is a new journal (funded through Scott Alexander's ACX grants program) that publishes speculative or non-traditional articles on scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them).

We just sent out an article for review - "What are the Red Flags for Neural Network Suffering?" - that may be of interest to some in the LessWrong community, so I wanted to see if anyone would be interested in joining us a gardener to review the article. It is free to join and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so it's no worries if you don't plan on reviewing very often but just want to take a look here and there at what kinds of articles people are submitting). Another unique feature of the journal is that comments are published along with the article after the main text.

To register, you can fill out this google form. From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review form for recording votes/comments.

Happy to answer any questions about the journal through email or in the comments below. Here is the abstract for the article.

What are the Red Flags for Neural Suffering?

By [redacted] and [redacted]

Abstract:

Which kind of evidence would we need to see to believe that artificial neural networks can suffer? We review neuroscience literature, investigate behavioral arguments and propose high-level considerations that could shift our beliefs. Of these three approaches, we believe that high-level considerations, i.e. understanding under which circumstances suffering arises as an optimal training strategy, is the most promising. Our main finding, however, is that the understanding of artificial suffering is very limited and should likely get more attention.
Againstness
by CFAR!Duncan — 2022-08-02T19:29:14.221Z

Epistemic status: Mixed

The concepts underlying the Againstness model (such as the division of the autonomic nervous system into the sympathetic and parasympathetic subsystems, or the bidirectional relationship between physiology and stress response) are all well-established and well-understood. The relationship between SNS activation and the experience of stress is somewhat less well-established, but still has significant research behind it. The evidence supporting physiological interventions for stress reduction is slightly less firm. The formal combination of all of the above into a practical technique for changing one’s psychological state and reasoning ability is therefore tenable, but vulnerable to disconfirmation.

We often pay insufficient attention to the fact that our minds live inside of our bodies, and cannot help but be powerfully influenced by this fact. The fields of economics, decision theory, and heuristics & biases have plenty to say about human irrationality, and disciplines like embodied cognition and evolutionary anthropology are uncovering more and more about how our physiology affects our thinking, but there’s currently not much bridging the gap, and where such connections do exist, they often offer little in the way of concrete guidance or next actions.

The Againstness technique is the tip of what we hope will prove to be a very large iceberg, with lots of useful content for developing physical rationality and overcoming metacognitive blindspots. It’s less an algorithm, and more a set of reminders about how to deal with the reality of being a program that wrote itself, running on a computer made of meat.

Mental shutdown

Sometimes, under certain kinds of stress, key parts of our mental apparatus shut down. Depending on the circumstances, we might have trouble thinking clearly about consequences, making good choices, or noticing and admitting when we’re wrong.

This isn’t always the case, of course. Sometimes, stress is energizing and clarifying. Sometimes the pressing need to act helps bring the important things into focus, and empowers us to take difficult-but-necessary actions.

The trouble is, most of us don’t know how to choose which of these effects a given stressor will have on us, and—from the inside—many of us struggle to tell them apart. Have you ever found yourself incensed in the middle of an argument because the other party had t...
(Summary) Sequence Highlights - Thinking Better on Purpose
by qazzquimby — 2022-08-02T17:45:26.859Z

Rationality is a very useful thing to learn, but while there is good curated reading material, it's not always easy reading. More than once I've wanted to introduce someone to the topics, but couldn't hope for them to dig through a pile of essays.

This is an attempt to trim some of the most important texts down into something very concise and approachable. All of the original texts are well worth reading, or I wouldn't be summarizing them. I make no claim to be able to do them justice, but I can try to optimize them for different readers.

Akash has written a much shorter summary of all highlights.

Images generated by Midjourney, prompted by the post title.

The Lens That Sees Its Flaws

Full Text by Eliezer Yudkowsky

When you look at an optical illusion, you're aware that what you're seeing doesn't match reality. As a human, you have the exceptional ability to be able to understand this, that your mental model of the world is not the same as the actual world around you. You are seeing a warped image through a flawed lens. Because you know this, you can manually correct yourself - "no, it's not actually moving" - and have a more accurate model than you would have on autopilot.

Our brains are riddled with systematic errors, mistakes people make so often you could bet money on it. But brains are not magical. The systems making those errors can be understood, anticipated, and corrected for.
The human brain is a flawed lens that can see its own flaws. By learning, noticing, and correcting for distortions, the lens can become far more powerful.

What Do We Mean By "Rationality"?

Full Text by Eliezer Yudkowsky

Rationality is about being right and succeeding.

You think you have milk in the fridge when you don't, and when you come home milkless from shopping you're disappointed. You had a false belief. Your mental map of the world didn't match reality, and so you're steered into an obstacle you couldn't see. Epistemic rationality is about making your map more accurate - to seek the truth - which makes it much easier to navigate.

Instrumental rationality is using that mental map to steer towards a future you want. To win.

A rational belief is one that is more likely to be true. There are other reasons to believe something, like the belief feeling nice, but you can't draw an accurate map based on what feels nice. Expecting it to be sunny because of the forecast is rational, while...
Progress links and tweets, 2022-08-02
by jasoncrawford — 2022-08-02T17:03:51.605Z
Link: https://rootsofprogress.org/links-and-tweets-2022-08-02

Announcements

Build Nuclear Now, a new policy campaign to enable advanced nuclear reactors (h/t @atrembath). Related, the NRC has (finally) approved the design for NuScale’s small modular reactor (via @NRCgov)
A new online PhD-level course in the economics of innovation from the Institute for Progress (via @calebwatney)
DeepMind has released structures for almost every known protein (200M+) (h/t @pushmeet)
Brad Harris’s podcast series “How it Began” available for only $15.00 through Aug 31 using promo code “humanprogress” (h/t @tonymmorley)

Opportunities

Convergent Research is hiring (via @AdamMarblestone who says they need “exceptional people with the right mix of ‘crazy’ and ‘grounded’”)
The Long Now Foundation is seeking a full-time Chief of Operations (via @zander)

Links

The Wonder World of Chemistry, a DuPont promotional film from 1936. Also, The Fourth Kingdom: The Story of Bakelite Resinoid (1942). (Highlights in this short thread)
Arnold Kling on GDP as a measure of economic activity, plus a followup on what GDP is and is not (a measure of well-being or even of production).
Isochronic maps showing how far a 5-hour train ride will take you from any city in Europe (h/t @_benjamintd)
Casey Handmer’s plan to refill the Colorado River from the Pacific (h/t @natfriedman)
Lant Pritchett attempts to explain the scarcity of rigorous evaluations of public policy (h/t @michael_nielsen)
Accountability in Florida for permitting delays (Washington Post). Related, the average San Francisco project takes nearly four years to be permitted (The Atlantic, h/t @DKThomp). Also, a thread from @itsahousingtrap on the irrationality of the housing planning process

Queries

What are the best movies about progress? (@MoritzW42)
Why are we seeing a shift towards older people across a broad variety of roles?
How is the LHC not an absolute disaster? (@albrgr)
Is there a Thiel Fellowship type program for PhD candidates to bail out of academia? (@tayroga)

Quotes

Intriguing take: the quantitative spirit of science originated in capitalism?

Retweets

A new philosophy of the future is needed (@elonmusk)
Problem-solvers vs. problem-sellers (@BoyanSlat)
Environmentalists should closely examine how Sri Lanka’s “organic“ policy played out (@stewartbrand)
Will people in 2122 be even more astonished at our quality of life than we are looking back at people 1000 years ago? (@sama)
Manchin, Biden, Schumer, and Pelosi hav

...
I want to donate some money (not much, just what I can afford) to AGI Alignment research, to whatever organization has the best chance of making sure that AGI goes well and doesn't kill us all. What are my best options, where can I make the most difference per dollar?
by lumenwrites — 2022-08-02T12:08:46.674Z

I don't understand this field well, I'm hoping you guys can help me out.