trentmkelly/LessWrong-43k huggingface dataset:
The Language of Bird
by johnswentworth — 2021-06-27T04:44:44.474Z

This is a fictional snippet from the AI Vignettes Day. I do not think this is a likely future, but it’s a useful future to think about - it gives a different lens for thinking about the alignment problem and potential solutions. It’s the sort of future I’d expect if my own research went far better than I actually expect, saw rapid adoption, and most other ML/AI research stalled in the meantime.

[Transcript from PyCon 2028 Lightning Talk. Lightly edited for readability.]

Ok, so, today we’re going to talk about Bird, and especially about the future of Bird and machine learning.

First of all, what is Bird? I assume everyone here has heard of it and played around with it, but it’s hard to describe exactly what it is. You’ve maybe heard the phrase “human concept library”, but obviously this isn’t just a graph connecting words together.

Some background. In ye olden days (like, ten years ago) a lot of people thought that the brain’s internal data structures were inherently illegible, an evolved hodgepodge with no rhyme or reason to it. And that turned out to be basically false. At the low level, yeah, it’s a mess, but the higher-level data structures used by our brains for concept-representation are actually pretty sensible mathematical structures with some nice universal properties.

Those data structures are the main foundation of Bird.

When you write something like “from Bird.World import Bookshelf”, the data structure you’re importing - the Bookshelf - is basically an accurate translation of the data structure your own brain uses to represent a bookshelf. And of course it’s hooked up to a whole world-model, representing all the pieces of a bookshelf, things you put on a bookshelf, where you’d find a bookshelf, etc, as well as the grounding of all those things in a lower-level world model, and their ultimate connection to sensors/actuators. But when writing code using Bird, we usually don’t have to explicitly think about all that. That’s the beauty of the data structures: we can write code which intuitively matches the way we think about bookshelves, and end up with robust functionality.

Functionally, Bird is about translation. It’s a high-level language very close to the structure of human thought. But unlike natural language, it’s fully formally specified, and those formal specifications (along with Bird’s standard training data set) accurately capture our own intuiti...
Deep limitations? Examining expert disagreement over deep learning
by Richard_Ngo — 2021-06-27T00:55:53.327Z
Link: https://link.springer.com/article/10.1007/s13748-021-00239-1

A recent publication by Carla Zoe Cremer, who's working at the Future of Humanity Institute:

We conducted 25 expert interviews resulting in the identification of 40 limitations of the deep learning approach and 5 origins of expert disagreement. These origins are open scientific questions that partially explain different interpretations by experts and thereby elucidate central issues in AI research. They are: abstraction, generalisation, explanatory models, emergence of planning and intervention. We explore both optimistic and pessimistic arguments that are related to each of the fve key questions. We explore common beliefs that underpin optimistic and pessimistic argumentation. Our data provide a basis upon which to construct a research agenda that addresses key deep learning limitations.

Abstraction: Do current artificial neural networks (ANNs) form abstract representations effectively?
Generalisation: Should ANNs’ ability to generalise inspire optimism about deep learning?
Explanatory, causal models: Is it necessary, possible and feasible to construct compressed, causal, explanatory models of the environment as described in Lake et al. (2017) using deep learning?
Emergence of planning: Will sufficiently complex environments be sufficient in enabling deep learning algorithms to develop the capacity for hierarchical, long-term reasoning and planning?
Intervention: Will deep learning support and require learning by intervening in a complex, real environment?

Personally, I'm fairly confident that the first three won't be major problems for deep learning. I'm much more uncertain about the fourth and fifth, since they correspond to types of data that seem quite difficult to obtain. (I'm happy to agree with the fourth in principle, but in practice the "sufficient complexity" might be well beyond the sorts of training environments AI researchers currently think about.)
What will be the aftermath of the US intelligence lab leak report?
by ChristianKl — 2021-06-26T20:33:22.098Z

I wrote a long post laying out the evidence for the lab leak theory. It's very likely that when the US intelligence community reports on 25. August on their data about the orgins of the COVID-19 they will conclude that it was a lab leak. This then will lead to the discovery that the virology community mislead the public and Fauci will be forced to resign. Once he's resigned a senate committee will have a lot of fun asking him how the funding decision for Baric & Shi was made in 2015 while the gain of function ban was in place and why the safety framework for gain of function research that was instituted after the ban was lifted in 2019 was never applied to the grant even so it was paid till 2019. They will also enjoy subpoenaing Twitter and Facebook for the documents about how those started censoring the lab leak hypothesis.

These events make it quite likely that we have a huge crisis of faith in the medical establishment. Milton Freedman used to say: “Only a crisis - actual or perceived - produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around. That, I believe, is our basic function: to develop alternatives to existing policies, to keep them alive and available until the politically impossible becomes the politically inevitable.”

There are likely two months to plan what to do once things blow up and it might be valuable to plan for the scenario. Is this the time where structural change to the medical system can be made? What solutions should be proposed to deal with malicious and incompetent scientists?

Unfortunately, there's a good chance this will happen while we have a new SARS-CoV-2 variant that would be better fought by giving a third vaccination dose. It might be pretty important to have Novavax around as an alternative to the current vaccines to give people alternatives to the current solutions.

Lastly, this scenario might be worth trading on. This isn't financial advice, but if you want to trade it might be worthwhile about coming up with your own probability for this event happening and thinking through what it will do to the markets.
[AN #153]: Experiments that demonstrate failures of objective robustness
by Rohin Shah — 2021-06-26T17:10:02.819Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Empirical Observations of Objective Robustness Failures (Jack Koch, Lauro Langosco et al) (summarized by Rohin): This paper presents empirical demonstrations of failures of objective robustness. We've seen objective robustness (AN #66) / inner alignment (AN #111) / mesa optimization (AN #58) before; if you aren't familiar with it, I recommend reading one of those articles (or their summaries) before continuing. This paper studies these failures in the context of deep reinforcement learning and shows these failures in three cases:

1. In CoinRun (AN #79), if you train an agent normally (where the rewarding coin is always at the rightmost end of the level), the agent learns to move to the right. If you randomize the coin location at test time, the agent will ignore it and instead run to the rightmost end of the level and jump. It still competently avoids obstacles and enemies: its capabilities are robust, but its objective is not. Using the interpretability tools from Understanding RL Vision (AN #128), we find that the policy and value function pay much more attention to the right wall than to the coin.

2. Consider an agent trained to navigate to a cheese that is always placed in the upper right corner of a maze. When the location of the cheese is randomized at test time, the agent continues to go to the upper right corner. Alternatively, if the agent is trained to go to a yellow gem during training time, and at test time it is presented with a yellow star or a red gem, it will navigate towards the yellow star.

3. In the keys and chest environment (AN #67), an agent trained in a setting where keys are rare will later collect too many keys once keys become commonplace.

Read more: Paper: Objective Robustness in Deep Reinforcement Learning

Rohin's opinion: I'm glad that these experiments have finally been run and we have actual empirical examples of the phenomenon -- I especially like the CoinRun example, since it is particularly clear that i...
Improving parliamentary information processing
by victorsintnicolaas — 2021-06-26T16:11:27.567Z

Crossposted from Equilibria Club

You can see organizations as information processing units: they eat and process information after which they create all of the products and services which we see around us. Parliaments seem to be especially constrained in their processing capacity, as they are at the center of governmental decisionmaking. I argue for various ways to reduce the amount and scope of decisions which parliaments have to make, e.g. using direct democracy or delegating legislative tasks to other institutions.

Information flow between organizations

Organizations may be coordinating lots of complex activities, but this is not revealed to customers (when things go according to plan). Customers usually end up seeing just a simple interface: if you pay us X we will let you use product Y.

Regardless of how much complex exchange happens within organizations, the markets between them are thin spaces of interaction. Such market interfaces can be described by function signatures. Imagine a company specialized in selling chairs and banana's; you may then be able to:

buy(10 EUR) -> chair

buy(100 EUR) -> expensive banana

call(1 EUR) -> refund for your expensive banana

Governments are also providing lots of services, for which either the inputs or outputs are often a lot more complex than your typical company interface:

declare_taxes(personal_details, tax_data) -> stay_compliant

request_subsidy(personal_details, 10 forms) -> subsidy

customer_service() -> help_with_taxes

Information flow within organizations

What about what happens within organizations? Things are actually no different there. In order to create order out of chaos, organisations are often split up into several parts. This is symbolized by the famous 2-pizza team concept from Amazon (see also Stevey's Google rant for amusing commentary):

get_materials_from_marketing(requirements) -> marketing_text

fancy_website_upgrade(requirements) -> website

salary_increase(charm, wit) -> raise

Large organizations such as governments are split up into many parts at many different layers of abstraction. On a high level, many governments are splitup between executive, legislative and judicial branches. On lower levels, different departments do different things, from marketing to bookkeeping to management. Ultimately, this enables speci...
Cliffnotes to Craft of Research parts I, II, and III
by Quinn — 2021-06-26T14:00:53.560Z

supported by CEEALAR

I happened to glance over at the CEEALAR bookshelf one day and saw a book called Craft of Research. Craft of Research is a book by 5 english professors that tries to provide a field-agnostic guide to, well, doing research. I previously shared my notes piecemeal in my lesswrong shortform, and today I'm collecting them into one top-level post. There will occasionally be reflections on the relevance of this book to the alignment community throughout.

The audience models of research - thoughts on chapter 2

Writers can't avoid creating some role for themselves and their readers, planned or not

Before considering the role you're creating for your reader, consider the role you're creating for yourself. Your broad options are the following

I've found some new and interesting information - I have information for you

I've found a solution to an important practical problem - I can help you fix a problem

I've found an answer to an important question - I can help you understand something better

The authors recommend assuming one of these three. There is of course a wider gap between information and the neighborhood of problems and questions than there is between problems and questions! Later on in chapter four the authors provide a graph illustrating problems and questions: Practical problem -> motivates -> Research question -> defines -> Conceptual/research problem. Information, when provided mostly for novelty, however, is not in this cycle. Information can be leveled at problems or questions, plays a role in providing solutions or answers, but can also be for "its own sake".

I'm reminded of a paper/post I started but never finished, on providing a poset-like structure to capabilities. I thought it would be useful if you could give a precise ordering on a set of agents, to assign supervising/overseeing responsibilities. Looking back, providing this poset would just be a cool piece of information, effectively: I wasn't motivated by a question or problem so much as "look at what we can do". Yes, I can post-hoc think of a question or a problem that the research would address, but that was not my prevailing seed of a reason for starting the project. Is the role of the researcher primarily a writing thing, though, applying mostly to the final draft? Perhaps it's appropriate for early stages of the research to involve multi-role drifting, e...
Why you should try a live reading session
by Nihal M — 2021-06-26T08:30:25.591Z

For the last 8 months, I've been running live reading sessions with some of my friends, co-workers, and members of the local SSC/LW group. In this post I wish to share some of my findings on it's effectiveness, advantages and potential failure points

What is a live reading session

I was inspired by this tweet to try out the deep reading sessions mentioned. The basic format is as follows:

Pick something to read
People show up for the session with no homework
People take turns reading out live and highlighting/annotating
Discuss what was read at regular break points and at the end of the session.

I'm calling them "live reading" sessions instead of "deep reading" sessions as I believe the killer feature is the process of "reading live" with others.

How I run them

I took the basic idea from the tweet, and have run three independent groups doing reading sessions on a regular basis. All the groups are still active, and my week is filled with 4-6 hours of these sessions.

Group A - (9 months old) with two friends reading full length books and occasionally essays
Group B - (6 months old)with co-workers and some friends reading essays and articles only
Group C - (1 month old) with the local SSC/LW group reading essays and articles only

I have modified the original formula for all these groups based on the group dynamics and with the goal of having continuously active sessions. The main difference is not having live annotations/highlighting.

What makes it work

Active discussions

I've run SSC/LW meetups which discuss some of Scott's essays and not everyone in the group was able to participate at the same level. Among other reasons, this was also due to different people not recalling parts of the essay or major points of the essay. Partial recall deters conversations, and having the essay read moments earlier makes it very easy for people to chime in.
Some discussion points that you think of while reading while reading are not necessarily present or remembered after the whole essay is read. Having the discussion much earlier in the pipeline from reading--->recall--->discussion is much more unfiltered.

Lower Barrier of Entry

In a typical reading club, people are expected to do homework of reading a specified section beforehand (Which is perfectly reasonable), so that all the time is devoted to discussion. From my experience in trying and failing to sustain such...
Anthropics and Embedded Agency
by dadadarren — 2021-06-26T01:45:06.880Z

I am not very familiar with the AI field, which is probably why I have only recently come across Scott Garrabrant's posts about embedded agency. To my surprise, I found it connected with anthropics - something I have been interested in for a long time. In particular, my solution to the paradoxes: Perspective-Based Reasoning (PBR) is highly similar to the non-embedded model (dualist agency). I feel this connection between the two fields deserves more attention.

Perspective-Based Reasoning

My purposed solution to the anthropic paradoxes (PBR) can be summarized as the following.

1. Perspective-based reasoning is more fundamental than objective reasoning.

It just means thinking from a perspective (e.g. your first-person perspective) is more basic than thinking perspective-independently. This is opposite to the common notion that objective reasoning is fundamental whereas perspectives and indexicals are additional information.

2. Perspective is primitive.

Perspective is a reasoning starting point. Just like an axiom, it can only be regarded as given, not further analyzed. This is why in anthropics, self-locating probabilities such as “I am this particular person” in Doomsday Argument or “Today is Monday” in Sleeping Beauty Problem are meaningless.

3. Reasoning from different perspectives shouldn't mix.

Like statements derived from different axiomatic systems, reasoning from different perspectives should not mix. So switching to a different perspective halfway in reasoning could lead to inconsistency. It is also the reason for perspective disagreement in anthropics. (something all halfers have to recognize)

This idea leads to a double-halfer position on the sleeping beauty problem. It suggest there is no reference class for the agent itself (or the moment now). PBR rejects the doomsday argument, simulation argument, and arguments related to fine-tuned universe.

Similarities with Dualist Agency

1. A dualistic agent exists outside of its environment. It primitively carves the world into "agent" and "environment" with an input-output relationship. This is mirrored in PBR regarding the perspective (self) as primitively defined. The input-output relation in the AI context is mirrored by our perception and subjective experience as highlighted here. It is why there is no valid reference class for self

2. Dualists agents don't tend to model t...
Internal Information Cascades
by JenniferRM — 2021-06-25T16:35:13.870Z

Epistemic Status: This was written in 2010 and existed in LW's editing purgatory ever since. It doesn't seem extremely wrong. ("Remember 'light side epistemology'? Pepperidge Farms remembers.") Perhaps the biggest flaw is that I didn't hit the publication button sooner and this meta-flaw is implicit in the perspective being described? Maybe the last 10 years would have gone better, but it is hard to reverse the action of publishing. Some links don't work. I fixed a few typos.. Perhaps link archiving will one day receive more systematic attention and links here which once worked, and now do not, can be restored via some kind of archival reconciliation?

A classic information cascade is a situation where early decisions by individuals in a group bias the decisions of other people in same group in the same direction. When people talk about information cascades they're generally talking about "dumb herd behavior".

If one person observed something and justified a behavior of their own in a robust way, their behavior, observed, is indirect evidence of their observations and reasoning. If 10 people update based on a shallow observation of the first person, their own behavior is not actually 10 times more evidence. It would be more informative for there to be 11 independent people who all just happened to think and act alike independently.
The classic example of multi-agent information cascades is probably an economic bubble, where early investment decisions are "irrationally" mimicked by later speculators creating a temporary increase in demand for the investment. Purchasing behavior expands through a community like a wave and the "genuine" opportunity here is to buy ahead of the wave and "flip" the investment shortly afterwards to a "greater fool" who found out about the bubble later than yourself. Aside from obvious moral concerns, the selfish worry is that bubbles are basically self organized Ponzi schemes where the last round of investors is the largest, and they lose a substantial amount of money to early adopters who successfully executed the "flip" maneuver before the crash. The "average investor" has no easy way to tell that they are a late entrant who will lose money.
In this community, RichardKennaway called attention to a paper about classical cascades in academic citations and Johnicholas wrote about similar potential in voting and ka...
Announcing the Replacing Guilt audiobook
by gianlucatruda — 2021-06-25T13:38:54.756Z

tldr; Complete narration of the Replacing Guilt series is available here: anchor.fm/guilt

I discovered Nate Soares' Replacing Guilt series in late 2019 via the Bayesian Conspiracy podcast and found it immensely valuable. Over ~40 posts, Nate channels the guilt-based motivation that is common in rationalists / EAs into more productive emotional drives.

I read the series on my Kindle, thanks to lifelonglearner's ePUB version, but was disappointed that no audio version existed. To make it more accessible, I reached out to Nate and got permission to produce the official audiobook.

The rationalist community has a fantastic tradition of volunteer narration — Slate Star Codex , AI to Zombies, and, of course, HPMOR. These free resources have added immense value to my life, so narrating Replacing Guilt feels like my tiny way of reinvesting in our rationalist commons.

For a 2-minute summary of the series and why you might be interested in it, check out this post (or listen to the audio version).

You can find the individual episodes at anchor.fm/guilt. The complete audiobook can be streamed in any podcast app, or you can download the mp3 file here.
Welcome to Norfolk Rationalists [Hampton Roads, Virginia]
by Willa — 2021-06-25T02:56:55.319Z

Hi!

The meetups I organize are open to all, and unless mentioned otherwise are focused on building a social group and community in the area. You do not need special skills nor background familiarity to attend these meetups, if it seems like something you'd be interested in please don't hesitate to show up, we welcome all at our main (social) meetups!

If there are enough group members and meetup attendees, and enough interest in specific ideas then I'm happy to organize smaller more focused on something specific / productive kinds of meetups and events.

Each social meetup will include 1 or 2 suggested readings so that we can have a common shared topic to start from, but meandering as far as our interests take us is welcome!

Note that even if you are only interested in smaller, specific / focused meetups, it is strategically beneficial to build up a social rationalist community in your area, because that's where such groups and events typically draw from. Thus, if you don't want to attend social meetups here, please still reach out to me so I can connect you with anyone else who may share your same interest in the area, if they are available and willing.

For a little more info about me and this area, please see this post or click on my profile.

We have a google group! Norfolk Rationalists: norfolk-rationalists@googlegroups.com
Join so that you can receive a calendar invite for any scheduled meetups.

Cheers,

Willa