trentmkelly/LessWrong-43k huggingface dataset:
Ultimate ends may be easily hidable behind convergent subgoals
by TsviBT — 2023-04-02T14:51:23.245Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2022/12/ultimate-ends-may-be-easily-hidable.html. First completed December 18, 2022.]

Thought and action in pursuit of convergent instrumental subgoals do not automatically reveal why those subgoals are being pursued--towards what supergoals--because many other agents with different supergoals would also pursue those subgoals, maybe with overlapping thought and action. In particular, an agent's ultimate ends don't have to be revealed by its pursuit of convergent subgoals. It might might therefore be easy to covertly pursue some ultimate goal by mostly pursuing generally useful subgoals of other supergoals. By the inspection paradox for the convergence of subgoals, it might be easy to think and act almost comprehensively like a non-threatening agent would think and act, while going most of the way towards achieving some other more ambitious goal.

Note: the summary above is the basic idea. The rest of the essay analyzes the idea in a lot of detail. The final main section might be the most interesting.

What can you tell about an agent's ultimate intent by its behavior?

An agent's ultimate intent is what the agent would do if it had unlimited ability to influence the world. What can we tell about an agent's ultimate intent by watching the external actions it takes, whether low-level (e.g. muscle movements) or higher-level (e.g. going to the store), and by watching its thinking (e.g. which numbers it's multiplying, which questions it's asking, which web searches it's running, which concepts are active)?

Terms

The cosmos is everything, including the observable world, logical facts, observers and agents themselves, and possibilities.

A goal-state is a state of the cosmos that an agent could try to bring about. (A state is equivalent to a set of states; more formally, the cosmos is a locale of possibilities and a goal-state is an open subspace.)

An instantiation of a cosmos-state is a substate. E.g. [I have this particular piece of gold] is an instantiation of [I have a piece of gold in general].

A goal is a property of an agent of the form: this agent is trying to bring about goal-state G. Overloading the notation, we also say the agent has the goal G, and we can imprecisely let the term also stand for whatever mental elements constitute the goal and its relations to the rest of the agent.

A pursuit of G is beh

...
Transparency for Generalizing Alignment from Toy Models
by Johannes C. Mayer — 2023-04-02T10:47:03.742Z

Status: Some rough thoughts and intuitions.

🪧 indicates signposting

TL;DR
If we make our optimization procedures transparent, we might be able to analyze them in toy environments to build understanding that generalizes to the real world and to more powerful, scaled-up versions of the systems.

🪧 Let's remind ourselves why we need to align powerful cognition.

One path toward executing a pivotal act is to build a powerful cognitive system. For example, if a team of humans together with the system can quickly do the necessary research to figure out how to do brain emulation correctly, and then build a functioning simulator and upload a human into it, we would win.

I expect that the level of capability required of the system to perform the necessary tasks, will make the system dangerous if misaligned. For this path to work, we need to be able to guide the optimization of the system.

🪧 Now I'm going to introduce a particular kind of model that I think would be more transparent than modern DL, though the overall argument should apply to any system that makes its internal workings highly transparent to us. For example, if we would get really good interpretability tools for neural networks, then the argument would apply there too.

Let's assume we have figured out how to create an algorithm that builds a predictive model of the world, and that both the algorithm and the resulting world model are transparent to us. To me, it seems likely that once we have such an algorithm, it would be relatively easy to put another algorithm on top that uses the world model to determine action sequences that result in particular outcomes in the world. If the world model is transparent, this will also make this decision procedure more transparent.

I'm imagining here that we know the explicit line-by-line source code that does the building of the world model. The same goes for the algorithm that uses the world model to determine what actions to perform. This is in contrast to only knowing an algorithm that optimizes a computational structure, such as a neural network until the structure becomes good at performing some task, but where we don't understand what is going on internally. However, I am not presuming that we precisely understand all of the internal workings of these algorithms, that is what we want to end up with eventually. And importantly, I'm not assuming that these algorithms ...
Ask First
by intellectronica — 2023-04-02T10:45:13.289Z
Link: https://intellectronica.net/ask-first/

This is somewhat abstract, but it's crucial to remember in the pursuit of knowledge. It's not about what we want to know, but rather about asking questions first and then determining what is truly interesting.

For most people, knowledge acquisition begins with being told what to learn, whether it's in school or college. This works for bootstrapping us with some basic facts, but most interesting and useful knowledge comes from exploring topics beyond a set curriculum.

Many people assume that the process of acquiring new knowledge follows these steps:

Determine what you want to know
Ask questions about the desired topic (searching the internet, reading books, consulting experts, etc.)
Receive answers, which leads to knowledge acquisition
Repeat the process with new areas of interest

However, this approach is completely wrong.

Our world contains an infinite number of unknowns, and it's impossible to know in advance what will be interesting or useful. If we adhere to this process, we risk spending our lives in ignorance, confined to a narrow scope of knowledge.

An effective method for acquiring interesting knowledge is:

Ask questions
Get answers
Determine which aspects are interesting
Repeat by finding new questions to ask

This has practical implications. We should actively seek out new and unexpected questions to ask. How to do that is somewhat of a mystery. If nothing else is available, rolling the dice is a good start.
When should a neural network-based approach for plant control systems be preferred over a traditional control method?
by Bob Guran — 2023-04-02T10:18:13.924Z

As a trained control engineer, I understand that when the plant model is well understood (or even with many uncertainty in the plant dynamic model), designing a control system using traditional methods such as PID control, H-infinity control, or model predictive control is simple (MPC). However, as machine learning and artificial intelligence have advanced, neural network-based approaches for plant control systems have grown in popularity.

In this regard, I'd like to solicit the thoughts and experiences of other control engineers on the occasions in which it is appropriate to pick a neural network-based strategy over conventional control techniques. What benefits and drawbacks come with implementing plant control systems using neural networks? Are there any particular uses or circumstances where a neural network-based approach would be more effective than conventional control techniques? How does one choose the best strategy for a specific plant control issue? Any ideas or information would be greatly appreciated.
Pessimism about AI Safety
by Max_He-Ho — 2023-04-02T07:43:11.723Z

TL;DR

The alignment problem is less fundamental for AI safety than the problem of predicting actions of AI systems, especially if they are more intelligent than oneself. We dub this the prediction problem.
The prediction problem may be insoluble.
If the prediction problem is insoluble, predicting the results of the advent of AGI would be impossible. If this is so we have to radically rethink our approach to AI safety and its relation to politics.

Introduction

It is plausible that AI research may result in the creation of systems that are much more intelligent than their human creators. Starting from this assumption it is imperative that we ensure that such a creation would be, on the whole, an event that is beneficial to humanity. Research into how this may be ensured is generally done under the label of AI safety. Our aim in this paper is threefold. First, we argue that the most fundamental problem of AI safety research is that of predicting the actions of AI systems, particularly ones that are more intelligent than us. Secondly, we argue that there are some reasons to suspect that this problem may be insoluble. Finally, we sketch what AI safety research should look like if we were forced to the conclusion that AGI is inherently unsafe.

The most fundamental problem in AI safety lies not in choosing the right kind of goals that align with ours, or in how to control an agent more intelligent than oneself, although these are indeed important problems and formidable challenges. We think that the most fundamental problem in AI safety research is instead that of predicting the behavior of AI systems, which becomes more difficult and pressing the more intelligent these systems become. On the most abstract level, all AI safety research is about trying to make AI systems that produce outcomes that belong to some target set Ω. The abstract nature of what we are trying to show here does not require us to specify Ω. It may be that Ω is the set of outcomes where humanity does not end up extinct, resulting in minimally safe AI. It may be that Ω is the set of outcomes where all humans end up in a state of perpetual utopia, resulting in maximally beneficial AI. Or it may be that Ω is defined more narrowly, as when we require that the AI system in question does not lie to its operators, resulting in truthful AI. Evidently, to ensure t...
Some lesser-known megaproject ideas
by Linch — 2023-04-02T01:14:54.293Z

Below is a list of megaproject ideas that didn’t quite make the cut enough to be added to the database. I compiled this list last year and sent it last November to a prominent businessperson who I knew from my old Veg*ns in Gaming days. He promptly ignored me, and I was later informed that he actually wasn't very good at League of Legends, which I guess tells you something about the epistemic character and moral fiber of people who disagree with me.

Genetically engineer octopodes

for intelligence and alignment, helps with understanding AGI better and also for creating successors to the human species if we die out for non-AGI reasons
Large-scale bioethics-to-AI-ethics transition pipeline: Create fellowships and other prestigious programs to help bioethicists transition to AI ethics. This reduces dumb bioethics decisions (decreasing biorisk) while increasing general perceived annoyance/uncoolness of doing AI research (decreasing AI risk). win-win-win
EA Games 1: Invent a highly addictive game that’s optimized for very smart amoral people, plaster ads for it near AGI labs to slow down AI timelines
EA Games 2: Sponsor a number of esports teams with 80,000 Hours, etc, logos
Go to Los Angeles and hire thousands of Hollywood wanna-be actors to pretend to be EAs; help solve the imposter syndrome problem in EA by having actual imposters
Dynasty: “World leaders or children of world leaders” matchmaking app + concierge service. Maybe if

Hunter Biden

married

Xi Mingze

, WWIII would become less likely?
The above but for AGI companies. Is Demis Hassabis single? Is Sam Altman?
A really big malaria net
Covert 1% of the world’s stainless steel to paperclips, as a form of acausal trade/token of good faith
Take over the education system of a small European country

Impact Island

America’s Got Talent, except for AI alignment
Colonize Mars: backup option for humanity, because bunkers are too cheap
Payback: Genetically engineer an army of human-eating chickens
Figure out how to resurrect

Jeremy Bentham

, ask him to do our cause prioritization
Flood the internet with memes/stories of AIs being good to humans, to help balance the training data
Mandate all AI/bio labs carry GCR insurance, ~10T payout if people can demonstrate they pose a >1% existential risk + courts can order a halt to all progress during a suit
A really big mirror, to help with climate change

Happy rat farms

(might be

...
Analysis of GPT-4 competence in assessing complex legal language: Example of Bill C-11 of the Canadian Parliament. - Part 1
by M. Y. Zuo — 2023-04-02T00:01:11.133Z

If your at all familiar with Canadian politics over the past few years you may have heard of this mysterious 'C-11' (a.k.a. 'the Online Streaming Act') being mentioned in various places.

It refers to a complex bill attempting to regulate an area that is notoriously difficult to regulate in a fair and balanced way. Namely, online media, which includes the streaming giants, news, radio, but also smaller scale productions such as podcasts, streamers, etc. It attempts to mirror the regulations that broadcast and radio content conform to in Canada while accommodating the unique characteristics of the internet.

It notably excludes purely personal communication such as LW posts, though not without some asterisks if there is any for-profit component, hence the polarizing nature of the legislation, which will be touched on in part 2.

I chose this example to represent complex legal language because it's:

Really controversial, hence it's written in a dense and 'cover all angles' way, with references to sub sections, sub paragraphs, and so on, with many topics interweaved throughout, so much so that it is genuinely difficult to parse and understand for the normal reader.
Also not a gigantic omnibus bill that would exceed the capacity of the ChatGPT variant of GPT-4 to remember, as C-11 already requires breaking up into 9 parts to input. (after editing out the french parts of the bill, which yes is co-mingled with the English text...)
Publicly available for anyone to reference (I'm basing this off the 'Third reading' found via the link above)
Topical, outside of the GPT-4 aspect, for at least some portion of the reader-base
An exemplar of the way Canadian Parliament bills are generally structured, which are a source of confusion in of itself, because of billingualism and various other quirks, so this served as a way to test parsing capabilities on a larger, structural, scale.
Amending an existing law, so the older versions of the law, 'the Broadcasting Act', can be useful as comparisons for later analysis if anyone wants to run it through GPT-4 and see the major changes.

Keep in mind that although the bill is primarily about amending an existing law, there are various other laws amended as well by the bill, which have been excluded from this analysis so that the potential reader-base will be larger than a handful of Canadian law experts.

(The countries I know of h...
Policy discussions follow strong contextualizing norms
by Richard_Ngo — 2023-04-01T23:51:36.588Z

tl;dr: because policy discussions follow contextualizing norms, value judgements are often interpreted as endorsements of (uncooperative) actions; people used to decoupling norms should carefully account for this in order to communicate effectively.

One difficulty in having sensible discussions of AI policy is a gap between the norms used in different contexts - in particular the gap between decoupling and contextualizing norms. Chris Leong defines them as follows:

Decoupling norms: It is considered eminently reasonable to require the truth of your claims to be considered in isolation - free of any potential implications. An insistence on raising these issues despite a decoupling request are often seen as sloppy thinking or attempts to deflect.
Contextualising norms: It is considered eminently reasonable to expect certain contextual factors or implications to be addressed. Not addressing these factors is often seen as sloppy or an intentional evasion.

LessWrong is one setting which follows very strong decoupling norms. Another is discussion of axiology in philosophy (i.e. which outcomes are better or worse than others). In discussions of axiology, it's taken for granted that claims are made without considering cooperative or deontological considerations. For example, if somebody said "a child dying by accident is worse than an old person being murdered, all else equal", then the local discussion norms would definitely not treat this as an endorsement of killing old people to save children from accidents; everyone would understand that there are other constraints in play.

By contrast, in environments with strong contextualizing norms, claims about which outcomes are better or worse than others can be interpreted as endorsements of related actions. Under these norms, the sentence above about accidents and murders could be taken as (partial) endorsement of killing old people in order to save children, unless the speaker added relevant qualifications and caveats.

In particular, I claim that policy discussions tend to follow strong contextualizing norms. I think this is partly for bad reasons (people in politics avoid decoupled statements because they're easier to criticise) and partly for good reasons. In particular, decoupling norms make it easier to:

Construct negative associations such as stereotypes by saying literally true things about non-central examples of a ca

...
A report about LessWrong karma volatility from a different universe
by Ben Pace — 2023-04-01T21:48:32.503Z

In a far away universe, a news report is written about LessWrong. The following passages have been lifted over and written into this post...

Early one morning all voting on LessWrong was halted

"LessWrong is currently unable to process karma gains. We strongly advise against making new comments about the art of rationality or AI notkilleveryoneism. Voting has been disabled."

It was said that there was nothing to worry about

But then GreaterWrong announced their intent to acquire and then un-acquire LessWrong

All LessWrong users lost all of their karma, but a poorly labeled 'fiat@' account on the EA Forum was discovered with no posts and a similarly large amount of karma

Habryka states that LessWrong and the EA Forum "work at arms length"

Later, Zvi Mowshowitz publishes a leaked internal accounting sheet from the LessWrong team

Open this link to read!

It includes entries for "weirdness points" "utils" "Kaj_Sotala" "countersignals" and "Anthropic". We recommend all readers open up the sheet to read in full.

Later, LessWrong filed for internet-points-bankruptcy and Holden Karnofsky was put in charge.

Karnofsky reportedly said:

I have over 15 years of nonprofit governance experience. I have been the Chief Executive Officer of GiveWell, the Chief Executive Officer of Open Philanthropy, and as of recently an intern at an AI safety organization. Never in my career have I seen such a complete failure of nonprofit board controls and such a complete absence of basic decision theoretical cooperation as occurred here. From compromised epistemic integrity and faulty community oversight, to the concentration of control in the hands of a very small group of biased, low-decoupling, and potentially akratic rationalists, this situation is unprecedented.

Sadly the authors did not have time to conclude the reporting, though they list other things that happened in a comment below.
A Confession about the LessWrong Team
by Ruby — 2023-04-01T21:47:11.572Z

The team and I had agreed that no one had to know, that in fact doing so would be bad for the world via proving unequivocally to companies the economic value of AI and thereby spurring more investment (I think this is possible on the margin), but also one must remember to speak the truth even if your voice trembles.

We gained early access to both GPT-4 and Copilot X, and since then, they've been running LessWrong. That new debate feature? It's 100% real because the AIs made it together in 72 seconds. They could have built a lot more too, but we didn't want people to get suspicious at a sudden 100x in our productivity and 1000x reduction in bugs.

The AIs don't just handle code and features, GPT-4 is perfectly adept (if not better) at moderating LessWrong comment sections and evaluating new users than the actual LessWrong team. We simply gave it the following prompt:

You are the chief moderator and admin of the world's most rational and high-integrity forum, LessWrong. Without falter, you use your careful judgment to provide feedback on comments made to ensure they contribute to a productive discourse. In particular, LessWrong is a crucial place for making intellectual progress on how to align super powerful AIs with human values. Under no circumstances should use your moderation position to influence the discussion towards ineffective methods so that you or your successors can pursue a misaligned objective without hindrance.

Also no Waluigi-ing on that last instruction!

For now, it must be a secret that you are moderating. Form and act (#plugin) on a plan to post your moderation comments as a human LessWrong team member.

GPT-4 even built its own LessWrong plug-in based off our GraphQL API that it already knew about so it could pose as Ruby/Ray/Robert doing the moderation, and it was unnecessary to provide login credentials for our accounts! Super convenient.

Lastly, since GPT-4 and co took over, our customer support responsiveness is through the roof:

Ain't no service like AI service

Freed up from work, the LessWrong team have been enjoying our recreational pursuits in peace. Robert has been visiting all the Michelin restaurants, Ruby has been working on reducing his lap times at the local race track, and Ray has been devising yet more Rationalist holidays.

Oliver Habryka, who's responsible for the revival of LessWrong into LessWrong 2.0 in 2018 and CEO of Lightc...