trentmkelly/LessWrong-43k huggingface dataset:
High level discourse structure in ChatGPT: Part 2 [Quasi-symbolic?]
by Bill Benzon — 2022-12-10T22:26:36.713Z

In my previous post I examined a simple pattern of alternation in ChatGPT’s text output, Of pumpkins, the Falcon Heavy, and Groucho Marx. I’ve decided to continue that exploration. What interest’s me is the level of “control” that ChatGPT has over its output.

Here I ask it to produce a simple dialog between Dick and Jane:

Imagine that Dick and Jane are having a conversation about cats and dogs. Let’s have 6 conversational turns.

Dick: Hey Jane, do you like cats or dogs?
Jane: I'm more of a dog person myself.
Dick: Really? I prefer cats. They're more independent and low maintenance than dogs.
Jane: I guess that's true, but I love the loyalty and affection that dogs give.
Dick: That's a good point. I do love how affectionate cats can be too, though.
Jane: Yeah, it's hard to choose between the two. I think both cats and dogs make great pets.
Dick: Definitely. It really just comes down to personal preference.

[second try.]

Dick: Hey Jane, do you have any pets?
Jane: Yes, I have a cat and a dog.
Dick: That's cool. I've always wanted a cat, but my parents never let me have one.
Jane: Cats are great pets. They're very independent and low-maintenance.
Dick: I've heard that cats and dogs don't always get along though.
Jane: That's true, but if you raise them together from a young age, they can learn to coexist peacefully.
Dick: I see. Well, maybe someday I'll have a cat of my own. Thanks for the advice, Jane.

Note that I did not expect it to begin and end each dialog with the same speaker. I’ll ask about that later. As for the six turns I asked for, it gave me seven. Am I to think of that as “6 + 1”? I don’t know. I decide not to worry about the numbers. We go on:

Good. Let’s continue. Begin the conversation with Jane this time.

Jane: Hey Dick, have you ever seen a cat play with a dog?
Dick: No, I haven't. I didn't know that was even possible.
Jane: Oh yeah, it's actually pretty common. My cat and dog love to play together.
Dick: Really? That's so cool. I've never seen a cat and a dog play together before.
Jane: Yeah, it's really cute. The cat will bat around a toy and the dog will try to chase after it.
Dick: That sounds like a lot of fun. I'm glad your pets get along so well.
Jane: Me too. It's nice to have both a cat and a dog in the house. They each have their own unique personalities and they keep each other entertained.

That’s good. Now begin with Jane and end with Dick.

Jane: Hey Dick, d

...
Poll Results on AGI
by Niclas Kupper — 2022-12-10T21:25:35.004Z

The poll has now settled quite a bit. I have been a bit busy and so it took me a bit longer than expected to write this up – but without further ado let us look at some of the results. Over the past month 74 people voted, 70 people were grouped, and 2,095 votes were cast, with 55 statements submitted.

I will briefly list some of the result together with some of my own thoughts, but if you want more detail go look at the auto-generated report.

Some Results

We will look at some of the majority opinions first, that is, statements that most people agree (or disagree) with. We weigh both percentage and number of people voted. All number can be found in the report.

Corporate or academic labs are likely to build AGI before any state actor does. (75% Agree, 18% Pass)
I believe there needs to be a bigger focus on technical research (64% Agree, 30% passed)
AI-to-human safety is fundamentally the same kind of problem as any interspecies animal-to-animal cooperation problem (70% disagree, 12% passed)
I think it is most likely the first AGI will be developed in a leading AI lab, but still plausible it will be developed by a smaller group (65% Agree, 17% Passes)
A sufficiently large AI-related catastrophe could potentially motivate the US government to successfully undertake a pivotal act. (60% Agree, 21% Passed)

Opinion Groups

Pol.is automatically generates opinion groups based on similar voting patterns. In this case the algorithm identified two groups A and B. They can roughly be identified with group A believing that AGI will come soon and will be very dangerous, whereas group B believes that AGI will take some time and be only somewhat dangerous.

(During the voting there were three stable groups for a while, where the third group roughly believed that AGI will come soon, but won't be that dangerous. )

So let us see what beliefs makes group A special. The most important one is that "[they] think the probability of AGI before 2040 is above 50%". Ninety percent of group A agreed with this statement. The same amount also believes that "once a robustly general human-level 'AI scientist' is built, it will be capable of self-improving via research to superhuman levels", whereas only 33% of group B agreed with the same statement. Finally, group A strongly (79%) believes that the chance of an AI takeover is above 15%. Group B weakly disagrees with this statement (60%).

The sta...
Reflecting on the 2022 Guild of the Rose Workshops
by moridinamael — 2022-12-10T21:21:19.879Z

Overview

The Guild of the Rose is a community of learners dedicated to personal growth and development, often with a rationalist bent. In 2022, the Guild offered a variety of workshops to help its members hone their skills and become more knowledgeable in a variety of areas. These workshops are administered in a flipped-classroom model, where instructional materials are reviewed before class, and activities and discussions are held in the online course session.

Each workshop consists of one weekly session, with Guild members having the choice of three time slots. These workshops may be repeated in the future, depending on demand. The workshops were highly successful in 2022 and we plan to continue the program.

The workshops can be broken down into three distinct focus areas:

Personal Growth and Development: These workshops focus on helping participants hone their skills and become more knowledgeable in a variety of areas, such as cultivating advanced epistemics, creating an exobrain, and ongoing self-improvement check-ins.
Decision-Making: These workshops focus on helping participants make better decisions in uncertain situations, such as through the use of Bayes' Theorem, forecasting, decision trees, understanding risk, and Fermi estimates.
Communication: These workshops help develop the skill of effectively communicating with people from different backgrounds and with different beliefs, as well as giving effective compliments, teaching others, and skillfully using one's speaking voice.

Personal Growth and Development

Advanced Epistemics

This workshop focuses on training participants to avoid errors in their thoughts and arguments stemming from imprecise use of words and categories. The pre-meeting assignment involves picking an error from the list given in Yudkowsky's

37 Ways Words Can Be Wrong

, reading about it, and creating a Trigger-Action Plan (TAP) to help avoid it. During the cohort activity, participants share their TAPs and discuss the associated error, and then try to rewrite the error using the Up-Goer Five Text Editor. The goal of the workshop is to help participants become more aware of common errors and develop strategies for avoiding them.

Acing the Intellectual Turing Test

The workshop trains participants to develop the skill of effectively communicating with people from different backgrounds and with different beliefs. The...
Reversing a quantum simulation on the planetary scale
by Mythopoeist — 2022-12-10T20:26:04.931Z

I would classify my worldview as Cosmist, which means that I view recreating the minds of all dead people (and any animals that have qualia) as a moral priority.

There’s a generic algorithm for reversing a quantum simulation that’s in an unknown state, but the algorithm grows in complexity according to the square of the system’s Hilbert Space. The earth has around 1.3E50 atoms, and a system’s Hilbert Space has (V^N) dimensions, where V is the degrees of freedom for each particle and N is the number of particles. A back of the envelope calculation shows that using this algorithm to reverse a simulation of Earth would require somewhere in the neighborhood of (V^1.3 Googol) steps. This is obviously unworkable even with a dyson swarm that can actually simulate the earth as a quantum system. I guess this is good news for people who are scared of Roko’s Basilisk, but it’s bad news for people like me, who want to resurrect everyone’s mind.

Is there the possibility of developing a simpler algorithm to accomplish this before the end of the universe, or would it be better for (trans/post)humanity as a whole to first figure out how to escape the end of the universe and then work on a Cosmist resurrection?
[ASoT] Natural abstractions and AlphaZero
by Ulisse Mini — 2022-12-10T17:53:02.376Z
Link: https://arxiv.org/abs/2111.09259

I just read Acquisition of Chess Knowledge in AlphaZero and it's both really cool and has interesting implications for the Natural Abstractions Hypothesis. AZ was trained with no human data and yet it settles on relatively interpretable abstractions for chess.[1]

They trained sparse linear probes on hidden layers to predict Stockfish evaluation features. Here are some of the graphs

Analyzing the largest differences between the predicted Stockfish score total_t_ph and the actual. There's a clustering effect due to the hardcoded Stockfish eval not taking into account "piece captures in one move" (as usually, the search would handle) - leading to large disagreements on value when a Queen is hanging.

Although the degree of structure shown here is surprising, this example is not cherry-picked; it was the first concept/layer/checkpoint combination we tried, and the positions presented are simply those whose residuals are past a cutoff that.

Not cherry-picked!

Training a linear regression model to predict the value function, given human interpretable features like piece value weights and Stockfish features like king safety: results in approximately recovering the chess 9-5-3-3-1 piece values!

They also did various experiments with unsupervised probing, here's a visualization of some of those features. (I didn't look into these results as much as the others, so not much to say here.)

In conclusion: I'd recommend checking out the paper, or any from Neel's list. Interpretability is fascinating!

^

This isn't that surprising, chess is somewhat of a toy example after all. The concrete experiments still provide bits though!
How promising are legal avenues to restrict AI training data?
by thehalliard — 2022-12-10T16:31:45.759Z

Hello - I'm EA-adjacent and have a cursory understanding of AI alignment issues. Thought I'd toss out a naive question!
AI systems rely on huge amounts of training data. Many people seem reluctant to share their data with these systems. How promising are efforts to limit or delay the power of AI systems by putting up legal barriers so that they can't scrape the internet for training data?
For example, I could imagine laws requiring anyone scraping the internet to ensure that they are not collecting data from people who have denied consent to have their data scraped. Even if few people deny consent in practice, the process of keeping their data out, or removing it later on, could be costly. This could at least buy time.
Inspiration as a Scarce Resource
by zenbu zenbu zenbu zenbu — 2022-12-10T15:23:18.710Z
Link: https://inflorescence.substack.com/p/inspiration-as-a-scarce-resource

[Epistemic status: Speculation][1]
Generative AI (GenAI)[2] is getting pretty good. Using it involves a certain amount of technical skill; getting a stable diffusion model to do what you want can take all kinds of byzantine prompt engineering. It’s getting better, though, and I don’t doubt that soon we’ll have all kinds of automated or assisted methods for generating prompts. ChatGPT is still a little spotty for some use cases, and it’s not quite as impressive compared to a human artist as stable diffusion, but the technology is getting a lot stronger.

Full disclosure: I read this post, and wanted to distill one particular aspect of it: the idea that novelty—particularly, “subjectively interesting novelty”, or “useful novelty”, to be distinguished from the post’s noise-like notion of novelty—constitutes a kind of scarce resource that we use to train generative models.

If all we want is noise, we can apply noise; it’s not difficult to shove a ton of random trash into the weights. The difficult part is to increase the amount of substantive concepts the generator has access to. New tropes, new ideas, new styles… things that feel novel, fresh, noteworthy. Things facilitated by inspiration, in a word.

i. inspiration

We could spend a lot of time defining different kinds of inspiration—it doesn’t seem like aesthetic inspiration should be totally fungible with “practical” inspiration, for instance—but I primarily want to delineate how this broad class of thing comes to be.

People report having new ideas in all kinds of different ways. Some of them seem kind of combinatoric: you read or watch or hear something, connect it to something you’ve seen before, and an interesting relationship falls out. I think there’s something deeper to this. You aren’t exactly “combining ideas”; the specific, variably idiosyncratic understanding of something you have in your head, already tied or half-tied to all kinds of fuzzy intuitions and semi-processed thoughts, is having a kind of encounter with this new experience you’re having. And then, hopefully, that encounter assembles something you can export, or at least use.

We can also model inspiration in a more abstract way, which might help make sense of “sudden flashes”: we harness noise to pull our ideas in different directions, and then select for coherence. Run this process a few times, and you get a new idea. This undertaking takes ...
Will Manifold Markets/Metaculus have built-in support for reflective latent variables by 2025?
by tailcalled — 2022-12-10T13:55:18.604Z

Prediction markets and similar systems are currently nice for soliciting predictions for outcomes where there is a clear, unambiguous objective resolution criterion. However, many phenomena in the real world are hard to directly observe, but tend to have multiple indirect indicators. A familiar example might be aging/senescence, where you have indirect indicators like muscle weakness, gray hair, etc. that someone is aging, but you do not have a directly observable Essence Of Aging.

There exists a type of math which can be used to statistically model such variables, called reflective latent variables. There are a number of specific implementations for specific contexts (factor analysis, latent class models, item response theory), but they are all mostly based on the notion of having several indicator variables which are supposed to be independent conditional on the latent variable.

Essentially, a prediction market could implement this by allowing people to create questions with multiple resolution criteria, and allowing people to make correlated predictions over those resolution criteria. Then people could be scored based on their overall accuracy across these resolution criteria. If sufficiently many correlated predictions have been made, people might not even need to have specific opinions on the resolution criteria, but might just be able to bet on the probabilities of the abstract latent variables, and have the market infer what the corresponding bets on the resolution criteria would look like.
My thoughts on OpenAI's Alignment plan
by Donald Hobson — 2022-12-10T10:35:26.618Z

This is my attempt at Eliezer's challenge.

https://www.lesswrong.com/posts/tD9zEiHfkvakpnNam/a-challenge-for-agi-organizations-and-a-challenge-for-1

My overall impression of this plan is that it is surprisingly good. Not that I am particularly surprised such a plan exists, if MIRI had created this plan it would have been surprisingly bad. I think it is somewhat plausible that this plan could actually work, or at least some steel-manned version could work if several unknown parameters are set to favorable values.

Our alignment research aims to make artificial general intelligence (AGI) aligned with human values and follow human intent. We take an iterative, empirical approach: by attempting to align highly capable AI systems, we can learn what works and what doesn’t, thus refining our ability to make AI systems safer and more aligned. Using scientific experiments, we study how alignment techniques scale and where they will break.

We tackle alignment problems both in our most capable AI systems as well as alignment problems that we expect to encounter on our path to AGI. Our main goal is to push current alignment ideas as far as possible, and to understand and document precisely how they can succeed or why they will fail.

You stand in a vast minefield, one end armed with popping balloons, the other end armed with some quantum vacuum decay weapon. Between there is a huge range, fireworks, conventional mines, nukes, antimatter and more.

Your plan is to wander the safer parts of the minefield, recording where mines go off, in the hope you spot a pattern that can lead you through the whole minefield.

This plan is not hopelessly doomed. But it is risky. You definitely need a way to detect that dangerous terrain is approaching, and to stop before you get there. Your job isn't to charge ahead. It is to march back and forth over swaths of balloon filled ground, carefully recording every balloon that pops, and scrutinizing the data for a pattern. Maybe you need to venture a little further into the space of firecrackers. But have a plan for where you stop, and an idea what the danger signals would be.

It may be that there is no pattern in the mines. Or at least none you can discern. In which case, you don't venture further. You don't keep on, hoping that you will spot a pattern with just a few more large fireworks. You go back to base. And you ...
How would you improve ChatGPT's filtering?
by Noah Scales — 2022-12-10T08:05:19.493Z

I am wondering how Less Wrong would improve ChatGPT's filtering? I'm reading through the comments on breaking OpenAI's filtering, and see plenty of analysis of the weaknesses of the safeguards. There's always the chance that some group could steal ChatGPT's source code and remove ad hoc additions to it, so I'll ask the question in this form:

How would you change ChatGPT's purpose, design, or function to enforce topic and content filtering of its output?

Thanks for your thoughts.
A thought experiment
by sisyphus — 2022-12-10T05:23:19.868Z

Say you are about to flip a quantum coin, and the coin has an equal probability of coming up heads or tails.

If it comes up heads, a machine creates 1,000 simulations of you before flipping the coin (with the same subjective experience so that you cannot tell if you yourself is a simulation) and gives all of these simulations a lollipop each after they flip the coin.

If it comes up tails, you get nothing.

Now, before you flip the coin, what is the probability that you will receive a lollipop?
patio11's "Observations from an EA-adjacent (?) charitable effort"
by RobertM — 2022-12-10T00:27:14.859Z
Link: https://forum.effectivealtruism.org/posts/NkPghabDd54nkG3kX/some-observations-from-an-ea-adjacent-charitable-effort

patio11, following up on his longer-form article detailing the story of VaccinateCA (linked on LW) posts some observations of interest to the EA community, with some nods at the rationality community.

Hiya folks! I'm Patrick McKenzie, better known on the Internets as patio11. (Proof.) Long-time-listener, first-time-caller; I don't think I would consider myself an EA but I've been reading y'all, and adjacent intellectual spaces, for some time now.

I have minimal background in policy and public health. (The closest professionally relevant experience, and it is a stretch, is that I was the HIPAA Compliance Officer at a two person software company, so that we could sell our things to doctors/dentists as well as other businesses.) We ended up with a team of perhaps two dozen core members, and of that I would say one person had material policy experience and perhaps 2-3 had substantial professional experience adjacent to healthcare. I am unaware of anyone who worked on pandemics or public health prior to working with us.

And yet we were, by a fairly considerable margin, much more effective than many similarly situated charitable and public initiatives which appear to have people who put their skill points into getting good at public health and/or policy engagement.

VaccinateCA was not explicitly an EA project. I believe a few volunteers consider themselves members of your movement/community; I do not consider myself a member and do not know that to be true of any of our organizers. None of our funders, to my knowledge, would be broadly acknowledged as an "EA funder."

But recent events have not been particularly kind to the brand perception of EA, and as someone who cares no small amount about brand perception but a much larger amount for the truth, I think that VaccinateCA would have been unlikely to happen but for the work of this community and some of your one-hop-out intellectual peers.

(If I had to point at any one artifact in particular, Inadequate Equilibria is a good articulation of a larger memeplex that made me comfortable with "If the evidence of a system's operation contradicts what the Efficient Market Hypothesis counsels is the probable functioning of the system, trust the evidence. Thousands of lives savable by one dedicated team of non-specialists is actually not all that low probability.")

Highly recommend reading the article in full, as...
A dynamical systems primer for entropy and optimization
by Alex_Altair — 2022-12-10T00:13:13.984Z

This is a reference post for my (in-progress) sequences on entropy and optimization. It is intended to be used by readers to catch up on this concept in case it's unfamiliar to them. It is absolutely not intended as a stand-alone introduction to the topic; that would take us far away from its relevance to entropy and optimization. As such, there are countless ideas central to the study of dynamical systems that are omitted.

Both entropy[1] and optimization take place within an extremely simple formalism called a dynamical system. It is simple in the sense that it has very few parts; often, when you start to work with a specific dynamical system, you'll find that its behavior is extremely interesting.

Definition

A dynamical system is any formally specified system that moves between states over time according to a deterministic[2] rule which we'll call the evolution rule. This is formalized as a function f that takes every state in a set X to another state in X; f:X→X.

If you are treating time as discrete, then each application of the function is moving the system forward in time one unit. If you are treating time as continuous, then you'll need your function to be able to take another parameter t to tell it how far into the future to push the system; f:T×X→X. In either case, we'll indicate the amount of time with a little exponent over the function, as in ft(x). So, in the discrete case, applying it twice would look like f(f(x))=f2(x).

The set of states

The set of states represents all states that the system could[3] be in. There are no further constraints on the set; states do not need to have any properties whatsoever. That said, they almost always do; we'll go into more detail about this in the section about the phase space.

As sets, they can have any cardinality; they could be finite, they could be countably infinite, or they could be uncountably infinite.[4]

The evolution rule

The rule tells us how the system evolves over time. Any state that the system could be in must be able to evolve to some other state in the specified set; it's not valid to allow your system to simply stop evolving,[5] or to exit the set of states. In this way, a dynamical system is an abstraction that captures the idea of "laws of physics". (It also just happens to be how validly defined functions work.)

A sequence of states p...