trentmkelly/LessWrong-43k huggingface dataset:
Attempting to Deconstruct "Real"
by herschel — 2023-07-09T16:40:53.140Z

I see some current contentions which have a relatively invisible crux with respect to the concept of real. I'm increasingly doubtful that the concept means very much. I think ‘real’ or ‘reality’ is probably a mostly needless reification over several different and not fully compatible distinctions. Note this is largely a cold take in a good part of my circles, but it’s not yet common knowledge, and so seems worth laying out explicitly.

It doesn’t seem like a primate nervous system comes out of the box with a notion of “reality”. Experiences are meaningful with respect to our values, capacities, and channels for meaning, regardless of their truth values or the accuracy of our world-models.

Certainly, there are lies, and there are dreams, and there are hopes that do not come true. On the other hand, there is mathematics, and there are models of deep time (let alone QM or Tegmark levels), which we cannot touch but still believe in.

Physicalism has no account of the truth of logical statements, nor does logical consistency of the predictive power of physical models. Nor, importantly, do either of these on the emotional and psychological reality of violence, music, winning, or love. These are just some important kinds of realities.

There have been various attempts to construct reifications of the various realities which we encounter, in terms of one another, or grounded in some metaphysical ontic, which is never to be found. I have yet to see an account of ontology which can do anything but wave its hands or stomp its feet when it comes time to narrow down what “reality” actually is. I see this as making the same mistake as “is it really a blegg?”

This is the basic mechanism of reification. I claim that reification is simply the process of the construction of salience or weight, phenomenologically.

Furthermore, I have yet to see an account which captures all of the realities under one roof. “That which cannot be denied” might almost suffice, but some of these views lead people to deny experience whatsoever.

This doesn’t mean you should start believing in magic. (Though, having messed around with this stuff, some kinds of magic seem to be much more real than I had originally thought.)

I’ll be direct about what I’m contesting here: it’s not clear how we should regard the ontological status of prayer, meditative exper...
Quick proposal: Decision market regrantor using manifund (please improve)
by Nathan Young — 2023-07-09T12:49:01.904Z

Epistemic status: Some fundamental errors but worth leaving up

I think the error with this piece is that it is a hammer looking for a nail, rather than the reverse. Austin makes the point well here. Rather than saying "how to institute Futarchy," I guess it should say "how do we solve problems people already have"

Summary

I would like there to be a regranter using prediction markets and I want your thoughts on how that process should work
Manifund, a new granting org, is connected to the charity prediction market, Manifold, so is a uniquely good candidate
Create prediction markets for each proposed project, scoring it against some metric if it happens or doesn't
Order all projects according to which offer the most additional value if they happen than if they don't
Fund all projects until the Futarchy regrantor uses its allocated budget for that period
A discussion of different metrics
A discussion of how this might be different from current grantmaking orgs
Please correct any errors (even grammatical errors, I think I'd publish on here more if I didn't have to send it round to 10 people to check grammar first)

What is Futarchy and why is this a good opportunity?

Futarchy is a decision-making system where prediction markets are used to make decisions on what actions to take. You agree on some some kind of metric that you care about, and then predict how decisions will affect that metric. Futarchy generally implies a deterministic system, though this proposal is more about ironing the kinks out of the system so it could initially be advisory initially[1].

We can imagine running decision markets on a Manifund regrantor. Manifund is a new regranting org. Individuals get regranting budgets and then publicly allocate them. I guess that if they use their budget well they can argue to be given more.

Manifund

Manifund is a good test bed for Futarchy for three reasons:

They already run an impact market system, so one could try and do some kind of futures markets on the impact certificates. I don't know the legality or technicality of this.
It's associated with a play money prediction market, Manifold - Austin Chen is a cofounder of both Manifold (the prediction market) and Manifund (the regrantor) hence the same part of the name. I think it'd be much easier to implement features than any other comparable one.
The Manifund team ship things very quickly, and are into mechanis

...
Where are the people building AGI in the non-dumb way?
by Johannes C. Mayer — 2023-07-09T11:39:12.692Z

I am somewhat baffled by the fact that I have never ran into somebody who is actively working on developing a paradigm of AGI which is targeted at creating a system that the system is just inherently transparent to the operators.

If you're having a list sorting algorithm like QuickSort, you can just look at the code and then get lots of intuitions about what kinds of properties the code has. An AGI would of course be much, much more complex than QuickSort, but I am pretty sure that there is a program that you can write down that has the same structural property of being interpretable in this way, where the algorithm also happens to define an AGI.

And this seems to be especially the case when you consider that when building the system we can build it in such a way that we have many components and these components have sub-components such that in the end, we have some pretty small set of instructions that does some specific task that is then understandable. And if you understand this component, you can probably understand how this set of instructions behaves in some larger module that uses this set of instructions.

Everything interpretability tries to do, we would just get for free in this kind of paradigm. Moreover, we could design the system in such a way that we have additional good properties. Instead of using SGD in order to find just some set of weights that performs well, that we then interpret, we could just constrain the kinds of algorithms we design in such a way that they are as interpretable as possible, such that we are subjected so strongly to the will of SGD and what algorithms it finds.

Maybe these people exist (if you are one please say hello), but I have talked to probably between 20 and 40 people who would describe themselves as doing AI alignment research and never came something like this up even remotely.

Basically, this is my current research agenda now. I'm not necessarily saying this is definitely the best thing that will save everyone and everybody should do this, but if zero people do this, it seems pretty strange to me. So I'm wondering if there are some standard arguments that I have not come across yet, while this kind of thing is actually really stupid to do.

There are two counter-arguments to this that I'm aware of, that I don't think in themselves justify not working on this.

This seems like a really hard program and might take just

...
What to read on the "informal multi-world model"?
by mishka — 2023-07-09T04:48:56.561Z

People on LessWrong often talk in terms of an "informal many-world model". For example they talk about the worlds where the alignment problem is relatively easy vs the worlds where the alignment problem is really hard.

I wonder what are good references for this "informal multiverse model of reality" with multiverses of worlds with different properties? What's the history of this line of thoughts.

(And I also wonder about possible mathematical models of that, how much people are talking about those, and how much people keeping some of those mathematical models in mind? I've seen some sheaf-based models which seemed to be like that.)
Whether LLMs "understand" anything is mostly a terminological dispute
by RobertM — 2023-07-09T03:31:48.730Z

Epistemic status: speculating on what other people mean when they make certain claims, without having done any checking to validate. Probably (mostly) true anyways.

Thought prompted by Tsvi's latest post: most discussions of whether LLMs "understand" things seem to conflate a few claims under the term "understand":

the weights encode an algorithmic representation of the "thing" in question
the weights encode some kind of much more general reasoning ability, which is sufficiently capable that it can model a large number of not-obviously-related abstractions, including the one in question
#2, but in a way which also includes some kind of map-territory distinction
#2, but in a way which resembles human "system 2" thinking, where it's capable of doing the laborious explicit reasoning people sometimes do when other options aren't good enough

There are probably other things meant by the claim that I haven't considered.

I suspect that some people who say that LLMs don't understand things are thinking of 3 or 4, maybe 2. 1 seems pretty obviously true (we have numerous examples: modular arithmetic, Othello, mazes, etc). I wouldn't bet against 2; evidence in favor includes some speculation[1] that one reason the GPT-series of models are so much better than everything else is the substantial amount of code in their training data[2]. I don't think 3 or 4 have happened yet.

Once one's gotten past the linguistic and philosophical confusions, such arguments could in principle be resolved with empiricism. In practice, most such arguments probably aren't worth the time.

^

I don't remember where exactly I saw this claim. If anybody has a source I'd appreciate it.

^

The causal chain being something like "code is more structured than natural language, and that structure includes a lot of very powerful, generalizable insights, useful across many domains". I'm not sure if this was spelled out in the source or if that was me filling in the blanks.
"View"
by herschel — 2023-07-08T23:19:22.014Z

The term view is used fairly often among meditators in a subtle but distinct way. View refers to the embodied, intuitive perspective that a person holds in a given moment. (Note there can sometimes be multiple views functioning at once, which I mostly don't describe here.)

Colloquially, ‘view’ and ‘perspective’ are used synonymously. I’m trying to distinguish ‘view’ as phenomenological (and sometimes metaphysical), as opposed to a common usage of ‘perspective’ which is generally cognitive and especially verbal. (For now I’ll call the latter sense a perspective, for lack of a better word.)

A verbal expression of a belief is straightforwardly of a different type from the experience which inspires that verbal explanation. Furthermore, ‘view’ encompasses a much greater phenomenological breadth. We often discuss differing perspectives,

Perspectives often end up being expressed as relatively simple endorsed claims about politics or science, eg “the COVID vaccine is safe and everyone should take it”, or “this market is going to be overtaken by AI in the next few years”. Sometimes ‘perspective’ refers to network of priors that a person has, which are hard to reduce to distinct claims. This sense leaves out the full breadth of human apprehension of meaning that a person normally experiences.

Some examples:

A mathematician reflects on the beauty of a proof. To them, the proof is profound and elegant, it pulls on an extensive body of knowledge that they’re familiar with, as well as a long history of great minds who have been working on the problem.

Another person with much of the same knowledge might be able to follow the steps and say “yup that seems right”, but the view in which they experience it is quite different (as well as less intense, pleasant, etc.) from that of the the first person.

A meditator meditates on the Koan “who am I?” and realizes there is no boundary between him and the world. He finds the walls laughing.

The objective circumstances are basically ordinary: a person sitting in a room. Countless people tried without much luck to meditate and have profound experiences like this. What separates the meditator in this case is that they have encountered a highly unusual view, with different (or in this case, absent) boundaries between self and world, which is basic to most ordinary human experience.

A group of people sit at

...
H5N1. Just how bad is the situation?
by Q Home — 2023-07-08T22:09:33.928Z

How bad do you think the H5N1 situation is? I tried to search on LessWrong and EffectiveAltruism, but haven't found that much discussion.

When I was reading about H5N1, a lot of my assumptions were broken. I was shocked how vulnerable we are to the most "predictable" threats.

Should/can rationalists take this threat into account? Can this threat warrant even, I dunno, some community-wide preparation?

Edit: this comment gives specific predictions. Can somebody explain the justification behind such predictions? What exactly keeps the risk relatively low?
A Two-Part System for Practical Self-Care
by Jonathan Moregård — 2023-07-08T21:23:42.987Z
Link: https://honestliving.substack.com/p/a-two-part-system-for-practical-self

Category: Life optimization

Epistemic status: Seems reasonable, not sure how this could possibly backfire.[1]

Sometimes I feel blue. The world turns dull and grey, and I feel like I don’t have the time or energy to accomplish anything. This is occasionally accompanied by social anxiety, where asking for help feels like a big challenge. Figuring out who to reach out to and what to say becomes almost impossible.

When I am feeling blue, it’s very hard to figure out how to recover. I’m way more capable when I feel happy. A year ago, happy me spent an hour or so thinking about what I could do to make life easier for feeling-blue me.

Up until this point, happy-me hadn’t put in any effort to improve life for feeling-blue me. When I'm in a good place, it's easy to only optimize for situations where I feel well, forgetting that I tend to start feeling blue a few times every winter.

The hour spent improving life for feeling-blue me was one of the best investments I’ve done into myself and my life. The system I came up with has 2 parts. The first part is an automatic check-in, helping me realize that I am feeling blue.[2] The second part is a list of things to do in order to start feeling better.

Automatic Check-in

The automatic check-ins are simple. I get a notification twice a week telling me to spend 5 minutes writing about how I feel. I need these automatic check-ins since I tend to slide slowly into feeling blue, in a way that makes it hard to notice.

Event if you don't have a system in place, you can still set up reminders like these. Use an alarm, a recurring reminder, a recurring calendar event or similar.

The check-in notifications also remind me that I have a “feeling blue” list. If I notice stress, an “ugh” reaction[3] or similar, that's a good sign I need to use my feeling blue list.

Feeling Blue List

The second part of my system is my feeling blue list.

On my feeling blue list, I have a list of 2-3 people I can reach out to in order to get support. For each person on the list, I have a pre-written message that I can copy-paste and send to them. Something like “Hey, I’m feeling down and would love some support. Do you have time to hang out or talk a bit on the phone?”. This makes asking for help effortless, even if I’m stuck in social anxiety.

The list also contains a number of self-care activities, each satisfying the following criteria:

They

...
Really Strong Features Found in Residual Stream
by Logan Riggs — 2023-07-08T19:40:14.601Z

[I'm writing this quickly because the results are really strong. I still need to do due diligence & compare to baselines, but it's really exciting!]

Last post I found 600+ features in MLP layer-2 of Pythia-70M, but ablating the direction didn't always make sense nor the logit lens. Now I've found 3k+ in the residual stream, and the ablation effect is intuitive & strong.

The majority of features found were single word or combined words (e.g. not & 't ) which, when ablated, effected bigram statistics, but I also wanted to showcase the more interesting ones:

German Feature?

Uniform examples:

Logit lens:

Stack Exchange

Ablated Text (w/ red showing removing that token drops activation of that feature)

In the Pile dataset, stackexchange text is formatted as "Q:/n/n....", and removing the "Q" greatly decreasing the feature activation is evidence for this indeed detecting stackexchange format.

Title Case

Ablating Direction (in text):

Here, the log-prob of predicting " Of" went down by 4 when I ablate this feature direction, along w/ other Title Case texts.

Logit lens

One for the Kids

Last Name:

Ablated Text:

As expected, removing the first name affects the feature activation

Logit Lens:

More specifically, this looks like the first token of the last name, which makes since to follow w/ Jr or Sr or completing last names w/ "hoff" & "worth". Additionally, the "(\@" makes sense for someone's social media reference (e.g. instagram handle).

[word] and [word]

Ablated Text:

Logit Lens:

alike & respectively make a lot of since in context, but not positive about the rest.

Beginning & End of First Sentence?

On

logit lens:

"on behalf", "oncology", "onyx" (lol), "onshore", "onlook", "onstage", "onblur"(js?), "oneday" ("oneday" is separated as on eday?!?), "onloaded" (js again?), "oncomes"(?)

High Level Picture

I could spend 30 minutes (or hours) developing very rigorous hypotheses on these features to really nail down what distribution of inputs activate it & what effect it really has on the output, but that's not important.

What's important is that we have a decomposition of the data that:
1. Faithfully represents the original data (in this case, low reconstruction loss w/ minimal effect on perplexity)

2. Can simply specify desired traits in the model (e.g. honesty, non-deception)

There is still plenty of work to do to ensure the above two work (including normal hyperparamete...
Eight Strategies for Tackling the Hard Part of the Alignment Problem
by scasper — 2023-07-08T18:55:22.170Z

Thanks to Michael Ripa for feedback.

TL;DR

There are two types of problems that AI systems can have: problems that we can observe during development and problems that we can’t. The hardest part of the alignment problem involves problems we can’t observe.
I outline seven types of solutions for unobservable problems and taxonomize them based on whether they require that we know what failure looks like and how it happens mechanistically.

Solutions that require both knowing what failure looks like and how it happens

Ambitious mechanistic interpretability
Formal verification

Solutions that require knowing what failure looks like

Targeted latent adversarial training
White-box evals

Solutions that require knowing how failure happens

Mechanistic interpretability + heuristic model edits

Solutions that require neither

Untargeted latent adversarial training
Anomaly detection
Scoping models down

I argue that latent adversarial training, mechanistic interpretability + heuristic model edits, and scoping models down all may be highly important, tractable, and neglected and loft out a few ideas for new work.

Two ways that AI systems can fail

Consider a simple way of dividing AI failures into two groups.

Observable failures: Failures that we can encounter in training and development. These include failures on the train set, test set, some types of adversaries, or anything else that we might think to test our AI system on. These can be very bad, but they are problems that we can get feedback on – we can spot them and then work to fix them.
Unobservable failures: Failures that we will not encounter in training and development. These failures are somewhat scarier and harder to address because we will not have feedback to help us solve them. These can include trojans, some adversarial examples, misgeneralization, and deceptive alignment.

Because one cannot get feedback on them, unobservable failures are the harder part of AI alignment, and this is why the AI safety community is especially interested in them. It is important not to ignore the importance and difficulty of fixing observable problems, and there might exist a very valid critique of the AI safety community’s (over)emphasis on unobservable ones. However, this post will focus on unobservable failures.

What are the ways that we can try to tackle unobservable failures? One good way may ...
"Concepts of Agency in Biology" (Okasha, 2023) - Brief Paper Summary
by Nora_Ammann — 2023-07-08T18:22:43.228Z

This is a brief summary post of a recent paper titled "The Concept of Agent in Biology: Motivations and Meanings" by Samir Okasha (2023).

In both Biology and AI Alignment, the concept of agency comes up a lot and appears relevant to properly understanding the respecting "territory" - while also being used in confusing, and sometimes confused, ways.

Many more person-hours have been invested in progress in Biology compared to AI alignment, so it seems apt to keep an eyes out for any insights (as well as mistakes) from the discourse in Biology that might help improve the quality of discourse around the notion of 'agency' in Alignment.

I decided to write this summary post in order to make the information in the paper more accessible to the Alignment discourse. The paper itself is already a succinct (5 page!) summary of decades of discussions about the appropriate use of the notion of 'agency' in Philosophy of Biology. I can recommend to read it fully, though I have tried to capture all relevant ideas/concepts in this post.

Okasha himself is a well-respected Philosopher of Biology who has a range of substantive prior work on this and related topics (e.g. Agents and Goals in Evolution (2018), Evolution and the Levels of Selection (2006)).

0. Overview

"Biological agency" is a topic that has recently been gaining attention within Philosophy of Biology. In this paper, Okasha sets out to discuss "what exactly we mean by the term 'agent'?".

To bring some clarity to the discourse, he:

Distinguishes two motivations for why scholars typically invoke notions of 'biological agency', and then goes on to
Differentiate three distinct concepts of agency.

1. Two motivations

Motivation 1: 'Agency' as a way to point at what is distinctive about living organisms relative to other biological entities at different hierarchical levels (e.g. organs, tissues, species or ecosystems)

Biological organisms seems to possess certain 'agent-like attributes' (e.g. decision-making, learning about the environment, performing actions) that other biological entities, at different hierarchical scales, do not.
This is sometimes motivated by scholars wanting to counterbalance or oppose "gene-centric or reductionistic views of evolution and development".

"The worry that the gene-centric paradigm either 'leaves out the organism' altogether, or wrongly depicts all organi

...