trentmkelly/LessWrong-43k huggingface dataset:
A framework and open questions for game theoretic shard modeling
by Garrett Baker — 2022-10-21T21:40:49.887Z

Background

The Shard Theory of Human Values is a new research agenda which I think has promise in serving as a model of value formation in humans and self-supervised ML agents. In brief, it claims that humans are made up of possibly self-preserving contextually-activated-heuristics called shards, that the sum of these is what we mean when we say 'human values', and that self-supervised ML agents will have this same format.

But there's a dreadful problem... there's currently no way of attaching numbers to its concepts! So it's difficult to prove any given shard-theoretic explanation wrong. Here I present my brief thoughts on the subject, and try to give an informal overview of a framework I think will have promise for these purposes, as well as some questions I think are likely solvable via experiment or further theoretical thinking around these lines.

Shards and contexts

Suppose we have two shards: Fatman and Littleboy. Fatman likes doughnuts, and will spend its bids on actions which in expectation will achieve that end. Littleboy likes being thin, and will spend its bids on actions which in expectation will achieve that end.

Fatman has more bidding power when the agent is in a room which it models includes a doughnut in it cd, and less when the agent is in a room adjacent from one which it models includes a doughnut in it cadj, and still less when the agent doesn't model a doughnut in any rooms c∅.

Littleboy has more bidding power when there are mirrors in the room its currently in cm, and if there's nothing remarkable about the room it finds itself in, it has a bit more bidding power than Fatman c∅.

This can be thought of as each shard having portfolios over contexts in which its activated.

Context	Fatman	Littleboy
Doughnut cd	10	1
Doughnut adjacent cadj	5	1
Mirror cm	1	10
Unremarkable room c∅	1	1

There are two ways we could aggregate the contexts. One way is to just add them up. If there's a doughnut in the room, and a doughnut in an adjacent room, because there's a doughnut in the room, Fatman gets 10 bids, then the adjacent doughnut gives Fatman 5 more bids, for a total of 15 bids.

Another way is to multiply together bids. In this case, this would give Fatman 10×5=50 bids.

It's also unclear if there should be a form of context-starvation between highly related contexts. If there's two doughnuts in a room, ...
Cooperators are more powerful than agents
by Ivan Vendrov — 2022-10-21T20:02:35.224Z

Epistemic Status: Very speculative, intended to provoke thought rather than convince.

A crux of much AI safety research is the theory that

agents are powerful

; that we should robustly expect agents that plan and seek power to outcompete other entities in the long term. Among agents,

higher levels of agency are better

- we should expect entities with planning and metacognition to outcompete entities that merely learn from direct experience or mimicry.

I claim this theory is not well-supported by empirical evidence, and that it’s at least as plausible to expect cooperation, not agency, to be the key property that determines which entities will survive and outcompete.

I don’t mean to say that cooperation and agency are mutually exclusive; in fact they are complements, and I expect future intelligent systems to exhibit high levels of both. I mean that power-seeking and cooperation are mutually exclusive, and if the world selects for cooperation more strongly than for agency, the

instrumental convergence

arguments for power-seeking may not go through.

Evidence from Biomass

A clear notion of power that has been the subject of extreme optimization pressure for billions of years is biomass. An organism’s biomass measures the amount of matter they directly control (with a lot of corner cases and “extended phenotype” considerations we won’t get into - let’s just count carbon atoms).

The vast majority of the world’s biomass (>99%) is controlled by bacteria, plants, and fungi, entities which are not particularly agentic. But perhaps we should exclude them because they are not really in control of their component matter, not in the same sense that animals are - e.g. they are either much slower at moving their carbon atoms around (plants, fungi) or are so small and simple that they just don’t count (bacteria).

Let’s very roughly normalize for organism size and speed, then. Once we’ve normalized, the most outlier species in terms of biomass are

Humans, with a single species composing something like 50% of all large animal biomass (most of the remainder being livestock under our direct control).
Eusocial insects, especially ants and termites,

which compose 2% of insect species but the majority of insect biomass.

What makes these species so powerful? Not agency, not their ability to plan. Their distinctive feature relative to ...
Intelligent behaviour across systems, scales and substrates
by Nora_Ammann — 2022-10-21T17:09:32.512Z

Tl;dr

In an

earlier post

, I introduced a metaphor for thinking about the epistemic landscape of AI alignment, and then described three epistemic strategies for making progress on the alignment problem.

In this post, I will double-click on the third strategy: learning from intelligence-in-the-wild. In particular, I will explore in more detail a core assumption this epistemic bet is based on—namely, that intelligent behaviour, as exhibited by different types of systems, both natural and artificial, share underlying principles which we can study and exploit.

An epistemic bet: Learning from Intelligence-in-the-Wild

Earlier

, I defined the epistemic strategy of learning from intelligence-in-the-wild as follows:

Finally, the third approach attempts to chart out the possibility space of intelligent behaviour by looking at how intelligent behaviour manifests in existing natural systems ([3] in Fig. 1).

Instead of calling it a strategy, we could also call it an epistemic bet. This framing is helpful in emphasising the fact that in research, we are in the business of making bets. We cannot be entirely confident a given approach will bear fruit before we try it. But we can be more or less confident it will, and thus make informed bets. Furthermore, the term "bet" makes appeals to

epistemic pluralism

more intuitive (by reference to the reasons for why diversifying, say, one’s investment portfolio is a good idea).

In the case of this specific bet, the hope is that by studying intelligent behaviour as it manifests in existing biological or social systems, and by recovering principles that govern complex systems across various scales and modalities of implementation, we can gain substantial insights into how to design intelligent behaviour (with certain desired properties) in artificial systems.

Premise: Intelligent behaviour across systems, scales and substrates

What reasons do we have to expect this strategy will be fruitful?

The promise of this epistemic bet is, in part, premised on treating intelligent behaviour[1] as a “real”, naturally occurring phenomenon, rather than, say, some theoretical or linguistic construct. Furthermore, intelligent behaviour - as we can observe it in different types of systems - shares underlying principles of functioning. As such, we can observe and investigate the phenomenon of int...
Deepfake(?) Phishing
by jefftk — 2022-10-21T14:30:04.188Z

[EDIT: I now think this probably wasn't a deepfake, but a recording the attacker made when using the same attack against the previous person. See the comments.]

I think someone just tried to phish my Facebook account, including a fake video of a FB friend. Here's the conversation:

Them, via FB Messenger, 9:32am:

Please ,I was trying to login in my instagram page on Facebook my new phone and they ask me to find someone to help me receive a code, Facebook gave me two friends suggestions and you one of them, the other person isn't online. will you Help me receive the code please?

Me:

I'm sorry you're having trouble logging in! Just so I can make sure your account hasn't been hacked, how did we meet?

Them:

[Calls me over FB Messenger, audio isn't working but it does look like them. I'm completely convinced at this point.]

Me:

Audio wasn't working, but I did recognize you
What do you need me to do?

32665, over SMS:

NNNNNNNN is your Facebook password reset code [this number has previously sent me FB resets]

Them:

Send me the code sent to you minute ago

Me:

Hmm, those look like the code to reset the password to my account?
Can we call again?

Me:

[I try to call them back, doesn't go through]

Them:

Nahh it's for my instagram

Them:

Having bad connections here

Them:

Send me the code ?

Me:

sorry, I'm still worried your account has been hacked -- can we do another call?

Them:

[Calls me over FB Messenger, audio is still not working, and the video feels slightly off. Ends quickly on their end. Possibly it's even the same video from last time?]

Me:

We're you able to hear me?

Them:

My connections

I've reported their account as hacked.

Things that made me suspicious:

I don't think FB has any sort of account recovery that looks like this

This is exactly what an attempt to hack my FB account would look like

9:30am, even though that makes it 6:30 where they live

Video call didn't have any audio

They couldn't receive incoming video calls

Text did't feel like them, though I don't know them that well.

Here's a screenshot I took during the second video call:

Even with all those things, the video call would normally have been very convincing, and it did briefly convince me. I could easily see it fooling someone who didn't know about deepfake video.

Comment via: facebook
acronyms ftw
by Emrik — 2022-10-21T13:36:39.378Z

Everyday acronyms are some of the most usefwl innovations for making language more efficient. Conlangs are too hard to make well and too expensive to coordinate switching to, but we can profitably innovate on the margin.

Never really understood why this didn't take off tbh.

By default there's huge inertia to how fast language evolves to fit the new purposes it's being used for, and that inertia is proportional to the number of people who use the archaic term for that purpose. Ever notice how we still have terms like "probability", "Nash equilibrium", and "independent impression" despite how long they are and how often we (should) use them?

But we don't have to settle for default growth! As long as we have common knowledge that trying to innovate this is a good thing (as opposed to everyone believing that coining new terms is the ultimate act of hubris and needs to be punished), we can constantly be trying out new forms. I'd prefer if enthusiasm were the default response to someone trying to make up new words, and punishment/disappointment only be issued after evaluating the suggestion as bad enough that you'd want to discourage the effort.

I think ideally we should spend a much larger fraction of our talking time on debating which terms to use. Or better yet, have a central real-time voting database, sort of like a weaker form of an assurance contract implemented via a crowdaction platform. On the platform, people (preferably LW user accounts) can informally commit to use a new term conditional upon N other people also committing to it.

Communities that are comfortable with "internet slang" have a massive advantage due to their willingness to use new terms. Sometimes they even compete to use the newest terms. Imagine the power we'd have if we could learn from some of that culture! What we have instead is a very strict informal regulation on who's allowed to even suggest new terms, lest they be exiled for the crime of status-grabbing above their station.

A selection of acronyms maybe you'd like to use

One danger with importing acronyms like these instead of inventing our own is that we're clogging up the short-word space with code that's not optimised for our own semantic distribution, and we might wish to reserve room for our own potential shibboleths there. So the list below only contains words I presently want to see used, minus some that I'm pretty confide...
Crossword puzzle: LessWrong Halloween 2022
by jchan — 2022-10-21T12:41:58.676Z

Just in time for Halloween - the spookiest themed crossword puzzle of the year!

Interactive web version

Printable PDF

The comments below may contain unmarked spoilers. I don't expect many people will solve it without doing at least some Wikipedia lookups, but you're welcome to try.

Enjoy!
Weekly Roundup #2
by Zvi — 2022-10-21T12:10:00.919Z

Experimenting with pushing the non-Covid roundup on Friday instead of Thursday, and trying out dropping non-Covid from the title, and I plan to start pulling more focused posts out of the bank of written stuff in a week or two. Over time I’ll figure out how much belongs in a digest, how much is time sensitive, and how much doesn’t and isn’t.

Communication Styles

Alexey Guzey wants efficient communication.

One could respond that you have no respect for your time. Why is it obvious what you are going to say a third of the way in, yet you still say the other two thirds of it unless I interrupt you?

If the norm is ‘most of the time I will interrupt you first’ then of course that is fine. The problem is this is often considered super rude, and many people hate it. You cannot assume people will efficiently interrupt you, since the third possibility of ‘they don’t feel permission to do this’ also exists. Given that, perhaps check in a third of the way through until you know they’re not like that?

There is a key difference here.

Alexey is modeling this as ‘take an interest by exploring various intuitions and finding out what is hard or interesting.’ Amanda is modeling this as ‘the mansplainer thinks he has actually solved the problem.’

And yeah, if you are going in thinking you have solved it, that is not good. Stop it. If you’re learning and exploring the problem, because that is the curious and potentially helpful thing to do, then that is great on principle. The problem is that if it is interpreted as the other thing, then that won’t do at all, and you’ll need to stop it. Right now is a bad time, in general, for this move in these kinds of spots, even the good version.

When talking to me, yes, absolutely, if you are confident you know where I am going with something, or it would save time in expectation to ask if your hunch is right, or You Know This Already, please interrupt me. Then, if you are wrong, update on that. Unless, of course, I’m clearly having fun saying it. In that case let me finish, ya know?

Same goes for when there is something you disagree with. It is maddening to me when there are norms that one can’t note disagreement for fear of interrupting flow.

Strategic Use of Strategic Oil Reserve

Biden continues to release more oil from the reserve as of mid-October. More interesting is the plan to refill the reserve at ...
Improved Security to Prevent Hacker-AI and Digital Ghosts
by Erland Wittkotter — 2022-10-21T10:11:39.344Z

In a previous post on “Hacker-AI and Digital Ghosts – Pre-ASI”. I discussed multiple problems contributing to our vulnerabilities – facilitating the development of an AI application (Hacker-AI) that could threaten our freedom, privacy, and quality of life.

I hypothesized that a Hacker-AI/Digital Ghost finds and uses (new) vulnerabilities in existing software or modifies software to create backdoors or sleeper code. The core hypothesis was that Hacker AI is flexible enough to find ways to manipulate every system (i.e., any OS, CPU, or device type). With Reverse Code Engineering, it could not just steal crypto-keys but also modify other software and try to avoid detection by manipulating the OS and removing all data traces it inadvertently generates (for becoming a Digital Ghost). Finally, this Hacker-AI could make itself irremovable on every device it visited.

My previous post also suggested that some governments are highly motivated to create Hacker-AI first because an irremovable AI on all devices could repel late-coming AIs and become the only Hacker-AI. This Hacker AI could make conventional war-fighting impossible, mainly by disrupting the supplies and logistics of nations that were taken by surprise.

A Hacker-AI cannot be prevented by governmental regulation or international treaties/diplomacy. We must have a reliable and fast technical solution to counter any side’s first (cyber-)strike.

The following 10 problems, issues, or vulnerabilities do not follow a particular order. It is assumed that preventing digital ghosts and irremovable Hacker-AI implies confronting these issues/problems technically.

10 issues or security-related problems

The issues introduced in the mentioned post are briefly repeated here:

(1) “Software is invisible”: Software can only be seen/validated indirectly; it requires trust in the entire ecosystem. Rephrased: “Software must be trusted (blindly); someone else will (hopefully) see the problems.”

(2) “Software is covertly modifiable” Emphasis is on “covert”; detection of modification is not reliable. There are too many blindspots where covert (temporary) modifications could happen.

(3) “Every tool/component could be compromised”. We must question our trust in whatever we use. Rephrased: “Any software can be dangerous.

(4) “Attackers know more (about vulnerabilities)”. Attackers know more about the weak spot they use, while...
Two Guts
by chanamessinger — 2022-10-21T10:01:46.108Z
Link: https://chanamessinger.com/blog/two-guts

Me: “I don’t know what I should do next? Lots of things seem good, but I can’t think of particularly good reasons for them to cash out to what I care about.”

Hallucinated frankenstein interlocutor: “People in your position tend to undervalue judgement and gut thinking relative to explicit reasoning, you should go with what your gut says is good."

Me: “I live inside my head and I can definitely say that much of the time my gut is saying what it thinks high status people would want to hear or thinking about the last three examples and none before or is taking into account at most two out of the whole space of considerations, why am I trusting it again?”

HFI: “Sure, but if you just go with explicit reasoning, you’ll miss a bunch of unquantifiable intuitions that contain a lot of data, and those are valuable.”

Me: “Yes, but again from inside my head I need you to hear me when I say my gut is like the people who answer “should there be homosexuals in the military” differently from “should there be gays or lesbians in the military” and I don’t think I’m like unusual, this is like the whole point of the rationality project.”

This is a kind of conversation I have a lot (if this confuses you, see

Appendix 1

), which at times has left me feeling like maybe I’m the only person in EA/rationality who thinks they have a brain that makes mistakes by default, that can’t be trusted to make good decisions without actually thinking about them. This is definitely not the case!

There are a bunch of ways in which HFI and I above are not understanding each other (

Appendix 2

), but one is that, like half of a cow, there are two guts in play.

Sometimes people in the world are like “trust your gut instinct about people” and they mean “trust your split-second first impression.” And sometimes they’re like “in the end, you have to go with your gut” and they mean “after marinating in all the relevant ideas, thinking for a long time, sleeping on it, probably doing a lot of explicit reasoning, if you have a deep sense of what you should do, it’s worth trusting.”

And I often heard people as saying “trust your snap gut”, and I was like “my snap gut judgment is that that is insane. Do you know that I don’t know anything about this.” But at least in one case, and likely others, they meant “trust your reflective gut”, the gut that’s had time to sit with everything and digest it (the meta...
The importance of studying subjective experience
by Q Home — 2022-10-21T08:43:25.191Z

I think people severely underestimate the importance of studying subjective experience (qualia).

In my opinion, people think a lot about "the mystery of qualia" (the hard problem of consciousness), but not enough about "the miracle of qualia". I.e. why qualia is cool and interesting. It's not as trivial as it seems. Subjective experience is always with us, so it's hard to notice how strange it is (even without any "hard problems"). Down below is the list of reasons why qualia deserves way more attention than people give it. For convenience, I split the post into 4 parts.

Epistemic status: I haven't read a lot of relevant literature. Some claims are based on indirect evidence: we clearly don't live in a culture were subjective experience is valued as much as I think it should. Discussing this topic may be an exercise in prioritization.

Qualia signals missed knowledge

Subjective experience (qualia) tells us that there are differences between objects which feel very simple, but lay beyond any current knowledge. That is to say we don't have any model of those differences. We don't know the (mathematical) space of our experiences.

Subjective experience is like a big carrot hanging right before your eyes. Telling "hey, I'm hiding some new knowledge you're completely missing out". If I were a mathematician or an AI researcher or an artist, I would be going insane trying to guess what could resemble the workings of subjective experience.

Examples

Some examples of differences which feel simple, but don't have an explanation/a model:

Differences between tastes, smells, tactile sensations and other (direct enough) physical experiences.
Differences between human voices. (See Timbre) A lot of subjective differences of sounds are not explained.
Differences between simple impressions of people. Your simplest opinions/feelings about other people are not explained.

And much more. Any experience teases you about the knowledge you lack. Every second. And yet a lot of people don't react to this "teasing" or don't even feel it. Don't try to advance their understanding of experience.

Qualia is core to value

Imagine your favorite piece of art or your favorite place. Your favorite story, fictional or real. Your favorite fictional character. Your memories about personalities of your friends. All of this is about qualia.

Anything that is meaningful or interesting in life is created by the subjecti...