trentmkelly/LessWrong-43k huggingface dataset:
Big-endian is better than little-endian
by Menotim — 2024-04-29T02:30:48.053Z

This is a response to the post We Write Numbers Backward, in which lsusr argues that little-endian numerical notation is better than big-endian.[1] I believe this is wrong, and big-endian has a significant advantage not considered by lsusr.

Lsusr describes reading the number "123" in little-endian, using the following algorithm:

Read the first digit, multiply it by its order of magnitude (one), and add it to the total. (Running total: ??? one.)
Read the second digit, multiply it by its order of magnitude (ten), and add it to the total. (Running total: ??? twenty one.)
Read the third digit, multiply it by its order of magnitude (one hundred), and add it to the total. (Arriving at three hundred and twenty one.)

He compares it with two algorithms for reading a big-endian number. One is using the same process as for a little-endian number, but from right to left. I agree with him that this is worse than the little-endian algorithm, because it is easier to read a number in the same direction as the text that surrounds it, which in English is from left to right.

The other big-endian algorithm is the one I observe myself as usually using. For "321", it is:

Count the digits (three), and convert that into an order of magnitude (one hundred). (Running total: ??? hundred ???.)
Read the first digit, multiply it by its order of magnitude (one hundred), and add it to the total. (Running total: three hundred ???.)
Read the second digit, multiply it by its order of magnitude (ten), and add it to the total. (Running total: three hundred and twenty ???.
Read the third digit, multiply it by its order of magnitude (one), and add it to the total. (Arriving at three hundred and twenty one.)

The point raised by lsusr against the big-endian algorithm is that we must count a number's digits before we can start reading them. He doesn't say explicitly why he dislikes this, but I can see three reasons:

It makes the algorithm more complex.
It makes the algorithm slower.
It means we cannot begin the algorithm if we only have access to the beginning of a number.

Making the algorithm more complex is bad, but not very bad, because it is still fairly simple. It being slow to count all the digits in a number is a real problem, but we can usually solve it by separating groups of digits using commas or by using exponential notation. Finally, only having access the beginning of a number is not a very comm...
The Prop-room and Stage Cognitive Architecture
by Robert Kralisch — 2024-04-29T00:48:17.473Z

This is a post on a novel cognitive architecture I have been thinking about for a while now, first as a conceptual playground to concretise some of my agent foundation ideas, and lately as an idea for a project that approaches the Alignment Problem directly by concretising a sort of AI-Seed approach for an inherently interpretable yet generally intelligent architecture on which to implement a notion of "minimal alignment" before figuring out how to scale it.
I don't strongly expect this to work, but I don't yet see where it has to fail and it feels sufficiently fresh/untried to at least see where it breaks down.
I omitted most of the technical details that I am considering, since this post is about the conceptual portion of the architecture, and purposefully not fully polished (to get around some of my publishing anxieties).
I am hoping to treat this as a living document or to make follow up posts of updates and larger clarifications. Because of that, I really appreciate criticisms and challenges to any ideas you find in here.
Anyway, here goes:

PSCA

The Prop-room and Stage Cognitive Architecture is a neuro-symbolic architecture for embedded agency, PSCA for short. It is inspired from the observation that the human brain seems to run an ongoing simulation of the “current scene”, in which many sensory and emotional modalities are unified into a single referential context. Furthermore, the level of resolution at which elements of this simulation are being simulated depends on their relevance to the current objective that the human is attending to.

The proposed architecture aims to emulate this particular ability of the brain by specifying a “Stage”, which is a hypergraph representing the “current scene”, that contains present elements of relevance-adjusted resolution. The features of those elements are parameter nodes that are connected by edges corresponding to the causal relationships within the current scene.
The idea is now to use constraint satisfaction in order to increase certainty about beforehand uncertain nodes in the graph, which includes nodes referring to the agent’s next action. The current objective is also part of the scene, playing an important role in constraining the expected action. One interesting feature about this architecture is that it naturally concretises objectives, so that the general goal of e.g. finding some...
How are Simulators and Agents related?
by Robert Kralisch — 2024-04-29T00:22:30.751Z

In this post, I will provide some speculative reasoning about Simulators and Agents being entangled in certain ways.
I have thought quite a bit about LLMs and the Simulator framing for them, and I am convinced that it is a good explanatory/predictive frame for behavior of current LLM (+multimodal) systems.
It provides intuitive answers for why the difficulties around capability assessment for LLMs exist, why it is useful to say "please" and "thank you" when interacting with them, and lines out somewhat coherent explanations about their out of distribution behaviors (i.e. Bing Sydney).
When I was wrapping my head around this simulator classification for GPTs, contrasting them against previously established ideas about classifying AGIs, like oracles or genies, I noticed that simulators seem like a universally useful cognitive structure/algorithm, and that most agents in nature seem to implement some version of a simulator to predict the behaviors of systems that are entangled with those agents’ regulation targets.
And on the other hand, targeted simulation tends to be biased towards agents for various reasons, whereas untargeted simulation tends to sample simulacra until it reaches a stable pattern - which may either be an inert state in simulated space, or an agent-like pattern that keeps up its own boundaries/is self-evidencing.

To put it plainly:

I suspect that agents tend to converge on implementing simulators as part of their cognition, and that simulators tend to naturally discover/implement agents.
What follows in this post is a sort of handwavy explanation of this suspicion. I have no idea if this is true, but I think it is sufficiently interesting or even educatively wrong to deserve posting. Please feel free to engage and criticize as much as you like, this is speculative conceptual work and I may not know more than you about any given point.

Agents convergently implement simulators

Simulators are one natural solution for the kind of compression task that embedded agents are faced with: The environment is complex, and it changes, and the agent is always interacting with just a relatively small portion of the environment, the selection of which tends to change continuously, but sometimes more radically (i.e. when a predator shows up).
What kind of realistic algorithm can make “useful” predictions[1] on this kind of sensory input sequence?...
Extended Embodiment
by Robert Kralisch — 2024-04-29T00:18:12.892Z

I find that an especially illustrative thought experiment regarding embodiment is to imagine a superintelligent Stone that can talk. Let’s say that this Stone can somehow perceive its environment but is, as you might expect, incapable of moving around. The Stone is not a very powerful optimiser of its environment as long as it is just lying around at the beach or under some rubble, but this story quickly changes if the Stone is found by a human.

I like to imagine this in a tribal setting, where the Stone is brought back to the tribe and figures out how to play the role of a holy spirit or something of the sort. One could also go with a modern setting, but I think the point is more clear with the tribal one, because taking control of this setting is easier for us to imagine.

In any case, the Stone could expectedly quickly figure out how to manipulate the humans’ behavior. In relatively little time, the tribe effectively becomes the arms and legs of the Stone, as well as its extended eyes and ears. Even some of the cognition can be outsourced, for example by giving vague instructions in some cases but correctly figuring that the humans will end up making the right decisions.

It’s tempting to imagine how the Stone would efficiently assimilate other tribes, establish institutions to outsource management and generally find ways to make its control over the planet wider and more stable, instrumentalizing humanity.
One can almost see this body, made out of humans, with institutions as organs, slowly stretch around the planet and tighten the grip of optimisation pressure towards whatever goal the Stone pursues.
However, the body analogy might distract from the fact that the Stone could just be waiting to “upgrade its body”, to engineer more efficient tools. Maybe the Stone is attached to humans or has a somewhat similar morality, but if not, it might well replace most humans with machines eventually.

In light of this thought experiment, what can be said about the Stone’s embodiment? Would it be correct to say that the Stone retains its body throughout all of this, or is this a progression from a small Stone body to eventually having the planet Earth as its body?

One could say that the Stone still interfaces with the world from its original body, only able to perceive a very limited amount of data and give a very limited amount of verbal instru...
Referential Containment
by Robert Kralisch — 2024-04-29T00:16:00.174Z

This is an idea I am toying around with for understanding resolutionally adjusted causal modeling - this is just a bunch of intuitions and pointing towards a somewhat clear framing of a fundamental thing. I am sure there are already plenty of accounts for how to approach this kind of task, but I like to figure stuff out by myself, at least initially.
This could definitely benefit from some visual aids, but I prioritized publishing.

“Referential containment” relates to how one might sensibly divide a system into parts or how a system with many parts might be understood as a system with fewer parts - it relates to the translation between different resolutions.

The mathematical intuition here is that if we think of a physical system as a complex causal graph with many edges and nodes, the nodes being information units and the edges representing the relationships of that information, then if we are tasked with dividing this graph into two areas, the dividing line should cross as few edges as possible.

The two resulting pieces will then have the fewest possible edges between them, so the explanation to account for these edges will also be the shortest. Most “references” are “contained” within the respective objects. This also works for hyperedges, and we might want to assign weights to different kinds of edges to indicate how “costly” they are to cross.

To expand this a bit towards more practical examples we could think of material objects and their surface and permanence. Solid objects are usually referentially contained in their locality, which partially guides our understanding of how to differentiate such objects from their surroundings. If I make a cut through an apple, the pieces are no longer connected as strongly in their locality so it becomes easier to think of them as two objects.

However, this is contextual. The locality of the apple halves is not entirely statistically independent, it rarely happens that two apple halves are moved more than a few meters apart before being eaten. Also, I chose the context of locality here, but one could choose among a great number of contexts that will provide different referential boundaries according to which I can detect objects/phenomena/patterns.

In practice, there will likely be too many dimensions of references to track to begin with (as they can be almost arbitrarily constructed), and my...
Disentangling Competence and Intelligence
by Robert Kralisch — 2024-04-29T00:12:50.779Z

I struggle quite a bit with publishing, so today I am just going to publish a bunch of hopefully relatively refined drafts. Any comments and questions are welcome, I would be grateful to elaborate on any area where I appear unclear. If you like, you can also let me know some thoughts about formatting this to be easier to read. This applies to all my future posts.
This post captures most of my current thoughts on deconfusing intelligence.
note: the two parts overlap somewhat, but I didn’t quite synthesize them yet. Part 1 had been included in our publications in AISC before, but I think it's cleaner to post it seperately.

Part 1 (comments on conceptual confusion):

The “ability to achieve goals in a wide range of environments” as per Markus Hutter and Shane Legg (https://arxiv.org/abs/0706.3639) is often used as a definition of general intelligence, but in my opinion it better captures the notion of general competence.

How might the two concepts differ?

Most agents can achieve significantly fewer goals in many environments if their embodiments are changed (say, if their legs were removed), even when retaining the same cognitive capabilities. Is it therefore right to say that an agent has become less intelligent as a consequence of losing its legs?

I believe that including the embodiment/interface of an agent within our measure/concept of its intelligence inflates the concept and leads to confusion.

An example of such confusion would be the notion that companies are already superintelligent agents/entities and thus serve as useful analogies/frameworks for thinking about how to align superintelligent agents in general. While I agree that companies can be more competent than individuals in many domains, they are interfacing with the world very differently, which is something that needs to be accounted for before characterising this difference in competence as purely/mostly a matter of intelligence.

Competence is contextual and compositional, there are many factors that can contribute to it. The sensors that an agent/system possesses, its actuators, processing speed, data storage capacity, and so on, are things that I believe should be understood separately from intelligence. If we study the factors that make up the construct of competence, both as individual factors and in their interaction/relation with each other, we become capable of generating more p...
List your AI X-Risk cruxes!
by Aryeh Englander — 2024-04-28T18:26:19.327Z

[I'm posting this as a very informal community request in lieu of a more detailed writeup, because if I wait to do this in a much more careful fashion then it probably won't happen at all. If someone else wants to do a more careful version that would be great!]

By crux here I mean some uncertainty you have such that your estimate for the likelihood of existential risk from AI - your "p(doom)" if you like that term - might shift significantly if that uncertainty were resolved.

More precisely, let's define a crux as a proposition such that:
(a) your estimate for the likelihood of existential catastrophe due to AI would shift a non-trivial amount depending on whether that proposition was true or false;
(b) you think there's at least a non-trivial probability that the proposition is true; and
(c) you also think there's at least a non-trivial probability that the proposition is false.

Note 1: It could also be a variable rather than a binary proposition, for example "year human-level AGI is achieved". In that case substitute "variable is above some number x" and "variable is below some number y" instead of proposition is true / proposition is false.

Note 2: It doesn't have to be that the proposition / variable on it's own would significantly shift your estimate. If some combination of propositions / variables would shift your estimate, then those propositions / variables are cruxes at least when combined.

For concreteness let's say that "non-trivial" here means at least 5%. So you need to think there's at least a 5% chance the proposition is true, and at least a 5% chance that it's false, and also that your estimate for p(existential catastrophe due to AI) would shift by at least 5% depending on whether the proposition is true or false.

Here are just a few examples of potential cruxes people might have (among many others!):

Year human-level AGI is achieved

How fast the transition will be from much lower-capability AI to roughly human-level AGI, or from roughly human-level AGI to vastly superhuman AI

Whether power seeking will be an instrumentally convergent goal

Whether AI will greatly upset the offense-defense balance for CBRN technologies in a way that favors malicious actors

Whether AGIs could individually or collectively defeat humanity if they wanted to

Whether the world can collectively get their collective act together to pause AGI development given a clear eno

...
Things I tell myself to be more agentic
by DMMF — 2024-04-28T17:44:39.789Z
Link: https://danfrank.ca/things-i-tell-myself-to-be-more-agentic/

I'm sharing this here since being agentic has become somewhat of a meme in EA circles. Despite its meme status, I find the idea very helpful and empowering, though difficult to put into action.

This is written as a personal self-affirmation — my hope is that by writing it up and sharing it publicly, I will internalize these words more and act on them. In the context of this post, being agentic means being more ambitious, doing more things out in the world, putting myself out there more, and tolerating higher levels of rejection, discomfort, and failure.

Your life can become much better or worse based on your actions and, more generally, the randomness of the universe.
Nobody knows what your talents, ambitions, and knowledge base are like except you.
There is no divine providence for your life. Just being kind, talented, or good at your job won’t provide you with opportunities or good outcomes — good outcomes don’t just happen automatically because you wish them to; you need to actually go out and create the opportunity for yourself. If you were famous or the child of a billionaire, perhaps people would proactively think of opportunities to bring to you. However, since you are not, it is crucial that you advocate for yourself. If you don’t show the world your talents or share with individuals what you are seeking, others won’t know what you have to offer or are searching for.
You have a distribution problem. If you have twice the network and outreach, you’d probably have twice the amount of goodness in outcomes/opportunities. Actually, likely more because these good things compound and make the next thing more likely.
Ask people for ambitious, specific requests for support rather than just hoping for general support. When you make a general request, it requires the person to brainstorm all possible ways they could help, which is too broad. In contrast, specific requests narrow down their thought process and reverse the onus so they need to come up with a reason why they cannot help, making it counter-intuitively more likely for them to offer meaningful support. If they can't meet the specific request, they often think of an alternative solution they can provide, which they wouldn't have considered under a general plea for support. It may feel uncomfortable to make these asks, but it typically isn’t costly for someone to say no or ignore you.
Many world-renowned supe

...
Estimating the Number of Players from Game Result Percentages
by Daniel L — 2024-04-28T17:42:03.247Z

Recently I got into the daily word puzzle game Couch Potato Salad. At the end of the game, it shows the percent of players who “nailed”, ”sailed”, ”prevailed”, ”exhaled” and ”failed”. Once, I played the game shortly after midnight when the new puzzle becomes available. I nailed it (ha!), but noticed that the game results suspiciously neatly split into 33.3%, 66.6% and zeroes for the rest. Aha, I thought, there must be only three or six or nine or twelve of us playing the game that early in the morning. That got me interested in figuring the number of players based on the score.

Generalized as a number of respondents based on the poll results, I thought the following logic should work.

The number of respondents is the minimum positive integer that when multiplied by each of the voting percentages produces a near-integer number.

ChatGPT had concurred, while pedantically pointing that “this concept is akin to finding the least common multiple (LCM) of denominators in fractional representations of those percentages.” Ok.

I concocted the following Pythonic implementation:

N = min(
[n for n in range(1, Nmax)
if all(
[abs(v[i]*n - round(v[i]*n)) < 100*e
for i in range(0, len(v))
])
])

Where

e is margin of error, (0, 1)
v contains the list of voting results, sum(v) must be equal to 1.0.
Nmax is the maximum reasonable number of respondents

I tried it with actual numbers from that game results from a bit later:

>>> N = lambda v, Nmax=10000, e=0.001: \
min([n for n in range(1, Nmax) \
if all(
[abs(v[i]*n - round(v[i]*n)) < 100*e \
for i in range(0, len(v))])
])
>>> N([0.333, 0.666, 0, 0, 0])
3
>>> N([0.125, 0.125, 0.75, 0, 0])
8
>>> N([0.409, 0.136, 0.182, 0.045, 0.227])
22
>>> N([0.821, 0.103, 0.051, 0.026, 0])
39
>>> N([0.671, 0.075, 0.171, 0.04, 0.043])
374

I don't know what was the actual number of people played, but intuitively this seems alright. It was still early in the morning.

ChatGPT suggested using numpy for efficiency, but agreed that my one-liner is Pythonic and elegant.

Implementation aside, what do you think? Is there a different / better way to think about this problem? How would you approach this?
The Science Algorithm - AISC 2024 Final Presentation
by Johannes C. Mayer — 2024-04-28T14:55:50.504Z
Link: https://www.youtube.com/watch?v=eKypHdMveCU

I gave a presentation about what I have been working on in the last 3 months. Well a tiny part of it, as it was only a 10-minute presentation.

Here is the vector planning post mentioned in the talk.