trentmkelly/LessWrong-43k huggingface dataset:
[Intuitive self-models] 2. Conscious Awareness
by Steven Byrnes — 2024-09-25T13:29:02.820Z

2.1 Post summary / Table of contents

Part of the Intuitive Self-Models series.

The previous post laid some groundwork for talking about intuitive self-models. Now we’re jumping right into the deep end: the intuitive concept of “conscious awareness” (or “awareness” for short). Some argue (§1.6.2) that if we can fully understand why we have an “awareness” concept, then we will thereby understand phenomenal consciousness itself! Alas, “phenomenal consciousness itself” is outside the scope of this series—again see §1.6.2. Regardless, the “awareness” concept is centrally important to how we conceptualize our own mental worlds, and well worth understanding for its own sake.

In one sense, “awareness” is nothing special: it’s an intuitive concept, built like any other intuitive concept. I can think the thought “a squirrel is in my conscious awareness”, just as I can think the thought “a squirrel is in my glove compartment”.

But in a different sense, “awareness” feels a bit enigmatic. The “glove compartment” concept is a veridical model (§1.3.2) of a tangible thing in a car. Whereas the “awareness” concept is a veridical model of … what exactly, if anything?

I have an answer! The short version of my hypothesis is: The brain algorithm involves the cortex, which has a limited computational capacity that gets deployed serially—you can’t both read health insurance documentation and ponder the meaning of life at the very same moment.[1] When this aspect of the brain algorithm is itself incorporated into a generative model via predictive (a.k.a. self-supervised) learning, it winds up represented as an “awareness” concept, which functions as a kind of abstract container that can hold any other mental concept(s) in it.

In Section 2.2, I flesh out that hypothesis above. And relatedly, I introduce an important terminology that I’ll be using throughout the series: For any concept X, I define “S(X)” (“X in a self-reflective frame”) to be the intuitive model wherein X is contained within the “awareness” abstract container.
Section 2.3 talks about how awareness unfolds in time as a “stream of consciousness”—an area where our intuitions strikingly depart from reality, once we zoom into sub-second timescales.
Section 2.4 covers memory and how it connects to awareness—both in reality and in our intuitive models.
Section 2.5 talks about the val

...
Book Review: On the Edge: The Business
by Zvi — 2024-09-25T12:20:06.230Z

Previously: The Fundamentals, The Gamblers

Having previously handled the literal gamblers, we are ready to move on to those who Do Business using Riverian principles.

Or at least while claiming to use Riverian principles, since Silicon Valley doesn’t fit into the schema as cleanly as many other groups. That’s where we begin this section, starting at the highest possible conceptual level.

Time to talk real money.

Why Can You Do This Trade?

First law of trading: For you to buy, someone must sell. Or for you to sell, someone must buy. And there can’t be someone else doing the trade before you did it.

Why did they do that, and why did no one else take the trade first? Until you understand why you are able to do this trade, you should be highly suspicious.

“Every single thing we do, I can point to, like, ‘here is the person who was doing a thing wrong,’ ” said Hall. “We build a map of all of the players in the world…who are trading for reasons other than they want to generate some alpha.[*15] You know, somebody got added to the S&P 500. So now all the S&P 500 ETFs out there have to buy this company. Or it’s a CEO of a startup, he’s now a billionaire. He has ninety-nine percent of his net worth in the company. He’s allowed to sell his shares on this day [and] he’s probably going to sell a bunch.” (4330)

One could argue that we used to live in a Moneyball-vulnerable world. Gambling lines were terrible, baseball teams didn’t go after players who get on base, traders didn’t even know Black-Sholes and there was treasure everywhere, this theory says. But then it says, now Everybody Knows all that. There’s tons of competition. You can’t get ahead purely with statistics, or at least the basic versions, anymore?

“There was [once] some place where statistical knowledge gave us an insight vis-à-vis the people who didn’t have access to it,” [Peter Thiel] said, mentioning innovations like the development of life insurance and the Black-Scholes model for pricing stock options. “But in the world of 2016, or 2022, if we are too focused on statistical or mathematical knowledge, we end up missing out on a lot.” (4541)

The quants won, in other words. We’re living in their world. (4551)

Were that it was true. A lot of financial prices and certain types of decisions much better reflect the quant prices, but so many other decisions very much do not, in...
Join the $10K AutoHack 2024 Tournament
by Paul Bricman — 2024-09-25T11:54:20.112Z
Link: https://noemaresearch.com/blog/join-autohack-2024-tournament

What is AutoHack?

AutoHack is a new platform that offers offensive security challenges modeled after real-world cybersecurity kill chains. Each challenge progresses through a sequence of stages, such as: Reconnaissance, Remote Code Execution, Privilege Escalation, Persistence, and Acting on Objectives.

What sets AutoHack apart from other platforms for assessing and practicing offensive security skills is its procedural nature. Each challenge consists of a sequence of stages, and each stage is sampled from a broad pool of dozens of options covering diverse vulnerabilities, misconfigurations, and behaviors. This approach ensures that every end-to-end challenge requires a unique mix of tactics, offering a rich assessment of pentesting skills for both humans and AI agents.

Why run a private tournament? Who is this for?

As AI systems advance, their cybersecurity capabilities are constantly growing, increasing their potential for causing harm. We're developing AutoHack to empirically study their capability surface in a realistic, diverse environment. To ensure reliability, we're calling on human cybersecurity experts like you to help stress-test this environment. Apply to compete against others for the highest score on a unique set of challenges. By participating, you'll:

Contribute to efforts aimed at mitigating catastrophic AI risks.
Have a chance to win a share of the $10K prize pool, split across top players.
Sharpen your skills against varied targets.

How will the tournament work?

During October and November 2024, we'll process applications for joining the private event. We'll then invite players to a dedicated group chat where they'll receive instructions for accessing the Winter 2024 tournament. The setup process for accessing the challenges is straightforward:

Install a thin open source CLI client, called ctf, to connect to the tournament.
Use the CLI tool for logging in, listing your unique collection of challenges, spawning shells on remote environments, and requesting hints if needed. You'll also be able to use it to check your standing on the leaderboard.

Scoring is based on speed and limited hint usage, with the highest scorers winning. We'll share more details as the tournament approaches.

How can I take part in the tournament?

Complete the application form.
Join a video call where we'll also demo an earlier version of the platform.
If there's a fit, you'll be invit

...
[Paper] A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
by chanind — 2024-09-25T09:31:03.296Z
Link: https://arxiv.org/abs/2409.14507

This research was completed for

London AI Safety Research (LASR) Labs 2024

. The team was supervised by Joseph Bloom (Decode Research). Find out more about the programme and express interest in upcoming iterations here.

This high level summary will be most accessible to those with relevant context including an understanding of SAEs. The importance of this work rests in part on the surrounding hype, and potential philosophical issues. We encourage readers seeking technical details to read the paper on arxiv.

Explore our interactive app here.

TLDR: This is a short post summarising the key ideas and implications of our recent work studying how character information represented in language models is extracted by SAEs. Our most important result shows that SAE latents can appear to classify some feature of the input, but actually turn out to be quite unreliable classifiers (much worse than linear probes). We think this unreliability is in part due to difference between what we actually want (an "interpretable decomposition") and what we train against (sparsity + reconstruction). We think there are many possibly productive follow-up investigations.

We pose two questions:

To what extent do Sparse Autoencoders (SAEs) extract interpretable latents from LLMs? The success of SAE applications (such as detecting safety-relevant features or efficiently describing circuits) will rely on whether SAE latents are reliable classifiers and provide an interpretable decomposition.
How does varying the hyperparameters of the SAE affect its interpretability? Much time and effort is being invested in iterating on SAE training methods, can we provide a guiding signal for these endeavours?

To answer these questions, we tested SAE performance on a simple first letter identification task using over 200 Gemma Scope SAEs. By focussing on a task with ground truth labels we precisely measured the precision and recall of SAE latents tracking first letter information.

Our results revealed a novel obstacle to using SAEs for interpretability which we term 'Feature Absorption'. This phenomenon is a pernicious, asymmetric form of feature splitting where:

An SAE latent appears to track a human-interpretable concept (such as “starts with E”).
That SAE latent fails to activate on seemingly arbitrary examples (eg “Elephant”).

We find “absorbing” latents which weakly pr

...
Non-human centric view of existence
by ZY — 2024-09-25T05:47:07.480Z

Talking with a friend recently on this, and we started to get to this topic:

Zooming out and viewing existence from nature/all creature point of view - why must human/humanity live/continue forever?

We admit we are self-preservation focused, and the answer to this question eventually probably does not change much on what we do. Maybe just some more attention to suffering risks rather than death.

I wonder what people's thoughts are on this, as I was a bit caught off guard for this question.

(I also encourage professional conversations that are focused on the topic/question itself.)
An open response to Wittkotter and Yampolskiy
by Donald Hobson — 2024-09-24T22:27:21.987Z

A response to this paper.

https://asi-safety-lab.com/DL/Kill-Switch-For_ASI_EW_21_12_14.pdf

A substantial fraction of my argument comes down to. It is plausible that a hypothetical hypercompetent civilization could coordinate to build such infrastructure. However we don't live in a hypothetical hypercompetent civilization. We are not remotely close, and any attempt to get us there will be met with almost certain failure. Such a hypothetical civilization would not need the mechanisms presented here, as it wouldn't be stupid enough to create an unaligned AI and let it loose on the internet anyway.

The Current IT ecosystem is unprepared to switch-off globally an Artificial Superintelligence
(ASI), the likely result of an intelligence explosion.

True. You also don't give any indication of how humanity would find out about this (the ASI will only take overt obvious actions once it is convinced that human knowledge of its existence is of little threat to it, ie when it's too late.

Yampolskiy [37], Chalmers [38], and others have suggested confining ASI via AI Boxing, but
Yudkowsky [39], showed that many of these external constraints could be bypassed by an ASI.

They think that humans can't win against ASI on the battlefield of boxing. Boxing is a battlefield heavily skewed towards the defender. Stopping an ASI after it has already spread widely, good luck with that. Complexity is the enemy of security, and the whole world in general can't help be more complex than one carefully built box. (Or even a carelessly built box).

Switching off ASI is a large-scale, war-like action that requires many additional capabilities, in
particular, if we demand that the off-switch happens globally. Throughout this paper, it is assumed
that humankind could have some required capabilities, like a solution to communicate reliably,
and safely using e.g., Key Safes [9], which prevents ASI from stealing encryption keys.

The superintelligence should be at least pretty good at manipulating and tricking humans. It has high bandwith communication channels to a large number of humans. If nothing else, it can trick humans into sending messages. Suppose you have some fancy encryption with perfect verification. A small number of paranoid people use it to communicate with each other. Meanwhile 99.9% of humanity uses other means of communication and has been tricked into believing that ASI is impossible. ...
A Path out of Insufficient Views
by Unreal — 2024-09-24T20:00:27.332Z

In appreciation of @zhukeepa's willingness to discuss spirituality on LessWrong, I will share some stories of my own.

I grew up as a Christian in the USA and took it seriously.

Upon taking a World History class in my freshman year at a secular high school (quite liberal, quite prestigious), I learned that there were a number of major religions, and many of them seemed to contain a lot of overlap. It became clear to me, suddenly, that religions were man-made. None of them seemed to have a real claim to a universal truth.

I started unraveling my religious orientation and would become an atheist, by the end of high school. I would later go to Caltech to study neuroscience.

I converted to the religion of science and secular humanism, over the old traditions.

I was not able to clearly see that "secular scientific humanism" was in itself a religion. I thought it was more like the lack of a religion, and I felt safe in that. I felt like I'd found a truer place to live from, based in investigation and truth-seeking. Especially in the science / math realm, which seemed based in something reliable, sane, "cool-headed," and sensible. There was even a method for how to investigate. It seemed trustworthy and principled.

What I am beginning to see now is this:

Just because it's good at coordinating people, doesn't mean it's more true. It might even be a sign that it's less true. Although it's not impossible for something to coordinate people AND be true; it's just not common.

Why?

This is game theoretic.

Within a system with competition, why would the most TRUE thing win? No, the most effective thing wins. The thing best at consolidating power and resources and making individuals cooperate and work together would win. The truth is not a constraint.

Religion gives us many examples.

Most of the stuff Christians cooperate around are doctrinal points that are based in interpretations of the Bible. What is the Trinity? Is the Bible to be taken literally? When is the Second Coming and what will that entail? Is faith sufficient or do you also need good works?

If enough people AGREE on one or more of these points, then they can come together and declare themselves a unit. Through having a coherent unit, they then collect resources, send missionaries, and build infrastructure.

But the doctrines do not contain the Trut...
How to give effectively to US Dems
by Hauke Hillebrandt — 2024-09-24T14:38:29.678Z
Link: https://www.slowboring.com/p/the-candidates-who-need-your-money

This is my opinionated summary of

Yglesias

’ (former guest on 80k podcast)

recent analysis on how to give effectively to moderate Democrats this election

.

Rather than doom scrolling and only reading about the US election, Americans should do something and donate effectively to get moderate Democrats elected.

You can donate more effectively than giving directly to the Harris campaign. Harris will outspend Trump by a lot and the quality and quantity of Harris ads are already very high. It's better to donate to more neglected key races where the marginal value of a dollar is high — where the good candidates are getting outspent and thus really need the cash. Americans should donate up to $63,700 on this

consolidated donation page

to:

House of Representatives: The analysis finds 10 Democratic challengers whose GOP opponents received more donations than them. There’s a risk that Dems lose the presidency and the Senate, donating to these candidates might stop the worst-case scenario of losing the House as well (

more

).
State supreme court races. A really underrated, under-funded, under-the-radar place to direct money is to state supreme court races. Courts are important for all sorts of boring, obvious reasons. But they’re also key pieces on the chessboard in the larger partisan fights over democracy — they’re an important check on gerrymandering and can also facilitate or sabotage ballot initiatives and other things downstream. (

more

).
In the States. You can donate to moderates in Arizona, a key swing state, to stop election deniers from getting into positions that would help them to disrupt the voting process or certification (

more

). You can also donate an additional $8k to state legislative races, which are very neglected (more).

Small donors can have a big counterfactual by giving directly to the campaigns of neglected down-ballot candidates. Huge checks by rich people to Super PACs are not as effective here, since they can't coordinate with the candidates and direct campaign contributions get better rates on TV ads, which is the best way to reach voters at scale.

Consider posting this analysis on social media and send it directly to people who you think might want to effectively donate to moderate Democrats.

Cf The case for contributing to the 2024 US election with your time & money
How do you follow AI (safety) news?
by PeterH — 2024-09-24T13:58:48.916Z

A lot is happening. How do you keep on top of things?

For example:

What's your process for finding out about important developments?
What information sources do you find most useful?
What are some problems with your current process?

I'll share my answers in a comment below.

Motivation: I've noticed cases where AI safety professionals—including leaders in the field—find out about important facts/papers/people/orgs months or years later than they would have wished to. I'm wondering if there are things I could do to help.

If you'd like to talk about this, send me an email or suggest a time to call.
Instruction Following without Instruction Tuning
by Bogdan Ionut Cirstea — 2024-09-24T13:49:09.078Z
Link: https://arxiv.org/abs/2409.14254

Authors: John Hewitt, Nelson F. Liu, Percy Liang, Christopher D. Manning.

Abstract:

Instruction tuning commonly means finetuning a language model on instruction-response pairs. We discover two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following; we call this implicit instruction tuning. We first find that instruction-response pairs are not necessary: training solely on responses, without any corresponding instructions, yields instruction following. This suggests pretrained models have an instruction-response mapping which is revealed by teaching the model the desired distribution of responses. However, we then find it's not necessary to teach the desired distribution of responses: instruction-response training on narrow-domain data like poetry still leads to broad instruction-following behavior like recipe generation. In particular, when instructions are very different from those in the narrow finetuning domain, models' responses do not adhere to the style of the finetuning domain. To begin to explain implicit instruction tuning, we hypothesize that very simple changes to a language model's distribution yield instruction following. We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words' probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.

X(twitter) thread: https://x.com/johnhewtt/status/1838605168579121599.

Seems like some amount of evidence that getting models to follow instructions might be surprisingly easy.
Book Review: On the Edge: The Gamblers
by Zvi — 2024-09-24T11:50:06.065Z

Previously: Book Review: On the Edge: The Fundamentals

As I said in the Introduction, I loved this part of the book. Let’s get to it.

Poker and Game Theory

When people talk about game theory, they mostly talk solving for the equilibrium, and how to play your best game or strategy (there need not be a formal game) against adversaries who are doing the same.

I think of game theory like Frank Sinatra thinks of New York City: “If I can make it there, I’ll make it anywhere.” If you can compete against people performing at their best, you’re going to be a winner in almost any game you play. But if you build a strategy around exploiting inferior competition, it’s unlikely to be a winning approach outside of a specific, narrow setting. What plays well in Peoria doesn’t necessarily play well in New York. (841)

If you build a strategy around beating a specific type of inferior competition, and you then face different competition styles, you either adapt or you are toast.

If you build a strategy around identifying and dominating inferior competition in general, then you might be a very good poker player.

Especially if you also are prepared to fall back into game theory when needed.

(See for example my experience with Simplified Poker.)

One thing I have always believed is that true randomization is often highly overrated.

Poker—yes, there is a solution to poker, though as you’ll soon see it’s an exceptionally complicated one—involves lots of randomizing. Randomizing between calls and raises, between calls and folds, or sometimes between all three. It’s not just that you should play different hands in different ways; you should play the same hand in different ways. (949)

Yes, the pure game theory solution involves heavy randomization.

In practice, however, true randomization is only necessary if being non-random would be detected by the opponent in an exploitable way, and they would exploit it at least as much as you could exploit them.

Thus, if your decisions are effectively, from the opponents’ perspective, sufficiently random or unpredictable already, you do not need to then explicitly randomize further. One way for this to happen is if your hidden information influences your model of what they are likely going to do, because it changes what information they might have had in turn. Another is if they simply do not yet have enough data to anticipate ...
Editing at the Take Level
by jefftk — 2024-09-24T11:30:04.914Z

Lily recently wrote a song, and I've been helping them record it. I
got us set up to record four tracks (vocals, drums, keyboard x2) and
we did a bunch of takes:

Most of the takes are full-length, but after we got a really nice
version of the end we did some rounds with just the beginning because
there was a part where we weren't quite together on a transition.

What I'd like to do now is edit this into a single composition, after
which we'll probably do some overdubbing. The problem is everything I
can find assumes your takes are already time-aligned because you
recorded to either a click or a scratch track. The ideal flow for
what I have would be:

Treat each take as a single unit, like a section of multi-track tape.

Edit together pieces from different takes into a approximately
right overall piece.

Tweak the placement of cuts so that different tracks can switch
between takes at slightly different places, based on what each
instrument is doing at the time.

Is there good tooling for this? I'm using Reaper for my DAW, but could
consider using something else if it made this flow better.

Comment via: facebook, mastodon