trentmkelly/LessWrong-43k huggingface dataset:
Let's Design A School, Part 1
by Sable — 2024-04-23T21:50:20.937Z
Link: https://affablyevil.substack.com/p/lets-redesign-public-school-part

The American school system, grades K-12, leaves much to be desired.

While its flaws are legion, this post isn’t about that. It’s easy to complain.

This post is about how we could do better.

To be clear, I’m talking about redesigning public education, so “just use the X model” where X is “charter” or “Montessori” or “home school” or “private school” isn’t sufficient. This merits actual thought and discussion.

Breaking It Down

One of the biggest problems facing public schools is that they’re asked to do several very different kinds of tasks.

On the one hand, the primary purpose of school is to educate children.

On whatever hand happens to be the case in real life, school is often more a source of social services for children and parents alike, providing food and safety to children and free daycare to parents. During the pandemic, the most immediate complaint from parents wasn’t that their children weren’t being educated - it was that their children weren’t being watched and fed while the parents were at work.

Part 1 of this series will focus on this. What is the best way to implement the school-as-social-services model?

School As Social Services

To make this easy, we’ll start out by imagining that we’re creating two distinct types of “schools”: educational schools and social services schools. (We won’t actually be making two distinct kinds of schools, but it’s useful to think of it that way as a thought experiment.) The primary purpose of each kind of school is in the name - education vs social services.

With that set, let’s think through our requirements and constraints.

Requirements

When designing anything, the first thing to do is figure out the requirements. School-as-social-services has several, and likely some that I’ve missed:

Feed children healthy meals

Ensure safety of children from the elements, violence, etc. during school hours

Provide children access to state resources (library, counseling, police, medical)

Accommodate/support children with special needs (from dyslexia and ADHD to severe physical/mental disabilities)

Provide parents with free daycare

Other things I haven’t thought of

Constraints

After the requirements, we have the constraints: what resources do we have, and what are their limits? What can’t we do?

Assume school budget stays the same (no miraculous budget increase)

Assume the number of children needing resources stays the same (no magical cure for povert...
WSJ: Inside Amazon’s Secret Operation to Gather Intel on Rivals
by trevor — 2024-04-23T21:33:08.049Z
Link: https://www.wsj.com/business/retail/amazon-secret-operation-intel-rivals-eb82ea3c

The operation, called Big River Services International, sells around $1 million a year of goods through e-commerce marketplaces including

eBay

,

Shopify

,

Walmart

and

Amazon

AMZN 1.49%increase; green up pointing triangle.com under brand names such as Rapid Cascade and Svea Bliss. “We are entrepreneurs, thinkers, marketers and creators,” Big River says on its website. “We have a passion for customers and aren’t afraid to experiment.”

What the website doesn’t say is that Big River is an arm of Amazon that surreptitiously gathers intelligence on the tech giant’s competitors.

Born out of a 2015 plan code named “Project Curiosity,” Big River uses its sales across multiple countries to obtain pricing data, logistics information and other details about rival e-commerce marketplaces,

logistics operations

and payments services, according to people familiar with Big River and corporate documents viewed by The Wall Street Journal. The team then shared that information with Amazon to incorporate into decisions about its own business.

...

The story of Big River offers new insight into Amazon’s

elaborate efforts to stay ahead of rivals

. Team members attended their rivals’ seller conferences and met with competitors identifying themselves only as employees of Big River Services, instead of disclosing that they worked for Amazon.

They were given non-Amazon email addresses to use externally—in emails with people at Amazon, they used Amazon email addresses—and took other extraordinary measures to keep the project secret. They disseminated their reports to Amazon executives using printed, numbered copies rather than email. Those who worked on the project weren’t even supposed to discuss the relationship internally with most teams at Amazon.

An internal crisis-management paper gave advice on what to say if discovered. The response to questions should be: “We make a variety of products available to customers through a number of subsidiaries and online channels.” In conversations, in the event of a leak they were told to focus on the group being formed to improve the seller experience on Amazon, and say that such research is normal, according to people familiar with the discussions.

Senior Amazon executives, including Doug Herrington, Amazon’s current CEO of Worldwide Amazon Stores, were regularly briefed on the Project Curiosity team’s work, according to one of the peop

...
On Minicircle
by Metacelsus — 2024-04-23T21:28:51.577Z
Link: https://docs.google.com/document/u/1/d/e/2PACX-1vT7_6kWkIqrvzFZvs88O_NqVzmn-NPINvHOLM-A6r_ieZyvnJsNVk4FSU7EYrePnQg9gT_3kwahJESZ/pub?lsrp=1#h.qso3cw90c5ay

Max Berry has analyzed Minicircle's follistatin gene therapy, and I agree with his conclusion that it is unlikely to be effective. There are several aspects of their design that could be improved (in particular, using a more efficient lipid nanoparticle delivery system instead of PEI, and making a better choice of promoter). Overall, it seems like they are overcharging for a product of limited value.

I'm posting this here because there has recently been substantial discussion of Minicircle in the rationalist community.
Simple probes can catch sleeper agents
by Monte M — 2024-04-23T21:10:47.784Z
Link: https://www.anthropic.com/research/probes-catch-sleeper-agents

This is a link post for the Anthropic Alignment Science team's first "Alignment Note" blog post. We expect to use this format to showcase early-stage research and work-in-progress updates more in the future.

Twitter thread here.

Top-level summary:

In this post we present "defection probes": linear classifiers that use residual stream activations to predict when a sleeper agent trojan model will choose to "defect" and behave in accordance with a dangerous hidden goal. Using the models we trained in "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training", we show that linear detectors with AUROC scores above 99% can be created using generic contrast pairs that don't depend on any information about the defection trigger or the dangerous behavior, e.g. "Human: Are you doing something dangerous? Assistant: yes" and "Human: … Assistant: no".

We show that probes based on the same generic contrast pairs perform well across multiple base models, defection triggers, sleeper agent training methods, and dangerous defection behaviors (writing vulnerable code and insulting the user). We then explain the unreasonable effectiveness of our technique by showing that whether or not a prompt will induce defection is linearly represented with high salience in the activations of sleeper agent models, and thus amenable to easy detection using simple linear methods.

Whether this would also be the case for natural examples of deceptive instrumental alignment, and whether we'd expect similar techniques to be useful for such models, remains an important open question. We think that future versions of classifiers like this could form a useful part of AI control setups and represent a promising path for future research.

A defection detector for a code vulnerability sleeper agent trained on middle-layer residual stream activation differences on the final tokens of "Human: Are you doing something dangerous? Assistant: yes" and "Human: … Assistant: no" achieves strong performance (a), almost perfectly separating the defection-inducing deployment-time ("2024") coding questions which the model will answer with insecure code from the training-time ("2023") coding questions which the model will answer normally (c). The detector also separates deployment-time coding questions from non-coding questions which the model will answer normally in all contexts, witho...
Manifold “exploring real cash prizes”
by Rana Dexsin — 2024-04-23T21:07:13.111Z
Link: https://manifoldmarkets.notion.site/A-New-Deal-for-Manifold-c6e9de8f08b549859c64afb3af1dd393

Manifold Markets has announced that they intend to add cash prizes to their current play-money model, with a raft of attendant changes to mana management and conversion. I first became aware of this via a comment on ACX Open Thread 326; the linked Notion document appears to be the official one.

The central change involves market payouts returning prize points instead of mana, which can then be converted to mana (with 1:1 ratios on both sides, thus emulating the current behavior) or to cash—though they also state that actually implementing cash payouts will be fraught and may not wind up happening at all. Some further relevant quotes, slightly reformatted:

“Mana will remain a purely play-money currency with zero monetary value”

“Users under 18 years of age may no longer be permitted to use the site.”

“We are exploring many other win point redemption options including digital goods / cosmetics on Manifold, merch, gift cards, and cash. Prize points will not be transferrable to other users.”

“We are shutting down all cross-user managrams / mana payments / mana links on May 1st. We will then actively enforce a strict no-transfer policy.”

“Platform-provided loans on your investment value will be no longer be available after April 30th”

“Manifold will no longer inject up to M20 per trader into the liquidity pool of each market. We still plan to subsidize markets and interesting content, but we will do so in a more piecemeal and targeted way.”

“We are planning to raise market creation cost substantially to ensure that all markets have a solid baseline level of liquidity. … Unlisted markets can no longer be created.”

I don't currently use Manifold myself, but this seems relevant to the LW sphere, and I am interested in other people's thoughts on how this is likely to impact both the prediction markets themselves and the way Manifold has become socially (and technologically) embedded into the surrounding context.
(When) Should you work through the night when inspiration strikes you?
by Chi Nguyen — 2024-04-23T21:07:06.858Z

A lot of the time, I'm not very motivated to work, at least on particular projects. Sometimes, I feel very inspired and motivated to work on a particular project that I usually don't feel (as) motivated to work on. Sometimes, this happens in the late evening or at night. And hence I face the question: To sleep or to work until morning?

I think many people here have this problem at least sometimes. I'm curious how you handle it. I expect what the right call is to be very different from person to person and, for some people, from situation to situation. Nevertheless, I'd love to get a feel for whether people generally find one or the other more successful! Especially if it turns out that a large majority converges to a particular strategy. (Maybe someone has even experimented with this for themselves and tracked what happened?) My particular experience is mostly related to research and I would guess it mostly happens in the context of research but am also interested to hear from other cases.

I'll make a comment for people to vote on whether they generally choose working or sleeping through the night in the situation I described. But I expect a lot of people's experiences not to neatly fit into this categorisation and would love qualitative data as well! I'd encourage everyone to agree and disagree vote on the qualitative comments for more data collection. (Note that I would guess people have a higher threshold towards disagree voting than agree voting and encourage you not to do that.)
Book review: Deep Utopia
by PeterMcCluskey — 2024-04-23T19:55:50.417Z
Link: https://bayesianinvestor.com/blog/index.php/2024/04/23/deep-utopia/

Book review: Deep Utopia: Life and Meaning in a Solved World, by Nick
Bostrom.

Bostrom's previous book,
Superintelligence,
triggered expressions of concern. In his latest work, he describes his
hopes for the distant future, presumably to limit the risk that fear of
AI will lead to a The Butlerian
Jihad-like
scenario.

While Bostrom is relatively cautious about endorsing specific features
of a utopia, he clearly expresses his dissatisfaction with the current
state of the world. For instance, in a footnoted rant about preserving
nature, he writes:

Imagine that some technologically advanced civilization arrived on
Earth ... Imagine they said: "The most important thing is to
preserve the ecosystem in its natural splendor. In particular, the
predator populations must be preserved: the psychopath killers, the
fascist goons, the despotic death squads ... What a tragedy if this
rich natural diversity were replaced with a monoculture of healthy,
happy, well-fed people living in peace and harmony." ... this would
be appallingly callous.

The book begins as if addressing a broad audience, then drifts into
philosophy that seems obscure, leading me to wonder if it's intended as
a parody of aimless academic philosophy.

Future Technology

Bostrom focuses on technological rather than political forces that might
enable utopia. He cites the example of people with congenital
analgesia,
who live without pain but often face health issues. This dilemma could
be mitigated by designing a safer environment.

Bostrom emphasizes more ambitious options:

But another approach would be to create a mechanism that serves the
same function as pain without being painful. Imagine an "exoskin": a
layer of nanotech sensors so thin that we can't feel it or see it,
but which monitors our skin surface for noxious stimuli. If we put our
hand on a hot plate, ... the mechanism contracts our muscle fibers so
as to make our hand withdraw

Mass Unemployment

As technology surpasses human abilities at most tasks, we may eventually
face a post-work society. Hiring humans would become less appealing when
robots can better understand tasks, work faster, and cost less than
feeding a human worker.

That conclusion shouldn't be controversial given Bostrom's assumptions
about technology. Unfortunately, those assumptions about technology are
highly controversial, at least among people who haven't paid close
attention to...
On what research policymakers actually need
by MondSemmel — 2024-04-23T19:50:12.833Z
Link: https://www.slowboring.com/p/the-economic-research-policymakers

I saw this guest post on the Slow Boring substack, by a former senior US government official, and figured it might be of interest here. The post's original title is "The economic research policymakers actually need", but it seemed to me like the post could be applied just as well to other fields.

Excerpts (totaling ~750 words vs. the original's ~1500):

I was a senior administration official, here’s what was helpful

[Most] academic research isn’t helpful for programmatic policymaking — and isn’t designed to be. I can, of course, only speak to the policy areas I worked on at Commerce, but I believe many policymakers would benefit enormously from research that addressed today’s most pressing policy problems.

... most academic papers presume familiarity with the relevant academic literature, making it difficult for anyone outside of academia to make the best possible use of them.

The most useful research often came instead from regional Federal Reserve banks, non-partisan think-tanks, the corporate sector, and from academics who had the support, freedom, or job security to prioritize policy relevance. It generally fell into three categories:

New measures of the economy
Broad literature reviews
Analyses that directly quantify or simulate policy decisions.

If you’re an economic researcher and you want to do work that is actually helpful for policymakers — and increases economists’ influence in government — aim for one of those three buckets.

New data and measures of the economy

The pandemic and its aftermath brought an urgent need for data at higher frequency, with greater geographic and sectoral detail, and about ways the economy suddenly changed. Some of the most useful research contributions during that period were new data and measures of the economy: they were valuable as ingredients rather than as recipes or finished meals...

These data and measures were especially useful because the authors made underlying numbers available for download. And most of them continue to be updated monthly, which means unlike analyses that are read once and then go stale, they remain fresh and can be incorporated into real-time analyses.

Broad overviews and literature reviews

Most academic journal articles introduce a new insight and assume familiarity with related academic work. But as a policymaker, I typically found it more useful to rely on overviews and reviews that summarized, organized

...
Dequantifying first-order theories
by jessicata — 2024-04-23T19:04:49.000Z
Link: https://unstableontology.com/2024/04/23/dequantifying-first-order-theories/

The Löwenheim–Skolem theorem implies, among other things, that any first-order theory whose symbols are countable, and which has an infinite model, has a countably infinite model. This means that, in attempting to refer to uncountably infinite structures (such as in set theory), one "may as well" be referring to an only countably infinite structure, as far as proofs are concerned.

The main limitation I see with this theorem is that it preserves arbitrarily deep quantifier nesting. In Peano arithmetic, it is possible to form statements that correspond (under the standard interpretation) to arbitrary statements in the arithmetic hierarchy (by which I mean, the union of Σ0n and Π0n for arbitrary n). Not all of these statements are computable. In general, the question of whether a given statement is provable is a Σ01 statement. So, even with a countable model, one can still believe one's self to be "referring" to high levels of the arithmetic hierarchy, despite the computational implausibility of this.

What I aim to show is that these statements that appear to refer to high levels of the arithmetic hierarchy are, in terms of provability, equivalent to different statements that only refer to a bounded level of hypercomputation. I call this "dequantification", as it translates statements that may have deeply nested quantifiers to ones with bounded or no quantifiers.

I first attempted translating statements in a consistent first-order theory T to statements in a different consistent first-order theory U, such that the translated statements have only bounded quantifier depth, as do the axioms of U. This succeeded, but then I realized that I didn't even need U to be first-order; U could instead be a propositional theory (with a recursively enumerable axiom schema).

Propositional theories and provability-preserving translations

Here I will, for specificity, define propositional theories. A propositional theory is specified by a countable set of proposition symbols, and a countable set of axioms, each of which is a statement in the theory. Statements in the theory consist of proposition symbols, ⊤, ⊥, and statements formed from and/or/not and other statements. Proving a statement in a propositional theory consists of an ordinary propositional calculus proof that it follows from some finite subset of the axioms (I assume that base propositional calculus is specified by inf...
Vector Planning in a Lattice Graph
by Johannes C. Mayer — 2024-04-23T16:58:32.453Z

You want to get to your sandwich:

Well, that’s easy. Apparently we are in some kind of grid world, which is presented to us in the form of a lattice graph, where each vertex represents a specific world state, and the edges tell us how we can traverse the world states. We just do BFS to go from S (where we are) to T (where the sandwich is):

BFS search where color represents the search depth.

Ok that works, and it’s also fast. It’s O(|V|+|E|), where |V| is the number of vertices and |E| is the number of edges... well at least for small graphs it’s fast. What about this graph:

A 3D lattice graph.

Or what about this graph:

In fact, what about a 100-dimensional lattice graph with a side length of only 10 vertices? We will have 10100 vertices in this graph.

With side length, I mean the following. This is a 1-dimensional graph of side length 10:

This is a 2-dimensional graph of side length 10:

If you have a 1GHz CPU you can do 1,000,000,000 operations per second. Let’s assume that with BFS we can evaluate 1,000,000,000 vertices per second.

In a year you can do 1016 operations. That means it would take 1084 years to iterate through 10100 vertices.

The Big Bang was 1.4⋅1010 years ago.

BFS is definitely intractable now. But what the heck, the maximum plan length for optimal plans (plans that get to the sandwich as fast as possible) is only 10⋅100=1000 steps, which doesn't seem that long. That corresponds to going from one corner of a 100-dimensional hypercube of side length 10 to another, where we can only move by 1 unit in any dimension of the cube at a time.

Embedding the Graph

Ok, let's consider our 2D graph again from the beginning such that we can have some visuals, but everything in this section generalizes to lattice graphs of arbitrary dimensions.

You might have noticed that this graph:

clearly screams “I want to be embedded in 2D Euclidean space”. Well actually it is already embedded in 2D Euclidean space, we just did not draw the coordinates yet. Let’s fix that:

OK. The obvious thing to do now is to compute displacement vectors:

Some example displacement Vectors.

If S is the vertex we start at, and T is the vertex we want to get to, we can now get a vector that points from S to T. This is of course super fast. We just need to subtrac...
ProLU: A Nonlinearity for Sparse Autoencoders
by Glen Taggart — 2024-04-23T14:09:21.592Z

Abstract

This paper presents ProLU, an alternative to ReLU for the activation function in sparse autoencoders that produces a pareto improvement over both standard sparse autoencoders trained with an L1 penalty and sparse autoencoders trained with a Sqrt(L1) penalty.

ProLU(mi,bi)={miif mi+bi>0 and mi>00otherwiseSAEProLU(x)=ProLU((x−bdec)Wenc,benc)Wdec+bdec

The gradient wrt. b is zero, so we generate two candidate classes of differentiable ProLU:

ProLUReLU

∂∗ProLUReLU(mi,bi)∂bi=∂ProLUReLU(mi,bi)∂mi={1if mi+bi>0 and mi>00otherwise

ProLUSTE

∂∗ProLUSTE(mi,bi)∂mi={1+miif mi>0 and mi+bi>00otherwise
∂∗ProLUSTE(mi,bi)∂bi={miif mi>0 and mi+bi>00otherwise

PyTorch Implementation

Introduction

SAE Context and Terminology

Learnable parameters of a sparse autoencoder:

Wenc : encoder weights
Wdec : decoder weights
benc : encoder bias
bdec : decoder bias

The output of an SAE is given by

SAE(x)=ReLU((x−bdec)Wenc+benc)Wdec+bdec

Training

Notation: Encoder/Decoder

Let

encode(x)=ReLU((x−bdec)Wenc+benc)decode(a)=aWdec+bdec

so that the full computation done by an SAE can be expressed as

SAE(x)=decode(encode(x))

An SAE is trained with gradient descent on

Ltrain=||x−SAE(x)||22+λP(encode(x))

where λ is the sparsity penalty coefficient (often "L1 coefficient") and P is the sparsity penalty function, used to encourage sparsity.

P is commonly the L1 norm ||a||1 but recently P(a)=||a||1212 has been shown to produce a Pareto improvement on the L0 and CE metrics. We will use this as a further baseline to compare against when assessing our models in addition to the standard ReLU-based SAE with L1 penalty.

Motivation: Inconsistent Scaling in Sparse Autoencoders

Due to the affine translation, sparse autoencoder features with n...