trentmkelly/LessWrong-43k huggingface dataset:
The Great Data Integration Schlep
by sarahconstantin — 2024-09-13T15:40:02.298Z

Midjourney, “Fourth Industrial Revolution Digital Transformation”

This is a little rant I like to give, because it’s something I learned on the job that I’ve never seen written up explicitly.

There are a bunch of buzzwords floating around regarding computer technology in an industrial or manufacturing context: “digital transformation”, “the Fourth Industrial Revolution”, “Industrial Internet of Things”.

What do those things really mean?

Do they mean anything at all?

The answer is yes, and what they mean is the process of putting all of a company’s data on computers so it can be analyzed.

This is the prerequisite to any kind of “AI” or even basic statistical analysis of that data; before you can start applying your fancy algorithms, you need to get that data in one place, in a tabular format.

Wait, They Haven’t Done That Yet?

Each of these machines in a semiconductor fab probably stores its data locally. The team that operates one machine might not be able to see the data from the next one over.

In a manufacturing context, a lot of important data is not on computers.

Some data is not digitized at all, but literally on paper: lab notebooks, QA reports, work orders, etc.

Other data is is “barely digitized”, in the form of scanned PDFs of those documents. Fine for keeping records, but impossible to search, or analyze statistically. (A major aerospace manufacturer, from what I heard, kept all of the results of airplane quality tests in the form of scanned handwritten PDFs of filled-out forms. Imagine trying to compile trends in quality performance!)

Still other data is siloed inside machines on the factory floor. Modern, automated machinery can generate lots of data — sensor measurements, logs of actuator movements and changes in process settings — but that data is literally stored in that machine, and only that machine.

Manufacturing process engineers, for nearly a hundred years, have been using data to inform how a factory operates, generally using a framework known as statistical process control. However, in practice, much more data is generated and collected than is actually used. Only a few process variables get tracked, optimized, and/or used as inputs to adjust production processes; the rest are “data exhaust”, to be ignored and maybe deleted. In principle the “excess” data may be relevant to the facility’s performance, but nobody knows how, and they’re not equipped to...
"Real AGI"
by Seth Herd — 2024-09-13T14:13:24.124Z

I see "AGI" used for everything from existing LLMs to superintelligence, and massive resulting confusion and illusory disagreements. I finally thought of a term I like for what I mean by AGI. It's an acronym that's also somewhat intuitive without reading the definition:

Reasoning, Reflective Entities with Autonomy and Learning

might be called "(R)REAL AGI" or "real AGI". See below for further definitions.

Hoping for AI to remain hobbled by being cognitively incomplete looks like wishful thinking to me. Nor can we be sure that these improvements and the resulting dangers won't happen soon.

I think there are good reasons to expect that we get such "real AGI" very soon after we have useful AI. After 2+ decades of studying how the brain performs complex cognition, I'm pretty sure that our cognitive abilities are the result of multiple brain subsystems and cognitive capacities working synergistically. A similar approach is likely to advance AI.

Adding these other cognitive capacities creates language model agents/cognitive architectures (LMCAs). Adding each of these seems relatively easy (compared to developing language models) and almost guaranteed to add useful (but dangerous) capabilities.

More on this and expanded arguments and definitions in an upcoming post; this is primarily a reference for this definition of AGI.

Reasoning

Deliberative "System 2" reasoning allows humans to trade off "thinking time" for accuracy.
Aggregating cognition for better results can be added to nearly any system in fairly straightforward ways.

Reflective

Can think about their own cognition.
Useful for organizing and improving cognition.
Reflective stability of values/goals has important upsides and downsides for alignment

Entities

Evokes the intuition of a whole mind, rather than piece of a mind or a cognitive tool

with Autonomy

Acts independently without close direction.

Very useful for getting things done efficiently

Has agency in that they take actions, and have explicit goals they pursue with flexible strategies
Including deriving and pursuing explicit, novel subgoals

This is highly useful for factoring novel complex tasks
One useful subgoal is "make sure nobody prevents me from working on my main goal." ;)

and Learning

Continuous learning from ongoing experience
Humans have at least four types; LMCAs currently have one and a fraction.[1]
These are all str

...
AI, centralization, and the One Ring
by owencb — 2024-09-13T14:00:16.126Z
Link: https://strangecities.substack.com/p/ai-centralization-and-the-one-ring

People thinking about the future of AI sometimes talk about a single project ‘getting there first’ — achieving AGI, and leveraging this into a decisive strategic advantage over the rest of the world.

I claim we should be worried about this scenario. That doesn’t necessarily mean we should try to stop it. Maybe it’s inevitable; or maybe it’s the best available option — decentralized development of AI may make it harder to coordinate on crucial issues such as maintaining high safety standards, and this is a major worry in its own right. But I think that there are some pretty serious reasons for concern about centralization of power. At minimum, it seems important to stay in touch with those. This post is deliberately a one-sided exploration of these concerns.

In some ways, I think a single successful AGI project would be analogous to the creation of the One Ring. In The Lord of the Rings, Sauron had forged the One Ring, an artifact powerful enough to gain control of the rest of the world. While he was stopped, the Ring itself continued to serve as a source of temptation and corruption to those who would wield its power. Similarly, a centralized AGI project might gain enormous power relative to the rest of the world; I think we should worry about the corrupting effects of this kind of power.

Forging the One Ring was evil

Of course, in the story we are told that the Enemy made the Ring, and that he was going to use it for evil ends; and so of course it was evil. But I don’t think that’s the whole reason that forging the Ring was bad.

I think there’s something which common-sense morality might term evil about a project which accumulates enough power to take over the world. No matter its intentions, it is deeply and perhaps abruptly disempowering to the rest of the world. All the other actors — countries, organizations, and individuals — have the rug pulled out from under them. Now, depending on what is done with the power, many of those actors may end up happy about it. But there would still, I believe, be something illegitimate/bad about this process. So there are reasons to refrain from it[1].

In contrast, I think there is something deeply legitimate about sharing your values in a cooperative way and hoping to get others on board with that. And by the standards of our society, it is also legitimate to just accumulate money by selling goods or services to others, in o...
Evidence against Learned Search in a Chess-Playing Neural Network
by p.b. — 2024-09-13T11:59:55.634Z

Introduction

There is a new paper and lesswrong post about "learned look-ahead in a chess-playing neural network". This has long been a research interest of mine for reasons that are well-stated in the paper:

Can neural networks learn to use algorithms such as look-ahead or search internally? Or are they better thought of as vast collections of simple heuristics or memorized data? Answering this question might help us anticipate neural networks’ future capabilities and give us a better understanding of how they work internally.

and further:

Since we know how to hand-design chess engines, we know what reasoning to look for in chess-playing networks. Compared to frontier language models, this makes chess a good compromise between realism and practicality for investigating whether networks learn reasoning algorithms or rely purely on heuristics.

So the question is whether Francois Chollet is correct with transformers doing "curve fitting" i.e. memorisation with little generalisation or whether they learn to "reason". "Reasoning" is a fuzzy word, but in chess you can at least look for what human players call "calculation", that is the ability to execute moves solely in your mind to observe and evaluate the resulting position.

To me this is a crux as to whether large language models will scale to human capabilities without further algorithmic breakthroughs.

The paper's authors, which include Erik Jenner and Stuart Russell, conclude that the policy network of Leela Chess Zero (a top engine and open source replication of AlphaZero) does learn look-ahead.

Using interpretability techniques they "find that Leela internally represents future optimal moves and that these representations are crucial for its final output in certain board states."

While the term "look-ahead" is fuzzy, the paper clearly intends to show that the Leela network implements an "algorithm" and a form of "reasoning".

My interpretation of the presented evidence is different, as discussed in the comments of the original lesswrong post. I argue that all the evidence is completely consistent with Leela having learned to recognise multi-move patterns. Multi-move patterns are just complicated patterns that take into account that certain pieces will have to be able to move to certain squares in future moves for the pattern to hold.

The crucial different to having learned an algor...
My career exploration: Tools for building confidence
by lynettebye — 2024-09-13T11:37:55.843Z

Crossposting from my blog

I did a major career review during 2023. I’m sharing it now because:

I think it’s a good case study for

iterated depth decision-making

in general and reevaluating your career in particular, and
I want to let you know about my exciting plans! I’m doing the Tarbell Fellowship for early-career journalists for the next nine months. I’m excited to dive in and see if AI journalism is a good path for me long-term. I’ll still be doing coaching, but my availability will be more limited.

Background

I love being a productivity coach. It’s awesome watching my clients grow and accomplish their goals.

But the inherent lack of scalability in 1:1 work frustrated me. There was a nagging voice in the back of my head that kept asking “Is this really the most important thing I can be doing?” This voice grew more pressing as it became increasingly clear artificial intelligence was going to make a big impact on the world, for good or bad.

I tried out a string of couple-month projects. While good, none of them grew into something bigger. I had some ideas but they weren’t things that would easily grow without deliberate effort. (Needing the space to explore these ideas prompted me to try

CBT for perfectionism

.)

I always had this vague impostery feeling around my ideas, like they would just come crashing down at some point if I continued. I wasn’t confident in my decision-making process, so I wasn’t confident in the plans it generated.

So at the beginning of last year, I set out to do a systematic career review. I would sit down, carefully consider my options, seek feedback, and find one I was confident in.

This is the process I used, including the specific tools I used to tackle each of my sticking points.

Deciding Which Problem to Work On

I’m a big proponent of

theories of change

, and think that the cause I pick to work on heavily influences how much impact I can make. I also need to match my personal fit to specific career opportunities, so it’s harder than “just” determining what the most important cause is and choosing a job working on that.

My solution was to write about what I thought were the most pressing problems in the world. I kept that info in the back of my mind later when brainstorming options and considering theories of change for specific options.

Example

My short answer from Jan 2023:

AI seems like the...
Contra papers claiming superhuman AI forecasting
by nikos — 2024-09-12T18:10:50.582Z

[Conflict of interest disclaimer: We are

FutureSearch

, a company working on AI-powered forecasting and other types of quantitative reasoning. If thin LLM wrappers could achieve superhuman forecasting performance, this would obsolete a lot of our work.]

Widespread, misleading claims about AI forecasting

Recently we have seen a number of papers – (Schoenegger et al., 2024, Halawi et al., 2024, Phan et al., 2024, Hsieh et al., 2024) – with claims that boil down to “we built an LLM-powered forecaster that rivals human forecasters or even shows superhuman performance”.

These papers do not communicate their results carefully enough, shaping public perception in inaccurate and misleading ways. Some examples of public discourse:

Ethan Mollick (>200k followers)

tweeted

the following about the paper Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy by Schoenegger et al.:

A post on

Marginal Revolution

with the title and abstract of the paper

Approaching Human-Level Forecasting with Language Models

by Halawi et al. elicits responses like

"This is something that humans are notably terrible at, even if they're paid to do it. No surprise that LLMs can match us."
"+1 The aggregate human success rate is a pretty low bar"

A

Twitter thread

with >500k views on LLMs Are Superhuman Forecasters by Phan et al. claiming that “AI […] can predict the future at a superhuman level” had more than half a million views within two days of being published.

The number of such papers on AI forecasting, and the vast amount of traffic on misleading claims, makes AI forecasting a uniquely misunderstood area of AI progress. And it’s one that matters.

What does human-level or superhuman forecasting mean?

"Human-level" or "superhuman" is a hard-to-define concept. In an academic context, we need to work with a reasonable operationalization to compare the skill of an AI forecaster with that of humans.

One reasonable and practical definition of a superhuman forecasting AI forecaster is

The AI forecaster is able to consistently outperform the crowd forecast on a sufficiently large number of randomly selected questions on a high-quality forecasting platform.[1]

(For a human-level forecaster, just replace "outperform" with "performs on par with".)

Red flags for claims to (super)human AI forecasting accuracy

Our experience suggests there are a numb...
OpenAI o1
by Zach Stein-Perlman — 2024-09-12T17:30:31.958Z

It's more capable and better at using lots of inference-time compute via long (hidden) chain-of-thought.

OpenAI pages: Learning to Reason with LLMs, o1 System Card, o1 Hub

Tweets: Sam Altman, Noam Brown, OpenAI

Discussion: https://www.transformernews.ai/p/openai-o1-alignment-faking

"METR could not confidently upper-bound the capabilities of the models during the period they had model access."

"Apollo Research believes that o1-preview has the basic capabilities needed to do simple in-context scheming." (Apollo prompted the model to misbehave; this was an evaluation for scheming capabilities, not propensity.)

Apollo found that o1-preview sometimes instrumentally faked alignment during testing . . . , it sometimes strategically manipulated task data in order to make its misaligned action look more aligned to its ‘developers’ . . . , and an earlier version with less safety training proactively explored its filesystem to test for the presence of developer oversight before acting on its misaligned goal.
How to Give in to Threats (without incentivizing them)
by Mikhail Samin — 2024-09-12T15:55:50.384Z

(available as a paper.)

TL;DR: using a simple mixed strategy, LDT can give in to threats, ultimatums, and commitments - while incentivizing cooperation and fair[1] splits instead.

This strategy made it much more intuitive to many people I've talked to that smart agents probably won't do weird everyone's-utility-eating things like threatening each other or participating in commitment races.

1. The Ultimatum game

This part is taken from planecrash[2][3].

You're in the Ultimatum game. You're offered 0-10 dollars. You can accept or reject the offer. If you accept, you get what's offered, and the offerer gets $(10-offer). If you reject, both you and the offerer get nothing.

The simplest strategy that incentivizes fair splits is to accept everything ≥ 5 and reject everything < 5. The offerer can't do better than by offering you 5. If you accepted offers of 1, the offerer that knows this would always offer you 1 and get 9, instead of being incentivized to give you 5. Being unexploitable in the sense of incentivizing fair splits is a very important property that your strategy might have.

With the simplest strategy, if you're offered 5..10, you get 5..10; if you're offered 0..4, you get 0 in expectation.

Can you do better than that? What is a strategy that you could use that would get more than 0 in expectation if you're offered 1..4, while still being unexploitable (i.e., still incentivizing splits of at least 5)?

I encourage you to stop here and try to come up with a strategy before continuing.

The solution, explained by Yudkowsky in planecrash (children split 12 jellychips, so the offers are 0..12):

When the children return the next day, the older children tell them the correct solution to the original Ultimatum Game.

It goes like this:

When somebody offers you a 7:5 split, instead of the 6:6 split that would be fair, you should accept their offer with slightly less than 6/7 probability. Their expected value from offering you 7:5, in this case, is 7 * slightly less than 6/7, or slightly less than 6. This ensures they can't do any better by offering you an unfair split; but neither do you try to destroy all their expected value in retaliation. It could be an honest mistake, especially if the real situation is any more complicated than the original Ultimatum Game.

If they offer you 8:4, accept with probability slightly-more-less than 6/8, so they do even

...
Open Problems in AIXI Agent Foundations
by Cole Wyeth — 2024-09-12T15:38:59.007Z

Originally stand-alone, added to the AIXI agent foundations sequence, underlying research supported by the LTFF.

I believe that the theoretical foundations of the AIXI agent and variations are a surprisingly neglected and high leverage approach to agent foundations research. Though discussion of AIXI is pretty ubiquitous in A.I. safety spaces, underscoring AIXI's usefulness as a model of superintelligence, this is usually limited to poorly justified verbal claims about its behavior which are sometimes questionable or wrong. This includes, in my opinion, a serious exaggeration of AIXI's flaws. For instance, in a recent post I proposed a simple extension of AIXI off-policy that seems to solve the anvil problem in practice - in fact, in my opinion it has never been convincingly argued that the anvil problem would occur for an AIXI approximation. The perception that AIXI fails as an embedded agent seems to be one of the reasons it is often dismissed with a cursory link to some informal discussion.

However, I think AIXI research provides a more concrete and justified model of superintelligence than most subfields of agent foundations [1]. In particular, a Bayesian superintelligence must optimize some utility function using a rich prior, requiring at least structural similarity to AIXI. I think a precise understanding of how to represent this utility function may be a necessary part of any alignment scheme on pain of wireheading. And this will likely come down to understanding some variant of AIXI, at least if my central load bearing claim is true: The most direct route to understanding real superintelligent systems is by analyzing agents similar to AIXI. Though AIXI itself is not a perfect model of embedded superintelligence, it is perhaps the simplest member of a family of models rich enough to elucidate the necessary problems and exhibit the important structure. Just as the Riemann integral is an important precursor of Lebesgue integration, despite qualitative differences, it would make no sense to throw AIXI out and start anew without rigorously understanding the limits of the model. And there are already variants of AIXI that surpass some of those limits, such as the reflective version that can represent other agents as powerful as itself.

This matters because the theoretical underpinnings of AIXI are still very spotty and contain many tractable...
On the destruction of America’s best high school
by Chris_Leong — 2024-09-12T15:30:20.001Z
Link: https://scottaaronson.blog/?p=4979

This is a rare case when I think it's actually worth sharing an article that directly comments on the culture wars on Less Wrong:

I’d like you to feel about the impending destruction of Virginia’s Thomas Jefferson High School for Science and Technology, the same way you might’ve felt when the Taliban threatened to blow up the Bamyan Buddhas, and then days later actually did blow them up. Or the way you felt when human negligence caused wildfires that incinerated half the koalas in Australia, or turned the San Francisco skyline into an orange hellscape. For that matter, the same way most of us felt the day Trump was elected. I’d like you to feel in the bottom of your stomach the avoidability, and yet the finality, of the loss.
Optimising under arbitrarily many constraint equations
by dkl9 — 2024-09-12T14:59:28.475Z
Link: https://dkl9.net/essays/general_lagrange.html

Say we have a multivariate function to optimise, like f=x2+y2+z2, under some constraints, like g1=x2+y2−z and g2=y+z−1, both to equal zero.

The common method is that of Lagrange multipliers.

Add a variable λ for each constraint function — here, we'll use λ1 and λ2.
Declare the set of equations ∇f=λ1∇g1+λ2∇g2.
Bring in the equations g1=0 and g2=0 (etc, if there are more constraints).
Solve for λ and, more importantly, the inputs x, y, z.

Lagrange multipliers annoy me, insofar as they introduce extra variables. There is another way — arguably more direct, if perhaps more tedious in calculation and less often taught. I found it alone, tho surely someone else did first — probably Euler.

Lagrange, anyway

For the sake of a standard answer to check against, let's use Lagrange multipliers.

The gradient of x2+y2+z2 is [2x,2y,2z]. Likewise, ∇(x2+y2−z)=[2x,2y,−1], and ∇(y+z−1)=[0,1,1]. So step 2 gives these equations:

2x=2xλ1
2y=2yλ1+λ2
2z=−λ1+λ2

It readily follows that λ1=1 or x=0.

If λ1=1, then λ2=0, and z=−12. By the second constraint, y+z−1=0, find that y=32. By the first constraint, x2+y2−z=0, find that x2=−114, which is a contradiction for real inputs.

If x=0, then, by the first constraint, z=y2, and, by the second constraint, y2+y−1=0, so y=−1±√52 and z=3∓√52.

Determinants

With one constraint, the method of Lagrange multipliers reduces to ∇f=λ∇g. ∇f and ∇g are vectors, which differ by a scalar factor iff they point in the same (or directly opposite) directions iff (for three dimensions) the cross product ∇f×∇g=0 iff (for two dimensions) the two-by-two determinant |∇f∇g|=0.

With two constraints, the method asks when ∇f=λ∇g+μ∇h. That would mean ∇f is a linear combination of ∇g and ∇h, which it is iff ∇f, ∇g, and ∇h are all coplanar iff (for three dimensions) the three-by-three determinant |∇f∇g∇h|=0.

As it happens, the cross product is a wolf that can wear determinant's clothing. Just fill one column with basis vectors: ∇f×∇g=∣∣∇f∇g[^i^j^k]∣∣.

Likewise, with zero constraints, the "method of Lagrange multipliers" — really, the first-derivative test — asks when ∇f=0. Fill a three-by-thr...