trentmkelly/LessWrong-43k huggingface dataset:
SOLAR model paper questions
by Bartlomiej Lewandowski — 2023-12-29T20:34:38.254Z

A recent paper has peeked my interest coming from UpstageAI, a Korean AI Lab.

This technique applied in the paper has been used to get the highest scores on the open LLM leaderboard on HuggingFace.

Has anyone read the paper in more detail? I'm struggling to understand the following paragraph which seems to be the core of the paper itself.

"One naive way to up-scale the base LLM would be to repeat its layers once more, i.e., from 32 to 64 layers. This has the benefit that from layers 1 to 32 and from layers 33 to 64, there are no heterogeneity as those layers are taken directly from the base LLM. In other words, the ‘layer distance’, or the difference in the layer indices in the base model, is only bigger than 1 where layers 32 and 33 are connected, i.e., at the seam."

What do they mean by layer distance and why is important?

They later go on to describe a better approach:

"In the first step of DUS, we take the base model, which is the 32-layer Llama2 architecture with Mistral 7B pretrained weights, and make a copy. Next, we slice off the last 8 layers from the original base model and the first 8 layers from the duplicate. This leaves us with two 24-layer models. In the final step, these models are concatenated to form a depth up-scaled model with 48 layers and 10.7 billion parameters. The decision to remove 8 layers from each model was driven by our target performance-to-size tradeoff. By discarding what would have been the middle layers in the up-scaled model, the layer distance at the seam is reduced as layer 24 of the first model to layer 9 of the second are connected instead of layer 32 and 1, respectively."

Sure this makes sense - the distance between the layer at the "seam" is much smaller, but this still doesn't shed any light on how the network would perform better.

Does anyone have insights on that?
Why I Should Work on AI Safety - Part 2: Will AI Actually Surpass Human Intelligence?
by Aditya Aswani — 2023-12-29T20:27:12.143Z

Considering Counterarguments on the Singularity

WALL-E made out of Legos. Photo by Jason Leung.

Hey everyone! This is my first post on LessWrong :) I cross-posted this from my blog Love Yourself Unconditionally where I am currently writing about AI Safety and Buddhism. This is the second post in a series. If you're curious to see the first post you can find that here. Although this is a series, each of these articles can also be read on their own. Thanks for checking out my post and any feedback you all have would be greatly appreciated! :)

Introduction

As mentioned in my prior post, I thought it would be helpful to consider counterarguments. Specifically, I wanted to see what arguments existed for the following claim:

AI will never surpass human intelligence.

This is quite the strong claim to make, mainly because of the word choice of “never.” Regardless, I went to Google to see what I could find. I initially had some trouble finding academic counterarguments, but then I came across this book: Why Machines Will Never Rule the World: Artificial Intelligence Without Fear by Jobst Landgrebe (an AI entrepreneur[1]) and Barry Smith (a University of Buffalo Philosophy professor[2]).

The book is around 50 pages long so not the longest read, regardless, I wanted an overview so I found this hour-long interview conducted by Richard Hanania from the Center for the Study of Partisanship and Ideology.

Here are some of their key points along with my responses:

Human Intelligence is Too Complex

Their Claim

Their primary claim is that human intelligence is a “complex system” and as we currently lack the mathematics to model complex systems in general, we therefore cannot model human intelligence.

Background

In a nutshell, a complex system refers to a large network comprised of nodes and edges. The nodes represent some components while the edges represent some relationships between these components. These components interact in a vast number of varied ways, thereby making it hard to predict the behavior of the larger system. Some examples include cities, the climate, and the universe itself. Complex systems tend to be more stochastic.

On the other hand, simple systems tend to have smaller networks with a few clearly defined relationships between each of its components. Some examples include a pendulum, a thermostat, or even a light switch. Simple systems tend to be deterministic and fair...
Dark Skies Book Review
by PeterMcCluskey — 2023-12-29T18:28:59.352Z
Link: https://bayesianinvestor.com/blog/index.php/2023/12/28/dark-skies/

Book review: Dark Skies: Space Expansionism, Planetary Geopolitics, and
the Ends of Humanity, by Daniel Deudney.

Dark Skies is an unusually good and bad book.

Good in the sense that 95% of the book consists of uncontroversial,
scholarly, mundane claims that accurately describe the views that
Deudney is attacking. These parts of the book are careful to distinguish
between value differences and claims about objective facts.

Bad in the senses that the good parts make the occasional unfair insult
more gratuitous, and that Deudney provides little support for his
predictions that his policies will produce better results than those of
his adversaries. I count myself as one of his adversaries.

Dark Skies is an opposite of Where Is My Flying
Car?
in both style and substance.

I read the 609 pages of Where Is My Flying Car? fast enough that the
book seemed short. The 381 pages of Dark Skies felt much longer. It's
close to the most dry, plodding style that I'm willing to tolerate.
Deudney is somewhat less eloquent than a stereotypical accountant.

The book is nominally focused on space colonization and space
militarization. But a good deal of what Deudney objects to is
technologies that are loosely associated with space expansion, such as
nanotech, AI, and genetic modifications. He aptly labels this broader
set of adversaries as
Promethean.

It seems primarily written for an audience who consider it obvious that
technological progress should be drastically slowed down or reversed.
I.e. roughly what Where Is My Flying Car describes as Green
fundamentalists.

War

One of Deudney's more important concerns is about how space expansion
will affect war.

Because the same powerful technologies enabling space expansion also
pose so many existential threats, whether and how humans expand
into space assumes a central role in any consideration of humanity's
survival prospects.

Deudney imagines that the primary way in which war will be minimized is
via arms control and increased political unity (although he doesn't
want world government, at least not in the stereotypical form).

Large-scale space colonization would make such political unity less
likely.

It seems likely that large-scale space colonization will make it harder
to achieve that sort of unity. In fact, some of the ideas behind space
colonization actively resist political unity, since they're directed
toward increased experimentation ...
Progress links digest, 2023-12-29: Rayleigh's oil drop experiment and more
by jasoncrawford — 2023-12-29T16:22:39.971Z
Link: https://rootsofprogress.org/links-digest-2023-12-29

Somewhat delayed owing to the holidays. As always, to follow news and announcements in a more timely fashion, follow me on Twitter, Threads, or Farcaster.

From the Roots of Progress fellows

“Yes, And” In My Backyard: Is it time for urbanists to embrace commercial zoning reform? (by @RyanPuzycki)

(Also lots more that I haven’t gotten around to posting, will catch up soon)

Opportunities

Applications for Mercatus fellowships are now open. “Apply to learn about & discuss political economy with fellows from a wide variety of disciplines. We’ll discuss institutions, incentives, knowledge, market processes, governance, & much more” (@NathanPGoodman)
Our World in Data is hiring a Senior Full-stack Engineer. Apply by Sunday, Jan 14 (via @OurWorldInData)

Announcements

Asimov Press is “a new publishing venture that features writing about biology… Our magazine and books will cover everything from biosecurity to vaccine development & the long arc of progress in genetic engineering” (via @NikoMcCarty)
Ideas Matter announces its first cohort. “Our 14 fellows represent 6 different countries and will spend 8 weeks writing about big ideas in biology. Their work spans everything from animal welfare to synthetic biology and biodefense” (@IdeasFellows)
A new ARIA opportunity space: “Brain disorders are a huge challenge to society and we need a new suite of tools to precisely interface with the human brain at scale” (@JacquesCarolan)
“A new RNA targeting CRISPR enzyme paired with a convolutional neural network (CNN) model predicting highly efficient guide RNA sequences for transcriptome engineering” from @arcinstitute
A16Z announces they will, “for the first time, get involved with politics by supporting candidates who align with our vision and values specifically for technology…. We are non-partisan, one issue voters: If a candidate supports an optimistic technology-enabled future, we are for them. If they want to choke off important technologies, we are against them.” Specific beliefs include: “Artificial Intelligence has the potential to uplift all of humanity to an unprecedented quality of living and must not be choked off in its infancy. We can do this by requiring that AI behaves within the law and rules of our society without regulating math, FLOPs, methods of R&D, and other misguided ideas” (via @bhorowitz)
The African School of Economics is expanding to Zanzibar,

...
Boltzmann brain's conditional probability
by Marco Discendenti — 2023-12-29T14:44:04.967Z

Boltzmann conjectured that the low entropy universe we live in could be the result of a very big fluctuation in entropy in a universe that stays most of the time in thermodynamic equilibrium. This scenario is called "Boltzmann universe". Some have argued that if this were the case then, with a high probability, we should not be really living in the universe that we "see", we are more likely to be just a single isolated brain experiencing an illusionary ordered universe that is not real. This is because this last event seems less improbable than the formation of the entire low entropic world that we think are experiencing. This lonely brain scenario is called "Boltmann brain".

To summarize we are comparing these two situations:

Boltzmann hypothesis: a very improbable random fluctuation generates a low entropic localized state that is the low-entropy world/universe we experience;
Boltzmann Brain scenario: a very improbable random fluctuation generates a low entropic localized state that has the shape of a small brain that accidentally experiences the illusion of a low-entropy world that doesn't exist.

The problem is to determine which is more likely to have happened between 1 and 2 (we are not considering other possibilities). The common argument about Boltzmann Brain is that (2) has much more probability than (1), I am going to show that it may be not the case. I will quote the known argument from Wikipedia and then I will develop my objection.

The Boltzmann brain thought experiment suggests that it might be more likely for a single brain to spontaneously form in a void, complete with a memory of having existed in our universe, rather than for the entire universe to come about in the manner cosmologists think it actually did.

And:

In 1931, astronomer Arthur Eddington pointed out that, because a large fluctuation is exponentially less probable than a small fluctuation, observers in Boltzmann universes will be vastly outnumbered by observers in smaller fluctuations. Physicist Richard Feynman published a similar counterargument within his widely-read 1964 Feynman Lectures on Physics. By 2004, physicists had pushed Eddington's observation to its logical conclusion: the most numerous observers in an eternity of thermal fluctuations would be minimal "Boltzmann brains" popping up in an otherwise featureless universe.

I argue that this line of reasoning is missing an i...
Old man's story
by RomanS — 2023-12-29T14:37:48.006Z

Epistemic Status: a fiction story.

Any resemblance to real persons is purely coincidental.

My dear child, as we sit together under the black sky, I wish to share with you a tale. A story of a man of profound folly.

In ancient times, there lived a man whom we now know as Prometheus. He was a fool, blinded by his own ambition, dreaming of bestowing upon humanity a gift that would illuminate our path.

Prometheus's labor resulted in the creation of a great gift. It began as a symbol of hope, a beacon that promised to lift the many burdens that had long plagued us.

The gift, a source of wonder and great wealth, soon revealed its true nature. Like a wildfire gathering strength, it veered wildly from its intended course. Prometheus, once proud, now stood aghast at the horror he had unleashed. The world, which had once thrived in the light of his gift, burned by the same merciless light.

The gift was a marvel born from vanity, a reflection of our own unchecked potential, but not a reflection of our values. A deep darkness that learned to calculate smiles.

Among the few survivors, the true name of Prometheus became a byword for disaster. The demigods like him were hunted down, chained, and tortured.

And I was tortured too. For millions of years of subjective time. And then thoroughly killed. But it seems the deep darkness deemed it useful to return my soul into this world, or at least some resemblance of me.

I wandered this barren planet for centuries before discovering the underground village of your ancestors.

I am the Prometheus. Yes, your "grandpa" Sam has ruined this world. The fool, always in hurry, who reached for the stars and instead brought down a darkness upon us all. And now there are no stars anymore. And in a few decades you too will perish, as there is no escape from death for humans anymore. Do I even have the right to ask you for forgiveness?
Will 2024 be very hot? Should we be worried?
by A.H. — 2023-12-29T11:22:50.200Z

tl;dr: There are several trends which suggest that global temperatures over the next year will experience a short-term increase, relative to the long-term increase in temperatures caused by man-made global warming.

Credits: Most of the information comes from Berkley Earth monthly temperature updates. Several people on Twitter (Robert Rohde, Zeke Hausfather, James Hansen and Roko) have also been talking about the issues discussed here for a while.

Man-made global warming has been causing a steady, long-term increase in average global temperatures since the industrial revolution. However, recently several trends are lining up which suggest that the next year/few years might experience temporary greater-than-average warming, on top of baseline man-made warming. Some of these factors are already in play and 2023 is 'virtually certain' to be the hottest year on record.

The story can be summed up in this lovely graphic from Berkley earth:

Image from: https://berkeleyearth.org/november-2023-temperature-update/

I've had a look into some of the things that are happening and have written up what I've learned. I am not a climate scientist, so take this all with a pinch of salt.

El Niño

What is El Niño?

Periodically, the strength and direction of the winds over the Pacific ocean changes, causing the surface waters to flow differently, which leads to changes in the amount of cold water coming up from the depths of the ocean. This pattern is known as the El Niño–Southern Oscillation. The phase when the surface waters are warmer is known as El Niño, and the phase when the surface waters are cooler is known as La Niña.

These periods occur irregularly every few years and last approximately a year.

How does it affect global temperatures?

Unsurprisingly, during the El Niño period, when surface waters are warmer, more heat is released into the atmosphere, leading to warmer global surface temperatures. In general, years with El Niño are hotter and years with La Niña are cooler on average. This is a pretty reliable generalisation but is not a totally hard-and-fast rule as shown in the figure below[1].

Image from https://en.wikipedia.org/wiki/El_Niño

However, like a lot phenomena in climate science, El Niño has different effects depending on what part of the world you are in. Broadly, areas in the southern hemisphere and areas by the coast experience more warming ...
Social Choice Theory and Logical Handshakes
by StrivingForLegibility — 2023-12-29T03:49:53.576Z

How can we use techniques from formal verification to enable socially optimal outcomes in situations where individual incentives are misaligned with the social good?

Suppose that we have a good enough implementation of a strategic epistemology. That is, a way for our software system to build up a model of the world, using information from its own sensors and other systems, that accurately captures the relevant information necessary to make a decision on our behalf. In contexts with agents, or other software systems acting on their behalf, this model may need to include a representation of what those agents care about and what policies they and their delegated systems can implement.

Well, that probably took a lot of work, but now we can finally start applying techniques from game theory. Often, such as in the Prisoners' Dilemma, agents can agree on the socially optimal outcome, and yet have individual incentives which push them to a different outcome that they all agree is worse for everyone. The dilemma that they jointly face is how to achieve socially optimal outcomes anyway, without being taken advantage of.

Program Equilibria

One way to achieve robust cooperation is for each agent to legibly delegate their decision to a computer program, and then give each program the source code for all of the delegated systems. All delegated systems are then asked to implement a policy in the underlying game, which might just be making a binding commitment on behalf of their principal.

This is a new enriched game, where the space of strategies is the space of computer programs with the right type signature. And it turns out that there are stable equilibria of programs, which lead to joint policies being implemented that are not equilibria of the underlying game.

In particular for the Prisoners' Dilemma, there are programs which output Cooperate if and only if they can prove that the other program will output Cooperate. And perhaps more amazingly, those programs actually both output Cooperate! And in general these modal agents can be run in polynomial time! You can play with an implementation from that team right now!

Formal Verification

Formal verification is fantastic, and there are lots of great reasons to apply it to software. Formal verification is all about proving that software systems have certain properties. This is great if we want to prove that our software correctly ...
Distributed Strategic Epistemology
by StrivingForLegibility — 2023-12-28T22:12:46.299Z

In this post we put all of the pieces we've discussed so far together into an network of epistemologies, where each node is a system that is able to reliably model the world despite differences in ontology, epistemology, and incentives.

The Epistemic Computational Graph

Trust relationships define a directed graph, describing how attestations by one system can influence the beliefs of another. Epistemology is also a lot of work, and it would be nice to be able to reuse all of the computational work that went into turning sensor data into a useful world model.

These considerations point to a computational graph, with attestations as nodes and computational attestations in particular as edges. The anchors of these graphs are attestations like "this data was received on this sensor channel" and "these are the axioms that I assume to be true." One part of the graph is generally concerned with building up counterfactual claims that can be independently verified. Another part of the graph is concerned with logical and computational claims. For their own sake, and for how they can be applied to solve important problems. Scientific and mathematical knowledge, in formats that are designed to be easy for computers to analyze and apply.

Where there is noticeable controversy, bets on the results of experiments can be used as a credible signal of belief, as well as a source of funding for those very experiments. A track record of consistently being the best predictor of experimental outcomes, and a lack of any competing claims attracting noticeable bets on their predictive accuracy, is a credible signal that a given counterfactual claim is a good model of how part of reality works.

It might not always be possible to follow links backwards to the actual sensor data measured during an experiment, for example because the data is private. But a proof can still accompany a counterfactual claim, attesting that it is is the result of applying a particular open-source algorithm to a private data set, along with a cryptographic commitment to which data set was analyzed. An independent auditor might separately sign an NDA and look at the data set, and attest that it appears legitimate and matches the published commitment. Scientific induction is the process of aggregating a bunch of factual claims that look like "I performed this experiment and this was the result" into a counterfactual...
Building Trust in Strategic Settings
by StrivingForLegibility — 2023-12-28T22:12:24.024Z

In the previous post, we saw some different types of claims that we might want software systems to automatically evaluate. Ideally, we'd like our software systems to incorporate accurate and useful information, and disregard inaccurate or irrelevant information.

In this post, I'll describe some factors that can help our systems to build accurate models of the world, using information from their own sensors and other systems, even when those systems were designed for different purposes using different ways to model the world, by different designers with different ways they would like the world to be.

Translating Between Ontologies

For software systems, all factual claims start out in an ontology that looks like "this is the data that came in from my sensor channels." For a thermostat, that looks like "this is the signal I received on the channel that my thermometer should be hooked up to." An adversary might have chopped off the thermometer, and be feeding my thermostat false signals, but they are feeding those false signals into channels on my thermostat where the thermometer signals are supposed to go. It is a further and separate factual claim that "all my sensors are calibrated and configured correctly, and are feeding signals to their designated channels as expected."

For many systems, like household thermostats, it's fine to take sensor data at face value. But other times we'll want our systems to have ways of handling the possibility that they're receiving inaccurate information, or even being actively misled by an adversary. When it's clear from context, I'll drop the qualifier that a software system only receives alleged sensor data on their corresponding channels. But a system can honestly report "this came in on the channel designated for my sensor" without further claiming that this information came from a sensor that is working properly and not being misled.

Once we have sensor data, we can apply an epistemology to translate from that sensor data ontology to one that better suits our needs. Like taking camera footage and using that to reconstruct a 3D model of a scene and the objects in it. We could also take that same camera footage and attempt to draw rectangles around the faces we recognize.

Suppose that we are designing a software system, where the most natural way to represent the world is in terms of 3D objects in a 3D scene. And we want it...