skysight-inc/HackerNews huggingface dataset:
I have written a ton of evaluations and run countless benchmarks and I'm not even close to convinced that we're at

> the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”

so much as we're over-fitting these bench marks (and in many cases fishing for a particular way of measuring the results that looks more impressive).

While it's great that the LLM community has so many benchmarks and cares about attempting to measure performance, these benchmarks are becoming an increasingly poor signal.

> This is a nerve-wracking time to be a knowledge worker for sure.

It might because I'm in this space, but I personally feel like this is the best time to working in tech. LLMs still are awful at things requiring true expertise while increasingly replacing the need for mediocre programmers and dilettantes. I'm increasingly seeing the quality of the technical people I'm working with going up. After years of being stuck in rooms with leetcode grinding TC chasers, it's very refreshing.
> People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark

This seems like a bold statement considering we have so few benchmarks, and so many of them are poorly put together.
Yeah it became a rocket science with high barrier of entry. Like DevOps.
But I'd argue two points here. First, the average American household spends approximately $5k a year on gasoline [0]. That's still quite a lot of gasoline burned up. Add to that another ~$1k of natural gas the average household is burning up every year. I don't really call that small consumption. On a worldwide scale, each of these American households is burning up way more fossil fuels than the bottom 50%+. THEY ARE the rich of the world. But my second point is that switching to an EV and heat pumps (HVAC, dryers, water heaters) and induction (or electric) stoves are actually an improvement over their fossil fuel burning counterparts, either by ROI, function, or most likely, both.

I'm not suggesting anyone drive less even. Just switch to superior electrified products (and support renewable electricity+batteries as appropriate).

[0] https://www.jdpower.com/cars/shopping-guides/how-much-do-peo....
How long does it take the operator to sift through those 10,000 attempts to find the successful one, when it's not a contrived benchmark where the desired answer is already known ahead of time? LLMs generally don't know when they've failed, they just barrel forwards and leave the user to filter out the junk responses.
I am not up-to-speed on CoT side but is this similar to how perplexity does it ie.

- generate a plan
- execute the steps in plan (search internet, program this part, see if it is compilable)

each step is a separate gpt inference with added context from previous steps.

is O1 same? or does it do all this in a single inference run?
We just increased our server resources because a lot of questions are getting asked (thank you!). Hoping to keep the wait times manageable
If you mean STS-51-B, the commander duct taped the hatch. After that flight a padlock was added for a more permanent solution.

> “I remember waking up at the beginning of a shift and seeing duct tape on the hatch," Gregory told Ars. "I did not know what the origin of it was, and I didn’t pay any attention to it. I may have, but I don’t recall asking Overmyer about it.”

Also this detail gives me pause:

> The Space Shuttle has been retired for 13 years, but the padlock remains in the fabric of US spaceflight with Crew Dragon. A commander's lock is an __option__ for NASA's crews flying to the International Space Station on Crew Dragon, as well as private missions.

Why even make it optional? Seems better to officially take responsibility out of the commander's hands and make the lock mandatory. This would greatly reduce risk to onboard trust dynamics, since the commander is just following orders instead of signaling mistrust toward the crew.

This seems like one of those "who needs all those lifeboats??" situations. Why not avert the Titanic now?
There are people who are trying to strip away that complexity. FastHTML and HTMX are showing the direction of where the wind is blowing in my opinion.

Now the fact that it looks and feels almost like a new flavor of JSP is another matter. I think the HATEOAS model was tried tested and true and just needed a bit more refinement rather than the total overhaul that SPAs+JSON APIs brought in.
The GlassFish server is rolling forward again. What happened?
The tanking is more closely aligned with new tax rules that went to effect that make it much harder to claim dev time as an expense.
You can use it, I just tried a few minutes ago. It's apparently limited to 30 messages/week, though.
Finally, a Claude competitor!
The Turing Test (which involves fooling a human into thinking they are talking to another human rather than a computer) has been routinely passed by very rudimentary "AI" since as early as 1991. It has no relevance today.
This is the sort of reasoning needed to solve the ARC AGI benchmark.
Many people use DC-DC to down-regulate a 48 volt battery system to 12-volt for their RV where they remove the original battery pack and replace it with Lithium. Not everyone does that, some use a 12-volt lithium system. Less 4/0 cables needed when adding inverters and other fun stuff to the RV.
This is a common side effect. Try the extended-release form, i.e. Glucophage XR.
I realize I didn't mention monad transformers at all in my original post, I only linked to them!

I should have mentioned that, as you say, monad transformers should be the default effect system choice for 99% of people
Have you tried Claude 3.5 Sonnet?
I am perplexed by SSR being lumped in with the rest of your examples as an approach that increases complexity.
Frank conversation about entrepreneurship [video]
You wouldn't do that to this model. It finds its own mistakes and corrects them as it is thinking through things.
Why can't you just omit all modern content related to physics? There's lots of training data that doesn't mention any of Einstein's work.
Maybe if you ignore the vast nuclear arsenals that various countries have aimed at each other at all times..
SAP Completes Acquisition of WalkMe
> Max. discharge current vs. time: 30A-40A > 44s

I figure that might be workable for a few power cycles and a few big solder joints, but it would probably be a frustrating experience for anything more than quick fixes in the field.
I like your phrasing - "any task limited in scope enough to be a 'benchmark'". Exactly! This is the real gap with LLMs, and will continue to be an issue with o1 -- sure, if you can write down all of the relevant context information you need to perform some computation, LLMs should be able to do it. In other words, LLMs are calculators!

I'm not especially nerve-wracked about being a knowledge worker, because my day-to-day doesn't consist of being handed a detailed specification of exactly what is required, and then me 'computing' it. Although this does sound a lot like what a product manager does!
What's new in F# 9 – Languages and Runtime Community Standup [video]
Any general-purpose JVM is already going to have JNI entry points which bypass privacy. They can certainly restrict JNI as a whole and make users jump through hoops to enable it, but they're not going to be tearing it out for a long, long time.
Well, article title implies single battery (and not parts of it being replaced regularly) that lasts lifetime. With current technology the only way to achieve that it it to make it explode rather violently. In all seriousness, the issue with that battery's approach is only if it can fit any specific bike, otherwise idea is good since 18650 batteries are highy standardized. except new chemistry right around corner, and new chemistry means different voltages etc
yeah was wondering what 4o with a CoT in the prompt would look like.
Yes, and note the large price increase
Familiarize yourself with a tool which does half the prompting for you, e.g. cursor is pretty good at prompting claude 3.5 and it really does make code edits 10x faster (I'm not even talking about the fancy stuff about generating apps in 5 mins - just plain old edits.)
Like taxes, I'm a fan of proportional fines, at least for major offenses. The purpose of fine is to deter the perpetrator. If you fine a (m|b)illionaire 20 quid like the rest of us, how much of a deterrence can that be? I'm also a fan of increasing the fine exponentially for frequent and/or repeated offenses.
@Dang, @Paul Graham

This is where it really helps when people who can affect state budgets can really help. I am guessing the Governor of Texas would like more Tech investment. This would be a great time to speak up and tell him that killing innocent people for political gain might keep some ethical people away. Sorry for my rant, but this kind of thing is so surreal it shakes me.
Interesting sequence from the Cipher CoT:

Third pair: 'dn' to 'i'

'd'=4, 'n'=14

Sum:4+14=18

Average:18/2=9

9 corresponds to 'i'(9='i')

But 'i' is 9, so that seems off by 1.

So perhaps we need to think carefully about letters.

Wait, 18/2=9, 9 corresponds to 'I'

So this works.

-----

This looks like recovery from a hallucination. Is it realistic to expect CoT to be able to recover from hallucinations this quickly?
I think their tools are overhyped - not worth the price, you pay for the brand they have built by basically PR (repair scores for iPhones).
Use Python or JS. The models definitely don't seem to perform as well on less hyper prevalent languages.
> And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowledge worker for sure.

If you have to keep checking the result of an LLM, you do not trust it enough to give you the correct answer.

Thus, having to 'prompt' hundreds of times for the answer you believe is correct over something that claims to be smart - which is why it can confidently convince others that its answer is correct (even when it can be totally erroneous).

I bet if Google DeepMind announced the exact same product, you would equally be as skeptical with its cherry-picked results.
> Unless otherwise specified, we evaluated o1 on the maximal test-time compute setting.

Maximal test time is the maximum amount of time spent doing the “Chain of Thought” “reasoning”. So that’s what these results are based on.

The caveat is that in the graphs they show that for each increase in test-time performance, the (wall) time / compute goes up exponentially.

So there is a potentially interesting play here. They can honestly boast these amazing results (it’s the same model after all) yet the actual product may have a lower order of magnitude of “test-time” and not be as good.
A near perfect on AMC 12, 1900 CodeForces ELO, and silver medal IOI competitor. In two years, we'll have models that could easily win IMO and IOI. This is __incredible__!!
Yep, all these AI announcements from big companies feel like promises for the future rather than immediate solutions. I miss the days when you could actually use a product right after it was announced, instead of waiting for some indefinite "coming soon."
Adaptive Tree Structures Tackle Blockchain Scalability Issues
My first interpretation of this is that it's jazzed-up Chain-Of-Thought. The results look pretty promising, but i'm most interested in this:

> Therefore, after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users.

Mentioning competitive advantage here signals to me that OpenAI believes there moat is evaporating. Past the business context, my gut reaction is this negatively impacts model usability, but i'm having a hard time putting my finger on why.
Many of us nowadays don't carry keychains. I have keypad locks on my home and with a Tesla, no need for any keys. I love not having that extra clunky thing in my pocket!
I'm skeptical because "we fired half our programmers and our new AI does their jobs as well as they did" is a story that would tear through the Silicon Valley rumor mill. To my knowledge, this has not happened (yet).
Honestly I think every year that goes by in which the entire market is not consumed by F2P, battle royale, microtransaction-ridden shovelware, we're lucky for whatever we get left.

Though perhaps I'm just a bit cynical.
THAT'S why I hammer the point he is being a useful idiot.

He is doing their work and trying to force regulations that are good for that terrible end state. He's working so that government help it, not put some controls, which we already have today btw
No model is perfect, the less I can see into what it’s “thinking” the less productively I can use it. So much for interpretability.
Do you have plans to have an actual local application to do this? Chrome-only web tools are not sustainable and a deal breaker.
It's been less than 10 years since they had an all time high for summer sea ice extent.
I think that Memristors are perfect for use as configuration RAM for FPGAs and FPGA-like things. Something that you want to be able to update, but not frequently, and read all the time.

Of course, then the question becomes one of refreshing their state, like DRAM.
Well, until the cops start actually serving the public rather than being (again) security theatre, I stand by my skepticism. I understand other countries have much less organized-crime-like police cultures.
The problem is one often follows the other. If you're economically incentivized not to rethink your premises despite mounting evidence that they are wrong, it's hard to pivot.
I agree, if you keep the heavenly filament model you should probably restrict the view point origin to the center to have at least have one plausibly correct interpretation.
This is a Mozilla $6B+ wasted money problem.
That's unfortunate. When an LLM makes a mistake it's very helpful to read the CoT and see what went wrong (input error/instruction error/random shit)
(Sorry for the delay, I haven't checked HN is a while and just noticed this)

So some proxies (forward and reverse) will strip this header for different reasons.

The reverse proxies sometimes strip it to improve caching and/or because they were configured to have a allow-list of headers and this (being a less common one) was excluded.

The forward proxies will sometimes strip it for privacy reasons (it can be used in fingerprinting) or for the same reason (they have a list of allowed headers and this isn't one).
> A prime example is the case of Robert Fortune, a Scottish botanist, who in 1848 undertook a covert mission for the British East India Company.

For anyone else reading, I'll mention that I found this book extremely engaging:

Rose, Sarah. For All the Tea in China: Espionage, Empire, and the Secret Formula for the World's Favourite Drink.
Apologize is perhaps the wrong word, thank you.

But has he come out and said "this new evidence means my older theories are wrong?" The other comment about his review of Questioning Collapse makes me think not, but even that was from nearly 15 years ago.

I think it's important that Diamond acknowledge the new reality, as he is probably the person most responsible for spreading an idea (one that has a lot of implications for the present day real world) that turned out to be wrong.
What's the statement that OpenAI are making today which you think they're violating? There very well could be one and if there is, it would make sense to talk about it.

But arguments like "you wrote $x in a blog post when you founded your company" or "this is what the word in your name means" are infantile.
Saying "competitive advantage" so directly is surprising.

There must be some magic sauce here for guiding LLMs which boosts performance. They must think inspecting a reasonable number of chains would allow others to replicate it.

They call GPT 4 a model. But we don't know if it's really a system that builds in a ton of best practices and secret tactics: prompt expansion, guided CoT, etc. Dalle was transparent that it automated re-generating the prompts, adding missing details prior to generation. This and a lot more could all be running under the hood here.
I think what I’m saying is a bit more nuanced than that. LLMs currently struggle with very “wide”, long-run reasoning tasks (e.g., the evolution over time of a million-line codebase). That isn’t because they are secretly stupid and their capabilities are all hype, it’s just that this technology currently has a different balance of strengths and weaknesses than human intelligence, which tends to more smoothly extrapolate to longer-horizon tasks.

We are seeing steady improvement on long-run tasks (SWE-Bench being one example) and much more improvement on shorter, more well-defined tasks. The latter capabilities aren’t “hype” or just for show, there really is productive work like that to be done in the world! It’s just not everything, yet.
>https://www.frontiersin.org/journals/endocrinology/articles/...

Data is controversial. It's not "not enough" data or anything like that either.
>I feel that you desire agency on both parties.

That is such an artificial way of talking and it is making me not want to talk to you. You are actively cutting of communicating with me if you keep talking to me this way.

Now you will blame me, and not yourself.

I have lived all my life, cooperated with people of all kinds. Never used NVC.
in general, I am in favor of language features that make it easy to prove that common errors will not happen. Conversely, I am against language features that make it easy to make new classes of errors that are hard to reason about.

Haskell manages to do a lot of both. The kinds of problems I ran into in my Haskell error were much, much weirder than the problems I run into in other environments - things that when I explain them to other programmers they often don't even believe me.

On balance, for me, the new problems were worse than the old problems, but your mileage may vary.
I was looking into the pattern matching example in the article with `Either` type. If we need to unwrap and check for all the cases one by one would it become a callback hell?

I was going through a Scala codebase at work that uses `Future`s and `map`ing and `flatMap`ing them. Sometimes the callbacks went 5-6 levels deep. Is there a way to "linearlize" such code?

I come from JS/TS background and have not much experience with pufe functional languages. But I love how TS handles discriminated unions - if we handle a branch and `return` early, that branch is removed from the union for the subsequent scope, and I was wondering if something of that sort can be achieved in Haskell/Scala.
Here you go: https://www.simplypsychology.org/emotional-labor.html

> Hochschild (1983) suggested that jobs requiring more emotional labor are performed primarily by women. These jobs typically involve creating feelings of well-being or affirmation in others – responsibilities usually assigned to women.

Hochschild, A. (1983). 1983 The managed heart. Berkeley: University of California Press.
José Valim nails it: Remix's concurrent submissions have a fundamental flaw. Without causal ordering, you're stuck with race conditions and stale data, leading to poor UX. They need to rethink their approach or embrace causal ordering to fix this.
Generally yes but the languages are similar enough that the refactoring could be automated. Of course, that would create a fair amount of QA and testing work but if you have good test coverage you can mitigate that.
Like everything, the devil's in the details.

In our case, non-idle cumulative thread CPU time (including system activity) is a nice metric. If it exceeds 70%, start sending some 503s (or 500s depending on the client). A single core can easily service 100B requests per day, a few times more than we needed to, so if we exceeded that by enough margin to saturate every core to 70% then the right course of action was some combination of (1) scale up, and (2) notify engineering that a moderately important design assumption was violated. We could still service 100B requests per day per core, but the excess resulted in a few 5xx errors for a minute while autoscaling kicked in. It's not good enough for every service, but it was absolutely what we needed and leaps and bounds better than what we had before, with a side-benefit of being "obviously" correct (simple code causes fewer unexpected bugs).

The core idea (in tweaking the numerator in exponential smoothing) isn't any easily refutable definition of CPU time though. It's that you're able to tailor the algorithm to your individual needs. A related metric is just the wall-clock time from the beginning of a request being queued till when it was serviced (depending on your needs, maybe compared to the actual processing time of said request). If you ever get to total_time >= 2 * processing_time, that likely indicates a queue about to exceed a critical threshold.

> how many other requests are running or waiting to run

I referenced that idea pretty explicitly: "If you have more control over the low-level networking then queue depth or something might be more applicable, but you work with what you have."

Yes, that matters. That doesn't make other approaches invalid. Even then, you usually want a smoothed version of that stat for automated decision-making.
I don’t think HATEOAS is harmful but acronyms like it contribute to the feeling of overwhelm.
No, this is just how Microsoft names things.
> I remember in school and in old textbooks hearing the question, “What happened to the Mayan people? They totally disappeared!” and at the same time reading that there are dozens of Mayan languages spoken in the Yucatán to this day.

These aren't technically incompatible; for a better-understood example, you can look at Turkey, which speaks the language of the long since dead Turks that conquered it without leaving any demographic impact. It would be accurate to say that the Turks completely disappeared from Turkey, at the same time that the region continues to speak Turkic languages.

I think an analogous situation holds in Hungary.
Thanks. We get that question a lot. I think currently it distinguishes;

- Shuttle allows you to make well designed sharing pages in your branding (wallpaper, logo). Looking to extend those customization features further.
- Share pages can also be used to request files from others without them to require an account.
- We allow you to connect your own storage account (Backblaze B2 or Amazon S3) turning it into a truly pay as you go storage
- It's generally cheaper than Dropbox

Obviously Shuttle is just getting started and Dropbox is a big company that's been there for a long time. Instead of competing with Dropbox, we're currently looking into optimizing for niche like photography.
Surprising that at run time it needs an exponential increase in thinking to achieved a linear increase in output quality. I suppose it's due to diminishing returns to adding more and more thought.
Out of curiosity, could you describe the data set here?
https://haskell-beam.github.io/beam/ is fantastic, but good luck understanding it if you don't already know some Haskell
> Available starting 9.12

I don't see it
Incredible results. This is actually groundbreaking assuming that they followed proper testing procedures here and didn't let test data leak into the training set.
Does that mean human intelligence is cheapened when you talk out a problem to yourself? Or when you write down steps solving a problem?

It's the exact same thing here.
Pretty much.

With both more time and vastly more money than Sierra, BO has yet to reach orbit with anything. Medium lift rockets are not particularly scarce...vs. there's only one decent non-Russian, non-Chinese manned orbital vehicle. And it took even SpaceX almost a decade to go from Cargo Dragon to Crew Dragon.