skysight-inc/HackerNews huggingface dataset:
Ask HN: How do you version your data?
I'm working at a company that processes data through multiple distinct stages, and struggling to figure out what tooling to use for versioning and maintaining an auditable history of changes.

I'd be interested to hear about first hand experiences with how you version production data, use it safely during testing or experimentation, and maintain audit trails.
Gemini for coding does not work for me. It gets so many things wrong
I wouldn’t call either “state of the art”, it all depends on what you’re doing and what you want to achieve. Many compilers use several different IRs to achieve different things
We're having another kid, but yes, I'm going to need to find a way to force some socialization. I think the comments section has it right: regularity is the most important aspect.
Economic Injustice Is Anchoring Itself in the Arab World
It does not beat Claude Sonnet 3.5 on SWE Bench (42 to Claude's 50). It chooses 4 benchmarks of the 100s of available benchmarks and then decides it "beats" Claude Sonnet 3.5.
Potentially useful threads for your consideration.

Data Version Control - https://news.ycombinator.com/item?id=41888937 - Oct 2024 (52 comments)

Data Version Control - https://news.ycombinator.com/item?id=33047634 - Oct 2022 (59 comments)

Oxen.ai: Fast Unstructured Data Version Control - https://news.ycombinator.com/item?id=34831547 - Feb 2023 (63 comments)

Show HN: Oxen.ai – Fast Unstructured Data Version Control - https://news.ycombinator.com/item?id=34825056 - Feb 2023 (5 comments)

Ask HN: How do you version your data? - https://news.ycombinator.com/item?id=13683539 - Feb 2017 (55 comments)

With regards to tooling, https://github.com/pachyderm/pachyderm may satisfy this use case.
I've always assumed people are largely the same, socially. I.e. we enjoy being with people we mesh with, and don't enjoy being with people we don't mesh with. The thing is, you can like people you don't mesh with. You might not have a single bad thing to say of a person, yet still have no interest in hanging out with them.

And then, surprise, being forced to do things you don't enjoy is draining. Especially when it involves long periods of politely choosing your words and facial expressions to pretend you're not bored/frustrated.

The only thing that makes an "introvert" is when the genre of person they mesh with happens to be more numerically rare. There is a ton of variety in humans. It doesn't mean anything significant that your "group" is small.
Show HN: Eunomia, Open Source Data Governance for LLM-Based Applications
Like many of you, we've spent the last two years dealing with requests to sprinkle LLM-powered features everywhere.

One recurring problem we faced in almost every project was related to data governance. In most cases, it was extremely hard to implement the needed granular control over the retrieved data.

We developed custom solutions each time, but when we realized most of the solutions could be reused in subsequent projects, we started thinking about creating a modular framework.

Today we're releasing that framework! We've already built some modules using open-source solutions such as Microsoft Presidio (1) or custom fine-tuned models like our own Financial BERT (2), but the goal is to let you play with it and add the modules you need for your own use case.

Leave a star on GitHub if you find this problem interesting, join our community if you want to stay in the loop, and let us know what you think about Eunomia! We'd love to hear your feedback!

(1) https://github.com/microsoft/presidio
(2) https://huggingface.co/whataboutyou-ai/financial_bert
> evidenced by simultaneous price increases across jurisdictions with vastly different zoning laws

“Only in particular areas, especially New York City and California, do housing prices diverge substantially from the costs of new construction. The bulk of the evidence examined indicates that zoning and other land use controls are responsible for prices in high cost areas of the country” [1].

Inflation has raised construction costs; climate change the cost of homeownership. Beyond that, it’s almost all zoning.

> rapid price changes following interest rate shifts

Prices didn’t fall when the Fed spiked rates. The monetary-first hypothesis for housing requires epicycles to gain explanatory power over zoning.

Our housing crisis is a political choice.

[1] https://realestate.wharton.upenn.edu/working-papers/the-impa...
https://github.com/soutaro/rbs-inline

I myself am unsure where I stand on RBS. I wouldn't mind more use of it in my gem dependencies, but would probably not like it if it was enforced everywhere.

For now I'll stick with improving my test/spec-writing skills, and maybe some runtime type checking like https://literal.fun/
I would not underestimate the amount of resources necessary to support this brick-and-mortar stores model. You need to deliver those goods to each shop, then handle the returns of unsold items... Not to mention costs of employees (and their costs) — but that may be debatable, of course (creating jobs).
[dupe] https://news.ycombinator.com/item?id=42511267 | 191 points by bratao 16 hours ago
Apple doesn't give two shits about the web platform. Google only does because of cultural inertia.
I don't think that's true. Because the main way I learned those things was poking in the repl. Hitting tab and stuff. And reloading changes to files or overlays and debugging what happened by inspecting things.

It's the same as learning Haskell. Outside of syntax and some basics, you don't need to have deep knowledge to use ghci. And Nix and Haskell are both just substitution-based evaluation (lambda calculus) which imo is 80% of understanding them.
Stop following bad trends in 2025
I use mpd and ncmpcpp but arguably I don't need much in terms of features, I just play all the things on shuffle.

No point in using my phone as when I'm on the LAN, I don't need it, and when I'm out of it, it's counter productive to connect to it (because Canada still has third world mobile data rates) and better off replicating the entire library on-device.
Ask HN: Should I switch away from SQLite if I only use JSON fields?
I noticed that in 2024 I completely stopped using database tables with more than one column. Everything just goes into one single JSON column, which I always call "data".

So to query all entries from the table "cars", I do:

SELECT * FROM cars
WHERE data ->> '$.color' = 'blue';

That seems a bit redundant, considering all my tables have just the data column. So if the DB knew this, the query could be just

SELECT * FROM cars
WHERE color = 'blue';

Should I start looking for a database that works like this? Are there any databases like this out there? It would have to be a database that stores the data in a single file like SQLite. Because I would not want to give up that convenience.
Related discussion [https://news.ycombinator.com/item?id=42511267] 191 points by bratao 16 hours ago
How the average car went from £1000 to £29,000 in 75 years
Great Book. I presume you are reading the 3rd edition (all 64-bit). I suggest also getting a used copy of the 1st edition (all 32-bit) to compare and contrast. This is the only book that i have all three editions of :-)
> it's simple-minded to assume (and misleading to claim) that zoning is the cause of high prices in general

Why? Almost all the evidence points to this. And it makes intuitive sense. Take builder financing costs and add 2 years of reviews (super optimistic!) and you’ve added tens if not hundreds of thousands of dollars to the cost per unit.

> it may be a primary driver of rising prices in some areas

Value or population weight your data. Nobody is complaining about a housing shortage in rural New Mexico.
I see the value in backlinks (I assume you mean backlinks between said markdown files, right?), and graphview might come in handy once in a while...but am not seeing the value brought by databases here? Unless the db becomes part of a top/management layer for said files, which adds other things like maybe possibly faster search or easier query of content (that is, easier than find/grep, etc.)? Not knocking if DBs can be used, with all apologies, am genuinely curious how DBs could help here? (Context is that if this person is using simple markdown files in folders/repos, then they probably don't need/want more than that and typical file management tools)
Am I correct in seeing this as a custom language for a custom browser? That kind of makes this a bit more impressive but equally more challenging if nothing else to find other members to help with develop!
> Make high-demand areas less attractive and low-demand areas more attractive

Figure out how to sell Atherton on affordable housing and the Bronx on gentrification and you’ll have a solution.
Dark Ages : The idea that the decadence and "Fall of the Roman Empire" led to 1000 years of European social and cultural decline and stagnation, in which Christianity and feodalism have played a large part. It goes with "people only bathed once a year etc".

You might have seen a largely shared picture showing "Scientific Progress in the West" which shows "progress" vs time. It goes dramatically up during Antiquity, dramatically down between 400 and 1500 and exponentially up afterwards. Even without the ideologies on both sides it's easy to see that defining an entire millenium (and the Renaissabce) as underpinned by a single concept is simplistic.

It's a lot of teenage atheists vs people with alt-right-like views and right-wing Catholics.

Interesting is that a similar debate was already there at the time ("back when we were pagans we didn't have troubles with Germans pillaging Spain and Tunisia", vs the positive retelling of the Fall of Rome in a Christian light by iirc Augustine).

Another factor here is the lack of sources, making a lot of things hard to understand. Wars and political instability are one reason, but also for example the fact that a lot of less-important paperwork was written onto some kind of papyrus, which generally doesn't age well outside of dry deserts, contrary to the better, but much more expensive parchment.
Thanks for the clarification. Confusing terminology!
Aesthetics is a lot of it, but also the ability to build things like sidewalks or other public infrastructure. If people can build up to the property line, you’re going to have a hard time making many neighborhoods accessible or healthy without having to move structures.
Coding is very personal and fashion driven. Everyone's brain work different. It's all random events that forms our thinking. A software project is too big when you can no longer see the full picture - then you should break it down to independent parts. State is your enemy.
I'd like to get a deeper understanding of dynamic recompilation and implement a few PoCs based on different stages of techniques.
A friend of mine wanted to be an electrician, but once he realized he'd be working minimum wage in an apprenticeship for four years, he went back to school for a white collar job instead.

That's the real bottleneck.
Thank you! I shall play with it this weekend :D
Americans are exhausted by political news
Fair, but not strain in terms of storage space. More in terms of disk writes and how write intensive it could be, which can be a problem especially for SSDs.
If you're working within git, maybe `git notes` fit your use case? You can basically attach notes to various Git objects, without changing the objects themselves.

https://git-scm.com/docs/git-notes
> fact that there's almost nothing that overlaps with Christmas besides the timing

The drunken revelry?
its mafia racketeering thats increasing prices, lack of government oversight (due to them being the mafia) and algorithmic price fixing. youre welcome. all this talk about monetary policy, supply and demand, and interest rates is cute.
I have been programming since 1963 and I was a Master Software Engineer at Capital One before I retired. I have my own ways of managing cognitive load. I usually use Lisp languages, Python, and Haskell and I tend to write my own short libraries and then treat these libraries as black boxes unless I want to extend them. I obviously use a zillion libraries that other people write also, but there is a comfort and ease using my own code.
I recently made Safari my default browser for this reason. If Firefox’s days are numbered, I figured I’d get ahead of it than be forced. Ublock origin isn’t available for safari, but I was able to use the same content blocker that I use on iOS and it seems to be doing the job.
> Otherwise you can run these systems with a potato.

Crikey: This gave me a laugh like none other in a while. For anyone else who doesn't get the reference, you can build a very basic battery from a potato, e.g., https://stemgeneration.org/potato-power/

Now, I would love to see a YouTube video where someone tries to power a portable music player from a battery. Could a PiZero be done?
Seems fine here for me, Firefox on Android. It's not great for me on the desktop though, unless I use reader mode.
The Ardour DAW is kinda like this. Downloads of prebuilt images through the website are paid, but you can download and build yourself from source. (Most Linux users probably have free access via distro package manager anyway.)
Sounds like the source of truth for the enum members may have been in the database.

(But yes, exhaustiveness checking for sum types is a great feature.)
One of the things I like about HN is the civilised discourse and your reply is a great example of that.

I get what you mean re the unexpectedness of a fact and level of fun, too. So thanks for explaining -- makes sense and I understand your original comment better :)
This sounds like something the younger generation is having to relearn. When you power down a machine, leave it off for at least 30 seconds. One to let the various capacitors drain & discharge, and Two to let any network devices see that the machine is off so they update their internal tables.

You want your cold boot to truly be a cold boot.
Prices didn’t fall as much as a simple model would predict, but sales volume fell a lot. IMO, that was driven by sellers withdrawing from the market, because they weren’t willing (or perhaps able) to sell for what buyers were able to qualify for mortgages to pay.

https://tradingeconomics.com/united-states/existing-home-sal...

There is also a zoning and policy effect, of course, but it’s a mistake to think that mortgage rates do not inversely influence transaction prices.
Those who do are valued not for their programming skills but their knowledge of the (very complex, largely undocumented) system and its business environment.
Luckily, it isn’t.
You can sort of achieve that the other way around by starting an incognito session when you put on background music. Haven't worked out how to do that for things with Spotify integrations though.
It's not Cobol programmers that are rare, it's a shortage of people that can understand the business logic. Cobol is often used in very complicated business operations.
A lot of people are incentivized to keep these rules in place to keep their property values up. Artificial scarcity hurts those not in the game the most.
Sounds like an interesting website. Would you mind sharing the link? Asking for a friend.
I used codestream with two of my previous teams and absolutely loved it. I don’t remember if you can keep annotations private but I see plenty of value in allowing the rest of the team to see what questions/note you have. In any case, I believe they open sourced the whole thing so you could see how they handled code changes
I'm torn between: Learning systems (POMDP, RL, etc) for planning, Quantum computing basics, or jumping whole hog into the a learning-from-scratch type course track.

With RL+Planning, my goal is to eventually be able to tradeoff classical estimation and planning with end-to-end systems, and use the best parts of either for a better-than-either pipeline. At the jobs I've had I've seen a weird false dichotomy that produces decent results once committed, but there's always this tension that "the other way" is better, producing weird politics.

Most recently, a classical planning pipeline was completely broken by an ML-based estimation of orientation of obstacles. On inspection they were using quite possibly the worst estimator you could use, and ignoring all kinds of good data, because (I think) they were counting on perfect "measurements" from the ML-based vision system. That kind of thing must happen all the time - get stuck in a local minimum of "tune just a little better" when a good filter on top of a noisy estimator could solve all your problems. There's probably a million such examples in planning systems too.
Fine-tune classifier with ModernBERT in 2025
But then could you trust it not to hallucinate functionality that doesn't exist? Seems as risky as out-of-date comments, if not more

What I'd really like is an AI linter than noticed if you've changed some functionality referenced in a comment without updating that comment. Then, the worst-case scenario is that it doesn't notice, and we're back where we started.
I think it’s worse than a single generation because you need wages to be competitive for long enough that people are encouraging their children to try it. A good decade won’t do that, and especially not without improvements in our healthcare and retirement systems since physical trades are harder and the richer boomers are pushing to deal with the savings gap by raising the retirement age for everyone younger than them rather than restoring taxes to actuarily-necessary levels. Being a plumber at 60 isn’t for everyone.
If I remember correctly, other examples of OSS code with premium addons and license check, on the top of my head:
Bitwarden, Metabase, EmailEngine
But within nature there are examples of this kind of mimicry - i.e butterfly wings pretending to be predators eyes.
The Online Gambling Addiction
Construction cost may be a poisoned data source, does it take into account real estate purchasing costs? Or do individuals only build when they can afford a home at a price point greater than the 75th percentile home price in metros?
Syncthing and local files are a much better strategy. Internet access is just too unreliable even when its ubiquitous.
If it was as wide as the OP said then it means errors, panics, and especially Unsafe wouldn't exist. Even if we ignore unclean sources (say network errors/etc), this isn't a Proofed language where programs in it cannot possibly fail.

Besides, it likely is very possible to write programs that cannot fail in Rust. This usually means encoding state into the type system (enums included), but few go through that work. They know what corners they're cutting. Further proving that they know their program can fail.

Hell, Rust itself can fail. I struggle to imagine how this is perceived as some long-con from the Rust PR team to convince people Rust programs cannot be written with incorrect logic.
I don’t think I’ve ever seen a neighborhood that made any particular public accessible use of the area between the sidewalk and the buildings. There are occasional utility easements, but those don’t need much space. Some neighborhoods have no fences and let pedestrians walk between the houses. Other than that, setbacks seem to be mostly for aesthetics and front yards.
If you're on mobile, try switching the page to desktop mode.
Does this phenomenon have a name — things get progressively worse but people find ways to make do in the short term, and are relatively content with the state of affairs; then the ways to make do break under continued pressure, and people are worse off than if they didn’t make do and fight the worsening conditions initially?

In this case, it’s the enshittification of the internet — a technological wonder exploited, commodified and destroyed for commercial gain. A network built for science and then for human connection, turned into a cesspit of corporate spyware, addiction, and behavior manipulation. If we rejected this earlier, we would all be better off — less spied on, less addicted, less manipulated, probably more connected and happier. But we found ways to make do like ad blockers that have grown into essential privacy protectors. That will now gradually stop working.

But it’s a broader phenomenon. It can be seen in tech, healthcare, politics, economics climate change, corporate management — everywhere.

It’s like a proverbial cousin of the boiling frog. In this case, the frog is aware of slowly boiling water and makes itself content by temporary means. Then, of course, it boils, but rather suddenly and painfully compared to the usual boiling frog tale.
You should try again. Gemini rates highest on coding at lmarena.
There are some ways to abuse solely the ability to stop an inflight web request, and being able to see what url it was for.

But, that did require a specific permission.

And the permission/ability to inject arbitrary JavaScript into any page is still there. As are other abilities that can be abused.

Meaning, the security argument for removing blocking onBeforeRequest was always a diversion. It is not nearly the highest risk thing in the api.
Flarum is great [1].

Looks good, works on mobile, continuously updated.

Try it out.

Edit: Oh wow, downvoted for posting a good recommendation?

1: https://flarum.org/
An interview with David A. Mundie, creator of RxOL - the first programming language for recipes in 1985. He shares his vision of simplifying cooking through technology, his thoughts on recipe formats, and perspectives on the future of kitchen automation.
When people mean DBs in these note taking tools, they are usually referring to the Notion like databases: a lightweight Airtable-like table allowing a few data manipulations and analysis. Basically a CSV file with a nice UI on top if you will
I had access to PluralSight, so I started there. Then after I bought Apple Developer Account they have a lot of videos and tutorials for free.

But I've been developing software for a long time so it was pretty easy to start :)
Thank you ; it’s kind of you to write such a thoughtful, thorough reply.

In my original comment, when I wrote timezone, I actually didn’t really mean one of many known civil timezones (because it’s not), but I meant “timezone string configuration in Linux that will then give TAI time, ie stop adjusting it with timezones, daylight savings, or leap seconds”.

I hadn’t heard of the concept of timescale.

Personally i think item (3) is worthless for computer (as opposed to human facing) timekeeping.

Your explanation is very educational, thank you.

That said, you say it’s simple to get TAI, but that’s within a programming language. What we need is a way to explicitly specify the meaning of a time (timezone but also timescale, I’m learning), and that that interpretation is stored together with the timestamp.

I still don’t understand why a TZ=TAI would be so unreasonable or hard to implement as a shorthand for this desire..

I’m thinking particularly of it being attractive for logfiles and other long term data with time info in it.
It still fails my private physics testing question half the time, where claude 3.5 sonnet and openai o1 (both web version) most of the time passes. So I'd say close to SOTA but not quite. However given deekseek already has the r1 lite preview, and they can achieve comparable performance for much less compute (assuming the API cost of close models roughly represent the inference cost), then it's not unreasonable to believe deepseek may be close to release very good test compute scaling model that is similar to o3 high effort.
I use Funkwhale, which exposes a nice web UI for uploading new music. Saves me the hassle of manually adding music files to a server, and it also allows for a multi-user setup where users can keep some music to themselves.
But knowing what Brian was considering at the time is useful, both due avoiding redoing that and for realising that some constraints may have been lifted.
Apple is the only one standing between legacy Web and ChromeOS Development Platform.

Ironically, without them WebGPU wouldn't even have ever existed.

https://webkit.org/blog/7380/next-generation-3d-graphics-on-...