skysight-inc/HackerNews huggingface dataset:
Ask HN: How do you track configurations across a machine learning pipeline?
Suppose I have an ML pipeline that generally consists of:

1. Some initial ETL to get most of the building-of-the-analytic-dataset out of the way, say in SQL.

2. The rest of the ETL, which mainly consists of creating the feature matrices to the exact specifications I need. This includes, but is not limited to: imputation of missing values, applying the rule of n for categorical predictors, raw-text-to-embedding cleaning and transformations, standardizing/centering, etc.

3. A retraining script which consists of: loading the "assets" saved from step 2, performing hyperparameter optimization, cross-validating, and saving the best model, according to some performance metric.

4. A CI/CD script to deploy the latest model on K8S.

There are certain configurations that are useful to track throughout this pipeline, especially since I'm trying to minimize CI/CD friction. For example, if I choose to impute value v for column X, not only do I need to know that in my retraining process, but I also need to know that in my scoring REST API in order to properly transform input. My goal is to be able to quickly define new ways of transforming input and test them in my core retraining framework (e.g. using simple count embeddings versus TF-IDF for text mining).

Here is the main way I've thought about doing this - I'm curious to see what you all prefer personally:

A. Have a master config JSON file that is loaded in each step of the pipeline and changed depending on what command-line arguments were passed (e.g. `--embedding_type count` versus `--embedding_type tfidf`.

Apologies if this problem isn't specific to machine learning, but just generally to SWE.

Edit: I suppose another way of doing this, without explicitly have configuration files, would be to go all-in on scikit-learn (or a similar tool) with something like its `Pipeline` object.
meh. just look at "Amazon dark patterns" : https://news.ycombinator.com/item?id=17651488
If you want to learn more, the biological process is called “autophagy”.
C++ has a whole scripting language embedded into itself. They're trying to reinvent lisp, of course a much less efficient lisp.
Quick calculus :

TLDR : 100+ billions is a hard reach today, but 50+ billions might not impossible

There is approximately 63,824,448 km² or liveable land on earth today.[1]

Let's cut that in half to account for farms/land lost due to global warming/industries/etc... We get 31,912,224km² of land usable for habitation.

Let's assume we cover the remaining with a huge city with the density of Hong Kong (6,690 people/km²). This is for the whole city, not the most populated quarters.

We arrive at a population of 213,492,778,560 people, so twice as much as the parent's estimate, with half of the remaining liveable land usable for farms/industries/wildlife parcs/inefficiencies of repartition...

Would that be enough ? According to a Sweden study [2], we use 1.6 billion hectares (16,000,000 km²) of farmland today, for approximately 7 billions people. It means that for 200 billions, we need 486,857,142 km². That's a huge increase, and currently the Earth can only sustain 44,000,000km² (same source as earlier, FAO), so an order of magnitude lower.
According to a study on the effect of diets on land [3], switching to a vegetarian/vegan (not much change in the result) diet allows to cut the land use by ~4.
So we're left with 121,714,285 km² of required land : a bit better, but we still fall way short. That's going to be our actual bottleneck.

Let's cut population by 3, and we arrive at ~40,000,000 km² of required land for 66 billions people.

It would be a pretty grim dystopia, that's for sure.
Most of nature would be erased, there would be a lot of natural catastrophes, and clearly the Earth would be way less welcoming than it is now.

But 66 billions people on Earth, given 'sufficient' technological (and most notably political) advancements is not completely out of whack, especially if we manage to produce clean energy (or at least clean on Earth) and to radically improve our crops (with genetic engineering, hydroponic culture, exploitation of oceans maybe ?)

Bottom line : this is napkin calculation, a lot of factors have been left out (water to give a main one)

[1] : https://www.quora.com/If-every-person-on-earth-was-to-be-giv... (I know, Quora is not the most reputable source, but it seems decently sourced)

[2] : http://www.fao.org/fileadmin/user_upload/newsroom/docs/en-so...

[3] : https://www.elementascience.org/articles/10.12952/journal.el...
How to Design Programs
https://htdp.org/
As an avid death metal fan, I think that overall this article isn't wrong, but there's much more to be learned from the subjective experience of violent music. Which is to say: just ask a fan, and they'll tell you.

Using "negative" emotions to evoke a positive response is way, way older than metal. It's a technique as old as humanity itself, and is present the oldest stories we have (e.g, Homer, Gilgamesh and Beowulf.)

Storytelling and emotions are not straightforward things and the emotional response we get is not linearly tied to the subject matter.

To suggest that a metalhead must be negative and violent because they listen to Cannibal Corpse is tantamount to saying that someone must be suicidal because they enjoy a good performance of Hamlet.
"Dad when are you going to remortgage the house?"

"What? You're five years old, how do you know about mortgages"

"Well.. I was just thinking... Interest rates have never been lower"
Trump plans to end giving citizenship to children of unauthorized immigrants
To put this in perspective, the DJI Phantom 3 mostly passes: https://reviewmeta.com/amazon/B013U0F6EQ
> Lots of negativity here, as is unfortunately typical at HN far too often these days.

Well, people are not bullish about this website, so they're expressing their views (which happen to be negative). Check out the 'Show HN' posts, there's a ton of positive feedback on many projects.

Maybe HN crew can somtimes be aggressive and opinionated, especially when it touches the economy and politics, but Show HNs and product reviews in general are the most honest you'll get on the net. This part works pretty damn well.
More nematodes and earthworms then. Both compete seriously for the crown of higher biomass on terrestrial invertebrates. Is said in many textbooks of ecology that there are much more Kg of tiny nematodes in an area of forest than Kg of deer.

And much more rodents also, of course.

We could assume that more or less a half of the biomass of each tree in a temperate forest is under the surface. Number of animals is not a good measure to explain the current situation.
>After reading that line, I would expect a somewhat federated blogging platform where you can set up your own server, and it joins the 'network of independent microblogs'.

This is exactly what I was hoping for and now I'm super disappointed.
I think this is good. Not a physicist but a lot of the SI definitions seem contrived and arbitrary [1] rather than based on physical fundamentals.

[1] https://en.wikipedia.org/wiki/SI_base_unit
Download free noise rock albums in HQ mp3 format, stream online or buy on CD/vin
Disagree, the embedded world is still firmly routed in C and this world is larger than you realise.

Consider that every electronic device you have has multiple microcontrollers inside running C code.

I would argue that if you want to get your code in the most peoples hands, you have to know C. Just looking around the (engineering) office I'm in at the devices using C:

- My PC.

- The router(s)

- The printer(s)

- the USB hubs.

- the mobiles and tablets

- the lights (yes).

- the A/C.

- The door entry system

- the microwave

- the desk phones.

- the monitors on my desk.

- The dishwasher

- The coffee machine

- The clock

The C world is much larger than most web people realise.

(the list goes on and on...)
Ask HN: How to get stateful behavior in a stateless distributed environment?
In the past I worked a lot with standalone applications that could handle events such as OnChange, in which the change of an internal variable or system state could trigger an action.
Now, with a distributed architecture I want to know the best practices on handling events. I have read about Observer Pattern, PubSub, but my point goes beyond. Imagine, for example, a system in which a microservice must be updated when a value is changed on the database.
Also this is a question about real time values in a stateless architecture. When you need to store current states of the system in a database, you want other parts of the system to watch this database for changes. How to do that in a efficient way?
I use react-scripts-ts. There's an open issue discussing the future of it now that CRA does it on its own.

https://github.com/wmonk/create-react-app-typescript/issues/...
Currently, none. If you wanted to do it right, the correct way would be to treat so-called stablecoins like a bank and audit them accordingly, except with a 100% capital requirement.

It’s highly entertaining to me how many issues with cryptocurrencies have to be resolved with classical financial instruments and regulatory processes. It’s almosf as if these systems were slowly built over a few centuries and are actually pretty good at what they do, and that you can’t just throw computers at the problem and pretend that you know better.
Definitely! I'll sleep better when we have all the bulk data shipped to everyone who wants it. That will happen by March 2024, or earlier for any states that switch to official digital publishing.

(I mean, Harvard isn't a bad home for this -- I work in a building with books that predate the printing press, and I work on stuff like Creative Commons-licensed forkable textbooks. Libraries are cool places. But Harvard definitely shouldn't be the only place that preserves this data set.)

As far as preservation-friendly formats, our bulk data download format is xzipped jsonlines, which is tuned for NLP (highly compressed, parseable in a few lines of python with low memory requirements) rather than preservation:

https://case.law/bulk/download/

Internally we have a preservation format where each volume is stored as a bagit bag containing METS XML for the OCR and case-level data, plus color and black and white images of each case. These are much harder to work with, so it's not a focus to share them right now, but we can definitely share if someone makes a case for it.
Show HN: Collect by WeTransfer
That's nice. I didn't say there was no value in intuition. I just said it was unreliable and I demonstrated an example of why its unreliable.

Its better to focus on solutions that don't rely on intuition because intuition is based solely on experience and often can't be applied to everything. This is why there's a strive to find general solutions to things. Intuition can help you, but you should not base your understanding off it. Intuition can lead you astray easily as in my example. People just assumed it was correct for many years until Galileo actually proved it through experimentation.

In fact, I found this explanation way more complicated than just stating that you are performing trigonometry in two different domains. Your sine is scaled by an imaginary value and your cosine is scaled by a real value. The radius equates to an exponential. When you move it pi units you get sin = 0, cosine = -1 which you use algebra to get Euler's identity.

You don't need to introduce complex intuition in order to get it. Intuition is often a crutch people use to try and think they understand something and just blame others when they don't get it much like this post. If intuition was so important, why not submit it as a legitimate mathematical way of proving things?
I agree with you regarding existing software - I don't think it's worth discussing too much about, since that software will either live on "forever" or be outright replaced, rather than being rewritten.

I was mainly referring to new projects, where C continues to be selected because of a myriad of reasons, none related to technical uniqueness or superiority.

This is truly infuriating, because we will collectively pay for those poor choices in security vulnerabilities.
APFS is still (afaik) only used on SSDs. So your HD boot volume is probably HFS even on Mojave.
Porn-loving US official spreads malware to government network
Financial security is a much higher bar than 1%.
It's a shame you've been downvoted on this point, and it's more evidence Hacker News is becoming more like Ars: upvotes are votes of agreement as opposed to votes that you providing insight.

And your comment is not only extremely insightful, it is exactly right - the current global political climate is a populist reaction by people tiring of being told by elites what they should and should not have and what they should and should not believe.

All of the fixes being proposed are, if we are honest, some form of rationing or another. That will necessarily reduce the quality of life of a large group of people, and they won't accept that no matter how strongly bureaucrats insist they should.

So such insistence will need to be backed by force, because it will be resisted by force.
http://www.sciencemag.org/news/2017/05/where-have-all-insect...

https://www.telegraph.co.uk/science/2017/08/26/windscreen-ph...
There's also an excellent video from Veritasium, a science and education YouTube channel, about what's being done to define the kilogram

https://www.youtube.com/watch?v=Oo0jm1PPRuo
Let's see how glib you are when humanity succeeds in wrecking the environment enough to begin exterminating itself.
Show HN: Share interesting articles and get jobs
That would be devastating for individual organisms, but what would the large scale ecological impact be? Chernobyl is home to quite a bit of wildlife as I understand it.

Also, if we're talking about a world after the exchange of thousands of ICBM's I'm not sure how much relative impact the melting down of the world's nuclear power plants would have.

The problem has been considered before, though probably as a general safety issue rather than trying not to further irradiate a barren world.
https://en.wikipedia.org/wiki/Passive_nuclear_safety
A friend of mine once said: “There is a reason why Death Metal is very popular in the Nordics and Reggae in the Carribean”

I thought that was insightful.

PS: I really like Death Metal, Unleashed in particular
For me it’s all about the dissonance and I almost never listen to lyrics in any form of music and find it an annoyance. If there was a death or black metal musical, I wouldn’t like it probably, for example. The voice to me is another instrument that is no better or worse than other instruments but unique in that voices are obviously much more varied than an instrument is.

But I think it’s worth getting a survey of some sort from folks into metal seriously. We’ve already shown that metal fans are among the most dedicated listeners of a genre, for example, according to Spotify’s stats. The Metal Maniacs BB more than a decade ago did an informal survey among the members and found that at least in that crew most metal fans are pretty darn introverted, which is a finding that most people not into the music and cliques would be surprised by. That board may be up on the Wayback machine but maybe it’s be of some value to researchers.

The self-report aspects of a study are handy enough for the purpose of identifying further research if you have no clue what to investigate next at least.
So who get's to define evil? It's incredibly dangerous game to restrict free speech in this way and force political alignment .. no one is forcing anyone to agree with anything anyone says, but to outright ban certain topics of discussion is a dark road to follow.
That seems very expensive for what is a very simple site with very low costs. Why not $5/year?
Impressive! Interesting as a benchmark to test the performance of different compiler implementations.
So where's the code? Or is "on your own web site that you control" fake news too?
Same with us at last year at missiveapp.com, the interview was relaxed, still could not articulate what we were doing based on the questions. They even talked about some of our metrics at the end, those were wrong, like from the previous time we applied.

We were rejected, the reason: No market.

To YC's defense... they interview so many incredible founders.

We grew between 10% and 30% every months since. So in the end all good.

Our team was also rejected in 2013 with conferencebadge.com... that time we bombed the interview and were really bad. The business has now grown to 7 figures a year.

Both businesses are still managed by our small team of 3 co-founders and one employee and are quite successful. We own 100%.

What if we would have made the cut... who knows...
How Hubspot reduced user churn and reached $100M in ARR
This does not cut it as a response. The earth does not have capacity for us and all the living plants and animals. A healthy future should include biodiversity and not 9 billion people. Things are not getting better for most of the planet.
> Is it simply a question of finite developer time?

Maybe. Or it could be because they don't want to. People equipped to solve one problem may not want to solve another, related problem. It might not be a priority, interest, or motivation for them. That's not apathy; that's decision-making.
You're not. Well, it's not officially deprecated, but it pretty much is, yeah.
How do you control the content your children are exposed to once they go to school?

I remember being shocked about a decade ago that a group of 3rd graders had seen The Hangover(a rated R movie) without the permission of their parents. When I asked how they were able to do this without their parents knowing, they almost laughed at me.

The idea that their parents could effectively censor the content they consumed was not something they understood. There was always someone with a phone, computer, laptop, USB key, DVD, etc.

When kids are young and you(or a loved one) has eyes on them 24/7 it’s easy to control their lives, and the content they are exposed to. I’m very curious as to how parents effectively censor the world for older kids who are around other kids, without turning into the dreaded “helicopter” parent.

It seems like an overwhelming challenge.
Was going to mention meat. If you take public transportation for your job and only drive occasionally (let's say to visit family, friends, and shop) it easily dwarfs any gas you use. Like 2-4x the pollution you cause via gas.
Well, that is a strange school, such a viewing would cause nationwide outrage if it happened in my country.
There's seemingly a lot of negative responses to articles concerning constexpr in modern C++ (you can see some here in this thread). I suspect this is partially because there's a general outsized dislike of C++, and partially because typically these articles focus on toy problems where the capacity to run at compile time isn't actually beneficial, rather than cases where constexpr can provide some real benefit towards production code.

Like any syntactic construct in a turing complete language constexpr presents a convenience - in this case the ability to rectify compile time generated data and runtime code paths/data. There are some practical cases I've seen where it can be a real improvement to hygiene (i.e. you can replace a compile time script/program generating data with one unified code base). e.g.

1. Perfect hash generation. Rather than using a separate gperf run you can generate your hashes during compilation of the relevant TU.

2. Format string lookup tables. For fast formatting you can push an index and positional arguments to a queue, and handle formatted string construction in another thread. Broadly speaking lookup tables in general can be generated at compile time rather than static data imported into code.

3. Complex starting conditions for physics simulations. If you have, say, some initial matrix that doesn't change between simulation runs - you can exercise existing code to generate this matrix at compile time rather than needing a separate process run or script.

These approaches are not without their downsides - particularly if the translation units in question need to be recompiled often, or if you're using whole program optimisation - but the benefit of keeping generated data in lockstep with the code consuming it can be worthwhile.

(That being said, I personally find articles like this incredibly edifying.)
I forgot the Discover feed is publicly accessible, it's a good way of getting a sense of what the community is like.

There's also the 2 min video tour that they just released today, and should probably have been added to the front page of the website:

http://help.micro.blog/2018/video-intro/
> I'd like to believe C has a stronger future in the academic world because it is simpler than C++

C is simpler than C++ until we need to manage memory or need to handle basic data structures. When that happens, C requires reinventing the wheel over and over again.
> Hopefully this means no logins, also, but that's less clear.

The only things we have behind logins are what we are contractually required to, yes. This gets pretty fine-grained -- if you do a logged-out search across jurisdictions, requesting full text, the json contains error fields for the specific fields we aren't allowed to share without a login yet.
You mean besides the fact that the volume encryption password was stored in plain text in the disk?

https://thehackernews.com/2018/03/macos-apfs-password.html
Not only can you not price match those situations but often times they get advertised as "exclusive" and maybe have one small feature change to warrant it.
Well, there are two debates.

The debate about executive power is a critical one to have, that's been really brewing for probably about 14-16 years or so I would say just off the top of my head, and isn't limited to this particular issue.

However, this particular debate has to be on the merits. If the fourteenth amendment is found to not protect birthright citizenship, then the President actually does have the power to limit it by executive order. Nearly all immigration functions fall under the executive branch, unless Congress chooses to legislate on the issue -- on this particular issue I can't recall a specific law that addresses it and I suspect that given current leanings in the senate at least it would be very difficult to pass such a law.
Yes -- in fact some webperf techniques are HTTP/1.1 - specific, and are actually _anti-patterns_ in HTTP/2. Spriting is one such example.
The only way to stop this is to drastically reduce the human population immediately. No shitty carbon tax will even remotely address this problem. Anyone want to give that a shot?
Well, war is good for the war racket, but generally not good for trade in the vast majority of other goods.
Domain sharding is another.
Oracle Autonomous Database
Centuries of fear-mongering by those warning of ecological disaster and yet we've experienced no global ecological disasters. But you want government forcing change despite many, many examples (within a century!) of that resulting in political, and subsequently human, catastrophe. You might want to think this through...
That first album is great considering it's just Schumiez playing/producing/recording/singing everything. Fermented Offal Discharge has an amazing neoclassical solo. The second album is even better in my opinion, a really talented drummer and bass player in addition to more complex writing. There's so great non-traditional time signatures on that album too. God I wish they would release a new record. It's been over a decade I think.
Thank you, yes, it was helpful for me to understand the possibilities. I'll dig more into that.
U.S. restricts exports to Chinese semiconductor firm Fujian Jinhua