skysight-inc/HackerNews huggingface dataset:
I say this as someone who built data infrastructure before and after the invention of data lakes, and I have done it every way - the old and the new.. For the record, for a lot of scenarios, the "old ways" actually do still work fine. But there are new opportunities/possibilities too.

I really understand what you are saying... I know the hype problem, I lived it. It makes me both frustrated and sad - because the hype is annoying, there is a lot of vaporware - but there is also something real that is happening too which is part of the story of the evolution of data architecture/infrastructure. My strong advice is, being open-minded is helpful - learn and take what was good/real and leave behind the stigma/hype. Something real and useful happened in terms of architecture, so take the benefits - but of course, don't compromise on delivering real, working solutions.

Regarding "data lakehouse", I struggle with the buzzword term also, but once again, I recommend looking at what is good/real and ignoring the stigma of the buzzwords. One way of looking at it is the literal translation - a data warehouse made from the components used to make data lakes. To be honest, it is a marketing term, but it is also an architectural pattern we had even before the term existed - for example - you could use a data warehouse product such as Vertica, and back it by HDFS - guess what, there's a data "lakehouse".. and most of the big database vendors can do this trick now - a full traditional data warehouse engine sitting on top of lake storage infrastructure.

There are several "real" use cases for data lakes. Precursor architectures could be seen to be "operational data stores" [1]. Data lakes are real, they are one approach to solving some problems.

These use cases could include: 1) raw, long term storage of large volumes of diversely structured data for staging/historical purposes; 2) data discovery/exploration of this data to identify patterns/models and relationships (this is both an AI and analytics use case for power users and BI/analytics/data scientists, etc.); and 3) an opportunity to change the paradigm of traditional ETL - instead of pulling from sources, one way you look at it is, allowing many diverse/distributed sources to push their data into the lake for powering analytics, exploration, AI model building, etc. It makes sense as part of lambda/kappa architecture as well - some of the "push" in can come from streaming sources as well.

Use case #1 is very much a "data infrastructure" kind of use case that we do anyway in data warehouses - especially those that do ELT (vs. ETL) - staging databases. If you want an architecture that actually helps make some sense of such a use case of data lakes more formally, one could look into Dan Lindstedt's data vault architecture [2]. While data vault modelling doesn't necessarily require "data lakes", the "raw" part of the data vault architecture use case overlaps nicely with data lakes.

[1] https://en.wikipedia.org/wiki/Operational_data_store

[2] https://en.wikipedia.org/wiki/Data_vault_modeling
Compliant Mechanisms
Because every software-focuses business that does one thing really well thinks that to scale they need to build a “platform” or “ecosystem”.
NASA is testing the first of its new moonwalking spacesuits
We have a battery-powered drill that my girlfriend brought with her when we moved in together over 7 years ago. We use it very rarely, probably once every 4 months. We don't even have a charger, and the battery has never been changed. Still enough charge to screw into particleboard. I hope someone here can explain how this is possible :D
There is a coinvent sidestep to this issue.

Distributing sexual images of someone without their consent is a clear crime against that person. Children can not consent.
Probably, although my question was not about the bloat but the claim on the article that users prefers the UX of SPAs.

Should rephrase that, they have big JS payloads to make the website feel close to an SPA as possible.

> my guess is that if Github were created today, most of it would probably be an SPA

Github.com today has many components that are pure JS(most new features), writing it today from scratch my guess would be they would have go almost all client-side JS.
And some thoughts on the background for the app - https://medium.com/dhiway/secure-qr-scans-with-the-seqr-app-...
Depends on the use case. YAGNI? The presented function is very simple to read and understand what it does.

Of course in Python you could parametrize the predicates or the attributes or attribute values, and it would still be fine FP code.
https://en.wikipedia.org/wiki/Boltzmann_brain

See also: https://en.wikipedia.org/wiki/Timeline_of_the_far_future
Whether usage is commercial or not is a big part of the first factor in the fair use test: https://en.wikipedia.org/wiki/Fair_use#1._Purpose_and_charac...

You could arguably implement Java without a license from Oracle provided you were using the GPL, as OpenJDK is released under the JDK, and presumably includes the entire Java API. The problem for Google, of course, is OEMs would've balked at being forced to open source all of their proprietary modifications to Android.

Wikipedia says the OpenJDK was released in 2007, so Google could've legally used the Java API under GPL terms, but chose not to, and also chose not to pay for Java licensing.
I've worked in environments across the version-control gamut. The best-run places have had monorepos. But, for the life of me, I would not trust the companies that didn't have monorepos, to operate a monorepo.

To go mono is to make an org-level engineering/cultural commitment, that you're going to invest in build tools, dependency graph management, third-party vendoring, trunk-driven development, and ci/cd infrastructure.

Can you make a mono repo work without all of those things? Yes, but you are sacrificing most of its benefits.

If your eng org cannot afford to make those investments (i.e. headcount is middling but there is zero business tolerance for investing in developer experience, or the company is old and the eng org is best described as a disconnected graph), forcing a monorepo is probably not the right idea for you.

Monorepo vs microrepos is analogous in some ways to static vs dynamic typing debate. A well-managed monorepo prevents entire classes of problems, as does static typing. Dynamic typing has much lower table stakes for a "running program", as do microrepos.

edit:

It's worth noting that open source solutions to build tooling, dependency graph management, etc. have gotten extremely good in the last 10 years. At my present company (eng headcount ~200), we spend about 2 engineers-per-year on upgrades and maintenance of this infrastructure. These tools are still quite complex, but the table stakes for a monorepo are lower today than they were 10 years ago.
Hey everyone, this is a weekend project I built - it lets you search through US Presidential candidates’ speeches for topics and see what both candidates have said side-by-side (on a largish screen).

My thinking is that with all the divisive misinformation making their rounds these days, hearing what both candidates have said first-hand on the same topic presented in a side-by-side format, could help inform people of all perspectives in an unbiased way.
I was going to say the same thing. GP and I could design the bet with assumptions like "most of the US" and "good weather" but that's riddled with issues from a contractual standpoint.

So how about it has to take GP from his home to a destination 100+ miles away, the car has to park itself in a legal spot, and it must reverse at least once during the trip? And I'd like to see it done at least 3x, including once when it's not perfect weather (could be foggy, moderate rain, light snow - whatever).
It's absolutely for the latency, it's almost a faux pas to play the games I mentioned online at a high level without an Ethernet adapter (since there's a decent chance you'll get a match with others who are). Additionally, the Switch is the sole major gaming platform which has avoided the bloat endemic everywhere else; there are no games on the platform anywhere even close to triple-digit GBs in size, or even half that.
Ask HN: What are some good ideas for distributed systems project for college?
I am in my third year of university studying computer science and I need to come up with an idea for the final project on which I would work for more or less 7 months. I would love my project to be a distributed system utilizing AWS since I've experience with that.
I think the main lesson is just to not tolerate third-party protocols. Having a uniform RPC interface with integrated authentication, authorization, and delegation makes it much easier to get your security situation under control. If you're out there with your MongoDB password in a secrets vault, you're already in an unsustainable situation.
>A) lying to cover up getting information from The Russians, or B) telling the truth and the “analysis” that says the Russians hacked the DNC server is actually false.

Assange was still communicating with and receiving DNC documents from his source after Seth Rich was already dead. Assange was not telling the truth. It was obviously option A.

A journalist should try to hide their sources. A journalist doesn't try to hide their actual source by pretending their source is a dead person and as a result opening that dead person's family up to repeated harassment.

https://www.nytimes.com/2019/04/20/us/mueller-report-seth-ri...
Show HN: Unofficial Terminal Interface for AWS
Ask HN: Is there much future in being a Swift developer?
I'm a web developer who really likes the look of Swift-- I would love to be writing code in it.

My concern is that it's only used by a narrow range of products -- Android has taken something like 90% market share in mobile phones and I worry that in the long run Swift and the Apple frameworks aren't a good investment as a developer.

What are your thoughts?
I've idly wondered how good those order pullers are, both those working for the store itself and those working for Instacart. Memorize the store layout, reorder your list, and you could blaze through it -- not the efficiency of an Amazon warehouse, but more efficient than a casual grocery shopper. (Even one like me, who rather enjoys grocery shopping and knows where most of my common items are.)
>hackers tied to Russian intelligence hacked the DNC

That was debunked this summer.
With all the recent talk about Samsung TV's spying, privacy concerns about Google and others, etc... I'm surprised you hold this position.
Historians are politically biased, with about 90 percent of them leaning left. See Jonathan Haidt for more information on political bias in the humanities. On such a politically charged issue they are not to be trusted and are a corrupted group of individuals. This is exactly why we need a politically balanced team to do any vetting on a platform like Facebook.

Also, to say that protesting in support of a Robert E Lee statue is not showing support for Lee is prima facie absurd.
Bleh. When you need a in-browser tool to use a library, you really know it's messed up.

Isn't there anything like structopt https://docs.rs/structopt/0.3.19/structopt/ for Python?
Did BMW learn a lot from the nine production years where their chassis cracked so badly that the wheel supports reliably tore away from the rear subframe? Or the control arms bearings that would disintegrate every 50k? The rear differential bushings and the GUIBO flex disc that both require regular replacement to prevent the car destroying itself?

Not to mention the little stuff, like the window regulators and sunroof self-destructing, the ignition tumbler corroding and disabling the car's internal bus so the lights freaked out, or the headlight mountings disintegrating.

As the owner of an E46 that I purchased used at 100k and now sits defunct at 155k I'm baffled by your comment. I'd fix everything on that car and four unrelated things would break the next month. "Preventive maintenance" wasn't oil changes and brake pads... it consisted of half-replacing major systems at regular intervals.

Maybe they just got their shit together on the newer models, but yeesh.
Your point about not being an early adopter is well taken. We fully expect a segment of the market will be disinterested in using a relatively new product like ours. If we were a customer considering a new secrets manager, we would likely weigh similar concerns. I think the onus is on us to prove to these larger customers over time that we can be trusted (from a security/availability/longevity standpoint) and we certainly intend to do so.
So if Oracle implements a proprietary API in GPL software does that mean that all Oracle software is copyleft?
You mean my garbled sentences, or something else?
> Where are the expert witnesses in this case?

Expert witnesses play a role in trials, they play no direct role in appeals where fact claims (to the extent they are reviewable at all, which is normally limited when, as in this case, there is a jury-trial-by-right, because otherwise you obviate the right to a jury trial, though its worth noting one of the issues in this case is that the Federal Circuit tossed aside the jury verdict using a standard which is not usually appropriate for review of fact questions in such a scenario) are decided by review of the trial record and, to the extent that the trial record is not sufficient, remand to the lower courts for further proceedings with legal guidance.

> Do we really expect judges in their 60s and 70s to understand basics of coding in order to come to the right conclusion?

We expect the parties to have developed their fact claims at trial or, failing that, to be able to explain to judges why any issues needing factual evidence are insufficiently developed in the trial evidence such that if they were critical it would require remand (and the reason better be something like "we were improperly prevented by the trial court from presenting evidence" or "this is a issue that somehow was allowed to be raised for the first time on appeal so we had no opportunity to present evidence on it at trial".)
While everything you said is true, the constitution is suppose to be what protects the individual from this kind of mob thinking.

General Warrants have been unconstitutional since the founding of this nation, only through some very bad "3rd party rule" exemptions to many constitutional amendments is this even thought about as acceptable.
Well, I do live nearly 3000 miles away from WV, so all I really know about it is the stories that rise to the top of the national news circuit.
It seems to be possible: https://en.wikipedia.org/wiki/Timeline_of_the_far_future

> 10^10^10^56: Time for quantum effects to generate a new Big Bang, resulting in a new universe.
If Google wins on fair use, then re-implementing much (but not all) of a 10,000-element API is fair use. That leaves the industry somewhere near "APIs can be copyrighted, but that doesn't change anything".
We don't think of Secret Manager as a competitor, but more of a destination for your secrets. We have integrations with both AWS and GCP Secrets Manager so you can write directly to them from Doppler.

We also can write to AWS Parameter Store.
I’ve Been Watching My Kid’s Teacher on Zoom, and I’m Horrified
Bingo. It doesn’t matter what he says, they just want to keep asking the question anyways to drive division to garner support.
Gitlab CI does a single clone of each repo per runner executor and then does fetch/clean for subsequent builds. It's pretty quick in our use cases.
You can still run msvc6? Is it only 32-bit on command-line or what?
> can you prove in court you received a given email? No, because a SSL connection is transient and you can't replay it to show that Google's certificate digitally signed that email in GMail.

A TLS session replay is unnecessary. If you've got the email with its headers intact then you can prove that you received that email.

You might want to learn what email headers are available to you. You'd be interested in learning about ARC-Seal, ARC-Message-Signature, ARC-Authentication-Results, and DKIM-Signature. Those headers will let you cryptographically validate that a message is authentic -- that the email is unaltered as Google received it from whoever sent it.

You'll also be interested in learning about Key Transparency: the signatures are created using Google's PKI which should be listed in any public key transparency log. That will let you prove that the signatures you validated were created using something that only Google knows.
I've had to dive into this recently as part of my work (I'm in academia).

While the original vision seems to have failed from a lack of incentives, the idea behind RDF as way to model connected data and its many serialisation formats is actually quite helpful. It seems to have found a niche in academia when it comes to sharing and integrating large graph datasets. I just consider RDF and most of the other semantic web standards as a framework for modelling and serialising graph data. It's pretty good at that.
Bazel? Gradle? They’re good at figuring out the minimal changeset and caching very aggressively.
Professional Degree vs. Academic Degree: What's the Difference?
Pros:

- You can track changes across projects.

- Finding source for a project can be easier.

- You can more easily switch to code dependency instead of binary dependency. With things like gradle becoming able to pull and build git commits this is less of a win.

- Its easier to kick off downstream builds because everything is in the monorepo and you can more easily track dependencies.

Cons:

- Its a technical challenge. Git isn't great at it although its getting better.

- If you do want to lean into linking source instead of binaries you now need to have a more consistent build system across your codebases.

I prefer working in them and in my opinions its approaching a personal preference choice.
"Should not have been accepted" - is typical xenophobia. It's not an argument at all.

What sectors have seen a depression of salaries as a direct result of Romanians entering the market? And you have the huge hurdle of proving that those salaries are depressed specifically because of the Romanian labor, and not because Asian products or other global trends.

Also - restricted migration fails to attract the right labor, driving up the cost of labor unnecessarily. Sometimes it gets ridiculously stupid... to the point that local consumers(also local labor) cannot afford to consume products, because local labor(also local consumers) refuses to work for less. It gets to a point where local businesses cannot pay their local labor and invest into productivity gains(required to keep the pay high enough).

It's a complex clusterfuck... and blaming Romanians or Mexicans is just an easy "solution".
So would const generics for arrays make map/fold somewhat faster because there would be no need to create an iterator?
Is Your Favorite Podcast Tracking You?
That's weird!

My company, Megiddo Hyperloop, just chose West Virginia too!
Information security is just a super wide field. To pick a couple famous examples: what Google Project Zero does, and what the "Swift on Security" person does, have almost nothing to do with each other.

They both matter, though. Basic blocking and tackling at the IT level is important, especially to large old institutions. Apple is obviously an apex technology company, but they're also a 45 year old public corporation... I'm not surprised they've got some vulnerabilities lurking in their subdomains.

Patrolling DNS and 3rd party corporate applications is not usually what people think is sexy security work, though. Problems avoided are harder to sell than problems discovered or bad guys defeated.
Louise Glück is the 2020 recipient of the Nobel Prize in Literature
> That sounds like a slam dunk good thing for society.

That particular application is a good thing. But the existence of such capabilities can be a net negative, and by a huge margin.

Crime is something like a "cost of doing business" of a free society. Take all the crime away with an all-seeing Eye of Sauron and you risk a technological dictatorship that opens death camps for the currently unpopular race/class and crushes all resistance.
Skytap | Full-Time | REMOTE (US/Canada) | Mid/Senior Full Stack Engineer (Ruby/Rails/JS/React)

Skytap, a cloud infrastructure startup based in Seattle, seeks a talented, mid-level or senior Rubyist for a full-time, remote opportunity as a Rails / full-stack engineer, anywhere in the US or Canada.

I’m the hiring manager for this position. My team of 5 (including this new opening) manages a product of our own that sits atop Skytap’s cloud platform and helps to drive one of the most important segments of Skytap’s business. We work closely together, wear lots of hats, operate with a high degree of autonomy, and are generally pretty awesome, if I do say so myself. :)

More info: https://jobs.lever.co/skytap/0d7943d0-a111-4ac5-a907-5fe0704...

Don't hesitate to reach out if you have questions or would like to chat: my HN username at skytap.com.
Well, you're right: in fact, in some cases Monty cannot open any door! That is, if players A & B both chose goats, then the remaining door is the car, and Monty cannot open it without ruining the game.

For this reason, both players should be playing "blind", without seeing the other's choices or the door Monty opens for the other player. Anything else doesn't make sense for the game.
A Delhi Man Stands Between Sony and PS5’s India Release
A great find. Thanks.
I haven't seen a single positive article about Google from ArsTechnica (and plenty of ones that were overly sensiationalistic, especially aroudn Android which I'm pretty familiar with professionally).

Is there any more independent media article describing what went on in the hearing?
Another technical fix might be separating Submission karma from Comment karma. I'm guessing that good Comment karma is associated with civility and adherence to community norms. That assumption could be validated by someone with access to the underlying data.

I agree that continuous oversight by quality moderators like dang are probably more important than technical tweaks.
We just build and test everything every time. Maybe it costs us more $$$ on CI, but it costs far less in developer time. That being said, I mostly work at startups and larger companies will certainly cross a threshold where this isn't feasible.
Can you copyright Harry Potter the character? Or can anyone use recognizable Harry Potter reproductions in commercial context, be it alternative books, movies or merchandise?
The IA has an archived copy: https://web.archive.org/web/20160923123125/http://blog.bumbl...
I’ve pondered the same thing in my time working for web startups.

I think a good goal to instead shoot for is being able to extremely trivially build and ship a “latest” of your entire product.

You know your product doesn’t grow in sane predicable ways, and you know you won’t have resources dedicated to maintaining internal dependencies, so why try to half-ass a complex internal dependency tree? What ALWAYS happens is unexpected feature requests break clean APIs setup by engineers, which creates all this fake work of “making the breaking change” across the dozens of dependents.

Something like a monorepo could help prevent issues like this, but it’s not a guarantee you’re making it easier to ship one latest version of your software.
I am quite at the opposite of the article author. I see HN as a news aggregator, especially through the excellent hckrnews.com. I do post a few comments, read answers, seldom answer back. I certainly receive enlightening answers and upvote them. I often receive downvotes, and generally ignore the noise. I do not thing it is possible to have a real followed argument on HN and possibly on any open forum on the web.