skysight-inc/HackerNews huggingface dataset:
Looks broadly similar to the bottom layer of the system I work on, except that blb only claims that it "should" scale to very high levels whereas the one I work on has already been running at many-petabyte scale for years.

Not too surprisingly, a lot of things that seem "impossible" at smaller scale become every-day things for a large enough system. For example, you will get some kinds of inconsistencies that necessitate various forms of active GC or scrubbing. You will have hot spots, which you need to explicitly deal with instead of relying on statistical distribution guarantees. You will have to migrate whole racks' worth of data at once as equipment (not just disks and hosts but also network switches and power infra) get upgraded. And of course you'll have to monitor the hell out of it so you can fix these problems as they occur instead of having them multiply until your system is irretrievably broken. Don't take claims of super-duper scalability (from blb or Minio) on faith. Look for these "extras" as evidence that the system actually has been run at scale. BTW they don't seem to be there in the blb source.

What does seem to be there is a reference buried in the deployment docs to a "master" component, not mentioned in the architectural overview. It seems to be responsible for assigning partitions of the blob space to curators and directing clients to the right one. Seems like a bottleneck but OK, let's take a look anyway.

// Since we don't persist the address and last heartbeat time, when a master
// failover happens, the new leader cannot service requests until it hears
// heartbeat from the curators. See PL-1102.
(from master.go lines 35-37)

Hm. That seems like a pretty big disruption, even if it's rare. Also, what kinds of heartbeats are these? I think it's generally a bad idea for systems like this to implement their own liveness checking. That's a hard problem, there are tried and true specialist-written systems for doing it, other systems that do it themselves are almost certainly drifting away from their own core competency. This comment is getting long enough so I won't do a full analysis of the blb heartbeat system, but I invite others to look at it with an eye toward how much load it imposes on masters in a large cluster, how reliable failure detection is, and what things should be done (but aren't) when heartbeats fail.

It looks like a pretty good start to a distributed blob store. The basic architectural principles are sound, the code looks pretty clean and well commented, etc. OTOH, seems a bit light on tests, and the lower-level implementation details suggest that in its current form it might not handle even a hundred-node cluster all that well. Caveat emptor.
Beautiful Paper Flower Easy Making Tutorial
> toward in-flight on-demand programming at no charge

Huh? I've never ever seen a charge for inflight entertainment, and I fly United all the time. Maybe different routes, then.
Then maybe stop using our personal data for advertising? DuckDuckGo can do it, so why can’t you?
> How far from optimum the human mind is, and how complex it is to improve upon it.

Nature is pretty good at optimization. We can often build devices that out perform animals on certain characteristics, but that's because we've limited the scope of what they do and we aren't concerned with the energy requirements. We also know that, as interesting as our brains are and as powerful as they are at certain kinds of problems, they aren't the best at all of them. For instance, chimps appear to have a significantly better short term working memory. It is possible that we are already close to optimum for the kind of intelligence we are interested in. I suspect that if we manage to significantly improve upon it, it will be because we don't have to concern ourselves with the energy requirements.
Or use an Adblock.
Definitely - and the intent of the business model and the business itself. I've worked on products before that on the surface (and even what I was told by the folks running it) could have done good. But the company had no intention of using it for good. They had every intention of lying about where they were really making their money though.

I feel like all the justification we see in our industry of "it's just a tool, it's not our fault if it's used poorly" can often end up being a variant of the banality of evil. What if its most obvious use case is the malicious one? What if the company paying you to make it has the malicious use case of the tool in mind? Would the company and the product exist/be sustainable without the malicious use case? Are there ways to curb the malicious use case that are being ignored - intentionally or not? We've got to start asking ourselves these questions. The ACM and IEEE would insist that it is part of our core duty to be asking these questions.
Just another reason to use noscript or other blockers. Why was it a good idea to turn browsers into receivers for arbitrary script code again? I guess we should at least endorse GNU's LibreJS [1] and enforce subresource integrity to check the script executed by your browser is verifiable, even though LibreJS aims for OSS principles. As a bonus, you'll find that web sites sending crap script aren't worth visiting anyway ;)

[1]: https://www.gnu.org/software/librejs/
The first part of this reads like an intro to algorithms class where I think they cover all of these things. Directed graphs, back edges (cycles), convert to a DAG (no cycles), strongly connected components, topological sorting, etc. Nice to see it applied to solve the problem at hand.
> If you can kill people with your comments, it should be against the law, no question.

You can't, though. You can call for violence, and that is already against the law. The conflation of words with violence is dangerous.
That's the popular quote, but presumably something was preventing war in Europe prior to the creation of the EU in 1993.

The European Coal and Steel Community (grandfather of the EU) was created in 1952 to regulate production of these materials, and address the risk of massive re-armament. I'm not convinced how relevant that is after the US detonated the first atomic weapon in 1945.

IMHO, NATO has been the factor in preventing war in Europe during the last 70 years.
The deployment infra constraints mentioned in parent post are generally managed by tooling we also publish[1]. The encoding ends up similar to Helm’s, but happens through this tooling just by virtue of taking a dependency on a published jar, npm package or conda package, and that removes a lot of programmer error/guess work/maintenance. (I think we’d be open to also emitting Helm’s formats in our packaging tools, if that’s interesting to someone reading this feel free to open an issue on our repo and reference my post.)

[1]: https://github.com/palantir/sls-packaging
Modern Software Development or Slow as Molasses
I don't want to just come out and say you're wrong bluntly but...well this debate has happened on HN many times. I'm a little shocked you worked in the industry and can seriously assert all marketing does is consolidate the power of the top n%. More than just about any other service in finance (except maybe credit/lending and consumer banking), market making drives prices down for everyone. It's a prime vehicle for delivering efficiency.

I can understand arguments that the work wasn't fulfilling, or you didn't enjoy your colleagues...but what you're saying seems to be incorrect, strictly speaking. Were you working in marketing making where most/all trades were OTC?
There are people living on a few dollars a day... You really think we can't provide a cot in the equivalent of a walk in closet with communal bathrooms, and simple meals like beans and rice with a few vegetables for $10k/year?
Bare Bones of the Great Ape-Man Hoax (1953)
Oskar Schindler was a Nazi party member who owned an ammunitions factory and saved 1200 Jewish people in Nazi Germany (the movie Schindler's List was based on his life story), but I couldn't find any mention of him sabotaging his own wares. I would imagine this would've led to more suspicion on his factory, where he secretly employed many Jewish workers illegally.
I don’t think I did any harm, but I came close to regretting my work.

Shit job at a startup, no money and burnout central, then I heard that the CIA was investing (well, their front company, which was an open secret). I decided that that was plenty enough for me and got a different job.
Things like this are clear steps on a road. Article 13 was the camel nose through the tent. One could call it drastic at the time but shooting every loser involved in the Beer Hall Putsch for treason would have saved the world a lot of trouble. A lot of them were purged anyway.
Whenever this comes up, the answer seems to be "for programming language research and proofs in academia" and not much else.

It's a pity. SML is in many ways a joy to work with. The language doesn't change, which is a nice thing in itself in the current climate, but it has many things people still look for in modern languages (type inference, pattern matching, a module system). Multiple compliant compilers are available that produce fast code. The syntax is delightful, relying on neither brace delimiters nor syntactic whitespace. The language and Basis library are small enough that a developer can realistically aim to learn the whole thing. It is a comfortably idiomatic general-purpose functional language: lacking object or procedural syntax but relaxed about side-effects and occasional use of mutable types.

There are problems. Everyone will have their own list of things they expect a language to have, that SML lacks (mine: a map container in the standard library; record updating without rebinding all the elements individually; universal printf/toString, no matter how hacky). There is no standard foreign-function interface for calling native libraries, and no standard memory model - each compiler has its own. There is no standard build system (though I dunno, maybe this is a good thing). Libraries are sorely lacking, although those written many years ago usually still work. Because the language is a stem for academic developments, there are several variations that are intriguing but not actually compatible - CakeML seems to be an example, as it doesn't support functors or records.

I find SML pretty nice for writing small command-line utilities - like a functional alternative to Go or Python.
I'm of the same opinion. If we actually create something that much better than ourselves, it'll just be another step in our evolution, albeit an unconventional one in that it will involve passing on memes instead of genes.
To my knowledge that's basically what Google Contributor does: https://contributor.google.com/v/beta

I don't think it ever caught on.
They're a complete mystery. There is extensive evidence that US and international militaries and space programs have released. That evidence points to physical objects in the sky that respond to our aircraft and behave in a way that physicists have said is not possible given our level of technology.

Maybe they're from a small African nation that's hiding their technology in plain sight. Maybe they're time travelers. Maybe they're the result of some distortion in the earth's magnetic field. Maybe they're aliens.

It's a scientific mystery but it doesn't get the proper funding and respect because of the stigma. This is a huge loss.
Introducing Watermill – Go event-driven applications library
Where do you live and why are US drones a reason to criticize Israel?

You're right that the term is a bit nebulous, but this isn't a new problem. E.g. criminal law already treats terrorism differently in many countries. As is often the case in law, courts would have to figure what should be considered terrorism. Courts in your country could decide differently than in mine, which can be a good thing or a bad thing depending on your point of view.
> technically did not do anything wrong.

I didn't share everything that happened in two years time but just some vague summary. Is hard for someone that does nothing to do something wrong. In all the meetings we had his first thought was "let's find an open source alternative that does that for us" even in some highly customisable and domain specific solution that won't be available in GitHub for you to use. Or not any "how to do ...." search in Google can solve.

Yet, after more than two years, is still have lacks on how our infrastructure and the application works. And he still propose useless open source or SaaS solutions that don't fit at all our usecase because he doesn't even know what the applications do.
I wasn't making the accusation, rather asking if you may have missed some opportunities by coming across as it to strangers.

Being a contractor, its a perceived strength to be as flexible as possible.
How to debug your front-end/back-end interface with the DevTools network tab
Very true - but a great seat post and saddle can help - I stuck a carbon seat post on my alu bike and it was amazing, and even on my full carbon I moved to a carbon seatpost and carbon (unpadded) saddle - soaks up the vibration even though I have the tires at 100psi. People often comment on the lack of padding on the saddle, but with a nice pair of bib shorts it is fine. And 250g for the seatpost and saddle ;)

To clarify - vibration that it best dulls is 'road buzz' which you can really get on a alu bike, esp with high pressure tires. A carbon seatpost and seat can be as little as £50 for both - I got my seat post from wiggle and seat from china - happy with both - and recommend to all my friends.

Yes this will have little impact on 'bumps' - though the saddle def soaks some of them up.

I posted this to refute that 'only' tires have an impact on ride - frame, wheels, seat post, saddle, bars and bar padding, shorts etc can all have a positive impact on ride - and allow you to ride at a high tire pressure and get all the benefit of lower rolling resistance.
I can't be the only one who feels The Register _IS_ a tabloid. Not tongue-in-cheek.
Also received in Italy 3 hours ago.
What you don't realize is that this is how most of the world was until it became horrifically overcrowded everywhere. Although admittedly, I have no idea when India wasn't overcrowded. In the States, every place has gotten to be a gridlocked mess nearly all the time and it is terrible.

I drove to work today just to enjoy what should be "normal" traffic levels, not the 5mph, stop and go, everyone cutting you off, nightmare.
Yes, but then you get scam sites which show 1 decent article and trick you into paying for garbage.
Yes, the more elite the outfit the more security conscious and vindictive it will be. It is defies belief how much they can get away with.
thank you. this is exactly my point. the only reason this is implemented is to shield those in power further and allow them to hunt down anyone who they deem in violation of what constitutes their version of free speech.
Do we? Our physics today suggest there are hard limits. We do have some creative theories (Alcubierre, wormholes) to work around those limits, but they could run into issues and in current forms require massive energy that could well be impossible to generate or harvest. Laser/solar sails seem most likely to me, but could be limited to small payloads and the general vicinity of our solar system. As much as I like sci-fi and real science, it seems quite likely that there are physical limits that keep intelligent life constrained to their solar system. Space is really really big.
> excess food in San Francisco needs to be sent to people in Bangladesh.

Most of that food in San Francisco is coming from other countries to start. The U.S. imports 50 percent of fresh fruits, 20 percent of fresh vegetables and 80 percent of seafood.

We EXPORTED 133 billion dollars of food in 2017, the bulk of that being soybeans, beef and poultry.

Neither of these practices are sustainable and the US isn't alone in this practice. People don't want a can of corn that was grown locally in the past year or two, they want a fresh cob grown in the past few weeks which has to be shipped from another hemisphere. They don't want frozen fruit, they want to go to the grocery and pick over visually appealing fresh fruit that just finished its trip from China to their local retailer in Bumsville, Idaho.

Did you know that in the U.S. alone the cattle we raise eat enough grain to feed a BILLION people? The average cow will eat EIGHTEEN TONS of feed per year. Don't get me wrong, I love a hamburger or some beef sausage on my pizza but...
> You'd need a deduplicating filesystem

Until all your packages decide to use different versions of their dependencies.
> would even cheer for fascists

That is what you said.
No, what you actually wrote in the other posts - such as "I would cheer for fascists" - is what got them downvoted, not what you now claim to have meant, because the voters are not clairvoyant.
It’s actually non drivers who pay. Gas taxes cover less than 50% of road cost on average.
What Happened to Tumblr Overnight
I agree on the unnecessary use of javascript and the madness of inviting anyone to run code on your own machine, but not on the browsing experience with js off. I used to do that but stopped given the high number of websites that are dysfunctional or simply show a blank page with js off. Why some developers think they need js to display a blog article is beyond me (well, web "developers") but the web is now pretty unusable without javascript. Even adblockers break many websites.
What in the world are you talking about? Are you claiming that farmers never use trains to ship produce? Because that’s just patently false.
Speak for yourself. For me, Google News shutting down in Europe would be a disaster.
Just make it an all you can eat plafond.

I know I am worth ~50$ a year to the Ad business. I'll gladly pay $60 a year for access to content. That $60 gets redistributed to content providers based on the share of my viewership they get.

Basically, cut the middle man (ad companies in this case).
Anticipation is future predicting
Someone should figure out a way to:

- Track how many sites a user visits, without knowing which sites the user visits. Call this number N.

- Charge the user for N visits.

- Distribute the money to the original websites, without anyone knowing (not even their banks?)

Sounds like a nice task for a cryptography researcher.
Get good with git: bisect
FCC to Release Report Wed Telling You If Your Broadband Provider Is Screwing You
The End of Trust (McSweeney's 54)
Unless we want the internet to continue centralizing into the websites that don't need a business model (like VC funding), or to the few that people are willing to take out their wallet for (like Netflix), we need a new solution if we're going to unanimously block ads.
You might want to look into board games a bit. That industry still tends to thrive on making a quality product they can sell (unless it's on Kickstarter, then sometimes it's just about having really cool looking minis, with gameplay an afterthought) and not so much about how they can microtransaction the game to death.

It pays much worse than game programming though, unless you make a major hit, so you'll need to keep a day job, especially now that there's way too many games released every year that it has the same visibility problem as like, Steam does.

But you get to work with all sorts of people, and do so in person (like I'm going to an event in a couple weeks where I will be playtesting other designers' board games, and even a couple games by publishers themselves, and get to talk to and be friendly with the people who work for the publisher), and I've found it generally more fulfilling than my time in video games, personally.

Although that being said, I do miss video games and I've started working on one in my spare time again recently.
The Snowden Legacy, part one: What’s changed, really?
Don't you end up paying more for the gift cards than you saved by flying a day later?
Social Media Lessons
Kakoune TV – Asciicasts of VimGolf Solutions in Kakoune
I read 'Modem' instead of 'Modern' and it still made sense.
Good on you for making the right move.

I think we as an industry need to stop stretching for excuses for why greedy behavior might unintentionally do something good. It's not a worthwhile cause and it doesn't justify the behavior if the "good" is a theoretical and unintended side effect. The "good" being theorycrafted on the nature of this product is indeed both highly speculative, oversimplifying of the human cost of changes to this economic model, and is obviously not the purpose of the product. That "good" is contingent on and integrated with a number of factors too complicated to boil down to a single sentence: it's something that looks good when oversimplified and makes ourselves feel better if we try not to think too hard about it.

When we ask ourselves whether a product we work on is ethical or not, we have to account for the intent of the product, and not merely stretch to find theoretical ways to justify it to ourselves. We do need to think about unintended side effects - particularly when they could harm others - but we shouldn't be using our theoretical and simplified best-case scenario of unintended side effects as a way to justify the intention and practical use cases of the products we make.

We should ask ourselves: if the "good" of an unintended side effect could be cut out of a product and we'd still make it, is it actually the good of a product? Or is it an excuse we're making to ourselves to make us feel better?
Trying to solve AI bias is good, even if I don't think this is the best approach
That's a non-issue because a microkernel system call interface is stable. Drivers will simply continue to work in perpetuity. That seems like a huge help to me.
Preliminary report on the prediction of "afternoon effect" (1942)
Yup; I prefer the hard ride and not to feel like I'm peddling a tractor.
https://www.eff.org/deeplinks/2018/10/eus-link-tax-will-kill...

Do you have any source for that? According to the EFF even the pretty URL itself is enough:

> The Directive is extremely vague on what defines a "link" or a "news story" and implies that an "excerpt" consists of more than one single word from a news-story (many URLs contain more than a single word from the headline).
> I don't accept that this means condoning any action those people might take.

That's because you turned "cheer for fascists" into "granting some leeway to people with less than scrupulous ideals". Even just playing such word games is lending support, and that's even more than condoning something.
https://www.eff.org/deeplinks/2018/10/eus-link-tax-will-kill...

> The Directive is extremely vague on what defines a "link" or a "news story" and implies that an "excerpt" consists of more than one single word from a news-story (many URLs contain more than a single word from the headline).
OCAML has this area more than SML does, however.
> Seriously, the most valuable skill the internet has helped me foster is my bullshit o-meter.

How do you know how accurate it is? How many false positives or false negatives does it generate?
The rise of fascists is EXACTLY WHY we should make sure such values are protected, not an excuse to abandon them. Do you really think you can prevent the rise of fascism by forbidding speech? How would that work?
Monkey Sort: Cloud Stocks Sundered, Can Facebook Save Instagram? And More
They're not talking about vibration damping, they're talking about attenuating impulses.

To dampen vibrations as soon as possible, you want very high stiffness.