skysight-inc/HackerNews huggingface dataset:
While developing our ETL processes in Flink we encountered this issue with data skew and there was surprisingly little information around on how to deal with it.

After a bit of brainstorming and trial and error we ended up applying some of the work that the engineers at Alibaba made open source.

If you work with or are looking into Flink this might be an interesting read for you and might save you the same headaches we had.
Used to use ccache to do stage1 gentoo rebuilds to support the labs @ the local university. 120 build nodes made short work of it (even for 2004)
This is similar to the Scott Adam's advise- which was great:
https://dilbertblog.typepad.com/the_dilbert_blog/2007/07/car...

If you want an average successful life, it doesn’t take much planning. Just stay out of trouble, go to school, and apply for jobs you might like. But if you want something extraordinary, you have two paths:

1. Become the best at one specific thing.
2. Become very good (top 25%) at two or more things.

The first strategy is difficult to the point of near impossibility. Few people will ever play in the NBA or make a platinum album. I don’t recommend anyone even try.

The second strategy is fairly easy. Everyone has at least a few areas in which they could be in the top 25% with some effort. In my case, I can draw better than most people, but I’m hardly an artist. And I’m not any funnier than the average standup comedian who never makes it big, but I’m funnier than most people. The magic is that few people can draw well and write jokes. It’s the combination of the two that makes what I do so rare. And when you add in my business background, suddenly I had a topic that few cartoonists could hope to understand without living it.

I always advise young people to become good public speakers (top 25%). Anyone can do it with practice. If you add that talent to any other, suddenly you’re the boss of the people who have only one skill. Or get a degree in business on top of your engineering degree, law degree, medical degree, science degree, or whatever. Suddenly you’re in charge, or maybe you’re starting your own company using your combined knowledge.

Capitalism rewards things that are both rare and valuable. You make yourself rare by combining two or more “pretty goods” until no one else has your mix. I didn’t spend much time with the script supervisor, but it was obvious that her verbal/writing skills were in the top tier as well as her people skills. I’m guessing she also has a high attention to detail, and perhaps a few other skills in the mix. Probably none of those skills are best in the world, but together they make a strong package. Apparently she’s been in high demand for decades.
It's complicated.

Static typing easily leads to a over-specification of types.

With dynamic typing the least specific type that will make this work is automatically implied. You can often make huge changes to the underlying data structures and things will just work.

Basically: I need a Car Object with these exact fields vs give me a thing with the field "foobar" that contains something that I can concatenate with another things.

(Though static languages with structural typing like TS muddle the water here as they can represent some of the power of dynamic languages. That might be part of why TS is so popular. The problem is that types in dynamic languages can easily get super complex without feeling complex while for static languages you might need very advanced features (dependent types and other scary stuff) to represent some cases.)

The big downsides when working with dynamic languages: You need to have tests or at least some reliable way to test you program. If not, refactoring can be indeed worse than what you know from static languages.

Also static types make it easier to safely make changes to a program without actually understanding most of the code as you can just follow compiler errors while dynamic languages require you to have some model of what is going on in your head. So for people that jump projects a lot, static typing also allows for easier refactoring while dynamic typing allows for more productivity when one is more intimate with the code.
Byrne's translation of Euclid, you can find a digital version here: https://www.c82.net/euclid/
Make uses timestamp, so if you checkout code or switch branches even if the files are the same, make rebuild them. Also when you switch branches the ccache cache still have copy of .o files.
I think it becomes even more obvious with team sports. Among males, Sweden is a top-tier ice hockey-nation but an absolutely disgrace in rugby while the opposite is true for New Zealand. There's a significant overlap in the talent pool for ice hockey players and rugby players so there's no reason why Sweden and New Zealand would be equals in ice hockey and rugby. Access to the sport makes all the difference.
> As such it’s not unexpected that older things thought to be correct turn out to be wrong. That’s part of learning and refining knowledge.

Of course.

Now imagine me at 8 learning one thing "said to be taken for granted because <<science>>!" and me at 20 wondering how was it possible that everything was completely wrong and ancient history as we knew it wasn't much different from reading Nostradamus.

It all literally radically changed in 15 years, we got it completely backwards, it wasn't simply "a period of understanding", everything was simply wrong.

Everything, not just something.

One example among too many: we thought and we were taught and students at university were taught that the size of the brain mattered and was unequivocally linked to intelligence (we know now it's simply a correlation), so Sapiens obviously had the biggest brain among humans and hominids.

That was false too!

Neanderthal had bigger brains.

Does it mean they were more intelligent?

*we don't know!*

Another example: many dinosaurs had feathers, when we discovered it, I was already a young adult.

And not just some of them, but hypothesis (is) that many, if not all non-avian dinosaur species also possessed feathers in some shape or form.

Stonehenge and all the other cromlechs were believed to be built by the druids until the 60s of 20th century!

There are 3 thousands years difference between the construction of Stonehenge and the first records of the druids, still the knowledge of them being "magical places" is still the most common among people.

> Sorry you had shit school teachers that didn’t help you understand that

Don't be so dismissive, if you read papers from the 80s, when I was a kid (I-m actually from the 70s), you'll see it wasn't my teachers, it was scientifically accepted, you could read about that in every book you could lend your hands on.

No one contested that knowledge in public, it was "the truth".

The only ones contesting those theories were the religious people, my city hosts the Vatican, you can imagine they were quite vocal about it.

It wasn't so easy to shed some light on the topic even putting some effort into it.

We were simply sitting on bogus knowledge, everything prior to 6-7 thousand years ago is obscure and cloudy, everything could either be and not be, except maybe aliens.

We don't know, Hancock is obviously not a scientist and his theories are laughable, but could a civilization have existed before the ice age and have survived it?

We don't know!

We even miss a billion years of Earth geological history, it's almost a quarter of its entire existence.

https://www.nature.com/articles/d41586-021-02450-5

It's easy to understand why people can believe to crook theories about Atlantis and the saviour from the sea, it's actually interesting and if mainstream theories can be proven utterly wrong every 2 decades, who can say they are not true?

It's the reason why religion still goes so strong, we need to be upfront and say that we don't know anything and the things we know are probably mostly wrong or we risk that history in the future will be taught by the Hancocks.
There are two sides to the coin with not being able to find things - sometimes it is, as you say, because you are bad at searching for things, but sometimes things haven't been made in an easily searchable way. To name just three:
- sections with ambiguous names;
- multiple sections with the same name; or
- information is split across separate sections/pages.

Worst case is that there is detailed documentation but it's all in the comments of twenty different Jira tickets.

I do agree that user docs and dev docs should be separated though, as should specifications and development logs.
The Great Renaming FAQ 1985 – 1988
CD Projekt Red shows off The Witcher 3's 'next-gen update' ahead of launch
If you have perfect description of dedepencies, and you develop alone, ccache is not useful.

But how many times have I seen perfect description of deps?

gcc -o hello hello.c
I dunno, if I was writing the Spotify recommendation engine and wanted to trick people into thinking it was good, I'd do the same thing.
> There aren't enough extraordinary runners to make him rank #1000 or even lower.

I do not want to diminish Bolt's achievements, but how do you know this?
Why not debian binaries? It even had source packages back in 2004.
The commenter was referring to having compilation and linking in separate stages, as is standard in Makefiles to enable Multi-Threaded compilation. As in

cc -c main.c -o main.o
cc -c init.c -o init.o
cc init.o main.o -o final_binary

In that specific setup, ccache does indeed not provide a speedup, since those .o file are kept. The Makefile simply checks if the source file has a younger modification date than the object file and recompiles only if it's older. In that sense the Makefile does in fact cache the results. Once we go beyond single-user and onward to bigger projects, ccache starts making sense.
> Participation trophies are for protecting people from knowledge that there were winners and losers in a competition and that they are one of the losers.

Have you ever seen a participation trophy?

I have some from youth sports. They're a tenth the size of the actual placement trophies. They were not an attempt to fool people.
Thanks. When you retire your old job is not far away.
> As most blogs and websites don't export the full content of their posts, that leads to a mixed reading experience. Quite often, you click the title of a post within a conventional reader only to see that there's nothing there apart from another link to the original post.

It’s the opposite for me. Almost every blog exports the full text. It used to be the other way, but that was in the GReader times.

> The feed reader for people that want to be in control

I find that a questionable headline with no self-hosting option.

And it misses screenshots, I never sign up blind for anything.
But you could run the JVM. Which does the same thing also since decades.

The difference is: The JVM is already proven in production since a very long time.
PHPs model would be okay if they used green threads/goroutines. That way, you can have a far larger number of workers, as most of them are blocked on database connections or HTTP requests to other services.
It's hard catching bugs in go when your eyes tend to glaze over tens of thousands of 'if err != nil' statements causing bad signal-to-noise ratio.

I actually like Go over Rust, but really, really wish the language provided some more ergonomics and fixed several of its terrible gotchas. (the for loop issue for example)

Also wish Go gets RAII some day.
It was my experience that building a native Android project with older NDKs benefitted hugely from introduction of ccache. Especially if you had multiple branches of the same code (release/nightly) that shared a significant code base.

That was pre-Android Studio times. IDK what is the situation now.
Project Solar just raised Series A to effectively cut out the sales rep. I put one in my roof at 1/3 the cost of other companies… even less the cost of Tesla which is priced pretty well.

But Urban Compass initially tried to cut out the overpaid realtor before eventually raising VC to be the largest realtor themselves. Let’s see if Project Solar pivots to become what they have initially set out to replace.
I love how you use the phrase "permacrisis narrative" as if climate change and its consequences are made up constructs. They are not. They are established science at this point. You are doing noone any service by claiming otherwise, even obliquely.
Though, let's be honest, that's not the reason for Valve investing in Linux. Users wouldn't care what's running underneath their games as long as the games run reasonably well.

Back about a decade ago there was real concern that Microsoft might lock away modern graphics APIs behind some sort of entitlement and Valve simply didn't want to be dependent on Microsoft.
On ccache site, there is section "Why bother?", the very first line:

>If you ever run `make clean; make`, you can probably benefit from ccache. It is common for developers to do a clean build of a project for a whole host of reasons, and this throws away all the information from your previous compilations. By using ccache, recompilation goes much faster.
Oh nice! Seems like "send articles to Kindle" is its main selling point?
Have we been getting children’s playthings all wrong?
https://support.torproject.org/misc/misc-4/ links to TODO which says "We've been saying for years not to run Bittorrent over Tor, because the Tor network can't handle the load;" and links to a dead page "Why Tor is slow". I googled that and found https://www.whonix.org/wiki/Why_is_Tor_slow#Misuse_of_the_To... and https://tails.boum.org/doc/anonymous_internet/tor/slow/index... which sound like they were both sourced from that defunct page.

So, if you call it a stupid myth, please provide some credible proof. The Tor developers as well as the maintainers of two highly respected tools built upon Tor disagree with you.
I think that in when using a fixed-width font in a text-only editing program, the assumption is that a tab is "made" (visually) of a number of spaces. That number is a frequent ... debatable setting, in software development teams, but at least it's a number of spaces.

Not being able to visually line up indents made out of spaces vs tabs (in any mix) breaks a very basic usability/interaction pattern, and is really rather bad. :(
Sustainable income being self-employed (4 years and counting). Especially since I had spent about 4.5 years failing to find a way to pay my bills (was burning through my savings).
Hi Richard, thanks for the heads-up.
> I guess the only answer, as some others have mentioned, is a predictable organization system for the documentation, which crucially is actually taught to newcomers.

Precisely - just as you'd on-board new hires with information about codes of practise for development (code styles, nuances of internal git practice, etc.) you ought to at least have a "Contribution Guidelines" or similar doc that sets out how to structure and record information in the docs.
One point he’s making, that many people miss, is that there was a time when people learned to memorize large bodies of information for ready recall. Note the oral traditions of indigenous peoples, the exam system in historical east Asia, or even the way western schools used rote learning until recently.

Socrates is partially lamenting the expected loss of this super ability. And he was right: it is rare to memorize things reliably now since we have so much reference material

I think he underestimated how many ever-changing programming apis we would need to “learn” today — otherwise he might have more sympathy for the use of reference material
Sure. However, I don't see how the exact number of previous mass extinctions is particularly relevant to the idea that we're probably experiencing one right now, and we (humans) are the primary cause.
i'd really wish game mods would be run with WASM. the current solution of downloading native code from third party websites always makes me highly uncomfortable.
You never played football, tag, or even ran track in PE? I'm also not entirely sure where you're getting your comments about his high school stuff from. He won both the world youth and junior championships while in high school, and was the fastest man alive in his age group - a record that stood until quite recently. He ran a 20.4s 200m in 2003 at the age of 17. His ultimate personal best and still world record would be 19.19 in 2009.

Incidentally he also wasn't interested in running. He was much more interested in football, but seeing one kid Bolt across the field led others to convince him to try out track and field.
What does bad bot mean?
That seems like a leap to me. I bet there's plenty of people who could take over that life and rough set of preferences and do a better job of being happy.
So this is all about one pioneering brand: The Crockpot.

Is the subtext that this is marketing - the gifting season is upon us, and the competition from newer brands e.g. Instant Pot, fierce?
Never made that claim. You are an expert on both CLR and JVM... what is your take on WASM?
Because people here talk a lot about extinction, I just wanted to point out that 99% of all species ever to have lived died out. What remained are the one we currently are in company with.

Also, after each mass extinction, there was a massive radiation of new species, for example the Cambrian explosion. No new species without extinction. Evolution, baby!!!
The limit for this sort of exercise is "holding everything in memory". Because training of neural networks require that one updates the weights frequently. An NVIDI A100 has a bandwidth of 2 Tb/sec. Your home ADSL something in the order of 10 Mbit. And then there's latency.

Mind you, theoretically that is a limitation of our current network architectures. If we could conceive a learning approach that was localised, to the point of being "embarrassingly parallel", perhaps. It would probably be less efficient, but if it is sufficiently parallel to compensate for Amdahl's law, who knows?

Less theoretically, one could imagine that we use the same approach that we use in systems engineering in general: functional decomposition. Instead of having one Huge Model To Rule Them All, train separate models that each perform a specific, modular function, and then integrate them.

In a sense this is what is currently happening already. Stable Diffusion have one model to generate img2depth, to generate an estimation which parts of a picture are far away from the lense. They have another model to upscale low res images to high res images, etc etc. This is also how the brain works.

But it is difficult to see how this sort of approach could be applied to very small scale, low contextual tasks, like folding@home.
I’m assuming the lazy downvoters had watched the recent PBS SpaceTime episode that talked about the gravitational lens approach in the context of exoplanet imaging, and just assumed that anything involving earth for lensing was gravity based.
FM radio really is very good quality even still, it must have been spectacular back in the 30s. In fact, it's so good it's stymied the uptake of DAB radio, at least in the UK. Digital radio in fact is often worse as it is so compressed. Funny, they are trying the same 'stealing the frequency allocation' trick again, but delayed for now.

https://www.theregister.com/2020/07/03/analogue_commercial_r...
Sometimes going from one kind of soul crushing to a different kind of terrible can be nice.
Artemis I – Day 8: Orion Exits the Lunar Sphere of Influence
I think you may be confusing gratitude with "toxic positivity". Gratitude is not a weapon wielded to prevent growth, that's more like "toxic positivity", where you must be positive no matter what. Gratitude is about drawing attention to the goodness in life, whether someone is bestows that thing upon you depends on your definition of it.

There are lots of definitions of gratitude, some focus on the social aspect, others the moral aspect[1], but I have found keeping a gratitude journal to be helpful in managing my depression and anxiety, by teaching me that "happiness" and "contentment" are just as valid as "unhappiness" or "sadness".

I understand the feeling of "phoniness" you might feel practicing it, but it gets easier over time. Sometimes I'm grateful my partner made me a coffee. Or I'm grateful that a friend called to check in with me, when I was unwell. Sometimes I'm grateful my pull request was merged. It's about little things.

It's worth mentioning: being healthy, having a family, a high-paying, stable job is not "ordinary bullshit" for many people. I say this not to guilt you into being grateful, but to remind you that you seem to have a lot of goodness in your life.

[1]: https://positivepsychology.com/gratitude-appreciation/
It sure can: https://github.com/GeorgLegato/Txt2Vectorgraphics

It doesn't output color though, only B/W.
The business bros offering 10% to their CTO are probably also lousy product managers too, and know very little about what to build anyway.
Check out https://0xax.gitbooks.io/linux-insides/content/index.html

See also: https://old.reddit.com/r/linux/comments/z26h5h/best_book_to_...
On the bright side, outside of a professional context people might care about the multiple interests and dimensions, and might find it fascinating enough to share experiences together.
>I'm not sure that price is going to be a blocker.

Eh, idk if I want to spend $350 on what is arguably a half baked, subpar Google-Assistnat/Alexa.
How? And please don't tell me you achieve this by redefining the meaning of value in this context, that would be very boring.
I've started seeing things like this recently too. For example I was browsing through a gallery of artwork by an artist and one was blurred because it contained "sensitive material". The warning stated "food" and when I clicked to reveal the image I found that it was simply a photograph of some freshly baked bread.
To be clear, while gratitude does improve happiness, the thing that I find compelling is that it improves actual tangible outcomes (e.g. goal achievement, relationship health, doctor visits, etc.)

Also, expressing gratitude for the things you have does not preclude recognizing areas that could be improved. I am grateful that I am able to walk around and use my body for exercise and sport without trouble; this doesn't mean that I think my creaky knees and back ache are good things, it just means that I appreciate those good things that I do have.
Actually the EU is thinking about exactly doing this.

https://energy.ec.europa.eu/topics/renewable-energy/solar-en...
I am grateful that my family is coming to visit me soon. I haven't been able to see them due to Covid travel restrictions and am looking forward to hugging them and cooking with them.
I used to use it for gentoo locally even, but I don't remember why and I don't think I only used it for stage1. I do remember that even locally it had a huge impact on build times.
Well, given the time frames we're talking about they might lack the remaining lifespan needed to see a project like this through to the end. Definitely an option to get such a project rolling though.

And your second point is valid too - we shouldn't underestimate the internet's ability to give niche subcommunities a chance to flourish even when people are physically far apart.
That would be the relevant example.

Compile-time code restriction seems dumb. Programers can write any program they wish, stopping them for some ideological reason isnt effective. As with all the half-baked metaprogramming people restort to anyway.