skysight-inc/HackerNews huggingface dataset:
The core idea with these compressed columnar "big data" formats is that they minimize storage accesses. Nothing else really matters as much. If you're gonna have to load a sizable chunk of the file to get to the bits you need, the format you store it in starts mattering less.

What this gets right: Part of the reason you want to store columns together is that similar values compress well, so you could reduce your IO: smaller files are faster to load into memory. However in many cases (e.g. Arrow, Parquet) lightweight compression formats are preferred here, e.g. run length encoding (1,1,1,1,1,1,5,5,5,5,3,3,3 -> 6x1,5x4,3x3) or dictionary encoding (if your column is enum-like, you can store each enum value as a byte flag) because they can be scanned without decoding, amplifying your savings.

What it misses on (IMHO):
- There's a metadata field but it doesn't contain any offsets to access a specific column quickly. So if you have 8 columns of 2GB each, to just get to the 7th column you have to read 12GB first which is quite wasteful. If you store just an offset, you could be reading a handful of bytes. Massive savings.
- Within each column, how do you get to the range of values you want? Most columnar formats have stripes (i.e. stored in chunks of X rows each) which contain statistics (this stripe or range of values contains min value A, max value B) that allow you to skip chunks really fast. So again within that 2GB you have to read not much more than you strictly have to.

If this reminds you of an on-disk tree where you first hop to a column and then hop to some specific stripes, yeah, that's pretty much the idea.

-----

Sidenote: I've generally concluded that "human readable" is only a virtue for encoding formats that aren't doing heavy lifting, like the API call your web app is sending to the backend. Even in that case, your HTTP request is wrapped in gzip, wrapped in TLS, wrapped in TCP and chunked to all hell. No one complains about the burden caused by those. So what's one more layer of decoding? We can just demand to have tools that are not terrible, and the result is pretty transparent to us. The format is mostly for the computer, not you.

When I hear about stuff like terabytes of JSON just being dumped into s3 buckets and then consumed again by some other worker I have a fit because it's so easy and cheap these days not to be that wasteful.
Hopefully this is something we'll be able to search on and find out what was really being done.

It would be nice if that database actually had links to any of the research papers that resulted.
Looking at the video, his apartments wiring probably dates to the 50’s or so. Which would be the norm here. Hard to do worse.

Most landlords here are pretty low maintenance (both ways) as long as you pay on time and willing to bend. Did not blink an eye when I asked to go roof WISP and drilled my own hole next to the cable line. No roof penetration so they didn’t care.

I’m thinking of asking about a weighted panel to cover the 4-9pm PG&E quadruple pricing via a LFP bank.
I'd also like some sense of zoom. Like graticules of 1k or 10k; they could even change. If it makes the screen too busy, make it just ticks on the edges.
I wish headlights + taillights were required to be always on, or at least on by default.
Also: A man openly celebrating International Women's Day in 1981, while not impossible, is certainly unlikely. Similarly, "Nice Day With Liz" on a Sunday, seems like mentioning a platonic event. A man opening having female friends and meeting them 1-to-1 on a Sunday in 1981? Again, not impossible, but another point suggesting a female author.
> I don’t know if any of the implementations are up to the mark. FSD isn’t.

It is very clearly called "Beta" software.

You can agree or disagree if Tesla should be allowed to put that out there, but I don't think it should come as any surprise that software labelled as beta is not 100% flawless.
Groups of spinal cord neurons that allow for brain-independent motor learning
It’s a bit weird to me to mention compose and kubernetes without bringing up Docker Swarm which feels like a nice middle ground. We actually run it at my very small SaaS company and it seems like a good fit for us though it’s still relatively early days for us; it’s been a very slow transition.
It has more to do with a hidden class structure below your average white family, working for much less, providing goods and services at a discount. Inflating the means of the average white family in a way that wouldn't be possible today because we're not as purposefully racist or sexist as we used to be.

In order to make a better comparison one should compare what was affordable to the average black family in the 1960s (or whatever previous time period you choose for which we have figures) to today.

There's other factors like the aftermath of WW2, the real and "soft" power the US held back then in comparison to today, and the role women played in the economy.

Today's world is a lot different from then but the truth is that the wealth and power we've amassed since then should have us in a better position, not worse. The factors at play in today's economy are not benefiting the common man nearly as much as they should.

We're beginning to look more and more like a post-scarcity economy every day yet our laws, regulations, and culture are not evolving with the times. The fact that land and homes are vastly inflated compared to regular every day goods and even the "durable" kind is strong evidence that our entire economic system stands on precipice of great change.

Either we're just getting started on a great collapse or a great sea change in politics and economic systems. I very much doubt things are going to go back to the way they were... Ever.

We as a society can decide to spread our prosperity around or we can increasingly suffer for the lack of it by continuing the status quo.
VOSK Offline Speech Recognition API
They buried the single-command show-don't-tell quick start examples on their website:

vosk-transcriber -i test.mp4 -o test.txt
vosk-transcriber -i test.mp4 -t srt -o test.srt
vosk-transcriber -l fr -i test.m4a -t srt -o test.srt
vosk-transcriber --list-languages

https://alphacephei.com/vosk/install#usage-examples

More information on their website:

1. Supports 20+ languages and dialects - English, Indian English, German, French, Spanish, Portuguese, Chinese, Russian, Turkish, Vietnamese, Italian, Dutch, Catalan, Arabic, Greek, Farsi, Filipino, Ukrainian, Kazakh, Swedish, Japanese, Esperanto, Hindi, Czech, Polish, Uzbek, Korean, Breton, Gujarati. More to come.
2. Works offline, even on lightweight devices - Raspberry Pi, Android, iOS
3. Installs with simple `pip3 install vosk`
4. Portable per-language models are only 50Mb each, but there are much bigger server models available.
5. Provides streaming API for the best user experience (unlike popular speech-recognition python packages)
6. There are bindings for different programming languages, too - java/csharp/javascript etc.
7. Allows quick reconfiguration of vocabulary for best accuracy.
8. Supports speaker identification beside simple speech recognition.

https://alphacephei.com/vosk/

It's pretty fast and accurate. Should get several times realtime transcription speeds, on CPU using one of the "small" models:

https://alphacephei.com/vosk/models
How on earth does it take 4 hours to explain? I'm sure it's a lovely video but anyone care to tldr?
I have a friend who works for the IRS. They basically can't spend more than a minute per client. Its basically "okay cool" move on. They gotta do like 40 cases a day.

The tricky part is that you're basically playing a lottery. If they audit you (and there's a very strong chance that in your lifetime it won't happen), you're likely to hit a bunch of fines. And if you know someone in the IRS they'll just tell you what you owe. And if they don't, basically nobody gives a shit.

That's what it is. Is the risk of a failed audit that much of an issue? Or not. And if you don't have the money / assets to pay the fines, they'll likely just give you a loan or maybe a reduced amount. The IRS rather get some money than no money.

Its... weird.
you do realize that it is literally first time that so many ballistic missiles are shot together and nobody even knew till today if it's possible in general to defend against such attack ?
The third prong is vague and the attack depends on whether the applicant is a founder or an employee and whether the applicant's endeavor is technical or not. Almost always there's a basis for those with PhDs to qualify for an NIW.
Most articles about public speaking, including this one, have interesting advice, but they miss the point on the most important factor in a presentation: the content.

If you do everything that the author recommends, but you don't have anything valuable to share, your talk will flop. So make sure you have something important to tell people.

What I'd really like to read is an account of how to make interesting content, how to arrive at great conclusions while working on great problems.
Not sure if related but I often imagine a map of concepts when trying to understand or remember things.

Linux kernel, software skills, career progression. All maps.
in a way, this framework represents how far off the reservation you're going. You're either building in the known, challenging existing assumptions, or creating new landscape.

for apple:

building in the known: Mail

Challenging existing assumptions: airpods

new landscape: iphone
People used to say backdoors and security bugs were less likely on Linux because 'open source'.

Sociological theories of computer security are magical thinking.
You haven’t mentioned what kind of laptop they own.

Everyone I know who owns MacBooks uses them for: photography editing, video editing, graphic design, programming.
Ah, I miss it so much. Adding to the conversation on touchscreen keyboards, yes I know that technically they can be faster but the frustration of dealing with auto-correct and miss-typing constantly ends up being a slower, more frustrating experience. There was no better typing experience than my BB.

Still today, I don't use the phone much except for SMS and core productivity stuff. Mind you, I have the benefit of working at a computer.
The average household also includes empty nesters and people who have moved out yet haven't started a family yet. The average number of children per family in the 60s was 2.3, which means quite a few families had 3 or 4 children.
> 1960 the average single family home was 1300 square feet, now it’s more than double that

It's the same house. They put drywall up in the basement and walls around the porch and call it a sun room.
The latest version 12 FSD beta seems significantly better, to the extent that experience with previous versions is less relevant. There's no real reason to expect these systems to get worse, they'll only improve. Today's videos versus those of even just a year or two ago shows huge progress.

Not better than doing yourself today in general, but it will be in more and more situations. https://www.youtube.com/watch?v=fpoXr_z_6a4 is illustrative of the current level of ability. A good example where the self-driving compares favorably with poor human driving is at 10:15 in the video.

There are some pretty major gaps (for example, limited ability to reverse and read signs), but I wouldn't bet on them not being resolved in the next five years.
I skipped paying mine for a year, when I was younger. I had done an early withdrawal on a 401K to finance a vacation to Europe, and I knew I owed, but I didn't have any money and I didn't want to deal with it, so I just didn't. Nobody ever bothered me about it. I squared up with the IRS the year after and they put me on a payment plan, no trouble.
I think the argument is that inflation is complicated by globalism.
I don't think so. The "benefit" isn't a monetary on; the obligation is to help the H-1B worker return home so if the worker is already home, there's no obligation.
I've flagged my share of stories on the topic, not because I consider myself to be on any "side" of the issue – other than the "side" of humanism – but because any thoughtful discourse on the topic is damn near impossible, and endless repetition of the same tired old stuff adds nothing to the site.
> Israel's nuclear capacity

shhhhhhhh we're not supposed to say the secret part out loud!
But the LLM predicts the output based on some notion of a likelihood so it could in principle signal if the likelihood of the returned token sequence is low, couldn’t it?

Or do you mean that fine-tuning distorts these likelihoods so models can no longer accurately signal uncertainty?
> Deep sleep is not improved by consistent sleep/wake times, and adjustments to noise/temperature/light conditions?

Deep sleep definitely benefits from consistent sleep/wake times. Your body tries to play catch up if you break from your normal rhythm but it’s not entirely effective.

I think the parent comment might be exaggerating the futility of sleep hygiene because they have a product to sell. :) Proper sleep timing, duration, and sleep hygiene are inarguably valuable for proper sleep architecture.
The Curve 9320 is the best phone experience I've ever had in my life. It fell short on web, but at the time I didn't care. BB10 phones that came later were more of a swiss army knife adapted to modern needs, but they felt somewhat more sluggish.
"She said a more effective measure would be to aggressively curtail fossil fuel extraction and their use in Canada and around the world."

And not simply swapping for more-renewable energy sources, because there's only so much sunlight hitting the earth, and we're not the only life here.

Using less, far less if you're a spoiled American like me, is the way to go. We can lead healthy, meaningful lives on far less energy.
Managers have different authority.

Also, some managers were born before mobile phones were ubiquitous.
Only if you think it's possible to get all of the information in someone's head out of their head without consuming significantly more of their time and energy.
Because they're sustaining record quarterly profits.

Ask the average person: "would you rather pay $200 more for the base model, but get 8gb more ram?" The person would respond "what is ram?"

People say they hate how media always portrays negative news. But bad news always gets the most views.
I'm honestly surprised they didn't roll out their own CA.
One of my favorite books! If I remember correctly, Grant had no aspirations of being a soldier, and was not one of the best students at the Military Academy, and even after he graduated and became a soldier was still...green for a long time.

One thing I found intriguing was how gentlemanly he was able to conduct himself in war, when dealing with his own men (who sometimes barely had any training at all) and when dealing with the enemy.

The manner in which he writes his memoirs, his manner of speaking, is much different than how I speak, but is pleasant to read. There are sections that I reread many times. If I recall correctly he had no confidence at all in his own writing ability, and only began writing to pay the bills (writing short stories about the war for the newspaper).
I like your idea of storing columns as JSON arrays. I might play around with that. Thanks for giving it a look.
> yes I think it way quicker per ship, because ocean liners in particular wanted fast turnaround times

But container ships also want fast turnaround times. This is not a convincing argument.
Unless I'm misunderstanding, Arrow solves the data representation on disk/memory, both for pandas and polars, while I'm writing about type inferencing during static analysis, which Arrow doesn't solve.

Having a type checking system respect arrow schemas is indeed our ideal. Will polars during mypy static type checking invocations catch something like `df.this_col_is_missing` as an error? If so, that's what we want, that's great!

FWIW, we donated some of the first versions of what became apache arrow ;-)
You have to also treat write ups like this as mostly marketing for the firm. There was a discussion recently about A16Z (https://news.ycombinator.com/item?id=39901289) that made me think about that angle here as well. All these frameworks may have useful information but they are also simplistic. I think this is marketing for their Arc program, which is a selling point for early stage startups to consider Sequoia. In other words, this is about improving their deal flow. It also probably helps filter the flow, since now anyone seeking early stage funding from Sequoia will do the work to try to fit to this framework before wasting anyone’s time, and their pitch will also be easier for Sequoia to understand and evaluate.
You only processed 500m tokens, which is shockingly little. perhaps only 2k in incurred costs?
I don't believe so. The only exceptions are for those under 14 and over 79.
One maybe a stupid question. If your thinking is visual, wouldn't you have to close your eyes to be able to think? Or how can you still see? Is it like augmented reality?

Actually since I think, I think by hearing a continous record of text, when I do that, it does make it hard to listen to people. I have to attune that record exactly to what people are saying and it is something I have had challenges doing my whole life. Then I have to fight, put effort in, to be able to keep it on track. I prefer watching films with same language subtitles for that reason. Then I read the subtitles and doing that my thought track is somehow in sync. But it is impossible to keep the thought track in sync with realtime voice. I could only repeat it out of sync.

If people had subtitles, on top of their heads as they were talking I could pay attention much better in social situations.
https://www.youtube.com/watch?v=YsXCVsDFiXA&t=3228s
It makes sense to me now, but the first time I heard the term it brought to mind Wreckx-n-Effect. I am glad there was no connection.
(By 'people', I mean me and also the person who introduced me to Linux.)
Also seems like the stopped car could emit a signal that self-driving cars could pick up.
But why not 3x12hr shifts rather than (what appears to be from outside perspective), 7x12hr shifts?
>Attractive women are 'hit' by this particularly hard - all those guys trying to get them to bed, get another score, the curve is exponential with beauty.

Bill Bradley's Life on the Run discusses this. Bradley was the

* #1 high school basketball player in the country

* #1 college basketball player in the country at Princeton

* Rhodes Scholar

* NBA player

* US Senator

* ran for President

From high school on, Bradley avoided the many, many women who wanted to date him just because he was a celebrity. In Life on the Run, written during his NBA career, he wrote about how such experiences gave him an understanding of what life is like for beautiful women: "the unnaturalness of being a sex object".
I'm a SWE by day. But on my personal Mac all I do is have a tons of Chrome tabs open, Discord, Apple Notes and Spotify. Once in a while I'll compile and run random projects off GitHub (nothing too heavy) for fun.

I can get by with 8GB on the M1+ laptops.
An absolute pittance compared to what Elon Musk is able to extract from US taxpayers to fuel his own personal wealth.
> I mean, if you're writing Erlang/Elixir code, then you kind of do have "classes, instances, and ancestors"

and

> In short: I think it's only an accident of history (of Elxiir copying Erlang's delegate-module approach to receive-selector "inheritance", instead of going for a macro-based approach) that held Elixir back from blossoming into an ecosystem of "stupid OOP tricks" that are just as messy and ridiculous as Ruby's.

Directly contradict each other unless you ignore what I wrote.
Yes this does seem to be the culture, at least for doctors working towards the more competitive specialties (such as surgeons). Things get much less hectic once they reach “consultant” at age 35 or so, but it’s an absolute grind until then. With no guarantee of success! If you see your GP with a bunch of extra paper on the wall, chances are they are one of the many that didn’t win the game.

What’s a good alternative though? These are high paid, high status, jobs for life. Of course it’s going to be as competitive as all heck to get them. Do we shame Olympic athletes for the dedication and sacrifice they must provide to obtain their goals? I’m not saying the current system is great, but any alternative could well be worse.
In the engineering world outside software, our cad tools generate rich interactive functionality into PDFs, including but not limited to 3d models for those doing mechanical work.
I appreciate the effort you put in the comment, and úsed sources rather than just voicing unsubstantiated criticisms.

But still, the mentality in this thread made me double-check if i'm on hacker news or bureaucrat news.
I liked this analysis a lot.

However, OpenAI is a bad example of the future vision paradigm. In fact it fits 1/6 of the characteristics they identify in this paradigm. A better example would be something like Tesla.

I would say OpenAIs backing of the GPT model family in itsself is a better example of the future paradigm than anything to do with OpenAIs founding or non profit status that sequoia discuss. Going all in on GPT gave them a big headstart (although google should have been fighting much earlier if they had listened to internal researchers).
You should see the "0.5 A presses" video: https://www.youtube.com/watch?v=kpk2tdsPh0A
I don't think old age is a factor; the issue is more when someone very young applies for a work visa, such as a prodigy applying for an O-1, because there is general skepticism about the applicant's qualifications/achievements.
PMF to someone who never experienced a true PMF is grossly simplifying a product.

There is so much execution and luck in every step to get there even if you have the right idea
I disagree, I find the total keyboard experience much better on Android.

To each their own, obviously. But... yeah, IMO it's pretty bad on Apple.
...or just being in the hospital environment for a month. The older theory they mentioned here:

> "The earliest twin studies really helped to debunk this theory that autism was caused by parenting," Morris says. Under this theory, moms took the brunt of the blame, supposedly for being "cold and distant and detached from their child."

...might not be entirely wrong. They've since shown genetics is a factor, and I'll ignore the institutionalized misogyny part, but being in a hospital for a month probably did mean having much less parent/care-giver interaction/touch during a critical period, and who knows how much of an effect that could have.
You still haven’t addressed the greater point. Instead you are fixating on the accuracy of least important number in the point the original commenters post. I suspect it’s because it’s the only one you can pick at. The point being can the average single income household of 3.x in 2024 afford house, car, groceries, etc at the same ability as the average single income family household of 3.x in 1963?
>The memes of Arby’s 5 for $5 becoming 4 for $10 are more informative than the CPI numbers. Don’t let the shock at the grocery store wear off – it’s real and painful despite what the news tells you.

"The thing I have noticed is when the anecdotes and the data disagree, the anecdotes are usually right. There's something wrong with the way you are measuring it". —Jeff Bezos <https://sports.yahoo.com/amazon-ceo-jeff-bezos-explains-2123...>
Those seem like different things, I thought Ansible was more for configuring individual boxes.. then you would use something like Helm to configure Kubernetes.

Tech stack at my last workplace was insane like that. To deploy, they woud have a Buildkite job would launch Terraform, which would then be configured by Ansible, then k8s would deploy on the cluster. I think they could have avoided all this ess k8s and probably Terraform and gotten away with one big box and shell scripts (which they had plenty of anyway), and used swarm/compose, but what do I know.