skysight-inc/HackerNews huggingface dataset:
I'm curious why krep runs faster with large files in a multithreaded manner?

Naively, isn't IO the bottleneck?

IE, I'd think that loading a file would be slow enough that krep would be IO-bound?

Do you have a typical ratio of IO time to search time on a modern disk and CPU?

What about a producer-consumer model where one thread reads files and creates an in-memory queue of file contents; and a different thread handles the actual searching without pauses for IO?

Edit: If you're truly CPU-bound, another variation of producer-consumer is to have a single thread read files into queues, and then multiple threads searching through files. Each thread would search through a single file at a time. This eliminates the shared memory issue that you allude to with overlap.
I did a quick search and it seems there is neither?

- "SPHEREx relies on an entirely passive cooling system — no electricity or coolants are used during normal operations"

https://www.jpl.nasa.gov/news/6-things-to-know-about-spherex...

edit to add:

- "The telescope is passively cooled to below 80 K in low-Earth orbit by three nested V-groove radiators. An additional radiator cools the long wavelength focal plane temperature below 60 K to reduce detector dark current."

https://arxiv.org/abs/2404.11017v1
At least I usually do prefer to do the EDA plotting by writing and editing code. This is a lot more flexible. It's relatively rare to need other interactivity than zooming and panning.

The differing approaches probably can be seen in some API choices, although the fastplotlib API is a lot more ergonomic than many others. Having to index the figure or prefixing plots with add_ are minor things, and probably preferable for application development, but for fast-iteration EDA they will start to irritate fast. The "mlab" API of matplotlib violates all sorts of software development principles, but it's very convenient for exploratory use.

Matplotlib's performance, especially with interaction and animation, and clunky interaction APIs are definite pain points, and a faster and better interaction supporting library for EDA would be very welcome. Something like a mlab-type wrapper would probably be easy to implement for fastplotlib.

And to bikeshed a bit, I don't love the default black background. It's against usual conventions, difficult for publication and a bit harder to read when used to white.
I parted a 2.56 kWh secondary-battery system at $773.66 for my business' self-driving computer product, but it's not one complete unit or as sophisticated as this product. It seems like a decent price for what you're getting. Especially in terms of space consumed.
No, it was fairly well known a decade or more ago as a very good way to remove bias for how low-effort it is. We were using a variation of it at work back then for an unrelated subject.

How much it's used in political polling I can't say.
Lisp code is an AST. Once that's internalized, the parenthesis fade in the background. Mentally, instead of editing code, you're just arranging the branches. So, when reading, you can usually ignore whole sections as they will evaluate to a single value (side effects are possible, but strongly discouraged)
Ask anyone maintaining even remotely interesting project in C# on GitHub and the answers would have likely surprised you.
> tech companies from the last 10 years that were successful?

Discord, Anduril, CoreWeave, Veho, MoneyBox, Peak, Marshmallow.
It seems that they port the code manually, probably with the help of LLMs.

https://github.com/microsoft/typescript-go/commits?after=dad...
Ask HN: Would you reapply to a company that ghosted you?
Having gone through multiple rounds of interviews at different companies large and small, only to be ghosted is an incredibly frustrating experience. It's happened five times now. Including interviews I thought went really well, and one where the manager verbally assured me I'd be invited back for the next round. Not only is it unprofessional, it's a complete waste of my time invested.

Would you apply to another position at a company that did this? Personally, I have decided to blacklist every company that does this on general principle, not only for the perceived lack of courtesy, but it seems to be a harbinger of corporate dysfunction. What are your thoughts on this practice?
If you think the bad actors are only on one side, you are in a bubble that is feeding you bad information.
Shameless plug: I'm actively working on a similar project, Datoviz [1], a C/C++ library with thin Python bindings (ctypes). It supports both 2D and 3D but is currently less mature and feature-complete than fastplotlib. It is also lower level (high-level capabilities will soon be provided by VisPy 2.0 which will be built on top of Datoviz, among other possible backends).

My focus is primarily on raw performance, visual quality, and scalability for large datasets—millions, tens of millions of points, or even more.

[1] https://datoviz.org/
"Snowflake Cortex didn’t manage to answer any of our questions" :))
I don't mean to nitpick but I'm not arguing that it's burdensome or difficult to remember. I meant it when I said:

> once you know what the rule is I agree that it makes access control obvious

The rule is memorable and grokable, I agree with you! I don't use "silly" to characterize it as the wrong choice, I strictly mean it's a quizzical choice.

Using your hidden file analogy in Unix: if I were to invent a new file system and I decided to use lower case letters to represent hidden files instead of borrowing the "." convention, wouldn't that strike you as a silly choice given the ubiquity of Unix-like filesystems?
Your example is a conceptually simple filter on a single list of items. But once the chain grows too long, the conditions become too complex, and there are too many lists/variables involved, it becomes impossible understand everything at once.

In a procedural loop, you can assign an intermediate result to a variable. By giving it a name, you can forget the processing you have done so far and focus on the next steps.
Hacker News is just 19 years old? I remember my professor at university mentioning it when I told him I am using .net for my project. According to him I should use OS language like Java and recommended me to HN. I thought it is much older. Time flies.

I wonder if he is raeading HN still...
The second pictured person stole women's luggage and took their clothing from the luggage, while working for the President.
The amount of CVEs for C++ apps is off the charts compared to java lol
I'm certain the host heavy lifting is done by numpy which is a python wrapper around Fortran and C. The visualization heavy lifting is done by pygfx/wgpu-py. wgpu-py has C. I think wgpu-py compiles to WASM to run in the browser. More and more packages are taking this route.

[1] https://github.com/pygfx/pygfx
[2] https://github.com/pygfx/wgpu-py
That happened because capital wanted it to happen. Blaming academics is just fascist garbage that avoids addressing the real issue.

Of course capital is saying now it can fix all the problems they caused. I’m sure you believe them too.
I want to say no, but I also will say that I can only remember the name of one that did (plenty of others have, I just don't remember the name).
Thieves are planning an inside job. They forge the surveillance video ahead of time, do the theft and submit the forgery to be timestamped while it's happening.

Also, a lot of surveillance systems are purposely kept offline to prevent them from being compromised, but your system doesn't allow that because they would need external connectivity to get signatures.
Can vouch.

It just works and has low resource utilization.
We won't see eye-to-eye on this, who exactly were the reasonable intellectuals that were going to save us? Maybe Hayek or Mises, Keynes won the day and look where we are.

The past century is defined by Progressivism, from the New Deal and subsequent Great Society, which was hoisted by the World Wars and complete centralization of society into the administrative state run by supposed experts.

At every level it's been experts and intellectuals of all sorts, think someone like Paul Krugman - there's countless others, who we're told have it all figured out and we should simply accept all their demands and if anyone disagrees well they're rubes and what do they know.
It's far from a perfect solution, but I applaud businesses that have tried to improve the situation through packaging changes. IHOP is a stand-out here, in my experience. Their packaging is very sturdy and isolates each component in its own space. I've occasionally been surprised at how hot the food is.

Hah. I went to find a picture and apparently they gave an award to the company that designed the packaging: https://www.directpackinc.com/2018-ihop-vendor-partner-year/
... with all of us.
Yes, but average time to unicorn status has been increasing since the time of Dropbox and Airbnb. I assume you can draw more inferences from companies that are earlier-stage, too.
> Java. Java-Script. What's the difference?

I can't find the Wayback Machine address, but I was going to link to one of the redacted Jerkcity comics where "Johnny from Arkansas" asks the exact question and Rands is happy to help the "American Taliban." Its title was something like "Clean the Dead Moose out of Panel 1"
To answer three of your questions:

- It is passively cooled rather than using an expendable coolant- "SPHEREx relies on an entirely passive cooling system — no electricity or coolants are used, simplifying the spacecraft’s design and operational needs."

- It is a Medium-Class Explorers (MIDEX) mission - Investigations characterized by definition, development, mission operations, and data analysis costs not to exceed $180 to $200 million total cost to NASA.
I think the cost of ground support eats into the budget length. The original estimate for project was $241M, so it was a large MIDEX

- It is in a Polar orbit around Earth at the day-night (terminator) line

https://www.jpl.nasa.gov/press-kits/spherex/

https://explorers.gsfc.nasa.gov/missions.html

https://spaceflightnow.com/2019/02/14/nasa-selects-mission-t...
You don't ever need to "understand everything at once". You can read each stanza linearly. The for loop style is the approach where everything often needs to be understood all at once since the logic is interspersed throughout the entire body.
Of course I would. Companies can have recruiting issues that aren't related to engineering. They can be forced to freeze a hire and not advertise it. It is an irrational decision to do blacklist them.

Who is harmed by blacklisting a company? The company will just hire someone else, and you potentially won't have a salary.
Oh really, pull up your savings/investment portfolio and compare YTD results
Why is this on a site that wants you to sign up for marketing, rather than the GOV.UK petitions site?
Notes on type layouts and ABIs in Rust (2018)
Neat. It certainly makes oncall and maintenance easier! It is likely more resource efficient by e.g. minimizing idle compute, maximizing cache hit rate, etc.
> competitivity
"competitiveness" might be the word you want.
You were saying?

"The French nuclear submarine FS Tourville photographed in Halifax Harbour today.

I'm not sure when a French submarine last visited Canada. I certainly believe this is the first time a Suffren-class boat has been to Halifax."

https://x.com/CDNPolicyHawk/status/1899184320092233883
Yeah. If I remember it correctly, you need to compile the GC to run on WASM if the GC extension is not supported.
The closing marks are way too positive and should say: if you need a machine that has working audio, webcam, lasts more than 5.8 hours and does not crash when compiling Rust code, this machine is not for you.

We’re getting closer to great working ARM64 laptops, but we need to get past the issues listed under The Bad and The Ugly.
Every system I'm familiar with that used liquid nitrogen to cool the IR instruments has had a operational lifespan based on the coolant. JWST is one such. "The coolant will slowly vaporize, limiting the lifetime of the instrument from as short as a few months to a few years at most."[0]

[0]https://en.wikipedia.org/wiki/James_Webb_Space_Telescope
Percolation Theory [pdf]
As another comment mentions, SPHEREx is passively cooled. But fwiw, plenty of infrared space telescopes use consumable coolant:

> Notable infrared missions that carried consumable cryogen include IRAS (1983), ISO (1995–1998), Spitzer (2003–2009 in cryo mode), Herschel (2009–2013), WISE (2009–2011 in cryo mode), and Planck (2009–2013). Each relied on a finite liquid helium (or solid hydrogen) supply to keep detectors cold and reverted to a warmer operating mode or ended once their coolant was depleted.
Aren't you missing the entire point of exploratory data analysis? Eigenfaces are an example of what you can come up with as the end product of your data exploration, after you've tried many ways of looking at the data and determined that eigenfaces are useful.

Your whole third paragraph seems to be criticizing the core purpose of exploratory data analysis as though one should always be able to skip directly to the next phase of having a standardized representation. When entering a new problem domain, somebody needs to actually look at the data in a somewhat raw form. Using the strengths of the human vision system to get a rough idea of what the typical data looks like and the frequency and character of outliers isn't dumping the job of exploratory data analysis onto the reader, it's how the job actually gets done in the first place.
The Scientific Controversy Tearing Families Apart
I've written parsers and compilers in Rust. I used DAGs in a fairly standard way for stuff that needs it like references. I also work on a complicated VM with a custom memory allocation strategy professionally. Some of my other Rust projects include: a frontend web application that runs in the browser, a client library that has facades for Node, Python, and Ruby, and a Kafka Consumer that handles ~20 GiB/second of data throughput with relatively modest resourcing.

What he's saying here doesn't make any sense. It sounds like they threw in someone who doesn't know Rust at all, didn't bother to ask any questions, didn't reference any existing code, into trying to write custom memory management strategies and data structures and then bounced off the surface. That isn't how you do things in any language, it's bizarre and sounds like Rust was set up to fail. I wouldn't expect this scenario to succeed in any language on a non-trivial project like the TypeScript compiler.

What's even more bizarre is TypeScript actually has better support for ADTs than Golang (which is especially impactful when writing things like type checkers and compilers). I don't even _like_ TypeScript and I can see that. I've written 5-figures-ish of Golang for some work projects like a custom Terraform provider and it's horrific to model just a JSON schema in Golang's type system. Some of the problem is Hashicorp's terrible SDK but not all of it by any means.

Usually the problem is someone just not knowing how to write Rust. The modal subjective experience of writing Rust code as an experienced user is "Python with nice types, async, and executes fast." If you need to handle stuff with weird non-deterministic lifetimes you can use stuff like `slotmap`. If you need DAGs, use `petgraph`. I think pitching Rust as a "low-level systems language" might be misleading some people on how ~95% of Rust users are actually working in the language.
“Nice” doesn’t mean “well suitable for writing a compiler in”. It’s strange to think that all languages should be equally good for writing all kinds of things, and choosing a web language for a non-web task is doubly strange.
That's the gist of his whole talk – that doing things "the UNIX way" (which can be defined to various degrees of specificity) has been cargo culted, and that we should reexamine whether solutions that were pragmatic 50+ years ago is still the best we can do.

The specific reason I mentioned it was because his initial example was about how much more ceremony and boilerplate is needed when you need to pretend that USB interfaces are actually magic files and directories.
> This spreadsheet was the primary data file

>> - as crappy as that is, thats pretty common.

No. This is not a common practice. An Accounting / ERP system with full audit trails and data entry compliance is. Exporting data from such a system into Excel for analysis is. But Excel as the [re-reads] Primary data file is not a practice at all.
The Agents SDK they linked to comes up 404.

BTW I have something somewhat similar to some of this like Responses and File Search in MindRoot by using the task API: https://github.com/runvnc/mindroot/blob/main/api.md

Which could be combined with the query_kb tool from the mr_kb plugin (in my mr_kb repo) which is actually probably better than File Search because it allows searching multiple KBs.

Anyway, if anyone wants to help with my program, create a plugin on PR, or anything, feel free to connect on GitHub, email or Discord/Telegram (runvnc).
It's really good to read those stories.

I applied twice (once alone, once with a really amazing co-founder) and it felt like applying to a large faceless corporation, both times.

Automatic rejection after 30 days with zero feedback.

Kind of sucky, to be honest.

I'd just love to hear that the idea was shit, the application was shit, that my face was ugly, or anything, really.
I have built myself a much simpler and powerful version of the responses API and it works with all LLM providers.

https://github.com/Anilturaga/aiide
The Spitzer used passive cooling, but only as a method to reduce the amount of coolant required. It still needed a coolant.

"One of the most important advances of this redesign was an Earth-trailing orbit.[1] Cryogenic satellites that require liquid helium (LHe, T ≈ 4 K) temperatures in near-Earth orbit are typically exposed to a large heat load from Earth, and consequently require large amounts of LHe coolant, which then tends to dominate the total payload mass and limits mission life. Placing the satellite in solar orbit far from Earth allowed innovative passive cooling. The sun shield protected the rest of the spacecraft from the Sun's heat, the far side of the spacecraft was painted black to enhance passive radiation of heat, and the spacecraft bus was thermally isolated from the telescope. All of these design choices combined to drastically reduce the total mass of helium needed, resulting in an overall smaller and lighter payload, resulting in major cost savings, but with a mirror the same diameter as originally designed. This orbit also simplified telescope pointing, but did require the NASA Deep Space Network for communications"[0]

[0]https://en.wikipedia.org/wiki/Spitzer_Space_Telescope
Go has buildmode=c-shared, which compiles your program to a C-style shared library with C ABI exports. Any first call into your functions initializes the runtime transparently. It's pretty seamless and automatic, and it'll perform better than embedding a WASM engine.
As long as these gravel-filled piles are deep enough to avoid frost heaves, they should be fine.
Very cool! I think I'll need to sit down with this for a while before I fully understand it but it looks like something special. New interface/format concepts for basic computing tasks such as file storage, sharing etc. are in severely short supply considering the shortcomings of our current paradigm. I appreciate the clear and minimal homepage too.
and before that, there were literal community message boards. corkboard at the grocery store. people gossiped before there was Facebook. the only real innovation was going from verbal communication to the written word, but now with TikTok, we're back at verbal communication, just with a lot of extra steps. there's no other innovation that's happened in the world since writing, isn't there?
I was interviewed by The Atlantic about the "shaken baby syndrome" scientific controversy. More context here: [1, 2]

[1] https://cambridgeblog.org/2023/05/a-journey-into-the-shaken-...

[2] https://news.ycombinator.com/item?id=37650402
> You are told explicitly at the beginning what the type of the result will be

I would argue that's a downside: you have to pick the appropriate data structure beforehand here, whereas .distinct() picks the data structure for you. If, in the future, someone comes up with a better way of producing a distinct set of things, the functional code gets that for free, but this code is locked into a particular way of doing things. Also, .distinct() tells you explicitly what you want, whereas the intention of set() is not as immediately obvious.

> There are no intermediate results to think about

I could argue that there aren't really intermediate results in my example either, depending on how you think about it. Are there intermediate results in the SQL query "SELECT DISTINCT Author FROM Books WHERE Books.PageCount > 1000"? Because that's very similar to how I mentally model the functional chain.

There are also intermediate results, or at least intermediate state, in your code: at any point in the loop, your set is in an intermediate state. It's not a big deal there either though: I'd argue you don't really think about that state either.

> and no function call overhead

That's entirely a language-specific thing, and volatile: new versions of a language may change how any of this stuff is implemented under the hood. It could be that "for ... in" happens to be a relatively expensive construct in some languages. You're probably right that the imperative code is slightly faster in most languages today, and if it has been shown via performance analysis that this particular code is a bottleneck, it makes sense to sacrifice readability in favor of performance. But it is a sacrifice in readability, and the current debate is over which is more readable in the first place.

> a single pass over books

Another detail that may or may not be true, and probably doesn't matter. The overhead of different forms of loops is just not what's determining the performance of almost any modern application. Also, my example could be a single pass if those methods were implemented in a lazy, "query builder" form instead of an immediately-evaluated form.

In fact, whether this query should be immediately evaluated is not necessarily this function's decision. It's nice to be able to write code that doesn't care about that. My example works the same for a wide variety of things that "books" could be, and the strategy to get the answer can be different depending on what it is. It's possible the result of this code is exactly the SQL I mentioned earlier, rather than an in-memory set. There are lots of benefits to saying what you want, instead of specifying exactly how you want it.
This is a very sweeping generalization.

The party that got voted out was mired in corruption scams and their premise to power was fighting against corruption.

Politics in India can be a lot more nuanced than you claim it to be. What you are saying is one part of the equation.
War had always been horrible. Drone attacks are no exception.

No one is gonna give up the option, given their effectiveness.
Notably those were all far-infrared telescopes, which need even lower temperatures (liquid helium) to escape thermal noise.
Diskonaut: Terminal Disk Space Navigator
Yet people wouldn’t write a Fortran compiler in Fortran, or a MATLAB compiler in MATLAB.
what i'm being is sarcastic in the face of someone who refuses to step back from my single tablet to everyone else in this thread mentioning the same thing

and like i said, you're welcome to my tablet, i'll mail it to you, then you can be certain you are correct.
The GC extension is supported within browsers and other WASM runtimes these days - it's effectively part of the standard. Compiler developers are dropping the ball.
No sane optimizer is going to emit the functional code as a gajillion function calls.