skysight-inc/HackerNews huggingface dataset:
Hello David, and thank you for your comment, analysis, and the issues you opened. I will get to them all.

- SQLite tooling: You will not need it unless you are debugging something, then you can change the header or just use the `--activate-other-options --transform-to-sqlite3` parameter to transform a Pack file to SQLite3, and use the `--activate-other-options --transform-to-pack` to go back. This way, you get a true SQLite3 database that you can browse as you wish. For most people, mixing Pack with SQLite was just a call for problems for the SQLite team (imagine people coming and asking to fix their Pack file from the team; that would not be fair) and a harder future for Pack to update.

- Metadata is not stored in Pack. I don’t want the metadata of my machine attached to a file. It’s a never-ending nightmare to match source and destination OS metadata. There will always be something missing, and Pack tends to get everything perfect or nothing. Storing metadata adds extra weight that most users don’t care about and complicates the ability to store other types of data alongside files. It may get added as an option in the future if many people need it.

- There is a version field. It is currently in Draft 0, and it is written using a custom VFS. Look here for more information: https://github.com/PackOrganization/Pack/blob/main/Source/Dr...

- All future versions of Pack must handle previous versions and must only write the latest version. So any files created right now (Draft 0) will be read correctly for ever to come.

- Each Draft proposal will get its own version, and if it gets final, it will be set to final.

- Two byte after 'Pack' header in little endian as (1 (Draft) shl 13 + 0 (version 0) = 8192). Final would be 0, so the first Final version will be 0 shl 13 + 1 = 1. and the second will be 2. It is by design, so any Draft version gets a higher number, preventing future mixups.

- 8 MB chunks are the default; Pack may choose smaller or bigger (16 MB for many small files or 32 MB for Hard Press).

- Random access is proper as unpacking steps take into account what you want and decompress a Content just once for many neighbouring files. But even for reading just one file, here is an example: if I want to extract just a .c file from the whole codebase of Linux, it can be done (on my machine) in 30 ms, compared to near 500 ms for WinRAR or 2500 ms for tar.gz. And it will just worsen when you count encryption. For now, Pack encryption is not public, but when it is, you can access a file in a locked Pack file in a matter of milliseconds rather than seconds.
Right, so I want a list of results where I can assess the information from each in the context of the source it came from, its reputation, etc. And other metadata like how old the article is.

It's a non starter to have that critical context stripped out and be given an opaque answer by an AI without having seen the underlying information in the context it originated from.
Because there’s nothing out there but swarms of paperclip optimizer ai drones, converting all the universes mass into paperclip fueled war machines for when they encounter a competing paperclip swarm
Same in the UK fwiw. Medicine is socialized, but not dentistry.
Thanks so much for building this. Very cool. In addition, making clear that the feeds are embeddable in Obsidian is just great. Will see how well this searches and will start using today!
That is not a very compelling argument.
Thank you so much for the kind words, refreshing.

Here is the latest sqlar result on Linux source code on the same test machine in warm state:

sqlar: 268 MB, 30.5 s

Pack: 194 MB, 1.3 s

Very good result compared to tar.gz. And much better than ZIP, considering sqlar gives random access like ZIP and unlike tar.gz.
I considered sqlar as a proof of concept, and it inspired me to create Pack as a full solution. I always agreed with the great drh (creator of SQLite) points about SQLite as a file format, and Pack is a try to demonstrate that.

I made Pack to give people a better life (at least behind their desks), and as you do, I hope people get to use it and find it useful.
Yeah, I'm disagreeing with you. It's not CO2. I won't say it's impossible, but it's pretty implausible. We already know how bad leaded gasoline was at poisoning an entire generation. Modern cars are cleaner, but miles driven keeps increasing and the amount pollutants being emitted are poisoning people.
What are you using to index the blogs and podcasts?
Very nice, I've been doing some similar things at work to help map out a data warehouse migration project.

Have you considered outputting to a MermaidJS format?

https://mermaid.js.org/syntax/entityRelationshipDiagram.html
What's the value proposition for TypingMind as a commercial product ($3500 to run locally for 5 seats)?

But let me contrast that last "native app" with Machato and MacGPT:

== Machato ==

Machato is feature-full for system prompts and transcripts, connecting to to OpenAI, Claude, and any "server" endpoint that's OpenAPI API compatible, and surfacing parameter and token settings per conversation right on your text entry bar. You can also point a given conversation to a local ollama endpoint such as Mixtral 8x7B and it works as well.

The best feature is the selective forking and suppression of exchanges within conversation threads.

https://untimelyunicorn.gumroad.com/l/machato

== MacGPT ==

MacGPT is highly integrated throughout MacOS, and works with either OpenAI key or a ChatGTP Pro login. It's quite similar to BoltAI mentioned elsewhere in this thread, but in addition to the OpenAI key based mode, also works with a ChatGPT Pro subscription in a ChatGPT web UI pop-up.

https://www.macgpt.com/
I also got a C in my cs relational databases class. I also got more out of that class than just about any other in CS. I'm still kind of mad about it though. We had a group project assignment and our group had far-and-away the best project, of which I did 90% of the work. My bad grade came down to just one thing: the class required us to use Oracle and I refused on moral grounds.
Krustlet implemented the Kubelet API rather than the containerd-shim API. That means that with containerd shims, you can use the default K8s kubelet, containerd, and extend your containerd runtime with a shim for executing Wasm. Kubelet is much higher in the stack and requires much more to implement it and everything below it in the stack.
If you're interested in what it would take to put relational databases back on the rigorous footing of relational algebras, the Third Manifesto is a good place to start.

https://www.dcs.warwick.ac.uk/~hugh/TTM/DTATRM.pdf

I find it somewhat sad that an implementation of a database with a proper D language hasn't broken out and become a ubiquitous tool for the profession. There were some proprietary versions shortly after the manifesto's publication, but it never caught on.
There is I believe a specific context where removal is good and others where that is not the right measure

I'm working on some java 8 cide that was written by folks still living in java 1.4 land, lots of opportunities there to reduce 20 lines into 3 (plus those devs kinda insisted on doing things in painful and unnecessary ways). In parallel, I'm working on a Greenfield project. Even on the Greenfield, there are times when negative code added is good. Not all programmer activities are equal

It's like measuring gas consumption during emergency braking, during acceleration, and during normal highway cruising. Sometimes it firs, or kinda fits, other times it's just the wrong metric. Programming is not a homogeneous activity
As soon at the Doubleclick CTR drops they'll be reversing this change like they did on the recent change to stop profiling anonymous YouTube users.
So they're formalizing techno-feudalism in to law. Human beings cannot exchange value with each other. The only way to exchange value is if a third party (corporation) owns your currency and gives you conditional access to it. Big yikes.
My nephew is on the spectrum (fairly severely). His mother and father are obsessed with air quality and used filtration and forced ERVs for the entire pregnancy and upbringing. Please do not make such strong medical claims without any scientific backing, it is quite harmful to kids with autism and their parents who are already overwhelmed with quackery.
» This isn't a success story, it led to mass layoffs.

Did the company valuation go up? Because that is the only thing McK gets the fees they do. They're also at Google and Facebook, and have done mass layoffs there even though their enormous profits are not under threat.
Me.

It is the point: if you trust a project based on "who" made it, my friend, that is the start of the big problem we are facing in this current situation of tech. Just look at the code, build it yourself, and check the license.

Pack is made to be a private option; future locking end encryption options will solidify that. Trusting the author is not the correct way to verify the security and safety of such a tool.
You may already be aware of this, but the T14s is "just another" Intel box, whereas the X13s is Snapdragon/arm64. I despise the random nomenclature of their product lines for that very reason. But IMHO a blog post about Ubuntu on more Intel stuff would not make the front page versus "here's the current state of arm64 on the desktop" is something I have keen interest in. I still have high hopes someone is going to have better luck with Microsoft's Dev Kit since it comes with a lot more ram (recently https://news.ycombinator.com/item?id=35757277 et al https://hn.algolia.com/?query=windows%20dev%20kit )
Using 32GB of RAM, but it is far more than they need.

7-Zip was used as others, just gave it a folder to compress. No configuration.

As requested, here are some numbers on tar.zst of Linux source code (the test subject in the note): tar.zst: 196 MB, 5420 ms (using out-of-the box config and -T0 to let it use all the cores. Without it, it would be, 7570 ms) Pack: 194 MB, 1300 ms Slightly smaller size, and more than 4X faster. (Again, it is on my machine; you need to try it for yourself.) Honestly, ZSTD is great. Tar is slowing it down (because of its old design and being one thread). And it is done in two steps: first creating tar and then compression. Pack does all the steps (read, check, compress, and write) together, and this weaving helped achieve this speed and random access.
On the other hand, VLC goes out of their way to give updates to users of old OS versions. Why won't Google and Microsoft let them do it?

According to VLC there are still millions of users of outdated Android TV and I agree with them: it's unreasonable to ask all these people to buy a new TV if it's still working. I get that manufacturers don't update the OS for more than n years (if at all). I get that some developers decide to not deliver updates for OS older than n versions. But wanting to deliver updates for these users should not be impossible.
std::quick_exit
Kagi uses the Google search API.
Zephyr: https://huggingface.co/HuggingFaceH4/zephyr-7b-beta

Their platform for easy distribution and management of models has sped up the ecosystem more so.
A Return to Blu-ray as Streaming Value Evaporates
Reading many files (81K in this test) is way slower than reading just one big file. For bigger files, Pack is much faster. Here is a link to some results from a kind contributor: https://forum.lazarus.freepascal.org/index.php/topic,66281.m...

(Too long to post here)
Because you can't really die from it? I know you can actually die, but for those situations it will be handled by the emergency room.

Not saying I agree with the policy.
The refreshed air in the cabin is driven by engine bleed air (or a dedicated pressure pump) so even at idle power there is nothing preventing a high rate of cabin air exchange. There might still be less air flowing on the ground due to 0 cabin air pressure differential, but speed of the plane isn't a relevant factor.
At least one Go crypto developer publicly expressed concern about this very issue in 2021: https://github.com/golang/go/issues/49702
I pay for Kagi and was similarly only vaguely aware that they have AI features and I had no interest in using them.

I started adding a question mark to my query out of curiosity and their instant answers are really good. Also they have links to their references, which I often check to verify the answer.
> I don't have an iPhone

Yeah, we all knew that. You and about 95% of the other complainers in this thread obviously don't know what you're talking about and keep getting exposed as bad faith actors. It's tiresome and only serves the opposite of what you intend. When someone sees claims like yours and then finds the first result on Google proving you're full of shit, they don't believe the next complaint about Apple that could be legit.
> Not even that, I'm not a civil engineer, but it looks pretty hard to add a ton of concrete without really helping a skyscraper.

I am also not a civil engineer but I can think of a lot of ways to add concrete to a skyscraper in ways that are somewhere between nonhelpful and burdensome.
It's basically Perplexity which is a very good product. Google is on the money here.
More documentation will be published soon. For now and about CLI: https://pack.ac/cli-documentation
> For example, I read somewhere that ~70% of atheists admit to praying in the last year (or something like that. I don't remember the source). I think it's fair to say that the urge or feeling of wanting to pray is common.

I am an atheist. I am very strict about the fact I do not believe in a God. My beliefs are strongly based in science, and I also believe Gods are fictionally created by people who need something more in their life.

However saying all that, there have been 3 major personal crisis in my life, and I have prayed in every one of them.

I cant explain it, I dont understand it. But I am hopefully intelligent enough not to close the door on it and let my mind ponder the whys of it from time to time.

JR1427, your comments here have provoked thought. Thankyou :)
Hey,

The choice of parameters was solely done to be clear, and not what people used to. -f meaning force is not clear; -w meaning overwrite, seemed like a better logical choice, to me.

Nice point on -h. Yes, I did not want to go crazy. After all, almost all (CLI) people use pack as `pack ./test/`. Options are for advanced people like you. Most people will use the OS integration that will be published later on.

--press=hard is the only option there is. There may be more, but with Pack you do not need to choose a level (like 1..9 with ZIP). Just let Pack do its thing, and you will be happy. Hard Press is there for people who want to pack once and unpack many times (like publishing), and it is worth spending extra time on it. Even then, Pack goes the sane way and does not eat your computer just for a kilobyte or two.
"...For reasons I can’t quite understand, we’ve sort of collectively decided that it’s okay to put our Kubernetes control planes on the public internet. At the very least, we’ve sort of decided it’s okay to give them a public IP address..."

Is that what Google does with Borg? ;-)

"...Many of the developers at Google working on Kubernetes were formerly developers on the Borg project. We've incorporated the best ideas from Borg in Kubernetes, and have tried to address some pain points that users identified with Borg over the years..." - https://kubernetes.io/blog/2015/04/borg-predecessor-to-kuber...
Why? They tune their algos/metrics to specific metrics.

Main metrics for their business is engagements(how many queries you do on google.com) and Ads clicks, and not how precise answer you got.
Same in the UK. You get GP stuff on the NHS for free, but dental and optical is only subsidised if you're on state benefits. If you go to the GP with a health issue that's related to the mouth in some way, you'll be told to see a dentist. I've never understood it.
Kinda like Deltec's "Deputy CEO"? (Tether's bank), or even Deltec itself:

At the start of 2021, according to their website, it was a 55 year old bank. By the end of 2021, it was a 70 year old bank!

The bank's website is a WordPress site. And their customers must be unhappy - online banking hasn't worked for nearly two years at this point.

Anyway, their Deputy CEO gave this hilarious interview from his gaming rig. A 33 year old Deputy CEO, who by his LinkedIn claimed to have graduated HEC Lausanne in Switzerland with a Master of Science at the age of 15... celebrating his graduation by immediately being named Professor of Finance at a university in Lebanon. While dividing his spare time between running hedge funds in Switzerland and uhh... Jacksonville, FL.

The name of his fund? Indepedance [sic] Weath [sic] Management. Yeah, okay.

In this hilariously inept interview, he claimed that people's claims about Deltec's money movements being several times larger than all the banking in their country was due to them misunderstanding the country's two banking licenses, the names of which he "couldn't remember right now" (the Deputy CEO of a bank who can't remember the name of banking licenses), and he "wasn't sure which one they had, but we might have both".

Once the ridicule and all this started piling on, within 24 hours, he was removed from the bank's website leadership page. When people pointed out how suspicious that looked, he was -re-added-.

The bank then deleted the company's entire website and replaced it with a minimally edited WordPress site, where most of the links and buttons were non-functional and remained so for months thereafter.

I mean fuck it, if the cryptobros want to look at all that and say "seems legit to me", alright, let em.
It looks clean and pseudocode-like. It helps readers from around the world with different languages understand. FreePascal compiler is very good too.
then just click the link and visit the page like you would have without the summarization? It doesn't seem like you've lost anything
I just do not want to follow Wirth's law.
*On Developer Marketing*

Why?

He who has control makes the rules --- and typically, this isn't developers.
Reminds me of “Flashed face distortion effect”:

https://en.m.wikipedia.org/wiki/Flashed_face_distortion_effe...

https://youtu.be/VT9i99D_9gI?si=QHhzR00tib3ZuX-5
You should try Kagi's FastGPT [0]. You need not trust the answer, as it gives the URL sources for each paragraph to verify yourself.

[0]: kagi.com/fastgpt
Can you be more specific? Just saying it's not compelling doesn't add anything to the conversation
Best monthly purchase I make now is Kagi.

Replaced our Netflix membership with it.

Recommend so you don't need to be worried by what Google does with your search.
Something I wonder about such attempts is, that as soon they push cryptocurrencies into the more and more shady terrain, the sooner people might start to use them for paying without converting them to normal currencies.

Like for example, someone does a thing for someone else (maybe helps them with some task or does some kind of commission, or even some real world thing) they give them a token of some crypto currency as payment. Then they take that token to the next person and pay them with that token for another service, item, whatever.

Everything without even paying a dime of real world tax. That might be really dangerous and could destroy societies if governments don't make money anymore to do stuff for their people.
Sorry. Site is very new, and custom made, and needs to be worked on mobile.
Hell yeah indeed!
You can find me on the forum too if you liked to talk Pascal: https://forum.lazarus.freepascal.org/index.php/topic,66281.0...
As far as I can tell this is just the truth, so I'm curious who downvoted and why. Guilty consciences?
Meet the AI-Censored? Naked Capitalism
Three Body Problem (TV Series)
AI as a mechanism for allowing Google (or whoever) to deliver the same information we already have through an inevitable additional opaque layer of bias and advertising would be seen as a horror if so many people weren't impressed by the gee whiz factor of it.
As far as I know, PAQ strives to give the best compression for archival purposes. Pack, on the other hand, tries to give the best compression, while keeping speed as instantaneous as possible.

You should try them for yourself.
bork the stego with random yellow pixels.

https://security.stackexchange.com/questions/155782/defeatin...

https://www.theregister.com/2018/06/27/german_researchers_de...

https://github.com/dfd-tud/deda

https://news.ycombinator.com/item?id=14501894
I've had these enabled since launch and I think they are great. It usually provides 2-3 stanzas of summary, each with a carousel of sources. It's a useful format because in a ranked list it is not always immediately obvious when there are multiple classes of results, the union of which answers your question. With the summary result, the correspondence between sources and relevant facts in more obvious.
Bing has also included AI-powered search results since last year: https://blogs.microsoft.com/blog/2023/02/07/reinventing-sear...
Hello, and thank you for the notes. Unfortunately, your points seem to be mostly wrong, so let me clarify them a little. Do not worry; many people misunderstand SQLite and its abilities.

- Single implementation: Sure. Working with SQLite convinced me that nobody cared to reimplement it, as it worked so well that nobody wanted or needed to rework it. I may write an unpacker just to prove that it is not hard at all to read SQLite format. The complicated part is the SQL engine (and many other features that are not used in Pack), and for Pack, you can live without it.

- SQLite does not require a disk. It has a memory option. Pack can have piping and properly will. I did not implement it because, well, it is too new, and I do what I feel is needed first. You can subscribe to the newsletter on the site (https://pack.ac/notes) or follow GitHub.

- Of course you can read SQLite without reading the whole file. It is a database, not a tar file.

- SQLite is highly optimized to read the lowest amount of data, and it has layers of smart caching. There is a reason it is used on almost any device that has a computer on it, even smartwatches.

- Of course the archive is safe from changes in unpacking. It will be opened in read-only mode, guarded by the OS and file system, and Pack also uses code isolation, which prevents calling write on any file.

- There are a lot of tools that help repair a damaged SQLite file. Pack too is guarded with transactions. The file will not get corrupted unless the disk goes corrupt; the mentioned tools come handy then. And in today's world of SSDs, the risk is shrinking rapidly.

- On unpack, Pack reads, decompresses, checks, and writes in a multithreaded. So yes, parallel reading is possible and done in Pack.

- I suggest trying Pack for yourself. It gives you the feeling you need to have to be sure.
The cryptosystems the article says are too big to use, Kyber and Dilithium, are both lattice-based.

A solution to this problem may also be lattice-based: SQISign is mentioned, though that is too slow. Or it may be something else: Unbalanced Oil and Vineger, and Mayo, are not lattice-based.
I highly recommend the Third Manifesto. I could link this under most posts in this thread but I'll limit myself to two.

https://www.dcs.warwick.ac.uk/~hugh/TTM/DTATRM.pdf

The only problem there is that you might want to use a D language, and well. You can't. There was a product called Dataphor which one can find some writeups on but, baffling though I find this, there are no robust open-source relational databases which use a D language.
damn shame there doesn't seem to be a search facet for saying "only podcasts" since I don't need another search engine for webpages
It doesn't seem plausible to me, based on my anecdotal experience with an ASD child who played in the woods all the time.