skysight-inc/HackerNews huggingface dataset:
Interesting that the two main "alternatives" mentioned here are columnar data storages. The problem I see with both Arrow and Parquet are that they are binary-based files. Sure, binary files can be VERY efficient. But the "magic" behind files like CSV, JSON or YAML is that they are both "sufficiently" both human and machine readable.

Maybe a CSV killer would be a human readable columnar based file format.

Nevertheless, the article basically discusses the issues encountered with the manual "editability" of CSV files, not so much with its performance. It also mentions parquet or arrow and concedes that they require specialized format to read/write. If we are looking to that, then there are a lot of options such as sqlite format, BerkleDB (used by some cryptocurrency projects) among plenty of others.
You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc.

These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. other data formats — the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically. You actually need to fully parse it (serially), and only then will you know where the row boundaries are. If you've ever dealt with trying to write ETL logic for datasets that exist as multi-GB CSV files, vs. as multi-GB any-other-data-format files, you'll have experienced the pain.

> The new format should not be grossly larger than the one it is replacing.

Self-describing formats like JSON Lines are big... but when you compress them, they go back to being small. General-purpose compressors like deflate/LZMA/etc. are very good at shearing away the duplication of self-describing rows.

As such, IMHO, the ideal format to replace ".csv" is ".jsonl.gz" (or, more conveniently, just ".jsonl" but with the expectation that backends will offer Transport-Encoding and your computer will use filesystem compression to store it — with this being almost the perfect use-case for both features.)

-----

There's also Avro, which fails your point #1 (it's a binary format) but that binary format is a lossless alternate encoding of what's canonically a JSON document, and there are both simple CLI tools / and small, free, high-quality libraries that can map back and forth between the "raw" JSON document and the Avro-encoded file. At any time, you can decode the Avro-encoded file to text, to examine/modify it in a text editor.

The data-warehouse ecosystem already standardized on Avro as its data interchange format. And spreadsheets are just tiny data warehouses. So why not? ;)
The hoops, to me, make it seem like you're going to be working with puzzlemaster trivialords, which sounds like the definition of hell on earth.
Not Amazon. Definitely Google.
It’s just a 2 year stint my guy

You’re not married or anything close to that and you don’t have an hour commute anymore either

Maybe don’t work nights and weekends. The rest of your team lowers the bonus performance expectations, so just coast and collect yours
It would need to have 30 collisions before anything even took place, which realistically isn't going to happen.
Thanks!

I think it's quite likely that you will heavily overfit the network if you only use a handful of images, where it might end up reconstructing the exact paintings. I only have experience using 5,000+ images.
Recycle Cardboard into Anything with 3D Printing
You are also incorrect. It's both. For instance upon experiencing a novel, pleasurable experience, one you didn't know to anticipate, dopamine levels will increase above baseline. It will then drop below tonic levels before returning to the baseline, creating mild, even unnoticeable discomfort, driving longer term desire / motion to find / anticipation for more of the thing in the future.
Doesn't everything in the sewer eventually end up back in some body of water that feeds the streams anyways?

It's all pipes, Jerry.
There is a CSV standard (RFC 4180), the problem is that a lot of programs don’t follow it. DataGrip don’t quote all text fields and Python remove some of the \r\n or \n\r (don’t remember which).

As long as you are following RFC4180 it works.

I ended up exporting from Oracle DB using JSON and converting to CSV for one of our contractors to be able to import the data to MongoDB.
But having a cold Sprite after 6 months of zero sugar is amazing, it's like fireworks on your tongue! You can't have that kind of experience if you have soda or sugar everyday.
That's ignoring two big points in the OP: that CSV variants are ill-defined and conflicting; and that existing CSV data is too poorly-specified.

There is no reason to try to be "backwards-compatible" with existing CSV files - we don't have a single definition of correctness to use to check that the compatibility is correct. Every attempt to be parse existing data would result in unexpected results or even data loss for some CSVs in the wild, because there is no way to reconcile all the different expectations and specifications that people have for their own CSV data.
All of that is true, but at the scale we’re talking about those are rounding errors.

Sure, housing in the Bay Area costs $36k more per year and yeah, you might shell out $10k in a year for medical if you have a baby and yeah, your kid might cost $20k per year in private school, but when you’re taking home $200k more per year, an extra $50k doesn’t move the needle much.
I've created an open source TypeScript dashboard
Perhaps they are aware and just decide to do it anyway, plenty of examples in US history.

Maybe read some history and be less baffled?
As someone that works with maven and npm.... what?

This is valid JSON

[["bob", "jones", 1, 22],
["frank", "was", 32, 45]]

That's unreadable and unparsable by a human?

Not only is JSON often more parsable, because it's structured it also becomes a lot easier to query.

I grep and awk xml and json stuff all the time. I also have the added bonus of being able to use `jq` for json content.
I think what curbs the immigration to Spain is its economic situation(15% unemployment rate), plus I wouldn't say that Spain has tough immigration rules.
In 2020 11% of of people living in Spain were foreign nationals: https://www.ine.es/prensa/cp_e2020_p.pdf

Out of these 11% only a third were from EU (including Great Britain), 6.5% of the foreign nationals were from Latin America, taking 5 out of the 15 countries from which most Immigrants come from, and the biggest year to year increase in immigration is from Venezuela (39.8%), Colombia (31.3%) and Honduras (29.4%).
"One of these passengers tried to kick out a window"

I am duct taping this person. Then when someone says I should not do this, I am duct taping that person.
No, but it would be easy to tell that you associate with your co-worked by correlating Bluetooth beacons, which many apps do for exactly this purpose.
Genomics Testing on the ISS with HPE Spaceborne Computer-2 and Azure
> I strongly doubt the public servants making those decisions have it in the power to increase the rate of immigration tenfold based on individuals meeting criteria without political support.

It is clear that there is high political prioritization of rejecting as many applications as possible, despite what the criteria actually say and whether the criteria are met. Once people manage to appeal to the real courts, rather than the politically controlled asylum process, most of the decisions are overturned. https://www.theguardian.com/uk-news/2018/sep/03/inhumane-thr...

The number could be significantly increased by simply taking their own stated rules at face value rather than rejecting valid applications.
You shouldn't rely on CloudWatch to discover and resolve issues
> access through cat/grep/awk, and easily load into any programming language

Until the CSV fields contain commas themselves. Even if fields are surrounded by "".
Consider trying less hard. Do mediocre work. They’re all still employed for doing mediocre work. It may be very difficult to produce poor quality work - you’ll have to overcome your “I’m brilliant” ego to do so — but you can trust that your peers, by your description anyways, are not encumbered by such ego.
If you're uploading your photos to someone else's computer, you've already accepted this state of things. It's a trade-off for convenience.
> Subsequent in vivo (within a living organism) testing on mice showed the DNA-origami traps were capable of targeting individual virions inside the body, disarming them without disrupting bodily functions, and finally destroying them with natural immunological mechanisms.

It's wonderfully surprising that body can have bunch of free floating DNA constructs inside without triggering strong immune response.
To quote a sibling comment, who looked into the horses' mouth:

> Apple’s method of detecting known CSAM is designed with user privacy in mind. Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child-safety organizations. Apple further transforms this database into an unreadable set of hashes, which is securely stored on users’ devices.

https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
>just accept it as it is and coast

This is why you go to faang. Take the money.
There is more and more 24V in marine systems to deal with higher loads in recent years. Your 28' (8m) day sailor may not need much more than a starter battery, but once you're at 40' (12m) and above things start changing. E-motors in the marine world generally use 48V (AFAICT).

48V is also being looked at in the auto world as well, at least for "low voltage" stuff:

* https://www.automotiveworld.com/articles/48-volt-likely-to-b...

The IEEE's 802.3bu PoDL can handle sending 12, 24, and 48V, and automotive is one of its deployment spaces:†

* https://blog.siemon.com/standards/ieee-std-802-3bu-2016-powe...

* https://en.wikipedia.org/wiki/Power_over_Ethernet#Standards_...

† 10BASE-T1, 100BASE-T1, 2.5GBASE-T1, 5GBASE-T1, and 10GBASE-T1; .3cz task force is working on 25/50/100 Gb/s.
That seems right to me.

I don't know a single person making SV money from Europe. Difference between dev salaries between eastern and western Europe has shrunk somewhat, thought.
Quit your whining and fix the damn problems. You sound like an entitled brat. If you're so smart compared to everyone else on your team, then you should have no problems fixing things. Why do you have to work nights and weekends? Are they asking you to? If not, then why put that on yourself? And even you work occasional nights and weekends, so what? Part of the joy of software development is the positive feedback loop of fixing something and moving on to the next broken thing. I did nights and weekends in the past because it was fun! I enjoyed problem solving.

As someone in a movie once said, "Now you can either accept that, or you can get to work. That's all it is. You just begin. You do the math. You solve one problem... and you solve the next one... and then the next. And if you solve enough problems, you get to come home"
Define "garbage." If I know what my data looks like, I can anticipate the edge cases ahead of time. Plenty of CSV exports work this way, they don't need to be general if the schema is already imposed by the system.

Have you ever worked in embedded systems? Writing XML files and then zipping them on a platform with 32 kilobytes of RAM would be hell. CSV is easy, I can write the file a line at a time through a lightweight microcontroller-friendly filesystem library like FatFS.

I know this is HN and we like to pretend we're all data scientists working on clusters with eleventy billion gigs of RAM, but us embedded systems folks exist too.
> The hoops you have to jump through to snag one of these jobs

This right here is exactly the problem. In the interview you are expected to write a fault tolerant k-way distributed sort and publish it to production, in 3 hours. Once you are embedded in your team they'll have you fixing typos on the landing page.

"Our interview process is good at finding people who are good at interviewing, not good at their job." ~ Someone I follow on Twitter
> I’ve hired a lot of excellent remote developers in European countries. The trick is to pay them American-level wages, which are often substantially higher than what they could get locally.

Are there any websites that specifically cater to advertising these kinds of positions?

As someone in London currently working at a FAANG it's becoming clear that the only way I can hope even to match my current salary is to A) get hired by another FAANG or B) get hired remotely by a US-based company.

If there is an option C) that I haven't thought about, please do share :)
Maybe a violation of TOS, but I guess it's one of these things that would not be enforced at all. Same way as Oracle can't do much about Google using Java or Microsoft doesn't do anything about Wine.

Regarding the second point - that's true. I've heard there are some plans to add payments to the Aurora store/F-Droid (which are alternative app stores) but right now you can't use paid apps.

I consider this to be a plus though - gives me a chance to switch to open-source / self-hosted apps.
Interesting thought experiment that feels good after struggling with some thorny data corruption issues (or whatever inspired this), but this what they call a boil-the-ocean problem. There's no point in thinking about CSV beyond the impact on things that you directly touch as it will never rise above the background noise of the problems faced by all the people whom you hope to inspire.
uBO shows you what it has blocked on each site. So you can see which sites would subject you to ads without actually having to expose yourself to the psychological terror of seeing those ads.

It also blocks things that you don't see (tracking), improves performance and prevents malware infections.
It is in the urine. Drug companies have long resisted any investigation of the impact of their drugs once they have exited the first patient. The necessary implication would be that waste from patients on certain drugs should be treated as hazardous, or even radioactive. That's Pandora's box.

A parallel question is whether the body of a deceased patient should be treated as hazardous. We do tend to bury or cremate them without a thought towards whether these drugs will survive and impact the local environment.
The author might as well be talking about the asteroid 16 Psyche.

They need the supply chain, transportation, electrical, and financial systems necessary to exploit those resources, none of which Afghanistan has.
What I don't understand about discussions in this thread is that RF is treated as almost magic and 'new technology' can make it do anything. Electromagnetism is virtually perfectly understood and there is nothing more it can do, definitely not responsible for nausea ffs without burning you from the outside... (unless it's ionizing radiation at the other side o the spectrum from RF but radiation sickness is also well understood and would be super hard to miss). Not to mention sound that apparently is being heard...

If someone really is targeting US diplomats worldwide, with those sets of symptoms, why does it have to be a sci-fi tech instead of, old as time itself, plain poisoning
Reading the comments here makes me think nobody even bothered to read the article.

Right in the first half they mentioned that there was a physical representation of the illness in the brain tissue when they did a MRI scan.

This is NOT a shared paranoia, because you can't see them on a brain scan. This seems like a very real and frightening disease.
Hard to know what the best options might be without knowing what it is you are trying to avoid in the US.
> The warrant would properly only be to search iCloud,

iCloud is encrypted, so that warrant is useless.

They need to unlock and search the device.
That CIA says it, isn't a reason to believe it.
I fix the software I use. Random commenters are not entitled my time and attention. I've learned to avoid rolling out my own crypto and suggest others learn it too. One shouldn't be proud of reinventing the wheel unless it's significantly better wheel.

So, is it?

edit: they use libtls, so I guess they aren't "free" from third party code after all.
> "a FAANG"

What's that? There's only five of 'm, just give us a name. ;)
The least fun csv-like format i have seen was mandated by a Big Spanish Bank. It was original a fixed fieldwith flatfile. Numerals were 0-leftpadded and fixed decimal (decimal point not written). Except that someone decided they needed to open it in Excel so semicolons were added at the end of each field. Then they discovered that some data didn't actually fit in the fixed with fields. So certain fields where variable width.
End result was a file that looked like shit in Excel, and couldn't be parsed by a fixed width parser.
i switched to Ubuntu 10 years ago
He has said that in interviews, "there is longevity in my family" if I recall correctly. He won the genetic lottery and can get away with things the rest of us can't.
I think Apple is the only major company who tries at least a little to resist Orwellian dystopia. Business model of other companies cannot allow resisting.
If you believe "the government can compel any company to do anything", then this system is no worse than any other.

Although how do you think Linus Torvalds "manged to get away with refusing" adding backdoors? https://www.techdirt.com/articles/20130919/07485524578/linus...
> Subsequent in vivo testing on mice showed the DNA-origami traps were capable of targeting individual virions

... in mice.

Sorry, that's an in-joke:

https://twitter.com/justsaysinmice

https://jamesheathers.medium.com/in-mice-explained-77b61b598...
The "Elm at Microsoft" link is broken and when I google for it, I can only find an archived package on Github.
I want to echo your concerns. This thread is full of gaslighting and a lazy and strange assumption that organizations are maliciously accepting friendly fire due to their own tech or tools. Same with "government workers wanting an early medical retirement to get their pensions".
> I'm on a team that... is staffed with below-average IQ people.

I would definitely not want to work with you.
In terms of power usage versus CPU performance, nothing beats M1 indeed. In terms of pure performance, Ryzen 7 5800X beats M1 in nearly every benchmark.
That's ~200W versus ~20-30W though.

In terms of SSD performance, the M1 SSD upgrade from 256 GB to 1TB cost about 500 EUR, while a 1TB Samsung 980 SSD (with very similar performance) costs about 120 EUR total.
Such a goal could only be accomplished by removing the Taliban from power. So, it really was about nation building the entire time.
Reality is also there is just a lot of regular stuff that needs to happen. Handle this report, deal with this type of issue, improve tooling for X.
Whatever human ancestor was a pure herbivore was some millions of years ago, and in the process we started to rely on meat for nutrients such as B12 and creatine. We also absorb heme iron, which is exclusively found in meat, far better than plant sources of iron, while our stomach acidity is closer to vultures and owls than other primates. I very much doubt humans could survive without at least some meat without agriculture.
Apple could scan on the cloud. Rumor is that Apple wants to use E2E on iCloud, and this is a necessary step to shut up the government's biggest critique of E2E and deploy it before the government can figure out a different excuse. We'll see if that pans out.
Sad that android one didn't take off. Not having clean android images or universal images really hurts smartphones as a platform.
Will London eat itself? Are dark kitchens and dark stores as evil as they sound?
Yep. Pretty much any large group of people will inevitably grow to feel largely incompetent.
Even if that was true, why should we shift the blame away from an entity with plausible ability to solve it, and towards the great mass of people who are impossible to hold accountable?
In Germany, even good software engineering jobs pay on average 2-3x less than their US equivalents.
I remember my first day at Goofacelix when all my cowokers were forced to submit IQ tests to me.
If that's the case, we should be seeing correlation: higher amounts of medication in the waterways of older US cities, where sewer and stormwater systems were built to an earlier idea of best-practices.

This should be relatively straightforward to investigate.
Yeah, provided I don't find alternative open-source / self-hosted apps. I'm not saying it's easy, but it seems doable in medium/long term - if you value privacy and owning your data.
> It's actually not

It's Joe Wong's joke, that it's like peeing in the snow in a dark winter night, while there is a difference but it's really hard to tell.
So, they aren't "free of third-party code now" after all. Good for them.
This is only applicable in the US?
For some additional context, the person in that story was 13 years old.
> The whole idea of a perceptual hash is that the more similar the two hashes are, the more similar the two images are

This is already proven to be inaccurate. There are adversarial hashes and collisions possible in the system. You don’t have to be very skeptically-minded to think that this is intentional. Links to examples of this already posted in this thread.

You are banking on an ideal scenario of this technology not the reality.

EDIT: Proof on the front page on HN right now https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
To be honest, I'd rather let Tim Cook personally review every photo I take and show them to his friends at parties than use that tech stack.

I hate to say it, but if these are the best alternatives then I guess I have to be ok with losing my privacy. I'll keep pressing Apple and donating to the EFF and Fight for the Future.
Remember that part about China moving quickly to recognize them?
> I fully expect airplane windows to be able to withstand a kick from a child.

Would you bet your life on it?
Inconveniencing an out of control teenager for a few hours to ensure the safety of all passengers seems like a pretty sane response.
Maybe because someone just pulled the numbers out of his ass?
> - No US Company will never "outsource" to europe to save 50% per full time worker

Even European countries outsource to Europe for at least a 50% save on FTE.

Edit: and if you count countries like Ukraine as European, it's even more.
> If someone uploads child porn to iCloud and then shares it with someone else Apple is possessing and distributing child porn.

If someone sends CSAM using Federal Express, is FedEx legally liable for "possessing and distributing" that material? Does FedEx need to start opening up packages and scanning hard drives, DVDs, USB sticks, etc. to ensure that they don't contain any CSAM or other illegal data?

I really struggle with the lengths that people are going to to justify these moves. If we can justify this, it's pretty simple to justify a lot more surveillance as well. CSAM is not the only scourge in our society.

Reminder: Apple tells us that they consider privacy a "fundamental human right"[1]. That simply does not square with their recent announcement of on-device scanning, and some would argue that it does not square with scanning or content analysis anywhere, especially on behalf of government.

[1] https://apple.com/privacy
Most is kidney passthrough into pee. Second is combined sanitary AND rain runoff sewers - which bypass treatment when there is high rain. Third is treatment plant pass through - many drugs are not eliminated in sewage plants (it varies a lot with the drug)
I was illustrating how there’s a switching cost for me, and not for Apple. They already have my money.