skysight-inc/HackerNews huggingface dataset:
There are better formats but the change needs to come from the top down, I can't see it working from the grass roots. Put it this way, I'm not going to turn around and refuse a non-technical client who volunteers at a non-profit because they can only give me data in Excel rather than whatever I decide the superior format is, or tell them it's impossible when the next developer will click a few buttons and have me taken for a fool, nor will I push hard to sell them up on a data mapping service or cloud pipeline if it's well outside their needs or budget. Unlike topics like web accessibility that's easy to argue on its morals and ethics, or mobile first that's easy to argue based on market share and audience reach, refusing to work with an organisation's data because you don't like the very simple, very established format is a very tough sell. If Excel could natively export to JSON, that's what I'd ask for, or their custom-built website from the 00's had a clean REST API, then it wouldn't even be a discussion.

There are workarounds for sure; VB macros, unzip XSLX and parse the XML, write scripts to automate Excel-to-whatever using DCOM, or import into an intermediary service and re-export into the format de-jure, but that all takes time and costs money too, and often causes confusion when even spoken about. Asking for a CSV takes a couple of seconds and is easily understood by most people. Anyone experienced with importing / manipulating CSV data can deal with the variance in delimiters and escape sequences without major issues. It's a headache at times but easier than the alternative of alienating or confusing clients who are looking for a simple solution to whatever issue they have today.

On the other hand once the data is in the target system, if they're still asking for CSV exports I do probe to ask why, and try and figure out if there's a better way. Reporting is the usual reason, and plenty of the CRMs I work with have built-in reporting that can replicate and improve upon whatever spreadsheet they are using, and have APIs that can interface with cloud-based reporting services. But there's a lot of inertia against change in most small-to-medium organisations, not everyone is a data expert and you can't sell someone something they can't use or understand. Ultimately people win, and the solution ends up being a balance of hopefully incremental technological improvements that they can still integrate into their day-to-day. Not every organisation is able to undergo a full digital transformation with time, budgets, and skills at hand.

I agree with the sentiment but I'd hate to see a future where everyone's locked into proprietary ecosystems - not that that is what's advocated for in the post, but we have CSV because that's what the big platforms seem to allow, not because they don't know there are better options. There's no technical reason Excel couldn't export to WordPress or SuiteCRM. Take CSV away and it gets harder, not easier, to move between platforms.
While Google and Amazon might both suffer from a horrendous 10 year old code base all built in house, I can assure you that Google does not have high turnover lmao
> but it’s not ideal if you’re trying to run a lean team that ships products quickly.

Good. The longer I do this, the longer I realize that unless your in a race-to-the-bottom industry (something like vying for eye balls to increase ad spend), then this is categorically a bad thing. I would prefer the quality of engineering required in knowing how to do some of these things yourselves.

Take your time. Understand what you are doing. Some of the world class technologies we all know and use daily use minimal dependencies.

And with more malicious code entering more package repositories, it might be apt advice more than ever today.
I have never seen the iOS show me an ad. I've seen some third party apps, but not the OS or core apps.
You're thinking of cryptographic hashes. There are many kinds of hash (geographic, perceptual, semantic, etc), many of which are designed to only be slightly different.
> The fact Google is too lazy to stop a bunch of conglomerates from monopolizing the internet is the most disgusting part

It would make them somewhat hypocritical if they did.
> That would be among the dumbest options you could possibly choose.

You don't get to dismiss others' concerns by saying "it's a choice", and at the same time dismiss choices others present...
SQLite also requires external dependency and special execution environment and are not as portable as CSV.
Imagine going to a country by building infrastructure and not with bombs to destroy it..hmm I wonder what would the populace prefer.
It’s that the situation can’t be a “slippery slope” if there’s no evidence of there being one prior
Thanks so much for trying it out and for the kind words Dan!

> setting goals for the hour, seeing the faces of six strangers on mute, and a timer ticking down. It drove focus and accountability.

Goal-setting is so powerful and yet so difficult— clarity of goals + setting the right level of challenge for the time period are two of the key characteristics of flow state (as defined by the great Mihaly Csikszentmihalyi). Re: "seeing the faces of six strangers", we think that this experience becomes more powerful as you come back to Flow Club over time—you'll do sessions with some of the same hosts and a mix of new faces, acquaintances, and friends, both from outside Flow Club as well as ones that you make in Flow Club.
Why use a VPS when you could have your own metal?
I'd add the constraint that it needs to be understandable by most people. A typical non-tech office worker can use CSV without any trouble. Sqlite has an extremely steep learning curve for someone unfamiliar with databases.
OP joined Amazon without a doubt. (Build everything in house, working nights and weekends, rigid team-switching rules.) I just interviewed there and the onsite presented more red flags than a Soviet military parade. Would definitely recommend treating them somewhat differently from the other FANG companies
I feel like you are both chasing red herrings.

This is not about child porn nor about what AV software can or cannot do.

It's about normalising mass surveillance and implementing populace control. They don't want another Snowden to scare the public with reports about "backdoors" and mass privacy violations.

They want the coming generation to perceive it as normal. Because all phones do it, hasn't it always been this way, and think of the children.

Oh, these dissidents in $distant_country that will be muted and killed using Apple's shiny surveillance tech? Well, evil governments do what they do. But we are not like that. Over here it is only about the child predators. Trust us.

Apple has been an opponent of these developments for decades. Now they are spearheading it.
I believe belt extenders are used to demonstrate how to unlatch the belt during the pre-flight safety briefings. You could easily use one to restrain someone's legs.
Just tell them they are falling into the 9 ladies can make a baby in one month process. That usually gets their attention.
that’s not true. i just gave an example of switching from what i was used to, to something new that i thought was superior.
Original paper: https://www.fradenlab.com/app/download/9774776565/s41563-021...

Edit: sorry for the previously bad link. Didn't notice.

From what I get, it looks like the traps basically smother the viral shell and prevent it from interacting with any surfaces.

They tried a couple methods to achieve this. Either assembling the shells around the virus, or starting with pre-assembled shells with a hole in them (an icosahedron with a pentagon missing.) For the latter case, they were able to neutralize more than one viral particle by making the shells big enough.

My main question is how the viruses find their way reliably into these preassembled traps.
The Taliban were not part of the groups that fought against the Soviet occupation.

https://en.wikipedia.org/wiki/Taliban
This thing is pretty well publicized, people do get it but not in the way you're implying.

If you have children and you explain to the other parents that you're just really passionate about privacy and are dumping all your iDevices for...reasons...your kids are probably not going to have friends come over to visit much after that.
Good lord. Why is everyone so upset by this CSAM scanning? Google scans every email you send, Facebook tracks every gesture and post you click. What makes you think that apple hasn't been scanning, or at least had access to, all your stuff in the cloud anyways? The instant you trust your data with a third party, you forfeited your privacy. The whole CSAM thing isn't a revelation.

If you're upset by the thought of a third party entity looking at your stuff then stop using products by these companies. Use FOSS for your computer programs and OS's. Use a phone OS like graphene, /e/, or something else privacy oriented.

The only way to stop being a commodity is by sacrificing convenience and the only way to gain that convenience back is to build your own tools. Talk to the old timers here that have been using minimal OS's and building custom scripts for the past 20 years. They'll tell you that you get a level of speed, efficiency and customizability you can't get with name brand OS's.

The learning curve is worth it considering you build knowledge and skills not a lot of people today have. Not to mention you gain technological independence, which is worth its weight in gold.
I wonder why we need more gas stations.

I own several vehicles of various vintages and there is not a single one that I couldn't crank up and drive from an existing gasoline station in my home town to any other connected, accessible location on my continent. Indeed, I can actually drive the 4WD vehicles I own deep into the countryside where there are no gasoline stations and never have a single fuel supply related problem with proper planning and equipment.

The fuel tanks on my vehicles range in volume from 10.5 US gallons to 33 US gallons and fuel economy numbers range from 32 mpg down to 12 mpg. That means for any of my vehicles I have a range of more than 300 miles (10.5 * 32 = 336 miles and 33 * 12 = 396 miles).

I will encounter lots of existing gasoline stations along any route I choose.

I think the idea behind building new stations has to revolve around making it more convenient for people moving to new subdivisions to fuel their vehicles locally. If you don't build out gasoline station infrastructure near new construction then those who choose to live there are required to monitor their fuel consumption closely enough that they always know when they should gas up on the way home. They should be doing this anyway IMHO.

I don't think we need new gasoline stations in summary because I think all places are already accessible with our current infrastructure.

And I would like to expand on or clarify a statement in the article that is very important and was not stated well.

From the article about halfway down:

>>But it’s not all good news. The long-term shift to electric and a mass closures of gas stations will create toxic “brown sites” across the country. A typical gas station can spill up to 100 gallons of gasoline annually, ... ...these poisoned plots will present an environmental problem of monumental proportions.

This paragraph makes it seem as if the existence of a gas station is environmentally fine until that station is closed or decommissioned at which point it becomes a brown site requiring remediation. That is not true at all. Once you fill the tanks and begin serving gasoline to customers there will always be some spillage at the pumps. The gasoline station becomes a brown site when it begins operation but you are typically only able to begin remediation when the station is shut down and no longer serves gasoline or when there is a documented leak from the underground tanks that makes it critical to shut down the station for repairs. Therefore, all existing gasoline stations are already toxic environmental sites and the problem is indeed monumental today.

I see no reason to add to the problem by building new gasoline stations.
Or maybe there are multiple competing interests, and the interest who wants an endless war lost out to public pressure and loss of faith of 2 different presidents.

The war in Afghanistan was still fairly popular among the general public until very recently.
That reminds me of the old joke about hell being just a sales demo.

http://www.jokes.net/heavenandhell.htm
Combined sewers that are in perfect maintenance include provisions to overflow by design.
It'll work until someone upstream upgrades their CSV writing library and your process breaks. But it's still writing CSV.
There is no scarcity of open source Minecraft imitators/clones, but Minetest seems to be among the most thoroughly developed. Veloren has some interesting demo videos, for example, but its modding interface is barely a sketch at this point. (The plugin API documentation covers interaction with the chat window, and events for players joining or leaving… and that's about it. There is no way to plugins to actually interact with the game.) A closer match for Minetest would be Terasology[0], which has a modular design based on Java rather than Lua.

[0] https://terasology.org/index.html
> provide for the common defense, promote the general welfare, and secure the blessings of liberty to themselves and their posterity

Good definition of the goals of self-government. I prefer the phrasing of the Declaration of Independence, dated as it is.

Regardless of how you say it, it's an idea that is either very difficult to communicate or that many just do not agree with. The "intellectual elites" must "protect their interests" (unfairly quoted from someone else on HN) regardless of what those who disagree think.
pigz is no match for zstd...
EMBRACE PARQUET.
"Instead, I'm on a team that has and horrendous turnover and is staffed with below-average IQ people."
"This company builds EVERYTHING in house, and the toolset is like going backwards in my career 10 years."

I am guessing this is either Amazon or Google. I would think that this is most likely Amazon ("horrendous turnover": 50% of people who join Amazon leave within the first 2 years). In LinkedIn you will routinely find people from Amazon showing their badges stating that they completed X+ number of years. Have you seen people from other places routinely doing this? That itself is a dead giveaway to the critical eye.

Btw, your IQ comparison is extremely derogatory. So please edit that out.
4 weeks later the feature is delivered to prod, nobody bats an eyelash.

What do you do when eyelashes get batted?

Seems like you'd get accused of "not being a team player" or some such bullshit if you say the truth, which is: "The PM set that deadline after consulting with us, and us telling the PM that 2 weeks wasn't feasible."
Why not simply use the ASCII characters dedicated for this? STX,ETX, RS, GS, etc
I appreciate all the comments (even the criticism). In so many ways, this post has already helped me see things more clearly.

One final thought is that I know I can quit - that isn't the question. I could make more money (with less stability) consulting, or find a middle ground at a Series C+ company.

What I probably should have said in my OP is that quitting feels wrong. I've never quit a position after three months and this is honestly the first time everything is telling that quitting is the right decision. As dysfunctional as this team is, quitting would feel like letting them down, and it just isn't something I've done before.

But again, these comments have given me perspective and will make me give this more thought.
There is a class of hashes known as locality-sensitive hashes, which are designed to preserve some metric of "closeness".

https://en.wikipedia.org/wiki/Locality-sensitive_hashing
Lots of consumer innovations have been the result of military spending!
That's extremely cool. Well done. :)
It's a good link, but it really doesn't seem like very much sugar at all..
Mitigating the Collision of Apple's CSAM NeuralHash
As I understand it, there was a rift in the community over the leadership's tone in messaging and communication. Some Developers also felt like they could not contribute in a meaningful way or have constructive discussion with maintainers.

See posts like these for more details and comments:

https://news.ycombinator.com/item?id=16510267

https://news.ycombinator.com/item?id=22821447

There are also several great alternatives now. F# with Fable and Elmish for example: https://zaid-ajaj.github.io/the-elmish-book/#/

And Bucklescript TEA with Ocaml or Reason: https://github.com/OvermindDL1/bucklescript-tea

Some others have also gone on to develop new languages taking inspiration from Elm: https://www.mint-lang.com/
Climate Change Startup Landscape
Seems like someone just can’t admit they’re wrong.
Jitsi is discussed in the comment below yours in this sub-thread.

I have extensively tested Jitsi and it does not do what I need it to do (support E2EE video conferencing for teams larger than 50 people).

People recommend Jitsi all the time, but I have yet to see any real-world cases of people actually using E2EE Jitsi meetings for large teams.
> It's Time to Retire the CSV

> This column obviously contains dates, but which dates? Most of the world

It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting).

> A third major piece of metadata missing from CSVs is information about the file’s character encoding.

It's bloody the time to retire all the character encodings and always use UTF-8 (and update all the standards like ISO, RFC etc to require UTF-8). The last time I checked common e-mail clients like Thunderbird and Outlook created new e-mails in ANSI/ISO codepages by default (although they are perfectly capable of using UTF-8) - this infuriated me.

> If not CSV, then what? ... HDF5

Indeed! Since the moment I discovered HDF5 I wonder why is it not the default format for spreadsheet apps. It could just store the data, the metadata, the formulae, the formatting details and the file-level properties in different dimensions of its structure to make a perfect spreadsheet file. Nevertheless spreadsheet apps like LibreOffice Calc and MS Excel don't even let you import from HDF5.

> An enormous amount of structured information is stored in SQLite databases

Yet still very underused. It ought to be more popular. In fact every time I get CSV data I import it to SQLite to store and process but most of the people (non-developers) have never heard of it. IMHO it also begs to be supported (for easy import and export at least) by the spreadsheet apps. A caveat here is it still uses strings to store dates so the dates still can be in any imaginable format. Fortunately most of the developers use a variation of ISO 8601 conventionally.

And by the way, almost every application-specific file format could be replaced by SQLite or HDF5 for good. IMHO the only cases where custom format make good sense are streaming and extremely resource-limited embedded solutions.
Don't think so. One is a promising-sounding experiment, the other is a serious effort by a company with a track record of delivering effective vaccine doses by the billion that already has something ready for human trials.
I've had the same thought, but alas. Prices are terrible.
Are you trying to stop abuse of children, or enforce a standard that the idea of images of children is bad?

If you’re actually trying to stop abuse, having the computer create fake CP seems like an ideal outcome, since it would avoid the need for abuse of children.

Flooding the market with fakes and then directing consumers of the fakes to whatever mental health resources are available seems like it would fit the claimed problem far better than what apple is currently trying.
Thats just a CSV with extra steps.
That's what both the soviets and the americans where doing in the 70s there... didn't work back then either
Your post wreaks of working at Amazon. I hate to tell you, but Amazon is the lowest hanging fruit of all FAANGs. They hire and fire at above average rates to try to find the best and then bait the rest to stay on with stock.

If you're at Amazon and you think things are going to get better, let me tell you, they aren't. Find a way out immediately unless you're okay basically moving forward with the same schizo management and dev policies.

Hope you find some work you enjoy!
Unit testing coverage is a development practice, not really a defense.

Testing, coverage, static analysis, methodologies, etc... are a way to make prevent things from going wrong in the first place, defenses are here to protect you if things go wrong. We need both.

The article focuses only on the "defense" part, the development practices could make another article.

Note that testing and defenses can go together: test to make sure all paths are taken before shipping, and see if defenses trigger. Fuzzing in particular takes that approach.
President Biden didn't consult with other world leaders before acting on Afghanistan. This was an actual statement during a press briefing. A quick talk around the table would reveal what China's intentions would be (such as what we're doing here). This amounts to nothing but giving away Afghanistan to China by the Biden Administration. It would have been more appropriate to assist our allies to working with the Afghan government while the US still has some occupation.
Thanks for checking it out! I don't think that it's because "[you're] just old"— it's definitely a matter of personal preference, and we know this style of working isn't for everyone.

Re: open offices, that's a good point— I actually think open offices are distracting if people are walking around behind you or chatting in the background. But if everyone else is also hard at work (and therefore not really watching), I definitely feel a small lift from that ambiance/energy.
To be honest, I don't care that much. Google, Facebook, Microsoft, they all do the same thing or worse. I started using an iPhone because it had the best tech at the time and it was the most convenient. The privacy stuff is nice to have, but it's not a big deal to me.
That won't happen again if Chinese interests are now at stake.
I think there's two conversations here: known, working CSVs as part of a workflow; and CSVs from random, unknown sources.

There's no such thing as schema-less - there's undefined schema
As soon as you open it in Excel, it's garbage anyway, since it will replace date-like items with nonsense, drop number digits, convert anything it can, reencode monetary unities, and so on.

If you don't open it in Excel, you can have as strict a parser as you want, just like any other format.

But neither is going anywhere anyway.
Afghanistan has been mining and exporting valuable minerals for several thousand years.

There's more mining technology available now, but it's perfectly feasible to do large-scale mining without it. It's just a question of how badly you want the rocks.