skysight-inc/HackerNews huggingface dataset:
Ask HN: How many % of Machine Learning projects “fail”?
There are all these reports/surveys claiming ~80% of ML projects fail:

- Jan 2019: https://blogs.gartner.com/andrew_white/2019/01/03/our-top-data-and-analytics-predicts-for-2019/
Gartner predicted that through 2020, 80% of AI projects will remain alchemy, run by wizards and through 2022, only 20% of analytic insights will deliver business outcomes.

- May 2019: https://content.alegion.com/dimensional-researchs-survey
Dimensional Research - Alegion Survey reported 78% of AI or ML projects stall at some stage before deployment, and 81% admit the process of training AI with data is more difficult than they expected.

- July 2019: https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/
VentureBeat reported 87% of data science projects never make it into production.

There used to be similar claims in 1990s about failure of Software Development projects:

- 1994: https://www.standishgroup.com/sample_research_files/chaos_report_1994.pdf
The Standish Group’s CHAOS Report in 1994 claimed that only 16% of the software projects succeeded.

That claim was doubted: https://ieeexplore.ieee.org/document/1438340

Overtime, CHAOS reports became nuanced: https://www.standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf

For building ML-based software applications/product-features, my questions are:

- How "failure" is defined for ML projects, when to call a project failure?

- Is failure rate really 80%?
When you publish the code, the chances of someone getting it to run increases from 0% to a much much higher number. Maybe 50% of those who really try can get it done. That it is hard to get to 100% does not at all invalidate this value.

It is fine to not do much support and bugfixing on such a codebase - but one should make sure to state it clearly in the documentation.

I try out published research code (in machine learning) all the time. Almost every time I have to debug and fix things. But in a couple of hours or days I can usually get it fixed, normally without any help. That is a shorter time than it take (in the best case) to even ask for access to code from a researcher.

Reduce barriers to entry. Make it easy for others to build on your work. Your chances of impact and value created from your work will be higher.
Yes. Also the blinkenlights are very satisfactory. And very nice casing.

What is the clock speed at the end of the demo? It sounds like about 8 Hz.
Signal forcing you to give your phone number to anyone you want to contact is a much bigger security problem to me, especially when in my country it's both somewhat hard to acquire an anonymous sim card and a relatively small sum of money can get you a lot of private information from just having someone's phone number.
That's very insightful. It's also interesting that the tendency is to do the opposite: if it's your code you may not want to trash it completely and instead work on an incremental refactor, while if the codebase is foreign it's tempting to just get rid of it and do it your way.

Anecdotally I've been recoding from scratch one of my applications at work (converting from python to rust) and while it's a lot of work and it put new features on hold in the meantime I'm fairly optimistic that it's going to be a big win in the long term. Not only will the Rust code perform better and be more maintainable, the rewrite lets me address architectural flaws with the previous solution, some caused by lack of foresight and others by tacked-on features that weren't expected when the code was originally designed.
Neither does Signal (publish server side code). See the top comment in this thread. Their last server code dump was 9 months ago.
Of course. The vigorous policing is decentralized, so you can choose which community standards you would like your interactions to be guided by. There are safe spaces, there are crazy freezepeach places, and there are many places in between.

A US server may have content that is illegal in Germany, a Japanese server may have content that is illegal in the US, and an LGBTQ-friendly server may not allow subscribers from a server full of "gendercritical" people. You can pick where you feel at home, don't get ick thrown in our face, and don't get accused of ick for things that seem harmless to you.
Couldn't Google just GPT3 the titles and not have to pay?
> it disproportionately affected EU businesses detrimentally

Well that’s a very interesting take. Enforcement is overwhelmingly targeted at foreign companies, and primarily from the US and EU. I guess you can be the judge of whether it’s a just coincidence that those are the countries that the EU has the most contentious trade relationships with. Aside from China of course, which I’d say you have correctly observed would simply not tolerate the EU trying to enforce its laws extraterritorially.
> Isn't that going to hurt the b-tree performance by having a very high fanout?

High fanout helps minimise the depth of the tree and therefore helps performance as it minimises the number of nodes to check.
You think threatening violence against journalists is "top notch"?

Is that what Hacker News is now? Has it really descended that far into actual fascism?
Show HN: Scaring my friends with infrared night vision
How to pick a qualitative research approach
"The French Competition Authority held that the deal Google offered to news sites—let us index your site for free or we won't index it at all—was an abuse of that market power and contrary to the spirit of the new French law."

This is why this law is so terrible. If the government wants news media to keep its role in society, it should not force a private entity to pay a "failed to monetize the web"-tax.

News companies cannot demand payment while they demand that Google indexes their pages for them. The lobbying of the news industry it truly terrible and has made me question if I want to pay for any news media when the national news service is perfectly fine and accessible for free. I want to support independent journalism, but I don't want to sponsor the moneymaking schemes they're inserting into the law.
Thanks for your answer. I read your response as: If well written and well researched, self promotion material is presented on HN, there is no need for the reading audience to be aware of this. Ok.
We moved away from UUIDs for different reasons. It is hard to work with UUIDs when multiple entities have them. You do not know if you are looking at a user or a group if both is just a UUID. We created our own IDs with prefixes and using base58 encoding for the random part, for example: user-AhnPpn7CbbwPykmux11LJ39Jk. It has the same amount of randomness in it and we can understand what we are looking at.
Luckily they are working on transitioning away from requiring a number. That will be an improvement.
Or even worse: little touchscreens that resemble a physical knob/button, even with haptic feedback!

All physical buttons/switches in my 20 year old Mitsubishi still work. I have a feeling the newfangled, fancy LCD-switches don't last that long, and will be very expensive to fix when they break.
There’s a spec called ULID that’s pretty much this with default base32 encoding

https://github.com/ulid/spec

I’ve also worked on a UUID-ULID bridge for Go

https://github.com/sudhirj/uulid.go

And seeing as this is just 128 bits it’s quite easy to move seamlessly between formats and representations.

I’ve found this concept especially useful in nosql stores like DynamoDB, where using a ULID primary key makes objects time sortable automatically. It’s also quite easy to query for items by zeroing out the random component and setting only the time stamp bytes.
I like Telegram, but it would be much nicer if secret chats could be synchronised between devices.

The distinction isn't just "mobile". Each mobile device a person uses has a separate secret chat, not synchronised with their other devices.

There's someone I talk with on Telegram where we have two secret chats open all the time, one for their iPhone and one for their iPad. Both channels are connected to my phone.

I have to guess which chat to write in for the other person to notice that I'm writing to them...

That's because when using their iPad they often don't have the iPhone nearby (no need for most purposes). It's lucky I don't have two mobile devices too, as then we would have four secret chats.

When I'm at my computer, we can only use the non-secret Telegram chat, which has a nice GUI but unlike my phone cannot connect to either of the secret chats. This isn't really a problem because we use the non-secret chat for most topics anyway, because of the search and synchronisation.

(Aside, the ability for the other party to delete old messages in Telegram is not very cool imho. I know someone who fell out with a friend and their friend deleted their entire multi-year conversation history, which was heartbreaking. Once you have a long history of messages with someone I think it should be part of your personal archive to look back on. It feels permanent until that point, so it's a surprise to find it can all disappear.)
No. Don't be crazy ...
That's awesome. Though I probably won't be using it much because I use the reMarkable to avoid the HN! :)
Can someone explain why on Earth news media worldwide is so adement on wanting commission for traffic to their sites?
I mean, were I a search engine provider faced with this demand I would just say "Well I'm terribly sorry you feel that way, we will no longer feature you in search results" and let their viewership plummet.
Same with social media, if someone demands commission on traffic towards their site I would just make it impossible to share links towards them.
It's their loss, not one of Google or Facebook.
It is however unfortunate how desperate publishers have gotten for revenue and I sympathise with them, but they're picking the wrong fights here. Journalism is important, both general and investigatory, and they have a massive job of figuring out monetization going forward to keep the lights on.
The article is directly comparing apple iphones against logitech webcams.

iphones benefit from: a processor that costs more than the entire webcam. A firmware and software stack with a billion dollars behind it.
Switching to IPv6.
(Someone correct me if I'm wrong!) I think about differential dataflow as the solution to "I can't batch data operations, because I don't know when my various inputs will land."

If everything exists at 7am, and/or you don't need the freshest computed values, this is not the solution you need.

If data A is ready between 2-4am, data B at noon, and data C sometime between 8am-6pm, this allows you to abstract that uncertainty into code, then let the system solve it on a daily basis.

This is not a problem everyone has. But it is a problem most people working with inventory or events have! And it's usually a problem people feeding things to ML have.
That is great. Pre-registration is far from standard in many fields though? And more and more papers are involving some level of statistical analysis?
I strongly disagree. Without a shadow of a doubt, the most successful/important AI model of recent times is the Transformer (from Brain). As another example, the fastest accelerators, TPUs, are from Brain. If you only look at "amount of output" and use # publications at e.g. NeurIPS as proxy, Brain vastly outperforms Deepmind. Deepmind really exceeds at flashy and showy PR releases like AlphaStar, not so much at laying the groundwork for AI breakthroughs. (Though of course AlphaFold or AlphaGo were great).
I'm not too worried about the client being closed source, especially not when the server is open sourced.

For Syncthing there is of course the potential problem of the client leaking the secrets to the author, giving them unauthenticated access to the server.

Thanks for the pointer, i'll check it out, though i've had a "lifetime" license for Resilio for years, and it scratches my itch, so there's no pressure to switch.

I have used syncthing in the past for server to server synchronization, a task it performs extremely well, but previous attempts at creating a "road warrior" setup from iOS (with f:sync) all ended in clients taking minutes to connect to the backend, where Resilio would do it in seconds. I'll give it another try.
Gotta say I've seen many many long lived Kali VMs or laptops over the years. Whilst ideally ephemeral OS images would be great, not just for Kali, but for testing environments in general, that doesn't always meet reality.

This (pentest tooling) is one of the areas that seems a good fit with containerization (podman, Docker, lxc etc), as their use case fits nicely (single use ephemeral images with some isolation)
“The principle of unrestricted linking between websites is fundamental to search and coupled with the unmanageable financial and operational risk if this version of the code were to become law, it would give us no real choice but to stop making Google Search available in Australia,”
That's a big threat, but pretty risky if their bluff is called because I think that might reveal that other companies could fill the search need if google vacates the space.

edit: quote source https://www.theguardian.com/media/2021/jan/22/google-threate...
Chinese Electric Car Battery Maker UPS the Ante
Having just finished Charles Petzold's Code book, and reading all about relays and building up a fully working computer solely using relays, it's amazing to see a real physical working relay computer! Incredible work!
The tuning is almost identical to what we have in production. A few comments:

* I would probably go with ashift=12 to get better compression ratio, or even as far as ashift=9 if the disks can sustainably maintain the same performance. Benchmark first, of course.

* We came to the same conclusion regarding AIO recently, but just today I did more benchmark, and it looks like ZFS shim does perform better than InnoDB shim. So I think it's still fine to enable innodb_use_native_aio

* We use zstd compression from ZFS 2.0. It's great, and we all deserve it after suffering through the PR dramas.
> help protect against AWS nonsense

Not gaslighting, legit questions follow:

It's not clear to me how taking open source software and providing commercial managed services is "nonsense"? Or is it specifically when AWS does it that it's not okay?

Either way, is there something specific at play that makes it fundamentally different from say WPEngine running hosted Wordpress? Is it purely a function of scale?

If a startup decides that Kubernetes is worth providing a managed service for at what point does providing that managed service become "AWS nonsense"? Is there a revenue watermark beyond which what they're doing becomes unfair? Or is k8s ok because it's backed by a mega-corp already?
The source to their server seems open source licensed and available: https://github.com/signalapp/Signal-Server

Wikipedia also says their server is open source: https://en.wikipedia.org/wiki/Signal_(software)
You are talking about a v1 UUID.

A random number is a valid v4 UUID, provided you set the version and variant bits to the correct value. A random 128-bit number won’t have those bits correctly set (except by chance). Set them, and you’ve made a valid v4 UUID
The same bullshit as US medical prices, it seems:

- Hospitals have an interest in massively inflated high sticker prices to make as much money as possible from those who pay in cash, e.g. foreigners, and to make the debts look better for collection agencies - basically, assuming the true cost of a procedure is 1000$, when the hospital bills 1000$ and that goes to collections who buy the debt for 10% of value the hospital gets 100$, and if the hospital bills 10k $ and the collections agency pays 10%, the hospital gets 1000$ or the actual cost

- Insurances have an interest in high sticker prices because they will negotiate with the hospital to pay true cost + some markup anyway - so with a higher sticker price, they can claim that their insurance saves the buyers a higher percentage

- Employers have an interest in high sticker prices because they can market themselves as employers who provide a better health insurance than the competition

The people losing out in this gamble are those who cannot afford insurance and have to declare medical bankruptcy.

And in the case of hardware or even some "contact sales for a quote" SaaS it's the same end result: the ones who lose out are small businesses (who can't achieve the sales amount to qualify for cheap-ish rates), and the big companies with dedicated account managers have a nice life.
On the complexity of JSON serialization (2020)
Australia is small in terms of inhabitants, far smaller than France, the rest of the EU, BRICS and other relevant markets. The business impact of shutting down search for Australia will be relatively small.

So Australia just volunteered to be Google's example for what will happen if you try to milk Google...
Should we scrap STEM in high school?
This does not need to be a war between Signal people and Telegram people. It's not an either/or scenario. For those people who desire E2EE by default, well let them use Signal in peace. For those people who desire the greater flexibility of infinite independent chat clients and a higher uptime, let them use Telegram in peace.

The Signal community took a wrong turn into if it's not E2EE we hate it. There are people who should be using E2EE but it's not everyone and to insist on this narrative is going to drive a lot more people away from Signal than it is going to recruit.
If you think that's bad...

https://en.wikipedia.org/wiki/Geneva_Freeport
Uuid v5 is namespaced.
I spent part of my early youth in the North of Botswana.

Coming from a densely populated part of Europe, one thing that stayed with me, is the incredible size of everything. Maybe for an Aussie or someone from rural USA it is not so special, but driving to the nearest town took a day. Our car had an extra large water and fuel tank (making the backseats somewhat uncomfortable for us kids). Over a 1000km of nothingness (that's from Antwerp to Kopenhagen). Just dry shrubs, an accidental hut, A lonely baobab-tree, one tiny fuel-station, and sand. Dusty, red, dry sand (it gets everywhere).

I think Botswana has another "diamond" here: cheap, unpopulated, unusable land. With highly predictable, and heavy sunshine, makes for a perfect solar deliverer.

Now, if only we could transport it as easy as oil or gas, and it could probably deliver large percentage of Africa's energy needs in coming decades. Because as far as I know, Botswana is still one of the least populated countries in the world (per km²). So it's not as if the local economy uses giant amounts of energy on itself.
While the report mentions Threema, there are quite a lot more messaging apps. I found a nice list comparing them here: https://www.securemessagingapps.com/
Isn't the iphone camera a primary phone feature - used primarily for still photos, and secondarily for video?

I've seen apple billboards about ONLY the iphone camera, so apple is investing money in it.

Meanwhile for a laptop, people don't care (as much) if it takes 10 megapixel images and can see in the dark. Personally I would like a laptop without a camera (or with a hardware switch)

(and I've never seen a macbook billboard)
It does look pretty awful:

"Although the Treasurer, in making the determination, must consider whether there is ‘a significant bargaining imbalance between Australian news providers and the group comprised of the corporation and all of its related bodies corporate’ the determination is not invalid for failure to consider this (s 52C(3))."

So there is only one factor they're required to consider for that portion, but they can choose to ignore it without apparent consequences. You can tell this was written specifically as a blatant money grab, something more legitimate would have factors beyond "because we say so" even if only to provide cover.
All major components communicate via grpc so rust is not out of the picture. GC-less languages might make sense for storage-drivers.
Alphabet Is Grounding Project Loon–But Won't Call It a Failure
> Because rent has a tendency to always increase, as supply is necessarily limited. Even the very highest density cities like Hong Kong have extraordinarily high rents.

Hong Kong tightly controls land - only something like 25% is developed. As for rent always increasing, we’ve seen rents fall in major cities during the last year as demand plummeted relative to supply. We need to build more housing - rent control, as in a hard price ceiling, tends to disincentivize doing that, while also tending to decrease the quality of existing housing if rents are set too low.
FWIW this also exists in the Netherlands, it's very handy when I have visitors as I can go along to most of the places they go without having to pay, or just when I have an afternoon free and want to go do something.
I am no fan of Signal -- their well-known anti-federation rants are basically a dealkiller for me. This does not exclude Telegram being even worse.

But sigh that the "secure messaging app conundrum" is apparently limited to two terrible choices.
Good riddance. Google is a nuisance and one of the worst offenders in the global offensive against privacy. Anyone remember the "Don't be evil" thing?
I have been involved with open source since 2000. I have been a member of the ASF for many of those years and worked with Apache licensed projects throughout. I have been in the position Elastic is in where I managed the employer-sponsored team working on an employer-sponsored Apache licensed project, though thankfully not against a behemoth like AWS.

I have some sympathy for Elastic, but I just do not agree that AWS is doing anything wrong or against the spirit of open source. When they run into things that need to be fixed or improved in the projects they upstream them. The majority of work AWS does is in "operationalizing" the project for their cloud. It might be cool and interesting if they let us see how they do this but it is not like that work directly translates back to the open source project. Anyone that has worked with Google open-source stuff has run into the same issues where there is some behind the scenes magic that Google has internally that the projects can leverage to make them scale better etc.

It is Elastic's right to change the license if they so choose. If they own trademarks that they believe are violated it is up to them to take those challenges to a court and seek damages. Elastic is trying to have it all ways though. They want you to believe the new license is open source .. sort of true but not really. They want you to believe AWS has done something wrong. This I do not buy at all. What is special about Elasticsearch? The entire Internet and the Cloud is running on all kinds of open source software up and down the stack. It is in the interests of the behemoths that run these stacks to upstream fixes and improvements and generally they do. That does not mean they have to contribute the proprietary wrappers they have put in place around them as part of building their business model.

AWS needs to be focused on its customers, not companies like Elastic and Mongo. If its customers would prefer to operate an Elasticsearch service provided by AWS as opposed to Elastic, then that is a problem for Elastic to resolve.
> News companies cannot demand payment while they demand that Google indexes their pages for them

You have got it wrong.

Nobody asks of Google that they pay for indexing news sites in Google Search. And no news company wants to force Google to include their content in Google News.

The thing is there are two different services, Google News and Google Search. The news companies have a beef with Google News. Google has been saying: if you do not accept us using your content in Google News, we will remove you from Google Search.