skysight-inc/HackerNews huggingface dataset:
The leaked Google memo "We have no moat, and neither does OpenAI" is instructive here:

https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...

Original author is an ML researcher, and the crux of his argument is that most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language.

The remaining misses are facts that it has never seen, usually because they are so obvious that nobody thinks to write them down explicitly. And so more training data doesn't necessarily help LLM performance, unless you're either ingesting either extremely basic facts that are so obvious most adult discourse overlooks them, or you're ingesting expert knowledge that's highly specific and only discussed in a few forums. Reddit data could perhaps help with the latter, but a.) Reddit is usually not the place to go for expert discourse and b.) there are other better sources of data for it. You'd usually be better off training on trade publications, scientific journals, or fandom than Reddit.

Also the LLaMa/RedPajama approach makes this stupidly easy, because you can pass around patch sets to the model trained on specific mini-corpora, and then update the weights appropriately. Hence why the author of the Google memo believes neither Google nor OpenAI has a viable moat.
Thanks! So is this more like an arduino then?
LLVM is because they don’t want GPLv3. They just went ahead with a stack where they can be in control on their own terms.

Basically GNU gambled and lost, they wanted to make a big stand against software patents and some other powerful forces had different opinions.
Google drop their "do no evil" motto and jack up ad prices.

Raising prices is evil? Does this hypothetical online store never raise prices?
I know the podcast app Snipd (https://www.snipd.com/) offers some similar things. It will build chapters with AI generated summaries of each chapter.

Not sure if they are leveraging that for search and discovery or not but it looks like they do (https://www.snipd.com/podcasters)

A user can request it for any podcast and it doesn't seem to take that long the times I have tried it.
I think AOP is a fair tool.. for maybe 1% of of your day to day problems.

I'd say it's okay to use in combination with OOP when you (and your team) manage to strictly keep the Domain/Business stuff in OOP world and the infrastructural stuff (logging) in AOP. Whenever you start mixing any of these together or have overlapping concerns, then things will quickly get very messy. While logging is fine with AOP e.g. with Access control things easily get mixed up over time.

When I look back the last 1-2 decades, the pattern that I see is that one principle always won: Simplicity.

I'd like to end up with a little war story:

In 2017 I happened to work at a startup (~60 engineers) that started with a gazillion of Microservices in dozens of different languages and ecosystems. Then there was this Clojure guy that didn't even do that much of missionary work. When people started reading each others codebases they realized the simplicity of Clojure and later started porting/rewriting their things to it.
At the same time, everything Scala was frowned upon; I vividly remember trying to understand the request a supposedly simple CRUD service was making to another service and I couldn't understand the program flow for like 2h.
Needless to say that probably like 80% of the services ended up being rewritten in Clojure and the Clojure guy got called the Johnny Appleseed of the company.
Cloudflare proxy error here too.

Archive.org was astonishingly quick:

https://web.archive.org/web/20230614125253/https://programmi...
That sounds completely different. Blocking the word 'keygen' accomplishes absolutely nothing and is clearly stupid. Blocking build scripts absolutely stops a major attack vector.
> I genuinely don't understand the appeal of AI for search

If you're good at googling the flow is: Ask the question > Clock which result isnt spam and click it > Figure out how to dismiss the cookies gate without accepting the cookies > Dismiss the google login box > Dismiss the popover pushing you to install an app > Scroll the page or ctrl+F to find the answer

With ChatGPT it's just type your question and your answer is appearing right away.

Of course I understand ChatGPT shouldn't be used for this because it will lie to you and make things up. But I am saying that's why you'll see people who don't know that preferring it over Google.
I’m the author of the article and the CTO of Appsmith, an open-source low-code application builder platform. Since Appsmith is primarily a self-hosted software, we decided to forego Docker’s suggested best practices of single container, single process. We decided to run 5 services inside a single Docker container managed with supervisord! This keeps Appsmith’s “monolithic” philosophy alive while still modularizing the components inside.

Users definitely succeed a lot more with installing Appsmith on their servers, but I'd love to hear feedback on any pitfalls or unintended consequences that I probably didn't think about.

PS: @todsacerdoti thanks a lot for posting this article :)
Whole point of federation is to eliminate the need for centralized platforms, which will never not self-destruct like Reddit is currently doing.

If you realize that one casino is ruining your life and go looking for a different casino, you've failed to grasp the problem.
> The attitude of the police and the population at large is if you left ANYTHING in your car it’s your fault for baiting the car.

... Is that not the attitude of the police _everywhere_, though? I've never heard of the police anywhere taking much of an interest in items stolen from cars; it's not like they really have a realistic way to pursue that even if they wanted to.
Having had a few years of intensive, classical-trained piano lessons at a young age, the notion of "deliberate practice" has also just meant "practice" to me: break down what you're learning into its separate parts, figure out which part you're weakest at, try to further isolate that specific thing into its own exercise, then do it slowly, gradually increase the pace, gradually re-incorporate the aspects you removed, play that part with the preceding music, play that part with the postceding music, play it without the sheet music, etc. Each step should feel uncomfortable at first, and once you get comfortable with it you move on to the next step until you can play it perfectly without any thought.

Kahneman has already touched on this, but obviously not all skills are predisposed to this type of practice. IIRC, this method of learning is useful for skills with shorter feedback loops (e.g. music, dance, gymnastics, some aspects of sports, etc.) and neutral to practically useless for skills with longer feedback loops or that can't be clearly broken down into its constituent parts (e.g. investing, science, entrepreneurship & innovation, etc.)

I wish all skills could be learned this way—it's the method of learning I'm most comfortable with. But of course, life is not always so easy.
> There is no question that Reddit is extremely valuable as training data. How often do you append “reddit” to your searches?

Is it?

When I worked in networking, and later web dev work I found Reddit to be a TERRIBLE place for Q & A type situations.

Answers on Reddit are often skewed by truthy answers from people with limited perspective in the industry who are surprisingly sort of militant about a given topic.

For example the networking topics often were “small shop” focused and users unaware of very common enterprise level practices, and when they saw someone mention it they would react really poorly. The result was advice tailored to small shops that have limited staff, so you got ideas that “work” but would perform poorly at scale (at best) and at times could present security risks, miss opportunities where more performance was needed, better solutions were available with a sizable staff and planning. They weren’t wrong, but the answers were skewed.

Subreddits are weird, they might be about a topic but the community often moves that topic/ has a specific pov within that topic… people without those views tend to abandon those subs and those who remain aren’t necessarily “right”. Reddit isn’t stack overflow where being correct is at least on the surface valued where we can run code and see if it works. Top answer on Reddit could be wrong and anyone taking issue with it may simply not answer. Heck there are whole subreddits dedicated to wallowing in misinformation…

Now I don’t know if Q&A is the start and end of LLM training (not my area of expertise at all, I am open to the possibility that what I’m talking about isn’t a problem at all) but Reddit as a source makes me wonder what the results would be.
Very cool! I'm curious--I'd imagine that some long tail podcasts have transcripts that are too long to fit within a standard context window. Do you have some strategy for handling these?
Maybe if reddit developed some expertise in using all of the user data they have to tag, categorize, annotate every comment, they would have something of value to provide. Their recent actions only angered users and won't stop future LLMs from using their data for the reasons many others here have noted.
Very interesting read. If they're going all out (Reddit co.) it means this is important
Label data and build a meme classifier? Does not have to be perfect to be useful. But yeah, data curation is probably a huge endeavor at the companies making Language model that are fit for production. Like in practically all applications of Machine Learning.

But the Reinforcement Learning from Human Feedback (RLHF) is also one of the key tools to getting useful outputs.
The point isn't really to discover the opinions of redditors, it is to ingest the 'common sense' things that you would never find out from reading scientific papers or even books.
You can be compensated the same way we compensate the earth for real oil: by burning your output to create a toxic environment for you to exist in.
It's hidden from the perspective of code represented as individual byte stream files, but that doesn't need to be the only representation, though obviously it is the most common form. Or, your IDE could surface this information.
Man how insane is it that these guys and gals raised 400 millions on tech without any raw data, nothing to show for it? It was never reproduced? They just said they got it?
depends of where you go on reddit. i've learned a lot of things for my hobbies. for example r/espresso and r/roasting are a source of good information. there are also places like r/askhistorians and many many otheres. reddit is not just r/funny.
Aka “Gell-Mann Amnesia”

https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect
A heat pump is much more effective if you only have a small temperature difference [0]. If your house is well insulated and you have underfloor heating, you can heat your house with 40°C (sorry for European units) water to keep it warm inside. If you have poor insulation and small heating elements, you need much hotter water (>60°C).

A heat pump can have a carnot efficiency (theoretically achievable efficiency) of >10 to warm water from 20 to 40 degrees, but only ~5 if you have to heat it from 20 to 60 degrees (not sure about the numbers, but you get the idea).

[0]: https://energyeducation.ca/encyclopedia/Carnot_efficiency
could you please elaborate on what the actual realworld problems are with "slow performance", if my customer is in Germany, but the transmission of email happens from Newfoundland, Canada, is it really that terrible that it takes 100ms more to send the mail? do people notice?

a far far bigger impact is if mails arent transmitted immediately, but every 1, 5 or 10 minutes.

Am I missing something here, or is this just marketing for the sake of marketing?
Floored the Complete Documentary Film
>Apollo probably made more profit from Reddit than Reddit itself.

that's not saying much when Spez outright said reddit isn't profitable. Yes, a few devs working on an API scaper is cheaper than 2000 employees worldwide, for a company HQ'd in San Fransisco on top fo that (I still can't imagine what reddit needs with 2000 employees).

meanwhile I don't exactly think Spez is surviving paycheck to paycheck himself. If he's not already, he'll easily clear tends of millions of dollars of net worth when Reddit goes public. Or at least, that's what I would have said 4-5 months ago.

>you can't expect to generate profit from another business unilaterally forever and be shocked that they won't allow it.

on the flip side, I don't think Christian is suddenly broke either. Probably not "I can retire in prauge" levels of finances, but he's definitely made a lot of money.

But he's under no obligations to play ball with Reddit, especially when reddit is taking the ball home. I don't see this as him leeching off reddit when you take into account how many users his app brought to the business. Just like how you said RiF is your Reddit, I'm sure Apollo is others' reddit.

>As people have said countless times in other situations where people had issues with reddit; just use another platform.

I agree and am. But that's tangential to this article. Christian isn't exactly trying to kill reddit nor pitch his next Apollo app for the next big platform. He said his dream scenario is to wake up and see the API rates halved and he'll be happy and keep Apollo running. He clearly doesn't want reddit dead.

But as is: he seems more or less done.
UTF-8's original specification included 5-byte and 6-byte encodings to cover the complete astral plane (31-bit code points), but later specifications have marked those "invalid" today due to the current 21-bit limit of UTF-16 and to align both specifications for now rather than fix the bugs in UTF-16 (or scratch UTF-16 altogether). In theory, UTF-8 can even extend beyond 6-byte encodings (and UTF-32 into 8-byte encodings and beyond) if the next plane (63-bit code points) or the one after that ever needed to open up. (No one expects that any time soon, of course. Today Unicode is nowhere close to in danger of filling 21-bits much less 31. That would be a massive shock and the compatibility headache would be terrible with UTF-16 breaking or today's software breaking that hard codes the assumption that UTF-8 should never go past 4-byte encodings.)
https://www.id.me/business/identity-verification

> ID.me uses liveness detection and advanced machine learning technologies to determine match between applicant’s image on their government identity document and the applicant’s image on their selfie.
> , that data can and will get scraped if the intention is to use for a model.

How would that work from a legal perspective, though?

Let's say there's no paywall and Reddit's terms of use disallow unauthorized commercial use of their data. Wouldn't that be a violation of Reddit's terms and liable to some legal procedure?
It really really really needs to fix the topological naming problem. It also needs a UI overhaul and some convenience functionality to knock some of the rougher edges off common workflows. It's possible to do a lot with FreeCAD, so in that sense it's not particularly limited, but it's deeply annoying to have to jump through some of the hoops it puts in your way.
Basically the only thing that comes to mind is that our collective views and knowledge about topics can change over time. If we had a hypothetical LLM in the 1930's, it would have had quite different views on things like mental health, civil rights, and so on.

But like you said, the models would ideally be trained on many other data sources as well.
http://muratbuffalo.blogspot.com/2016/09/sonification-for-mo...

Sonification for monitoring and debugging distributed systems
Our CLI already uses genkey as a command so unfortunately that's no bueno:

keygen genkey

Can't do genkey genkey. That's just weird...
I guess I have my answer here (link suggested by another user here): https://news.ycombinator.com/item?id=35080777
I feel like I’m taking crazy pills: Reddit is on CommonCrawl, which means its all available without API access, permanently from AWS’s CDN. These discussions are completely moot because the path of least resistance was already available and in use by OpenAI
That would put suicide hotlines, many driven by volunteers, in serious trouble of being sued.
Airbnb is still great for the original use case, which is renting a room in someone else’s apartment. Zero hotels in NYC, London, LA, etc. are cheaper than a room on Airbnb.
In most companies, PMs make roughly the same as engineers at the same level, so your wife’s situation likely has other factors. Did she make the jump to another company? If so, that’s why, not the ladder switch.

Also, PM requires real skills too, as you can see from a lot of the comments here. Unfortunately most PMs don’t have those skills, so I don’t blame people who don’t know that.
> Compared to when I lived in Zurich essentially all government officials were willing to even just entirely speak English after briefly suggesting we speak German and only rolling their eyes slightly that I wasn't competently able to. Navigating the system there as a "first year resident" was zero problem by comparison.

Switzerland has 4 official languages in a very small territory and even more dialects variations so there is that.
What is a PM?

I'm told I am one. Because I am the manager of a piece of software the CTO has passed on to me.

Besides leading the develoment, what should I do?
I suspect this is due to a fundamental misunderstanding of what tools like ChatGPT actually are, what their capabilities are, etc. I think there is a large population of people who think they're simply more sophisticated versions of Siri. In fact, I'd go so far as to say that the vast majority of people don't even realize they're powered by different things, but instead just see the "marketing term currently known as AI" as a singular monolithic entity, and all related technology just gets lumped into the same category. That's miles away from understanding how "use a language model to interpret what I'm asking you to search for, and then plug that into a normal search engine, and read me back the most relevant result", is fundamentally different from "use a language model to predict the stastically most likely next sequence of tokens based on the ones you gave me."

I don't mean this as a way of thumbing my nose at people who don't know this stuff, but rather, to point out what a massive failure the completely nonexistent consumer education surrounding ML products has been.
> Plus one of the more normal characters investigating the phenomenon just happens to be a vampire who--surprisingly enough--is neither formulaic nor boring.

The writing-guide-esque "OK, now write down ten wild elements or characters that certainly do not fit in your world... flip page ... and now add one of them, finding a way to connect it to some other element you've already established" was almost comically transparent, but also so damn effective that I've added it to the ol' toolbox.

(I mean, I don't know that Watts literally did that sort of exercise, exactly, and even doubt that he did, but in my head that's definitely how that part got in there)
Even a very low penalty isn't something that the industry is willing to produce..

For example, the RISC-V is derived from the MIPS, the MIPS had some instruction to trap on integer overflow, it's a very minor penalty to have these instructions, and guess what? They haven't been included in RISC-V..
ppl have to see every CEO posts
Fair, thanks !

I realized that looking for reviews like this I would look for reviews in my local language, which you probably can't.
This is the logical endgame for the search engine form factor. Remember watching mom ask google questions using complete sentences and punctuation with a “please and thank you”?

The appeal of a conversational response was too great to wait for a solution to the problem of training data quality.
yes, but they were already there right? Not targeted out of country

and also just to be clear, I think this is equally bad when everyone does it
Do they have to start over from scratch or can they use all of the data they currently have and then either add more scraped data that has been curated by humans or just outright buy data that isn’t publicly available.

Considering that RLHF took GPT-3 from a text completion model to an instruction following chat bot, you could use expert feedback to fine tune the model in whatever domains you wanted or a mixture of domains to produce an even more generally capable model.
Yeah, it's definitely not a guarantee but there are already viable paths out of the mess. I just wanted to push back against the notion that it's somehow inevitable or a forgone conclusion that it will happen.

Personally, I'm hoping we will see a Cambrian explosion of new LLM models and approaches. We've seen some beginnings of this in image generation so it's not entirely implausible.

Another thing the study doesn't capture: What is the effect of combined human + AI content? It's plausible that an explosion (due to lowered barriers) of new human guided/augmented ai content could counteract the effect.
Yes. For anything from opinions of movies/games to DIY advice to nerdy stuff like the best watch to buy in a given price range or the best synthesiser. Or people's opinions on a particular episode of TV! Or basically any hobby.

It's not great for business stuff because there are way more people using e.g. AWS for their hobby project than running multi-million $ businesses on it.

What are the alternatives? Facebook and Discord are walled gardens. Quora is a shithole. The rest of the internet is blogspam and significantly more dubious than your average Reddit comment!
The really impressive feat here is not even that, but MS inventing time travel to acquire Github before OpenAI was a twinkle in Sam Altman’s eyes
> I would love to see Google leaving Europe. I bet within a few weeks proper maybe even better replacements appear in Europe. Maybe even one of mine :)

Big words, but then I google your github and apparently you are planning to substitute biggest search engine, mobile OS and email service with 3dbriks[0], that, hilariously, "works best in _Google_ Chrome"? Come on now, there are multitude of reasons why European tech is not competitive and "evil American corps monopolized everything and won't let us in" is not the main one.

[0]: https://github.com/holoduke/3dbricks
GeyKen?
While as a product owner I'm not primarily a people leader (although I do) I got a significant pay bump switching from a lead engineer to a product owner at my company.
I mean, that would be kind of insane, but it could actually work: OpenAI needs "clean", high-quality data as input for its AI; rather than harvesting data to sell to advertisers, they could and finance major internet infrastructure -- the equivalents of StackOverflow, Wikipedia, Reddit, Twitter, Facebook -- to harvest data to feed their models. That might actually make the world a better place.
Maybe it's time to convert base64 to high UTF8-areas. I guess there won't be any keyword-filter yet for ancient egyptian hieroglyphs. I'm curious whether there are already emoji-filters...
I think the governance of the freecad project has recently changed so I'm cautiously hopeful that it will finally start seeing some project management and UX love.

https://lwn.net/Articles/924953/

https://ondsel.com/blog/freecad-approach-to-software-develop...
Expected, by who? What a silly article title.