skysight-inc/HackerNews huggingface dataset:
How do I set up my own GPT search assistant?
Hi HN,

I have around 1000 research papers saved over the years that I would like to be able to search in, in a smarter manner than using CTRL+F. This is a personal tool for me to use doing work, and will not be shared with anyone else. It will run on my PC with 4 ghz cpu, 32 gb ram and 8 gb vram/gpu. Nvidia gpu.

I tried Pautobot but it is somewhat slow, which probably has something to do with my hardware. I also tried GPT4all but I couldn’t quite understand how I could train it on my data.

Basically I am looking for a search feature that can rank and return the best matches/paragraphs from these articles. I think vector searching sounds like the better option, but I have trouble finding an easy solution that involves vectorizing PDFs and then being able to search in them documents.

Pleased to hear your inputs

Note: I cannot write code.

Link:
https://github.com/nrl-ai/pautobot/
NC DMV replaced all of their HTML forms with chatbots and it drives me nuts because I used to be able to pay my taxes and fees in under a minute, but now I have a stupid chat bot interaction that takes closer to 5, because you can no longer just complete the form and submit it but have to answer queries from the bot one at a time with some BS fake typing delay.
Check Cortilia.it in Italy if interested.
Failing the moon landing wouldn’t kill our supply chains. It’s a silly comparison.

How do you think we get all our food and raw materials around? With trucks!
> pushed into taking these experimental treatments for a few months

It's a shame that this person was in such a difficult situation, but, barring some extraordinarily rare reaction, a few months would not cause long term problems. The full impact takes several years, by 3-6 months the very first changes are only starting to become visible and many of them are things like changes to skin quality and fat distribution which revert after cessation.

It's still possibly a failure that your acquaintance got on HRT or Lupron in the first place. Given that long term follow ups show, consistently, that less than 2% of trans youth "desist" and are happy with their treatments, the reasonable solution to that failure would be to require some sort of evaluation by a therapist who has been trained to spot when someone has gender dysphoria and belongs to that 98% of happy transitioners versus someone from the 2% group that will desist because their issues came from somewhere else.

Instead of that sort of reasonable response, we see calls for outright bans on all treatment with many also calling for banning treatment for adults and a reversion to conversion therapy as the only "treatment" - it's not a treatment because it doesn't work and just hurts people.

The reason for this, is that the vast majority of people who suddenly have such strong opinions on trans health care are not actually concerned about people's health. They've just found a way to make their prejudices against transgender people sound like reasonable concerns.
> Appeal to authority is the last resort of those who have no other ground to stand on. If you want to elevate the discourse, engage with what people are actually saying, with their ideas, instead of dismissing them as idiot outsiders.

I'm not appealing to authority, I'm appealing to evidence/experience versus ignorance and Big Important Feelings. As a scholar and an educator, I'm the first person to say how important it is to cite your sources and to argue from evidence.

So if you're going to write 12,000 words about how the education system sucks and how to get through it, wouldn't you say that some of those words should be devoted to citations? Or at least the recitation of facts? Or barring that, at least establishing the background of the author to dispense such advice? Is a single citation asking too much?

There is one external authority cited in this entire 30 page blog post: Paul Graham. I guess that really says it all though, doesn't it? 14 year olds don't know who Paul Graham is. It really tells me the intended audience of this is not 14 year old kids, but for HN.
why no https on this site?
A general question to anyone in the industry: how come after so much experience with over budget/over timeline projects, the industry still vastly underestimates those variables?
This is the EXACT same absurd UX issue for public transit buses making stops. And even more absurd? It's illegal (in Ontario at least) to not yield to the turn signal. The issue is that with an obscured curb-side light, you can't tell when a the road-side light goes from 'stopped' to 're-entering traffic lane'. I have no idea why we can't add additional informational signalling to these special-case vehicles.
> We know the data survived a narrative/biological co-evolution distributed process spanning thousands of generations over many millennia. It doesn't make it perfect, but it makes it pretty darn good with high likelihood,

This is fallacious. Already responded to this at length.

> which have yet to withstand the test of time and which show measurable signs of dysfunction.

Unlike the older traditions, which manifestly have horrible dysfunctions and consequences for a lot of people in a lot of areas. Should we never try to fix and improve those failings because we "dont know what might happen"? Or is that unknown worth it to actually have a shot at really making things better? Why are we acting like we can't have a reasonable idea of how good or bad an idea is based on reasoning through the idea, why is the null hypothesis that any societal change could cause civilizational collapse when there's no evidence of that? It's like the people constantly claiming gay marriage will cause the collapse of western civilization. Also, why are we weighing the known horrors of our inherited traditions so lightly you barely even acknowledge them, so much more lightly than the vague, unknown disasters of... civil rights or gay marriage. Who knows what disasters letting women vote or have their own bank accounts or get jobs might bring! The horror!

> See also the Lindy effect, https://en.wikipedia.org/wiki/Lindy_effect "the future life expectancy of some non-perishable things, like a technology or an idea, is proportional to their current age"

Lasts a long time =/= good.
I think you've missed the point about TFA's discussion on grades. Here's a link so you can go back: https://skunkledger.substack.com/i/125171481/b-dealing-with-...

Basically, high grades aren't the end-all, be-all of school. They aren't a way of defining ones' worth. The analogy of treating school like a job really applies here. TFA also points out the risk of bad grades.

I should point out that TFA's advice works for conventional high school, but less for prep school. I had so much homework in prep school that it was nearly impossible for me to pursue the kinds of things TFA advocates for. Had I known better, I would have insisted on going to public school. (But all my friends were going to prep school; another mistake from TFA that I fell into.)
True, I think if they fought today Zuck would win. However, given enough training, Musk can close the gap. If he trained hard and they fought a year from now I'd put money on Musk, hands down.
Last time I tried to tinker with Windows XP few years ago: you couldn't just update it after installation, but if you let it work for a few days, eventually it'd download and install updates automatically. And after those updates are installed, you can actually use Windows Update UI to install optional updates and other things.

It definitely was after 2011.
https://archive.ph/RwIze
I’m convinced this is group hallucination. It must be so interesting to work at OpenAI, knowing you didn’t change a thing, and seeing that because of random chance, some small fraction of 100M users have all tricked each other that suddenly, something is different.
It seems pretty obvious that it's no more dodgy than any other app they have. Despite people crying loudly about how untrue it is and how unfair it is when sites block people in the EU, the EU's regulations require a lot of work to comply with (even in the case where you are already complying with the intent of the regulations). Clearly here Meta's number one priority was getting this app out quickly. There are a ton of pretty basic features missing. If supporting a release in the EU required just weeks of work, I'd expect that they'd have chosen not to delay the release for that. Given the extra scrutiny that Meta is under, I'd put the amount of work required at more like months.
> First, in the moment, be conscious of what the risks are, and if they are worth it.

Is that a happy life? For every decision you take, every thought that crosses your mind, making a concious cost-benefit analysis?

> According to the author that's a failure, but I had a lot of really, really enjoyable meals. I ate the vast majority of them with friends and family. Many of them were either cooked by family, or were special occasions. Calling those meals "failures" now would be saying that it was a mistake to enjoy that time with my family and friends, and it absolutely wasn't.

Perhaps it wasn't the meals that were the failure, but the lack of compensating physical activity? Discussions of body weight issues seem over-emphasize blame of too much on food and diet and neglect sedentary lifestyle and lack of physical activity. In other words, barring serious medical conditions (many of which are treatable once diagnosed properly), you can have it both, enjoy good times with friends that includes eating really tasty food as well as not into health-affecting weight issues.
Capital is also produced by labor... so the aane thing applies to those workers except to a more extreme degree.
Reality is a notebook in the attic, if found, will at least get a few minutes interest. Only the crazy narcissists think anyone is going to care about their curated babble once they're gone.
Cloud Compute Cost Comparison
Performance normalised cost comparison of over 650 VM types, spread across 100+ regions and three hyperscalers.

A little HPC and FSI specific.

Feedback very welcome and happy to chat about the results to anyone who is willing to listen to me drone on about it :)
Seems like a pretty broad brush there
Sorry for the typeoh in the above.

Examples: compassion, loyalty, endurance/perseverance, social/independence, love/hate, trusting, cynicism/optimism, etc.
There’s nothing invisible about this. We talk about it all the time in literally every discussion about primary causes of high house prices, right up there with stupidly low interest rates for way too long. Unlike interest rates which can be easily changed, restrictive zoning is primarily a Boomer NIMBY problem, pulling up the ladder behind themselves.
My milkman (London) has been driving an electic float https://en.wikipedia.org/wiki/Milk_float since forever.

This is not reinventing the wheel, more like rediscovering the wheel we have actually had for 100 years.
TiddlyWiki! - One of the best and most underrated apps anywhere - seriously, check it out - I promise you'll be impressed even if it's not ultimately the best choice for you.

I'm in the process of migrating OneNote over to TW now. TiddlyWiki is extremely solid and stable, totally open (and hackable if you want to do that), capable of doing a surprising variety of things well (esp. with some plug-ins to add new capabilities), includes decent encryption, and stores your notes in a plain HTML file, though there are several ways to keep/sync TW to the cloud, too (requires a backend server or file sharing, obviously...) Being entirely JS/Browser based is the reason my 20-year-old TiddlyWikis still work as well today as when I created them.

The biggest advantage is that YOU are totally in control, with NO need to rely on any Evilcorp's support or services to access to your most important info.
Why, because you actually have to build something, not code that millions of people can download an app.

Here in Austin TX, there is a company, ICON that is trying to mass 3D print homes. It is not going so good. They are slower than framing, drywall & mason crews. Also final finish is ugly in my opinion.
So irrelevant metric to judge the impact of vans by number of packages delivered.
Gotta say, knowing Elon's sense of humor, he'd love that Threads has 69 million users.
There is no meaningful sense in which Homebrew is (or ever will be) "finished": it's constantly being updated with new package versions, by both demand and design.

Analytics are an essential burden-reducing component of the Homebrew maintenance workflow; the maintainers would not have time to make improvements like the one you're requesting if they spent their time on the things that analytics do for them.
> The best practice is not to allow passwords for root login.

My sshd is configured to not allow passwords logins at all: it's U2F/Yubikey only.
I think in principle this is a good idea, getting actual usage patterns not just from a loud minority seems very useful to the people making the OS. As long as they don't collect egregious stuff I'm very okay with this.
Nice try, OpenAI
You just got hit by a spam filter. I've turned that off now. But please send questions like this to hn@ycombinator.com, as the site guidelines ask (https://news.ycombinator.com/newsguidelines.html). They're off topic in the threads.
I worked in construction for ten years and then worked at a construction management software startup for a year, where maybe I could see the writing on the wall better than others.

The problem is that nobody can align the incentives of the tradespeople and the office folks when it comes to integrating into an information system together. You have to cross multiple organizational boundaries between owner, builder, contractor, and subcontractor of which very few people have a bottom up understanding. Mostly the people with the incentive to put everyone on a single platform (owner / builder) think very low of the tradespeople, which is why tradespeople are beginning to make more money than the accountants and PMs.
Guaranteed some of them will start grifting, taking bribes, dodging taxes and selling snake oil (perhaps even a little ai of their own making), while others will protest endlessly to deaf ears, struggle paying rent or starting families and living a life of misery, and those in high paying jobs, while still worked to death and lacking freedom, will turn a blind eye to everyone's suffering. Free pizza and bean bags.
Without concrete examples, I do wonder if much of this is perceptual. I love using ChatGPT, but once the amazement that it works as well as it does has worn off, one ends up spotting the flaws more than before.

I feel that advocates and critics of ChatGPT are both right, to a degree, but looking at the models responses from slightly different angles: it wouldn't be surprising if users' angles shift over time.
I just found the first English translation of the Septuagint (the Greek translation of the Old Testament which pre-dates the Masoretic Text) translated by Charles Thomson. It's quite lovely. A good one to add to your collection.

Volume 1: https://archive.org/details/oldcovenantcommo01thom/page/n5/m...
Volume 2: https://archive.org/details/oldcovenantcommo02thom/page/n3/m...

I'll also point you to Brenton's translation of the Septuagint, which is also a good addition if you don't already have it. This particular version has the original Greek and the English translation in parallel:

https://archive.org/details/septuagintversio1879bren/page/4/...
That's exactly what they say in the beginning:

> First off, this is an observational study, not a randomized experiment. I think we may be reaching the limits of what analysis of observational data can do in the education debate and, given the importance and complexity of the questions, I don't understand why we aren't employing randomized trials to answer some of these questions once and for all.
Comments like this are why I don't really buy the "distraction" framing. Folks will bend over backwards to find any pretext justifying the murder of people they don't like, no distraction necessary.
Wow, I had no idea what any of that meant. I thought these were scripts helpful for... the tiny fraction of people that work in tech, but live off the grid?
Exactly this. Using automation, you could make $60/hr nearly two decades ago. Without automation, a few bucks an hour and a lot of your work was rejected for payment.
This keeps happening
Wonder if this is the same as the discussion from 35 days ago on "OpenAI Employee: GPT-4 has been static since March"

https://news.ycombinator.com/item?id=36155267
I mean, given that this site itself almost certainly doesn't comply with GDPR, yeah, I think you're right.
> - most of the industry is decentralized small contractors. You can't go viral with construction, you can't build monopolies to extract rents later on.

At least around my area (in the PNW somewhere), and specifically for residential construction, the opposite happens. The builder has a bunch of small contractors he uses -- but they are only legally separate. And disposable, to be crude. They arrangement is a monopsony -- the plumber only has one customer: the builder. Same for the electricians, framers, etc. Even though they're not employed by the builder, they might as well be. Hell, even the realtor who sells their houses is an independent contractor with only a single customer. Guarantee she isn't making 2.5% per sale.
If Medicaid and Medicare were "socialist" in any way whatsoever, we would have Medicaid for All for a fraction of the cost. These programs are so expensive because the "healthcare" system in this country is designed to funnel money into the oligarchs' pockets. It is a capitalist scam that presents the single greatest barrier to American entrepreneurship.

The UK NHS is a socialized healthcare system. The profit motive is not allowed to corrupt any part of the system. Pharmaceutical prices are strictly controlled (none of this $26k/year Alzheimer's drug nonsense). It is fully nationalized. Nationalizing an entire industry in peacetime is often (but not always) associated with socialism.

Even Canada's single payer system is not socialist in any sense of the word. It's just a government-run insurance plan. There are many multi-payer insurance systems in the world that perform far better than ours. Single-payer is not a panacea without price controls. Price controls are somewhat socialist. Funnily enough, the last US president to enact price controls was Richard Nixon, known friend of Marxists across the globe /s.

There's nothing socialist whatsoever about Social Security. It does not redistribute wealth. The tax is infamously capped at the first ~$130k of income you make per year. Any income above the cap is untaxed by Social Security. Your benefits are proportional to what you paid into it. Social Security is definitely not "wealth distribution." It is an income tax. It's income distribution if anything, but that's still pretty ghoulish framing for a stipend for retired folks that barely keeps them out of poverty.

> I've seen estimates that in the most progressive states, welfare benefits can amount to as much as $42,000 a year for a small family.

No citations, just some half-remembered statistic from a right-wing propaganda outlet that conveniently adds up to the weed number. Very believable. Very representative of the average person's lived experiences. I cannot overstate how pathetic you have to be, in order to be resentful about welfare benefits while making a tech worker's salary.
> offshoring

I wonder if we'll reach the point where "AI" ends up with the same negative connotation as "Made in China".
After the construction industry is done answering that question, I have the same question for the software industry. ;)
We are trying to codify this: https://puthir.org.

My son (and our first learner) launched his portfolio (real work, not fake stuff) back in December and he added his new math toy last month. We expect his portfolio (+ his experience, skill and knowledge) to grow over the next couple of years. Here is his Show HN post: https://news.ycombinator.com/item?id=36603838.
Note that this post is from May 28, before the release of gpt-4-0613. By "the last two updates", I believe the poster is referring to some UI changes, that possibly also included some underlying model changes(?)
Not exactly related, but it makes me wonder what kind of metrics you can devise for tracking "regressions" on LLMs.

I feel like OpenAI definitely has thought about this at length, but I'm curious about what matters most for them / raises OpsGenie/whatever alerts. Internal model metrics? Customer usage patterns? Random test conversation diffs?
This also affects Gitlab Pages. If your site is hosted there, your visitors are getting 503s.
Observing large groups of humans acting like large groups of humans is a very interesting pastime.

Some people even make it their life's work.
They've definitely changed something about the models, and it is in their interests to do so, both to create a low-latency experience, but most importantly, to save money.

While GPT-4 is still workable, GPT-3.5 flatly refuses requests these days, claiming that as an "AI language model" it couldn't help me write code.
is this your summary or chatGPT? Serious question, no judgment
If you have an argument for why supply and demand doesn't apply to housing, I'm here for it.

And I say this as someone with a large house that's nearly doubled in value since I bought it with only 5 or 6 years left on their mortgage. Personally high house prices benefit me.
You are talking about ambiguous wording in contact language, which is very different than it being ambiguous whether someone was entering into a contract or not. The latter is not a thing; both parties have to reasonably understand they are entering into a contract for it to be valid.

It can be implicit acceptance (such as paying a cash for something without first signing anything), but the action that constitutes entering the contract must be clearly-defined.

This is a bad ruling.
Yes, you're right. I meant, users who choose to opt out are more competent users (or just more paranoid) and might sooner understand what telemetry means and/or does. Usually, and I am assuming, competent software users tend to have much more unique and specialized usage patterns than the average joe.
I've seen posts similar to this one maybe every week or two in various GPT forums. I suspect it's just an illusion, where you remember all the amazing hits that GPT4 had when you first started using it, and remember fewer of the times in the past that GPT4 gave a sub-par answer. In fact, it reminds me of the illusion that "Hacker News is turning into Reddit" and I think that it happens for a similar reason.
Deffo agree and have witnessed the changes first hand
Didn't they fine-tune it for instruction taking and function calling (basically to the use-cases people started having)?

Maybe that had some unexpected side-effects.
i wish there was a search engine that could find sites like that. there's so much value in people's personal experiences. For instance, i've searched for blog posts talking about individuals experience with setting up green houses and what temperatures they can keep overnight vs daytime. all the major engines refuse any attempt to find something that's not on a super high level domain or news site.

most of the web is unreachable. sigh....