skysight-inc/HackerNews huggingface dataset:
Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs
Hey HN! We’re Kevin and Steve. We’re building PromptTools (https://github.com/hegelai/prompttools): open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts.

Evaluating prompts, LLMs, and vector databases is a painful, time-consuming but necessary part of the product engineering process. Our tools allow engineers to do this in a lot less time.

By “evaluating” we mean checking the quality of a model's response for a given use case, which is a combination of testing and benchmarking. As examples:
- For generated JSON, SQL, or Python, you can check that the output is actually JSON, SQL, or executable Python.
- For generated emails, you can use another model to assess the quality of the generated email given some requirements, like whether or not the email is written professionally.
- For a question-answering chatbot, you can check that the actual answer is semantically similar to an expected answer.

At Google, Steve worked with HuggingFace and Lightning to support running the newest open-source models on TPUs. He realized that while the open-source community was contributing incredibly powerful models, it wasn’t so easy to discover and evaluate them. It wasn’t clear when you could use Llama or Falcon instead of GPT-4. We began looking for ways to simplify and scale this evaluation process.

With PromptTools, you can write a short Python script (as short as 5 lines) to run such checks across models, parameters, and prompts, and pass the results into an evaluation function to get scores. All these can be executed on your local machine without sending data to third-parties. Then we help you turn those experiments into unit tests and CI/CD that track your model’s performance over time.

Today we support all of the major model providers like OpenAI, Anthropic, Google, HuggingFace, and even LlamaCpp, and vector databases like ChromaDB and Weaviate. You can evaluate responses via semantic similarity, auto-evaluation by a language model, or structured output validations like JSON and Python. We even have a notebook UI for recording manual feedback.

Quickstart:

pip install prompttools
git clone https://github.com/hegelai/prompttools.git
cd prompttools && jupyter notebook examples/notebooks/OpenAIChatExperiment.ipynb

For detailed instructions, see our documentation at https://prompttools.readthedocs.io/en/latest/.

We also have a playground UI, built in streamlit, which is currently in beta: https://github.com/hegelai/prompttools/tree/main/prompttools.... Launch it with:

pip install prompttools
git clone https://github.com/hegelai/prompttools.git
cd prompttools && streamlit run prompttools/ui/playground.py

We’d love it if you tried our product out and let us know what you think! We just got started a month ago and we’re eager to get feedback and keep building.
> disrespecting privacy rights

It's not as if these companies are kicking in your door and violating your right to privacy. You're accessing their site with a device that is configured to transmit whatever you have it set to.

If you don't want cookies, disable cookies. If you want greater control, go and configure it yourself. Stop forcing your preferences on everyone.

The reality is that outside of a vocal contingent on HN, most people simply do not care. They won't pay a cent for their ad supported services. And I for one hate the endless consent popups and GDPR hoops I have to jump through. As an expat in London, I can't read many local news stories in the US because those sites simply block the traffic instead of trying to comply with a foreign law.
More discussion: https://news.ycombinator.com/item?id=36947393
I don't think this is a good idea at all. This is like saying you should buy lifting shoes, a belt, a pair of gloves and bands before you go to the gym for the first time. Literally any setup that does the job is fine when you're just getting into it — you don't know what you don't know and having a cheap entry point is a good way to figure out if you like it at all, what kind of a workflow works for you, etc.
I usually right click these and choose ublock origin's block element to make it go away. I actually don't know what that means as I never answered the question one way or the other.
He referenced his interview with NewsNation.
Semi-related but it's shame default cron in most (all?) kinda fucking sucks at everything.

There is no builtin spreading so */5 job on 500 nodes will get you nasty spike in CPU load. Logging options are pretty much "parse emails and hope for best" and some syslog spam that's mostly noise if everything is working fine. And fucked up environment vars that trip many script-makers. No metrics whatsoever
Chiliz/Socios.com | The Sports Blockchain | On-site or local remote (several locations, mostly Europe) | Full-time | https://www.chiliz.com/ & https://www.socios.com/

We are building the web3 infrastructure for sports and entertainment. Socios.com is our fan engagement app, where they can participate to activities around their favorite team.

With Chiliz Chain, we are opening our ecosystem to all Web3 builders.

We are currently hiring in Spain (Madrid+remote ES), Malta (Gżira+remote ML), France (Lyon+remote FR), and Turkey.

Tech Roles:

* Engineering Team Leads

* Frontend Software Engineers

* Backend Software Engineers - PHP Stack

* Senior Backend Software Engineers - Java Stack

* Senior Platform Site Reliability Engineer (SRE)

* Senior Platform Software Engineer

...and more!

Join us!
https://www.chiliz.com/careers/
Yes, it seems weird so many HN folks seemed to pine for the office when the statistics show a different picture.
New replication video coming from China

https://twitter.com/lereguy/status/1686363900651151360
> The simple truth is we don’t have a culture of replication because no one is paying and no one is paying because no one is demanding it as part of the incentive system that the money is part of…

A darker comment: the non-replication of results implies that nobody's actually making use of the science output, and an alarmingly large fraction of unreplicated positive results are simply a waste of time.
>is that primary and secondary schools that are in not very good areas are not ambitious for their pupils.

And where do you think the vast majority of immigrants live?
In Boulogne Billancourt?
Preprints that explained OPERA's faster-than-light neutrinos
> I'm really glad I chose it over Lexical and other options. Congrats

Do you mind expanding on this? I've been comparing the two and found the lexical data model easier to understand.
About three years ago I was looking for an editor and while TinyMCE was the absolute best, its pricing model was as bad as CKEditor's, so I ended up testing Tiptap in a Vue project.

It's really a great and flexible editor, with a couple of rough corners, but overall the only serious alternative to the above-mentioned editors. And since it has a sensible licensing model it is an overall good choice.
Rsyslog also supports sending to elasticsearch IIRC.
Not sure if it counts as noteworthy, but I'm submitting this comment via my TUI
web browser that I've been writing in Nim.

https://git.sr.ht/~bptato/chawan

Also, there exists another Nim web browser project; from what I can tell,
it's in somewhat earlier stages of development.

https://github.com/xTrayambak/ferus
The Useful Definition of Effort
> All of these videos show the same part of the flake being repelled by both sides of the magnet

I'm afraid we are not seeing the same, then. In this [0] video I can clearly see a different corner of the flake being repelled at 2:07 ~ 2:13 versus 2:27 ~ 2:30.

However, I agree that the other videos people like you have posted on this thread show materials that are not ferromagnetic.

[0] https://targum.video/v/2023/8/1/2534a4408ccce9c13a811e94f16d...
> Case sensitivity is unnatural and weird;

Case sensitivity is naive and simple, case-insensitive systems are complex and require Case Mappings [1] and Normalization [2] and those are ugly, unless you want a half-assed system that is case-insensitive only when you only use ASCII.

[1] https://unicode-org.github.io/icu/userguide/transforms/casem...

> There are different case mappings for different locales. For instance, unlike English, the character Latin small letter ‘i’ in Turkish has an equivalent Latin capital letter ‘I’ with dot above ( \u0130 ‘İ’).

> In general:

> case mapping can change the number of code points and/or code units of a string,

> is language-sensitive (results may differ depending on language), and

> is context-sensitive (a character in the input string may map differently depending on surrounding characters).

[2] https://unicode-org.github.io/icu/userguide/transforms/norma...
If you pay people enough you can get them to do nearly anything and there is currently a job shortage.
So they pick you up first and then if you don’t tip them up front they don’t let you in the car?

This doesn’t sound a little suspicious that this supposedly happens on a regular basis? Picking you up from a major airport where there are plenty of other drivers?
Clockwise | https://getclockwise.com | REMOTE first, office in San Francisco, California, WeWork subscription for distributed employees | Series C w. $76M+ in funding from Bain, Coatue, Greylock, Accel, Atlassian, Slack Fund, and other blue-chip investors.| US Work Authorization Required

Clockwise optimizes your team’s calendars to create more time in everyone’s day. Using AI, Clockwise creates space in our schedules to be present when we’re collaborating and focused when we’re working alone.

We call it a time orchestration platform, the modern way to bring schedules together in harmony at the organizational level.

Open Roles:

* Senior Backend Engineer (x2)
Pay Transparency Salary range: $160,000 - $230,000

Clockwise is looking to hire two Senior Backend Engineers. We value engineers who build for the end user, and are passionate about enabling our business to accelerate and scale. We're a humble, fast-moving, and fun team building a consumer-centric culture with empathy, authenticity, focus, drive, curiosity, and enthusiasm.

*Growth Marketing Manager
Pay Transparency Salary range: $105,000 - $155,000

Clockwise is looking for a dynamic, data-driven growth marketing manager responsible for our traffic and user acquisition strategy to bend the curve on our user and revenue growth. This role sits at the interaction of demand generation and user acquisition and you’ll need to balance both motions in service of our company goals.

Apply here: https://www.getclockwise.com/careers

If you're having trouble managing your work time, get started using Clockwise for free at: http://bit.ly/getclockwise

"Is Clockwise Worth the Hype?" (https://www.getclockwise.com/blog/worth-the-hype)

Let's help the world make time for what matters! We look forward to hearing from you!

The final salary offered to a candidate may be higher or lower dependent upon several factors including, but not limited to, market rate, the qualified pool of candidates, internal compensation, candidate experience, interview performance, references, and budgetary constraints.

Ctrl F keywords: Java, Python, Kubernetes, AWS, Jenkins, Terraform, Postgres, Kafka, Redis, Tensorflow, Docker, DevOps, Backend, Growth, Monetization, Software, Product Marketing, Demand Gen, User Acquisition, SEO,
> Website analytics can be incredibly useful for designers and developers

I'd have sympathy for these people if they weren't also primarily responsible for the many darkpatterns, traps, and user-hostile aspects of modern interactivity.
I guess they could bet heavily against themselves, try to get prediction markets to coalesce around the result being genuine, them come out saying "oops, we were mistaken"? Bam, rich!
Its easy to believe RTO would be a bandaid to a degraded social life or non-optimal at home situation. And for some it might even be true. But by large your social life can and should thrive outside your job. You just have to work for it.
> People listen to a talk, or read an article, because they want to learn something new.

That’s surprising. /s
> "Reduce US car accidents by 50%"

That's not something that needs AI at all. That's a political problem, and the solutions are not ones that Americans like.
> baking something in an oven under vaccuum for a week to get.rid.of.all.the.water.

vacuum oven also works great for drying 3d printer filament!
Didn’t GRRM use Word Star?

https://www.reddit.com/r/freefolk/comments/s770y1/this_is_th...
I agree with the first part of what you're saying to an extent - it makes sense that higher monetary cost items have higher costs in general - that's pretty much just describing scarcity.

I don't think your second statement logically follows though, because environmentally damaging costs are not all equal.

As a thought experiment, being able to extract a bunch of X mineral from a 500 mile deep quarry might be both extremely expensive and extremely environmentally damaging in that localised area, but if the mineral can be used to generate a bunch of clean energy or capture carbon or whatever, then in net it can work.
No, they are putting them there because it’s easier and they don’t want to be fined.
SEEKING WORK | Europe | REMOTE | US timezones compatible and preferred

I'm an independent consultant who helps businesses achieve their goals with my AI/ML expertise. I craft effective technology solutions tailored to specific business needs, working with a range of clients from early-stage startups to publicly traded companies.

If you have a data problem you need solved in an effective and pragmatic way, don't hesitate to reach out.

Past jobs: Google AI, Buyside Quant Researcher

Availability: In two weeks

Technologies: Python data stack (pandas, numpy, sympy, etc.), ML (Pytorch, sklearn, lightgbm/xgboost), C++

Email: hn@chrispotempa.com

Site: https://chrispotempa.com
They don't pay corporate income taxes without a profit but they still pay plenty of tax:

1. Sales Taxes
2. Payroll Taxes
3. Property Taxes
etc
TSDBs don't work like that. They generally will choose to store a slope, not absolute values. They're very efficient at storing tons of data like this.
The “0 to 1 Trap” and Seven Other things I learned from Peter Thiel
Sorry, I can only make games that are fair to each player. Not a completely rigged game where the ones with most points can stay on top using rhetoric, lies, and financial corruption.
> how about directing your hate towards the bad players of the web?

That's like hating a rock for rolling downhill. Regulation is the only way.
I'm guessing he forgot null check
JavaScript is the fastest scripting language in existence and it's not close.

It's strange that people automatically suspect it has to be the culprit as opposed to e. g. bad application programmers or (more likely) product owners putting more focus on features at the cost of optimization.
SEEKING WORK - Contract, Remote, based in NYC area, backend engineer

I am a backend expert, having built and maintained highly available systems for several companies over the last 7 years, in addition to over 20 total years of software development experience. I learn new things extremely quickly.

You will like working with me because I:

* hold your goals as my priorities
* communicate verbosely yet synthesize well
* my rates are reasonable in exchange for the value I deliver
* I guarantee I will get the job done

Concrete examples:

* built the streaming data architecture for realtime game player information and game stats for one of the top 3 US sports leagues (can’t name drop!) using Redpanda/Kafka on Azure+Terraform, and implemented its developer portal using Gravitee to provide first class streaming API support and monetization.

* Implemented a vector DB for one of the leading AI image hallucination platforms (millions of users) to provide an image similarity service.

* replatformed dozens of jobs to AWS Lambda for a startup hedge fund, vastly improving QoS and maintenance cost. Line 1 to platform-in-prod time 3 months.

* built a ~50 node, AWS, Dockerized, Terraform-deployed distributed system to provide realtime, transformed trade data from crypto exchanges to a Kafka pipeline and data lake (fodder for an analytics platform). Met 5000msec SLA across arbitrarily many exchanges and ticker symbols. Blue-green deployment for complete CICD safety. Cloudwatch metrics proved SLA was met and provided general system oversight.

* built a framework to collapse dozens of singleton ETL programs into one config-driven, simple C# application performing the same functionality at ~80% reduction in maintenance cost year over year.

Contact: jon.north@adiuvat-consulting.com
More: www.adiuvat.io

Keywords: Python, C++, R, SQL, NoSQL, Go, Golang, AWS, Azure, Lambda, Cloudwatch, Docker, Git, SVN, Subversion, Linux, Windows, Bash, Kafka, Terraform, Redis, Postgres, MySQL, SQL Server, LDAP, Active Directory, OAuth2
It wouldn’t surprise me if Waymo/google decide they just want to remain on the tech side, and avoid all of the headaches that the operations side brings.

Waymo and Uber have a partnership. Soon in Phoenix, when you call an Uber, you might get a waymo car.

We might see a three group market:

Waymo, Cruise, etc provide the driver. Uber, Lyft provide the platform. Hertz, Avis, Enterprise manage the physical cars/cleaning/etc.

https://waymo.com/blog/2023/05/waymo-and-uber-partner-to-bri...
> I feel the ideal environment is contextual and it’s probably a hybrid work environment where people who want to work remote can but have to touch base in-person a few times every month

This necessitates living near the office/only hiring people that live near the office. I will not let my employer dictate where I live and I will not live in some of the most expensive real-estate markets in the world
Maybe, but by being a paying customer with lots of options (including collective negotiation), you have a lot more leverage to get them to do what you want.
Essentially all of my knowledge of materials science comes from reading HN's discussion of this over the last ten days, but I've read repeatedly that these kinds of synthesis processes are not very reliable. Even with well known materials with well known "recipes," someone trained can follow a procedure apparently to the letter and get no result, and then repeat the procedure apparently the same and get a good result.

Semiconductor yield at high end fabs can be lower than 20%. If you tried that process at a new fab and wound up with a complete failure, would you say "well that process just doesn't produce any semiconductors" or would you think you might need to refine the implementation of the process.
I cannot find the specific posting for the Senior Low Latency Linux Engineer on your careers site. I believe we are a good fit: https://news.ycombinator.com/item?id=36957947
NextUI Version 2.0
Location: Belgium
Remote: Yes (EU/UK timezones)
Willing to relocate: No
Technologies:
* JavaScript, React, Next.js, Tailwind CSS
* NodeJS, Postgres
* Docker, Terraform, AWS, Azure
Résumé/CV: https://registry.jsonresume.org/ianchanning
Email: ian_channing[at]hotmail.com

Hi,

I'm a Tech Lead, full-stack developer with 19 years of experience.

I've successfully built web platforms for multiple startups across AI, mechanical engineering and statistics.

I work with early stage startups using whatever technology is most suitable for their domain.

I love functional programming and building things from scratch. Check out my JavaScript perceptron from scratch: https://github.com/ianchanning/neural-network-js. I like to share my knowledge through understanding and explaining complex topics - I have 26k Stack Exchange reputation https://stackexchange.com/users/128896 (if that still gets you any kudos).
TestFit | Senior C Software Developer - Procedural Generation | Full-time | Remote (CST +/- 2)

We design buildings in milliseconds. Our real-time automation allows real estate developers and architects to design projects in a fraction of the time required with traditional CAD software packages. It can be hard to describe so I'd encourage you to get a quick glimpse of our neighborhood layout tool at https://testfit.io/release/testfit-3-07-subdivision-configur...

We are hiring a senior software engineer to join our small team and add more functionality to our procedural generation systems written in C99. If you like solving tough spatial problems using 2D vectors and geometry, managing your own memory, and optimizing algorithms to run in under a millisecond, then we might be the company for you. At TestFit were are seeking to create a company of people who are capable, humble, and hard-working.

You can apply at https://jobs.lever.co/testfit/62bb6dfe-fd34-41c3-8cf3-5318a0...
Incredibly bearish stance from HN here.

Uber creates major consumer surplus for both riders and drivers.
The foundational math skills are linear algebra, calculus, and statistics. They are bog standard math anyone with a university education in the sciences should be comfortable with. The only math that's possibly more obscure are the higher level statistics tricks like graphical models, but those can be picked up from a textbook.
When using the Uber app, the only time that I have been able to apply a tip is after the trip is complete.

How would a driver insist on a specific tip before even accepting? This makes no sense to me.
>A problem with all this tech is that to the extent it can be used to make business problems easier to solve, it makes it much harder to introspect what running software is doing, which from a software freedom perspective tends to raise hackles. My hope would be that in general, this is more used like 'corporate TLS interception' insofar as your personal device does not do it, but I'd expect that mobile device vendors use it before too long.

A fair point. Although, given that corporate environments have been doing this sort of thing securely (in a variety of contexts beyond just browsing) for a long time without such remote "attestation," I'm skeptical that this provides much value other than for tracking/spying.
this changes everything (if it's true)
I feel like these memes completely ignore the fact that being an engineer exists as a career and is basically the trades job but for far more pay and with more thinking involved.
It's much more eye opening as well, productivity per hour of labour has doubled from the baseline right before 1990, salaries have definitely not increased at the same rate.
3dverse | Montreal, CAN | Onsite/hybrid (within CAN)

About us: We are building a real-time cloud-based 3D engine-as-a-Service to reduce the pain in building, collaborating on, and deploying 3D content at scale. We are a team of 12, almost entirely technical. Our stack is a combination of C++, Typescript and sprinkle of Go.

Role: Full-stack or Front-end Engineer (Mid to Senior). We are looking for a web dev to help us release our first public product, a web-based collaborative 3D viewer, built on top of our engine.

Stack: Typescript/React/Postgres. 3D or game dev experience is a plus (Unity/Three.js). Need a solid grasp of networking and web performance tradeoffs.

What we're looking for:
- An experienced builder with code and projects to show for it. You care about code quality and doing things well.
- A product mindset. You have opinions about design, UX and how to build products people love to use.
- A curious nature and a desire to tackle hard problems.

Tell us (or better yet, show us), what you've worked on and what your interests are. Contact me at mark at 3dverse.com

https://3dverse.com/careers/full-stack-web-developer
Contrary to Sensational Reporting, Summer Heat Is ‘Unremarkable’
I am sure you understand that not taking a path is not the same as not knowing the path...
Location: Boston
Remote: ok
Willing to relocate: yes
Technologies: Mostly C# and Winforms.
Résumé/CV: Last year developed CCTV software in C# for traffic project.
Email: ely@castellano.io

I'm also open to contract work.
Go 1.21 Release Notes