skysight-inc/HackerNews huggingface dataset:
Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
Hi HN!

We’re Guillaume and Lewis, founders of Dataform, and we're excited (and nervous) to be posting this on HN.

Dataform is a platform for data analysts to manage data workflows in cloud data warehouses such as Google BigQuery, Amazon Redshift or Snowflake. With our open source framework and our web app, analysts can develop and schedule reliable pipelines to turn raw data into reliable datasets they need for analytics.

Before starting Dataform, we managed engineering teams in AdSense and led product analytics for publisher ads. We heavily relied on data (and data pipelines!) to generate insights, drive better decisions and build better products. Companies like Google invest a lot to build internal data tools for analysts to manage data and build data pipelines. In 5 minutes I could define a new dataset in SQL that would be updated every day and then use it in my reports.

Most businesses today are centralising their raw data into cloud data warehouses but lack the tools to manage it efficiently. Pipelines run manually or via custom scripts that break often. Or the company decides to invest engineering resources to set up, maintain and debug a framework like Airflow. But that’s just for scheduling and the technical bar is often too high for analysts to contribute.

We saw a need for a self-service solution for data teams to manage data efficiently, so that analysts can own the entire workflow from raw data to analytics. We built Dataform with two core principles in mind:

1. Bring engineering best practices to data management.
In Dataform, you build data pipelines in SQL, and our open source framework lets you seamlessly define dependencies, build incremental tables and reuse code across scripts. You can write tests against your raw and transformed data to ensure data quality across your analytics. Lastly, our development environment also facilitates the adoption of best practices, where analysts can develop with version control, code review or sandboxed environments.

2. Let data teams focus on data, not infrastructure.
We want to bring a better, faster and cheaper alternative to what businesses have to build and maintain in-house today. Our web app comes with a collaborative SQL editor, where teams develop and push their changes to GitHub. You can then orchestrate your data pipelines without having to maintain any infrastructure.

Here's is a short video demo where we develop two new datasets, push the code to GitHub and schedule their execution, in under 5 minutes.

https://www.youtube.com/watch?v=axDKf0_FhYU

You can sign up at https://dataform.co. If you're curious how it works - here are the docs: https://docs.dataform.co and the link to our open framework: https://github.com/dataform-co/dataform

We would love to hear your feedback and answer any questions you might have!

Lewis and Guillaume
My point is that people should at least know the basics of recursion. Mentioning "programming without loops" without even mentioning that recursion is an option in that specific case seems neglectful to me.
> tell if their results were what the user wanted

That's ranking.
How to Cut Down on Unwanted Junk Mail
https://www.nytimes.com/2018/03/08/smarter-living/how-to-cut...

I get very little paper mail now except for companies that I already have accounts with (and who seem to believe that “paperless” doesn’t apply to the marketing departments).
Intro to ATS Programming Language
I think my all-time record was receiving a tiny metal pencil sharpener all by itself in a box large enough to fit a computer monitor.
You'll find that a lot of businesses use Word and Excel because they have a large, large body of institutional knowledge built up in those systems, and especially in Excel, a set of business-critical files that nothing else can open properly. They aren't picking Office because it's what the kids fresh out of college know.
You wouldn't fly on an airline that has had zero fatalities in its 34 year operating history? Ryanair definitely has had issues (particularly in the early 2000s with pilot fatigue) but there's a lack of basis to suggest they're unsafe.

I won't fly on Ryanair myself, but it has nothing to do with their safety record (which is fair to good), it has to do with the way they treat customers and employees.
I never claimed otherwise ;)

However, I will claim that the lower operational expenditure (and lower liability for holding user data) makes profitability on-the-whole easier to achieve in Blockstack-powered apps.
A few years ago, a friend of mine, in her 20s, worked full-time as a technician in a university-affiliated research lab, she managed to find a steal of a studio apartment (half the price of a bottom-end one-bedroom in the neighborhood), and she bought awful cheap bulk food... but money was still so tight that, in the evening, when she was too tired to do anything else, she did these online "gigs" that paid only $1-$2 an hour.

Since she could write, she mostly did writing assignments (which I suspected were for SEO Web sites), in which they tell you a topic and how many words to write, you research and write the article, and you get paid a pittance. But $2 will buy steel-cut oats for the day.

Her time should've been worth more than $1-$2/hour, and I don't like the idea of companies arguably exploiting desperate people this way. Though it's not just companies: university researchers sometimes use Mechanical Turk workers to process data, and as research subjects.
I will assume you are asking this in good faith and not tone policing or sealioning.

95 - 99 % of the time when people refer to political correctness its because they are objecting to the changes in modern society they object to or that it challenges the prejudices they grew up with.
CuriousMarc's Apollo Guidance Computer (AGC) restoration is a must-watch: https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb...
I'm sure SAP will gladly spent a few billions of their money trying :D
Remember that not everyone writes code that is designed to run on a machine with lots of RAM. Loops vs Recursion can be a proper engineering trade off when resources are low.
Anonymizong search queries is extremely hard, if not impossible. See https://en.wikipedia.org/wiki/AOL_search_data_leak for example.
I canceled mine last year sometime, whenever it was they announced the most recent price increase. I thought I would miss it but I haven't really. I don't mind waiting a bit longer for the standard free shipping. In the rare occasion I need something fast I just pay for the two day shipping, which ends up being significantly cheaper than paying for Prime.
Wait, what? Ryanair have barely had an accident in 33 years.

https://en.wikipedia.org/wiki/Ryanair#Accidents_and_incident...
yes, of course. And obviously I would do it with loops, so you don't have to. Ultimately a Turing machine needs to have loops somewhere. The best you can do is abstract them.
"Facebook is controlling people's minds and ripping apart the social fabric. Also, you'll notice some 'share' buttons to the right of the article. Remember to 'like' us!"
Show HN: The Climate Fixathon- Online hackathon for makers to fix the climate
The Climate Fixathon (https://fixathon.io/) is a 4-week online hackathon for individuals or teams to use their tech skills to launch a website, app or service intended to help restore a safe climate for our planet. It runs from 2nd-30th August with a prize value of $10k+.

AWARDS

️ Awareness - Most likely to raise awareness of climate breakdown.
Action - Most likely to help people take action against climate breakdown.
️ Facilitation - Most likely to make climate breakdown related tech projects easier to create in the future.

WHAT YOU CAN WIN

Our 3 winning team will each receive $1,500 cash to spend.
Free Egghead.io licenses
Free Sketch licenses
Free Kirby licenses
Free Drawkit illustrations
100 trees planted by Offset Earth

Our 3 runner ups will each get $300 cash to spend.
25 trees planted by Offset Earth

We hope that given this opportunity, we’ll get to see hundreds of techies from around the world using their skills to help fix the climate. We can't wait to see what they make :D
Not necessarily. The fact that I brush my teeth in a certain way doesn't determine the kind of car I drive or whether I love my wife.
It's the opposite, actually - every loop can be trivially encoded as recursion, but loops can't encode many common recursive patterns without a lot of work.

Recursion also has the potential to be much clearer than loops in some cases.

Of course, there are also cases where one would do much better to not use recursion, but that's why I think people should know of it.
Moveable Is Draggable Resizable Scalable Rotatable
So schools should stop using tools that are useful in the actual real world? Ok, sounds like a great education. Instead of Photoshop, everyone can learn Gimp, instead of Word they can learn LibreOffice. Then when they fail to get a job because they have no idea how to use an Excel spreadsheet we'll chalk that up as a win for a free society, or something.

People will argue, well companies will just adjust and not use Office...no they won't. Switching costs are expensive, retraining is expensive, and let's face it Office is just better.
What company makes it hard to pick your flight that you’d need to resort to leverage?
Isn't it a fact that software engineering is inherently creative? While programming may not strictly count as art by most people's definitions, it is certainly not soulless or mechanical as your comment implies. In particular, Bach's music is so remarkable precisely because it combines mathematical patterns and subjective beauty.

I hate the phrase, but exceptional software engineers "paint with code."
I think they mean that the results are still from the top pages of the internet. They mean long tail of visited pages, not long tail of searches.

A unique search query could still land you on Wikipedia.
Wearable that translates thought into action (dev kit launching)
I don't understand why this is a paradox. Isn't it just obvious that no system can contain a complete model of itself? That would be an infinite regress and hence would require infinite information.
Its not about “disrupting” anyone. There are other forms of advertising than Google - there is Facebook, Amazon, commercials on TV, guerrilla marketing, getting “1000 true fans” and let your product spread organically, etc. if you are completely dependent on Google you are just one algorithm change from becoming invisible and you may need to rethink your business plan. It's kind of silly to start a business with your only method customer acquisition is depending on Google.

I’ve worked for B2B companies that actually had a sales team.
Why would you not need military force?
iirc Amazon worked like this for their first n years before the difficulties that resulted made a SOA more compelling.

I think there's a middle ground between microservices and a single monolith for a large global corporation.
I think it all depends on the adults in your life. Maybe if we'd had a leader who said don't do that instead of "well they can't feel pain and they're pests" things would have gone differently.
Yep, that's actually the last three lines I quoted from chapter 17 in a different translation.
Seems to go both ways though as I'm not getting any auto completion results for "donald trump stormy daniels" either. I'm guessing they scrub things that are highly sensationalized in the news.
Sliate
React Front End Developer Based in NYC
We are needing a strong Front End Engineer with experience with #react.js + #redux (must) and extensive exposure working in a environment where you contributed solely into planning, designing & provided technical output.

This role is Full Time, on-site (Manhattan). Send me through your application to misti@happyfuncorp.com
A recursive data structure is a data structure that contains a pointer to another instance of itself. Linked lists, trees, etc.
Ryanair have a stellar safety reputation.
Manually Suspend Browser Tabs
Really? I was in a 777 yesterday and felt like I was flying in a well-tested, mostly modern plane built with well-understood technologies. That, and I realized 737s (not Max, obviously) have loud interiors.
This doesn't forbid using recycled materials for cheaps products, you just don't advertise it.
Admiral Jeremy Boorda, U.S. Navy, enlisted as a sailor at age 17 in 1956; he eventually rose to become Chief of Naval Operations, the highest-ranking USN officer (save for when a Navy officer is Chairman of the Joint Chiefs of Staff).

https://en.wikipedia.org/wiki/Jeremy_Michael_Boorda
> Netflix beats them in videos

Idk... I feel like good video options are pretty well split between Netflix and Amazon with Amazon Video on the rise and Netflix in decline. Amazon still has random esoteric stuff I want when I look and they picked up The Expanse, which is a huge win. Netflix almost never has what I want if it isn't a Netflix Original, which are becoming increasingly mediocre (looking at you Stranger Things 3).

The difference I see is that Netflix has had only one job and they've gotten worse at it while increasing the price. Amazon prices have gotten worse, but they still deliver me any esoteric thing I want in less than a week, and it's usually something I can't get at a local store (usually books, and I live walking distance to Powell's and many other book stores which won't have the title I'm looking for).

For me it's a no-brainer, mostly because Amazon has a student prime price of $59/year which I can get as a 37-year-old grad student, but even without that Netflix wants the same amount per month for their basic streaming ($13/mo) as Amazon wants for a full Prime membership, which includes al their video streaming.
But doesn't the Law of Large Numbers disagree with this. Yes the first 5 Users (froma random sample size in the target market) could agree, when you get to larger numbers the real observations come out.
I'm not sure this answers the question. Clearly Vox has a progressive (maybe originally neo-liberal?) editorial bias, but that may or may not mean that they have dishonestly distorted facts. There is a big difference between editorial bias and dishonest reporting!
>People [wrongly] get called a paedophile for such behaviour and get pilloried in the social press.

I've never seen anyone being accused of being a paedophile for having sex with a 19-year-old. Do you have an example? This frankly seems like a backhanded slur on Turing.

> Not sure Churchill would make it on anymore either tbh.

Churchill was added to a bank note a mere three years ago.
The ‘classic’ ones were -300/-400/-500, and the NG were -600/-700/-800/-900. The MAX 8 is a replacement for the -800, and it has been called a 737-8 for a while – their FAA type certificates are 737-7, -8, and -9.
Depends, I usually stop at a store on my way home from work. Also, my neighbors tend to drive a bit more friendly then some of the strangers delivering random packages. The delivery trucks are always in a complete rush. I realize this is not just from amazon but it is the bulk and it is driving the behavior change.
I think a distinction needs to be made between monopoly and 'best product'. Search is winner take all in many respects. As a user, I don't particularly care about the second best (because I don't pay).

The best way to fix the situation is to get users to pay. And this could be done. Users can pay negative dollars (e.g. get paid). Why would a search engine want to pay it's users? Why would a retailer want to pay me to look at their ad? Why wouldn't they?

Example:. My wife and I buy a not insignificant amount of stuff from LuLuLemon. So I have a deal for Lululemon. Pay DuckDuck half what you pay Google to show me your stuff. And DuckDuck, pay me half what they pay you. Right now DuckDuck gets zero. Google probably makes 20+ bucks a month off me. In this math, DuckDuck nets 5$!

There is a natural equalibrium here. Advertisers quickly start paying zero to people that don't buy stuff, so users can't farm ads. But relevant customers get discounts.

So now I actually care about making the trade off between second best but effectively cheaper or best but more expensive. E.g. a functioning market.

To do this though, you have to start tracking users in some way. But Google already does this, so the situation doesn't change. Some form of surveillance is required to improve relevance. The difference here is, I can choose who surveils me and to what extent rather than being stick with the winner take all.
LLVM's Arm stack protection feature can be rendered ineffective
https://www.washingtontimes.com/news/2018/oct/30/hey-barack-...
For us, the recycling only gets picked up every two weeks, so that will be 50% more driving, not double.

Also, if the garbage trucks fill up slower, there should be some reduction in trips to empty them.
> Yes, my thought was that by breaking everything off from everything else, these silo'd services would suddenly have to compete with the rest of their market at fair terms

I think it's instructive to look at the rest of the market. How is Mozilla funded? Basically a single gigantic contract with Google. Even Apple accepts payment from google to become the default, and it's not cheap: https://fortune.com/2018/09/29/google-apple-safari-search-en... The same logic applies to pretty much anything Alphabet spins off -- there's little difference between ownership and those contract.

About the only competition this setup produces is the ability for Mozilla to walk away to a competitor bid, which they did for like a year before bailing out at the first opportunity. There's a huge incumbency bias in these contracts. The first parallel that comes to mind is employer provided health insurance. Everyone gets to bid, but the incumbent knows the claims history far better than the competition and we'd only expect them to lose bids to companies overly optimistic about that history. Google knows how valuable various traffic sources are, but their competitors have to guess, and only when their guess is higher than Google's does it pay off. Does anyone think Yahoo winning Firefox was a good deal? I haven't seen any analysis to support that.

> My hope is a complimentary effect to the above also happens: Google no longer gets gobs of personal data from its other services, allowing other search engines to approach its efficacy.

Wouldn't the most profitable thing for these broken up companies be to sell their slice of the personal data pie as many parties as possible? This seems like a net loss for privacy. How much extra would it be worth to set up an exclusive arrangement?
This just sounds like pissed off rant. Can you draw out your statements a little and explain them?
I'm also now sure where there is criticism that employees need to turn to 'federal assistance'. Amazon warehouse jobs have been on the high-end in terms of pay for this kind of job for many years. Well above minimum wage in most states.
Is this really true? At least in the US, I do not think the amount of cash (M0) is accelerating. Unless my reading of this is completely wrong: https://fred.stlouisfed.org/series/BASE That said the recession really throws this off.
Kind of a cool idea. The combined distribution would be even more interesting and maybe to include some more varied types of coding, most of the examples were OS, programming languages, dev tools. The author shared his script so it wouldn't be hard to adapt likely.

This would be a cool report for github or bitbucket etc to provide across public repositories. Especially when combined with any basic demographics. Of course, I don't think they collect a ton of demographics today so not sure how valuable it would be.

I bet you could build profiles on people and predict changes in their patterns based on life events and other changes (kids, marriage, aging, divorce etc). Of course, I bet many people would find that a little creepy, but the data side of it is pretty interesting, at least in my head it is.
Software is one thing, but middle schools providing iPads to kids makes my blood boil. How on earth can you make kids work on one of the most closed down ecosystems there are?

At least on windows I can install everything I want or even run my preferred OS on a virtual machine...
Ghost (https://ghost.org) is a lightweight WordPress with an Electron app for editing/writing. However, it isn't statically generated, I think all the content is stored in a database.
Wow 20 minutes away. That's impossible to get to. I feel your 2019 pain.
> Why aren't all the big companies Doing Search?

They are.
The jester's job is to criticise the tyrant in his own court. Just enjoy the show.
> there is a clear way to determine fault

Yes, but criminal intent is less clear. Someone can be declared at fault for hitting a pedestrian, yet not have been criminally negligent. And even if this happens a few times from really bad luck, it's reasonable that they'd get to keep their license [0].

> You can be 100% race blind and have the best intentions in the world ... bias due to the nature of the data

The question is what data? Are you feeding things in that have a clear causal relationship with your desired result? Or are you inputting everything you can hoping to discover correlations ? The latter is essentially trying to suss out informal groups, and engaging in any group-based discrimination means you cannot claim to have "the best intentions" - regardless of whether the group can be named as a legally protected one or not. If say you're try to base mortgage underwriting on credit card purchase data, it's wholly disingenuous to claim that the resulting fallout is "accidental".

[0] In a society where cars are de facto mandatory to get around. I will disclaim that this casual attitude is a large part of what keeps roads so hostile for everyone else, but it currently is what it is. Also historically we haven't had such a likely hidden factor as phone use.
To some "Authoritarian left" means anyone who things the government should treat people equally. Not saying that is what the original poster means by it but a lot of people in my experience use it they way I said.
From personal experience driving for Uber, I've noticed that facial recognition technology isn't that accurate if you're not white.
Yes, MAX is a tradename, -8200 is like the model-number and FAA uses that. Still grounded.
Wonderful choice!! Hoping for Rosalind Franklin next!
Given how they treated eg Alan Turing, I think you have a point :-)

(preventive anti-outrage disclaimer: I'm not suggesting that homosexuality is eccentric, I'm only suggesting that it might have been considered eccentric by many in mid-20th century UK)
...Yes?

Though I think what you may be actually intending to ask is is the airline bound to provide with a flight to your ticketed destination on another plane if they have to sub-in a MAX because of reasons.

That is a question with a far less straightforward answer; the last time I read terms they reserved the right to swap in hardware as circumstances warrant. You'll want to take it up with the carrier. Which I recommend everyone does; as they only pay attention when sizable customer groups start making a fuss.
Plus if you get a Target Red Card you get 5% every purchase.
A correction for you. 7060$ is purchasing power parity number, while the 11000$ is nominal.
Microsoft maintains it's own fork of react-native that allows this. https://github.com/Microsoft/react-native-windows

There's more experimental forks for macos and qt.
for what its worth, github suffers from the same issue. it's a "feature"