skysight-inc/HackerNews huggingface dataset:
It is pretty much the same as TFX - but with Spark for both DataPrep and Distributed HyperparamOpt/Training, and a Feature Store. Model serving is slightly more sophisticated than just TensorFlow Serving on Kubernetes. We support serving requests through the Hopsworks REST API to TFServering/Kubernetes. This gives us both access control (clients have a TLS cert to authenticate and authorize themselves) and we log all predictions to a Kafka topic. We are adding support to enrich feature vectors using the Feature Store in the serving API, not quite there yet.

We intend to support TFX as we already support Flink. Flink/Beam for Python 3 is needed for TFX, but it's not quite there yet, almost.

It will be interesting to see which one of Spark or Beam will become the horizontally scalable platform of choice for TensorFlow. (PyTorch people don't seem as interested, as they mostly come from a background of not wanting complexity).
My understand of Buck (and Pants) is that they are the result of ex-Google employees going to other companies (resp. Facebook, Twitter), realizing that Blaze was more or less the Right Way to do a build system (at least for their set of circumstances), and then being forced to re-implement the ideas from scratch (edit: or memory, or exfiltrated docs/code) because Blaze was not open source. Reusing extant software is great, but it requires that software to be available to you in the first place.
Server overload. You can get it from here for the price of creating a login: https://www.jstor.org/stable/1429971
Fearing More Violence, Sri Lanka Silences Social Media
> I'd love to see someone translate the concepts in both books to a useful postgres package.

This is some work on this. [1] is an amazing patch by some researchers in Europe that has already had several rounds of review. Their first published work was against Postgres 9.1! Their solution is way better than SQL:2011, but could be used behind-the-scenes to implement the features from the standard. Essentially they implement a temporal variant for every operator in the relational algebra (very elegantly btw, by combining standard operators with just two new functions). That approach solves some of the problems that led to Snodgrass's original temporal proposal getting rejected in the late 90s (and which are still holding back SQL:2011 IMO). Whereas the standard only gives temporal INNER JOIN, their work gives every join type, as well as aggregates (including a way to scale the inputs if the group only includes part of their time interval). It also lets queries compose better through subqueries, views, CTEs, set-returning functions, etc. In my opinion their work would make Postgres the most advanced temporal database in the world. I hope it gets the attention it needs!

I also have a very modest WIP patch myself at [2], which adds temporal-aware primary and foreign keys.

This is all based on the excellent work on ranges and exclusion constraints by Jeff Davis and others. Postgres has an excellent foundation, but needs the higher-level concepts so that users can more easily set up temporal tables and use them to answer queries and record changes.

If you'd like to learn more about temporal databases, I wrote a summary of the research at [3]. I'm also giving a talk at the Postgres conference in Ottawa next month.... :-)

EDIT: Btw, if all you need is transaction time (usually included for auditing and compliance), then the temporal_tables extension [4] should be all you need. Maybe also consider the pgaudit extension too [5].

[1] https://commitfest.postgresql.org/22/2045/

[2] https://commitfest.postgresql.org/22/2048/

[3] https://illuminatedcomputing.com/posts/2017/12/temporal-data...

[4] https://github.com/arkhipov/temporal_tables

[5] https://github.com/pgaudit/pgaudit
Labor regulation has historically followed market trends rather than driving them.

If you tried to pass a 40 hour work week in 1790 you would have been laughed out of the statehouse because it would have caused mass starvation since people had to work longer hours than that just to grow enough food to feed everyone.

Then industrialization rolls around and we slowly get to the point that labor supply starts to outstrip demand, and then we don't need everyone to work 80 hours anymore. Moreover, it doesn't cost much more to pay twice as many people for 40 hours as half as many people for 80 hours, but then you have built in slack if someone is sick or quits or dies because you have someone else to work a double shift temporarily. And research starts to show that not working people as long leads to more productivity. And there is less unrest and more consumption when you have everyone working 40 hours than when you have half of everyone working 80 hours and the other half unemployed. And the people who still prefer to work 80 hours instead of 40 just take two jobs.

Meanwhile you've had labor activists demanding a 40 hour work week for years, and once most businesses were either already doing it or getting ready to, opposition to the law becomes weak enough that they can finally pass it. Then for a hundred years they take credit for the thing that was already happening regardless.
I make heirloom yogurt from organic milk from a local dairy farmer, currently about 50 litres a week & increasing fast. Turns out real yogurt is real popular these days. Doesn't exactly pay the bills yet, but it's delicious!
Threatening to leave is not the best strategy at Google BTW. You never know (by design) if a counter-offer will be extended, and you're very likely to be taken up on your bluff. :-)
But so have resources consumption and money concentration.
This is my favorite side project that I've ever heard of. It's so cool how you stumbled into it and there's a lot of scope for other products too!
I'm curious about publishing a book, what are the typical length requirements? I have seen some books with 400+ pages. Is that what publishers require? Anyone has an experience where page count was much less?
Thank you for the informative reply. I had no idea we could get m.2 nvme on such a small board. Very nice!
If all you need is a pretty form interface, a place to show output, and dont mind a shameless plug, Gooey[0] can give a native UI without writing a single line of UI code

[0] https://github.com/chriskiehl/Gooey
I get the sense that Bazel will end up with a larger ecosystem than Buck, because Google’s Cloud business depends on open-sourcing more stuff, so they’ll put more effort into open-source Bazel.

Uber is migrating from Buck to Bazel, for instance.
The idea that several forms of social influx (immigration vs gentrification) have some things in common despite being received differently along a political continuum is probably worth some examination.

But I'd start by looking at whether "flood" or "deluge" are justified not in terms of positive/negative connotation, but whether they accurately represent the scale of what's going on.

IIRC, post-civil-war in-migration to the US hasn't ever resulted in more than 15% of the total population being immigrants. The CIA World Factbook says the US had 3.9 migrants per 1000 citizens in 2017. Wikipedia says there's a recent rate of about 1 million immigrants per year, about half of which are changing status for people already here.

I'm less sure where to get observations about the portion/influx of tech in the Bay Area or SF itself. But if I were guessing, I'd bet tech pushes up to 20-25% of employmees and activity, and that the influx over the last decade (or in the 90s) is a considerably steeper curve.

One might also consider some qualitative factors. An influx of people of average or perhaps even lower than average means primarily represents an impact on (a) competition in the labor force and (b) demand. Those two things ought to cancel each other out in terms of general economic impacts (or perhaps even work in tandem to drive growth), except where there are supply bottlenecks... say, real estate/housing, especially in a geographically constrained area. So, one might expect the impacts to be outsized in a place like San Francisco vs the United States at large (which has plenty of places with modest population destiny). An influx of people of average to higher-than-average means (say, workers in a field commanding a larger salary than many past residents) would massively magnify this problem.

So... perhaps there are good reasons to treat the volatility and dynamics of tech-focused in-migration to SF/Bay as distinct from the volatility and dynamics of immigration to the US.
Using Minio as an S3 Gateway in Front of Hadoop HDFS
Studies on this are very valuable. The IPCC's AR5 advises this kind of action to be funded and analyzes different policy approaches to enable action.

There is no justification at skepticism and entertaining back-of-the-envelope gotchas about the climactic and environmental value of tree planting and enhanced forestry at this late stage. It needs to be supported !

https://www.ipcc.ch/site/assets/uploads/2018/02/ipcc_wg3_ar5...

" Reducing emissions from deforestation; reducing emissions from forest degradation; conservation of forest carbon stocks; sustainable management of forests; and enhancement of forest carbon stocks (REDD) consists of forest-related activities implemented voluntarily by developing countries that may, in isolation or jointly lead to significant climate change mitigation. REDD was introduced in the agenda of the UNFCCC in 2005, and has since evolved to an improved understanding of the potential positive and negative impacts, methodological issues, safeguards, and financial aspects associated with REDD implementation. " (page 865)
I can't say I'm surprised. They're a bunch of data science people. Thinking about how to not let embarrassing things get indexed by scrapers is probably second nature to them in the same way that chaos engineering is second nature to plenty of programmers. I bet the question "should this be indexed" is the data scientist equivalent of "how could a malicious user exploit this".
It destroyed twttr beta app!
https://twitter.com/alollou/status/1120020332578996226
Show HN: Phylogenetic Tree of All Life
We are finally ready to reveal the Phylogeny Explorer project to larger public. Please visit us at http://explorer.phylogenyexplorerproject.com/

The following video by Aron Ra, project's founder explains how the project came to be and what hurdles does one need to overcome to build a tree of all living things - https://www.youtube.com/watch?v=vU7difOTyvk

We welcome contributors over at https://github.com/phylogeny-explorer/explorer
Ditto for me.
Did you know what typical minimum page count requirements are? I have a non-technical, more program management focused book with about 100 pages. Was told by a publisher it's too short... wondering how you approached that when self publishing, since your post mentions it's about 100-150 pages. Did you get any feedback from readers that it is too short?
Build a Feed Forward Neural Network in Python with NumPy
I wonder if the eventual fate of most lumber is to be burned, on the scale of tens of decades (as buildings deteriorate, are remodeled and older bits thrown out). In which case you're just postponing the need to sequester anyway.
> Only property owners should vote.

That's a good slogan for starting a war. One day, people like you will have their land taken from them. Will they still feel the same way?

The fact is, if you take away a person's vote, they have no voice AT ALL in their government. The vote actually happens to be the last thing PREVENTING war because the fact is, everyone would like to do what they want and not have other people tell them how to live. We have government to prevent people from hurting each other and doing mean things to each other, and though it may not be all that effective, anarchy never works and always ends in some kind of oligarchy.

People go to war whenever things are taken from them. Land, property, rights, etc. They are happy when things are given to them. Bread, circuses, etc. So if you want to start a war, take from people. If you want to prevent it, give to them.
> at an informal employee gathering at a bar ...

Why don't companies just get rid of these "informal" but sanctioned employee gatherings, particularly in a bar environment where intoxicating substances are likely being administered as well? That's just a magnet for these sorts of problems as are detailed in OP's blogpost. The workplace should be a temperate environment.
A drip coffee maker
Sri Lanka’s action “is a damning indictment” of companies that once portrayed the platforms as vehicles for liberation...

Last year, the government briefly blocked social networks after viral rumors and calls to violence, circulating largely on Facebook, appeared to provoke a wave of anti-Muslim riots and lynchings.

Government officials had repeatedly warned Facebook, which also owns Instagram and WhatsApp, that the posts could lead to violence.

Company officials largely failed to respond until the government shut down access, after which they promised to hire more moderators and improve communication.

And from https://www.nytimes.com/2019/04/21/world/asia/sri-lanka-expl...

Rather than trusting in Facebook and other companies to police their networks for hate speech or incitement that could arise as a result of Sunday’s attacks, the government was treating the platforms as too dangerous to remain online.
It's in the sourceforge folder: https://sourceforge.net/projects/supertuxkart/files/SuperTux...
Can you elaborate on why downloading from S3 at startup is a bad idea? And why not synchronous everywhere as opposed to always queues?

Good points overall that I'd agree with.
He is directly quoting the original blogpost's description by using that terminology, and he is responding to a post with the relevant link. That is the actual original description of the incident. It's not his own personal summary.

> "Specifically, in October 2017, at an informal employee gathering at a bar after a week-long company offsite, one of DataCamp’s executives danced inappropriately and made uninvited physical contact with another employee while on the dance floor."
This seems to say that societies where the rich consume more tend to consume their resources more quickly? I think the opposite is through though. The rich don't consume in proportion to their wealth, but doubling the income of plumbers and farmers does increase consumption.

Unless I misread the summary, I read it in a hurry
"I think it’s likely that DataCamp management thought or hoped that their post was enough to placate instructors, and that they essentially did what we asked in the letter."

Perhaps I am misinterpreting this sentence. But, there was a complain, that complain pointed actions that needed to be made, DataCamp did all they were told to do, and yet, the complainers are not happy, DataCamp didn't do what they needed to do (although they did everything they were asked to do).

Now put that together with the fact that the text basically doesn't give any real detail about what happened, and it seems yet another post by someone that enjoys being outraged and let everyone know about it.
I write nonfiction books.

I used to be a magazine/newspaper editor. About eight years ago, I made a career change and became a full-time programmer. But I wanted to continue to write. So I wrote a book proposal, found an agent who loved it, found some publishers who were interested, and went with a publisher who offered a two-book deal.

My first book, "Experimenting With Babies: 50 Amazing Science Projects You Can Perform on Your Kid" (http://www.experimentingwithbabies.com) ended up selling pretty well.

My second book, "Correlated: Surprising Connections Between Seemingly Unrelated Things," (http://www.correlated.org) did not do nearly as well. In fact, it bombed pretty hard. I assumed my book writing career was over.

But because of the strength of "Experimenting With Babies," I ended up getting two more book deals, each of which follow a similar format.

"Experiments for Newlyweds: 50 Amazing Science Projects You Can Perform With Your Spouse" (https://newlywed.science) came out earlier this month. I'm really proud of it. You should give it to any engaged couples you know.

And "Experimenting on Kids: 50 Amazing Science Projects You Can Perform on Children Ages 2-5" will hit bookstores in May 2020.

If you factor in advances, royalties, foreign rights, and other secondary deals, the amount I've made from my books is about what I make in a year as a programmer. So it's a nice chunk of change, but also a decent amount of work.

Advice to anyone else who's considering writing a book:

* You can do well with self-publishing if you're really good at marketing your own stuff, already have a strong platform, and don't need to depend on an advance. If any of those things aren't true, you're probably better trying to get a book deal with a traditional publisher.

* Writing the book is often not the hardest part. Selling the book is often the hardest part. If you just want to write, but you don't want to sell, this might not be the side project for you.

* If you have a literary agent, you typically pay 15 percent commission. And boy is it worth it.
It became Freescale (embedded PPC and ColdFire).
“Reimplementation from scratch” seems like a generous description of Buck. It was initially so like Blaze that I always assumed a xoogler exfiltrated at least the documentation of Blaze.
> I didn't say nothing can be done. I particularly don't care about "convincing" some guy on HN with poor reading comprehension.

You're the last person on HN who should tell other people about reading comprehension, at least judging from this thread.

> > You started by insinuating that most homeless are incurably schizophrenic.

> I didn't say all schizophrenics refuse help or that all homeless are schizophrenic.

That's a typical example of your reading comprehension, which is based on constantly shifting the goalpost in order to "save face." And that is the reason why you're so unconvincing. You did start by insinuating that most homeless are schizophrenic and that most schizophrenic refuse to get help. Both statements turned out to be plain false according to your own Internet research - which you obviously conducted because you're not very familiar with the topic, otherwise you would have mentioned your single data point, namely the paper you linked to, right from the start. You dynamically adapted your claims instead of doing the mature thing of simply admitting that you were wrong and less homeless people than you thought were mentally ill.

> There's a difference between "a solution" and "mitigation".

This kind of hairsplitting is an immature reaction when someone points out inconsistencies in a post.

The fact is that in the end you said almost nothing, but you managed to achieve that goal with erroneous reporting of statistics, peppered with several false dichotomies and a lot of insinuation. So some percentage of the (i) 20%-40% of homeless that are seriously mentally ill might (ii) be schizophrenic, and of those (iii) some other percentage might refuse treatment, and for (iv) those there may be no solution. Nobody has ever denied that in this thread, and what is even more stunning about this conversation is that you have provided 0 evidence for claims (ii)-(iv). Zero. No data, no statistics, not even an argument that would withold basic scrutiny. That's not a good track record, especially not for HN.

Now you're again shifting goalposts and there is a new insinuation, of course. Now suddenly you're coming up with general homelessness rates. I suppose the insinuation this time is that mitigation strategies of welfare states like Sweden, France, and Germany don't help against homelessness - which is, of course, another obvious nonsense that could not possibly be backed up by looking at homelessness rates alone. Again, you're using the wrong data to insinuate something while keeping the actual position as vague as possible. You could just as well have mentioned North Korea and Jordan as positive examples with alleged 0% homelessness. Or that Mexico has a much lower homelessness rate than the US. Or that a welfare state like Portugal has a much lower homelessness rate than the US, too. And so forth. Fact is, of course, that homelessness rates cannot be compared in this way and that public policies need to be evaluated much more carefully by looking at their concrete effects.

> Go educate yourself on the topic if you want to "prove me wrong" or something.

That's called "projection" in psychology.
Great episode! The part about memory centers sort of reminds me of GANs. Discriminate during day, generate during night.
CNN coverage: https://www.cnn.com/2019/04/17/world/trillion-trees-climate-...
The cloud provider problem is lock-in using proprietary services - but you don't have to do it that way either, pretty much everyone can run a virtual machine running whatever flavor of Linux you like...
Do you have a link to Alexander Vovin's presentation somewhere online? It sounds fascinating.
I see this as a very timely question. As ML has proliferated, so has the number of ways to construct machine learning pipelines. This means that from one project to another the tools/libraries change, the codebases start looking very different, and each project gets its own unique deployment and evaluation process.

In my experience, this causes several problems that hinder the expansion of ML within organizations

1. When a ML project inevitably changes hands, there is a strong desire from the new people working on it to tear things down and start over

2. It's hard to leverage and build upon success on previous projects e.g "my teammate built a model that worked really well for predict X, but I can use any of the that code to predict y"

3. Data science managers face challenges tracking progress and managing multiple data science projects simultaneously.

4. People and teams new to machine learning have a hard time charting single a path to success.

While coming up with a single way to build machine learning pipelines may never be possible, consolidation in the approaches would go a long way.

We've already seen that happen in the modeling algorithms part of the pipeline with libraries like scikit-learn. It doesn't matter what machine learning algorithm you use, it the code will be fit/transform/predict.

Personally, I've noticed this problem of multiple approaches and ad-hoc solutions to be most prevalent in the feature engineering step of the process. That is one of the reasons I work on an open source library called Featuretools (http://github.com/featuretools/featuretools/) that aims to use automation to create more reusable abstrations for feature engineering.

If you look at our demos (https://www.featuretools.com/demos), we cover over a dozen different use cases, but the code you use to construct features stays the same. This means it is easier to reuse previous work and reproduce results in development and production environments.

Ultimately, building successful machine learning pipelines is not about having an approach for the problem you are working on today, but something that generalizes across the all the problems you and your team will work on in the future.
This is the regular economy, which is not the topic of this thread. We are talking about wealthy people becoming rich without contributing. Renting has diminushing return for the society. Some things like automatic trading have none.
Really cool idea!
What tools do you use to write the books? And how long did it take for you to write one?
At that rate, Bezos could offset 5 years of the entire world's CO2 emissions?