skysight-inc/HackerNews huggingface dataset:
Suppose there was an organization, "Creation", that did mass surveillance on everyone on the planet. They don't disclose what they do with this data.

Let's say as a result of Creation's sharing of this data with local governments, crime is nonexistent.

Based on only what we've said is that a good or bad deal? Also, all of your examples were either purchased or have alternatives, so, yeah.
Ask HN: Storage for Raw HTML? ElasticSearch or Something Else?
Hi,

I am working on a project where we need to store the response of a web page and have later on do the data processing such as NLP tasks, topic modeling, sentiment analysis etc.

The average content/documents we are expecting is around 1-2 million per day, and soon that will be increasing to 10-20 million.

For this kind of data storage what do you suggest? Is ElasticSearch better for this kind of stuff or should we use HDFS/Ceph etc. for the storage?

Currently, we are using mongodb for persistent storage and ElasticSearch for indexing the data and serving that to our frontend but if some better option is available we can look into that as we are restructuring our most of the architecture and data pipeline.

Any kind of help/suggestion will be appreciated.

Thanks

P.S If someone can share some insights what kind of possible architecture does the brandwatch.com, mention.com or ahrefs.com got for their data pipeline would be really helpful.
Deconstricting the Administrative State
AWS has crossed 40% renewable energy, and is on track to hit 50% this year, as per https://aws.amazon.com/about-aws/sustainability/

Google is a bit harder to pin down, as per https://cloud.google.com/free/docs/what-makes-google-cloud-p...

Microsoft makes no comment about Azure specifically, see https://www.microsoft.com/about/csr/environment/renewable_en...
Stack overflow with more fluid discussions could the answer.
With a wiki it would be even more accessible and lively
I hope things go this way. I'm betting my future on it.
The structure of a problem specifies it; an unstructured problem is less well-specified, and that makes it intuitively more difficult to approach.

E.g. summation of a column of decimal numbers is a highly structured problem that's very easy to solve. Parsing the speech of yelling drunkards is not so structured, and much harder to solve.
Cronitor co-founder here. This is a great question! I would love to see others post some numbers here.

As Shane mentioned in the post 12.5% has been a consistent number for us as we have scaled up over the past couple years. That said, at this point I suspect we will see this percentage decrease a bit going forward. I'm basing this on intuiting that we're a bit over provisioned at the moment and won't have to scale our infrastructure linearly with user growth over the next year or two. Of course, that remains to be seen...

To add some more numbers to the conversation my full time job is CTO at a consumer facing tech company (Babylist.com) I just looked at our spending on IT/infrastructure for the last couple of months and it's around 1.5-2% of revenue.
Primary goal of Sciter is to be an embeddable UI engine.
You can embed it into C/C++, .NET, Java, Go, Rust, Python etc. applications. Each of them have mature File I/O primitives.

As of Sciter itself ... It does have file I/O to load/read resources and data persistence from script but these are just auxiliary components to its main function (UI).
This is how visitors from other planets look at us in regard to resource distribution on earth and space exploration.
The classical solution to the tragedy of the commons for rivalrous goods is propertization. It's the logic behind contemporary land rights, some very successful commercial fisheries management (north west Atlantic), and emissions auctions in the US (SOx and NOx implementation - but notably not the proposed carbon, which has other wierd factors bolted on)
I've wondered if one could build a device that outputs different combinations of distinct scents for training dogs. Like condition the dog to understand "sit" = "bacon smell" and "shake" = "cat". My theory was that dogs have more cognitive resources dedicated to processing olfactory information and so they could also be trained to understand and perform more complex tasks. Like, "bacon + cat + shoe smell" = "bark while running left"
I didn't know Mosh until now. Thank you so much for mentioning it! :-)
Emulators Written in JavaScript
Controlling a human-interactive program with another program is pounding a screw, but it does work for simple programs.
That seems to cover the people who derive the most value from the relatively nice place we inhabit, built on the aggregated effort and toil and misery of all those who have come before.
Notably, it's potentially easier to hold large companies to account compared to a bunch of small ones equivalent in total size.

For example, bad news about Uber discourages people using Uber all over the world. Bad news about some taxi's behavior only discourages using taxis (or a particular taxi provider) for that one city, usually.

In other words, large companies are more susceptible to reputational threat, which is is why, for example, they tend to be more careful about what their employees say online. If an employee of a random 10-man startup says something racist online, that's nowhere near as big of a deal as if an identified Googler says something really racist online.
Is Heroku a way to hack?
If I found this on iPhone should I be concerned?https://devcenter.heroku.com/
DeepXplore and new ideas for verifying ML systems
that was before people had lots of nukes
Or, to summarize, we need to take some of the money from the people who are currently extracting it from everyone else...

Money, and wealth, is not owned, it is collectively created. Without collective enforcement of monetary value, currency is useless. Without collective enforcment of property laws, property rights don't exist. The only non-collective value that exists is individual power, usually via violence. Unless you want to live in a society run by armed militias (see, for example, various failed states), you depend on collective enforcement. Taxes are not a matter of taking _back_, they are fees for services rendered for that collective enforcement. Your wealth would not exist without society. How much society charges to allow you to have that wealth is what we, as a society, need to decide.
The why and how of RPATH
"You're wrong by at least an order of magnitude, or perhaps several orders of magnitude."

Says the one making the wildly inaccurate claim. shrug And more recent articles reflect that.

http://bgr.com/2017/01/11/apple-airpods-sales-success/

https://9to5mac.com/2017/01/11/apples-airpods-and-wireless-b...

But, whatever.
G'MIC 2.0 released
Not only that, if you (only) have a large number of small investors, there will often be a lot of conflicting ideologies and interests, sometimes diametrically opposed. Hard for a company to have any long-term direction that way.
> In truth, films move them far more; they talk about The Matrix the way my friends once discussed Hemingway or Kerouac.

It seems the author is talking about how the times have changed, since the context of the sentence talks about how "these days" (things are different from how they were). This sentence is really about the kind of friends that a person would keep, if that person were the kind of person who would become a professor of literature at Princeton.

This may be colored by my own personal experiences interacting with faculty at various universities. While I loved studying and taking classes, there was something about the institutions themselves and the culture that drove me nutty. It was something that most faculty never really understood when they gave me well-intentioned advice.

> The intricate prose of Walden is a tough read in the age of tweets, so much so that several “plain English” translations are now marketed.

This caricature of modern society is one that you'd find perfectly preserved if you sent yourself 50, 100, or 150 years back in time.

The point about Thoreau is well-taken, but putting aside Ayn Rand for a moment (did he choose her to maximize controversy?), the popularity of writers is always fickle.
FSCQ is a really great example of a large system with proofs of correctness using extraction from Coq.

Another well known project is CompCert the certified C compiler [1]. Which has seen a fair amount of external testing and use in verification of GCC and Clang as a reference for checking invalid compiled semantics [2] (to say nothing of compiling programs).

1 http://compcert.inria.fr

2 https://blog.regehr.org/archives/1052
IIRC Seattle actually has an unusually lopsided ratio in terms of megacorp openings vs startup openings, I think it's substantially better in SF/SV, NYC, and Boston.
I've saved myself a number of headaches by taking the opposite approach and being conservative in what I accept: I ignore the quoting rules. Comma or newline in the middle of your field? Malformed. Double-quote by itself in the middle of your field? Fine by me. Most of the weird stuff comes from Excel anyway, and I have a reader for Excel files that tolerates those things just fine. My users are happy with it because I've removed a source of ambiguity for them. My users are also highly technical, so I can get away with this sort of thing.
HR is there to serve the organization, not the other way around.

If you are trying to hire a talented employee, and HR says no way, you better demand they justify it. It's your job to advocate for people on your team, even when they haven't come on board yet. Otherwise no one will.

It's far to easy for HR to auto-reject because of hard-wired criteria, they have no incentive to challenge their own rules for a good candidate.

I know a kid who once was a chip-runner at a casino. The way the job worked was he took cash/chips from poker players and went to the cage to get them chips/cash. He got paid minimum wage plus tips, and was on the hook if he got shorted. But the plus was he'd get $1 tip for each transaction, sometimes more.

But this kid optimized the whole job. He hustled like crazy, minimized cage trips by carrying 7 or 8 racks at a time, and even took optimal routes through the poker room. Watching him work was inspiring, the kid was so driven.

He was making a ton of money, far more than the other chip-runners, and someone ended up complaining. So the room decided chip-runners should pool all tips together. He quit in a rage the next day and chip-running service in the room became awful, the remaining chip-runners now saunter at their own pace, carry a single rack at a time and minimize their risk by slowly counting everything three times.

A couple years later the kid got busted for selling weed out of his house. He ended up serving a few years in prison. He's going to now fail any HR background check/screen. I'd still hire him in a fucking second, and expect him to rapidly work his way up from any entry level role.
It looks like a multibilion fine for Google is coming http://www.independent.co.uk/news/business/news/google-eu-fi...
Oh, well, Apache and PHP give me this out of the box. So even though there is not a great deal of custom error checking in my code as I said before, I always have enough information in Apache's logs to track down the problem. I guess with C and Go you have to roll your own.
Well, Chicago is the city that works. But Uber is a bigger gang than the mob. Should that scare us?

Really, things seem less corrupt now than they did with Daley so maybe there aren't big payoffs by taxis, just laziness from them and the police.
The best way to predict the future is to invent it ;)

Good luck!
Just stream of thought here:

Why don't we have a "government YC" (yeah, I don't mean DARPA-like, or the other mechanisms for finding funding from the government, which requires way too much navigation f bureaucratic paper-work and hoop-jumping) - an office of the government that can allow any individual to apply for seed funding for ANY idea in ANY industry that they can see through - make the process nimble and quick. Ensure that there is infrastructure/systems to allow for accurate tracking and reporting on the spend of monies given - think an EDD card that the startup entity gets and all expenses are tracked via the transactions on the card. Integrate it with banks and other private and public sector services with credits, like compute credits from amzn/goog etc... tax services, payroll etc...

And when you apply - it effectively sets-up an individual LLC in your name as the defacto entity to track your startups progress with giving you an EIN etc...

Heck, this should be a simple thing to architect if everyone in the business/startup world were truly supportive of, and willing to actually be, the "innovation" unicorn they all claim to be.
I've requested it, just for my own amusement. They told me no. Unless they didn't hire me based on something from that background check they didn't have to give it to you.
I read it as a practicality. It doesn't matter in the sense that; like it or not, it happened. Factually, the change took place and the effects are real. You can make a new change through market or regulations but currently things are now different.
Apple & Amazon and the others, probably not. They have their own markets and they're just doing well in them. Google might be a little different in that what they've done in some areas affects so many others, especially in small business communications and now into education. I'm just thinking about how kids are unaware that there is even another email provider out there because all they know is Gmail. To me, that's where things have tipped over into "kind of a problem."
The Googleverse is massive, and we don't even know how much data they've collected.
>gmail, maps, photos, chrome, android, translate, youtube, drive, waze

I think there are alternatives for all of those, many open source.
Real-time location tracking with near-zero battery impact
Thanks for elucidating the contents of the DeepMind paper and for hipping me to the Graph Convolutional Matrix Completion paper.
Wild and absurd as these doomsday predictions sound, land ownership and the acquisition patterns thereof is a matter of record and may thus be judged as actions rather than words.

What good explanations are there for billionaires acquiring such vast acreages as personal property rather than for commercial exploitation purposes? Insofar as such strategies do represent an attempt to commandeer the lifeboats, to what extent are others bound to respect their rights therein?
I fail to see how the planting patterns are fractal. A fractal pattern is one which repeats itself at different scales. I realize that the repetition does not need to be exact but I don't see how there is any at all in this situation.
In today's world of disposable gear, I think lasting over 11 years is pretty darn good.
“This new technique is really aimed at removing that little bit of oil in that water that needs to be removed before you can consider it treated.”

This is for the tail end of oil/water separation, after existing processes have done most of the work. Here's an overview of oil/water separation.[1] Note the line "Very small particles, such as those of 10 microns (micrometers) and less in diameter, do not rise according to Stokes’ Law (or hardly at all) because the random motion of the molecules of the water is sufficient to overcome the force of gravity and therefore they move in random directions." This is a way to apply magnetic forces to those tiny droplets.

The lines at the end of the article, about this being useful in oil fields, probably were added by some PR person.

[1] http://www.hydroflotech.com/oil-water-separator-theory-of-op...
How to Grow Human Organs in a Pig
Don't forget cable providers.
I wouldn't evaluate it on those terms. Whether the ends justify the means depends on the specifics of the means.

Not all bad means are equally bad. E.g. Taxes are a necessary evil that we accept, but we don't, as a general rule, let agents of the government just take whatever they want (hence the recent controversy over civil forfeiture).
They have a no-snapshots plan, which I am using.
Google is not a search company they are an advertising company. Their search product is a value add, it is the reason people go to google but it is also not what they sell. A business with a monopoly can raise the cost of their product to the price the market can bear. Google cannot raise the cost of their ads because advertisers will simply go to other ad platforms.

TLDR: Google is selling space on its website for advertisements. They do not have a monopoly on that. If they put their search behind a paywall Bing, Duckduckgo, etc will replace them.
At least in Chicago, public transit options exist.
"Whither Software Engineering?", discussed in the OP, sounds like an interesting historical piece about how long it took for mathematics to be integrated into engineering work. The author is John Allen, who wrote Anatomy of Lisp.

Can anyone find that paper online? https://news.ycombinator.com/item?id=962733 points to a blog post about it from 2009, but none of the links seem to work.
Why is that the only way to get a 10s failover? The TTL on the DNS records for Amazon RDS, at least for Aurora, is just 5s.
> All post polar shift predictions are based on theories from Gordon-Michael Scallion, Edgar Cayce and others, and should not be construed as fact.

https://en.wikipedia.org/wiki/Edgar_Cayce

https://en.wikipedia.org/wiki/Earth_Changes
>> consumers might actually be confused by the name

Only problem is those consumers are in a different country - one where Trader Joe's does not operate. I would bet Pirate Joe's would have happily closed down if Trader Joe's opened up a store in Vancouver, since that's all the Pirate Joe guy wanted in the first place.
UBI must first overcome the unwillingness of citizens to trust individuals to be responsible for choosing how to best spend public funds to aid themselves; and to accept that they may often disagree.
Yeah, most people I know do use their 'assistant' to some degree, but it's not even close to crucial for any of them. That said, Siri is a running joke among many of us, so if/when assistants become more important, Apple might have a problem if they don't make some big improvements.

Anecdotally speaking, anyways.
csv parsing on PicoLisp:
https://bitbucket.org/mihailp/tankfeeder/src/5a91a025d78eacf...
That's a terrible policy. You need to optimize for getting the best employees, not minimizing discrimination complaints. How often is a felon going to sue because the company once made an exception for a specific felon because of specific outside factors, and how can that cost you any suit?

"Your honor, my client, the recently released child rapist with an iffy work record, is suing because BigCorp says they normally don't hire felons, but a few years ago BigCorp made an exception for a felon who sold marijuana, got his college degree in prison, then after he was released worked at non-profits helping the poor while getting an advanced degree in computer technology and writing several highly regarded academic papers on improving user interfaces for the disabled. That's not fair!"

The specific facts matter, and HR should be fired if they have blanket polices that ignore them. HR needs to serve the business, not the other way round.
Who asks for a prospective employee's religion in Germany? Apart from church affiliated positions this is illegal here as well.
> More progress has been made against conventional multiplicative group Diffie-Hellman than has against curves.

How is "none" more than "none"?
All employees should be kept as happy as possible anyway, shouldn't they? Why would anyone want unhappy employees?
KLF - yes mate :)
This Scallion character is described as a "spiritual visionary." Which is fine but I'm not sure I want my metaphysics guru drawing maps of coastline shifts in a Forbes article.

Also, billionaire land purchases probably have more to do with the human desire to dominate than with a doomsday scenario. E.g. If financial markets and land rights go down the tubes, I'm not sure what 200,000 acres in Montana is going to do.
Awesome! Where can I get this keyboard?
Yeah let me guess this dude was white. Unbelievable, the world must be so different in that skin.
For anyone unfamiliar with the connection, Hershey's milk chocolate is made with a process that produces butyric acid:

https://en.wikipedia.org/wiki/Hershey_bar
"We hired a guy once who, a few weeks after starting, had to take three weeks off to serve jail time for his 3rd DUI'

You flunked your own background check.
Cool! So, would it be correct to say that this is in essence generates a disambiguation model for a language with a lot of homonyms due to having relatively few sounds (but also with some variation in morpheme boundaries, e.g. "an-i" vs "ani")?
Well, I wasn't clear above, but sadly nature reclaimed my uncle, he's now ashes. I thank you very very much for asking and wish you the best of luck. It's a very weird and sensitive subject.

About your comment:

1) indeed networking is massively important. You talk to someone out of the blue, it's hard; you say you've been directed by <acquaintance> and the discussion is now relaxed and open.

2) indeed, you have to forget about shame. Just insist politely. It's hard at first but with time it goes.

3) I feared becoming too emotional and trusting things that would reinforce my belief. So I stuck on the usual venues: pubmed, nih, nci, etc etc. And even then, every topic or paper, I would check citation, publication date, previous or more recent validation/invalidation. Research lacks counter studies and sample so one academic paper is not enough.

After this screening, I then pinged various people about these findings and what they thought about it, to again, not pursue my own dreams. The issue is, 10 doctors ~= 10 point of views.

3) clinical studies were barely starting in my uncle case (one doctor told me about israel), he was against this because he thought he would be a guinea pig suffering for their profit. Which isn't false but we could have delayed death a bit. We all felt it was way too quick.

ps: all of this just gives hint about the complexity of the real world. So many pieces working at the same time, politics, economy, research, companies, doctors, networks .. nothing fits perfectly, and if you have to turns the gears on your own it takes deep faith to never stop. I wish I did more, but since it's the first time, I couldn't know if I was crazy and foolish or if I should harass doctors a bit more. My opinion is now clear: never stop; be sensitive, but never stop. (that's also the quality of a leader.. which is as old as important in human groups)
I don't think it was though. A medallion would make you around 50k in profit a year. Medallion's were a great investment, but the greatnest of them depended 100% on the government regulations staying relatively the same.
That is exactly the question you should be asking, thank you.

Answer is they provide great rates for passengers - see all of the anecdotal evidence here - and great pay for drivers.

It literally is "too good to be true."

It's just an app--with a $20B valuation and about as much in free cash.
For a country that doesn't believe (in big enough numbers) about providing universal health care to all, a country that has some of the lowest welfare provisions in the developed world, I don't see how it would ever come about to go for UBI.

Are all these discussions just academic or is it a real posibility? As I just don't see it every happening and certainly not in the US - ever
Interesting project! Does anyone know what iOS is using for it Japanese transliteration predictions? Mine worked well for a long time, but in the past 3 months it's gone haywire for common kanji suggestions. The other day it had "機能" as the first/only suggestion for "きのう," and I had to dig down into the menu to arrive at the intended, "昨日." Had a lot of similar experiences recently.
Shoe HN:What's the point of being an entrepreneur