skysight-inc/HackerNews huggingface dataset:
Previous discussion on HN: https://news.ycombinator.com/item?id=5378008

from kevinpet: "The Monsanto patents are for the method of using a particular herbicide in combination with the modified plant seeds....
Has Monsanto ever sued anyone who was not trying to use the combination of the GMO crops and the herbicide they are tolerant to?"

from tptacek: "the behavior Monsanto pursues is planting unlicensed Roundup-Ready seeds and then spraying them with glyphosphate-based ("Roundup") herbicides. You can spray Roundup without a patent license. You can probably plant RR crops without a license. But if you do both (commercially, at least), Monsanto sues.
The point of RR seeds is that they resist Roundup, which is a broad-spectrum herbicide that will kill non-RR crops. In the commercial suits, it becomes tricky to argue that you planted RR crops unwittingly when you are later shown to have sprayed them with an herbicide that would have certainly ruined your harvest but for inbred RR resistance."

There are a lot of actions that, if we described the component behaviors, would sound legitimate -- but taken in aggregate are clearly wrong. It's legal to be in the forest and it's legal to discharge a firearm, but it becomes illegal if you knowingly target a person in the forest with that firearm. It's legal to change your clothes and wear a fake beard, and normally legal to go into a grocery store, but it becomes illegal if you disguise yourself to circumvent a store ban. And it's legal to have RR seeds in your field and it's legal to spray roundup, but it becomes illegal if you intentionally try to circumvent Monsanto's patents by using roundup to isolate their seeds and then intentionally replant those so that you'll be able to use roundup on your crops without destroying your crops and without paying the standard fee.
http://www.dc-energy.com/careers/who-we-are/people-profiles/...

It boggles my mind to think they're writing their systems in PHP.
Do you have evidence that teaching a child to read at 7 years of age is a significant detriment to their overall learning? Do you feel like the parent post that not teaching them to read at 4 years old is 'child abuse?'
some references to start you off re the benefits of the Finnish system where formal education starts at age 7:
http://www.npr.org/2014/03/08/287255411/what-the-u-s-can-lea...

http://en.wikipedia.org/wiki/Education_in_Finland
79% of all bets in Scotland on Independence have been for a ‘yes’ vote
> for most ambitious folks.

He seemed to be targeting a specific group that's considered "ambitious." One could argue that everyone is ambitious but I'm sure his point was that those people who aren't satisfied with low-wage jobs and want to do something they see as important, would consider 80K "not enough."
This is why HTML CSS and JavaScript standards are constantly evolving, to solve real world problems.
I think you're wrong on a bunch of points.

The reason developing countries aren't "dominating the world" is because of a bunch of things, and the life of a kid from a developing country is very far from the life of an unschooled kid in the west.

Unschooled kids (and I know a lot of them) in the west are usually fairly privileges. If not economically (which definitely isn't always the case) then they are privileged to have intelligent and thoughtful parents who have the time, energy and interest to try to nurture a love of learning in them.

I also think you're wrong in regards to creativity. Creativity isn't domain specific, but being able to apply your creativity does take a lot of domain knowledge. You can have endless domain knowledge, and not an original thought in your head.

There are plenty of opportunities for unschooled kids to enter the mainstream education system, if they need certification of some kind to work in their field of interest. I know a good number of unschooled kids who became biologists, computer scientists, etc. by going to college/university when it became relevant.
You're being unnecessarily argumentative. The web is filled with badly-encoded/re-encoded/mixed-encoded text. One of the worst offenders is Microsoft Outlook, which sends emails using local 8-bit windows codepage by default, instead of utf-8. Pass that through several mail gateways, display those messages on some php forum, and you get yourself a bad mess that cannot be decoded successfully by any python codec. chardet is useless in that case - there is no 'valid' encoding per se.

This library is taking the only possible approach, which is to segment the text and try to convert each segment into its most probable unicode representation. It seems to cover a larger number of encoding mixups compared to other libraries, that's great!

Comment to rspeer: in fix_text_segment(), I would limit the number of recursive passes on the text to 5-10. Right now it's using 'while True', which might take a very long time to converge on corrupted/binary data.
I think your assumption about the tax increase is way off. The deduction is for the _interest_ paid on a mortgage. Let's assume you buy a $400,000 home with 20% down and a 30-year loan with a 4.5% interest rate. Because it's a deduction -- not a tax credit -- your savings in the first year is only about $6600, and that savings decreases every year after that as interest becomes a decreasing portion of each payment.

I don't think people are going to avoid homeownership in droves because of $6600 per year -- especially people who are in the financial position to own a home. It would, however, remove some of the perverse incentives that make people buy homes who might otherwise be happy to rent one.
When you give chardet text in a 1-byte encoding, it sometimes ends up >99% confident that it's in ISO-8859-2.

Empirically, it's not in ISO-8859-2.

I think the problem here is that chardet is built on the assumption that "encoding detection is language detection" (from its docs). This assumption is necessary, and basically correct, when distinguishing Japanese encodings from Chinese encodings. It's even pretty much taken as a given that you can't have Japanese and Chinese text in the same document without contortions that most developers are unwilling to go through.

But European languages and encodings are much more intermixed than that. One document may contain multiple European languages, and these languages may be written outside of their traditional encoding.

I wouldn't know how to fix the European languages without damaging chardet's clear success at distinguishing East Asian encodings.
Why not start a company that hires all these outstanding engineers who, for whatever reason, don't have degrees? It seems like a fantastic way to jump ahead of your illogically-prejudiced competitors.
> And conversely, some of the most "happiest" place on earth, according to their citizens own estimations in polls, are places with tragic poverty and harsh conditions.

That's very far from my recollection of those polls where Scandinavian countries frequently come out on top. Common denominators of those countries include low wealth inequality, an extensive social safety net, and a grounded attitude towards the future. With all this, there's still plenty of struggle and passion and meaning and hurt in the microcosm of daily existence. Humans don't seem designed to be suspended in a state of constant happiness.
The presentation of the page ("copy pasta z", the fullscreen gif background, the popup in the middle, the banner in the top left) all strike me as being dodgy/malicious and from a site that I'd immediately want to close and never visit again.

I'm not sure why you see the need to be so 'internet' with it all when you could do a fully functional equivalent that would be easier on your servers too. Just make it clean and usable and it'll be far better.
> In left-right politics, left-wing politics are political positions or activities that accept or support social equality, often in opposition to social hierarchy and social inequality

Source: http://en.wikipedia.org/wiki/Left-wing_politics

vs.

> Right-wing politics are political positions or activities that view some forms of social hierarchy or social inequality as either inevitable, natural, normal, or desirable, typically justifying this position on the basis of natural law or tradition.

Source: http://en.wikipedia.org/wiki/Right-wing_politics

Which one seems to be more favorable to enacting your statement:

> exterminates and enslaves minorities

The equality faction or the hierarchy faction?

Here is an analogy:

You can claim yourself to be a pacifist (left wing) all you want. But if your actions are aggressive and cause war (right wing), then you aren't a pacifist, even if you claim to be one.
I am implementing imperial system on our currency.

12 cents = a dime.
12 dimes = a dollar,
12 dollars = a thousand dollar.
12 thousand dollars = a million dollar.
12 million dollars = a billion dollar.

can you tell me how much is a billion dollar in a dollar unit now?
You have to do some instruction around the edges, but largely the idea is that you offer different avenues for exploration and then they choose which way to go. Homeschooling generally has a fixed curriculum, often as restrictive as traditional schools - so that is the distinction.
There are liberals in the government? The last one was killed immediately, when he tried to stop the upcoming Vietnam theatre
So, actually, it seemed to make sense at first to limit it to, say, 2 or 3 passes. But then I read about Spotify's username exploit [1]. That made it pretty clear to me that any Unicode-transforming function should be idempotent whenever possible, so that you never end up with inconsistent answers about whether strings are equivalent.

I have also seen text that was encoded six times in UTF-8 (and decoded five times in Windows-1252). Although ftfy had to leave it as is; it didn't successfully decode because it was truncated.

[1] http://labs.spotify.com/2013/06/18/creative-usernames/
> neither side -- left or right -- in America is that close to the ideas of National Socialism

...

> False equivalence is a logical fallacy which describes a situation where there is a logical and apparent equivalence, but when in fact there is none. It would be the antonym of the mathematical concept of material equivalence.

...

> A common way for this fallacy to be perpetuated is one shared trait between two subjects is assumed to show equivalence, especially in order of magnitude

Source: http://en.wikipedia.org/wiki/False_equivalence

The party that attracts militias is probably not the same as the one that attracts pacifists.
Vuze, a very commonly used torrent client has to take encoded torrent titles and decode them. Sometimes the decoding fails, because of corruption or missing encoding type.

Their approach is then to attempt each type of encoding they got, ignoring errors, and show the result to the user so the user can decide (detect) which decoding worked and which didn't.

Can ftfy replace this functionality? Is it doing the very encoding detection which is currently done by humans?
... with too many students, more each year, and ridiculous amounts of homework assigned, so much that there's no way a teacher can even look at it all (on your own time at night) much less evaluate it, ...
A Continuous Delivery Setup
Hey Nam! Really appreciate the comment.

We were just being silly and hacking away at it adding random things that made us lol hahaha.

You're right though. Moving forward minimalist is what we hope to accomplish.

Thanks for the feedback ^_^
Oh awesome! thanks for the links!
I made the switch a year and a half ago and I don't regret it at all. I'd been programming professionally for > 12 years and had gotten to the point where I didn't really feel like I was encountering enough challenging problems to keep my juices flowing.

Now I deal with a lot of new (to me) problems every day and get to spend my time trying to help my developers be the best they can be and kick ass.

The hardest bit for me was making the mental shift from trying to do things myself, to coaching and encourage others to solve problems and learning to be okay with people not doing it the way I would have.

No regrets so far, but I also do some consulting on the side, and occasionally jump into understaffed projects to assist as needed.
Well they pretty much got Crimea, don't they ?
>In left-right politics, left-wing politics are political positions or activities that accept or support social equality, often in opposition to social hierarchy and social inequality

Yes, and these are the positions Mao, Stalin, and Pol Pot took. They killed people using the justification that the people they killed were standing in the way of these ideals.

The next demagogue to rack of a massive body could could come from either the left or the right.
How much should they get paid?
Who are these great people and where do I find them?
How much oversight should they be given without restricting them?
How much should I trust the teacher vs a few vocal parents? Vs the silent majority?

The reason why your model wouldn't work is that nothing concrete was stated.
> If capitalism requires you to hire PhD's who specialize in detecting complex financial bets

It doesn't. That's the point. That's why the power company is selling these financial instruments, not buying them: so that somebody else does the hard thinking.

The price is the signal. You don't need a PhD to determine the message implied by the fact that energy traders are willing to give good odds (i.e. pay more than expected) to bet that your grid will fail in a certain locale on a certain date.

> the power company serve human beings

The power company not only dropped the ball but dropped the ball after someone told it in no uncertain terms that it would drop the ball in the time and place that it eventually did.

If they clean up their act they stand to make good money by selling similar securities.

> the financial traders represent the worst of everything about capitalism

Correct incentives are the best part of capitalism.
Aren't submissions folded to the last one, if the URL is the same?

I could swear I have seen that even when the prior submission was a few years old.

Does the algorithm to decide that take comment activity into account?
1) Lack of good Unit tests.
2) Stupid colleagues.
3) Bad food.
Definitions of mathematics
Lots of truth to that. There's a reason many atheists have started holding their own "church" services, on Sundays.

http://www.bbc.com/news/magazine-21319945
Africa: is it possible he's walking miles to school because either his parents can't read or they don't have time to teach him?

There is nothing "elitist" about choosing what's best for your children. I went through public school, and looking back, it was a sorry affair. Math teachers were terrible. Science teacher had no clue. English teacher assigned comic books to make sure the slow kids didn't feel bad. Despite all these obstacles, I had a deep love for reading and spent about 2 hours a day after school at the library, going through everything that caught my fancy (my version of unschooling). However, the one redeeming value of my public school experience was the immense amount of open play given to me. Formal education sucked, but between recess and after school explorations with my friends, I learned soft skills that I still used today. Unfortunately, today's public school has eliminated recess; furthermore, parents are so paranoid that they no longer allow their children to be outside after school. There is a reason why the Finnish system focuses on open play rather than homework in the first 9 years of child hood. There is nothing elitist about demanding what's best for your children.
Charter and voucher schools are not free market schools, since the government is paying the bill.
There is at least a solution to that one, stop assigning homework. It has no positive effects before middle school that anyone has been able to find.

Are we wasting our children's time by giving them more homework?

Ozkan Erena, Daniel J. Henderson

Following an identification strategy that allows us to largely eliminate unobserved student and teacher traits, we examine the effect of homework on math, science, English and history test scores for eighth grade students in the United States. Noting that failure to control for these effects yields selection biases on the estimated effect of homework, we find that math homework has a large and statistically meaningful effect on math test scores throughout our sample. However, additional homework in science, English and history are shown to have little to no impact on their respective test scores.
Yes but people were dying from hunger as a direct result of the policies that Mao implemented http://en.wikipedia.org/wiki/Great_Chinese_Famine He is largely responsible for these deaths even if he didn't personally exterminate them.
Correct me if I'm wrong, but I don't think most people have an LTE phone.
There have been some pretty large scale invasions in the last 10-15 years. Those countries that were anti them faced a pretty dark few years in terms of international relations. At least that's how the deadlines were interpreted by me. http://en.m.wikipedia.org/wiki/Opposition_to_the_Iraq_War
The Pyra at GamesCom
Whenever there are ambitious plans for a poor country, people think of all kinds of reasons not to - the environment! the culture! the local economy! people's family homes! But they forget "these people are already screwed!". This canal is a risk with a bigger upside than doing nothing.
Ok, so by this definition, Stalin was certainly not left-wing, since he exterminated and enslaved minorities, which you appear to agree is not supportive of social equality. Yet Stalin is used in the same Wikipedia article as an example of left-wing activities.

Also, if pacifism is left-wing, then the Obama administration is certainly not left-wing or anything close to it, since he has repeatedly authorized military action. I suppose you could finesse this point by saying that neither major party in the US is left-wing by your criterion. But trying to apply this more generally, for example to claim that Stalin, Mao, Pol Pot, etc., were not "really" left-wing, looks a lot like a "no true Scotsman" argument.

In fact, by the pacifism criterion, I'm not aware of any government or political movement that has ever been left-wing; the closest thing to it would be some religious orders, like the Amish, who really do try to be pacifists. (Some people would put the Quakers in this category, but the Quakers have always flavored their surface pacifism with a strong pragmatic streak; they sold gunpowder to Washington's army during the American Revolution, for example.) But this makes "left-wing" a basically useless term, not to mention that you would have to explain to an awful lot of people that they have been using it wrong for a long time, since most people apply the term to Stalin, as I noted above, as well as Mao, Pol Pot, etc.

Finally, if left-wing means supportive of social equality for everyone, then once again, there are hardly any true left-wingers and never have been. Take trade unions, for example: yes, they're very supportive of social equality for their members, but are they equally supportive of social equality for management? Not to mention social equality for people who are not members but would like to learn one of the trades that are unionized? In my experience, unions do a very poor job with things like apprenticeship programs, because training new members means less job security for existing members.

Similarly, central planning in economics, which is supposed to promote more economic equality, ends up giving special privileges to the planners and those who are closely connected with them (like large financial institutions); witness the huge bailouts of investment banks after the crash of 2008, while ordinary people got foreclosed on. The closest thing to true left-wingers by this criterion are left-libertarians or anarchists (which, to be fair, the Wikipedia article does mention).
No, but being on equal footing is not enough.
This actually seems astoundingly misguided and dangerous.

One of the unique, defining qualities of humanity is an ability to build on the knowledge and experiences of generations before us. Why throw all of the knowledge of the past 5,000 years of human civilization out?

When will they learn math, or science, or history? How can you become a lawyer, doctor, or engineer without learning these subjects? Can they really be postponed until college?
It works quite well in other countries (for instance, Finland). We have a few problems in the US:

-- no profit in it (as observed above)
-- we don't really respect teachers or education in the US
-- school boards and local politicians want control over curricula and political agendas. Can you imagine some bright amazing young teacher teaching unadulterated evolution in some counties in Texas?

Kids are seen as a cost, not an asset or an investment, and we like to cut costs. And I do think there are broad swathes of the political establishment that are really invested in keeping America dumb.
Human consumption hinges on the fact that the humans have income to buy stuff to consume. Are you saying we shall tax the company profits in order to pay people basic income to provide means needed to consume stuff to create these required profits. And that the profits are staying in the tax jurisdiction as well as not getting tax breaks the owners lobbied for using previous profits.
Interview with Adam Ernst on functional programming in Facebook's iOS app
So cool. Basically they built React for iOS. I had wondered if there was anyone out there trying that on a large scale.
As someone said the other day, why own and run down your own fleet when you can borrow someone else's Sounds like Flytenow is aiming to be the first airline that doesn't have any planes.
Can't be too careful, they are the only country that ever managed to burn down the White House...
More plausible since it directly connects to the problem at hand. But when you replace humans with algorithms how do you determine what to tax and how much?
Mom jailed over truancy died from heart failure
I apologise for my impulsive reply. I should have better discipline (usually I don't reply at all because I know I'm pretty dickish)

This is how the site looks to me: http://s13.postimg.org/uduowd0if/sugar.png

In my opinion this is pretty terrible. We all know the importance of first impressions (case in point, people reading my initial response).
That sounds like a use case where the manual intervention is pretty important. That's encoding detection, where sometimes you have unmarked bytes. I would not, in fact, recommend ftfy there, and the README warns you against using it in that case.

There's some auto-detection you can do -- for example, you can distinguish UTF-8 from byte-order-marked UTF-16 with 100% accuracy, by design. You could also try chardet if you're okay with some amount of errors. Maybe show the detected encoding first.

Cases where ftfy is useful:

* Web scraping -- sometimes you get data that decodes in the encoding it claims to be in, but isn't quite right

* Handling data that has, at one point, been imported and exported in Microsoft Office, without every user consistently picking exactly the right format from like 20 inaccurately-named options

* Handling data that was stored by half-assed services written in, say, PHP, and not tested outside of ASCII

* Reading CESU-8 (the non-standard encoding that Java and MySQL call "UTF-8", for backward compatibility) in Python without breaking it even more. (This isn't automatic.)

* Handling data that's combined from multiple sources, in mixed encodings

* All the other situations in which mojibake arises in mostly-readable text, which there seem to be no end of.
Corporations can move out of the jurisdiction with corp tax much easier than humans. And judging by Apple etc, they are pretty good at getting tax brakes, buy legislation.