skysight-inc/HackerNews huggingface dataset:
Ask HN: Is there a library or a package that converts a web page to JSON?
I am living in a country where open data is a very alien concept for the public and government. Except for a handful of civic hackers who find it very difficult to create apps or services due to lack of open APIs or datasets. Data is available on government websites but its not technically open (as in a RESTful API). I am thinking about writing a library that can crawl all over a page and convert the HTML into JSON that can be consumed over REST. Is there something like that out there already? What are some of the things I need to consider before I start writing this?
OP wants a free service.
Picasa with public visibility for albums comes to mind.
It seems to be encouraged:

"We have per-level leaderboards which give people a meaningful number to golf on, and additionally, we have discretion to (automatically or after human review) award additional badges for going above-and-beyond on a level. We’ll also do that outside of levels, too — I can imagine issuing badges for people who, e.g., write OSS clients for our API, submit documentation patches, responsibly report security issues, etc." - http://www.kalzumeus.com/2015/08/20/designing-and-building-s...
Using S3 for image hosting would be overkill + I already plan to use my free tier of S3 for some project next year.
The example of lost US jobs, given in the article, is actually about off-shoring. Not related to H1-B at all. Lets get that under control first.

Secondly, the process I went through back in the 70s, the position should be advertised for some length of time and if there is a qualified US applicant that person should be hired instead.
why not create your own CDN if you have some hosting space or create one on AWS?
The risk of AI comes from the control problem. We have a vague idea on how to build AIs. The work is just to optimize the algorithms extensively so they can run in real time. But we have no idea how to control those AIs. We can push a "reward button" every time it does something we like. But we can't prevent it from killing the programmer and stealing the button for itself.

It doesn't matter at all who makes AI first. Without the control problem solved, no one will be able to use it for evil ends, even if they want to. But it's still incredibly dangerous if it gets loose in some uncontrolled setting.

So this project is the exact opposite of what you would want to do if you wanted to minimize AI risk. This is rushing towards the risk as fast as possible.
Δωρεάν+Οδηγός+SEO+Google+Πρώτη+Σελίδα+-+SEO+|+WEB+DESIGN
- Do you know you are an ad? south park ;)

I doubt it though. People got quickly desensitized towards advertising. Propaganda will follow.
The Story of a Non-Muslim Eyewitness in Mecca (Part 1)
Outsourcing website testing
Oops. Thanks for pointing out that typo - corrected!
Pandora One Apk V6.7 MOD Latest Free Download
Graph.ql – Better GraphQL Schemes
"Communications on eBay's platform have long used pseudonyms and automatic salutations, so AirBnB could easily implement that approach."

Well, not quite. Messaging on eBay is anonymized (but that can be defeated), but if a purchase is made, the Paypal transaction has the buyer's full name and email.

eBay is so zealous about forcing people to Paypal, you can't even mention the word "cash" in a listing or under Shipping And Payments.
We have no problems with OSS API clients. (I've mentioned previously, but given that it is 2015, I'd expect most developers given the task "Do something with a novel API" to immediately start looking for the OSS client rather than sending in raw HTTP requests.) Our community has several of them ready to go: https://discuss.starfighters.io/t/helpful-external-tools/136

The more the merrier!

We'd prefer if folks exercised discretion with regards to posting solutions; many of our players find the existence of them anywhere demotivating.
As the founder of HashiCorp, I feel qualified to answer this.

The adjacent comment about Atlas is correct. Atlas itself is admittingly poorly marketed (something we're working on right this very moment) right now. The best way to describe how we make money is: we target the enterprise user and give them the features, integrations, and support they need in their regulated, legacy-encumbered, etc. environments. Atlas in particular is focused on giving enterprises a flexible application delivery pipeline that is opinionated in a certain way, but flexible enough to support their diverse environments (all at once: containers, VMs, physical machines. Windows, Mac, Linux. Etc.)

Due to our adoption we have the rare benefit of primarily letting open source adoption happen organically, and having large companies approach us as they get more serious about our software.

Vault, in particular, is a good example of this. Vault suffers from an issue where most impressive people who use Vault can't actually publicly say they use Vault. So we have a hard time talking about how widely adopted it is. The best I can give you is from what I said at our HashiConf keynote: "If you traded stocks, used a credit card, or did anything involving a bank, then you've interacted with Vault-secured data." As specific as I can be as to how widely spread Vault is right now.

Anyways, a digression from the original question: enterprises have interesting needs they're willing to pay for. We address those needs in enterprise products (Atlas, others that aren't public yet) and support. We don't have any SaaS-like "enter your credit card and pay us" (other than the Vagrant VMware plugin which predates all of this), and instead do deals with larger companies in a way that most of HN would likely perceive as old school in some sense. :)

Note when I say "enterprise" I'm not trying to exclude anyone. If you're a relatively small company (I didn't click your name to find out), then still feel free to email us any of your concerns and we should be able to either help you ourselves or route you in the right direction.
gzip is fast, gzip -1 is even faster, gzip has low memory requirements, gzip is widely adopted. Those are the reasons of gzip is still being used, and why gzip has future. I.e. the gzip "ecosystem" is rich and useful, despite not being the best compressor in terms of compressed size.

P.S. There are gzip-compatible implementations with tiny per-connection encoding memory requirements (< 1KB).
I currently don't have hosting space and I plan to use free AWS for a project (later on).
I support a basic income. Honestly, I think it is inevitable, or a lot of people will starve as we move past the need for a lot of unskilled labor. There simply won't be work for everyone...if we adhere to the old notion that everyone has to earn the basic necessities of survival, well, the results will be catastrophic. It's unfortunate that we'll have to wait until everything else has been tried before settling on the one thing that could actually positively change outcomes for huge swaths of people. There's already a lost generation of people who will never escape their school debt.
Back in the day Yahoo Pipes would have been something I'd have recommended as being a tool in this space. Here's a blog post that lists some alternatives to Yahoo Pipes and some of them look interesting for your use case, though I don't know that any of them is exactly what you're looking for.

http://www.makeuseof.com/tag/12-best-yahoo-pipes-alternative...

Also, CloudScrape:

http://cloudscrape.com/
I don't want to make a big comparison right now, so just addressing your specific concern about Vault:

Unsealing doesn't cause an outage since you run them in HA-mode (side by side, leader elected).

As you said, if you gain root, both in Keywhiz and Vault, you can extract secrets at will. I can't comment to Keywhiz's capability (I'm sure some exists), but Vault will send audit logs to multiple backends so at least you can see it happening. The root token in theory can disable audit backends, but that action itself is also audited before it is disabled. That scenario in particular (root token compromise) is the worst case scenario. Ideally you throw that root token away into a N-key area as soon as possible. This feature is coming to Vault soon (expanding the shamir to arbitrary paths/secrets in Vault to support N-key operations).

Vault itself is used in places with very very high requirements of uptime (one user in particular can measure their overall downtime in seconds in the last quarter century). So we've had to go through some rigorous due diligence in such places to make sure it can withstand that. So, I assure you operationally that it won't cause an outage if it is properly configured in HA-mode. :)
Zero compression is very often the correct choice these days.
Schlüsseldienst+24+|+Stuttgart+|+Tag+und+Nacht
I knew I'd read it somewhere, but just wanted to check anyway.

Not finished yet but: https://github.com/vertis/stockfighter-trading-api-ruby

Edit: And I can't post to that thread since I don't have an account yet (client based on the documentation) ;)
Looks good. Do they have an API or are there 3rd party tools available ?
You won't find direct ML work from MIRI because:

1. AI / ML is not AGI.

2. Deep learning may be a tool used by an AGI, but is not itself capable of becoming an AGI.

3. MIRI believes it would be irresponsible to build, or make a serious effort at building, an AGI before the problem of friendliness / value alignment is solved.

So are they philosophers? Of a sort, but at least Eliezer is one who can do heavy math and coding that most engineers can't. I wouldn't have an issue calling him a polymath.

There are lots of individuals who disagree to various extents on point 3. Pretty much all of them are harmless, which is why MIRI isn't harping about irresponsible people. But the harmless ones can still do good work on weak AI. You should look up people who were on the old shock level 4 mailing list. Have a look into Ben Goertzel's work (some on weak AI, some on AGI frameworks) and the work of others around OpenCog for an instance of someone disagreeing with 3 who nevertheless has context to do so. Also be sure to look up their thoughts if they have any on deep learning.
Thanks for the feedback. Will keep in mind next time
How about Microsoft offering basic utility software with their OS releases, e.g. curl/ssh/awk/grep/sed/tar etc.
Hey Guys - I'm a big fan of your work, both with, and before Starfighter. Just a few initial comments:

>> If your use of the system decreases Fun for other players, it is a violation of the Zeroth Rule. If it doesn't, we have no strong objection to it.

>> Illegal market manipulation wait, we're not the SEC -- that's very clearly Fun.

>> [Fun] means what we say it means, and our decisions on it are final.

What constitutes 'fun' seems arbitrary. It might frustrating and very un-Fun to be told "Patrick, Thomas, and Erin say no" because the rules aren't well-defined. For example, I don't think that illegal market manipulation is fun - it's just deleterious - and part of the reason it's illegal in the first place. To what extent are you planning on taking user feedback on what 'fun' is? It's not clear to me whether or not it's 'fun' to craftily infer your internal network topology, for instance. Your game has a global audience, so you have to assume that linguistic/cultural/whatever norms aren't shared by all your players.

Also, the contrast on the code samples is way too low. Please make the text stand out more? I can barely read it.
How big of an explosion do you think a drone small enough to fly through your ear canal is really going to be? How does it even reach its target with such a minuscule mass? The target exhaling would blow it away (and likely destroy it)
The danger of nanotechnology is that it can be built better than biological life. It could outcompete it in it's own environment, or at least different environments.

Solar panels can collect more energy than photosynthesis. Planes can fly faster than any bird. Guns are far more effective than any animal's weapons. Steam engines can run more efficiently than biological digestion. And we can get power from fuel sources biology doesn't touch, like fossil fuels or nuclear.

We conquered the macro world before we even invented electricity. Now we are just starting to conquer the micro world.

But AI is far more dangerous. It would take many many decades - perhaps centuries - of work to advance to that level. It's probably possible to build grey goo, it's just not easy or near. However AI could be much closer given the rapid rate of progress.

If you make an unfriendly AI, you can't just shut it off. It could spread it's source code through the internet. And it won't tell you that it's dangerous. It will pretend to be benevolent until it no longer needs you.
Please send me an email. I will give you a invite code to the beta, which will let you create an account to put your client up on the forum thread. (HNers who did not already spend a metric truckload of time writing clients: give us until tomorrow afternoon, please.)
My experience hiring and working with Haskell developers - and those trying to get up to speed - is that one cannot just "learn Haskell in 2 months" as you might say, pick up Ruby after you already have significant Python, Java and C++ experience.

It is however relatively unique to Haskell and similar languages whose paradigm is very different to what most of the market is used to.

Everything is an optimization over a number of variables.

The upside of Haskell is that (if you make an effort) you get very good people and they don't cost as much because there is comparatively less demand for the language, so even on a greenfield project it makes sense to give it a shot. This is aside from numerous language-specific benefits which have been discussed to death in the never ending language wars.

The downside is that only a portion of the self-defined Haskell community can truly be more productive in Haskell than say, Python; and these people have many options and are themselves making trade-offs and optimizing across a variety of variables when picking a job (e.g. interesting machine learning work in Python vs boring CRUD work in Haskell, given the same salary, might rank about the same). If you find these people, if they want to work for you, you get good results. But you're bidding against Google and hedge funds, so you need to find an edge that they cannot take advantage of. For example, I saw almost no CVs with prestigious universities on them, because that's a common filter in prestigious organisations, who tend to miss out on the self-taught talent and the less well ranked alumni; those with "prestigious" CVs wait for organisations to get in touch rather than having to sell themselves. So it's probably a good idea to avoid using university as a filter.

One reason there are many Haskell job openings in comparably fragile settings (low budget, bad politics, low stakes etc.) is because in cases where resources are ample (thus allowing for time for senior people to learn a new language or junior people to learn professional software development skills), the risk-taking appetite is less and managers will hedge by going for the popular instead of taking a "Python paradox"/blub bet. A cynical reason for this might also be that you do not want to be dependent on a small number of talented people - having a large team of replaceable junior talent hedges against the developers acquiring strong negotiating power by virtue of adding too much value, or even just the risk that a car crash wipes out half your codebase.

In some ways the survivability of the organisation in the low-resource setting is then dependent on the Haskeller - if you help your employer increase its conversion rate or customer lifetime value 10-fold, or reduce dev ops and infrastucture cost by a factor of 10, you might well change the very dynamics of the business and justify your team and salary; so it might make sense to take a less attractive posting if you know this is a possibility. At the same time, there are numerous companies arguing this to try and get you to accept a pay cut.
Today QA is not manual testing.

Today QA is talking with product/UX, taking the end user and customer perspective, wearing a quality head end to end over features an cross devices, doing explorative testing for stuff that does not make sense to a customer (mostly what's created by the inference of different features or cross device interaction).
I am against climate change people because they invariably suggest that the governments should do something. Often this is a good opportunity for the governments to grab more power and take away our freedoms. In my opinion this can bring more misery to humanity than the climate change.

For example Indian government is pushing Solar power without must thought. Billions of taxpayer money is being spend on solar panels which we are not sure if they will work. Same amount of money could have produced more energy through conventional means, supported more industry and helped few more million people get out of poverty, I am not sure that by using Solar panels Indian government has saved million people in future.

I think a small invention like a floppy disk might have saved more trees that all government efforts put together. Environment might be changing but I would bet on people like Elon Musk any day than the government.
Computers are far more general purpose. Any computer can theoretically run any program, it would just be slower, at worst.

Developing specialized hardware isn't out of reach, because of FPGAs.
Even that I enjoy using Linux for my day-to-day computing. I have to have a dual boot set-up for school projects.

Windows 10 is perhaps the best Windows I have used yet (since 95). I have an occasional bug (Shutdown doesn't work, mouse-pad hangs,.. nothing a restart or a driver update won't fix).

PS: Besides VisualStudio and Chrome, I hardly use it for anything else.
The fear is that instead of an algorithm and a human double checking each other and refining each other that they will simply add. So we end up with the worst decisions of both human and computer.
What I've learned:

Accountability.
This can mean sharing with a colleague what you intend to complete at the start of your day, and then sharing what you actually completed at the close of the day. It can also mean finding a location where you feel the risk of your distractions being observed when you begin to indulge them (e.g. sit next to the CEO of your company; manage your work on a large, external monitor that's within your team's line of sight).

Sleep.
The poorer my sleep habits, the easier it becomes to focus on and become excited by the wrong things. What's tricky is that you may actually feel more energized or alert when sleep-deprived, but this is perhaps due to the extra releases of dopamine that help fuel you through the day, a mechanism that was once useful in hunter-gatherer days when survival depended on persisting through fatigue to complete a successful hunt.

Nutrition.
Breakfast, lunch, and dinner. I require all three if I wish to maintain consistent performance.
How to become an athlete
First-of-kind dopamine measurements in brain reveal insights into how we learn
Suicidal terrorists are hardly a threat to the existence of humanity. It takes quite a bit more than explosive vests to destroy all of mankind.
Second that. You won't be taken seriously if you you Sourceforge.
Survey's of AI experts give a median prediction that we will have human level AI within 30 years. And a non-trivial probability of it happening in 10-20 years. They are almost unanimous in predicting it will happen within this century.
It might frustrating and very un-Fun to be told "Patrick, Thomas, and Erin say no" because the rules aren't well-defined.

With respect, the rule is well-defined: it's a walled garden, we're the gardeners, and we will prune ruthlessly. You may not like that rule, but that's our price of admission.

Our rationale for it is simple: we don't want to spend our time being disciplinarians arguing with rules lawyers. I've spent most of my life on Internet message boards. I know how that conversation goes: "You didn't explicitly say that `festering pustule of a fascist` was an insult, so I am innocent of `harassing behavior.` Also, for behavior to rise to the level of harassment..." "I consider running a dictionary attack against someone else's password and then resetting it to something random to be valuable security research! How was I supposed to know that was out of bounds!?" etc, etc

It's not clear to me whether or not it's 'fun' to craftily infer your internal network topology, for instance.

You're always clear to ask. That one's Fun. Probing AWS' infrastructure to attempt to reconfigure that network topology, on the other hand, un-Fun -- it will get the game shut down for everyone.

Your game has a global audience, so you have to assume that linguistic/cultural/whatever norms aren't shared by all your players.

I am aware that we have norms which are not shared by all our players. We're going to enforce them regardless.

Please make the text stand out more?

Noted, I will see what I can do.
You sound like the people protesting the industrial revolution. Automation of all jobs should be the end goal.
I just tested this on my laptop, I used the first 5 million JSONLines of /u/stuck_in_the_matrix reddit dataset (~4.6GB).

For compression lz4 took ~22 seconds (~210 MB/s) and I got ~30% compression, gzip -1 took ~56 seconds (~80 MB/s) and I got ~22% compression.

For decompression lz4 gave me 500MB/s while gunzip gave me 300MB/s.

Commands used:

lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | gzip -1 > test.gz

gunzip -c test.gz | pv > /dev/null

lz4 -cd RS_full_corpus.lz4 | pv | head -5000000 | lz4

lz4 -cd stdin.lz4 | pv >/dev/null
In Wilmington, Delaware here. I agree with this assessment. My Dad is a retired R&D scientist in materials for DuPont. Safety has always been treated with the utmost seriousness by the company. It was common for them to fire employees as early as the 1960s for failing to follow lab safety procedures. They are also famous for taking care of their employees.
if they hurry up with this, it could save those unsellable, smog-kludged VWs.
Electron lifetime is at least 66,000 yottayears
Thanks for the random access discussion!

And you are right for embedded! .xz just doesn't work there.

I've also found that on the faster systems, for different uses of mine, when I want the compression to last as little as possible and the total round trip time matters (compression and decompression), gzip -1 gives the best resulting size for the reasonably short time I want to spend.
How are you estimating how far away AI is so accurately that you can disregard it entirely? The best we can do to predict these things is survey experts, and the results aren't too comforting: http://www.nickbostrom.com/papers/survey.pdf

>We thus designed a brief questionnaire
and distributed it to four groups of experts in 2012/2013. The
median estimate of respondents was for a one in two chance that highlevel
machine intelligence will be developed around 2040-2050, rising
to a nine in ten chance by 2075. Experts expect that systems will move
on to superintelligence in less than 30 years thereafter. They estimate
the chance is about one in three that this development turns out to be
‘bad’ or ‘extremely bad’ for humanity.
I've been trying to find books or online courses to learn about scaling the org structure of a startup and the phases it passes through. Can anyone recommend anything?
One man has spent 15 years photographing snowflakes
In other news, African-Americans are still discriminated against.
Perhaps it could display different, positive content when accessed from Volkswagen IP addresses, to ensure they wouldn't find out.
ISIS does not have access to many CS researchers nor server farms, as far as I am aware.