skysight-inc/HackerNews huggingface dataset:
Fascinating paper. As a math graduate I had no idea that the switch to axiomatic descriptions and formal logical proofs was so recent and controversial, and no longer working in math I had no idea that "core" methodologies (i.e. pure mathematics) are under threat (although this makes perfect sense given grant funding priorities).
Elimination of the "work for hire" doctrine wouldn't mean that you couldn't contractually transfer your IP. It would just mean that such transfers would have to be in explicit contractual language, instead of the vague "your employer lays claim to everything you create" situation we have under work for hire.
Or whether Google would allow WordPress blogs to use the .blog domain? From what I recall, they intended to make .blog exclusive to BlogSpot.
When it catches on with Mom and Dad and the country western Walmart crowd hopefully people will start making money.

And all we know is some artists are getting screwed. Warner, Universal, EMI might have made a deal where they get a decent percentage and smaller labels get peanuts, we have no way of knowing.
This is OT, but you really should track clicks on links (if you need to do it in the first place) in a different way than this:

<a href="javascript:mixpanel_track_link('http://liftweb.com/','link:liftweb.com');">my first encounter with Lift</a>
It's not that easy to secretly tape someone with a cellphone.
Great writeup. I especially like the Telnet REPL trick:

require("net")
.createServer (socket) ->
repl = require('repl')
repl.start("my-node-process>", socket)
.listen 5000, "localhost"
Small nit: Watson, not Deep Blue.
I found some pleasant similarities between Pascal and Go (Golang)
Also recommend Mechanize for Ruby (uses nokogiri under the covers).

http://mechanize.rubyforge.org/

http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites.

FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs the possibly altered DOM (after JS has run on the page) that you might see with Dev Tools/Firebug.
Prizes still generate negative results. It's just that someone else is paying for it.
Someone needs to build out the auction infrastructure where all market participants (institutional buyers, high-net-worth individuals, retail investors) can participate. Until recently Wall Street investment banks had a stranglehold on high-net-worth individuals, but that's changing.

SecondMarket is probably in the best position to implement this - they already have access to trading activity of mature private companies, and while they don't do auctions, they do run order books for large lots.
Here are my highly opinionated opinions about their respective use cases:

* BeautifulSoup: It was the best scraping library ever until python-lxml came around and stole the show. Despite that the manual said BeautifulSoup gives you unicode, damnit! it had some long-standing bugs which it gave you strings or incorrectly decoded web pages. I wouldn't use it anymore because lxml is strictly superior.

* lxml: The king of scraping libraries. It is a big library so it can be hard to approach. It's actually a collection of markup parsers; lxml.html, lxml.etree and some more I've forgotten about. I almost exclusively use lxml.html since it "works" and can handle invalid markup without complaining. Use it like this: https://gist.github.com/mattoufoutu/823821 lxml can parse both using jQuery-style selectors with cssselect() and XPath 1.0 with the xpath() method. XPath is hard to learn, but once you get it you have a really powerful parsing tool which makes your life simpler.

* pyQuery: Easy to get started with and use. But not as powerful as lxml+XPath.

* urllib: Many of the modules in Python's standard library are there because they have been there for a long time. :) python-requests or httplib2 are the best libraries for http.

* scrapy: A framework for scheduling and supervising scraper spiders. It takes care of everything from downloading pages, following urls, concurrent requests, handling network errors to storing data in a database or generating csv files. It doesn't parse html itself, but delegates that task to lxml. Personally, I've found scrapy to be very good when your problem fits with how scrapy thinks scraping should be done. If you try to depart from the scrapy-way then scrapy suddenly feels very "frameworkish" and limiting. For example, i spent a lot of time trying to get it to support delta-scraping -- periodically scraping the same site, but only download new or changed data -- but it felt impossible getting scrapy to work the way I wanted.

* mechanize: Python port of the Perl module WWW::Mechanize. It's good for tasks like scripting logins. If you want to automate login to a site with a username and password, without having to care about session cookies, then mechanize is the ideal choice. Look elsewhere for an html parsing library.

* Scrapemark: It has a fun and clever approach to scraping. But once again, not as powerful as lxml.
There is no freedom of contract issue--getting rid of "work for hire" doesn't mean employees can't contract to transfer their IP to their employers. It just means that these transfers must be specific and explicit.
Read up on how Reddit got started. They essentially made up a bunch of fake accounts and shared content to make it seem like they were bigger than what they were. That's how they got over the chicken and egg problem, but it might not work for this type of content.

A video might be a nice add to explain things and would add a nice content block to the home page.
I've just added the .io and i'll add .com in a bit. Adding in the loop might be a bit much if you're not bothered about certain services - e.g constantly updating the page just because a twitter name cant be found.
The "singularity" is never going to happen because it assumes intelegence is a single thing that can be increased with more processing power and better algorithms. Completely ignoring things like information theory, over fitting, insanity, training data, and chaos theory.

We can and probably will build reasonably general AI, but you far more likely to see each generation of AI being an ever smaller improvement than any sort of runaway exponential progression. Not to mention hardware progress has slowed to a relative standstill.
That's not really true about Newton by the way. Newton most likely believed he did not have a convincing argument he wanted to put his name to or advance that wouldn't also drag him into yet another round of having to deal with someone like Robert Hooke. At that point of his life, his ambition was greatly lessened and his success somewhat assured.

There is firm evidence he had at least 3 serious views (and one fairly archaic one not many want to talk about) on the cause of physical gravitation. It is only the "modernists" who need for Newton be be not beholden to any of them that elevate and reiterate "Hypotheses non fingo" to some sort of rallying cry as if he had it tattooed on his torso.
10 FPS on my Nexus 4 with Chrome Beta. 20 FPS without the GUI. Very nice!
There are many blind chess players. Competitive chess rules allow blind players access to special equipment (basically an auxiliary pegged board) so that they can 'feel' the position. Really strong chess players don't actually need sight of the board and I imagine the best blind players (there are many of master strength) don't make extensive use of this provision.

In the biannual international chess 'olympiads' (team competition between countries) a multi country blind players team competes. Some other disabilities are also represented in the same way.
> Scraping online content is so simple these days, if a website doesn't offer an API you still have alternatives ;)

Having written code to both leverage a site's (private) API and to scrape that same site (when the private API stopped existing), I would much, much rather use an API. Yes, the scraper works, but the scraper's code is much messier, and JSON keys, for example, provide some documentation in their own right. Looking at the scraper months later, there's a lot more headscratching. Scraping will work, but it also will leave you searching for many little pieces of information that would be exposed by an API but aren't by a static site.

Still, I will agree that scraping is much easier; I've used jsdom (Node.js) extensively, and, for my use cases, it feels like working in a browser (full DOM, scripts, etc.).
This looks great and that is a huge amount of effort. I have to ask about what I think is the elephant in the room: Surely you must have noticed that all your mountains are chopped off at the exact same altitude? Wouldn't the scene look way better if you just increased the maximum, with basically no effort?

It's as though you spent a month painstakingly mixing 64 channels of crystal clear audio, but, right at the end, threw your hands up and clipped the final mix into oblivion.
You can say the exact same thing about any scientific model. My view on this has always been that as long as the model accurately predicts experimental results, assume it is correct for your calculations until it is proven to be wrong.

Even then, we never stopped using classical mechanics even though they were proven to be wrong at a variety of scales. They just happen to very closely approximate reality in some contexts and are useful.

The fact of the matter is, we have tools that are correct as far as we know and they point towards thinking that every quantum system is computable. Until this has been proven wrong, the fallacy is believing the brain is different, not the other way around.
Lots of possibilities allowed by general relativity (e.g. computers orbiting black holes and information traveling backwards through time) that may or may not correspond to our universe allow for computation beyond that of a Turing machine. Until we understand quantum gravity it seems premature to write off those possibilities.

Even simpler, there is the question of whether the universe allows for arbitrary precision measurement of a physical quantity. If it does, then it may turn out that an observable physical quantity (e.g. the mass or charge of a fundamental particle) is a noncomputable real number. If it doesn't, then it may be possible to represent a neural network with noncomputable real weights, but impossible to actually measure them to enough accuracy to simulate the network on a given input.
Came here to say the same. At least they should detect a scroll wheel click and open the link in a new tab.

Or they could use the google approach and change the link in the last moment to point to a tracking service.
Complete Vagrant Python Env...
The contact list link... priceless.
Is one of the five computer science ninja arts making a website that is completely blank (like, literally nothing) with JavaScript disabled?
Quantum theory is a "hypothesis"? The entire foundation of modern electronics is based on quantum theory. It's something that has been tested over and over in labs and the regular world for over a century.
Awesome. Because if there's one thing humans are better at than computers, it's solving chess problems.
What are you using to determine whether *.io domains are "free" vs "taken"? It seems to be incorrect.

http://www.usernique.com/io/x
-- "free"

http://www.nic.io/cgi-bin/whois?query=x.io
-- The domain is unavailable for use.
A physical force like gravity? Why refer to it as a god then?
Wealthier states do more global travel? It's expensive.
"that would be like a car"
It's been tested and proven FALSE, just look at relativistic effects. We just don't have anything better.
You don't see Firefox available on your Nexus 7? Have you modded it in any way? Firefox works fine on the Nexus 7.

https://play.google.com/store/apps/details?id=org.mozilla.fi...

and the beta:

https://play.google.com/store/apps/details?id=org.mozilla.fi...
Singularity does not have to do with our perception of the intelligence of a machine. In singularity the importance/validity of that machine's perception of our intelligence is higher than our's. Therefore, I wonder whether we will be able to know it when we reach singularity.
Like everything else. Google.
Along those lines, the BBC documentary "The Story of Science" shows over its episodes that this is true. It's really a fantastically intellectual series and well worth acquiring and watching.
I'm surprised this article didn't mention scrapy. I had to do a lot of web scraping for a healthcare-related project last month and found scrapy incredibly fast and easy to use.
Entrepreneurs are humans too, or: US Marines do cry
One more with the same problem. Right click open in a new window, and then 3 tabs with about:blank.

That is also because using lift?
Ask HN: When should you add a contributors name to a copyright notice?
When someone contributes to an open source project, at what point should their name be added to the copyright notice for the file(s) they edited?
Sweet! I love this name: etabocagi

Sounds like an indigenous tribe or something
I think this makes sense. Even besides the celebrities on twitter, there are many party promoters, news outlets, bloggers, merchandisers, etc. that use twitter to advertise to their followers, and it provides a far more efficient advertising medium than the alternatives. The service charge for having a huge following would probably be well worth it (perhaps even negligible) to certain major account holders.
My only complaint with the wording is the use of unnecessary technical jargon. Why use the word "captcha" ?
The Economics of Weight Loss
Giving anti-biotics for every situation that doesn't even require them puts a huge burden on your body. I swear one of these days people will go the emergency room because they "have a pimple", and they'll immediately be given a round of antibiotics. Nothing would surprise me these days anymore :)

Antibiotics have become so commonplace that they're prescribed for just about anything.
Antibiotics don't cure what caused the problem, to heal you need to look beyond temporary band-aids.

They don't just kill bad bacteria, they also kill good bacteria. Without good bacteria, it's likely you'll develop food intolerance with repeated antibiotic use. You might develop absorption issues, and then nutrient deficiencies.

They should be given only when there's no other alternative, only when it's absolutely necessary, but it's given out like candy by lazy doctors (though I bet they do it because parents insist, since it gives the illusion of "doing something" to help.)

It's better to take really good care of your body so it can fight infections on its own, but then again people are told constantly to eat food that's not good for us, and to not eat food that is good for us, so it's no wonder so many people get sick so often.
I'm trying to work out what you'd want this for. I suppose as a quick "yes this does actually work" but I can't think of many less efficient ways of spidering things - whole headless browsers? Ouch.

If you needed to do this, you'd use something that already exists - if you just wanted to do it in Node, for some particular reason, why not just use plain requests and an HTTP parser?

I don't mean this to be overly critical, but just in case anyone's thinking this would be a good idea - there are better ones. Just to learn from, fair enough.
Well for starters it needs to support many more websites (Facebook, Google/Youtube, Reddit, etc.) and needs to actually do the check automatically.

And it must be able to register the usernames in bulk, bypassing registration limits using multiple proxies and bypassing captchas using deathbycaptcha.com or similar.

Finally, support for generating realistic-looking activity on the websites would also be really useful.

Add an API and it would become a solid service.
They make scraping as easy as finding the right jquery selectors (once you inject jQuery onto the page) but can be very slow as compared to a vanilla HTML only scraper.

In my experience, a phantom/casper implementation could take upwards of 5-10 secs. to process a single page (almost 5-10x slower). This, even if you disable load of remote images and plugins.
Others seem to include hijacking the back button. Could people linking to such sites please include a warning in the title so we can avoid them?
One possible motivation might be to knock github offline following a 0day on some software such that the patch/latest version can't be downloaded.
Technically, you'd have to say that brain's output is combination of computable system and a perfect random noise source.

That is because a simple source of quantum noise should not be computable. Any computable pseudo-random noise generator produces strings of digit with asymptotically constant Kolmagorov complexity while the Kolmagorov complexity of a true random number generator goes to infinity.

All that said, the person is article is certainly overreaching to imply we can know the brain definitely isn't computable. Clearly he does have any kind of knowledge that would tell him that. This might enough to say since the rest of his argument seems to hinge on this.
Because the discussion that came from it is really good.
just checking the return code on a GET request. src is on github if you're curious. You're right though, looks like the domain lookups need a bit of work.
Apparently etune.com is available... cooler than any of my domains, and randomly generated. Nice.
> Do you currently carry a lot of short-term or long-term debt?

Nope: Mortgage + car < 20% of income. Enough savings for 2-3 years of frugal living if the bottom dropped out, and could (for some unimaginable reason) not find other employment. Vacations on cash, not credit. Never finance "toys", things like TVs.

I watched peers (in terms of age), go right into things like multiple car payments + high mortgages + student loans + kids right out of school, and were thus mostly working to pay off debt, and never really get out of it. Unnecessary financial pressure can be crippling.
Should anyone care? Once someone launches a DDoS, their argument is invalid. And reporting the grievance might just encourage them.

I don't know anything about any of Github's attackers, but from other incidents I know of at other services, it could be anything. A billing dispute. Anger that something was taken down... or not taken down in response to an unreasonable request. Anger that an account was suspended... or that some feud-rival's account wasn't suspended. People throwing "do it my way or I'll take your site down" tantrums may not make sense to anyone other them themselves.
What are they, aside from just := and the lack of brackets?
Kurzweil is saying much more. In The Age of Spiritual Machines he states "Artificial Intelligences claim to be conscious and openly petition for recognition of the fact. Most people admit and accept this new truth." What cluster of running software by which algorithms produces this consciousness 16 years from now?
Can you provide some insights into performance? In my experience, a single page could take between 5-10 secs. to scrape.
"blue-sky R&D funding" is the definition of "massive waste" to many people. To you, it's probably feeding poor children or something else.
I think you're thinking about the OLD web, when adjusting fonts would break table-based, graphic-intensive websites. Unless you want to go even further back - to a time when gif backgrounds of the stars were common and text was displayed through Comic Sans.
Yeah, I agree. You could do a sortable list of services so the user could rank their importance and then pick the best result from the first X(10?) iterations given their preferences.
Obviously, that adds a lot of complexity, you just got me excited.
awesome, good find!
Motocross Madness... on the web? :D
Data Privacy: Why can't Chrome and Chromium users run their own sync server?
While it's hard to make a data driven argument in this field, I think the US government has been incredibly successful in picking winning new technological ideas, judged by my notion that the vast majority of the scientific and technical advances in the last 60 years have occurred here in the United States. The US government has long had a tradition of picking winners in science and technology, and I don't think the fact that we are also the leader in science and technology is simply coincidental to that.

Looking for counterexamples, ie technical advances that have occurred outside of the US due to lack of government support, the only one that comes to mind is the failures in green technology, but I think that small failures there are minor compared to the other successes.

I also think looking for scientific literacy in the executive positions of government is less relevant in the highly bureaucratic world of government funded research, as there executives have less control on individual funding than they would in the private sector. As long as the the executives respect the technical expertise of their bureaucrats, (and the workforce is knowledgeable), then individual domain knowledge is less critical than it would be in the private sector.
Should I build my complex backend with Netkernel?
Perfect Workflow in Sublime Text 2 (video)
Suppose one has the means to build a flexible General AI with modern silicon technology. How could this not result in a thing whose abilities couldn't then be continuously increased at the rate regular silicon technology increased?

The whole "diminishing returns will stop us" just seems like a comforting fairy tale for those who don't want to think about the consequences (which I suspect a bit of thought does show won't be as rosy as Kurzweill imagines).

Edit: Hardware progress is still mostly following "Moore's Law". The only that's not increasing is processor speed. But if we build a "reasonably general AI", how could that box's capacities not be increased by tightly integration with other similarly intelligent boxes?
I think new economies is a good proxy. The computer invented an entirely new economy. So did the automobile, computer networking, television, radio, the telegraph, electricity, air travel, moving pictures, recorded audio, the light bulb, washing machine, dishwasher, etc. All of these things created not just the immediate industry, but an entire ecosystem of products and services related to the industry.

You could probably also measure it in terms of essential inventions: we can't live without the internet, but if Instagram or Facebook went away tomorrow, we'd have lots of alternatives for sharing photos and talking to our friends. Instagram and Facebook make those things nicer, but don't allow us to do anything we weren't doing before. The internet (or, I guess, the web) allowed for entirely new forms of communication and commerce.

From my perspective, we haven't seen a new economy emerge in the last decade, at least. The internet (maybe molecular biology) was the last invention to really create a new industry, and that was the mid-90s. Maybe robotics is the next one, but it's hard to tell prospectively.