skysight-inc/HackerNews huggingface dataset:
Ask HN: Has Google/Bing won the search engine war?
Recently I've been working on a project that required me to search a limited set of domains.
1)I tried Google Custom Search but they restrict usage to 10000 API calls per day at $50/day. I cannot go higher than this even if I wanted to pay.
2)So I decided to write a crawler that will go to these domains and index their pages so I can put them in ElasticSearch and create a basic search engine.
3)When I created the crawler I came across domains that do not allow crawling their content e.g. Yelp, Craigslist, etc. have locked down their sites for crawling. However I see results from these on Google.
4)It seems the only option is to use their APIs (if they provide one) to get data from these domains. This can be a nightmare to maintain as the number of domains increases.
5)I want to respect the domains' policies and not use shady tactics to crawl their pages.

So essentially these domains allow google and bing to crawl their sites because they are big and established and not having their pages show up on Google or Bing would drastically affect their web traffic but smaller startups would be left out in the cold.

So my question is: What is the possibility of a new search engine emerging if the web is locked down for crawlers?
That system has blocked wikipedia before. Fortunately people made a fuss and that block was quickly removed, but accidental or deliberate censorship of a less well known site could easily go unnoticed for months or years.
Just the Apple Air, I'd bet.

Don't forget that Apple has spent years getting everyone on Xcode, and disciplining them to avoid the parts that would make a recompile & test straight-forward.

They also have a "good enough" office suite (for many people), as well as the iWork apps.

Such a device could potentially also run iPad apps without much change.
I have a couple of comments on wi-fi on camping sites in the UK. They usually always charge and are not cheap. Why is it in Europe for example France the wi-fi is generally free?
Spain: http://en.wikipedia.org/wiki/Internet_in_Spain#Internet_cens...
GPLv3 doesn't see very widespread use.

Fedora tracks licenses on a package by package basis. You can fairly easily determine what the license of a library you use is, and recursively examine dependencies to see if there is something objectionable in there.

FreeBSD ships GPLv3 ports (gcc47, ...). So... it's not just smooth sailing there, either.

No need to hate on either platform.
This Month in Data Science – March Edition
I like that Cloudera's identity crisis is coming to an end. It used to be "here's a boatload of processing power, you can do anything with it...go crazy". Now they're aligning themselves in the data warehousing space where hadoop/hdfs is undoubtedly the most cost effective solution. I can't say the same for other hadoop vendors...namely, hortonworks seems to be all over the place.
The unfortunate excerpts from this post that stood out for me:

> Without access to a lot of source code I can’t tell exactly what is going on, but...

> Until Microsoft fixes their compiler to not load their web browser it seems impossible to avoid this problem when doing lots of parallel builds.

In my past life developing applications and services on Windows, this scenario was too common — something is broken, you can't quite tell why because you don't have the source, and you can't do anything but wait for it to be fixed or find some hacky workarounds (which is tougher to do without the source).

This has been the real win on switching to an open source stack for me: you can take your tools apart, look inside them, and fix any issues. I know Microsoft has come further in this in the past few years, but it still sucks to be stuck in that kind of situation.
The evidence we have so far points to a higher churn rate the closer to the top you are. According to this [1], the top 1% had a churn rate of > 50% over a decade, while the top .01% had a churn rate of > 75% over the same period. The "Fortunate 400" mentioned by the article is something like the top .0001%, and had a churn rate of 99% over two decades.

I'd love to see deeper analysis though. What does the churn rate look like as a function of top percentile size? Do the choice of endpoint years matter significantly? Also, as you ask, what might be the optimal churn rate economically?

[1] http://www.treasury.gov/resource-center/tax-policy/Documents...
By telephone just gets you someone who is using the website on your behalf. If website is down, by-phone is down too.
It is not possible to censor what does not exist, right?
Introducing early version of Fourchette: test PR against a fork of a Heroku app
The new DocuSign experience, all in JavaScript
Perhaps I should have been more clear. I consider the fundamental concept of intellectual property to be the application of a body of law to the ownership of ideas. This body of law must be enforced by some coercive third party. (I defer to Stephan Kinsella's "Against Intellectual Property" - http://mises.org/journals/jls/15_2/15_2_1.pdf - for an explanation of why that is.)

Property rights over tangible things (whose use is physically exclusionary) do not require the defense of a coercive third party. That this is how the world works in the US today does not imply it is the only way. For example, I could defend my property with my own weapons. Or more likely, I could pay someone else to defend it for me.
Kernel freeze is April second so this means that they "should" be able to update the kernel to 3.14 but its Ubuntu and you never know.
1. Invest in Yourself.

2. Value experiences over things.

3. Avoid borrowing money as much as possible. Avoid debt.

4. Keep an emergency fund.

5. Invest in low cost Index mutual funds/ETF.

6. Read following books:
6a. Your Money or Your Life: 9 Steps to Transforming Your Relationship with Money and Achieving Financial Independence.
6b. The Millionaire Next Door: The Surprising Secrets of America's Wealthy
6c. A Random Walk Down Wall Street: The Time-Tested Strategy for Successful Investing
In general, each dollar one has makes it easier to obtain the next dollar. That's what this chart shows - wealth accumulation accelerates as wealth grows. The rich aren't necessarily doing anything wrong to achieve that effect - it's just how a capitalist system works.

If society finds this type of inequality to be objectionable, then it is society's responsibility to implement laws and policies that encourage those holding the money to reinvest it in a way that gives others access to that capital.
Search Servers to Power Your Web Apps
I'd like to know what percentage of the top 1% ever gets into the top 0.01%.
Ancient virus DNA remnants necessary for pluripotency found in humans
The 'churn' mentioned in the article is somewhat illusory, due by the use of a sliding scale. If you used a fixed threshold and identified those above that threshold, I think you'd see a much more stable picture at the top. The names at the peak may change, but that doesn't mean that the previous top wealth holders lost any, only that others gained faster.
You're totally right. Obviously, I've never gotten that far along.
But don't you have to log into 1Password on that machine in order to set it up?
For more information about Christopher Monckton, the author of this article, please see http://www.desmogblog.com/christopher-monckton and http://www.skepticalscience.com/Monckton_Myths_blog.htm

He's a well-known climate change denier whose views and writings have been widely debunked.
Is there a reason its start with CS253 then flies back to CS101 in section 3. My prior understanding was CS101 knowledge is required for CS253 to make more sense. It seems to jump around a lot aside from adding additional resources to help learning during specific sections.
Better hope they don't visit the real site after the fake site!
> The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile.

Rust's approach to serialization eliminates this unnecessary tradeoff, IMO: it uses the built-in macro system to parse the structure of your serializable data types and generates the specialized code at compile time. There is no special build step, you get the full speed of specialized code, and it's as simple as writing "#[deriving(Encodable)]" at the top of your data structures.

(As an added bonus, we're using the same infrastructure to generate trace hooks for precise GC in Servo, which eliminates a major pain point of integrating systems-level code with precise garbage collectors.)
SF's Entrepreneurship-in-Residence program to roll out across cities through EFA
Now this would be a reality show I would like to watch :) Thanks!
Ask YC: Are the intern application results out?
Re: http://blog.ycombinator.com/intern-at-a-yc-company
This would probably be more popular than Ingress if they did it right. And yes, combined
with the "other" AR (augmented vs alternate) it would be revolutionary.
Except the P4 was a very inefficient architecture which relied on clock speeds (and the power of Marketing) rather than actual computing power

Especially the first P4s, those were awful. A Tualatin P3 @ 1.2/1.3GHz would be much faster than a P4 @ 1.4 or even 1.6
How about riding the motorbike on the track? Best of both worlds? :)
Segment.io (YC S11) is looking for a lead mobile engineer
We're looking for an extremely good mobile engineer to help us change the way people approach mobile analytics.
This is a perfect position for someone who loves building clean APIs for others to use. You enjoy not only building mobile apps, but writing the libraries that power those apps. You're fluent in everything to do with iOS as a platform, and if you have Android chops that's a huge bonus, but definitely not a requirement.

You'd be working on current and new features of the mobile SDKs themselves, as well as helping to write code for where they touch our Node.js backend, where we process tens of millions of events per day. And you'd be in charge of our entire mobile offering, which is used in product by lots of big companies like Live Nation, Nest Labs and The Knot.

You'd be joining a team of very talented engineers that are in love with open-source. Both of those SDKs are open-source, as is tons of our codebase[1]. So if you enjoy working with smart people and getting to make a name for yourself releasing the things you work on to other developers, you've found the right place.

Not only that, but you'd be working on a few of our still-secret mobile projects that we're pretty confident are going to change the way people integrate third party code into every mobile app they write afterwards.

[0]: https://segment.io

[1]: https://github.com/segmentio

---

To apply, email us at jobs@segment.io with:

1. a description of an app you've worked on (code is a bonus)
2. a description of a mobile library you've built (code is a bonus)
3. a link to your GitHub if you have one
4. the most interesting mobile SDK you've worked with and why
GitHub Showcase: Clean Code Linters
> If "the 1%" almost entirely changes every year, well... what's the problem with that? You're always going to have a top 1%. Don't we want it to have a significant amount of turnover? Indeed, these numbers sound perhaps a bit low, if anything (is it really that unstable up there? That's not obviously good to me.). If it's not the same people all the time, it's not obviously useful to be talking about them as a distinct group.

If the people in the "pool" competing for each year's exorbitant profits (the 0.1%) remain the same every year, there may still be a problem. That pool is called "the 1%"
Depending on your timeline, the AirVision stuff from ubnt.com might eventually meet your goals. Their line currently doesn't do night vision, but it's supposedly on the way Any Day Now(tm).

It's pretty flexible, good software, good motion detection, etc.
Well, P4's clock was achieved with 31-stage pipeline, which was a bad idea. If you compare with what came after P4, there was more than slight increase in clock.
No. 1Password is just software; there's nothing to log into except Dropbox.
How to get noticed on Kickstarter
Great article I just found for those who are interested in marketing/funding on kickstarter.com.
How? They still have gobs of principal lying around making gains. Unless you mean it will help enable more progressive taxation and income redistribution via the federal government?
nice features would be social sharing and cloud syncing.
If someone hotlinked one of my scripts, I'd update the head of the script and use the opportunity for free marketing:

if(!window.location.host.match(/(www\.)?myapp.com/)) window.location = 'http://www.myapp.com';

I bet they won't hotlink after that.
totally agree, QAM does transfer multiple bits

in fact in today's world everything from modulation and multiplexing is done digitally using network processors which actually work with bytes internally, so technically it should be possible to give network bandwidth in bytes

but my point is transmission on network or data bus, always happens in "frames" and a frame of length x bits, usually contains y bits of actual data and (x-y) bits that contain parity/redundancy bits. transmission of 1 byte usually never takes exactly 8 bits during transmission, with an additional few bits to ensure correct reception
Yeah, maybe "result" is wrong. It's not a cause and effect. I probably should have said that having social and economic resources makes it easier to tell the truth, specifically in instances where the person knows they can easily withstand any negative consequences.

As an example: leaving a note when you accidentally damage another person's car.
That was my thought, too. It seems like if they do this then they'll be going down the same path as Microsoft did with the Surface RT and Surface Pro, right?
So not just proprietary software, but saas using a proprietary client? In for a penny, in for a pound I guess.
Technical: Haskell & MongoDB
Non-Technical: Basic Business for a Technical Entrepreneur (Gautam - you could be a good fit to design such a learning pth)
On the .Net side, Jil is doing similar, creating a custom static serializer per type. It’s able to do the code generation at runtime by emitting IL: https://github.com/kevin-montrose/Jil
> most of the time the content is available elsewhere,

As is often the case with censorship, what happens "most of the time" is not the concern - it's the rest.

The DMCA isn't exclusively a tool of censorship, but no effective tool of censorship is. For it to survive as an effective tool, it has to provide some value in other ways.
This isn't honestly terribly surprising. When the sky started falling, plenty of folks were happy to mention that bad times are only bad for people who can't afford to take advantage of them.

When stocks are crashing? That just means they become cheaper to pick up for the very rich, who get even richer when the economy straightens out again. Meanwhile, everyone who had to dump their investments to survive has to start over with less.

It shouldn't be hard to recognize that there's something terribly wrong going on here, but the tough part is trying to figure out just what to do to fix it. Hell if I know.
Ah, I was unclear. When I said restart your web server, I didn't mean to reboot the machine. Just restart the web server process. Nginx or Apache or whichever you're using.
1. You send a trade to your brokerage for GOOG

2. Trade gets routed to an HFT who will fill the trade

3. HFT notices a spike in GOOG interest over a few seconds and starts buying at 1133.90 driving the price up to 1134

4. HFT fills your limit order at the best price of 1134 which they themselves hold.

Despite what the other commentators here have said, limit orders are less safe than market orders to market manipulation. HFT's will buy up all that dumb flow with limit orders and then essentially run the prices to the limit orders in their favor.
Look forward to it :)
Everyone loves to cite Hedge Funds. I am no hedge fund expert but you mention a few benefits that they do provide...liquidity and information aggregating (which has derivatives such as market signalling) are very much benefits to society. What good is a stock to anyone if there isnt a liquid market to sell into (and yes the buyer should make money on that transaction). A personal hunch is that some hedge funds are breaking some laws but largely the hedge fund industry provides a net benefit to the economy. The stock market and futures markets are NOT zero sum games. They provide capital, incentive structures, and a host of other benefits to both businesses AND everyday people (like you who has a retirement fund with lots of assets that pass through hedge funds). Businesses can (and do) grow and appreciate your assets (and hedge fund assets), etc etc.

Your second example is dead wrong. Identifying undervalued resources is an enormous efficiency gain in a market. They lower their cost structure while keeping quality high. It ensures that top athletes salaries are in check because they truly need to be twice as good (or 10x as good) to be paid millions of dollars. Which makes game tickets, jerseys, etc affordable to more fans who in turn buy more tickets, fan paraphernalia, etc. Yes indeed the pool of players is set (at all of humanity) but the wealth generated from quality entertainment is not capped (or capped at the leisure money of a market). Those sports leagues are always looking to expand their market and keeping costs low is one major strategy to that.
WhatsApp is now a medium for storytelling
The big problem with the article is that the author conflates wealth and income, and switches back and forth willy-nilly, so as to render all of his claims -- without further clarification -- meaningless.
What percentage of the 1% ever "churn" into the 0.01% at all? Churn makes it sound like it might be random, but I don't think that is demonstrated.
NSA infiltrated Internet encryption tools even more deeply than thought
You can use npm modules on the client via Browserify. And there is a dedicated to client-side package manager called Bower.
Facebook is Myspace
The Fastest Growing Open Source Project?
Dan Grossman's Programming Languages class on Coursera includes a good treatment of OOP in the later weeks.

https://www.coursera.org/course/proglang
That said - as somebody looking for specific Python / Django for web development resources, this is a great approach! I could see the author forking the tutorial at certain logical points moving forward to broaden the audience, but love that it speaks to exactly what I'm looking to learn.
You don't have to work in a developed nation to work with developed nation clients.

Try and target yourself to those clients and avoid the local clients like a plague. It has been my experience that if you want to break into more profitable territory you need to break away from local clients completely.
If I were amazon, I would've prepared to do a price cut when Google announces one. Then when they announce it, just push the button. Why give up margin earlier than necessary? *

Of course waiting to do a price cut till their competitor does leaves a bad taste in some customers' mouths. But I bet they think they're far enough ahead that it doesn't matter. Would Netflix and similar big AWS customers really leave AWS over something like this?

* This strategy even has a name: "follow the follower". Very common strategy in sailing.
My apologies if you found that disorienting! You are right that CS101 becomes a prerequisite for CS253 (if you don't know python already). The reason I started with CS253 is because, personally for me, it helped a lot to know the top level concepts of "How web works" and the front-end (because that is what a normal user sees) and then go into concepts of server side languages which might seem too involved and dry to a beginner.
API-Driven Development
A lot of people are opposed to there being a distribution more extreme than their emotions tell them is reasonable.

Consider what most Americans say the "ideal" level of wealth inequality would be: http://danariely.com/2010/09/30/wealth-inequality/ and http://www.slate.com/articles/news_and_politics/politics/201...

Now consider how much wealth inequality there would be in an excessively fair society: http://www.daemonology.net/blog/2011-01-10-inequality-in-equ...

Clearly, people are determining their "ideal" without doing the math. This makes it hard to take their opposition to "this change in distribution" seriously as well.
I doubt that most years they are hurting terribly...

They wouldn't have to 'hurt' at all, in fact they could experience personal records for income every year, year after year, and still drop out of the top 400 (or top .01%), since it's a sliding scale.
Naming UI components in OOCSS
Nodality, The Node Based Calculator, for iOS and Android Tablets
Added to my list of tutorials to attempt and hopefully steam through. I love guided lessons like this.
I think that here in Uruguay, we don't have any kind of internet censorship, but we have a monopolic ISP owned by the state, so... At any moment they could turn on the censorship switch and there is almost nothing you could do about it.

I feel like this is the general situation on countries where you "don't" have internet censorsihip, you don't, today.
Not exactly. Remember that the Surface RT deliberately restricted the applications that could be run via Win32 on the familiar desktop. Just about every third party application needed to adopt the new RT APIs and run "Metro"-style.

If they go down this road, Apple will not put constraints like that on their Mac developers: I'd expect that all the most commonly used frameworks would continue to be available, and porting most applications would amount to a recompile against a new Xcode.
A new Corvette convertible in 1970 cost $4,849 when the average white male made $7,011.

A new Corvette convertible in 2014 costs $58,000 when the average white male makes $31,335.

Young people are going to stop caring about cars when all they can afford is a honda civic. Cars which are exciting, sexy, impressive, etc are simply out of reach for most young people. Unsurprisingly, no one cares which boring sedan they or their friends are driving.
Yeah, we're brainstorming how to throw in some video / audio (podcast) to keep it entertaining.
so how do you earn?
http://en.wikipedia.org/wiki/Sabre, http://en.wikipedia.org/wiki/Sword#Single_and_double-edged
I believe that. Even for 0.5mm QFPs, trying to do a stencil at home controlling the amount of solder paste is very difficult. Almost every attempt I've tried leads to at least a few bridges (easy to fix, but still annoying). With the professional machines you guys have it's probably much more reliable even with not so great stencils.

Do you end up doing a final DFM as well for most submitted stencil files?
Which is silly if the car is going faster that 15 mph (edit: actually 18.6 mph) . At these speeds what you are hearing is the road (tire) noise, not the engine.
52% of Apple employees are in Apple Stores through the retail division.
Yes, using 1Password means that you have to trust the developers. Open source would of course be preferable but the open source community has IMHO not managed to provide a comparable suite of apps. Personally, I am more worried about keychains as single points of failure.

Some time ago, the 1Password developers tried to explain why you should trust them:

http://blog.agilebits.com/2013/09/06/1password-and-the-crypt...

In addition, http://learn.agilebits.com/1Password4/Security/keychain-desi... might be interesting with regard to your question.
So to verify a key I need to install npm?

No thanks.

" It doesn't seem particularly safe for me to trust my valuable PGP keys to this system."

I agree
The Future with AR
You could have advised the client to use the experimental technology that facebook uses to build their front-end, something called hhvm ;)