skysight-inc/HackerNews huggingface dataset:
Partial plug, but very related to the topic: if you're doing large scale analysis on the web and you don't want to have to actually run a large scale crawl, use the CommonCrawl dataset[1]! Common Crawl is a non profit organization that wants to allow anyone to use big web data.

I'm one of the team behind the crawl itself. Last month (July) we downloaded 4 billion web pages. Thanks to Amazon Public Datasets, all of that data is freely distributed via Amazon S3, under a very permissive license (i.e. good for academics, start-ups, businesses, and hobbyists). If your hardware lives on EC2, you can process the entire thing quickly for free. If you have your own cluster and many many terabytes of storage, you can download it too!

People have used the dataset to generate hyperlink graphs[2], web table content[2], microdata[2], n-gram and language model data (ala Google N-grams)[3], NLP research on word vectors[4], and so on, so there's a lot that can be done!

[1]: http://commoncrawl.org/
[2]: http://webdatacommons.org/
[3]: http://statmt.org/ngrams
[4]: http://nlp.stanford.edu/projects/glove/
You should still be able to learn quite a bit with just the Chromebook. After reading the article, I'm thinking about getting one myself so I can get back into learning web development again.

Did you end up buying the Acer C720?
There are wireless power monitors which have their own smartphone apps. I don't know if any of them can be set up to give you that sort of notification, but some of the manufacturers make enough information available that you could write your own app.

http://www.belkin.com/us/p/P-F7C029/

http://developers.belkin.com/wemo/sdk
Thank you for answering. Though private and internal, you are still using the data for profit, is that correct? Does it mean that one can't sell the data, but can use it for analysis etc and still profit from it?

I saw a service recently that emails app store reviews/ratings for a fee. Not sure if they are scraping or getting the reviews some other way. The same idea can be extended to lots of things like Amazon reviews etc. Not sure of the legal stuff though.
The number of small businesses who (1) are not currently using Square and (2) are happy using $EXISTING_POS_SYSTEM and won't be convinced to use Square anytime soon and (3) would make significantly larger amounts of cash being offered something like Appointment Reminder as a standalone service that integrates with any POS system is a large enough market to feed patio11 and his family for a very, very long time.
What is tendancy ?
I scrape craigslist for side by side comparisons of stuff I want to buy from there. Eg: Cars, motorcycles, etc. Maybe even real estate would be a good target.
If you don't need salts you can just use the inbuilt base64 functions:

var numbers = [1, 2, 3, 4, 5];
var encoded = btoa(''+numbers).replace(/=/g, '');
var decoded = atob(encoded).split(',').map(Number);
console.log(encoded, decoded);
So i guess the script would need to include some kind of gradually escalating random "like" increase.
Incredible eye opening article..
That's the one thing that keeps eating at me. Would it not be trivial for a large corporation or government entity to throw enough money around to accomplish this? The only response I hear is that the market will magically prevent this.
Thanks man - really means a lot to hear that :)

Well, I'd like to support my free users without relying on donations. But you're not the first person to say this, so if all fails, it's good to know I have this an option!
Does this remind other people of MicroEval who also did YC a few years ago?

They have shut down now but you can see info at their site still http://www.microeval.com/.
Cool, have you heard of Guerrilla Queer Bar in Boston? Very similar concept.
(How) does this compare to AntTweakBar?
wonder what the limit is? with out getting suspended.
It's not just a web service -- it's any web service. The algorithm is up to you.

I'm not sure what value they're providing, exactly. Anyone can put algorithms on the web now: you write them in a standard language like PHP and upload them to any hosting service.
The only part of this robot I'm skeptical of is the "grind-on-demand" portion. As someone who regularly grinds my own chuck to make hamburgers, I've learned that in order to get a consistent, even mix between fat and meat, it's absolutely critical to grind the entire roast (actually it's ground twice), and then mix to evenly distribute things.

If this were truly grind-on-demand, it would be quite hard to guarantee a consistent product (which is a hallmark of most fast food restaurants).
Thanks for looking at the repo. A couple of things to keep in mind.
- This list is only a starting point I pulled from a white paper. I agree that it does need to expand, hence the ongoing and open source nature of the project.

- If you have a suggestion for how this could be done with machine learning, please feel free to submit a pull request. Just keep in mind that the project only aims to catch language that can be reasonably interpreted as self-threatening.

Lastly, if you have a problem with how the project is implemented PRs are more productive than comments. It is far from perfect, but I believe it is still better than nothing at all.
He shouldn't, Heaven likely has some interesting natural resources.
I'd like to know how a startup still looking for a founding engineer...

https://news.ycombinator.com/item?id=8120070

...gets written up in WIRED. Nice PR work.
I love scrapping and even made a subreddit for the purpose where I showcased few of my public work. Any scraping lover can join in.

http://www.reddit.com/r/scrapingtheweb/
>The thing is, it is in your browser.

I already excluded this part from my criticism. Please read my original comment.

>You can't say the same for Steam.

So, the solution to that is to create something like Steam, but better. Choosing an inferior technology is the opposite of a solution.

I predict that advertisers are going to overuse webGL just like flash. We're going to see webGL blockers crop up the minute it gets any kind of traction. Certainly, I am not going to subject my machine to another malware vector.
This would see a significant slowdown for any algorithm that requires large amounts of data, as a lot of them do. The core philosophy of Hadoop which made it successful was "move the computation to the data and not the other way around."

Although the algorithms may run efficiently, ferrying significant amounts of data to and from a web service could quickly negate any gains.
Thank you! Haha being too busy is a problem we all have ;)

btw, I like where you were going with your timer. I've had similar thoughts and was going to do something for the Chrome extension in the future. Stay tuned for that.
Wouldn't make more sense to push employers to move jobs to existing urban areas (Oakland, San Jose, and San Francisco) instead of trying to build more transit to suburbs, and creating even more urban sprawl?
http://www.phrases.org.uk/meanings/on-cloud-nine.html
Yes, of course. When there is a big, ugly design flaw in the program, blame the big, ugly workaround for it.

City planning is difficult. The engineering itself is actually pretty straightforward for skilled laborers, but that isn't the only thing to it. The problems usually stem from all the buildings being filled with irrational yard apes that hate and fear change, and have an unjustified interest in what their neighbors do with their property.

SF needs a higher density of residental building space. People who live in SF use their political power to make demolition and construction projects aimed at increasing housing density unprofitable, or at least less profitable than similar projects undertaken elsewhere.

Result: housing shortage. Solution: blame AirBnB and continue to ignore the underlying problem.

The critics should be blaming those actually responsible: every person in the Bay Area that has ever opposed new construction, by any means, for any reason.
Surgeons Removed Part of Her Brain, Say This She Became “Hyper-Empathic”
I guess not being able to erasing data on FB servers is now just an urban myth.
I also have a unique constraint on the value and re-run the generator if there's a collision.
0/0 is "undefined" rather than "infinity" (:
Thanks for the kind words. That's exactly what I was going for :)

If you ever need help, have questions or for whatever reason, find it's not working for you guys, I'd love to hear from you. You can always reach me at john at dayboard.
And then watch rents rise across the city as land-lords queue up for that sweet-sweet subsidy cash.

Perhaps the solution is to spread the economic activity out from where it is concentrated and build more high-density housing near the economic activity despite ensuing NIMBYism.
Link to actual post: https://news.ycombinator.com/item?id=8120514
Very interesting. Thanks for the response!
That's a fair point. So far there hasn't been any transcribing necessary because I have the customers reply to their receipt emails if they have a problem, but that's definitely something to consider if you have to do phone support.
Let me give you an idea and direction to at least look into for your product, either currently or in the future.

Feedback, for hospital stays is super important. You should look at adapting a version of your product to suit people having hospital stays both elective and non elective. Hospital's are already conducting patient quality surveys and something like this product could easily fit into that. (And the best part is you can find many ways to charge for this as the budget and the need is definitely there).

This could be by way of a dedicated app for each establishment where the hospital tells the patients to download the app and they can either use it while in the bed (depending obviously on their condition) or once they leave the hospital. Wouldn't be to difficult to private brand an app for each hospital or hospital system either. (Yes, certain age groups would have issues but enough wouldn't to make it worthwhile for the hospital..)

(If you end up doing this let me know...)
If it's running on AWS , isn't it possible to just develop your whole system in AWS and gain the efficiencies?
Compounding the problem, many Silicon Valley cities routinely authorize the construction of huge office parks for tech giants, while explicitly prohibiting the construction of any new housing. The people who work in those offices have to live somewhere, and it often winds up being San Francisco.

Adding insult to injury, they have then made every effort to stall, delay, or cancel the high speed rail segment that was planned between San Francisco and San Jose, leaving only the strained diesel CalTrain and the clogged freeways.

You can't have it all, Silicon Valley; you have to choose. If you want to be a world-class economic engine, you have to accept housing density. Otherwise, we can dig it all up and go back to apple orchards.
i have yet to meet anyone who wants the things they've 'liked' to show up in their friends feeds. it's a purely user-hostile feature that facebook insists upon because it drives their engagement numbers up.
When I went through this I was deeply involved the paperwork and the impression I had was that the whole process is far more nuanced than a computer-generated go/no-go. A minor paperwork error, DHS agent on bad day, or misunderstanding a qualification or another candidate deemed more applicable would do the trick.
but seemed much saner than trying to recreate an actual vintage 1985-era PC-AT.

That was only a 20 year old computer at that point. I can play media from 50 years ago easily. I'd argue that software are more appliances (they do things) than media, and appliances from 20, 30, 40 years ago are still serviceable and working.

Yet we are amazed when a Raspberry PI can serve web pages, despite the fact that it's many factors of more powerful than many circa-1998 web servers.
Nadella's challenge: Saving Microsoft
I remember using it on an Apple II (with a Microsoft Softcard and CP/M 2.2). It was running off a 140K floppy disk.
IAMer dump and load your AWS IAM database into text files, think git
We have a non-tech intern and import.io looks like like a great tool to get him chewing up data. I'm playing with it now. Why didn't it work out for you? Beyond the wrapped browser interface being a little funky lol. (Edit: eugh, selecting data for import is really clunky.)

Ask HN: Anybody got a visual scraping service they like?
What would it take in terms of resources, etc. to get to the point where Common Crawl was doing web scale crawls on a regular basis?
Or you could simply request a permanent deletion: http://deletefacebook.com
Show HN: Traverse City Platmap with Leaflet and D3
I am a researcher at a nonprofit research institution that does both basic and clinical research. When I work with MDs (or MD/PhDs), it is usually to analyze clinical and/or genomic data that they have collected from patients. Sometimes I am roped into building web applications either for data analysis or patient questionnaires. Once in a while, I develop prototype algorithms to aid in diagnosis from e.g., histology images.

The ratio of research to clinical practice for MD(/PhD)s is basically whatever you want it to be. At my institution, it seems more like 70% research / 30% clinical on average, but that's because it's a research institution with a clinic, not a clinic that does some research. Several of the MDs I work with have dropped clinical practice altogether for full-time research as well.

Also at my institution there are researchers and/or clinicians who spend most of their time on building a business. The administration very much encourages this because they get a cut of the IP royalties. In academia, your value is basically proportional to the money you bring in, so these people are actually treated like gods.

Basically, there are a LOT of jobs out there; if you are qualified and productive, you can easily find one that suits your preferences.

(I have no hard data on what specialties are more or less likely to automate. I would expect GPs/family practitioners to be the least likely, but who knows.)
One of the things that keeps me in the startup fray is the desire to earn the privilege of meaningful work.

Not only is most work meaningless to the people who do it, but worse... quite a bit of my work throughout my programming career has never even seen the light of day. I've spent years working on products that get cancelled for stupid reasons, often before even really testing the market for them.

Most of it also feels way beneath my intellectual ability.

It seems to me that not having your time wasted and working at your ability are supreme privileges doled out only to the rich or those "tapped" by the academic establishment.
I took a look, it is great. Here are my 2 cents. I do have one similar task management appl built totally from groud up. It grew out of Spread sheet to have a Web GUI. It does the following. You may consider (if not so far).

* Email tasks that are pending at the end of the day.
* Change priorities
* You can request update, or add items to other's list of TO-DO ( a TODO is a task with same task start and end date).
* Pull up the tasks in a calendar.
* Search tasks.
I deleted it 'cause I'm not comfortable having those details online. An extremely long story short, in our domain we're using the scraped data exactly as the owners intend, albeit via machines instead of people. Consult a lawyer.
Glad you like it! Thanks for kind words.
Unbalanced Access to Tech Media
AMD Seattle 64 bit ARM at “Hot Chips” California
The entire event was volunteer run, and I am actually thankful to the organizers that we got these recordings. There are always things that one can do better the next time...
Focusing on reducing driver overhead is the right way forward. If driver overhead is minimized, so that OpenGL is as thin a layer as possible over the real hardware, then changing the API becomes trivial: you can just wrap OpenGL and expose any API you want on top. Then, if someone writes a popular API wrapper, Khronos can take that API and standardize it as the next version of OpenGL, no design-by-committee required.
My experience: C has a spartan standard library compared to other languages like Python. Even something like getting a scheme going for decent memory management can take a lot of work. I found it to be a good way to get my head round using pointers, now I don't find them so scary.

Just curious, is there a particular language you were thinking of learning?
It's also useful for anything where they might retype the string. (Even if you tell them to copy/paste, some people will still retype.) Not that this necessarily applies to you, but for others considering similar schemes, avoiding I/1/l and such can be really handy.
I recently tried using this for a project but ended up switching. I don't really need the encryption ability. More of an issue though is that there is no way to force that the hashes are "short". If you use their example for hashing the default _id in Mongo, you'll end up with a fairly long hash. The only way I could get them short was to switch to an auto-increment starting at 1 and then they will remain short for as long as the number is small. This presented other issues and in the end I found ShortId [1] which made the whole process much simpler.

- [1] https://github.com/dylang/shortid
This just feels like more proof that Swift will be nominally better than ObjC for the next 5 years, until Apple's APIs are (re?)written to make use of reasonable Swift idioms.

Also, this is the community's periodic reminder that not all ObjC (and soon Swift) is iOS; people still write (and use! imagine that!) OS X apps.
There are some cases where you can't share vertices. For example, flat shading with face normals. So now you need to put that normal data into every vertex and you have to duplicate vertices as well, which is 9x data for a very common use-case.
He was contrasting it with diamond and graphite. I don't think animals produce those either. It seems more like the topic was why nature in general does not produce carbon fiber, not animals specifically.
Very Blade-Runner-esque.

A new life awaits you in the Off-World colonies! A chance to begin again in a golden land of opportunity and adventure!
Good question. My guess would be somewhere between 850-980 after setting up a new account. I topped out around 1100 or so before getting suspended.
Ask HN: Swift or Obj-C for new iOS App?
I am making my first iPhone application and have spent time learning both Swift and Objective-C as well as the necessary iOS frameworks. This question has been asked before, but given that recent X-code 6 Betas are likely more stable and the community around Swift is more developed--do you feel a new-comer's first iOS application should be developed in Swift or Obj-C?
The Future of Mapping: An Interview with Clement Valla
But yet we have phones and computers that are arguably as (un) responsive as their counterparts in 1998. I really wish those issues would be addressed rather than making things flashier with sexy transition animations.
My username at google's email service.
Yes I'm learning Clojure. I've done the Clojure Koans and have been working through problems on 4Clojure.org. I'm getting to the point where I want to start applying what I've learned.
Who's amazed at that? I imagine you'd be hard-pressed to produce someone that is impressed that "a tiny computer can run real computer programs".
https://news.ycombinator.com/item?id=8161711
The key question isn't if it makes a better burger than the average burger cook - it certainly has capabilities to be better in some areas, and recipes/ingredients can be tuned to make the product fit it's strong sides.

But - is it really cheaper than a minimum wage employee?

If the costs are comparable, it might get some traction, but a limited one - for example, I got a burger today in a rather off-peak time, and a single person was easily running the shop at the time, accepting orders, taking cash, and making the food. Such a scenario would offer a grand total savings of $0 for some shops, as someone would still have to be around; but it can replace some labor in shops with larger volume.

Now, if it'd be comparable in price to a restaurant-grade oven or coffee machine, that would mean disruption - but is it?
Setting up Sublime Text 3 for Full Stack Python Development – Real Python
Microsoft transforms bumpy GoPro videos into smooth works of art
The Skully Smart Motorcycle Helmet Sharpens Your Senses On The Road
Currently FXCM (the first broker we've integrated with - more to come) offer forex in the US and also CFDs on indices, treasury and metals elsewhere.
I bet it doesn't have a flag either.
The Sneaky Trick Behind the Explosive Growth of the Kardashian Game
Added it :) Didn't even realize that existed until seeing this thread.
This is very interesting work, regarding your last point, I came across this site which is searchable by employee name or government organization http://canada.landoffree.com/.

I think the salary and expense disclosure is only for Ontario, based on the sunshine list.
Aren't there quite a few free unblockers?
This is really nice! Going to give it a shot with my tiny (but 1/3-remote) team this week.

A few tiny styling issues: the 3D effect modals have sometimes results in the text being blurred on Chrome OSX (see https://www.dropbox.com/s/qna5gj9cm6zbovm/Screenshot%202014-...), and the input field background is a bit too low contrast (I occasionally was confused where to click in a form).
Good idea on the flow chart! Thanks
It says they're moving to a faster release cycle, so maybe you won't have to wait too long.
Personal opinion, but I think this is probably good for professors / post-docs, but a horrible idea for the software community as a whole. Algorithm patenting and closed gardens are probably the worst hinderances to software development that exist right now. It feeds the big boys, who can easily pay, while imposing additional taxes on new folks. Marching Cubes is the classic example of this, but its easy to think of others. Lets say they pattent holographic image recognition / stabilization, or O(N) methods for N-body simulation, or pattern recognition that order of magnitude outstrips SVD or K-means methods. Then everything will have a choice: pay them their pound, use it without paying, or find a functionally similar version that is so different as to not terrify everyone of lawsuit hell. Marching cubes, you only managed to set graphics back by 17 years, meet marching tets.
It's less to do with how underlying algorithm reflects the author and their network's pre-existing preferences and more to do with the entities that buy vast amounts of "promoted content" (political moderates don't buy "likes", except maybe when running for office) and what garners clickthroughs (surprisingly, outside HN "Key Advances in the OpenGL Ecosystem", "A Computational Introduction to Number Theory and Algebra" or even "Conway's Proof Of The Free Will Theorem" isn't really clickbait, but "What Celebrity X Did Next Will Make You Laugh, Cry and Orgasm all at the same time" does very well indeed)

Lowbrow celeb culture trash and political extremism was selling newspapers for decades before the internet.

If there's anything interesting it's the proportion of content political extremist organizations are sharing to win "likes" from relatively sane people that actually isn't political extremism, or even vaguely related to their agenda.
The i7 low power part obviously is the premiere low-power part. And exactly like bitL I would like a nice i7 very-low-TDP, preferably passively cooled desktop machine, grossly overpowered for the vast majority of tasks that people do. Especially given that anything of consequence I do is on big fat machines with quad-Xeons sucking thousands of watts in a rack somewhere.