skysight-inc/HackerNews huggingface dataset:
What change does the graph illustrate? Each 10 year period is double what it was before. It's not a linear function but it wasn't in the first place. Thanks.
How I Want to Be Remembered by Jack Handey
http://www.newyorker.com/humor/2008/03/31/080331sh_shouts_ha...
I'm sorry this post made it to the front page, actually, and it's part of the problem I'm talking about. There's nothing wrong with submitting bad stories, but they should never get 30 points. I find it a bit unfortunate that after commenting on average more than once per day, this submission is the one that achieved the highest karma, just beating out http://news.ycombinator.com/item?id=107729
Very cleverly done.

Unfortunately the output is terrible - both unusable in your code and unusable to learn from.
I'm going to vote for Nutch too. Have heard good things.

Also, if someone on this list wants to work on a cool web spidering project (probably using Nutch), send me a message. I'm looking for someone.
Exciting times in Rails land (review of some 2.1 features)
Yea, seemed a little fishy. OP needs to lay off the Omega 3.
Dismissing a news article based on association is, for me, not a very good method. Following the link that it provides is a useful method. In this case to the actual paper in question:

http://www.brain-surgery.us/mobph.pdf

If the Register ran an article on a startup company, would that invalidate what is really presented?

"Mobile Operator Association" sends up red flags for conflict of interest in this argument. Not that I dismiss, but would hope for a more independent assessment.

"the science suggests" is, to me, an appeal to the authority of a select number of papers and implies that the paper in question is not "science". I do not see a conflict on interest or false scientific reasoning in the paper in question. If anything, this paper could be considered "newer science with more data" with its 2008 publish date and recently cited studies.

The author directly contests the assumption that the long wavelengths used by cellphones result in less risk to the users. He also mentions a report of an office in the top floor being closed due to the "seventh working in as many years" being diagnosed with brain cancer that he attributes to a cell phone tower directly above the office.
When you put a device that contains chemical potential energy over your legs, you're taking a risk that the energy is released and interacts with your legs/groin. The longer and more often you put the device over your legs, the more opportunities there are for the interaction.

Everything else is just a mathematical probability. Let's stop wearing pants.
Transcript: http://www.dpmms.cam.ac.uk/~wtg10/importance.pdf
i see this is a comparison of some kind, but it's not very clear. in the movie downloading case, who is being hurt when they wouldn't have bought it anyway? or if no one, why is it bad?
Why is this downmodded and the parent downmodded? I was making fun of the parent. If the parent is bad, I was right. It should go one way or the other.
"DayJet performed extensive econometric modeling involving detailed census data to build geographically-targeted service roll-out business plans that accounted for diverse consumer demand and logistics scenarios to demonstrate how regional air transportation could be successfully disaggregated."

lol, Do you think they used this:

http://dack.com/web/bullshit.html
I'm sorry to seem melodramatic, but I didn't feel like I was only speaking for myself, and I hadn't yet seen a post like this. For a while I thought others were overreacting when they were saying that HN was changing too much, but lately I think I know what they're talking about. Several times I've clicked through and read about half of a story only to find that it was a waste of my time, and judging by the names on the comments, most of the story's score is coming from new members.

I'm not saying the community isn't good for a lot of people, I'm just saying that it's changed enough that it's no longer what I came here for. I would apologize for making this submission, but it seems that enough people agreed with me that it did make it to the front page.
How about setting up a dedicated place for the discussion of these things, so that they need never cross the front page again, and we can discuss them all we want without feeling bad about it?

Feature requests might work, maybe not. In any case, it would also be necessary to fairly ruthlessly kill navel-gazing articles on the front page, or at least move them to navelgazing.news.yc.com...
Dean Kamen's robotic "Luke" arm - How long before this can perform martial arts?
Correct, but your statement assumes that there is some other field that is better attuned to "brain cancer and tissue interaction with EMF over prolonged periods." I dont know.

It does mean that he is most probably aware of trends in brain cancer, such as certain increases. He is also in the front lines in dealing with the possible results of cell phone EMF damage. He also could possibly have very good communication with other brain surgeons dealing with similar problems.
Omega-3 fatty acids have many health benefits but overeating fish these days will lead to excessive mercury in your system. Also many of the Omega-3 pills are of low quality. Note that men are probably best off taking the oil with more DHA than EPA, while the inverse applies to women. That said, if most of your protein is coming from seafood, you should be well ahead of the health curve.

I find that frequent eating helps me survive extended programming sessions. Anecdotally, meaty pasta seems to be one of the best foods when I need to pull an all nighter.
Thanks. Still would like to use Python to be honest (any python suggestions?), but I'll give this a go. Going to do some more research and might post back findings if anyone would be interested in critiquing them. I'm creating this startup from scratch so if there is anyone interested in the crawler side of things I'd be happy to chat either about collaboration or sharing ideas.
I concur with the Nutch vote; but more specifically, take a look at the crawler code written in the src trunk for use with Hadoop. That is probably a good place to start. Also worth a look is Heritrix (crawler for archive.org). http://sourceforge.net/projects/archive-crawler
Sadly, this too is written in Java.

The only Python one I am aware of for which code is available is:
http://sourceforge.net/projects/ruya/

Edit: You might also want to take a look at http://wiki.apache.org/hadoop/AmazonEC2

Edit2: Polybot is another Python based crawler, but no code. However, the paper has some interesting ideas:

Design and Implementation of a High-Performance Distributed Web Crawler. V. Shkapenyuk and T. Suel. IEEE International Conference on Data Engineering, February 2002. http://cis.poly.edu/westlab/polybot/
I looked, but this is not distributed, so currently (whilst good) its limited to one server
Before the 1973 coup, Chile tried to find the right software for socialism
The most important advice is to constantly challenge your mind, but diet and sleep are extremely important in making your body and brain operate efficiently.
State of Prolog
I'll give some from the front page and the "More" page that I would have rather not seen on the front page:

http://news.ycombinator.com/item?id=150070 - highly-debatable claims presented as science

http://news.ycombinator.com/item?id=149708 - decent, but 30 points? Besides, the comments thread looks identical to something I could read on Reddit.

http://news.ycombinator.com/item?id=149467 - who cares?

http://news.ycombinator.com/item?id=149482 - already covered at least a dozen times. I would suggest beginning with a search engine...

http://news.ycombinator.com/item?id=149618 - ancient news

http://news.ycombinator.com/item?id=149196 - old news

http://news.ycombinator.com/item?id=149091 - one of the worst of hundreds of front page submissions regarding the mortgage crisis

I do understand that this is only my viewpoint and that others may have liked at least some of these submissions, but this is coming from someone who was never a member of a link aggregating site. I started reading Reddit towards the end of 2007, but that didn't last long, as I quickly tired of seeing the same headlines and C-C-C-COMBO-BREAKER!! posts, so I left.

Hacker News served its purpose for me for quite a while: giving me the ideas and motivation to create my own startup. I also like some of the interesting science and hacker-related articles. Lately, though, it just hasn't been what I've been looking for.
metahackernews.com looks to be available.
Good response. We've created a basic crawler in Python, but are looking for something more powerful too. Heritrix above looks good
If someone releases something under the BSD license, how do can you trust him/her? I'm especially suspicious of something like an operating system with its attendant functionality, such as the TCP/IP stack. This takes enormous effort to put together. If you are told that it's under the BSD license, then how do you know that this is true and that it would be safe to use it in your code for commercial purposes?
Nicely done, and thank you as always for the thought provoking read. Absolutely love your essays.

Enjoy your weekend!

Another thing, along the same web conversation line is the dodge. I have found this very curious and am always fascinated by it.

Essentially, there is a conversation going on. Somebody has posted up something they characterize as factual, the truth, majority view, correct, etc... Somebody else then has their say and after some banter back and forth, it's clear one or the other just isn't correct.

This does not mean the other parties are correct either, just that we have one that is known not to be correct in the conversation.

At this point, we should then expect to see some acceptance, but we don't. That's the dodge right there. A successful dodge means being able to bring that battered point up, in a given venue, over and over and over again.

Frustrating, and also highly entertaining, if you like seeing new forms manifest!

Some that I've seen play out: (some are likely subsets or minor variations of others -->this is just a list compiled over a coupla years talking politics with a close circle of people)

1. Subject change

2. Claim of too many opponents

3. Claim to either be a member of, or opponents of discussion being members of club, with said membership somehow required to complete what is otherwise a non-rational conclusion.

4. Silence

5. Claim of overall subject and or implications of subject complexity being greater than scope of discussion

6. Invoke false comparison

7. Transform rational argument into emotional one

8. The Bible says...

9. Invoke redefinition of common words

10. Invoke slippery label to redefine winners primary point at hand

11. Claim of invalid higher authority / Claim of no high authority

12. Obsfucation, where some degree of acceptance is seen, but wrapped in difficult to reference terms

13. "I forgot, could you remind me again?", used to force other parties to tire of the subject and move on.

14. Necessary party unavailable for comment.

Some of these could be valid, depending on the context. I've got those listed here, with the assumption that it's really a dodge and that context is not at issue.
In the GPL case, who is being hurt, when the (a) the GPL violator wouldn't have contributed to the project anyways, and (b) the GPL provider wouldn't have collected any money?

It's not a vague comparison; it's a cliche, about copyvio apologists tending to want it both ways.
>(I hear a buzz on all my computer speakers)

Was your previous phone GSM-based? That buzz is not specific to iPhones. All GSM phones have it (to a certain degree). Search for 'gsm buzz'.
The Register? The posted article is from the Independent on Sunday.
All PDA phones emit relatively high amounts of radiation.

You can find the radiation level of your phone at the link below. Enter the FCC ID that is located on your phone. The lower SAR, the better. Anything over 1.0 is relatively high and 1.6 is the maximum legal limit. I think the iPhone is around 1.0-1.1.

https://fjallfoss.fcc.gov/oetcf/eas/reports/GenericSearch.cf...
Wouldnt amount of water be more analogous to current, than voltage?
Just saw the DH2.5 - Denial comment. That's where the list of dodges is aimed at.
I'm not upset at all. I would also agree that in some ways, it has gotten better in the last year. I think that shortly after the switch to Hacker News, it was a great mix of startup, programming, and general interest news, but I think that balance has teetered a bit towards general interest and meta-discussion.

1. Thanks, I appreciate the reassurance that I have been a decent member of the community.

2-7. The problem is that I don't use other link-aggregating or site-stumbling sites, and I don't have any RSS subscriptions. I'm just not that kind of Internet user. For me, Hacker News was my portal to all the things I'm interested in, but lately that has shifted too much for my tastes. If nearly 400 comments, over 400 votes on submissions, and countless reads through stories and entire comment threads are not enough for my complaints to carry weight, then I'm sorry. I don't believe that you and I are the same type of Hacker News user.

8. When I have some time to burn and I've perused the front page, I do visit the "New" page.

9. This wasn't intended to be a whining post, and if you would like to browse my old comments and point out where I have whined in the past, I will certainly consider my behavior. This submission was merely to attempt to gauge how some of the older users feel about the shift in purpose of Hacker News (what new user would vote it up?), and to see if there would be any point where others would consider leaving, as I plan to.

No, I'm not really offended, so don't worry about it. And that was the main purpose of this submission: for others to share how they felt about what exactly would be the point where this site has changed too much. Either some changes are going to be made, or this site will follow the trajectory that so many other great discussion-based communities have followed.
My point wasn't "guilty by association", though given the reputation of the original source, I can see how it would be read that way.

My point is that progress in the medical sciences isn't charted through "exclusive reports" to newspapers. As anyone who clicks through the link you've helpfully provided will note, this is a story based on a white paper, not a peer-reviewed journal article.

I'm sorry, I don't know offhand what the "DH" code is for the error this newspaper article made. I suspect it's a high number. 23, perhaps.
What learning Lisp taught me about other languages
>>Why would they go to some country where they don't speak the language, know the laws, or have many contacts?

For the same reason companies expand to BRIC economies -- to make more money. Every growing business has their hands full with operations, however the challenge is to perform the balancing act of scaling up the business through expansion while keeping the existing business in order. I wouldn't call it insane though, risky is a more appropriate term imo. Nonetheless it is a sort of risk that many businesses manage to overcome successfully. I think the real question is whether pg manages YC as a business or as a hobby.
The current body of medical literature is at best ambivalent about the risk of long-term exposure. That's not an appeal to authority, that's just checking on the state of the literature.

Science is always open to revision, and so I don't think it's impossible that this is true (indeed, I use a handsfree religiously to minimize my potential risk). But there isn't any hard evidence, and one study does should not a panic take, especially given the profound and subjectively positive social implications of mobile computing and mobile communication.

The article has nothing new for me, except sensationalist rhetoric and carefully worded (and carelessly paraphrased) "Mobile Operator Association points to these studies", when really that's the gross body of the current medical literature. It's not cherry-picked literature, the bulk of the literature suggests it is not true. That's not a conflict of interest, that's just pointing to the independently funded science.

As for the story about the top floor of the tower, why would only their heads be getting cancer if they're being universally exposed? The plural of anecdote is anecdotes, not data.

Whenever we have a somewhat difficult to reproduce condition with less-than-certain explanation in terms of the physical method, I demand more solid evidence before taking drastic action. I also think that news outlets like the register should take more responsibility and demand more evidence before pulling out the dramatic horn kicks and warming up a headline like this one.

People have been expressing fear over EMF for a long time now, and in general these fears have been dismissed and unsupported by scientific examination. Until I see some hard evidence, I consider a cellphone is about as harmful as a microwave. That is to say, there are small potential risks (particularly with damaged units), but I'm more likely to be struck by a car while commuting to work.
I wish it was that easy to get a work visa to come into US :(. As PG said in one of the essays about why Silicon Valley is special, immigration is one area where it falls flat.

Getting a work visa is next to impossible these days. You apply by Apr 1 and you MAY know in OCT if your visa was approved or not - it is a lottery system.

6 months is a lifetime for a YC startup.

IMHO, there must be another solution, like setting up a YC hub in Vancouver or something :)
I don't think it's stored in a cookie... I'm pretty sure it's just your browser's history keeping track for you.
In some countries there are laws that don't facilitate the formation of seed funding companies. e.g. A funding company can't simply take 5% equity for so much money in a brand-new company. Something like that. One reason among many.
I think that the first 100 comment threads discussing strategies to combat noise were worth my attention. While the ideas have gone nowhere, though, and the discussions have kept coming, it has gotten a bit old.
exactly - the brightness is what costs, and a LED for a projector needs to be pretty bright.
Sorry, my bad. I tend to confuse one pseudo-science endorsing rag with another. I corrected the article, confirmed that the Independent also ran the story.

Thanks for the time you took to offer the correction!
This reminded me of an episode of 'From Our Own Correspondent' I heard a few years ago. When you move house in Switzerland, apparently, you get a visit from the 'hygiene inspector':

http://news.bbc.co.uk/2/hi/programmes/from_our_own_correspon...
That is a great question. I just might start my first real blog and make the first post be something along these lines, not just in regards to Hacker News, but other time-consuming activities as well.
The video game industry is built on C++.
My prosper experience:

I've had 9 loans, 1 of which has defaulted already (all originated summer of 06). I've got another loan that is constantly late and a couple others that are paid off already.

I was going to use this as a source of growth on funds that I had set aside for my son's college, but after a while all the listings were crap so I pulled my money out (that I could).

A friend of mine has about 300 loans now and has about a 10% default rate. He only buys A-B paper. He has a hard time manually finding "good" loans now and is starting to slow his investment.

I think that Prosper needs to look at their loan terms (is it still only 3 years?) and the high number of defaults in order to get better lending sources.
BeautifulSoup is not foolproof, meaning it does not always even approximate the way a browser would parse the HTML. One important failure is that it fails to recognize when HTML tags are inside of JavaScript strings (and so should not be considered). Whether this is important or not depends on your application.
Most companies try to grow in their home country prior to expanding abroad, and YC has yet to do any of that at all, as far as I know. Also, not all businesses try and grow all over the world in a short period of time. Some realize their limitations and try and concentrate on doing their niche really well, leaving other places to other people.

> I think the real question is whether pg manages YC as a business or as a hobby.

My offhand guess would be "as an experiment". Past comments have indicated that they do want to come out ahead with it, to justify its continued existence, but that it's too new to just start franchising it... it's not exactly like McDonalds in terms of their business model!

http://paulgraham.com/ycombinator.html
yeah, i agree. if setting up seed funding is too complicated, starting regular meets or something like startup school will still be very beneficial. In fact, that will help more than hard cash.
I would still choose this site over the alternatives. Before Startup News, I never visited news aggregating sites (except through Google searches), and it will probably be the same after.
There is an inflection point in 1995 when reserve requirements were dramatically lowered, permanently as it turned out.
You haven't answered my question.
This Blog is Rated-X
I am exposing some facts about spam that most consider a bad thing. Not the X-Rated you were looking for, sorry. Maybe I will show you a little skin as you read on.

Spam is not as bad as you think. I know it’s annoying to manage spam email, but that’s why we have tools to manage it for us. The good thing about spam is something that many people don’t realize. Web related spam is an awesome indicator that your site is becoming recognized! Yes, can you believe it? You’re among many sites that stand out in front. It could be bots or human type spam. Whatever the type, you’re still getting web traffic.

Webmasters and web authors will tell you that spam is not quality web traffic, and they’re right. It’s just traffic and an indication that your site is getting noticed and found. I would rather have spam on my site than a high bounce rate%. Why? High bounce rates are quick visits, more like hit and run accidents. Spam at times can increase your time on site average, whether it’s a bot or human.

Example Spam Statistics:

Average Spam daily = 30

Average Spam Monthly = 900

Average page views Monthly = 1840

As you can see this web traffic is not bad, but a good indicator that your web site is being found.

What can cause spam traffic?

There’s many answers here, but I will speak about one. The way spam finds it’s way to your web site is from inbound links, also known as backlinks. The more backlinks you have, the more exposure your site will have. This is a good thing.

You can get more info on backlinks by clicking on this link http://blogtbt.com/wordpress/pagerank-and-backlinks-oh-my/

Spam is not all bad, but annoying at times. You need to look at the big picture and realize that you’re doing something right to get noticed.
Digg comments about how to disagree, slightly different from News.YC