skysight-inc/HackerNews huggingface dataset:
There are a few different standards, but I don't think it's achieved the level of a W3C style consortium. Google's been working on initiatives, and even Facebook is involved. Appleseed and Cliqset are two products which I believe already implement several protocols, while Diaspora is meant to do the same from what I've read.
sort -u <filename>

would be faster than all that piping
I think this article is really all about how we should subborn our individualism to the rule of the collective than about the question of friendships.

Reality is, the loners and the individualists have friendships, and so they aren't really alone.

But they are not collectivists, and therefore they must be portrayed as having some sort of defect.

You see this in every collectivist society.... as no form of socialism can survive if the people recognize it for what it is and resist.
The widespread use of ad-blockers could kill Facebook, as ads are their main source of revenue.
clickable: http://zferral.com/api-docs
The canonical Hadoop example is counting the number of words in a large text file http://wiki.apache.org/hadoop/WordCount

You could use the same approach to simply take the unique tokens of the output of the final "count".

May be overkill if you you can read the file in less than an hour, but this approach (divide and conquer) may be a good inspiration.
Google Needs to Know What Oracle Really Wants With Android
IPhone/iPad Game source code for $100, interested?
5 months later, Apple released the iPod.

At the time, Apple knew they'd be releasing the iPod, but nobody else did. Given the shift in Apple's product line retail stores made sense.
You may certainly be right, but I wouldn't bet on it: for all the services you mentioned, I always would have been surprised if a person I'd just met were on any one in particular. Now when I hear that somebody's not on Facebook (even (especially?) non-techies), that's what's surprising.
thats what she said.
I guess a line count has to run through the whole file, so obviously your storage is the bottleneck. Let's say your storage manages 100MB/s constantly for the whole 35GB it would take about 6 minutes.

Your 'cat' is not needed by the way, sort takes a filename as argument. And sort can '-u'!

It would be interesting if sort would fail on this (why?) or how long it would take.

edit: Why on earth are you doing this as root?
It's so strange to me that all the technophobic people seem to deal fine with Facebook, whereas I have a really hard time with it. Oh well...
Maybe Google should migrate Android to Python
I'm waiting for the alpha release of APL on carts.
Related from last week, my own Ask HN: http://news.ycombinator.com/item?id=1580464

One of the comments in that thread pointed to this presentation: http://www.slideshare.net/padday/the-real-life-social-networ...

Which points out that the current Facebook does not match the needs of real social networking (ie. multiple networks, different levels of trust, etc.). These things are hard to implement, but I feel that whoever does get it right (along with a killer feature to get users, such as what Photos was for Facebook) will be a winner.

The other feedback that I have from average Facebook users is concern for privacy (ie. a prospective employer finding your party pics), inability to control access easily (ie. your grand parents seeing your photos from a party), an overflow of information and the feeling of a 'fad' wearing out.
Best case: Facebook becomes the AOL of social networking

Worst case: Facebook becomes the MS Exchange of social networking
Is Portugal's Liberal Drug Policy a Model for US?
How would that look from an end-user standpoint? If I'm a person who doesn't know a thing about protocols or technology and uses Facebook a lot, what causes me to start to migrate over to the distributed services? Particularly because one of the great strengths of Facebook is exactly that it's not distributed: all the people you know, in one place. So it seems like to replace one of the core functionalities provided, there has to be some centralized component. And then it's hard for me to see what benefit decentralization is providing for either the user or the service.

An alternative social architecture seems appealing in some ways but I just can't see the [realistic, might actually happen with a significant chunk of the Facebook crowd] vision of how we'd ever get there from here.
'Persuasive' technologies to help you be good
X-Plane!

http://www.x-plane.com/

In active development, unmatched flight model accuracy, huge community of aircraft, scenery, and plugin developers, fly online in VATSIM and IVAO, excellent systems and failure modeling...

You can even buy a USB-Key that makes it FAA-certified!
It would be pretty cool to try to instantiate each one, and then do a sizeof(instance) on it to see how large the memory footprint is.
Bloom filter + memmapped hash table is generally a pretty good way to go.

http://en.wikipedia.org/wiki/Bloom_filter
Yes, it does matter. But once again, it's a matter of degree.

She's talking about low six-figures vs empire rich billions.

She makes no claims if one makes below that.
Did Google punish Oracle by removing them from the search index?
I don't think 'Facebook will fail because other companies have failed' is the right answer.
In an odd way, given what the pundit knew and all things being equal from the opening of the fist store, they probably would have closed down in two years.

BUT, Apple debuted the iPod that october and put it in its retail stores so customers could get a hands on for the device. I always believed (after watching my friends buy iPods, followed by Mac purchases a year or so later) that the iPod was a huge growth driver for the Mac, by making it easy for consumers to get acquainted with the Apple brand.
I think a lot of us (at least who were born before the 90s) had these friendships as kids if their family didn't move much.

I had a only a couple of good friends, but I had them for 10+ years. We'd stay at each other houses, spend time with each other's families. Play, watch movies, get into fights, goof around and so on.

Then eventually, as girlfriends came around, as we went to different universities, different countries, it was hard to keep the same level of friendship. First they become just "acquaintances", then sadly, they become strangers...

The point being, I think a lot of us know what the article means, it is just that we failed to hang on to it.

In America people just move more. It is hard to foster good friendships without face contact. Don't matter what the resolution of your Skype video, it is just not the same.

So, we are stuck. We'll end up spending more time on HN, playing Warcraft, working.

Another sensitive issue is that a higher proportion of HN-ers are probably introverts. Nothing wrong with that. I am one too. That means it is easier by default to be lonely. But in the long run it also hurts.
Mongomatic: A New Ruby MongoDB Library Hits The Scene
I remember reading this article in my high school computer lab.

This is a perfect example of journalistic hubris that has only gotten worse in recent years. Rather than a non-sensational headline Cliff Edwards speaks as if he has a crystal ball. This happens all the time - looking at HN right now I see the sensationalist "RIP Java, 1995-2010" (6 points) and a more humble "Analysts: Oracle vs. Google May Hurt Future of Java as Dev Platform" (28 points). I'm happy that people seem to be up voting the latter.

Kudos for posting this, hopefully it will show that headlines like this tend to make people look stupid in hindsight.
The first thing I'd recommend doing is find the unique entries THEN sort. Unless uniq has vastly better performance on sorted files...
If you're going to do a twss in a humourless forum, please at least make it a funny one!
The title is a bit misleading. The author mentions in passing that he took an at-home ground school course and passed the FAA written exam before flying. Studying to pass that test, and studying to retain some knowledge that can be useful when actually flying, can take more than a week itself.

The other major omission is learning to fly in or through heavily controlled airspace. There are a lot of rules and procedures, especially radio procedures, to learn and practice, and they really need to be practiced with an instructor to get some of them to stick. An instructor can teach to you to communicate effectively with air-traffic control in ways that home-study courses won't. Expecting to learn that in a week and retain it is nuts.
uniq requires a sorted file
Personally, I view pundit status as largely irrelevant. A valid argument is a valid argument whether it came from a pundit or not.

I tend not to trust anyone unless I've heard both sides of the issue. Of course, some people make evaluating their arguments easier by acknowledging the other side. It's the ones who present their side of the story as the only possible way of viewing things that you have to be careful of. A quick skim of this article convinces me that it falls into the latter category.
"Java: Write once, run anywhere and hide from Oracle"
If you had tried to become premium, we would have refused your money
Why you should listen to doers, not talkers. Makers not breakers.

Unfortunately, people who know how to do usually spend their time doing instead of writing in magazines and blogging.
They've been having discussions with various standards groups which I forget the names of at the moment.
If this is just a one off job, I would guess that running sort -u $FILE will take less time than writing some specific software for the purpose.

Of course, if you want to learn hadoop then go for it. But it's probably more practical just to let it run :-)
I've heard that so many times and I think it's really surprising too. I recall being much more confused when I first started, but now I feel quite comfortable with it. I'd love to see a study showing which things on Facebook are initially grasped more easily by the non-techies than the techies. Might cause us to make our UIs differently ("worse" to us) if we're targeting a mass audience?
Though "Facebook will fail because pretty much all companies fail eventually" is a reasonable one.

No-one, looking at Apple in 1998, could have correctly predicted what they would be like a decade down the road. It's not unreasonable to suppose that guesses at what the big companies in the industry will look like a decade from now will often be wildly incorrect, no matter how obvious things seem.
Apart from myself, I don't know anyone who uses ad-blockers. Almost everyone I know uses Facebook, including housewives who don't even have a computer in their homes.
I don't doubt that a lot of people use ad-blockers, but I can't imagine there's a big enough overlap between ad-blockers and Facebook users to bring down Facebook.
I think it was the iPod that saved Apple from dying, but I honestly think it was Windows Vista that propelled them to the limelight (whether the criticism of Vista was valid or not).
With all due respect I think you miss the point of punditry.

The point of good punditry is not to convince someone you are right and have them follow you blindly. It’s to take an insight that you have and transfer that insight to other people by explaining the thought process that led you to that insight. In doing that a good pundit expects the people who receive their insight to then evaluate it on an on-going basis and those people need to use their own knowledge to judge the original insight’s accuracy.

So it isn’t about having the reader say “that guy is right” as much as it’s giving the reader something to think about.

To use this as an example I think this guy was right when he wrote the piece. He had no way of foreseeing the iPod/iPhone/iPad and their halo effect. But as a reader in 2001 people should have taken this piece with all the negatives it lists and weighed it against Apple’s strengthening product line. Then used all that knowledge to form a new opinion based on it.
Yes, they are. Java is _used_ in Android SDK only. You use java-based tools to develop and build Android applications and then convert them into Dalvik applications.

On devices, there is only Dalvik, no Java.
How to Get 5 Million People to Read Your Website - Matthew Inman, Oatmeal
It seems as if High School students are getting bored with Facebook as a whole, and it's becoming less trendy as their parents and grandparents join. If someone could come up with a site that would be more engaging somehow while retaining the exclusivity of the early-ish stages of Facebook, I believe they could potentially do very well.
You sound a lot like me, 8 years ago. Except I hadn't given up on my family yet. It took a lot of time and a lot of thinking to realize that they were toxic, and I had to learn to love myself enough to stop subjecting myself to their toxicity just for the sake of "family."

> P.S. - I can tell that you've been through a lot of pain yourself. Is there any way I can help you to deal with it?

That's such a sweet thing to offer. I'm good now, though - but maybe I can help you.

It's hard to find people in person for talking about this stuff -- they don't understand, or they're in it just like you, and can't help you OUT of it.

But what really changed my life was two books, and a third I read recently that might help you too.

1. When Things Fall Apart by Pema Chodron. It's nontheistic buddhism, for crises. Get the audio book. It's the next best thing to having a loving parent. Listen with your heart, and do the meditations (including the lo jong - no matter how crazy it sounds). You're probably cynical about this stuff - because life has taught you to be cynical, but this is not something to be cynical about. This book has made the biggest difference in my life - the change you wouldn't believe.

I'm not a fruity religious type, or a touchy-feely new age type, but what I learned from this book -- to not run away from my emotions, to not distract myself, to embrace it, and go through the fear, and to have empathy for everyone else (and, then, myself), to accept that reality is reality no matter how much I might like it to be otherwise -- has been life-altering in the biggest way possible.

If you met me today, you'd see an extremely positive, happy, confident person, who is responsible for her own choices, and who doesn't let shit get to her -- because she knows who and what she is. That is the exact opposite of who you would have seen 7 years ago. I can attribute that almost entirely to what's in this book, and my practice of it.

2. Toxic Parents. In case there was any doubt, this book will be your best buddy telling you that you made the right choice, and it's okay.

3. Drama of the Gifted Child. This one's a bit weird, but you know what? It's good and true.

You don't have to stay the way you are now; you can be happy, and you can be sad, but you will be able to be whole. Seriously. I'm proof.
I have it on my favorites, love it!, best explanation ever!
They hired many very good programmers. That's just crazy to say otherwise. But if you lead programmers into a dessert you can't blame the troops when you die of thirst.
The Most Expensive Home Theatre in the World
We present you a cinema of your dreams. Who wouldn’t like to have such a big screen theater in their house?
Get a few EC2 instances and use MapReduce(hadoop)
We used this method in high school (called it burnout then) and it never ceased to fail that a pretty girl would walk in when you got down to the bar and could barely lift it!
If you're blocking access to your site from specific IPs or ranges of IPs, then you fail at the internet.

If you have "IT People" who advocate such a policy, fire them on the spot, and then give them two hours to think about what they have said and plead for their job back.

I find it ironic that tor is routing around the internet damage from censorship, only to run into internet censorship from damaged people.
Unix sort already does a merge sort (search for it), which is your default solution if you just want to sort individual lines based on a field:

http://vkundeti.blogspot.com/2008/03/tech-algorithmic-detail...

Thus, sort -u <filename> is your go-to for simple jobs. (Note that you'll need to have enough extra disk space to hold all of the temporary files.)

If you need to do something more sophisticated (e.g. joining lines from a web server log into sessions, then sorting sessions), you can still use divide-and-conquer, but you have to be smarter. Divide the file into N parts based on some logic (i.e. divide into files based on session ID), then sort the lines in each individually, then merge the results back together.

This is what map/reduce frameworks are made to do, of course, but something like Hadoop may be overkill unless you plan to do this type of thing often.
Finding duplicates in unsorted data is pretty time consuming.
Off topic, but I have to ask: why is floating point division the appropriate operator to use here? Or is this some other kind of notation not aimed at Ocaml programmers?
I've never used it but there's a -S flag for sort that allows you to specify buffer size. So maybe look into increasing that as large as you can.
Parents like that it's widespread because they can find old friends, but the site becomes less trendy when High School and College student's parents are using it. They need to pick either being widespread or being exclusive, but they can't be both.
I love F#.

Same code in F# (one line, of course!)

let currentNetTypes = System.AppDomain.CurrentDomain.GetAssemblies() |>
Seq.filter(fun x->x.FullName.Contains("Version=4.0.0.0")) |>
Seq.map(fun x->x.GetExportedTypes()) |> Seq.concat |>
Seq.sortBy(fun x->x.Name.Length);;

To me this reads easier, but I admit to a biased opinion :)
You might check out bashreduce: http://github.com/erikfrey/bashreduce
> multiple networks, different levels of trust

I keep waiting for either Facebook to solve this problem, or for someone else to solve it. Google comes closest with their social network concept. On the one hand, I hope they pull it off. On the other, I try to minimize giving Google much more information about me than they already collect, so I'm not sure how comfortable I'll be on their social network. I suppose I'm hoping their success pushes Facebook to implement a similar concept.
APL contains some genuinely new ideas worth learning. I'd suggest looking into J, as a more accessible and modern APL dialect. (K / Q is also really, really cool, but J is less commercially restricted.)

I was just curious because it was so different, but the thing that really convinced me was going through some of the J "labs" (a bunch of math lessons included with J, that incidentally teach it along the way) and spending a couple hours picking apart this code (http://nsl.com/papers/origins.htm) to get into the APL frame of mind.

I would suggest not getting too hung up on the syntax, and instead focusing on the pros and cons of doing everything over collections by default, with implicit looping. "Thinking big", as Henry Rich calls it in _J for C Programmers_ (also a good resource). The ideogram-based syntax is an historical thing, and getting fixated on that is like thinking the only novel thing about Lisp is all the parens.

See also: "The World's Most Mind-Bending Language Has the Best Development Environment" (http://prog21.dadgum.com/48.html)
It's not just that you can't fly into busy airspace, it's that you can't fly into airspace where you need to talk to an air traffic controller for VFR flight (class B-D).

If you live in or near a big city chances are that the airports out of which you will want to fly will be close to airspace into which you cannot fly. It a hassle if you have to drive out to the boonies in order to start your flying trip or if you have to take the long way around a large block of busy airspace.
Objective C has a lot of run time overhead due to its "message passing" paradigm. So any time you need to use a Cocoa API, it's pretty slow.

I hear over and over from experienced Obj-C guys that it's almost never message passing that's your bottleneck. (Just like how it's almost never your compiler that's wrong, it's almost always that you really did forget a semicolon.) Keep in mind that most of Mac OS X is Objective-C and it seems competitive performance-wise. I believe the Objective-C runtime even has some caching to make the performance of frequently-sent messages almost on the order of a function call.

Asm guys used to say the same thing about new-fangled languages like C: all that code that pushes/pops the function call stack was a huge slowdown. In practice, you just don't ever spend your time in function call stack manipulation code.
Eczema relief using 5 methods to live disease free lfie | Relief
Lack of revenue. It's killed more social networks than competition ever has.
The Week In Review: Duke Nukem Finally?
I didn't move the whole time I was growing up and had the same friends the whole first part of my life before college and a simliar thing is happening right now with us going down the path of becoming strangers. However, I do think that sometimes a person does need to go out make new friends. I just don't have anything in common with my old friends from high school. For better or worse they are going down a different path. I really resonate with this article though and thoroughly believe I need close friendships to be happy, but I do not think that one should hold onto old friendships, especially ones from high school that you were likely forced into, at all costs.
Shove your data into postgresql or mysql with one of their bulk import utilities and let the db handle the hard work of indexing and filtering. That's what they are designed to do. You can then slice and dice the data to your hearts content. Make sure you have plenty of disk space as indexing you data will take up a good amount of space as will the overhead of storing it in the db in the first place.
How to predict a successful Google product (Hint: it’s the name)
As well as scheme.
In this context, /. refers to http://slashdot.org, the grandaddy of all hacker-oriented sites.
Do you know what percent of records you expect to be unique?

Also, what do you want to do with the unique records? That might effect what initial processing method is best for your goal.
What's a twss?
We are moving into virtual world (Internet, TV, etc.) there is simply not much need for physical interaction.
Facebook will either be killed by something fundamentally better - or it will be regulated to death.

Social networking is emerging as far too important a communication medium for one company to be allowed to dominate. It may take a few yeas for regulators to catch up, but if Facebook isn't unseated by something better and more open, it will get broken up by government or at the very least forced to inter-operate openly with its competitors.

I'm betting something better will come around before the government steps in, but either way I'd say Facebook will be taken down a peg or three within the next few years.
I still think (or hope) that something that better reflects a person's multiple networks can emerge and gain widespread use. For instance, I don't have all my coworkers on my Facebook friends list because I don't like to mix them in with my personal friends or family. That said, I don't use Facebook's wall all that much already because what I like to share with my family and what I like to share with friends are completely separate as well.
Karawang Info
Citizen Journalisme of Karawang
I'd have to say intercal's COMEFROM: http://en.wikipedia.org/wiki/COME_FROM
Biggest Clock in The World is in Makkah (Mecca) Saudi Arabia
Saudi Arabia will test the world’s largest clock in the holy city of Mecca (Mekkah / Makkah) during Ramadan 2010. Clock tower with four sides will be installed on top of a big skyscraper has a total height of approximately 600 meters or the second highest in the world after the Burj al-Khalifa in Dubai.
You should see the people at my company do it by hand with pieces of paper! With n approaching 20,000 sometimes. (Yes, we're working on automating this even as we speak.)