skysight-inc/HackerNews huggingface dataset:
It's usually more efficient to use distcp to copy the data to ephemeral storage:

hadoop distcp s3n://<id>:<secret>@bucket/input_data /input_data
hadoop jar myjar.jar org.myorganization.module.jobname /input_data /output_data
....other steps....
hadoop distcp /output_data/ s3n://<id>:<secret>@bucket/output_data

Doing it this way, all spills/intermediate steps live on (very fast) ephemeral storage. Additionally, the distcp command is pretty fast since it is a distributed copy - all nodes will saturate their connections on the copy.

Also, the authors advice to "make processing faster, decompress it, split it in to lots of smaller files" is probably not the optimal way to do it. Hadoop isn't great at storing lots of small files.

http://www.cloudera.com/blog/2009/02/the-small-files-problem...

I'm guessing that he gets a speedup because he is using raw s3:// as a filesystem rather than s3n://. If I understand the hadoop s3 connection, this means that hadoop will use a single connection to pull a single file - so a single 10GB file will max out a single node's network connection and the rest will go unused. Splitting the file up into 1 million line blocks lets multiple nodes pull simultaneously. But storing files using the s3n:// allows this anyway, and results in large files being split into 64mb blocks. It also allows you to store files larger than 5GB, or whatever the s3 file size limit is.

Now, a question I'd really like to know the answer to. What advantage is there to using EMR vs Whirr?

[edit: adjusted language - upon rereading it seemed harsh,which was not my intent. ]
IMac (Thunderbolt) outperforms most other desktops but one which is $200 cheaper
The fight was over control.

The trademark issue meant that it didn't matter what the community wanted, it could be 1,000,000 : 1 against, if the 1 was Oracle, they could stop (or force) anything they wanted.

Commercial interests had wanted to move to Eclipse before. Most developers see the Eclipse process as overly bureaucratic and unfriendly - and therefore were -1 on that.

The whole process has been an attempt to try to force-feed the development community into eclipse, whether they like it or not. Because they hold the trademark, they have the control.

Well, thing is, eclipse was disliked before, and it is disliked now. I seriously doubt any devs will be considering it.
NumPy in PyPy - status and roadmap
"... companies like Microsoft and Sun ..."

http://www.sun.com/
I agree, that's why I don't work in those types of organizations and don't recommend it.

My rule of thumb is this: if technology isn't a major component of what your company "does," you will never be treated with respect by your company as someone in technology.
I thought the long term trend has shown for a long time the size of chip components converging to the size of an atom by the late teens. A little hard to imagine how physical transistors can get smaller than that. Possibly qubits could built from smaller particles: photons, electrons
Aren't all fundamental physical constants like that - they are essentially meaningless values plugged into theories to make them align with observations?
The "x-figure" notation is essentially logarithmic with an offset: "6-figures" = 10^5. Therefore, "mid-six-figures" should be roughly 10^5.5 or $316227.
The concept of attaching notes/summary/gifts/messages/art/etc as a layer on top of websites has been done endlessly.

It seems like the main problem is the network effect. Why should I use yet another service if there's not enough scale for it to be effective?
Bin Laden was a bad person, sure. But there are a lot of bad people in this world. Why was he specifically killed, and why is he singular in his ability to motivate Muslims to terrorism? Why is he going to be any less effective in his death than in his life? I just don't see the value in his death. Terrorists in Afghanistan, Pakistan, and Iraq have no shortage of leaders, or of new recruits. The only real purpose I see in his death is making a new martyr - much more powerful than the hunted man who died.
> I just don't understand why you need to be quiet about it.

Because it's too easy to let your days devolve into storms of bitching and moaning in which nothing of value is accomplished.
It may depend on the journal. I've definitely downloaded PDFs from Web of Knowledge, with the Cell journals seeming to use it instead of having their own archival facilities facing the web.
I've found that if I listen to podcasts, I tend to flit around between coding, HN, facebook, flickr, etc. If I listen to music, I can stay focused for hours. So my recent hack was to create an rdio station combining the Tron soundtrack, it's remix, and (oddly) upbeat Lady Gaga tracks. I happened to get that mix by accident a few weeks ago at a coffee shop and was startled at how much time passed, and how much I got done.

I think the more general point is to identify the mental triggers that put you into a zone, and work on refining ways to put you there on demand.
Nah, this is quite unprofessional spam. ;-)
Crud.

I guess that means that I should abandon any sourdough bread making, since it uses millenia-old technology....
I permanently blocked all from my laptop and only surf from my iPad now. Works great for me.
I don't think this is a necessary point to make. Of course they have something to gain. If I did something that made that kid as happy as he is in that video, I'd sure be better for it on a bunch of levels. Marketing bullshit aside, doing things like this is just plain good for all involved.

Further, I agree with the other commenter.. I don't really see the entrepreneurial spirit is ripe with 'out for yourself' mentality. Sure.. people are hustling, but most of the entrepreneurs I know are doing it by focusing on creating real value for people. I'm pretty thrilled with the brand of hustle I've been seeing in the last few years in this respect.
The Last Post
It actually works.

It's what people normally use to calculate citation indexes and all other scienceometrics and I really don't see how it has anything to do with a wiki system.
Show HN: What we did during the royal wedding - Android Unicode Keyboard
Last weekend we were stuck indoors unable to go celebrate the royal wedding, so with a long weekend to spare we decided to do a little side project to solve a problem on Android that's been bugging us.

Its a supplementary IME for Android devices that lets you input any Unicode character with a simple hex interface, recently used characters and browse and search capability.

We've blogged about how it was built here:

http://www.cwd.co.uk/2011/05/04/unicode-keyboard-released/

And its on the market here:

https://market.android.com/details?id=uk.co.cwd.unicodekeyboard

Let us know what you think.

What did you do during the wedding?
China Establishes New Internet Regulator
So it sounds from the article like the people who initiated the Jenkins fork already declined EF administration. That means it is unlikely that it would remove the fork. Also, something the article didn't explicitly call out.. Is the Hudson trademark also part of this proposal or does that stay in Oracle's sole control?
The recording industry's bigger problem has been that for the past twenty years or more, it seems to have gotten the idea that its consumer base consists almost entirely of twelve year old girls, and at the same time the bigger companies have been absorbing or destroying all the smaller labels that actually produce the innovative music that people want to hear and would willingly pay for. You know there's a problem when kids are saying that all the new music is garbage and listen to music that was made before their parents were born. That's not to say good music is not out there, it's just really hard to find sometimes.

I have heard a few owners of small labels say that they actually welcome the emerging distribution models, and if some of the music is downloaded or whatever (within reason) then that's cool too. That means that they are reaching an audience that is increasingly hard to reach, and so have the potential to find revenue by less traditional means. So while the days of super-bands are probably over, small labels and bands can actually make a decent living now doing what they love to do - producing good music that people like.

Consider this: most bands make their money through live performances and licensing. When you look at the total record sale for a band like the Rolling Stones versus their overall net worth, it becomes pretty apparent that, considering their percentage of the sales, they could not have made all their money by moving boxes full of vinyl, CDs or whatever. The same is true for just about every artist you've ever heard of.

To the irritation of the big labels, though, these new models are quickly cutting out the guys in the middle who don't really care about the quality of the music. They don't seem to get the idea that if they made an effort to produce a product that people want, then (surprise) people will but it. Instead of putting a pile of money into five "artists" that suck, spread it out to smaller projects that collectively do well - like the small labels do it.

I actually heard one of these boring suits discussing the downloading problem on a business program, and he said that if illegal downloading continued on its present course then they would no longer be able to finance all the mega-expensive music videos. All I can say to that is that we should all fire up BitTorrent and make it happen.
There really is no reason to earn an undergraduate degree in physics (as I learned the hard way). It is just as easy to get into a physics graduate program with an ECE or EECS degree plus a couple semesters of upper division quantum mechanics and maybe a semester of stat mech - and you have the option of getting a decent job instead.
The school ponied up for some DEC laser printers to supplement, eventually replace, the daisy wheel printers, and the output both looked great (despite the DEC output being relatively "fuzzy" [1]) and allowed for foreign characters.

Knuth's book on TeX was a very enjoyable read.

[1] IIRC, the Math Dept. eventually acquired an Apple LaserWriter or two, and if you pushed the output there, the lines looked significantly cleaner.
Apologies -- had forgot about the hidden points, then hoped the sorting would take care of it.
Wish I had the karma for the poll because I was genuinely curious.
For those who want to know, at this point at least, Linode - 2, Pgrmr and ARP-1, Rackspace-0
It's also important to note:

http://news.ycombinator.com/item?id=2040593
I'm loving the recent explosion in self-published books, but as a (general) request: they are so, so much more useful when they come in multiple formats (mobi, epub and pdf).
After unsuccessful fight with distractions i decided to maximize pleasure, not productivity :)

Of course it works only if you are not a full time worker and there are for profit projects, that you love to work on.

BTW, i used to think, that i spend time here on HN in vain, but now half of my income comes thanks to one of comments here i read.
What's to prevent popular Tweeters from front-loading Wall Street or otherwise intentionally manipulating it through carefully-crafted tweets?
Very brief attempt at answering that: many believers tend to think atheism is a choice.

Most of the time, it isn't. My only choice was trying to see the world as it is. And the evidence at my disposal tell me that the Earth is very old, and that God does not exist (almost certainly so). I'm not at liberty to believe otherwise.
Its fairly predictable that the reaction to these would be "ugh browsers shouldnt let people do that" but the web is growing up and it is not purely a document delivery platform anymore, there are a lot of cases when a large scrollbar takes up valuable space within a small pane within an applications.

now along the same lines, when are we gonna get clipboard access
Sony calls in data Sherlocks to unpick megahack disaster
Shunning Nasdaq, LinkedIn Prepares for an I.P.O. on the Big Board
Honestly, I think that jQuery pre-1.6 did the right thing almost all the time and we all learned to deal with the boolean attribute edge case just fine. I feel like this change is taking jQuery in the wrong direction. We don't want to look back and say "1.6 is where the bloat began..."
http://en.wikipedia.org/wiki/DO-178B
Syd Sym iPhone game Post-mortem by Thaddaeus Frogley
Chief: Data on stolen police laptop safe because laptop needs power cord
What they don't mention is that the competing $200 cheaper desktop also only has a 23-inch screen.
You should remember that the existence of open source or home-brew solutions is not in itself going to prevent me (as a person who makes purchasing decisions) looking at a paid-for version. For example, I use Dans Guardian at the moment but am looking at Smoothwall to do much the same thing. I use Amazon S3 + s3ql for backups, but also use Nasuni for the same thing. I can make forms quickly and fairly easily with Django but prefer Wufoo for all sorts of reasons.

I've been told by my manager to favour off-the-shelf solutions - he feels it will allow us to hire more easily if I decide to leave. I prefer not to have the hassle of writing software myself, or downloading and configuring often very complex software which might not be that polished. Sometimes, I just want a support person to call who can fix stuff quickly so I can get on with my work rather than dealing with support issues. The people I work with are not very tech-literate so I need software which even they can use easily (otherwise I just end up doing their work for them).

There are many products I would never consider buying, and many top-rate open source products I cannot live without. But if you can encapsulate your expertise and knowledge in software and sell that to me and make my life easier, I will cheerfully part with money.
this comment thread is turning into quite the social experiment. is ADD a prerequisite for being a good programmer/hacker/etc...?
KILL IT!
Intel’s Big Mobile Power Play Could Be With Apple… and ARM?
Warner Bros. buys Flixster, owner of Rotten Tomatoes
"Check out the talk that that inspired the book" <- Find the the redundancy.
Interesting possibility, but I'm not completely sure this would work. I'm sure that a "jobs" section of facebook would gain plenty of interest, but I think LinkedIn appeals to people who would like to keep their job-related networking completely separate from their social lives.

The line between professional and personal life will shift, and this may be a generational thing. But who knows, the distinction may sharpen rather than blur. One of the big reasons I haven't left a huge web/video trail of my own stupidity on the web is that I'm old enough that the web was still an infant when I was in college, and nobody had cell phones with video cameras and recorders. Now, anything even slightly unusual on a college campus will be recorded by at least a half dozen people.

You could figure that people will blend social and professional lives as they they get more casual about these artifacts. But maybe they'll just learn to draw very fine and deliberate distinctions - kind of the difference between getting excessively drunk at a work event and getting blotto at a bar where your colleagues just happened to show up later in the evening.

Anyway, I went a little off topic there, but maybe there's a long term niche for social/professional networks that are explicity not social/casual.
Facebook Ads: How to Get 1 Cent CPC (and why you might not want to)
I've ported this to a Bash script which is simpler and more extensible. It uses env variables to facilitate customization.

https://gist.github.com/955437
Not really solid reasoning. The reality is that you can't prove either case. You could spend a lot of time trying to debate either side, but in the end it's probably just wasted effort. Pick a side, be happy, then get on with solving "easier" problems. NP vs P is still open.
I have definitely come around to the idea that most programming is art, not science. Not to say there is no such thing as computer science as applies to programming, because there is. Algorithmic research, language research, etc, all of these can fall under the domain of 'engineering', even given the description of this blog post.

But if you are using those things, then you do so as a tool, using tools is not necessarily the domain of engineering, but it is often in the domain of art.

I would very much like if my job title read "Code Artist."
45 HTML5 Media Players + Tutorials and Resources
Zero-config reverse proxies: HTTP  SPDY  0MQ
Apache Software Foundation subpoenaed in the Oracle vs. Android case
Half-Life was released in 1998. I can still play it--using the product key from my original retail purchase!--via Steam.

Can I definitively say that this will hold for all my current Steam purchases? No, of course not. I can, however, say that Valve has demonstrated the ability to keep giving me access to my games over a period of more than a decade.

In contrast, I have many games on CD that I can't play any more, because they don't work on modern operating systems. Some of these I've repurchased through gog.com, just to get working versions. Some I've repurchased even though they still work, just to get versions that don't require five CDs to install.

Buying a game from Steam is a calculated risk. It comes with advantages: No need to fiddle with discs, automatic patch updates, and access to all my games from any networked computer. It comes with one big disadvantage: The potential of losing access to my games at some unspecified point in the future.

That's a pretty good tradeoff for me. Maybe it isn't for you, and that's okay. When I consider Valve's established track record, however, I believe that the odds of Steam providing my games for long enough are very good--where "long enough" is "until the game ceases to work on newer operating systems and/or becomes available for repurchase for trivial amounts of money".

And if they fail me, there's always piracy. After all, I'm hardly going to feel guilty or morally culpable for resorting to piracy to reacquire access to something that I already own.
Ask HN: "deliberate practice" for startup founders?
What would be the equivalent of "deliberate practice" for startup founders. How do you get better at putting products out there? Getting Stuff Done would be one skill, building a team is another one that comes to mind. I don't know if this question even makes sense, but I was reading about how deliberate practice makes perfect. The way BB King played his blues ladders every day, all his life, and then went on stage and jammed away.
Oh, lovely story. I had precisely this happen in Physics at Uni. I wrote down every piece of data, wrote out my computations in full. When I submitted it, I got a D. I challenged my prof to check any one of the data points or computations and exhibit a single error (I knew he wouldn't because I had checked myself). He refused to check even a single one. Every single person for 20 years had fudged the results!!
Or GNUnet, as it matures.
Quebecer's really sent a message to the Bloc with the NDP sweep of Quebec. this must be adding insult to injury for Gilles Duceppe...
Android's Achilles' Heel (for the rest of the world) - The Sim Toolkit
How do you find cofounders?
I love entrepreneurship, but my projects consistently fail due to the exhaustion of juggling every ball. I've never sought others for help for three reasons:
1) My resume is practically non-existent, and I doubt I'd be able to negotiate a fair split with strangers.
2) I haven't befriended anyone knowledgeable enough about tech to cofound with that isn't otherwise engaged.
3) I don't have any money.

How do you find cofounders?
How would you overcome being worthless on paper?
It is a shame none of the modern smartphones support SIM Toolkit applications. They're used for example in the M-Pesa mobile payments system that is popular in Africa and elsewhere.

I wrote something about M-Pesa when I was in Kenya over the new year: http://www.qaiku.com/home/bergie/show/5035411811ce11e088577f...
3 ways to monetise YouTube videos that are better than ads
thats how it works in small startups -- You got to use what you have :)
Well, yes and no. You can read the original 1965 paper that birthed the theory here:

ftp://download.intel.com/museum/Moores_Law/Articles-Press_Releases/Gordon_Moore_1965_Article.pdf

Cost is certainly one of his concerns, but the original "law" has much to do with actually cramming more circuits onto a board.
Thank you.

I live in the Medical Device software domain. It gets tiring really fast when people keep proclaiming that software development can't be engineering. It certainly can; it's just that most organizations don't want to implement the discipline required.

You want very low bug counts, predictable software schedules and the ability to go home at 5 every night? Then implement a process to build what you need built, actually follow the process, track progress constantly and adjust when it starts going off the rails. Expensive and time consuming and requires buy-in from all levels of the organization? Yes, but TANSTAAFL.
Considering that I have a degree specifically in Software Engineering, I think I'm quite justified in calling myself a Software Engineer. If you feel universities aren't justified in graduating students as Software Engineers, then that's something that you should take up with the Dept of Education and various other groups regarding accreditation guidelines.

My point being that just because building software is not as well defined up front as more classical engineering trades, doesn't mean there aren't aspects of it that are very similar to engineering principals, especially considering how young the field is. I bet that when we first started building bridges and skyscrapers, nobody had a good idea of just how long it would take.
Maybe that what's the author wanted to pointed out: if you have garbage collection (at least the way it's implemented in C#/Java) you have to give away scope lifetime of object.
This is annoying because you want determinism for some kind of resources (file, lock, ...) and then you have to manually clean them, which is what you wanted to avoid in the first place with garbage collection.

In C++, if you declare the object on the stack, the destructor will be called when the scope is closed. Which is very handy for e.g. lock.

The thing is, garbage collection could be use together with scope lifetime (aka RAII) just let the programmer choose which is best/more convenient for him given his resources.
You are being downvoted only because you are criticizing the god-like-status Linus Torvalds.
It's less a misstatement and more a generalization. Stuff getting exponentially faster or cheaper is a very powerful idea.
You sometimes need to represent things differently to make them appear the same
A collection of cool terminal tools you may not know (with screenshots)
Wall ahead in multicore programming
My biggest hack: accepting that this battle will never be won. If I rely on a "system", I stop fighting the demons of fear, procrastination, and perfectionism. I adapt my methods, stay in the moment, break tasks into small digestible chunks, and calmly move forward.

Also, I'm not above cutting myself off from time-wasting sites in /etc/hosts when I really need to. :P
Chinese Facebook: Your Captcha eludes me
Potential anonymous gift senders? I think we have to appreciate the level of gratitude that people feel to this guy for creating jQuery.
There are far more coders working for companies where no one dies if they code has defects.

The article argues that most programmers are not engineers. At least I draw that conclusion.
Winning Attitude, Losing Attitude
... web developers sigh