skysight-inc/HackerNews huggingface dataset:
Hi, I work at commoncrawl, so I will try to answer your question. We store our crawl data on S3 in the form of 100MB compressed archives and there are between 40,000 and 50,000 such files in commoncrawl’s bucket today. The key to scanning such a large set of files efficiently on EC2 is to have your each of your Mappers (assuming you are running Hadoop) open multiple S3 streams in parallel to maintain some desired level of throughput. For example, assuming that you can maintain on average a 1MByte/sec throughput per S3 stream, and you start 10 parallel streams per Mapper, you should be able to sustain a throughput 80 Mbits/sec or 10 MBytes/sec. If you were to run one Mapper per EC2 small instance, and start 100 such instances, this would yield and aggregated throughput of close to 3TB/hour. At that rate, you would need 16 hours to scan 50TB of data, or a total of 1600 machine hours at $.085 per hour, costing you somewhere in the neighborhood of $130.00. Of course, you would then need to add in the cost of running any subsequent aggregation / data consolidation jobs and the cost of storing your final data on S3. So, the $100.00 number is generally in the ballpark but final numbers may vary :-)

As far as comparisons to Yahoo BOSS are concerned, no, we are definitely not comparable to Yahoo BOSS or other such APIs that run on top of an already built (and properly ranked) inverted index of the web. At this stage we only produce bulk snapshots of what we crawl, and we are focusing our engineering resources on improving the frequency and coverage of crawl (the results of which will hopefully start to bear fruit in early 2012). Perhaps at some point in the near future, we can partner with the community to build a rudimentary full-text inverted index of the Web that we can make available in bulk via S3 as well.
We'd still be left with a spam problem. Just a smaller one. And it would be nigh on impossible to implement and co-ordinate such a change.
This is such a generous SEO Scheme by Wired.com

"Photos must be properly attributed to the photographer and Wired.com, and we ask for a link back to the original story where the photo first appeared."
AGPL uses the same definition of linking as GPL so communication over a network API to a storage backend is not linking.

http://www.quora.com/Does-the-AGPL-extend-the-idea-of-linkin...
This is awesome. Just tried it out and it syncs to all my Mac desktops / laptop. However, I'm not sure how to access the content on my mobile devices (iPad and iPhone).
The fact that most business decisions are made by gut feeling implies that post-hoc pseudo-mathematical rationalization shouldn't be called out as bullshit?
I generally use the Ubuntu patched LCD packages via AUR and this page was very helpful in helping me get it set up. I've heard the Infinality patchset is really good too but I haven't tried it yet.

Personally speaking, since I am so xterm oriented, the best thing I did was install the bitmap font Tamsyn, which I just adore. Easy to read and use. It is available from AUR, but I just download it to a directory in my home folder and use xset to add it to my fontpath.

(And Arch's wiki has an incredible wealth of information. It is one of my favorite online resources, along with OpenBSD's FAQ.)
WHY A GOOGLE+ BRAND PAGE JUST GOT MORE IMPORTANT
This sort of thing is so pedantic, almost insulting to my intelligence. Do i really need a website to remind me how to get off the internet from some random hacker news guy with too much time on his hands? He's almost countering his own message here. This is not a quite place. It's still the internet. I have to click my space bar to get to a new line teaching me about going to a 'quiet place'. Why is this even being clicked up on HN? And then i have to tweet it too? It's a joke right?!
Sorry, our github repository had some accidental check-ins that we needed to remove. I will share the link to the code shortly.
There's Nothing Wrong With Having a Plan
CrunchBang 10 Statler review - Crunch, bang
I think you are under estimating the stench of corporate bullshit. Take how ever bad you think it is and multiply it by 10 then add a little bit and it's worse than that at the really bad places.
Why? What can you accomplish by running unsigned code that you can't accomplish by embedding a (e.g.) Python interpreter?
Last I checked IE, Fire Fox and Chrome all had a dev mode. Sure they might not have had them at first but a strict mode or a parse check or some thing of that sort would have been helpful all along.
Information Technology Solution South Florida
Paktolus ITS is a South Florida based IT solution company providing information technology services such as website design, web development, Internet Marketing (SEO, SMO, PPC) with computer repair, hardware networking in Miami & Tampa, FL
IT Services FL
Our IT Services includes remote & on-site support, hardware networking, backup & security, consulting services, webhosting services in Tampa & Miami, FL.
You can also currently block IPs, domains, URLs and specific message content. There are even globally distributed lists of such things.

This is already how we manage to block most spam on the edge. What you're proposing is just a small iteration on the existing defences. An expensive one, which wouldn't work unless you managed to get everyone doing it at the same time.
If you listen to the interview, Metzger actually seemed happy that there would be a paid version, thinking it meant that it would be maintained. Instead, the paid version didn't work for him and fixes didn't seem forthcoming. So he rolled his own and contributed. I'm not saying that your scenario wouldn't happen, but it doesn't apply in Metzger's case.
Maybe I've misunderstood something. I wonder how much spam is sent via compromised residential systems and how much is sent via less than perfect commercial systems?
How To Set-up Your Company's Google+ Page
Thanks for the clarification - it helps a lot.

I have to admit that you seem to be offering more value than I realized.

Kudos.
This ranking system is considered to be one of the most reliable around. Even if it is only a measurement of one website’s traffic.

What? This is even more ridiculous than the TIOBE index.
Not in the traditional startup way. They took some money from Jeff Bezos (in exchange for a little equity) a while back but it was so they could cash out. They didn't need it, and I'm pretty sure Bezos doesn't give a single fuck that they're doing this.

They aren't going to sell out, so Bezos wasn't looking at an exit as a return on his investment (and he doesn't need it either).
Ask HN: Good resources for beginning with F#?
Looking for any good beginning tutorials for breaking into functional programming using F#

I'm choosing F# solely based on the fact that .NET has been my bread and butter for a while now, and I figure my familiarity with the IDE and platform as a whole would make it a good choice to begin learning functional programming. Also, I can just get-going with it quickly, as I have .NET and VS on just about every machine I own.

Once I've got the basics down and can begin to "think" in more functional terms, I plan on migrating to another functional language (suggestions welcome).

Although I understand some basic functional approaches (passing around functions as arguments, functions that return functions, lambda expressions), please go ahead and assume I need a "My First Functional Program" style set of advice.

Bonus points if the tutorials can actually demonstrate real world applications of what they're teaching, but that might be a bit much to expect from a "beginning" set of tutorials.

Thanks in advance!
Interview with Oliver Chris: One Man' Two Guvnors
Although the Boston Dynamics robots seem more fluid and may have more flexible and natural overall movement, the ASIMO seems more autonomous. I haven't seen any of the Boston Dynamics robots actually performing a task adaptively (other than adapting to terrain and pushes/kicks). PETMAN looks really interesting but until it is turned loose untethered, ASIMO wins the best android contest.
Ganar Dinero con un Blog - 3 Opciones
Startups: You're Not Really Ramen Profitable, You're Ramen Sustainable
The Early Stages Of Success
Amazon should own up and pay. They have effectively stolen from this author. No, we can't guarantee that those people would have bought it for free, but if these were physical books, the law would require that they pay the author.

The agreement wasn't that they get to change the price any time they like. The agreement was that they get to match competitors. They violated that.
KEYS is indeed only a debugging command, as clearly stated in the documentation.
Ask HN: which mobile framework, if any, to use?
Im developing a mobile web app for iOS and Android phones and Im not quite sure which approach to take.

Here are my options as I see them:

1. Use JQueryMobile or Sencha

I've had some experience with JQM when it was in alpha. I liked the structure and UI "look and feel" stuff, but the built in controls (ListView) were slow and the fixed positioning did not work as expected.

Haven't tried Sencha, but the pure Javascript approach seems abit unnatural.

2. No Framework

This is the most tempting approach, but I don't want to reinvent the weel.

Are there any good mini-frameworks/libraries for messaging/fixed-tabbar/basic mobile ui ?

As an example, I really like the feel of the GMail mobile app, they seem to have the fixed positioning sorted also.

Feedback welcome!
"Must use archiving ISP mail servers" is sinister.
Quite true. There are all sorts of pointy and/or hard things around that you don't notice until someone is having a seizure.
The vast majority is from botnets, which compromise mostly of compromised residential systems. http://www.theblaze.com/stories/email-spam-down-82-percent-s...
Black Friday Golf
Yikes, I sure wouldn't use an undocumented feature to keep track of anything important.
Hi I work at commoncrawl. We have spent our time (in 2011) improving our algorithms, and hopefully this effort will start to show real results (with respect to crawl frequency and relevancy) in 2012. But you are right, it is pretty unlikely that our crawl will be able to be fully competitive with the likes of Google etc., multi-billion dollar corporations who dedicate huge amounts of engineering and hardware resources to stay competitive in this field.
I'm betting the next "big thing" with these social sharing sites is personality driven niche sites. Reddit, Digg and HN all formed around the personalities of their founders but as the communities grow large the community's personality itself takes over (giving the community a personality out of control of the founders). What I haven't seen is the loss of that democracy in favor of a benevolent dictator.

Imagine a community where someone like Arrington is publicly voting on the stories giving him a Caesar like "thumbs up, thumbs down" influence over the content. You see this to some extent with "power users" but I think it won't be long before someone intentionally starts a community to promote their own ego and successfully develops a following.

Think of it like grabbing a bunch of Twitter feeds in front of a crowd and saying "I like this one, who's with me?" -- and the crowd cheers. "This one sucks, right?" -- and the crowd boos. People like to follow, not lead and this concept plays off that dynamic.
If the content is great and accessible, and the visitor leaves the website satisfied, who cares?
#1 Fix the F-11 message. Is this an IE thing? Do a browser detect, and display the right message.

No, it's an IE, Firefox, Chrome and Opera-thing. All browsers use F11 for fullscreen and i don't think there's an api for reading which button they use anyway. :P
Myths about our minds
A more realistic comparison is with Google Trends.

http://www.google.com/trends?q=Linux+mint%2C+mint+Linux%2C+U...
MoneyUSAmutual.com - Payday Loans
Leadership Word of the Day: SOLITUDE
Ah, thanks for that. I was under the impression that the bar for linking across the network was rather higher.
Hmmm... won't install with ghc 7.0.4:

$ cabal install berp
Resolving dependencies...
cabal: dependencies conflict: ghc-7.0.4 requires containers ==0.4.0.0 however
containers-0.4.0.0 was excluded because berp-0.0.2 requires containers <0.4
Programming feels more like playing minefield than reading a book or communicating with somebody. You are constantly wondering "this and this implies that this would be true always, except if this won't be true, but this won't happen because of something, so i can safely change this to that, and nothing would break".

You should make comments and try to convey what are you doing to future human readers of your programs, and for this good communication skills are important, but the core of the program - the code - is pure logic and adhering to strict protocols, and without this you won't program at all.

So language skills won't automaticaly make you good programmer. Just try, it's addicting, you'll know if you want to do it :)
Phobos-Grunt launching today in a bid to collect Mars moondust
> Instead we should rely on computational complexity, because no matter what we do, unless quantum computers become a realy, there are limits to what we can compute when exponential complexity is involved.

Quantum computing does not help with exponential problems in general.
ubuntu has almost 20m users, the recent circlejerk around this pointless click rank is ridiculous
Then please come up with one. Try not to score too many checkboxes on the form:

http://craphound.com/spamsolutions.txt
Or make Enqueue use the iTunes library directly, as Simplier[^1] does.

[^1]: http://itunes.apple.com/us/app/simplier/id467077932?mt=12
اكبر موسوعة لمجلات ناشيونال جيوغرافيك العربية National Geographic arabic
Android, Samsung Winning In U.S. Market For Q3 – comScore
Ask HN: Why is my Question not visible to others?
I submitted a Question Yesterday but it does not show after I logout from HN.

I presume it was not visible to other HN users as-well.

When I am login it is visible to me.
Its a leap that folks desiring cloud storage will want to host their own some day. I think its a little like "I want an open-source car, so when they switch back to horses, I'm ready!"
(a) is primary. Problems in physics are much harder to solve than they were.

For instance, the thing that corresponds to general relativity today is quantum gravity. Other paths exist, but the two that are most traveled are string theory and loop quantum gravity. The math involved in either one is considerably heavier than Einstein's relatively theory, and neither one is in a place where you can calculate real observable things.

There's plenty of (b) to go around, but it's hard to believe we'd know what dark matter is, discover supersymmetry, or have an established "theory of everything" if we cleared away the cobwebs.
There is a proposal for a proof of work system that does not require the sender to pay direct monetary fees and does not, significantly, slow down sending emails since it can be computed while the sender is writing the email.

I can't remember what it is called, but I hope somebody else can chime in with the name.

The smart thing about it is that you can use it as a filter in your baysian spam filtering systems. That way it can be implemented gradually, with no need to cut out everybody who doesn't yet use it.
The plan? To release approximately 25 videos over the
course of the next year sharing 37signals'
stories inside the office and out.

Sounds like a reality tv show to me. Something like Pawn Stars or Orange County Choppers.
Call 404-919-7545 to learn about CMM Energy
I'm sure their agreement had more than one clause. It could very well contain a set of clauses that limits amazon's liability to whatever it collects from customers (in this case, $0).
I agree that the website is quite functional and presents great content but I don't believe it conveys great design aesthetics. Some of the problems that I instinctively felt were:

- Intuitively I do not expect the clickable diagrams to be 'clickable'. It is sort of hidden in a plain looking site. If you don't look carefully, you might miss it.

- The choice of colors are inconsistent across the site.

- Unlike what the closing note suggests, it would be a little challenging to adopt the table-based 2-column layout for a more 'responsive design' suiting smaller screens.
No doctype means quirks mode, does not matter what's on the first line.
If that you could care less is the most you can say about something, that's an insult. If you say that you couldn't care less, that's just a lie - you're responding to it.

http://en.wikipedia.org/wiki/Damn_with_faint_praise

I love David Mitchell, but this rant by other people is a long held irritation of mine: false pedantry:)
Let's not confuse what's right with the law. The two are only occasionally related.
Caldas novas cvc
CVC caldas novas e faça parte de uma viagem inesquecível, lembre-se que qualidade e formas de pagamento são coisas que sobram nesta agência dos sonhos.
Is it littering if you intend it to be found and returned? Its not litter to leave your wallet in a bar, or you car in a car park.
For anyone curious about capacity and pricing versus Dropbox, despite the differences between the services:

iCloud

5GB free, 10GB: $20/yr, 50GB: $100/yr

Dropbox

2GB: free, 50GB: $120/yr, 100GB: $240/yr
Whats the bandwidth if you put a thumbdrive in the bottle? Error rate?
Recent discussion: http://news.ycombinator.com/item?id=3192006 (19 comments)
Those who will maintain the site.
In any case, table layout is not that accessible.
It stops our mail server from being used as an open relay though?

How does blocking 1 specific port stop the issue anyway? They can just change the port they connect on?

As stated in the actual article...

Do these government agencies actually think blocking port 25 will reduce spam? Most of the issues stem from open relays.

I was offering an opinion on how to resolve those issues. Changing which port accepts the mail is in my opinion pointless.

It's like saying that most burglars come in through the back door so the government blocks everyones back door, they will just come in the front.
We're talking about the global spam problem, not your localised end user spam problem. Even if gmails spam filter was perfect and stopped all spam (impossible), they'd still have a problem because they'd be using vast amounts of human and computing resources to do it.
Compared to the amount of oil burned around the world for all the computers reading HN I think you can ignore the bottles
The Knife Maker
It's been a while since I've read this essay -- and I'm surprised by some of the reactions I'm reading in these comments!

The essay is just as good now as it was then.

The underlying ideas are not only correct, they're being actively validated in the increase in salaries for good programmers.

Someone in this thread says that programming is a commodity. Well, sure. Isn't everything?

But the key point in the essay is about making value directly, by making something people want.

Has that somehow become a dated notion? Isn't it, rather, being validated with every single startup that enriches its value-creating founders?

There's no 'magic' in the essay, of course. No secret key to success, no fairy dust; I'm sorry for anyone who felt there was, but such is youth.

The essay never said that all programmers will become millionaires.

What it said was that if you wanted to become rich, and you're a programmer, probably your best shot is to make a product that people want.

Google and Facebook and Twitter and more, they're all validating that argument by raising salaries for their best talent, to make creating value directly for the customer a less attractive option.
Here is more detail about their goals w/ this move:

http://37signals.com/svn/posts/2980-were-looking-to-hire-a-f...
You could pull the app after confirming that the vulnerability exists and collecting evidence. Or tell the vendor that it's there:

http://twitter.com/0xcharlie/statuses/133901782169550848
Its a walled garden; they can do anything they like. Live with it.
I hope they add some Python stuff soon. I'd love to help write Python lessons.
Anonymous Hackers launched DDOS attack on El Salvadoran sites
What you're asking for is greylisting.

Nothing stops you from doing greylisting today, and it can be a very effective means of stopping most spam. There are tons of software solutions for greylisting.

Some require manual approval first time. Some sends a message back to ask the sender to do something (anything from just clicking a link to entering a captcha) and relies on you to do manual approval now and again (to catch automated but valid messages).

Some just defers delivery and waits for a second delivery attempt (because most spammers practice "fire and forget" and just ignores errors). Our office mail server is in the latter category - first time someone e-mails us,a second delivery attempt needs to happen after 10 minutes for the message to get through (it's ok if there are attempts in between too, but the assumption is that most of even the few spammers that retry will go away too quickly to send another attempt after 10 minutes). It gets rid of the vast majority of our spam before our real spam filter even kicks in.
The spirit of Steve Jobs lives on.
I'm currently staying away from iCloud syncing as I'm hearing lots of bad stories from people using its _supported_ functionality, so I too definitely won't be using a undocumented feature.

Since Dropbox is both free for 2GB and rock solid why wouldn't you just use that? (And there is an iOS client too.)
Maybe, but lets be honest the price they would have to pay for this is what a few seconds profit? Compared to the risk of losing the next great author because they get scared away that is a very small price to pay.

Not to mention that he may have a case for piracy for commercial gain, in which case Amazon is liable for $300000 a copy (which, at 6000, is more money than most authors will ever see).
https://secure.wikimedia.org/wikipedia/en/wiki/Hashcash
Last update was in May 2010...
Nice, but it would impact legitimate mailing lists.

(In the 90s Bill Gates touted the idea that the recipient of the email would receive the fee, and routinely refund it to the sender for legitimate messages.)
It's trivially easy for you to greylist everything, and trawl through that everyday, and whitelist what you want and blacklist the rest. Anything whitelisted is automatically whitelisted forever. Anything blacklisted is automatically blacklisted for ever.

The problem comes when spammers forge From headers which leads to:

-1: Very many emails in the greylist everyday

-2: False positives if they use a From that you've previously accepted (ie that person gets infected)

-3: False negatives if you get a spam that you reject which was sent from an address that you really want whitelisted.

Some of the failure modes are similar to challenge-response systems.
There may be a start-up idea here: an application to allow authors to easily monitor changes to their ebook prices?
مشاهدة مباراة الزمالك واتحاد الشرطة اليوم
مشاهدة مباراة الزمالك واتحاد الشرطة اليوم
مشاهدة مباراة الزمالك واتحاد الشرطة اليوم
مشاهدة مباراة الزمالك واتحاد الشرطة اليوم
مشاهدة مباراة الزمالك واتحاد الشرطة اليوم
مشاهدة مباراة الزمالك واتحاد الشرطة اليوم
http://media-sports.blogspot.com/2011/11/blog-post_6450.html
Recycling is the bane of archaeology. Science or nature, choose one.
That mincing gate pretty much precludes selling it for serious applications. Looks like a coolie carrying a rickshaw...
Amazon doesn't talk about their numbers either. The only thing they do say is that RRS (reduced redundancy storage) 'stores objects on multiple devices across multiple facilities, providing 400 times the durability of a typical disk drive, but does not replicate objects as many times as standard Amazon S3 storage, and thus is even more cost effective.'

This is at the main page: http://aws.amazon.com/s3/ (search for RRS)
Google cached version: http://webcache.googleusercontent.com/search?q=cache:KJXbNFy...
I think this could fall mighty close to the category of jumping the shark.