skysight-inc/HackerNews huggingface dataset:
Not sure if this is exactly what you're getting at, I use capistrano, it was built for rails deployments and does require some scripting/setup but once you've got that down, i can push changes to any of my sites all day long. I have a few wordpress installs that i deploy with capistrano as well. Once you've got everything setup, there is no noticeable downtime.

http://www.capify.org ... to modify the database schema you can put migration logic in your scripts.
You could modify the plugin to exclude certain areas of the comment from validation. I'm thinking of Markdown, where indented lines are code -- you could just validate non-indented lines.
Impressive...I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content.

I was also sort of hoping to see from the stats what proportion of content was public-facing vs devoted to arguments between wikipedians...if you look at the stats for 'most edited articles' (accessible from the top link) it's interesting that of the top 50 most edited articles, only one, 'George W. Bush' is user-facing - and I suspect that only made it in because of persistent vandalism.

Still, with history and all included, there is some fabulous data-mining potential here, with which there's the potential to do some really innovative work. I'd hazard a guess that the size of Wikipedia already exceeds that of existing language corpuses like the US code...

/retreats into corner muttering about semantic engines and link free concepts of total hypertext as necessary AI boot conditions
A super-fast newsfeed in Redis
Twitter Adds Places To Its Geo-Tweets, Just Don’t Call It A Check-In
Is SS2 orbital? I thought it was just "up and down" like SS1. Virgin Galactic's site seems to confirm that http://www.virgingalactic.com/overview/spaceships/
@Cory Matthews:

Compared to IE? Yes. Compared to Firefox? Depends on the release schedule of both. Compared to Chrome or Safari? It lags significantly behind.
I'm not hugely concerned about swapping out application logic, since with a bunch of application servers it's possible to pull some out of the pool, upgrade them, then use the HTTP load balancer to redirect all traffic to the servers running the updated code.

The big challenge is making changes to the database schema and co-ordinating the deployment of those changes with the switch over of traffic to the new application logic.
I think what we see is a (hopefully temporary) situation where there are people so poor that they are willing to work cheaper than machines. Because Americans are not going to go back to doing that work, it's not really a situation where Chinese labour is competing against American labour. the Chinese labour is competing against American /machines/
Hey, you're sneaking around someone else's network. It's an occupational hazard :)
Additional issues:

My HN username was imported lowercased. Both Zak and zak exist as HN accounts - the former is mine and the latter is not.

When I try to update the username under the links section of my profile, I get a complaint about my password being too short. This form does not contain a field to change the password.
Hopefully this will clear up some of the Bundler rage that people seem to be feeling.
This headline reads like one for an Onion news story
The defensiveness of some of these comments is unsettling. Inhumane conditions are inhumane no matter where they are, and no matter what the situation is in the society; it is never right to do to one's workers what this article is describing, not even if there's a lot of competition for their jobs.

This is the kind of thing that explains why the United States is not lassiez-faire capitalist anymore; and it emphasizes that we should forswear our addiction to shopping cheap, in favor of buying what is produced in a morally licit fashion, and genuinely punishing corporate leaders over whom the US has jurisdiction and who knowingly go along with this kind of thing. I don't mean sanctions directed at the corporation: I mean punishing the individuals who sign off on or turn a blind eye to this, and I think that a crucial part of any such punishment, in addition to a long prison sentence, is the confiscation of the individual's personal fortune -- since this kind of person (well, almost always "this kind of man") sees money as score, and his objective is to die with a score high enough to leave his initials on the game's attract screen.

I want to emphasize here that this is not partisan United States issue, either. I am a serious Catholic; I vote Republican and like the Tea Party movement; and I know that this kind of thing is wrong, and that you don't have to be a Green, "pro-choice," or a gun-control advocate to recognize it as such.
If Foursquare or Gowalla want to stay on top of this game, they need to start partnering with pubs/restaurants or high-traffic franchises and start pushing event notifications to their users, rather then wait for them to check-in.

The user still does their usual check-in wherever.. but if they’re in close proximity to Pub XYZ who has wings on special, an amber alert of sorts would start beaming on their device to let them know of the special.

Therefore entice more check-in and revenue opportunity.

I’ve had this idea in my head for a while.
Microcontrollers, Electronics & Robotics: PICMAN
Or they can just remove all google's ads and replace them with their own.

This is what BT (the UK's effectively monopoly phone co) did.
The introduce a system (phorn) that monitored all your browsing and replaced the ads with ones by it's customers more targeted to your 'interests'.

It got found out when it removed all charity's ads following an earthquake and also replaced all the ads on a sesame street type kids site with ones based on the father's browsing the night before!

This was illegal but somehow the government never got round to prosecuting them - in spite of the EU telling them to.
Yeah, but I was thinking more brain mismatch. On the other hand, most CPAN libraries aren't so large that they use APIs which needs to be overridden; I was probably wrong. (Anyway, it was irrelevant to my point. :-)
If an individual decides to be highly competitive by working crazy hours, is that considered bad? Will her colleagues complain to the labor department? A capitalist country's inhabitants, who belong to a system driven only by output and profit, complaining about labor practices, is the biggest oxymoron. Surely, the Chinese industries tactics are understandable?

In order for the Chinese manufacturing industry to be competitive, they indulge in such practices. If you want to remain competitive, one way is to cut costs or increase production, the way they do or by way of technological innovation. Or create high quality products, that consumers would want to buy, instead of the cheap {quality, cost} ones made in China.

The opponents of outsourcing are surely barking up the wrong tree. The Chinese workers surely have a choice of not working there. Agreed, that if they choose to work, life is pretty much miserable. But what if they choose not to work there. Do they go back to their non-existent farms, or to the streets? If such labor laws are enforced, the obvious impact is that oursourcing would move elsewhere. There will always be some country willing to take it up, and not to mention the people who would welcome a respite from their poverty.

The Chinese industry has decided to stay competitive. The people are probably between the devil or the deep sea, and it's all about choosing the better path, which in this case, simply is working 15 hours a day.
Actually I was doing this last night and forgot that it was just the information box on Schedule B with your 1040 - you definitely don't send the actual form to the IRS. Just send the FBAR itself to the treasury.
Introducing Cascalog: a Clojure-based query language for Hadoop
This is a huge collection of primary sources from lots and lots of people. It's kind of like if you got a copy of every letter sent by people in Philadelphia in the 1770s - most of it would be crap but there would be some pretty neat stuff too.
Congrats guys! I love your product and hope that you continue to evolve it under Rally's wing.
In a related study, blind people showed no racial bias due to skin color.
And yet, I for one found it very pertinent to HN. After all the discussion as to whether the iPad really is a revolution or not, I personally find this as convincing a demonstration as possible that a very intuitive interface has been created. It's a useful data point to have in mind when discussing iPad ergonomy.
The Official Panic(.com) Basketball Team
Was that an attempt to explicitly reference <http://www.reddit.com/comments/4jxe/paul_graham_ate_breakfas...? (from April 2006)
Open Logic Sniffer shipping
They may be waiting to see if Google can be rolled. If that happens, the floodgates will open.
When I was studying for my first degree (in mechanical engineering) I had to do a one month long internship as a factory worker after my first year (in an EU country). It was truly an eye opener experience, both in terms of understanding how industrial production looks like from below, experiencing the life of a blue collar worker and as a motivation to take my studies seriously (to avoid having to have a job like that) as well.

I think short "slave work" internships for first year students should be more common.
Can someone provide feedback on this release as to if it "includes rails 3" coverage or is "totally focused on rails 3". I would prefer the later; a book that doesn't take into account migration or trying to explain 2 vs. 3.
Now it crashes on an attempt to register.
Possibly, since I'm posting this from Opera Mini. That said, everyone who says its buggy and slow are bang on.
I know we've had our differences at times but every once in a while you guys pull something off that makes me sit back and say to myself, "OMG, Humans rock!".

--the Cat
Italian judge: Google profit prompted guilty verdict
The End of Retail? (The Brick and Mortar Type Anyway)
Yes, I'm pretty sure all the measurements done in these sorts of studies are done using the redshift-blueshift method you mentioned. A rotating star has a "red" half and a "blue" half, and a planet transiting the star will cause first a blueshift then a redshift, or a redshift then a blueshift. I went to a talk on this subject a few months ago and was amazed by how sensitively they can pick this kind of thing up.

I think at that stage there was exactly one planet known in a retrograde orbit, but this new discovery shows that they're relatively common rather than being a one-off occurrence. Neato!

I don't think it challenges existing theories of planetary formation quite as much as is described in the article, since our existing theories of planetary formation are already pretty handwavey. I can easily imagine that a planet with a retrograde orbit could happen just due to collisions between large bodies during the planetary formation stage.

Oh, the other important thing to note is that the images are all "artists' conceptions". (Well, except for the one which shows two people in front of a telescope). Telescopes cannot possibly resolve a planet as a little dark disc in front of a big bright disc, so all these measurements just rely on the slight darkening of the star as the planet passes in front.
I work for the LoC. I think I'd of heard of this.

Edit: It is my considered opinion that a lot of what the LoC does is excessively ambitious. The project I'm currently working in is called the Born Digital project, which is an attempt to allow major broadcasters like PBS and CNN to submit broadcast material to the Library in real-time. The tools to do this, do not currently exist.
My score for Chromium is: "65-90 points of FAIL: Kittens die when your code is downloaded"
Sorry, I probably shouldn't have used the term 'standards compliant'.

Don't get me wrong, it's far better than IE, but whereas if you get something working in firefox, it usually works as is in safari/chrome, there's sometimes little quirks in Opera.

As a relatively small userbase use Opera, it'd just be nice if it did things exactly the same as firefox/Chrome - layout, scrolling, iframes, etc etc

I can't remember specific details, but I often notice weird layout quirks, or issues with scrolling (Not well defined in any standard).
Error establishing a database connection
Guess rule #1 is not to be too lean when it comes to your database capabilities
I don't get it. I can slide the local time around, and right next to the bar it shows the time in each timezone, and there's a bunch of grey bubbles that move around. Is it just that it's broken on chrome?

[edit]I tried it in opera 10.5 and firefox 3.5 and it was worse than in chrome, what browser does it work in?
Its my understanding China's constitution does protect unions. China is a nation of many provinces, each differing in their respect to national rules. Even within wealthy, better regulated provinces like Shanghai, behavior is highly uneven. I have seen cases where unions and other labor laws (maternity rights is a huge one) were heavily weighed in favor of the laborer. Its still not enough.
A few weeks ago I was looking at signing up for an smtp service rather than using gmail to send emails, due to the limitations in sending mail via gmail.

However, from what I understand, it seems setting up your own server is pretty much a guarantee that your emails will get labelled as spam.

Any idea on how spam filters treat email from Amazon?
Latest climate hack inquiry clears the CRU (again)
I'm very happy with 10.6. I was very happy with 10.5

I am quite happy with 10.4 actually. I have no reason to spend $140 for 10.6. I'll continue as long as there is a workable Omnigraffle version for 10.4 :)
First they want to track all of our books... now this? Next it'll be the newspapers. Just you wait.
You can reply directly to comments using the "reply" link. No need for @ style comments.
No. Its fair big data actually. Around 55M tweets per day currently, although it wasn't actually always that big. Maybe that doesn't sound that big, but its at least a few terabytes of data.

I think this is big for a few reasons. One, if its in the LoC then it is (probably) available to the public. Previously it wasn't. That's pretty big.

Also with the LoC having a public archive of it, we can keep better track of politicians who post things online- always a good thing since occasionally they shift their positions or outright lie.

As a data analyst, I'm really excited.
Right, because working more than 45 hours a week is a crime against humanity and here in the civilized West nobody ever works more than that rollseyes

I have a cushy office job but the friends from my adolescent years are now mostly blue collar.workers. Those who have their own businesses (one-man shops: plumbing, construction contractors, painters, those type of jobs) work 50 to 60 hours a week, plus client meetings and paperwork in the evenings and on Sundays. Those who work in factories work regular weeks of 40 hours and they put "Yay! I'm getting overtime!" statuses on their facebook pages when they get to work another 10 or so hours, either during the week (evenings) or on Saturdays. Of course sometimes they complain too when their boss asks them to work on the day after a big soccer game or whatever and when they had rather slept in, but in general the overtime is what makes the pay quite good - so it is mostly considered a perk to get to work overtime.

That is in Belgium, if that matters.
Silly though it is, it's also a neat (if unintentional) proof-of-concept for the viability of tablets in psychology experiments with higher vertebrate animals and small children.

I've noticed that studies of animal intelligence that employ computers seem to feature equipment that is either very expensive or very primitive (not through any especial fault of the researchers, I hasten to add - it was technology which had not yet caught up to their ideas and needs). The iPad and its forthcoming competitors offer a pretty robust and capable device at a very low price, which suggests enormous scientific potential since many new research tools are now no more difficult to produce than a fairly modest webapp.
One suggestion http://en.wikipedia.org/wiki/Zimbra#Software_license
Caution: may void warranty.
I love Github, I like the Github team, and I use Github issues every day. But I honestly can't wish it on anyone else, it's just terrible. It's slow, buggy, and not all that well organized.
> 5.6 Tb uncompressed, 280 Gb in bz2 compression format, 32 Gb in 7z compression format

Wow, I didn't know 7z was this much better than bz2. Is this the expected result, or is there something special with Wikipedia that plays to the strengths of 7z?
Industrial Revolution repeating itself: http://en.wikipedia.org/wiki/Industrial_Revolution#Social_ef...

Is a phase like this required for developing nations to pass through? Specifically, large developing nations? Or could it be skipped, or at least have its negative human effects mitigated?
> I wonder how big a content snapshot is, ie no article histories and no meta-material like talk pages or WP:xxx pages, just the user-facing content

I don't know how big it is uncompressed, but they do have a dump of just that part:

2010-03-16 08:44:40 done Articles, templates, image descriptions, and primary meta-pages.
2010-03-16 08:44:40: enwiki 9654328 pages (255.402/sec), 9654328 revs (255.402/sec), 82.9% prefetched, ETA 2010-03-17 03:08:26 [max 26568677]
This contains current versions of article content, and is the archive most mirror sites will probably want.
pages-articles.xml.bz2 5.7 GB
I see what you mean. I've never looked into it, so I can't say, but I imagine you're right that it would be quite expensive to do so.
They've long been the leader in the mobile market though, and while the PR has undoubtedly helped like crazy that probably would have carried them alone.
Google Will Launch Android-Powered gPad
Your comments are pretty fair and your feelings on what your family went through in '60s Taiwan I take as genuine.

However, it was not only hard work that bootstrapped Taiwan out of poverty. Some part of Taiwan's current comfort is due to moving its most extreme labor and environmental problems to other Asian countries, namely China. So now China and a few others are at the bottom of the supply chain...who do they pass the buck to to lift themselves out of these conditions?
Yeah, this is exactly the sort of problem bittorrent was designed to handle
> If Apple only allowed in EA, Activision, Ubisoft, etc - you'd have a point with your captive audience analogy. The PS3, PSP, PSN, etc fit in great with your argument.

But the App Store is different than that. More than just the big names are allowed in and are making money. And that's the net win.

I don't entirely agree with that. The AppStore only allows in people that Apple 'likes.' Granted, this is a larger group than say the vendors at Disney's theme parks or Walmart's retail contracts, and maybe that is a net win. But it's still a captive audience. Apple could achieve the same effect with the AppStore + random apps installed from the internet.

The AppStore would still be the same integrated purchasing experience, and I'm willing to bet that most people would still sign up to be part of the AppStore. But this also leaves the avenue open for people that Apple "doesn't like" to create apps that target the platform.

What would you think if Ford Motor Co decided to make all their cars only accept tires make in a proprietary fashion that tire makers had to license from Ford Motor Co? They could make the same arguments as Apple. They could claim that they are only doing this to 'ensure the quality of the tires used on their cars.' It would be their right to do so, just as it's Apple's right to restrict their platform. But I think there would be a ton of people upset about the move.

> Even though it's still a closed platform, more creators are making a living by creating and that's a net win in my book.

I agree that it's a net win because it's 'better than what we had before.' The issue is that it could be even better but Apple is purposefully holding us back from that because we should 'just be happy' that things are 'better now than they were before.'

Though it's a flawed analogy, it's sort of like rescuing us from slavery to make us indentured servants instead of going the full monty and setting us free. Sure being an indentured servant is a hell of a lot better than being a slave, but we could have been set free at the same time. [ I realize that in reality indentured servants were little better than slaves of a different name. I'm going off the flawed picture that most children's history books paint for us to build this analogy. ]
Sure, maybe a few TB uncompressed. Compression would be pretty astoundingly good on that sort of data though - retweets etc.
This may not be a perfectly generalized solution, but is it possible to structure your system such that when upgrading from version i to version i+1, version i of your app is compatible with both versions i and i+1 of your database and vice versa?

Say, for example, your factoring a column into it's own table. Don't just drop the column, set up a trigger to synchronize the original column value with a value from one of the rows in your new table.

You could then finally drop the column in version i+2.

I seem to remember finding a book on database refactorings that covered this technique in more detail. It was online so you could try Googling for it.
Stop trying to make the Internet boring
It looks like it specifically focuses on Rails 3. The requirements in Chapter 1 state:

"Ruby on Rails. This beta book was written using Rails version 3 (specifically
Rails 3.0.0.beta3 at the current time)."
The principles of Tufte's that I always come back to are:\

maximize the data/ink ratio - figure out how to show more data with fewer lines, symbols, colors

Clarify by adding data - show the broad trend but allow a viewer to drill down into specific areas of interest

Here is a good example of both points, 2200 data points coherently graphed:
http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0...
All the hippies are using GitHub (me included), but with free private hosting, I'm going to try Assembla out.

Good move, Assembla.
Notice that they said public tweets. This is nothing that someone couldn't scrape off the site with enough time and energy spent.
That could have been done with a tweak to push notifications, no?
I think what programming (or more accurately hacking) really needs is a better way for programmers to communicate while coding.

Easier sharing, collaborating, communicating, etc. Message boards, email lists, irc channels, instant messages, etc simply don't cut it anymore and never really were that effective at anything other than starting fights and pissing people off.
TRAVEL DEALS-LOCAL TRAVEL GUIDES
Library of Congress: We're archiving every tweet ever made
many are doing this only as a means to something else - education, rescuing family from poverty, etc

Well this is the crux right here, we can't put something like this is context without good data.
Do these workers actually make a living wage? What is the cost of living in this part of the world, and most importantly, do any workers actually escape poverty over time?
Things like these can be measured, and we should demand more from any news source.

I happen to disagree with your point of view, but it's meaningless if all we have is conjecture based on personal experience.

Edit: If your personal experience is actually from that level of poverty in that particular part of the world, then please, by all means, tell us what it's like.
It's a little different since these are only public tweets. You won't get the same level of private information that you would get if you were grabbing private letters from the mail service.
I agree. At the same time pre-mature optimization is waste.
On it...
Deleted articles are stored in the "archive" table. They may have changed this since then, but last time I checked toolserver (http://toolserver.org/) users had access to that table.