skysight-inc/HackerNews huggingface dataset:
Ask HN: Tools/approaches to automate complicated software pipelines?
What approaches and tools do you use to help organize and run complicated pipelines and batch jobs? For example, I'm making pipelines for bioinformatics analyses, which we want to be maintainable, self-documenting, reproducible, and easy for end-users to run. These would run a large number of different programs and scripts on some input data, potentially with different permutations of parameters.

Some of the kinds of features that would be nice are:
* Running jobs multiple times with different parameter settings
* Branches, so that later parts of a pipeline can be run with different parameters while sharing results from an earlier stage, or running the same analysis on inputs generated using different methods
* Running jobs on a cluster / integration with LSF
* Logging runs and keeping track of results
* Keeping track of default/standard parameters and recording documentation of options
* Tracking success of stages and automatically re-running if necessary
* Tracking program/data dependencies and versions
* Control / reporting through a web interface as well as the command line

Obviously all of this can be done with shell/perl/python/etc. scripts, and with make/Ant (although I think that there is more to this than managing dependencies in the manner that make is usually used). However, there's a tradeoff between the simplicity of a static script vs. flexibility, and it seems like writing a generic system with such features is not an entirely trivial exercise. Making specialized scripts for each pipeline is an option, but risks degenerating into a mess of special cases and gets harder to document and control.

One very lightweight tool I came across that looks interesting and seems to have some of this functionality is ruffus http://code.google.com/p/ruffus/ I'm surprised that there haven't been more efforts like that.

What other approaches and tools do you like to use?
If we were to use MySQL, we'd be using multiple complex queries to get the job done. From what I've seen in New Relic on Heroku, queries have generally taken about 40-50 total for one request on PostgreSQL.

We're building a general purpose method of running Ruby applications, so 300 ms is certainly a possible request time (including database, processing things, and rendering a view) for some applications that we may be running.
Watch the Rape video game controversy
The idea here is using "Stochastic Functional Programming" (see some of the work by Goodman, Mansinghka, Roy, and others at http://web.mit.edu/vkm/www/ ). Basically, you write down your AI problem in a "forward" direction, suggesting how the data came to be. You then "fix" the outputs. The engine generates a distribution on "possible program histories" that preserves the statistical properties you care about.

For people familiar with inverse methods, what they basically have here is a generalized inverse solving engine that obeys the laws of probability.

Of course, right now, this approach ("solving AI by running programs backwards") is a bit slow, but some startups are rethinking the entire computing stack ( http://www.naviasystems.com ) in an attempt to rectify that. [I'm one of the people at said company]
Another Brisbane boy here. HN meetup would be great, since we seem to have a few locals. I can probably organise a casual venue one evening in the Valley? Maybe chip in for some drinks or dinner?
Slightly misleading title: the article is about modifying GUI program's input and output. Which, if the massive success of "screen macro" programs, and LodeRunner are any indication, is a lost cause.

The electrons chopped down to make that article would be better spent on actual open-source programming.
The reason why I mentioned Oracle in the demo is precisely because of the whole CAP debacle. So many people talk about CAP as if the SQL database was itself an impossibility, much less Oracle's scalable offering - but you can pick up the phone and an Oracle salesperson will sell you the physical device that you're implying can't exist.

So, if somebody misquotes the CAP theorem, it's an indication to me that perhaps the thread isn't going to go anywhere. I'll point them to the place where they can read about it (the proof was where the CAP theorem was formalized, not Eric Brewer's original presentation), and move on. In this case, that wasn't fair to btilly, who just made a typo.

So, as I said in the presentation and elsewhere here, if you have a hair-on-fire problem with scaling your database, and are willing to be an early customer of our new database, and would make a good reference customer in a few months, we'd love to work with you today. Otherwise, you'll have to wait till we open it up more widely.
There's a lot of new interest in probabilistic programming languages, which is what Noah et. al. have developed. See http://probabilistic-programming.org/wiki/NIPS*2008_Workshop for a workshop from about a year ago that explored the space. Probabilistic programming languages enable you to express concepts such as recursion and universal quantification in a way that Pearl's graphs don't easily allow.
OneNote -- it has been said it's the best piece of software to come out of Redmond.
(admittedly for OLAP workloads)

Well, you say that as if the difference between OLAP and OLTP was a minor thing. As far as I know OLAP is usually done on column stores with quite different properties to your regular MVCC RDBMS.

But well, I am not a database engineer. So, at least that was a response, thanks. I'll be looking forward to what you can deliver.
Maybe it's just me, but maintaining a task list is about as troublesome as finding one that does just what you need.

I get e-mails for business related tasks. Outlook flags posts sent directly to me (as in, I'm in the TO: box, not part of a group or CC'ed) and I'll check them out.

Anything else is stuff I want to do. I don't often forget that I want to do something, thus no list necessary. If I do miss something... well there's always free time in the future.
Impressive, yes, but they're not the first self-domesticating species.

Cats pulled this stunt 10,000 years ago.
I asked him about this a few months (a year?) ago on the Business of Software site. However, the rate I suggested (a SWAG of $100/hour: hey , what do I know about SEO consulting rates :-) was too low to match what his BCC was bringing in at the time. I was duly impressed!!

Congratulations, Patrick.
+1 Minnedemo is probably the best networking event for startups in the cities
Crypto Graphics
This is great news. I blogged about this back in October - glad to see Google have taken this issue on.

http://benmetcalfe.com/blog/2009/10/my-gmail-password-scares...

It will be interesting to see how successful adoption of IMAP-via-OAuth will be.
There is a lot of theory behind fuzzy logic as well and you get the bonus of it being very simple to implement. Thus it's use in appliances where cheap/tiny processors are the norm.
Second the vote for OmniFocus. The key for me was synchronization between multiple computers and the iPhone. The iPhone app seems a bit like an afterthought (it's very slow to update, even when using their suggested tweaks), but synching between computers works great.
5 Rules for How Start-Ups Should Work With Consultants
What Does Functional Programming Mean?
Yes, I wasn't thinking about "DreamHost VPS" (although come to mention it I do remember them offering such a service).

With most (read:good) VPS offerings the support guys won't have access to your VPS password (root, etc) and so from that perspective this would be safer and more secure for the OP. Shared hosting, by it's nature, holds a much higher chance of the hosting company holding your password in the clear/decryptable state.

My apologies if I wasn't clear in my original reply.
http://www.makeramen.com/noodles/
Homoiconic: Significant whitespace
頑張って。
I haven't had cable in over 3 years, and I don't miss it for a moment. Between Boxee and Netflix Streaming via PS3, it's a complete waste of money.

Per the live events issue, that's largely a non-issue for me because it almost never fails that a friend is throwing a party or we're meeting at a bar. Olympics are the exception. For football games, fight night, et al, the atmosphere with a bunch of cheering fans is just so much more enjoyable.
There's also the much lighter FlashDevelop.

The AS editor built into Flash should not be used for more than a screenful of code.
For what it's worth, there are many mutual fund -> hedge fund relationships similar to what you have described. Average investors can't invest directly in a hedge fund, but they can buy into a mutual fund which has major holdings in one (or several). It's not the one-to-one relationship you described though.
I've got several of the sit/stand desks where I work. Like the Biormorph, but one surface. There are several manufacturers of these kinds of legs (Steelcase and Linak). Check your nearest large town's office furniture retailers to see if they have a used department - that and auctions of office furniture.
There are managed VPS offerings. Another option would be to hire someone to set this up (prob 1hr work for someone remote).

Sure, I totally understand that this is over-kill if you are just running a personal blog - but then that's what wordpress.com etc are for. I'm working on the assumption that most people here on HN are either running a startup, a public facing project or building their personal brand -- all of which IMHO warrant setting things up properly.
What if I want to do something else with my photo, that iPhoto doesn't yet have a button for? How does a universal interface look like, that allows arbitrary other apps to get photos from iPhoto?

The email export function is implemented with AppleScript. To add support for Thunderbird (for example) just means writing a similar script for it. Indeed, it appears that someone has already done so:
http://softpixel.com/~cwright/programming/iPhoto/

Note that this is a minor hack, to get the button within iPhoto to support Thunderbird. It isn't necessary to get the same function as a regular Scripts menu item. Such scripts can be written in AppleScript, Ruby[1], Python, and Objective-C.

This is not really a solution to the more general problem being discussed, but I thought it worth mentioning.

[1]: http://developer.apple.com/mac/library/documentation/cocoa/C...
Upvoted just for the fabulous essay you referenced.

I'm not sure what sort of investing you have in mind - small face-to-face angel participation, or working the treasury bond market. Nor do you say much about the size or timescale of the investments you'd like to make. And I'm in no position to give you expert advice either; all I can say from experience is that nonprofit financial advisers generally seem more helpful with advice about saving rather than investing (eg what sort of IRA and 401k options are available) while wealth management experts either charge a hefty premium for their opinion or function as sales staff for a range of vehicles and funds operated by an employer.

since patience is a key aspect of investment, I think you might as well park your cash for a few months and begin with the basic-but-essential advice offered by the SEC: http://www.sec.gov/investor/pubs/begininvest.htm. This will teach you the fundamentals of how to read a balance sheet or 10-k filing, how to make sense of popular metrics and acronyms and stuff like that. At the same time, take out a subscription to the Economist, Wall Street Journal or Financial Times. there's another newspaper called Investor's Business Daily but I don't rate that very highly. See what insights you can glean on your own for a few months, then buy a few serious-minded books rather than the get-rich-quick variety.

At that point, go talk to some financial advisers, and it will be much easier to assess the quality of their advice without feeling baffled by jargon or bamboozled by a sales pitch. I know, this is kindergarten-level advice from a first-grader - sorry if you've already done all this and are looking for a more technical insight.
but you can pick up the phone and an Oracle salesperson will sell you the physical device that you're implying can't exist.

Well, thing is, part of your claim (if I understood it correctly) is that I just throw my data and my queries at your wonder-device and it will magically be fast and scale. You never mention any special care I must take, you even explicitly say it's not partitioning.

This is decidedly not the case with what said Oracle salesperson will sell to me. The sale will normally include an expensive support-contract and probably a dedicated DB Admin. Because operating a RAC- or Exadata Cluster is, for all I know, far from trivial. You still have to make a bunch of critical decisions and tradeoffs upfront or it will probably suck.

Moreover you're effectively saying that you solved what Amazon and Google either couldn't or didn't bother to solve with their SimpleDB and Appengine Datastore offerings.

Still sounds like quite a bold claim to me. But as said elsewhere, I appreciate that you're standing by it and will keep looking forward to what you will unveil.

Just one more thing that I really don't understand: If you have this doomsday device at your hands, then why on earth do you bother with MySQL hosting as a secondary product?
I've tried a number of different options - online and local files - but I've settled on Vim's taskpaper plugin (see link below) and a tasks.taskpaper file that I keep in sync across machines using version control. I don't actually even use Taskpaper itself, since I don't really want another application for this - just a file with a reasonable, and hackable syntax. (The syntax is simple enough that it's dead easy to whip up a script for any kind of display or search or munging you might want to do with your tasks file.)

Taskpaper.vim plugin: http://www.vim.org/scripts/script.php?script_id=2027

Recommendations for highlighting taskpaper files: http://www.praytothemachine.com/evil/2007/11/06/taskpaper-an...
You could - but it would need to be every 1 minute. Google tears down your app after 60-90 seconds of inactivity.

Some folks have resorted to pinging every minute but Google discourages that since it ties up their resources. If everyone pinged their site once a minute then resources would never be freed....

Here's a good thread on the issue https://groups.google.com/group/google-appengine/browse_thre...
"The industry remains strong, Lem said, but it has seen waves of layoffs largely because of company mergers such as Switzerland-based Roche taking over Genentech in South San Francisco."

http://www.mercurynews.com/search/ci_14508960
"Only $4 of an iPod that costs $150 to produce is made in China"
not from this company

You still have Windows Phone, Android and BlackBerry.
The economics of non-profit newspapers
It was fun to build. :) Check out how the a/an changes when you switch to "enterprise" and how we add an extra question if you're non-personal.
Hmm, I've both watched the TED talk and read the article pretty carefully and as far as I can tell he's been saying "Science can give us answers to moral questions" all along.

I also don't see how Harris is defending classical utilitarianism. According to classical utilitarianism the needs of the group are far more important than the needs of the individual, and as far as I can tell Harris doesn't subscribe to that line of thought at all. He cites the golden rule as a decent guideline, and I think the golden rule and classical utilitarianism are not compatible.

He also used the words imbecile and delusional in very different contexts. He doesn't claim that (again, as far as I can tell) that people who don't share his point of view are imbeciles, he just considers some points of view idiotic, such as the extreme form of moral relativism where every action can be justified by simply accepting it as an axiom of "goodness".
What qualities will potential parents pay for?
IBM Card Sorters
You pretty much described http://culturedcode.com/things/iphone.

These were the best $10 I spent on an iPhone app by a very wide margin. It is simple, but not simplistic, and it is really well designed and executed.
https://www.rememberthemilk.com/

Quite simple interface. Write your list, easy to manage times with the shortcuts (just go: ^ then type your due-by date, ie: tomorrow or friday and it'll set the date), and you can print it off if you still enjoy the nostalgia of pen and paper
Congrats Patrick... I've followed your posts since the Business of Software days and you've been an inspiration to me.

I haven't had your courage though, I quit my job in Japan before working on my own projects... Too many hours of overtime killed me afterwards for side projects...
People problems in Japan
> "Where did he state that he feels there is a 'high chance?'"

He didn't but the fact cryptnoob felt the need to mention it suggests he/she is concerned about. I myself don't go around registering multiple accounts "in case something happens" to one of the accounts. Do you?

>"WHOIS records are only 'supposed' to be used to contact the site admin, etc"

Dude, cry me a river. I own a shit ton of domains and so I get that spam all the time. The spirit of the rules around public record of WHOIS data (for com/net/org at least) is that someone can be contacted for technical and administrative reasons about the domain. It's a reasonable rule and so if people fundamentally disagree with it perhaps they should lobby ICANN/etc.

From my own experience running a web hosting business in the past that most people who anonymize their WHOIS details are doing so for suspicious reasons.

> "So the fact that a person wants a site to be 'anonymous' means that it's by definition a sleazy site?"

No, and my apologies for not being clearer on that - perhaps the word "ghetto" wasn't what I meant. What I meant was shared hosting is like being in the ghetto - you are at the mercy of your neighbors on the same server. An account on the same box sharing warez forums is going to affect YOUR site's performance.

> "Should the site admin be forced to be contacted/harassed by possessive (and potentially violent) men that are trying to find where their girlfriend/wife that ran away is?"

As someone whose domestic partner is a leading voice in women's rights online, who receives regular abuse and has had numerous death threats, I can assure you I am very familiar with this subject.

There is a difference between anonymous (read:un-contactable) whois vs using a business address or mailbox where you can receive communication but is not your private residence, etc.
Is there a throttling component to this? I seem to recall that some people used to encounter problems when trying to use a script to access GMail (for backups). I don't see any API limits offhand.
I think your performance here stems more from configuration than from MySQL vs Redis. If you're using Redis in-memory, if you dedicate the same amount of memory to your database, you should get roughly comparable performance between MySQL/PostreSQL and Redis. The under-the-hood differences if you're in memory just aren't that great, particularly because modern machines have such ridiculously powerful CPUs. Yes, it sounds like you'll have to do table join(s); yes, the joined tables need indexes; yes, you have the SQL parsing overhead; yes, it's a B-tree rather than hash-based indexing. But I'd be very surprised if you ended up with the order-of-magnitude differences you're suggesting.

One of the things we're working on with FathomDB is ways to identify slow queries and help figure out what to do about them. If you're game, you can contact me with the queries and the table structure you were using in SQL and I'll try to figure out what's going on!
The Leaf is more of the "microwave oven" of vehicles. It's vastly more convenient for most trips, but there are some things it just can't do.

It's been said that the best range extender for an electric car is a rental car. The net present value of two weeks' car rental per year, forever, is probably less than the $10,000 difference between the Volt and the Leaf.

The Volt will also require regular oil changes and other engine maintenance. About the only regular maintenance for a pure EV is putting air in the tires.
The case against the hockey stick
I don't understand it: why would the intermediate goods not be accounted for? Some stuff goes into a country, some stuff goes out of a country. The difference in value is probably what people are looking for? Why should it matter if the goods are "intermediate" or not?
No, I'm going to buy a Google Chrome PC, possibly in tablet form but probably clamshell.
Not in any one single way :-)

I think you might want to look at http://www.toodledo.com/ - it is actually quite powerful (aka "complicated") but the amount of complexity is user configurable. So if you don't care for a certain category, you can just tell the system to ignore it.

The main advantage of something "online" is that it is easy to attach supplementary information to a to-do item - a file, a message-ID...

I use a hybrid of post-it notes, online calendar, paper notebook, toodledo, and e-mail. GTD it isn't - but it works.

[Edit - toodledo also allows you to assign tasks to other users. It is not quite implemented in the way I would like, but their deveoper is very user-friendly and they are always working on it, so I am optimistic.]
I dont believe you can compare Facebook and Dreamhost, since users feel more entitled to demand features when they are parting with $.
Are you staying in Japan, or moving back home? If you're staying, how did you pull that off (visa-wise) without a "regular" job?
The Hit List

http://www.potionfactory.com/thehitlist/
California USA vs. Ontario Canada - Which State (Province) Is In Worse Shape?
Benjamin Graham's "The Intelligent Investor." The Rich Dad, Poor Dad series. Learn solid economics, accounting and law; at least enough so you can identify good advisors in these areas.
Designing with color-blind users in mind
I haven't found anything better than a written list. There's something really tactile and satisfying about crossing items off it.

Every few days I rewrite the list onto the next page, drop things that are no longer relevant, expand items that are sticking around. The physical nature of the list, and the ritual of recopying it are really powerful, to me. Plus the fact that I recopy it frequently helps keep it from getting out of hand and naturally prioritizes it.

I say this, but I've been out of the habit lately. I need to start doing this again.
One of the best sources for investing wisdom is Warren Buffet himself. Read his annual letters to shareholders (http://www.berkshirehathaway.com/letters/letters.html) and his "owners manual" for stock in his company (http://www.berkshirehathaway.com/).

His way isn't sexy, but it works and makes sense.
org-mode In Your Pocket Is a GNU-Shaped Devil
37signals Backpack. http://backpackit.com/

For more information read this article: http://patrickrhone.com/2006/06/27/backpack-new-gtd-implemen...
Of course, this decade, we actually have OOP, so OO databases make a lot more sense.
Why Haitian bus owners spend so much on murals
NASA to aid regulators in Toyota probe
I am a huge fan of DevTodo:
http://swapoff.org/DevTodo

It's a todo list designed for development projects (you can put a todo list in any directory), but I use one in my home directory for just the basic stuff: add and mark complete.
The bill does not set new requirements at all! It just explicitly authorizes the SEC to adjust the income & net-worth requirements for inflation. The SEC was already empowered to alter the requirements independently of legislation, but has never done so.

The figures quoted are pulled out of their hat -- they just took the old figures and adjusted for the last 30 years of inflation since they were set. Nothing in the bill says the SEC has to do that.
How's this different than apache mq?