skysight-inc/HackerNews huggingface dataset:
Actually, Postgres doesn't have any great out-of-the-box solution for partitioning the database across machines. The usual suggestion is Slony, but that is no where near as robust and widely deployed as MySQL replication. The GPL licence for MySQL isn't really a problem for webapps anyway.

OTOH, Postgres does somewhat better than MySQL on a single box with multiple cores (it's fairly linear up to 8 CPU, which is much better than MySQL) - mainly because of the work Sun put into scaling it before they bought MySQL. (At least - that's according to some people from Sun who do a lot of performance work with both databases)
I still think that proper, individual desks are better.
If you can't hash join then you can't join over large datasets anyway, so sharding costs you nothing in that respect.

Right now, using off the shelf kit and doing nothing particularly clever, running a major commercial RDBMS you could do 10,000 commits/sec and handle 100T of data on a single instance. Sure it would cost you a pretty penny, but the thing is, unless running a database is the one competitive advantage your company has, you're better off keeping your people focussed on the thing that does make you money.
True in this case, but not always true, the issue often isn't upgrading the schema, it's that there can be no single schema that can hold the data because it varies on a per row basis. No RDBMS can deal with this well, no matter how mature, it's just not what they're designed for. This is trivial for an OODBMS which just stores raw objects, no matter their shape.
Karplus-Strong Guitar Synthesizer
A really interesting piece of audio software engineering. An incredible sounding guitar synthesizer!
Facebook Replaces Venture Capitalists as the Blogosphere's Punching Bag
I work with http://www.klicktrack.com/mdt/ which is a turn-key download store for artists and labels. We also offer a widget for integration with Myspace and blogs.
Wow. Who does that? I do not mean this in a sarcastic manner, but this is the first time I hear of it.
Hopefully they simplified things for the sake of brevity and they're using consistent hashing or some other predictable algorithm in practice.
States with higher salaries and more generous welfare programs tend to have higher costs of living. The extra spending is one factor which tends to drive up prices.

It works with the schools too. One of the reasons colleges tuitions have risen in recent years is because subsidies have increased. When there is more money being spent on a good or service, there nearly always an increase in price.

So, if the California government pays it's employees more money, and spends more on welfare than average, then the costs of living are necessarily going to increase. Of course, there is more to it than that. Highly paid technical workers can make things difficult for more poorly paid locals as well, but it's certainly part of it.
You don't change the number of shards, you change the number of machines and re-balance the shards across them. Shard != machine.

12 shards on 1 machine
6 shards on 2 machines
4 shards on 3 machines
3 shards on 4 machines
2 shards on 6 machines

And on and on...
You do? I thought the punchline was clear by the middle of the second paragraph. I wanted not to know, but eh.
Yes and no. This is the "small lie" mentioned in there where I said that there's only one sort of rows. The StringListColumn maintains a separate row of string values and each reference to a tag just gets an index to the tag, but tags are not "first-class" nodes in that they're not a separate node in the database.

The first time that I implemented a system like this back in 2004 I did things that way. That's in theory more flexible, but since we had a specific class of applications in mind in this case it's for our uses faster to check if an item has a given tag just by having a list of tags associated with each item. The typical access patter for us means that we're already looking at an item and just want to know if it has a given tag.
Life on the roller coaster, the highs and lows of being an entrepreneur
Two mentions on TechCrunch in one month is doing pretty good, then your server goes down and your host says all your data may be lost -- and it's your birthday!
Oh, not at all. It's all public record.

You found me off in some odd corners of the net, I see (Catholic agrarianism, in this case).
What I do is I have an algorithm. Based on the name of the website/service and its purpose, I can calculate what my password should be on it in my head.

This means that every account I have uses a unique password composed of both alphabetical and numerical characters.

I store each of them in my head, or re-calculate it if I forget them, but I actually tend to just remember them, maybe because they follow an algorithm (or rather, a set of rules) it makes it easier to remember them.

And, I'm not contingent on any software to remember my passwords.
Full security systems
Full security systems are available from professional suppliers like Building Register.
Pretty Printing XML (in Emacs)
It's easy to quantify, or should be. When a developer comes in in the morning, he or she has to recreate in their own mind the state that existed there when they left the previous evening before they can continue working. Does being able to leave books/notes/printouts open on the desk and windows open on the desktop make this process faster or slower? How much cash is an extra hour spent getting ready instead of doing actual work in the morning worth to the company?
That's why airbnb is on their way to becoming the largest hotel in the world without having to own any rooms.
That's the one basic requirement for use in a website backend these days.

There are plenty of quite profitable websites which do not have this requirement. It is almost peculiar to sites which are trying to show display advertising to groups of users larger than many nation-states.

You can make an awful lot of money with one commodity server if your business model supports it. I used to have an effective CPM of $80 and I know one which has in excess of $500. No, that is not a typo. (That is on six digits of pageviews per month.)

You know how much scaling you need when essentially get 50 cents a pageview? Not much at all.

FogCreek has, if I recall correctly, one database server. I haven't read how many total machines they're using recently but its a "count on your fingers and toes" number rather than a "zomg we need a colo facility to ourselves" number.
No it isn't, it's a nice to have but in no way is that a requirement. Most websites still run on standard RDBMs that don't scale well across many machines and frankly, most websites won't ever need to scale beyond one big beefy box. Stop drinking the cool-aid, it's killing your perspective.
Primates on Facebook
At this point no, but that's a hurdle we'll cross when we get there. In theory it would be fairly easy to split things using one machine to do the hashing from name to indexes and then segmenting items across multiple machines.
A quick side note. The number of biotech startups is mostly a function of the legal environment. The industry is more regulated that almost any other sector of the economy.
ATTORNEY REFERRAL Atlanta
Attorneys' Confidential Referral was established in 1987 as a consumer service to assist individuals in the educated and informed selection of an attorney.
I'm curious about this too, but the gap between now and being able to order SSDs on non-co-lo boxes means that we're not thinking about that too much just yet.

The place that would be most relevant would be if we were considering bypassing the file system altogether and moving to doing raw-I/O on the disk itself and tried to account for disk geometry, which would be less useful with an SSD. But in practice that's not on the near term radar anyway.
Sounds like if I were targeting your accounts specifically, I'd have a pretty easy time of it. As soon as I get hold of one or two of your passwords I can work out the algorithm and then start calculating your passwords myself.

While it might be a better approach than just using the same insecure password on every site, based on the attack suggested in this article, there isn't much benefit from your approach.
When you take complexity out of one part of a natural language, it ends up sneaking back in to another part.

English verbs by themselves are easy to conjugate, but English also uses a large number of modal verbs (I did write, he is hacking, we used to travel, she had better shut up) that these other languages lack. So while a native English speaker learning Spanish has to learn a lot of verb conjugations, a native Spanish speaker learning English has to learn a lot of modal verbs, and the special rules for conjugating verb phrases that contain them.
Great! Good luck to the team.
I'm not sure debt is a good metaphor for code complexity.

Debt grows logarithmically, if the complexity of your code base is growing logarithmically you're doing great.

The problem is software complexity grows exponentially.

And that's why you can't manage it like you can manage high interest debt.

You can try to manage it, but I have yet to see that management succeed.

Every single time a company has tried to manage technical debt, the debt simply goes unpaid.

How many people here who've worked on a sizable and mature code base can name temp files and variables that have turned absolutely permanent over the years?

That's why software complexity, in large organizations, should be approached like an infestation. Erase and eradicate on sight, don't try to save some for later.
He also has a webapp that does the same thing:

https://www.fourmilab.ch/cgi-bin/HackDiet
Ah, OK. So you start with enough shards "for ever" and not change the number?

Edit: no wait. If you can split a shard across multiple machines, what's the benefit of having more than 1 shard? Why not have 1 shard split across 1000 machines?
This will be considered off-topic, but I'll say it anyway: I really like the design of that site.
Readers in the US should be aware that by running this piece of software they are violating a patent that might still be valid.

Karplus-Strong is a really interesting example for the effect of software patents. It is a very efficient and powerful algorithm (3 voice polyphony on a Z80), but it has never been used in any successful instrument. IIRC only one commercial synthesizer manufacturer licensed the patent, but only as a defnsive measure without actually using it in a product. Just imagine all the musuc that might have been without this patent.
I totally agree... and so does the article.

Hot desking doesn't work for software engineering any more than it works for architecture. The article proposes an alternative to that, which is personal desk space on a "continuous" desk so that your space expands and contracts as your needs do.
Link to Gazelle at Microsoft Research http://research.microsoft.com/apps/pubs/default.aspx?id=7965...
Which commercial database? Any idea which web scale applications are using it, and why the other ones aren't?
Great concise article.

It's interesting how small humanity's core and even extended groups are. Modern civilization and population density is very recent in evolutionary terms, and our brain's just can't make good use of that much humanity.

But even the casual circle is interesting, I think of everybody on LinkedIn as a kind of casual circle. Business casual, where we're all advertising our business acumen to each other. Both as desirable employees, and as desirable people to work for.

Do large twitter circles advertise casual habits? To what end?
i found it suuuuper hard to concentrate when i worked in an 'open cube' layout. i can only imagine that losing more space becomes stressful, like trying to find routine and focus while working out of coffee shops. it's surprisingly difficult!

the past few months i've been working from home with my husband on a joint venture. we both appreciate the flexibility of interaction levels, from working in separate rooms to putting our heads together.

i think i'm not alone in this need for concentration and chill, while still appreciating the benefits of neighbor nudging.

the whole interaction thing is of course much easier with just two people, but i think having flexibility of space and schedule is what makes us successful. and respect. nothing like working with your significant other to truly learn communication skills.
I have a tiered password system, so I'm interested in the advice here and (candidly) unlikely to follow it. That being said, if you're putting email accounts in your low-security tier, revisit that assumption. Now. It makes every password which can be reset or recovered as secure as the least secure site you have ever signed up with.

Think of how much fun life would be if you woke up one morning and someone had compromised gmail, had godaddy send a password reminder, and then used your credentials to initiate and then authorize a transfer of your business domain to their registrar. Then a month from now you get a call: $2,000 or your site goes dark within the next minute.
i think this is one of the best articles/emails i read in a long time. this is advice for becoming a cockroach and equipping your startup to survive the nuclear winter. I've actually been waiting for PG to write something that addresses the current economics for startups and what the huge batch of first time entrepreneurs are facing. its great to be the inspiration for folks to quit their jobs at mega Co, when times are good for startups, however their needs to be an equal amount of guidance and insight for what to do when times are tough. Jcal has stepped up and let us all know how we need to shift our priorities to survive. I was deeply inspired.
Nope, or at least none that called themselves such. We tried neo4j, which exploded trying to import data on the order that we're working with and a couple of RDF databases, which survived the import, but were a couple of orders of magnitude off from the performance we were hoping for.

After writing some 8 different backends for our store class and none being within an order of magnitude of our own prototype for the sorts of applications we're doing, it seemed more fruitful to round out our own application rather than continuing the seemingly endless recurse of possible data backends which ranged from mildly to amazingly disappointing.

If you've got something specific that you've worked with in the past that you think would be worth our while to evaluate, I'd consider investing the time to try it out. But just that there exist more options that we could evaluate at the moment doesn't necessarily imply that it's reasonable to keep writing new backends, which sometimes take a non-trivial amount of effort.
Taringa - Pelicula: El Pianista (The Pianist) DVDRip Rapidshare
Good interfaces create good habits
Look at that "hasn't slept in six months" thing again. This tends to play on someone's emotions quite heavily.

To be fair, it is quite possible that in a media-based startup like a newspaper, the company did make these people a star. However, I don't think I'm seeing anger as much as "oh shit, what do we do now?"
Has anyone tried the Shangri-La Diet? Seems too simple not to try it. From wikipedia:

"The diet itself consists of taking 100–400 calories in the form of extra-light (not extra-virgin) olive oil or sugar water per day, either all at once or spanned throughout the day. This must be consumed in a flavorless window, which is at least one hour after flavors have been consumed, and at least one hour before flavors will be consumed.[4] The consumption of these flavorless calories supposedly lowers the set point, and therefore, lowers weight."

I just mention this because I've read the authors paper "Self-experimentation as a source of new ideas: Ten examples about sleep, mood, health, and weight" and it seemed really interesting. Available here:

http://sethroberts.net/science/
So the flaw is that if one site is hacked, all the sites are hacked? And because of that, people will start sending spam from my account?

That argument does not make sense. Most sites do not send things out using my name, and those that do have very limited options for spammers. For example flickr, or Hacker News.
Open the Future: John Henry was an Audiobook-Readin' Man
This experience reinforces my belief that it's better to be quick than brilliant...

I've come to the same conclusion during the past year and a half of working on Justin.TV. If I was writing a list of pieces of startup advice, this would be #1.
Yeah, English has some gotchas too, but I don't think you can say it's a zero-sum sort of thing where they all balance out exactly.
> Technology seems to make cultural borders disappear, [...]

Can you expand on this? I'm uncertain as to what you mean.
It's kind of hard to find an article with pictures - but check this out - 3min video with great pics of the tools:
http://www.colorado.edu/news/r/1124c0243883c267a7759da4bc4a2...
hjvh
That's great.

I've never heard of Dieter Rams before this thread; I'm not hip to the design world. But some designs do raise my pulse while I looking from the sidelines. Henry Dreyfuss' Western Electric Model 302 phone [1] is a good example.

That phone has the nostalgia effect; that's good in design. Design should be nostalgic, in the sense that it should recall things that you're familiar with and feel comfortable around.

It's useful to look at design as communication. Your trying to same something with the right metaphors. But in this case, it's with three dimensional metaphors instead of words.

[1] http://www.telephonearchive.com/phones/we/we302.html
I'm assuming this is a proprietary, so any other comments regarding RDF databases would be helpful. I've used ARC (arc.semsol.org) before, and it works adequately. Though I haven't run performance tests personally, ARC is based on PHP so it probably gets blown away by this C++ version.
Men divorce help
men divorce help, divorce advice for men, My Divorce Plan, men divorce, divorce help for men, divorce advice men, mens divorce advice, divorce tips men, help for men in divorce, divorce advice, divorce help
The above table wasn't too clear, but my understanding is that a shard exists on only a single machine, but that a single machine could host multiple shards.
This is really interesting...could you do the same thing using S3 instead of MogileFS? The advantage would be cost and simple scalability, especially if you're running on EC2 so you don't have to pay for all the back and forth transfer from EC2 to S3. Concerns might be latency issues and whether this would scale to billions of objects. Other than that, it would seem like you could run a pretty huge site off of just a couple beefy servers.
I find this interesting because for a previous project, we had to manage separate database instances for each client (regulations). Even though we had a database management tool that allowed us to push schema changes out to the various DBs automatically, it would be better to not have to push those changes out at all.
There are better products, impossible products, and unthinkable products. Better follow an established trajectory in an industry, impossible products find a way to relax one or two constraints that better have taken as fixed, and unthinkable products--typically developed by someone from outside the target industry or the result of repurposing a product from another industry--create a new category of product. Some examples of unthinkable products: IDDQ testing in semiconductors, the Reebok Pump shoe, or Ford realizing that a meat packing plant's "disassembly line" could be run backward to assemble a car.
> You'd have to go to someplace that's older,

Actually, you don't. As I pointed out, almost every college town has one, and it's reasonably cheap. Small towns have/are them.

I pointed out that every largish US city has them. (Oakland and Berkeley's are better than SF's.)

Heck - almost every south (sf) bay city has one or two. I forgot to mention that Menlo Park has at least one and I only remembered two of PA and SJ's - both have at least three, and I'm almost certainly missing more in SJ than PA.

> In other words, people aren't really choosing

Except that they are. They are readily available, a huge fraction of people spend a couple of years in them, and yet most people don't choose them for the majority of their lives.

It's interesting that (some of) their advocates can't see them unless they're the only option.

> Also, what's with the "diverse" scare quotes?

They're not scare quotes. I'm pointing out idiosyncratic usage. "Diverse community" typically means that the residents are ethnic and/or culture mix. What he's referring to is typically called "mixed use".
This Internet fix is no pipe dream
Am a bit surprised at the omission of the Lambda papers. http://library.readscheme.org/page1.html
"However, none of them seemed widely-used enough by large sites to inspire confidence. In the tests we read about and ran ourselves, none of the projects were stable or battle-tested enough for our needs"

Ok, just some hour ago I released the beta-3 of Redis (http://code.google.com/p/redis/ if you care) and I'm near to feature-freeze with exactly with this goal. To make it rock solid (I'm going to use it in my startup's web stuff with a lot of users/month, so I care about stability).

The question is: what's in your opinion the right path to make a system like Redis stable and reliable for the real world usage? What to publish on the site in order to inspire a good feeling about stability? Thanks
It's probably not the best way to go about it, but couldn't you implement that by just cutting the delay to zero and drawing from the beginning until the requested frame? It probably wouldn't work well if you wanted to step through the page backwards, but for just cutting to a specific frame it seems like it aught to do the job with minimal extra coding.
Question: can I store my homepage in a CDN like Amazon's new CDN??
Can I store my static html homepage in a CDN like Amazon's new CDN? How would that work? I know how it works with images and such, but not sure how it would work with html?
I've lived in SF, the East Bay (worked in Berkeley), Portland, Oregon, as well as Padova, Italy, and Innsbruck, Austria, and US cities, even the ones you cite, still aren't nearly as "walkable" as your average European town, although, yeah, they're better than something like Phoenix, Arizona.

> They're not scare quotes. I'm pointing out idiosyncratic usage. "Diverse community" typically means that the residents are ethnic and/or culture mix. What he's referring to is typically called "mixed use".

Ah, ok... actually that wasn't entirely clear to me either.
Don't try to run for office here (MA). You are way too reasonable.
High-Throughput Nanomanufacturing (with video)
You should rethink everything if you cannot provide those people personal offices who absolutely must be at the office every day.

One thing you're probably doing wrong then is thinking that everyone should be physically there every day.
I think there are cycles where technology get's to a state where basic research has huge value until you have enough basic research to to build a wave of new technology's. Once you have fully exploited the last wave of technology you are ready to build the next level of new technology. But, you rarely change technology's while there is still room to grow the old tech. Example: Modems starting at 300bps to 56k used the same basic technology, then DSL, then Fiber.

IMO, there is value to understanding the limitations to existing technology before you start doing basic research.
Parenting advice from AAP. This is a sample article on discipline
As an about-to-be-doctor, I still find myself knowing these things but wanting to review them, not only because these issues come up 10 times a day in every pediatric clinic, but also because I'm a parent. Good to be able to refer to. But Google would never find this mini-library of sound parenting advice.