skysight-inc/HackerNews huggingface dataset:
They explained the sharding in the text - I glossed over it on first read-through too. When they say "join" in quotation marks, they don't actually mean a join in the MySQL sense. Rather, the initial call to user_id_index.get_all reads all entity_ids for that user into the Python code (they say it consults all shards for this, but isn't the index sharded on user_id, so all entities for a given user_id live on one shard?). The Python code then uses whatever shard function applies to the entities table to query its database backends, selecting the relevant entities. Then the Python code filters the returned records by the indexed field (in case the indices are out of date) and returns it.

As for disadvantages - well, it's denormalized, for starters. ("Normalization is for sissies", says Cal Henderson.) If an indexed field changes, you need to update it in both the index and the relevant entities. There're also a bunch of little inefficiencies, places where they traded performance for scalability. Imagine if you naively plugged this engine into an app with only 10 records: instead of a simple index search, it'd have to go to the index table, fetch the relevant entities, go to the entities table, fetch them, filter on indexed value, and then return them all. But then, if your database fits on one machine, you don't have the same sort of engineering challenges FriendFeed does.
You've said it in a nutshell. Exactly how it really is. PG should include this some where. Many people are blinded by the ways media portray stories and the reality behind them!
I agree, but howerver spaces is the one exception, it's totally useless compared to a really working virtual desktop like gnome, kde, or even better wmii. I could never get over the fact that spaces magically throws you around between desktops when klicking on program icons, and is totally opinionated on on which spaces to start program windows.
> I don't know much about how state governments work - but it's interesting to see how some states can have no income tax and still be better off financially than high income tax states like Cali.

One interesting question is why should state and local taxes be deductible at the federal level?

Yes, I know that folks are paying them, but presumably they're doing so for the benefits. Why should folks in low tax states pay more federal taxes to cover state tax benefits that they don't get?
This is solely for one ITunes feature ('Genius' recommendations). It's 326 words.

At the bottom of the page, a link to the web site's 'Terms of Service': 3,258 words. To the right, there's ~40 links to other terms/policies.

There you can find the ITunes Store Terms and Conditions: 17,807 words, for US users.

When you launch the ITunes installer, you have to accept a License Agreement of 4,053 words.

So I don't see Apple doing any better than others here.
>> $1000 a person isn't even that much - each citizen of the USA would have to pay $35,462.89 to cancel out the country's national debt.

This is mixing apples and oranges. The $1,100 deficit is this year's deficit. The US's $35K/person is total accumulated.

>> i'd much rather bail out states than corporations.

Why bail out any of them? Whatever is subsidized you get more of. Bailouts will lead to even more fiscal irresponsibility.
Yes! I knew I wasn't the only one who thought that was worthless. Sure, it's a sweet deal from a tax perspective, but only if you trust the government.
I'd add the book "Thinking Forth", which I just read again recently:

http://thinking-forth.sourceforge.net/

The ideas present - which are much more craftsmanlike than an academic paper - are easy to absorb by skimming, and they're still relevant even after 20 years of research/fashion trends in programming.

Plus Forth itself is, in a lot of ways, the living embodiment of the YAGNI principle, which can make it look uncomfortably-to-incomprehensibly spartan to outsiders.
Thanks for sharing. That was very educational.

We store data in a really similar way at Spock, pointing all our reads at a big MySQL table which essentially just has an id column and a really big text column containing all the values, JSON-encoded, w/roughly the same number of rows as the FriendFeed table. Our DBA wrote a blog post about it a while back: http://www.frankf.us/wp/?p=17

The way you guys create/delete indices and avoid all the replication nightmares sounds super cool.

All of these solutions ultimately end up sounding like a poor man's Big Table.
>> But only 13% of savings actually went to the citizens. 87% of that tax money went to coroporations with large tracts of land (agriculture, warehouses, ...).

Are you saying that not taking money from people or corps is the same as giving it to them? I didn't steal $1000 from the person next store, and therefore I gave him $1000?

Corporations are leaving CA by the droves. Neighboring states are advertising in the local papers to lure them away. Over 7,000 companies have left in the last few years.

Your post makes it sound like CA's problems are mostly just an accounting matter of how the money is routed around ("Many would actually argue that prop 13 destroyed California"). Is this what you're saying the main problem is?
I live in a country where software and technology is not really blossoming. The government is trying hard to promote the tech business in the country. But, nothing is really working at all.

I stopped school for three years before I came to the US to further my studies here. In those three years, my plan was to start something that could make me millions. Soon, I realize that this is a dream, I only managed to make decent amount of money through freelancing. But, I can tell you that you hardly can go anywhere if you are working alone. Luckily enough, I did learn some programming in this period.

Then, I decided to come to the US. English is not my first language. I worked my ass out to get decent score for my SAT. And, luckily enough, I got accepted into one of the best CS school in US. Here, I met with my current cofounders with my new startup. And, this is another new process that I have to dedicate myself to learn something new again(Communication skills, new culture, etc.).

I never regretted to further my studies to college. I met new people here. I learned new knowledge(Remember, CS is not all about programming). And, I believe that this year in college actually helped me more than the three years I wasted.
A good analogy is Linux. The Linux camp can't understand why people don't switch. Haven't they done everything they were supposed to do, easy installers, graphical desktops, etc? Well no, like those bankers, they're just going through the motions without understanding that at the end of the day, it all boils down to human factors. You can't solve that with equations. You have to grasp it intuitively.
financial engineering is not like civil engineering. It is not about helping society. It is about making money

You don't think Halliburton, one of the biggest employers of civil engineers, is about making money? Interesting.
I've found this can be a really big "problem" as well. And it's not just the temptation of getting sucked yourself into a cushy job. It's that much of your network, your closest friends, also have that same hot degree and face the same temptation. Holding a startup together is hard enough in the best of times; when everyone on the founding team is continually bombarded by really tempting offers, chances are someone will crack. (Unless you're in a place where Silicon Valley where working on a startup is definitely "cooler" than that cushy job.)

My startup died when my cofounder got into Harvard Business School and I got into Google (we're both Amherst grads). 99.99% of America would not call that a failure. But the bottom line is that my startup's still just as dead.
Aurora: Class types and invariants in JavaScript
Ghosts in the machine - avoid using window.sun in Firefox as it starts the Java engine
"The most dangerous thing for the frontpage is stuff that's too easy to upvote. If someone proves a new theorem, it takes some work by the reader to decide whether or not to upvote it. An amusing cartoon takes less. A rant with a rallying cry as the title takes zero, because people vote it up without even reading it."

When I read that, I was immedately struck with the idea that story votes have a "velocity in voting" and that a story with a "high voting velocity" is probably fluff, while one that has a "slow voting velocity" may be significant and perhaps some measure of this could be taken into account, kind of like the personal shields in _Dune_.
It’s akin to saying that If you don’t have a degree you are more likely to be an entrepreneur because your odds of getting a good job in a big company are very slim.

I would have agreed with you had Bill Gates, for instance, never been to Harvard in the first place. If he had applied for a job at IBM and had got spurned. If Microsoft was something he came up with as a last resort after failing to bag a job elsewhere.

I presume that’s not the case.
Even easier for a startup. All the fun employees benefits like "2 weeks notice" don't apply until a company is a certain size.
By afford do you mean, can pay for without racking up student loan debt? I don't quite understand the qualification here.
That's reassuring, I thought I was the only crazy fool who was storing json objects in database columns =). We do something similar for some of our data models at thesixtyone.com. It's really nice for not having to bring down the site for schema upgrades.
It undoubtedly helps. There're plenty of counterexamples though: Steve & Steve from Apple, for one. And I don't think it's more that they have a higher chance of starting a business, it's more that they have a higher chance of finding the money and time to succeed with their new business.

Also - the elephant in the room is plain old intelligence. Ivy League colleges tend to admit smart people, at least as closely as they can determine (there're always a few doofuses who sneak in, but they are a smaller proportion than at community colleges). Smart people have a higher chance of trying to start their own business, and a higher chance of being successful once they've tried.
That wikipedia article says the 14th made some state debt void but says nothing about other state debt or contracts being non-dischargeable/non-changeable.

There's a 10th amendment issue (limiting the power of the federal govt to order state/cities to do things), some federal bankruptcy law changes (after a company did what Vallejo wants to do to labor contracts, congress changed what companies could do, but not municipalities), and state law provisions (which may cover labor contracts by municipalities).
Pretty much it's MySQL only.

Most of the MySQL posts here are actually about clever ways to do things most DBAs and developers have taken for granted for years. Do you wonder why no-one seems to be writing blog posts about "sharding" Sybase or DB2 or Oracle or Postgres...?
But would you, if MySQL could do hash joins?
It's not necessarily an either/or proposition at this point. Without a strong alternative, I would say go to college and continue to work on what you've built so far and keep your eyes and ears open for other opportunities. If something you have takes off or you find a very compelling opportunity and college is conflicting then make a choice at that point.
Smart move if you ask me...I mean, it sucks, sure, but you get to keep things afloat, and everyone's losing out anyway.
When they say it's sharded on "user_id" what they mean is that's the field that decides which database the record is stored. It might go something like: if the user starts with 0-8, store in DB1, otherwise, store in DB2. This is up to their Datastore controller to decide how to hash based on the user_id and the number of databases.
This is as good as requesting it. ;-)

The thing is - decisions at Google are all made in a data-driven fashion. So "this random person I met on the Internet wants us to implement keystroke playback" doesn't carry much weight. "Someone on Hacker News suggested GMail and GDocs implement keystroke playback, and then 900 people upvoted it, and then it got picked up by Reddit and Del.icio.us, and then the NY Times wrote a story about how it's the next big innovation" would probably be pretty convincing, though. So if you want it, convince other people it's a good idea, and convince them to express their support of it. Somebody at Google will see it eventually.
Silicon Valley: down but not out?
The iPhone in Japan remains one of the worst phone experiences available on the Japanese market, on just very basic important features.

"push email" is not a concept here, because at least with Docomo and AU, all email already arrives on average within a minute. The iPhone (and Softbank generally) taking 15 minutes and manual checking is a joke.

Signal strength is laughable. Softbank has the worst signal strength of the three major carriers, and then compared to other Softbank handsets the iPhone even gets worse reception still.

Emoji support, even having been added, is not working correctly, as the moji don't look anything like the ones that other handsets use, so you have no idea what the person originally sent you.

Email is not searchable, yet most japanese people use their keitai as their primary email reading device.

These issues may all be entirely Softbank's fault, and nothing to do with the device, but there is no non-Softbank iPhone.
This is true. Right now, people like Mark Zuckerberg are looked to for inspiration among young people. It's all fine and good that he's created something big, but has it really done anything for us? If I want inspiration, I'm more interested in looking to the engineers who put a man on the moon 40 years ago.

As useful as sites like Facebook, Twitter, and Youtube are (not to pick on any of them), their achievements are nothing compared to putting people in space, designing planes like the Dreamliner and the A380. or even the engineering that went into the Chunnel.
Yeah. They actually explained that in the article too - the shard number = user_id % num_of_shards. So user 1 is on DB1, user 2 is on DB3, etc. If they have 10 shards, user 11 starts back on DB1 etc.
"Cutting more than 500 calories/day doesn't lead to sustainable results."

Sure it does, if you KEEP IT low (enough). That's the problem with viewing a diet as something you do for a while, and the return to what you've always eaten. If you want to follow a diet to get permanent results, you need to do it for the rest of your life.
That's a fair point, but the other datacentre attracting factor I should have mentioned is the presence of surplus optical fibre bandwidth on the Oregon/Washington border:
http://www.nytimes.com/2006/06/14/technology/14search.html
I just recently wrapped up using a similar "store serialized data" set-up myself. In my example, I'm allowing the user insert/remove/reorder items on a list. This type of operation is pain to do with SQL operations. If you have a list, A,B,C,D,E,F and you want to insert G before C, then you either have to:

delete all, then insert A,B,G,C,D,E,F.
set "sortnumber" on G to "3", and increment sortnumber on all >= 3.
set "sortnumber" of G to be the average of B and C.

The latter is the cleverest, but eventually you run out of space in floating-point land, unless you have a Cron come in and clean everything up periodically. And of course, reordering (the example was an insert), and removing need to be considered as well. So it's a choice between lots of deletes and inserts (but really easy). Semi-annoying logic of reordering/removing and a few updates. Or clever hack that requires a cron to cover your ass.

What I did instead was stored the list as JSON, convert it to an array, and use array splicing functions to reorder things. Then, I convert it back to JSON and store it. It's worked extraordinarily well... It takes a fraction of the amount of time to do native data structure stuff than it does to touch the DB several times.
This Diet is way to complicated. A better way is simply:

* Don't drink alcohol. Alcohol has calories like fat.
* Don't drink stuff with calories. The best is plain water
* Cut back on carbs. Don't eat outside. Most food you buy is not suited for dieting
* Eat your veggies. Eat at minimum one pound per day. It is a must. It is healthy and fights your hunger.
* After you are down to the desired weight, don't get back to normal. Still eat your veggies and you are fine.
It's really nice for not having to bring down the site for schema upgrades.

All the other popular databases let you modify the schema online. This feature has been taken for granted for over a decade.
The hacker's diet is a perfect example of a watched metric improving just by being watched.

When I needed to lose weight recently, I watched what I eat (in a google docs spreadsheet) and weighed myself every day, and lost 10kg in about 10 weeks.
Of course, every system of education does not encourage characteristics of good enterpreneurs - independence and thinking differently. They are crerated to produce good workers - those people who will work for enterpreneurs. And I don't think this atricle encourages people to drop out of college - those who have bravity to do this will do it anyway, and others will finish grad school.
Digital business: didier grossemy
Didier Grossemy Founder and CEO of X2 WORLD. Didier Grossemy founded the X2 group of Digital companies. Didier Grossemy provide global Internet Marketing ...
Shanzhai: New electronics trend from China
Excuse my ignorance, but what's the Postgres solution to sharding?
OOo and Google Docs can open most Excel spreadsheets..
Science gets a boost from cheap, super-resolution snapshots
I can think of quite a few people who dropped out and just became losers. Just saying, while dropping out of an education might work for some people, I'm willing to bet that on average, it's a bad decision.
Coach Fitz's Management Theory (by Michael Lewis)
> Holy shit, is he serious?

If you have to ask...
I agree. I knew Jason back in the day, and frankly, he was a bit of a dipshit. he was hot and trendy, and knew it and played it. It sounds like he's grown up a bit. Maybe we all have (group hug?)
Lol, I wrote "RailsCron" in 2005-2006. Funny to see how the ruby community iterates.
We're doing something very similar to store RDF data. Our reason for doing it this way isn't performance but rather RDF's pretty intricate schema requirements (and opportunities). The drawback of using this scheme is that it roughly doubles the amount of data stored.

Contrary to Friendfeed, we have to use joins a lot because analysing data is the purpose of our application. We tried to do it with mysql, but mysql turns out to be completely unsuitable for the task due to its lack of merge or hash joins.

I'm quite surprised that someone like Friendfeed would change their entire data model for performance reasons instead of considering a stronger RDBMS (of which there are many). Their problem with index maintainance isn't exactly new. It's a solved problem that needs no wheel reinventing and doesn't merit the increased complexity of asynchronous index updates in my view.
It's to write efficient hash joins, partitioning and row versioning into your core database engine.
I have seen two or three in the app store, but you are right currently no background apps kills iPhone alarm clock innovation.

Though we do have a solution were working on!
Without locking tables?
And what happens if he claims to have forgotten the password?

What happens in legal cases, as they mentioned, where I have to hand over the key to a safe, and I've lost the key?
Law Makes Blasphemy Illegal
On Building A Stupidly Fast Graph Database
Crazy Quilt
I wrote this article pretty much for Hacker News since when there have been previous articles that have made it to the home page there have been questions about what exactly our graph database was.
Why My Printer Received a DMCA Takedown Notice
Exactly. It's a manifestation of selection bias. We pay attention to the Gateses, Zuckerbergs and such, and disregard the thousands and thousands who dropped out of college and ended up achieving nothing in their lives. It seems to me that Microsoft's or Google's successes are anomalies that one should not try to emulate...
I'll take a closed, consistent deployment experience (flash), over an open and free,yet inconsistently implemented one (the real nightmare that is css) any day.
Oh sweet merciful Bob that has got to be some of the nastiest markup I've ever seen.
It is interesting that a relatively advanced piece of software, message queues, which most run of the mill projects most likely have no need for, has an interface that is compatible with a simple, straightforward protocol that was designed for everyone and their dog. I mean, if you need message queuing, you most likely have someone on staff who can design, write, and deploy a protocol language (especially if they can write the queuing software also) and who doesn't need to use compatibility libraries.

Great, your memcached-compatible queuing service can leverage all those memcached client libraries out there, but somehow I don't think that client libraries are the hard part of the queuing service.
Actually, they do, e.g. http://highscalability.com/skype-plans-postgresql-scale-1-bi...

MySQL is popular with startups though, so it's not surprising that there's a lot written about it.

It's true that MySQL has some lame limitations, but I don't believe that there are any silver bullets out there. Google tried to switch their ads system from MySQL to a "real" database once, and it was basically a disaster and had to be abandoned in favor of MySQL (I wasn't working on it, so I can't really give all the details).

Another problem we had with MySQL that Bret didn't mention was that it would try to be "smart", and sometimes it would "randomly" (from our perspective) choose a very inefficient strategy, and we would have to waste a lot of time figuring out what it was doing and how to force it to do the right thing. The approach Bret describes basically avoid any MySQL "smarts" and treats it as a dumb, but fast and well tested B-tree. This gives us fairly reliable and predictable performance characteristics because we know exactly what it's doing (mostly).
Aren't you people generalizing a little too much, obv there are loads of people working on big, important problems like energy, medicine etc.
While many people use Facebook I don't really see many that hold Mark Zuckerberg as an idol.
I'm 22, soon finished up with first year in Business School and constantly fighting with myself if I have the patience to graduate. I have worked in the web industry since I was 16, done freelancing for couple of years, involved with few startups and currently a co-founder.

I know that I don't necessarily need the degree. For the most part it seems useless. I'm quite sure I could find a job, my financials are not that bad, I could even start a startup, but still feel that I'm not somewhat "ready" or have lucrative options enough to dropout.

My current solution is just to read for the exams and skip most, if not all the classes and work on other things. Good thing is that college education here is basically free, so I'm only loosing some time. But still, most of the time, I dislike everything with institutional education.
lol what if someone commits murder for the sake of obtaining a copyright?

that's not actually that far-fetched if you think about it. well...maybe. big evil corporations are, after all, the stuff of movies haha
This is a stupid nitpick, but is California's standard of living higher on average, or is its cost of living higher? Implying that teachers get paid more so they can afford California's "standard of living" rings more accusatory to me than if the author had attributed it to "cost of living," but I don't know if he's right, if he's wrong and deliberately wrote that to fan the flames a bit, or if it was an honest mistake.