skysight-inc/HackerNews huggingface dataset:
Tokyo Cabinet (TC) is what I was going to recommend as well. It's crazy fast and quite reliable. It also does zlib compression so your dataset will be smaller than with most other key/value stores. Be careful picking an open source key/value store because a lot of them are kinda half baked right now. (But TC is very solid.)

My first attempt would be as suggested above, to use a single instance of TC. If it ever started to get slow or you simply ran out of space you could even put a consistent hash in front of a cluster of machines. Though I wouldn't be surprised if a single machine would be enough for your current needs (20 billion).

The hash doesn't have to be very complicated, depending on your key. If your key is just a numeric auto incremented id for example you could implement a "hash" with modulo which just maps 0-x billion to machine 1, x-2x billion to machine 2, etc.

If you do this though you'll lose the ability to generate keys to search through your data. If you need to generate keys and your data is distributed I highly recommend sphinx search ( http://www.sphinxsearch.com/ ). Technically, it's a full-text search engine but I use it for normal searching. You can build an index distributed across multiple machines and even multiple cores of multiple machines. For example, if you have 2 boxes with 4 cores each, you can split your data into 8 chunks, 4 of the chunks running on one machine, 4 on the other with a single "distributed index" that ties them all together. When you do a search on your distributed index for say 100 items sorted by date, 8 threads will search the 8 chunks, return the 100 best results and the distributed index will filter out duplicates and return the 100 best results. And it'll do it incredibly quickly.

We're using sphinx with mysql. I'm not sure how you'd get the data from TC to sphinx (there may or may not be adapters for that I'm not sure) but if you have to pick between sphinx and TC you'd have to decide what's most important for you, low storage space with very quick lookup or very low latency search. If you need low latency search I suggest you simply use mysql for storage and index your data with sphinx.
Yeah. but it's text, so i'm thinking about S3, interesting storage optimizations (alot of it would be tag data, for example,so there are plenty of fkeys).

i'd shard it based on some kind of distributed scheme... though not sure what that might look like. Good tips though. :)
I always run trunk as well, seeing as how it always has more kickass stuff than the release. The only bug I've ever had to directly work around is the "EmptyQuerySet sucks" bug (http://code.djangoproject.com/ticket/7235).
Really, why? I haven't been following closely.
Nooyi, Ayer among best US CEOs
But why did they need to cache data to provide this service? The script that generates the RSS feed could have just queried the FB API and generated a feed on every load.
Right now I am employed by University Residences at Western Washington University building web apps while I work towards my degree. UR is only one of many departments that has a need for tech staff, I'm sure there are tons more job openings than you realize. I was offered a job at the Financial Aid office after I made a simple scraper that took their student job openings and made a UI with better usability. This is an example of the Do What You Love (and money will follow later) idea that is spread around here so often.

My advice is to have fun hacking away at as many little projects as possible. Maybe one will get someone else's attention who is in a position to help you out (like what happened with me, but I already am employed by the university). Or maybe you can figure out how to capitalize on one of your little projects. If all else fails you're always building your resume for when you apply around.
Well, this is an article about programming, not a program, and it's pretty insubstantial. All it really does is mention some buzzwords and and note that they're the foundation of the "pyramid of code quality". Nothing personal against the author, but it's pretty flat for something that's stayed on the front page through evening. I wonder why it's getting voted up.

This article on ssh (http://sigusr2.net/2009/May/07/hacker-utility-belt-ssh.html) barely scratches the surface, too. At the very least, I'd mention ssh-agent (a good intro: http://mah.everybody.org/docs/ssh) and visual fingerprints for host keys (http://gwolf.org/node/1864).

But, I really should be constructive and write something myself, rather than just griping here. It's much easier to complain about other peoples' writing than to write. One of these days...
"Sounds obvious yet rarely executed."

It's often hard. I've been trying to do this for some time. It works if the client is open to the idea that he or she doesn't really know what they want, because they don't know all the things they could have, or what things may cost (both in upfront time and money, and long term maintenance).

I've had most trouble where the client is a team, one of whom has the job of defining the app. They often want to pick your brains ("Is this possible?"), then ignore your advice ("Yes, but that will eat a lot of time and money, which will bite you in the ass later.") and focus on what seems "cool".

I don't think this is approach is rare, though, just maybe not as discussed as much.
Alpha Shapes
(250 megabytes of data transfer, 10 cents a megabyte above that)

Some day we'll tell our kids we paid 10¢/MB for a ubiquitous Internet connection and they'll commit suicide.
The busy Java developer's guide to Scala: Scala + Twitter = Scitter
Unfortunately, this site is basically worthless as it stands. You just don't get enough traffic to make any money on ads, and you aren't going to find a team of hackers to take over your code. Heck, my blog about random stuff in my hometown used to get more visits than CherryPeel does.

Let's imagine a scenario where someone says "y'all did a lot of good work - here's a couple thousand bucks, because I think I can make a go of this."

Who is this person going to be? A hacker isn't going to take over your code, and no business person can analyze this and come up with a deal that would be anything but insulting. And the same goes for a company - they can't offer you enough for it to be worth it to you, and if they want this, they will just build it themselves.

You should just keep this as your hobby site. Maybe it will take off someday - either way, I'm sure you learned a lot, and it was time well spent. It's a neat site and I've told my musician friends about it a couple of times.
Campus work study when I was still a student, then got hired on by a pair of professors to assist with their research after graduation. It was a wonderful experience -- after the summer they wanted me to come be a grad student in their lab for $20k a year, I sort of had other aspirations in life, we parted ways amicably.

Failing that there is always retail/food service/etc, but working on campus has a lot to recommend it over these other options. It pays better, is more convenient, has more potential to result in something with resume power, etc.
Seconding this. TC is solid.
Why do people like this ask for experts?

I know you want an expert, but you really have to approach this logically. If they're an expert, it's highly unlikely they're going to accept equity in some unknown, dubiously advertised startup rather than the sweet cash they are (or at least if they're a real expert, should be) making already. If they aren't making real money, then you should probably question their qualifications, and in the process, rethink what you're offering your future employees.

</startup_rant>
I'm a big fan of the ForwardAgent option. It allows you to, based on destination, pass your SSH key forward and have it used as the private key on the destination server as well. This allows me to ssh into a remote server, but keep my key based access rights on other servers (or github).
Why Tokyo Cabinet?

How quickly can you populate your database locally? Adding 10 billion objects at 300/seconds will take too long.
Space Storage
Storage Concierge is your premier Self Storage Units directory.We can help find the closest storage location near you.
Virgin America internet review
So you do have updates? Because updates makes a database more complicated.

If you need lat/lng then you need an R-tree and very few databases come with that.

Sounds to me like your going to have to write your own database - or more accurately your own index.

Basically store data as append only, and handle change as delete/append. Store just the offset to the data as the key.

I really hope your index can fit on one machine.

Do it like the facebook link I gave you earlier.

Your going to have a hard time with index merges (lat/lng merge with tags).

Maybe a better idea is don't store the object in the database, but just the offset to it (plus machine ID).

Then use any commercial database that has an R-Tree.

You have 200GB to 1PT of data to store. Which needs at least 25 machines.

But if you can store just the index, but not the data, maybe you can store the index on just one machine - that will makes things easier. (Obviously you'll have multiple machines with the same index for load balancing.)

This might help also: http://www.ddj.com/184410998 (instead of an R-Tree, you might be able to a regular index).
Poker is an easy way to make a lot of money
It's not. It's grossly distorted by government interference, as explained in detail by the article.
I found his advice about "hiring a law firm" interesting (see also http://www.quicksprout.com/2009/05/05/8-things-i-wish-i-knew... ), but I could not help wondering if it was a bit overkill, especially since I have seen a lot of advice here that suggests that most small companies do not need legal representation, especially when they are just starting out. Is legal counsel an unnecessary expense for a new company, or a necessity?
If I read it correctly, 300 records per second was his production rate. I've benched TokyoCabinet out to be much, much faster.
Everyone's. Linode had the most.
Anyone notice how the ex-British Empire dominates the list? How about a correlation to angloindex?
Rashmi Sinha among sexiest CEOs
Mapping Shoutcast Listeners IP Location
Perl Roles versus Interfaces and ABCs
Canada cracks H1N1 flu code
Brick Staining 
Brick Staining and Brick Restoration services. Brick Imaging offers a high quality, customized brick coloring and masonry staining process for all residential or commercial needs.
ScratchPad v5.01 S60v5, Change Your Phone into Paper Notepad
The Real Problem With Dependencies
From an Apple iPhone user point of view this makes good sense as most of them will upgrade to V 3.0 pretty quick. I also assume this will help Apple regression test V3.0 with the current code base of App store applications. It might be painful for current developers but I think they can still build for V2.2.1 and then test on V3.0 also.
1 million records in 0.4 seconds. 2.5M queries per second.

http://www.scribd.com/doc/12016121/Tokyo-Cabinet-and-Tokyo-T...

page 4.
I would look into hosting the system at someplace like softlayer, pay special attention to the cloud offering they have coming out next week.

For file store you maybe able to save some money using http://www.danga.com/mogilefs/ instead of S3.

For your datastore checkout http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-va...

Also worth taking a look at is http://hadoop.apache.org/core/
MySQL: Find records in one table that are not in another - revised
Yeah, i think there's a lot of validity in this comment... I fully expect to do stuff like abstract storage of lat/long to separate stores- so most append is really just creating a new related object- but how to store it is the fun problem :)

good tips. thanks. :)
If you have HTML/CSS and basic website skills you can get a fair amount of money from Craigslist very easily. Additionaly, see if your university web department needs a webmaster, generally they allow you to work from home and the pay is higher than other jobs on campus due to the need for technical skills.
It's a scam. (To those in the same situation) Talk to the competition before they see one bank get away with it and go all cartel stylee, blaming market conditions. Banks and other similar businesses don't want to compete on substance, but on style.
Intern at the Free Software Foundation
there tend to be interesting jobs if you look long enough. Eg I'm doing some computational game theory for a Poli Sci prof part time over the summer and perhaps beyond. Fun stuff
mass education is, by its nature, about teaching kids with IQ's of 100. There are many better ways to teach kids with IQ's of 85- and better ways to teach kids with IQ's of 115+. But because we're more concerned with the ideal of equality than concrete results we've made many avenues illegal.
Love the article and I'd just like to take a moment to lament the lack of Bushido Blade type games in the last 2 generations of consoles. Easily the greatest fighting games ever.

you can email me if you can't figure out a way of playing them >_>
TA work: $9/hr for menial labor. Can get you ins with professors for research.

Research: degree of pay varies widely. Our math department offers $12k for a year's worth of undergraduate research. I'd assume yours has something similar.

Private sector: Last I was getting $18/hr for JSP work, but I found the Wash. DC IT scene to be relatively soul-crushing. Also risks interfering with your studies because in most cases it drags you off campus.

If you're really hungry for cash, sell some stuff from home on Amazon.
Ask HN: What services are available for sending/receiving SMS/MMS messages?
Looking for a reliable low-cost service.
Another interesting mechanic would be different types of attacks: thrust, slash left, slash right, slash down. The player can see the attack beginning and has a small amount of time to choose the correct defense to that attack. Choosing the correct defense reduces the amount of focus lost. You could also add shields. If you press block with a shield it'll block any attack with minimal loss of focus. Shields would have hit points and break if used too many times.

Mount and Blade is the closest game I've played to this. It still has hit points, but that makes sense for a medieval game where you're fighting heavily armored humans.
I'm working on writing an open-source math textbook. You can find it at http://christopherolah.files.wordpress.com/2009/01/math1.pdf . I'd love any feedback. It is very much a work in progress, though development will be rather slow until June...
Here's a good comparison of key/value database alternatives: http://www.metabrew.com/article/anti-rdbms-a-list-of-distrib...
Workaholic sets world record
Poker is an easy way to lose a lot of money if you're not careful
Ask HN: Side projects: How do you find people to work with?
How do you go about finding new people to work with?

I'm interested in finally playing out some ideas that I have been working on myself, but don't really have the contacts to fully and successfully work them out.

I'm wondering how you went about starting to make connections and establish a group of collaborators online.
www.textmarks.com/

I believe they are free and make their money for advertising. They share the short code though, so you signup for a specific keyword at first.
A bigger problem was that it doesn't meet state standards

I don't even think that's a problem.
IPhone Prank App Hits App Store
India’s best known poet and Nobel laureate flunked school
If I've learned nothing else from Dan Cook of http://lostgarden.com it's that just thinking up a game-mechanic is practically useless - you have to make a prototype, play with it (as a game) and tweak it (as a coder) to bring out the fun. Like any other fiendishly complicated, non-linear function, the psychological response to 'fun' is not something you can just sketch out on a napkin and say 'yeah, that'll work'.

Here's hoping someone with a bit of time on their hands will take this spec, stick it into some RAD game tool and see what happens.
My recent experience with Braintree was the opposite. I received an excellent offer from CHASE Paymentech and presented this to Braintree. They were willing to waive the monthly minimum and sent me a schedule of straightforward low rates, however, they still weren't as low as CHASE Paymentech.

They even went so far as to say that I got a better deal with CHASE Paymentech and subtly suggested we use them until we felt we needed Braintree's value added services which seem to be the best in the industry.
The game Dragon Realms (play.net/dr) implements a system very much like this. It's actually a mud and not a video game, but that actually doesn't make a significant difference in this context.

DR has balance, so that the better balanced you are the better you can attack and defend. Some weapons are better than other at unbalancing an opponent, and you can also use spells to gain balance or unbalance something else.

It also has stamina, so that when you swing a sword over and over eventually you'll start swinging wildly, and eventually you'll fall over from exhaustion. This also negatively affects your balance.

The mechanic of position is also used, and as you play you learn how to respond to attacks or execute sequences of attacks to gain a better position.

Benzim mentioned having thrust, slash, etc, and DR implements this idea with thrust, jab, chop, sweep, slash, charge, etc.

The DR system differs from the one in the article in that you can face multiple opponents, though that's a skill you have to level up or else you'll be overwhelmed.

Overall the combat system is very engaging and different from the typical MUD system of typing "attack" and watching lines of numbers go by. Worth playing if you're into that kind of thing.
Gel Nails | Terms  & Conditions | We Have Everything You Need for Gel Nails
We have a wide range of nail gels and also have the biggest selection of UV gel nails! We sell all products for artificial nails: UV Lamps, Gel Nails, Nail Art, Tips, Brushes, Drills, and much more. We ship all over the world! Great Prices and Excellent Service.
A 64-bit OS build consumes more memory, even if your app does not.

Perhaps there are special cases where it still makes sense, despite losing memory (though your example use case makes me feel funny, and not in a good way, but I don't know enough about such things to argue), but in the general web hosting case, it is stupid to have a 64 bit OS with less than 4GB of RAM. As you note, the difference is much more dramatic in dynamic languages (Ruby, Python, PHP, and Perl applications simply explode in size when run on 64 bit systems...like 50%-75% larger).
Work study job at the school's help desk. Best job ever - tax free and a lot of free time during the job to finish projects.

Also, my dorm room happened to be right next to the building I worked in.
I don't know what your definition of a "media object" is, but I'll assume:

* you have audio, video, and/or image files
* each media file has a name/media_UID
* your "related objects" are more or less
fixed-format small records with elements of
type 'string', 'uint', 'int', etc.
* your "related objects" might be metadata
attached to the media, info resulting from
processing the media, related info from
the context in which the media was found,
or dynamic info about how the media was
used or referenced

If money were available I would do the following:

* buy a 1-big-table log aggregator like SenSage
(http://www.sensage.com) (distributed Linux-based
redundant large data storage/query engine)
* define a single DB table with the media_UID as
key and with all columns defined for all related
objects (note: I'm assuming fixed-column-set
for each "related object") ... with the understanding
that any given row may or may not contain a
"related object" of a given type
* I'd take the (relatively) static data for each
media file (e.g., media_UID, file size, file name,
media type, ..., # of unique faces recognized in
the media, make-up-your-own-field-here) and
insert it once, with NULLs for other dynamic
"related object" fields
* For dynamic info I'd insert media_UID and relevant
fields and a timestamp for the dynamic event
* ... and after this you'd have a queryable data set
that's constantly evolving
* you could dynamically update the schema as you need
more "related objects" or more extend their fields

I would buy an EMC Centera array (integrated with
SenSage for archival) and use it also to store the
actual media, keyed by media_UID.

After you've done this you can periodically run full-table SQL/Perl scans to aggregate the info you need -- that's what the SenSage tool is built for and does it blindingly fast. You could expose the aggregations as full data sets in a Postgres DB if they're needed multiple times ... or as throwaway dynamic results if required just once.

If there's less money I'd try to replicate the data store and aggregation in Hadoop or something similar.

As for SenSage speed ... European/US/international telcos/ISPs use it to store call-data-records and IP-records, scanning billions of records in minutes when law enforcement demands the info. http://news.prnewswire.com/DisplayReleaseContent.aspx?ACCT=1...
tutoring in the computer lab for 159.101 and 102
As far as business models are concerned, having data that is open is more likely to lead to better web monetization like contextual ads. The ability to have inbound links from referral sites and Google is important for ads.
Swordfighting is a lot like sex. Often it's not what you do - but what you say.

A: Soon you´ll be wearing my sword like a shish kebab!
B: First you better stop waiving it like a feather-duster.
Yes, good point. I guess he was referring more to the scientific end of things, where you write a program that runs for days on massive data, and there really isn't a user