This sounds like the beginning of a fun project. However, may I humbly suggest a few things?

(In 2005, I built a system to do interesting computations on each stock option tick. The volume for the time was pretty high. Each time the IBM stock, for example, moved a penny, this generated 400 quotes. While small by today's volumes, 100k quotes with an OAS calculation with a binomial matrix on each tick was a challenge. So I am not totally unfamiliar with these throughput issues.)

And Python was used, but only on the GUI side, presenting the results. While I didn't measure it, I feel quite confident in saying that Python isn't going to keep up. At the risk of being a premature optimizer.

Before you optimize the stock look-up function, you might want to look ahead a bit. veyron has commented below or elsewhere that a very important calculation is going to be your total holdings on each tick. This is how you will calculate your risk, or at least the fundamental start.

Another thing to consider is your goal in trading. How will you be making money? For example, will you be buying/selling on, say, 10 second intervals and collect profits from that? Or longer intervals? Shorter?

Whatever your overall strategy function is, you will spend some time in that calculation. Will you do it on every tick? This computation time might well at this point be unknown. If it is an interesting calculation, it might well be complex.

So how is a startup going to deal with this firehose of a problem?

Well, consider splitting up the processing. One obvious way to do this is to filter tickers you are interested in in one box and push a much more manageable stream to your main logic box. Perhaps start with one frequently-traded stock, tune your algorithms. This will be a rich learning experience.

Then, watch the CPU percentage. If it is above 60% at 0910 central, maybe start profiling your code. Your time-consuming parts will surprise you.

If it is going well, and you want to expand, consider adding one box for each stock that you plan to trade. You can do it for about $200 diskless, quantity one retail. Perhaps put the quote stream or a divided quote stream on a multicast channel, two NICs per box. Allocate a pre-determined amount of capital for each stock so they don't need to coordinate to give you your risk calculation.

Also, the idea of High-Frequency Trading is kind of a fractal concept. On the low end, you have a trading situation that is slightly more active than a day trader. On the high end, you have boys who rewrite network stacks in windows or linux for their boxes co-located with exchange points in New Jersey, where the cables are measured to the inch so everybody sees the tick at the same time. Those are the 100 nanosecond boys. These are the fellows who order servers by the multiple racks of 30. And they trade in a whole different style--their net hold time might average out to zero, but because of agreements they have with the exchanges, they have to stick with the market when it takes a dive.

So don't try to be them at the start.

In summary, success here will come more from the engineering than the computer science, and not trying to beat the big boys at their game.