skysight-inc/HackerNews huggingface dataset:
Is it me, or do you answer your own question? What is wrong with memcached as the focus of your personal project?

Meanwhile, I don't understand what's wrong with huge. Every huge open source project is built out of a thousand small projects, most of which are probably desperate for help, and many of which have an importance which is all out of proportion to their size.

If you and a couple of friends start some small free-standing project to (e.g.) build a caching tool... well, there's nothing wrong with that. You might have fun, you will get practice and learn something, and you might even invent something important. But most likely you will toil in obscurity. Nobody will know you exist, unless you go out of your way to become your own marketing machine -- and that's a lot of work.

But if you find a way to improve (e.g.) Drupal's use of memcached that makes a certain class of Drupal installations 25% faster and 20% easier to configure, several hundred developers worldwide will be thrilled. You might find it surprisingly easy to become known as one of the world's top ten experts on Drupal + memcached + Application X. And that will mean something, because people actually use Drupal.
Science of Love -- A DNA-Based Dating Website
Pick a language that you love. That will probably happen to be something not very mainstream. All not-very-mainstream languages suffer from the lack of good, quality libraries to one degree or another.

Fix it. Since you mentioned OCaml, I assume you like it. I suggest you look at Python libraries (very mature bunch) and port something you like over to OCaml.
Check out the lift web framework. It fits well with your web development skills, and it'll let you explore a new language, Scala.

Scala, as described by someone I know who has dabbled in almost every programming language I've heard of, is "Java and ML done right". It really is a gorgeous language. It has the OO-goodness of Java, Ruby, or Smalltalk, but the type-inferencing heaven of OCaml/ML. It combines object-oriented and functional programming beautifully. And it compiles to Java bytecode so you can interface with Java libraries if you really want to. It's worth checking out in itself, even if you decide not to get involved with the lift web framework.

The lift web framework was developed in Scala. (Disclaimer: I'm actively involved in the lift community.) It's the brainchild of David Pollak, who is an experienced master hacker. He had done a lot of work with Ruby on Rails and had grown disenchanted with it, and decided to write a better framework. (Check out his "Web Framework Manifesto": http://blog.lostlake.org/index.php?/archives/16-Web-Framewor...) He hunted around for a language to write it in and settled on Scala.

He's been developing lift for a little over a year now, and while it is still alpha software it has really matured a lot. The community around lift, while still small (and -very- friendly), has started growing recently. We just announced the lift 0.3 release, and there is an ambitious release schedule to get lift 1.0 out the door by early May '08, just in time for JavaOne. So it should be exciting times in lift-land.

So come on in, the water's great!

http://scala-lang.org/
http://liftweb.net/
http://scala-blogs.org/ (scala-blogs is a joint effort of people from the Scala and lift communities)
Take the class with the best prof and the students who are most fun to be around.

I would personally take networking, because I tend to find mathematical logic, as it is typically taught, quite dull. There are better ways to make sense of things. For example, people still teach Godel's Incompleteness theorems as he proved them, yet with the halting theorem, which many a drunk freshman can understand, it falls out quite naturally.
Beware extended metaphors. They are a royal road to stupid.

I couldn't keep reading after this bit:

"Windows promised to replace the risk and uncertainty of a world splintered by different platforms under just one operating system. Anyone who didn't play along with the official party line was ostracized to a Siberian gulag, or at least ridiculed as an enemy of compatibility and a threat to low Total Cost of Ownership."

As a Mac owner throughout the '90s, I've seen my share of deranged, over-the-top anti-Microsoft rhetoric, but this makes Apple's 1984 commercial look like a Ken Burns film.
Q&A with Peter Norvig: The evolution of Web search
Sorry, not relevant but since when does no one wear a necktie? Check out the picture in the link. I can now see the value of the tie; their general throat, neck, collar area looks a disgrace. You'd think with all the money they've just made, they might want to announce it with a ludicrously expensive and obnoxious tie. When I make my millions I'm going straight out to buy the world's best tie and become the world's most handsomest man when viewed from the navel to the chin.
Audio in Linux is "awesome"
I would suggest that you consider a couple things:

1) How can you break down the work tasks your site/crawler is doing so that it could be divvied out across N processes? In the case of the crawler, how would you handle breaking up your work queue so that multiple spiders could coordinate their efforts without all crawling the same pages and without skipping any? How do you recombine the results back together when they're finished? What happens if a spider grabs a URL to crawl and dies before it can report in the results - will it still get crawled or has it been lost?

2) Google's view of mass concurrency looks at hardware in the same sort of way. They considered questions like: What happens to the running processes if this disk drive goes offline? How can work being processed on multiple computers be recombined together? What happens when a critical computing node dies while you're waiting on it to report its results?

3) As pertains to database and other resources: how can the data needed to power the site be distributed across my computer resources as evenly as possible? How do I go about identifying and correcting bottlenecks?

It does not seem like you need Hadoop to either think through these problems or work on coding solutions that take this sort of thing into account. Though if you build a system capable of mass concurrency, I can understand the desire to want to test it out in a widely distributed environment.
Do you have enough traffic that scaling is really a concern? It's typically putting the cart in front of the horse to worry about that too soon. You'd be much better off trying to improve the site and get out the name otherwise.
Most web apps don't involve any kind of low level networking. You're dealing with HTTP all day and even the details of that are abstracted away if you're using a modern framework. The networking course might be useful, but it's definitely going to give you a big edge.
Schneier: Never use it (MS has added RNG that could have an NSA backdoor into Vista).
Building Skills in Python [Version 2.4] (free book)
(because of SEO)

How does registering for 10 years help with SEO?
Python Vs Ruby
It seems that Netflix is trying to offer a prize to stimulate innovation much like the X-Prize worked to raise interest in privately funded rocketry/space launching.
The Old New Thing : How did wildcards work in MS-DOS?
Engineering Magic: the Wi-Fi Divining Rod
A quick look at CouchDB Performance
Personas from Mozilla Labs give Firefox a sleek coat
EVE Online: About the boot.ini issue (the fallout)
I have taught AI many times now and I must agree.

You don't know how many times I have heard a student claim that they have made some type of major advance. But on inspection it usually comes down to something that requires exhaustive search.

I think that there is definitely a place for new ideas in the field, but I don't think they will come from complete novices. I think they will come from those who have CRITICALLY read the basic literature. And probably from those who have read more than just the basics too.

My own pet peeve on the field now is the over emphasis on asymptotic complexity. Sure its important, but its not exactly an AI result. The scruffies need to make a come back and start coding! See my own meager efforts (http://www.anaphoric.com)
I love the design of useit.com. To me its one of the best designed websites out there. By far. I hate most websites. I also like Hacker News. So im surprised to see you disliking useit. Jittery typography? What would be better? One giant block of uniform text?

How would you design useit?
The 15 Biggest Tech Disappointments of 2007
The 15 Biggest Tech Disappointments of 2007, guess what OS made it to number 1?
"Mr. Brin was most intrigued by the "engineering tricks" employed."

I'd like to know what kind of "engineering tricks" they're talking about?
You are absolutley right that the site doesn't need any form of scaling right now. I have never had the chance to write a massivley scalable system of anykind, I am looking for a challange and I don't have a lot of money so I am trying to figure out how to learn how to scale on a budget for fun , not for buisness. If the time comes though I will have some tricks up my sleave.
FixDap: A Public Collaborative To Do List (Japanese - Translated by Google)
Jiggy: Javascript App Framework for iPhone
See the way essays are presented on paulgraham.com. The limited line width makes it very readable -- if useit.com was reformatted this way, the enormous amount of text would be overwhelming. The main problem with useit is that the way it is presented encourages you to not actually read it. To me, this indicates that there is something wrong with the information, or at least it should be a lot shorter.
I would take the networking class. My first reason is that it is a survey course, and networking is a massive topic. Its helpful to have a wide exposure to different parts of it. Once you know a bit about the various parts you are in a much more flexible position to go back and specialize in what seems useful later on. But its handy to have an inkling of the direction you need to look in.

So, I would take the networking course because it'll give you a broader perspective. You'll never know what problems you'll run into, nor the solutions to those problems, but having a broad and diverse background allows you to direct your efforts into the proper direction for the specific problem.
Faster Chips are Leaving Programmers in Their Dust
NY Times reports research findings from Facebook users
Tried to post this on the blog, but wouldn't take my comments: I noticed the guy had Eclipse installed for python, but then also installed Netbeans for java. Does anybody else do this? I've always used Eclipse, but I figure something must be good about Netbeans to cause him to use 2 IDEs.
Ask YC: Will your product fly?
Per http://evhead.com/2007/12/how-to-evaluate-new-product-idea.asp, I'm asking fellow entrepreneurs to rate their product using EW's method.

Mine is called iTenna, mobile document management.

Tractability - Low/Med
Obviousness - Very High
Deepness - High
Wideness - Med
Discoverability - Low
Monetize - High
You are already thinking in the right direction if you understand how Hadoop works. But don't bother for now. No, really, don't!

Just be ready to profile and refactor your code when you do need to scale.

That said, you can improve the odds with practices that are almost cost-free, but will help a lot later on.

- Read Cal Henderson's book.

- The center of your design should be the data store, not a process. You transition the data store from state to state, securely and reliably, in small increments.

- Avoid globals and session state. The more "pure" your function is, the easier it will be to cache or partition.

- Don't make your data store too smart. Calculations and renderings should happen in a separate, asynchronous process.

- The data store should be able to handle lots of concurrent connections. Minimize locking. (Read about optimistic locking).

- Protect your algorithm from the implementation of the data store, with a helper class or module or whatever. But don't (DO NOT) try to build a framework for any conceivable query. Just the ones your algorithm needs.

Think this is obvious? Just the other day I heard of a project, staffed by so-called experts, that made every one of the mistakes I mentioned above. And in simulations, they cannot even keep up with the load they expect at launch.
"Logic might be more practical if you're applying for a job at Xilinx."

Mathematical logical has very little to do with digital logic.
It's definitely a good thing to understand how the lower levels of an abstracted system works. For example, read Joel Spolsky's explanation of "Shlemiel the painter's algorithm" problem: http://www.joelonsoftware.com/articles/fog0000000319.html

Perhaps a more applicable example in web applications is the use of things like Ruby on Rails ActiveRecord. If you had no idea how ActiveRecord worked, you might end up making zillions of database queries without even realizing it. I have a feeling this is part of the reason some people have trouble scaling Rails apps...

In the case of networking and web apps, at the very least it's a good idea to know things like the difference between TCP and UDP. You probably don't need to know routing protocols and that sort of thing though.

That said, I'm not sure which to recommend. I took some of both (2 networking courses, and basic logic was covered in discrete math and 2 AI courses I took... including the one I have a final for in 7 hours...) and I've found the networking ones to be much more applicable, but it's also good to know about logic as well.
Uh, why are people voting this down?

> some applications aren't viable at all without this critical mass and some are

That is exactly what "network effects", or "positive network externalities" means, and they're critical to many, but not all, information goods.
A better question: will it blend?
Looking at that syllabus, it covers a lot of useful topics for web apps: sockets, threads, TCP and UDP, the major application protocols (HTTP!), cryptography, authentication...

and Web 2.0... WTF!... I'm sorry but that's a ridiculous topic to have in a college networking course. Is "Web 1.0" a prerequisite?
Ignoring existing work in AI is as if you would try building a flying machine, ignoring existing knowledge in aerodynamics. You'd be right back where Leonardo was. People are doing that, nevertheless, but if you do that in AI, chances are you'll be just fabricating crackpot theories.
Make $300 Per User on Your Facebook App
after 10 long minutes:

tractability.Med obviousness.Med deepness.High wideness.VeryHigh discoverability.VeryLow monetize.VeryHigh
Scalable Internet Architectures by Theo Schlossnagle was a good introduction for me.

http://www.amazon.com/Scalable-Internet-Architectures-Develo...
1) Don't link to Digg... link directly to the original source.

2) Your title is ridiculous. Tracing back to the original page is like a reverse game of "telephone":

- "Make $300 Per User on Your Facebook App"

- "Add New Facebook App += $300 For Somebody's Bank Account"

- $16.8 billion / 56 million users = $300 per user

3) The math makes no sense at all.
Sorry, I don't quite understand what you mean. What exactly is the "input"?
Fall on your ass many, many times :)
Milemeter: Buy Your Insurance By the Mile
Tarsnap (online backups for the truly paranoid):

Tractability: Medium.

Obviousness: High.

Deepness: Medium.

Wideness: Medium/High.

Discoverability: Low.

Monetizability: Very high.
> - Avoid globals and session state. The more "pure" your function is, the easier it will be to cache or partition.

I'd like to emphasize this point in particular. Shared state is a big, inefficient, centralized bureaucrat and is the enemy of horizontal scaling. Statelessness and decentralization are best-friends-forever (they each have a denormalized copy of different aspects of the same friendship bracelet!), and you should figure out as many ways as possible to exploit statelessness and minimize unnecessary shared state in your application.

If you have the time, learn statelessness-in-the-small as well: play with a strict functional programming language (for your own edification, not necessarily for implementing this particular project!), and you will learn how to become keenly aware of the flow of state in any program, and how to maximize statelessness in any language. This will improve anything you ever program again, and will form a powerful mental model of state that will carry over by analogy to statelessness-in-the-large.
So this is basically a JavaScript wrapper for Cocoa on the iPhone?

Interesting, but I don't see much advantage, other than not having to compile.
The difference between the field of AI and the field of aerodynamics is the state-of-the-art in the latter has produced working results -- to say the least!

Meanwhile humans farm gold and solve CAPTCHAs.
The Bishop book looks very good, judging from the table of contents.

Perhaps similar, and available as a free ebook: McKay, Information Theory, Inference Theory and Learning Algorithms, http://www.inference.phy.cam.ac.uk/mackay/itprnn/book.html
Cat Herding is a bad thing
> the AI community hasn't been seriously trying to construct strong AI for 25+ years.

Now that's an excuse if I ever heard one. "Uhh... we weren't seriously trying... uhh..."

That's also why I'm not dating Natalie Portman!

> Considering that trying to build strong AI is a pipedream anyway (IMHO)

That's hilarious. Heavier-than-air flying machines, a man on the Moon, sending pictures through invisible AIRWAVES?!? Pipe dreams!
Maybe the OLPC project? Check this out:

http://wiki.laptop.org/go/Getting_involved_in_OLPC

I think there are a range of technologies in play here that you can work with, from close-to-the-metal C to application level Python.
Yeah, but from the perspective of converting digg into a bunch of news.yc communities surrounding other niches, I highly doubt that could ever happen. And any attempt to do that would just ruin the digg community.
This is really good. Thanks.
Some punk-ass bitches from MIT?

The rest of the world just says AI. It's like a million to one, we win. Don't make me have to call M.C. Hawking.
Since you asked:-)

Hecl: http://www.hecl.org

Thanks to Google, if you contribute something now, there's a chance you could even get some money out of it:

http://journal.dedasys.com/articles/2007/12/17/announcing-he...

Also, it's a small community and in my opinion a fun and very creative project. Making a programming language is a difficult but rewarding process.
You may get real-world news, but it's most-likely news of a certain political nature. It's obvious to me (if not others) that places like Slashdot are completely different _political_ sites than they were say five years ago. The topic matter may be close, but the nature of the way articles are presented, commented on, and ranked is completely different. It used to be you got great intellectual comments from all sides of the spectrum. Now that sort of disucssion is down-modded and in its place you get basically political sloganeering masquerading as comments.

In terms of general subject matter, sure, the Daily Me is probably not happening. What's happening in its place is a sort of Daily Political Me, where whatever is "hot" is filtered one way or another based on general political views of the site. This is a natural consequence of user-ranking.
Self Googling Is Now A Popular Online Sport
Peter Thiel's Fund Raises $220 Million
The Digital Marketing Mix
Amazing how people with all this knowledge still haven't gotten anywhere close to AI.

It might even be something simple, overlooked, done a little differently.
get a beer - go out with friends, etc etc. - get drunk as hell - enjoy till you get wasted and then go back to hacking
I'd recommend starting with a definition of intelligence. Once you clearly and unambiguously define it, consider you have 51% of the task done already.
Yeah. I really should have rephrased what I was trying to say. I know what I like: code style, languages, community, etc... I just want to branch out and poke around and see what other people are working on. I want to broaden my horizons if you will :)

Thanks for the input and sharing your guys' own projects / suggestions!
you can't fuck with nature, not this much...
Some analogies are like bananas.
Calculus has very little to do with Computer Science. I meant that Mathematical Logic will probably exercise your brain better than Networking for something like Digital Logic/FPGA design.
The shape of things to come: predictions for 2008
That's brilliant, amichail. Thanks.
Research suggests that spite is a uniquely human trait
Jeff Hawkins talk on Hierarchical Temporal Memory at IBM Conference on Cognitive Computing [video]
"Artificial General Intelligence"

Never heard that term before.

And to quote Office Space: "Why should I change my name, he is the one who sucks". So personally I won't adapt "Artificial General Intelligence" into my vocabulary.
I learned to program on the ZX Spectrum (Awesome home computer).
The spectrum had a great(ish) keyboard, where each key also had about 3 or 4 BASIC shortcuts on it, which you could get with various shift keys. Also it would recognize when it was expecting a statement, and just use the statement for that key instead of the letter.

For example, to type "for f", you would press the 'f' key twice, as it had the shortcut 'For' on it, and also was the letter f.
Similarly, to type "next n", you'd hit the 'n' key twice.

Perhaps why you sometimes see f and n in for loops.
"...	How can I learn to scale my project? ..."

Go work for google? [0]

".... I really want to start testing and developing methods for scaling and speeding a project of mine, and also just for general knowledge. I can't think of a way to do this without creating a bunch of amazon ec2 instances, or buying more computers and doing this from house. ..."

Think distributed.

Create a useful installable tool that utilises spare cpu cycles that are being under-utilised. Or get some experience with some of the existing systems. Some examples that I can think of are

- SETI ~ http://setiathome.berkeley.edu/

- Electric Sheep ~ http://electricsheep.org/

- PlanetQuest ~http://www.wired.com/science/space/news/2005/03/66757

Another approach is to take a look at seeing how you can create a Beowulf cluster ~ http://www.beowulf.org/showcase/index.html
You don't have to build it (though operacy is worth 10x reading) you can look at the software. If you are still at school or know somebody there you can see if anyone is working on parallel software.

[0] Take a read of this blog on the development of a personalised RSS crawler ~ http://blog.persai.com/ to see the kinds of problems you have to overcome (scaling, data integrety, storage, re-writes)