skysight-inc/HackerNews huggingface dataset:
I mentioned a possible solution here: http://news.ycombinator.com/item?id=33902, but I'm really commenting because there's a really cool connection between PG's approach of storing it as an in-memory tree and my approach of a 2048-bit index:

Y'know how in a typical CS datastructures course you'll have to build a heap, and then reimplement with an array? A heap is conceptually a balanced binary tree where every leaf is filled, while there's nothing tree-like about an array. However, your CS profs were trying to get across that any binary tree can be stored as an array, with the root in position 0, its leaves in positions 1 and 2, their leaves in positions 3, 4, 5, 6, etc. If a node doesn't have a leaf, that position in the array is empty.

Normally this is very inefficient - a totally-unbalanced (linked list) binary tree of length 20 would require a 1MB array to hold it. However, heaps are always perfectly balanced, so an array really is the most efficient implementation for them.

By restricting users to a certain number of replies to each comment (65,536), I'm effectively constraining an arbitrary tree structure to a 65,536-ary tree. Which can then be represented by an array. If held in memory, it'd be a very sparse and inefficient array.

But databases are excellent at storing sparse arrays. If I don't store a potential key, it doesn't take up any space in the DB. And with a good indexing system, I can reference any element in constant time (well, O(logN), but the log base for B-trees is so huge that it might as well be constants).

So, I'm really just applying some of the meta-principles of CS coursework to basic data structures. (Actually, I got the idea off a Slashdot posting and had to figure very little of this out on my own, but it's still really cool.)

Incidentally, there's apparently a deep connection between number systems and tree structures. You can read more about it in Chris Okasaki's Purely Functional Data Structures. Apparently, every number system corresponds to a data structure. For example, Peano arithmetic = linked lists, binary numbers = binary trees, fibonacci numbers = binomial heaps, etc. There's a short presentation on it here: http://www.informatik.uni-bonn.de/~ralf/talks/BCTCS.pdf, and I'd strongly recommend the book.
Why I'm applying for SeedCamp
As I said, corporations will probably now. It was just a thought - why limit yourself to corporations? Who is the target audience anyway? Somehow that famous sentence comes to mind: "we estimate the global demand for computers to three machines".
"They were never going to jump into a discussion and tell people to stop being jerks."

Really? Isn't that more or less what they did with Pica when they requested that he stop using certain epithets?
here's a link to a portfolio of nice application interfaces from the author of this article: http://www.lukew.com/portfolio/web_applications.asp
Clay Shirky has a couple good essays about this:

http://www.shirky.com/writings/group_enemy.html
http://shirky.com/writings/broadcast_and_community.html

I agree with most of his points. Better technology can not substitute for better people, although I observe a strong correlation between mental acuity and humility and, hence, 133t hacker skills.

One of the unfortunate things about communities (reddit included) is that they tend to gravitate towards a certain conformity of opinion which then rewards people who parrot the standard opinions and stifles innovation.

I was reading a blog post recently on how someone creeped to the top of Y-Combinator karma by observing karma-award patterns and posting articles which he thought people would award karma for.

This, obviously, is an effective way of winning prestige, but prestige has rarely been a very good correlate for anything. In fact, as Pg stated in his talk at Rails Conf, even at the individual level an increase in prestige frequently has a negative impact on innovation.

Redditors have been observing the same pattern in their own community. As more people have become aware of the community (the prestige has increased), the quality of the postings and karma system has declined to the point where inane 'Bush resembles Chimpanzee'-type postings dominate the main page.

There is ultimately only one solution to this problem - the creation of walls. The last online community I participated in went through a similar phase of evolution. The content was progressively diluted until a number of core members went and created their own community. This new forum/community now has a application for membership which requires applicants demonstrate writing proficiency prior to entry.

Tie-ins to current political machinations could be articulated, but seem unnecessary at time.now
Did they? I didn't know about that, but I'm not surprised. Still, it shows how tolerant they were if it took him to make them say something.
You probably don't even need to be using the word "bit" unless you're writing something a lot more performance intensive than a forum.
Feedback on startup name "Mindrosia" & logo suggestions
I was wondering whether this would be a good name to use for my Study Stickies service. It sounds cooler to me although I'm not sure as to the sorts of connotations people would associate with this name.

I'm also trying to come up with a logo for this service. Any suggestions? The logo at http://studystickies.com doesn't look good when small (e.g., for an icon).
Why not just store the parentid (can be another comment or the news topic), username (instead of userid so you don't have to access the users table), and votecount for each comment?

Also, each comment should have a newsid field that points to the original newstopic. The left-most posts would have the parentid and newsid that were the same (or parentid can be zero, depending on how you do it). But any replies would have the parentid to the parent comment, but the newsid would still point to the newstopic. Then you just run an SQL query to find all comments with the newsid for that newstopic. You won't have to join any tables. After that, with everything in memory, you can just sort the way it has to be. Finally, you can cache the page in memory or a file in case the page doesn't change for the next set of visitors.
On the topic of community makeup, here are some interesting findings that come from Yahoo's social scientists (using Yahoo lists as the dataset):

1) Small groups have higher numbers of core users

2) Private and semi-public groups have higher numbers of core users than public groups

3) People in private groups remain in the core for longer

4) A user who is in the core of one group is more likely to become a member of the core when joining another group

5) New members who eventually become part of the core are more likely to receive messages from other users who have been long-time core users

6) New members who eventually become light users are more likely to receive messages from other light users, if they receive replies at all.

This suggests some strategies for building a strong group of core users to transmit social norms. For example, having an idea of who the core users are in other similar communities and making a special effort to reach out to them if they visit your site.

Unhappily I lost the original paper and all I have are my notes, so I don't know exactly how they defined the core users offhand.
My first reaction is that the word is elegant but with 4 syllables is a lot to ask. It seems like the most successful names are simple with fewer syllables making it easy to remember.

For the logo, there is a visual conflict between the abstracted lines of text and the "study stickies". Maybe if you choose either just the lines or the letters that would help, or you could differentiate the two more with thinner abstract text and thicker font lettering. For the small version, you could just use the open book look with an abbreviation of your name. We did a similar thing at http://www.streamfocus.com (see the little logo in the address bar)
How do you get the full thread for a given parent? Selecting based on the parentid alone gives you only immediate descendants. Recursive queries are generally a no-no, as they drag your performance down fairly quickly. Selecting based on the newsid gives you every single comment on the item, and then you'll have to do a lot of work (basically building up the full tree structure in memory) to find out which are children. For a site with traffic like news.yc, it's probably acceptable, but it's still more complicated to code than a simple select.
I would like if there is a function to hide and expand the comments. For example one click on a comment will hide itself and all its sub-comments. It is a good way to organize the discussions, and I always believe a website is at its best when scrolling is minimized.
Special relativity and the startup paradox by Matt from Xobni
You would find the full thread by searching for comments with the appropriate newsid.

To simplify the recursive function (using data in memory, not in the database), you could also have another field, called indent, that stores the level in the comments.

You could find all the comments with one SQL statement (where newsid = news.id), then go through each level of each comment, and sort it that way.

You could even make the parentid field a double, so you would store 123.4 where 123 would be the parent and 4 would be the indentation level.

I guess the difference is, do you want to store everything in one field or have different fields for everything?

I think this gives the original poster many options to think about. To me, a recursive function would be easier to write than coming up with ways to combine multiple fields into one using MySQL.

I think the order of optimization is:
1) Optimize so you can use a cached result if it exists
2) If the cache is out of the date, try to make just one SQL call
3) don't use table joins
4) reorder the data as needed in memory, then save it to a cache

As far as the single SQL call goes, I don't think it matters whether you have 5 fields or 30 in the actual table as long as you only request the fields you need, but I could be wrong.

By the way, do you have any other interesting examples?
Does it really matter what Nokia says when the carriers are the gatekeepers?! Biggest thing with trying to create a mobile app startup is to find a carrier willing to adopt your app. That's a huge hurdle and very hard to overcome. Even if you sign on a single carrier, signing on other carriers will be next to impossible since carriers like to divide and conquer the marketplace and will adopt your competitor rather than allowing you to have a monopoly.

Maybe this new FCC law and Google will come to the rescue!

In either case, if you have a mobile app that doesn't need to touch phone hardware, you're much better off with trying to market an iPhone app.
I have the same problem with reddit and digg: I like to read exceptionally downmodded comments. When comments get modded down so much that they are not displayed, and also they generate a lot of responses, it's the kind of thing I can help but look at. Like a train wreck I suppose.

It actually sucks me into the troll MORE when they get such special highlighting. I should say I do like how top level troll posts get put at the bottom for reddit.
If anyone is interested in a blog post about the problems with comment and forum systems in general, here is something I wrote on the subject: http://www.digitalkarate.net/?p=20
napster?
Interesting point about email addresses being a disadvantage in OpenID. Is this a point against email addresses as usernames? I was squarely in the email-address-username corner, but this is making me reconsider.
Lowtax (he runs somethingawful.com) gave a talk about web communities at UIUC a few years ago and he basically said without good moderation large sites will just degenerate into crap. The talk(http://www.acm.uiuc.edu/conference/2005/video/UIUC-ACM-RP05-...) takes 15 minutes or so to get into anything serious, but I think there are some good lessons in there.
I think an apps login page should accept url, email, username or mobile phone number, but request the simplest possible mechanism in that context. Eg news.YC just asks for an username on sign up, no email.
Considering Microsoft has been dealing in vapor for years, clouds seem to be a natural line extension for them. Seriously though, does anyone really believe MS is going to be able to deliver something well-engineered?
While I agree with your point, it's not the point of the blog entry:

"...if it doesn't work when you've got one, it's extremely unlikely to work when you have dozens."

Also, there is no mention of the difficulty of gaining users, merely that your idea won't be any better when you've got them.

That's clearly not true in several cases.
I like Study Stickies much better. There is always some benefit of telling you what it is right in the name.

Also, Mindrosia in my opinion is one of those names that people just aren't going to be able to pronounce or spell right. My first startup was called learnection (learning + connection) and we constantly had this problem. Of course, if people aren't saying or spelling it that is one thing, but I wouldn't want to close off that route.

As for the icon, it doesn't have to be the same as the logo. Why not just make the icon a post-it note. (I say this not really understanding the service yet, so feel free to ignore.)
Mythbusting: Why Facebook isn't the reincarnation of Google
I like the metaphor. Perhaps without meaning to you've also explained one of the reasons it's attractive to take VC funding. Working those 3 years without pay really sets you back, but if you're getting a competitive salary, it doesn't and it's more like "other" jobs (and arguably more fun in most cases).
Auctomatic featured in The Irish Times
Alex, this is very interesting. Do you think there might be a way to get one's hands on the original? If there is a person/persons at Yahoo to contact I would do it, as this is something I have thought about quite a bit and would love to see more about the definitions of core, light, new, long-time, large and small.

You can email me if you like.
It's behind a subscription wall so we linked to a screenshot. Are we bad people?
The original is at http://www.ireland.com/newspaper/finance/2007/0713/118419634...
The idea behind this post could be done in an interesting way. It could have looked at real languages and examined why they--despite some promise--didn't succeed. Unfortunately thats not what the article did.

Instead it asks us to feign confusion why bitfuck didn't become a mainstream language. Maybe because it was a bitfucking joke from the start?
Try WorldPay! www.worldpay.com
New all-in-one blog widget combines brag badge, analytics tool and social bookmarking buttons
can you mention any company that can deliver something well-engineered?

If yes, list the products please :)
Actually, this is a really good logo idea. Instead of trying to do to much, simplify. You could have maybe 2 post-its together with "study" on one and "stickies" on the other, and drop the book. Have the two out of alignment too, of course.

You could take it even further and treat the two words as if they were writing on different post-its with different looks :)
check this out for sure!
The Politicians see a golden goose, and they want the eggs.

No, unfortunately they are not clueless. I think the politicians know exactly what they are doing. In regulating big business, they force big business to come to them for "favors" which they are more than happy to hand out for a price.
Just thinking off the cuff here, but couldn't one store a lisp-style list as a string in a relational database. If I was using Ruby on Rails, I could simply build a simple lisp interpreter in Ruby to parse the string, then capture it as an in-memory tree. This way I would only have active items in memory and could use some of the advantages of the standard web development platforms for whatever else I was doing.

Edit: See Ruby/Lisp for one such already built interpreter.
No, not a chance. Unless they buy a completely separate company not run by their engineers to do the work. Their standard production model seems to be: slap a bunch of features together so that the user gets at least an hour or so of non-crash functionality, then spend years slowly resolving the bugs, yet pile on more features with more bugs, repeat.
So I heard this talk at the Cornell Microsoft International Symposium on Self-Organizing Online Communities. The presentation was called "Implications for Future Research; Theory and Methods in Large Scale Analysis of Online Communities." And the three Yahoo presenters were Ravi Kumar, Andrew Tomkins, and Cameron Marlow. Not sure which one those three specifically, but I'm sure you could ask any of them.

I've developed this hobby of sneaking into academic conferences. It works best if when everyone else is wearing a suit and tie, you show up in shorts and a t-shirt. That way everyone just assumes that you're friends with one of the speakers, or else that you're Sergei Brin. :-)
except if you calculate the effective hourly wage, i'm working illegally in San Francisco where the minimum wage is something like $9.80/hr
Interesting examples of what?
Sure you could. You could also select all comments on the item and rebuild the tree in-memory, as Vlad suggests elsewhere on this thread. The code isn't even as hard as I'd thought it'd be:

comments = dbh.query('SELECT * FROM comments WHERE news_id = $newsid')
response_tree = {}
for comment in comments:
comment.children = []
response_tree[comment.id] = comment
for comment in comments:
response_tree[comment.parent_id].children.append(comment)

The nifty thing about storing it in the database is that you need no application-level post-processing, you can take advantage of the DB for ad-hoc queries (what if you want to show subtrees of all posts by a certain user, like the "threads" page here?), and it happens to map to a set of simple and fast operations provided by most databases.
I wonder if there may be a way to build the wall you describe in the general-public site. Pretend you could rate commenters, not publicly but privately--for your own private use. If you spot a particularly dumb comment on reddit, you could blacklist the commenter--privately, so that your action would affect only you. Similarly, when you notice a brilliant comment you will want to permanently highlight its author so that for you--and only you--the author has an advantage on your attention in the future. Commenters you perceive as intolerable would never have another shot at your attention while commenters you view as geniuses would forever get artificially modded up--but only to your eyes.

The same algorithm could work for postings. If I want more Ron Paul, peace be upon Him, I can privately champion those authors who celebrate his news items. You may have had enough of the distinguished doctor-representative from Texas, so you could exercise your prerogative to ignore (or artificially down-mod, just for you) submitters whom you view are overly sympathetic to Him.

Further thoughts on reddit's comment system...

Problem: Logical fallacies largely go unchecked. Just today, someone replying to my comment pulled a tautology so I reflexively linked to the appropriate Wikipedia page. As mentioned, ad hominem is a more popular sin and the list goes on. What if, when I clicked on the "Report" link, I could select from several logical fallacies? In this way, if enough others agreed with me, we could get an official-looking stamp on the offending comment--corresponding to what the community felt was the logical fallacy (e.g., straw man, false analogy, non sequitur).

Problem: Today, if I submit a comment and I down-mod all other comments I increase my comment's visibility. I want my comment to have visibility (that is why I wrote it) and I suspect other redditors may employ the cheat. So I face an ethical dilemma. We can talk about real-life karma, but why tease commenters to begin with? Solution: If you comment you can't vote in the thread.

Problem: I imagine a lot of redditors, like me, sort by top->flat and I suspect most of the remaining redditors sort by top->nested. So if I comment, I significantly increase my visibility odds if I reply to the front-runner comment. To the extent I care that people will read what I write, I will tend to favor the popular comments--even if they are not as worthy of a response as those nearer the bottom.
I believe richardkulisz was also dinged. His new username is "redditcensoredme". So it does seem to only be the most extreme cases.

There are some very popular forum sites that have existed for years and maintained civility. Any Reddit-type site could learn a lot from them. They almost all have a group of core users who act as moderators and enforce a set of published rules. It's critical that they not make up the rules on the fly, otherwise users won't respect the decisions.
How MySpace Conquered Europe
"The number of unique visitors to MySpace's European sites has risen more than 72 percent in the past year, to 26 million per month, according to researcher comScore. Traffic has grown even faster: Europeans viewed 2.5 billion MySpace pages in May (the most recent month for which figures are available), compared with 1.1 billion a year earlier, says market tracker Nielsen//NetRatings."
I thought the acronym B2B went out of style in 2000?
Just like focusing on a tiny idea is immensely difficult (and powerful), so is trying to focus on a few users. The temptation in both cases is to hedge your bets by concentrating on more.

This applies perfectly to my soon-to-launch project. My partner and I have made a conscious decision to shut ourselves up about taking over the world and just concentrate on getting a few core users that are in love with our product. Stage 1.
LOL @ Willarson's comment

Reading about ColdFusion stirs up feelings of nostalgia for me. I remember being a young teen interested in web application development and somehow I got my hands on the Allaire ColdFusion Web Application Construction Kit published by Forta and Que. None the less, that was a short lived experience (fortunately) and I had quickly moved on to perl.
Combining blogs and chat rooms
After some reflection, I suppose my only hesitation with this method is that it locks one into dealing with the comments in a particular way, whereas the lisp model I outlined is easily adaptable to a variety of schemas.

However, I'm not sure how else one would want to model things. Must think more...
Can't wait for the initial release!
Just curious: why are you in Tokyo?
Versionate debuts new wiki service
I just wrote a comment on my site. I think that someone needs to spread the word about the product - more than just emails. The list focuses on development and cooperation, but there needs to be passion and a voice - a right brain.
Thanks for the post.

Here are some thoughts.
mikesabat.wordpress.com
Evangelizing Your Business
Heresy! I loved PowerBuilder and I won't have some dude saying it's dead, even if I haven't seen it in 8+ years.
Only if Auctomatic tanks.
Be sure to check out the Romlet on the sidebar of the site that is linked to - though that's just one version (the other can be seen in preview format on Romlet.com)
Who re-implemented rsync? Who re-implemented finger? What about talk?

When you start looking, you'll realize that blogger, mozy, twitter, GTalk etc. etc. are nothing more than versions of UNIX command line programs :)

They say that a successful startup will implement and modernize one of the UNIX command line programs. I couldn't agree more.
nice writeup!
Once a problem is characterized the solution often follows naturally.

Luckily there are no physical laws that prevent reasonable solutions to business/social problems of this sort.