skysight-inc/HackerNews huggingface dataset:
Ask News.YC: Filtering spam-filled titles?
Hello friends!

In my project I'm about to launch, users will be able to submit things which have titles. These titles are instantly viewable by other people, so they will be spammed. I've heard Bayesian filtering fails when the content consists of only a few words. Is there an ideal way to filter these? How did Reddit do it?

The titles aren't combined with a link, so I can't follow a link. I can filter out titles with URLs in them, though.
YouTube probably requires more bandwidth per user than all the YCombinator funded sites combined. Maybe Justin.tv is the exception.
Weblocks - A Common Lisp web framework - 3 weeks later
I don't like XML, much less its derivative/associated file formats. But if I needed to transform from XML to XML, I would almost certainly use an XSLT stylesheet. It's the right tool for the job.
He's wrong. I was in the middle of the first Bubble, and I don't think things are nearly as bad now.

Lots of lame startups are getting funded, but for far less money apiece. And it is the scale of investment that defines a bubble, not the number of companies.
Ohh, pq ... will you _ever_ have positive karma? :-P
It's a bit more than that - sharding is fundamentally a way of spreading writes. You're limited in the number of writes you can make by the speed at which the disk rotates, so this quickly becomes a bottleneck, particularly in heavily-trafficked social sites. Sharding lets you spread your writes out among an arbitrary number of machines, so you can grow with traffic.

Big sites that are primarily read-hungry often don't need to shard at all, instead relying upon replication. Look at Craigslist - they have a single master database, about 20 slaves, and an archive cluster for postings that nobody looks at anymore. Or PlentyOfFish - 1 machine for billions of pageviews. Social sites with lots of user-created content need to shard much earlier - LiveJournal has a fraction of the users as Craigslist, but they had to start sharding around 2002/2003.

There're locality concerns as well, but they tend to be secondary to the write issue. Disks are big, and with good caching you can often avoid hitting the disk at all.
You missed PQ.
It used to be called horizontal partitioning before Google started using the sharding term with Google Filesystem. It's been around for a while. The new part seems to be notion of pushing it down into the application code instead of paying Oracle or IBM a few million dollars for a DBMS that handles it automatically.
That's not necessarily the case. For example, consider IRC, once the bane of many college students' academic records: Many sites and services actually have negative value, by being addictive time sinks.

Economic theories tend to neglect considerations like people's self-destructive behavior; or they re-define it to make the theory work ("it's not a bug, it's a feature!"). The classic example is of a destructive drug habit.

The addictive element creates a larger user base, but its value is highly debatable: EverQuest was nicknamed EverCrack and World of Warcraft is similar; addictive elements also play into gambling sites, collectible card games, instant messaging, and even sites like Reddit. Anything that sucks up your attention and wastes your time is a net liability. Television sitcoms create the "value" of watching someone else's fake life by withdrawing your attention from your own, real one. (And it initiates a cycle, as the more you watch, the less of a real life you have, the more you'll turn to the comforting artificial alternatives.)

Thus, I don't subscribe to the idea that merely because people are using something, it must have value for them. It has value for the purveyor of the service or substance; and due to corporate self-interest, that is often confused in business with value for the user.

If lacing a drug with plutonium made it a hundred times more addictive, most corporations would have no problem selling radioactive crack (and killing off all their users), as long as it made the numbers work for this quarter.

If life is made torturous so that selling an anesthetic is big business, claiming it "creates value" is begging the question in accepting that life should be like that in the first place. Almost all mass media, including sports, and including the aforementioned addictive games and services, are forms of anesthesia.

The real value would be stopping the death by a thousand cuts, not anesthetizing ourselves to it.
The ITConversations interview with Scott Cook is one of the most useful on the site:

http://www.itconversations.com/shows/detail523.html
I meant to post this as a good thing for web 2.0, but for some reason my comments never showed with the original post. If Dvorak is calling it a bubble it might be a good thing... considering his record.
One huge difference is that you don't have an insane fund raising model:
1) Buy domain
2) Go public
3) Build product
4) Get users
5) Monetize

It is harder to have an equity collapse when companies are small and barebones.
Is it ethical to keep a copy of the code or designs you did for your employer, for your reference?
Please comment on the two:

A) Extreme case:

[deleted]

B) Normal Case:

You keep a copy of your work and designs so you can reuse the ideas and perhaps even snippets of code. It is also a record of your work.
This is for 'normal' work that took lots of thought and effort, not patented stuff.

Qs: How many of you regularly do B? Please also comment on A
Often it's behaviors that suck, not people.

If you dislike something someone has said, the best way to deal with that is to reply explaining why. That's more work than just downmodding them, but it can pay off in the long term if you can turn someone into a useful contributor to the site instead of just making them leave.
The reason Facebook was down...   it wasn't a proxy issue
I would rather miss a market than put out something I was not fully proud of.
Right. There is no such thing as downvoting someone's karma. What you vote upon (unless you abuse the system) are individual comments and submissions. Karma is an overall score, not something you edit at will.
I think we're stuck in a Bubble Bubble. (Or is it the Metabubble?)

You'd think that after living through two real bubbles we might develop some higher standards. Instead, the word "bubble" is getting applied to every fad, era, technology, and press release that comes along.
Are unoriginal ideas bad?
I didn't mean they don't have courses claiming to teach it.
The market decides.

They'll decide what valuation to offer you; you'll decide whether or not to take their offer.
Disney Buys Club Penguin
Answer to the question: "How's your startup going?"
We all get this question a lot. I now have an answer that everyone can understand:
My startup is like mission control in the movie "Apollo 13." There are a bunch of nerds running around talking intensely about things that nobody understands. Machines are lighting up making sounds that continually overturn the already chaotic nerd anthill. We've solved a number of complex problems with very limited resources and lots of duct tape. A bald guy is loosely in charge. (Specific to our startup.) We will probably not make it to the moon but if Tom Hanks and Kevin Bacon make it back alive it will be a huge success. They just went out of radio contact on the dark side of the moon and we won't know if we'll ever hear from them again. There are a number of huge outstanding issues that we will have to solve at the last minute. The ending will be very happy or very sad.
Try posting your job opportunity in places like this one. I'm sure there are people interested in startups that read yc news but that are not working on a startup right now.
excellent article. I agreed with most of it--especially about startups attracting freaks and misfits. I can vouch for that. haaha
The startup I was in looked primarily towards referrals from the investors and the existing employees. We also submitted a gig to the 37 Signals board, and I think there were some good candidates there. Most of those people seem to be on top of their game and favor the simplicity and ease that 37 Signals preaches (which I consider to be a very good thing).
In the most successful startups the first employees are usually people the founders knew in school.

In fact, that's probably the main reason startup hubs tend to form around universities.
How would one know what data to place in shards? Are there any "RDBMS to Shard" resources out there?
I'm struggling with the same thing. A post i wrote recently (http://www.pumperknickel.com/?p=300) covers a couple of suggestions.

To date I've had most success with craigslist ads. Cheap at $75. But you need to be very clear (not like your typical job ad) in your posting to make sure you really get what you need.

Good luck!
Ask News.Yc: What is social bookmarking lacking today?
I know, i know...they are a tons of social bookmarking sites out now, but they all seem cluttered, slow, and confusing to me. Im trying to get a feel of what features poeple like, dislike or they would like to see implemented.

News.YC is sort of a bookmarking app (at least the way I use it).It works great for me because its fast, has a clean layout and it just so happens to deal with web pages I would typically bookmark. but the internet isnt all about startups and entrepreneurship.

Any input would be great. Thanks
Big Blue scraps servers for big power savings
Possible Source of Adpinion's Idea?
This is further evidence that ideas are worthless until you implement them.
Unwired Nation, BlueGrind turning online text into audio
YC News in the car - could be nice!

What about a radio station that reads the top stories on digg or reddit?
YC News in the car - could be nice!

What about a radio station that reads the top stories on digg or reddit?
Disney Acquires Club Penguin in $700 Million Deal
Which is, I imagine, exactly why you can't downmod replies to your own comments.
Scoble Declares 20's vs. 40's Age War
Scoble's Top 10 Facebook Apps vs. TechCrunch's Top 10 Facebook Apps
Data Visualization: Modern Approaches
I applaud your ability to maintain an optimistic viewpoint about humans even though all evidence points the opposite direction. What you're suggesting is completely unprecedented on the internet. What always happens is that an open forum is eventually overwhelmed by trolls and morons, and all the smart people give up. There has never been a case where a troll or a moron turned into a useful contributor because of thoughtful response from a good member of a forum. Just review the history of Digg, Slashdot, Reddit and almost any other open forum, all the way back to ancient USENET times...

Wasn't this forum created partially because reddit was overwhelmed by the hordes and all the interesting startup information was lost in the noise?
The article doesn't say it wasn't a proxy issue. There definitely was a security problem, but that doesn't mean it wasn't caused by proxy issues.
What about for articles off the front page?
You'll very much enjoy reading http://www.shirky.com/writings/group_enemy.html

But no, it was created primarily as a test for Arc. The fact that it's so beneficial and awesome is a wonderful side effect.
Great article and speaking a someone who is actively working with the Netflix prize data on a daily basis the summarize some of the more bizarre quirks of the dataset pretty well. The other problem that he didn't really get into is the sheer size of the dataset, the .txt files are about 4GB in size to hold the 100,480,507 ratings ...

Beyond that I think he missed the really interesting ideas behind the recommender system possibilities. Once a user has enough ratings on movies and there is enough of a cross section of other users rating the same movies the math of the problem starts to create a profile of the user. With that profile it is more likely to be able to find the things the user will find interesting instead of just what the crowd voted most popular.

The real trick is not to use some artificial error measuring method to validate how your algorithm is working but instead to focus on how the recommendation system changes the way the user interacts with your website.

The Netflix prize did a few things that where wonderful for the process of recommendation system, they brought a lot of attention to the field and they gave a huge dataset for people to compare their results with. In the long run, even if no one wins the grand prize, they will have give much more to the field of study then the $1 million.

Oh yeah .. and maybe this January there will be one more YC start-up doing recommendation systems.
It's better to discriminate comments instead of people.
One word: community.
Say someone wants to buy Loopt for $20MM...
Can Sam go back to the investors and say, "Here's your money back plus x, I'm doing it" since he has the majority ownership? Can you structure a deal such as long as they get the investment back, you can sell at whatever price you want?
OK, I was contracting in my previous life so I've seen these issues pop up many times before. Here's few words.

A) is illegal. You have no ownership of ANY code or IP that you work on at the company that employs you. That's what they're paying you for, after all.

B) is also illegal. Unless the company in question has open sourced their code and allows anyone to use it under some kind of a BSD-like license, you have absolutely no right to use that code or IP.

Not only are both of these cases illegal, they're also highly unethical. Also, if you're forming a company around someone else's IP, you're gonna crash and burn.

When I worked as a contractor, I would use my own libraries to save me some time but before I used them, I notified the management that I was using them and told them that I own the code and that I give them full right to use it without any licensing fees and I also told them I would use them on my other contracting jobs. They always accepted these stipulations an I've never had any problems. They loved the fact that I was not wasting my time and their money re-implementing the wheel and that I was solving their problems in record time. The key is to be open and put it all on the table before you start working.

Also, after I'd finish working on a project and if I was using my own computer, I'd wipe my computer clean of all of their code. I would do that on the last day of my job and I ALWAYS made sure that someone from the management was there to witness it. Sure, that might be going overboard but trust me... it's worth it. It shows them that you care about these sorts of things and it also makes sure that they won't have a strong case if they ever decide to sue you down the road because you end up inventing something amazing that's in their marketspace or you end up working for competition.
And their families have no idea if they'll ever be home...
Say someone wants to buy teen wag for $23MM...
Can the founders go back to the investors and say, "Here's your money back plus x, I'm doing it" since he has the majority ownership? Can you structure a deal such as long as they get the investment back, you can sell at whatever price you want?
dude loopt so scammy they dont have users lol
I've been working with a programming language called Erlang for about two years now, and their RAM based database called Mnesia almost forces you into this 'sharding' style.

In Erlang you also get the ability to have the same database on multiple computers or fragment the database into different servers that hold only a certain piece.

It certainly was a trick to unlearn normalization techniques I had been using for years, but once you get he hang of it the data really becomes or efficient.
Say someone wants to buy TEE nwag for $40MM...
Can the founders go back to the investors and say, "Here's your money back plus x, I'm doing it" since he has the majority ownership? Can you structure a deal such as long as they get the investment back, you can sell at whatever price you want?
For the perhaps less hardcore, and less Blub averse, F# is a caml implementation/extension? on .net. Less/no need to write homegrown, production quality c/c++ interfaces, since it integrates with the rest of .net. It will compile some/most ocaml code without too much change. I can't help but think using it, or at least being aware of it, can benefit some software startups.
Both are unethical and NEVER acceptable. Run, don't walk, the other way from those who rationalize this behavior.

(Why is it that when it comes to ethics, what was once clear cut is now "fuzzy"?)
Dvorak is so horribly bad at his job! I don't understand why they even let him on the Internet anymore.
A bug let users view the inboxes of other users. This may or may not have actually been caused by a proxy issue.
Ask Steve #4 - Idea Brainstorm for MusicExplained.net
So I just give them my best estimates of revenues and expenses and they'll make an offer based from that? How long into the future should I project?
Man.. $2.89 billion. Can you imagine it? Creating something and being able to say, "This brought in $2.89 billion this quarter." I wonder what it feels like.

What's it feel like to create something that turns a profit?
guys looopt sucks.. it is run by gay sam and has no users
Perhaps you can use your ideas but certainly not your code.

I managed to open-source some parts of my code (basically some generic libraries) under a very liberal license and to reuse it in other companies. Not without agreement with the employers, of course.
It could very well be that, for most people working 'inside the bubble', this one may, in retrospect, have ended very similarly to the last one. A lot of people essentially spent a bunch of years working for chump, in widely shared anticipation of riches that never materialized.

At the same time, since not nearly as many outsiders are betting their life savings on the 'bubblers', the economy wide impact of even the most devastating sector wipeout, won't be nearly as bad as dot bomb.

In that respect, how you'll come to view it will depend largely on where you stand.

It could also be that what we are seeing is the leading edge of PG's predicted trend for more young people to enter working life by doing a startup, rather than going to work for someone. This would probably in and of itself lead to an economic landscape looking a bit 'bubbly' by historical standards.

I guess only time will tell.
Another huge bandwidth consumer is http://www.ytmnd.com

It's a site that lets you upload a gif image, a sound, and some text. You pick a subdomain (for example, "something"). Then it puts all those on http://something.ytmnd.com. Each day, around 100,000 of these are viewed, so it chews up a huge amount of bandwidth.

Max is the owner of that site, and he's been living off of the advertising revenue at about the same living status as someone working full time at McDonald's. Last I heard, the advertising pulls in about $20k per month, and the bandwidth takes about $18k per month. He works all day and night on it.. it's amazing. I don't know why he hasn't sought investors or anything like that, but it's a huge community (even though it consists largely of teenagers).

Any investors who like investing in that sort of thing might want to contact him.
It may or may not be illegal or ethical, but it is rarely useful. I used to keep most of the code I've written for employers over the years, but I never once looked at it.
Thats a given... I was wondering along the lines of technical features.
You are totally wrong. In a contracting situation, the CONTRACTOR owns all the code he or she produces unless the contract specifically states otherwise.
But you've another problem: you're looking at investors to get you out of debt and that's another thing they don't do;)
The contract almost always states otherwise though. Unless maybe you're working in an area where the companies doesn't care about their IP or are just simply ignorant.
Article about the SuperHappyDevHouse in California
Well, you have to enable a community through the technical features, is the thing. The community has to be intrinsic in the tech.
All I see are a lot of statements without supportive reasoning. The article reads something like this:

"We're in a bubble!"

"CD-ROMs ... Pad Comp--SPREADSHEETS!"

"People are talking to each other on the Internet! Even in Second Life! It's crazy!"

"There are videos on the Internet!"

"People are making things on the Internet!"

"Google thought of something I wish I had!"

"Widgets!"
5 Distractions To The Nesting Entrepreneur
Same here. My feeling is that being ethically compelled to re-implement something you've done before is a blessing in disguise.
There's this new tech that could be useful in helping you two communicate ;-)
I disagree. I think the last bubble was marked by the high number of tech IPOs that were so over valued. Now, we're not seeing as many IPOs of these companies with virtually no product. Instead, we're seeing a lot of VC money being thrown around and a lot of acquisitions. So, I don't believe it's really going to burst. When everything catches up, a lot of VCs will be out of a lot of money, but if you have a good idea and a solid audience, you'll be ok.
Jerry Yang (Yahoo Founder) could learn from the NFL's Chad Johnson
Maybe limit the number of "voting points" you get per day based on your karma? Works great at Perlmonks. Whoever it was went through my posts as well. I just figured I had pissed someone off with my Disqus post :-)
Well, sort of true:

http://ycombinator.com/announcingnews.html

Yes, because of reddit. Yes, because of Arc. But mostly because it's a good way to learn about people.
Yes, I am from the philippines, met him at a uni here think it was a forum on entrepreneurship.
I notice I haven't read TMR since it became a pdf this year. Source text of the article seems to be available for those who prefer such: http://sneezy.cs.nott.ac.uk/darcs/TMR/Issue7/JaneSt.tex

I have to read about the company later, but let me at this point add a link to more such stories at the Commercial Users of Functional Programming web site: http://cufp.galois.com/
There is a bit of craziness going on... A lot of people are spending a LOT of money on eyeballs and a lot of people have lost track that eyeballs and revenue don't necessarily mean the same thing.

There also seems to be a general disregard for business models (which I know can come later, but it's GOT to come SOMETIME).
I commend you for your firm sense of ethics, but what exactly are you saying is unethical?

For example, once in a while I will come across a problem that I know I have solved before, so I will refer to code I had previously written for someone else to remind myself of how I first solved it. I don't see any ethical dilemma in this.
An mp3 Question and Answer session with Paul Graham on Great Hackers
This is the other extreme - which I didnt mention in the op.

I agree that copying code is wrong - but what about a good idea that you come up with, or a neat design or technique in code?

It seems unfair given that doctors, scientists, marketing and many other professionals can keep a record of their work. Maybe not reams of data, but something. (I am not 100% sure of the extent they are allowed to). All art professionals have portfolios.

Is your brain just a vessel to be washed clean before it is rented out again?

Again, I am not talking of secret algorithms that would break a company, not someone else's work.
Perlmonks is the best example I've ever seen of maintaining a great place for smart people. It's been running non-stop for a decade. It's very open (even allowing anonymous posts) but has community rules that mostly reflect common decency. Over the years I've seen countless disrespectful newbies and trolls turned into contributing members of the community. The solution there has been to have long-time members volunteer to enforce the published rules in a consistent and fine-grained way. It requires implementing a user hierarchy, but we already have that here with the "editors". Publishing some rules and systematizing the editor controls would solve most of the problems reddit has faced.
No, Justin.tv is not even close.