skysight-inc/HackerNews huggingface dataset:
Much better! Just adding pagination means I don't have to worry about missing new comments if I crawl too infrequently.

But the crawler is almost entirely unnecessary now. If you could make it convenient (ie no reload) to show the ancestors of comments (the context of each conversation) that pretty much covers all the features I built.
Why don't URLs in Ask YC posts appear as links? e.g. http://hystry.com/newsyc/follow/ in pchristensen's submission?
Isn't this a dup of: http://news.ycombinator.com/item?id=43074
Google Trains the Next Generation of Adword Professionals
Yeah, you're right. How about now? Kind of interesting modifying software on the fly in response to user input. I wonder how often this happens.
Kind of interesting modifying software on the fly in response to user input. I wonder how often this happens

Funnily enough, I was just thinking that. New version works well for me.
Awesome. Can you add the same thing to each user's comments page, too?
Yes, rsnapshot is excelent. The documentation is good too.

We also use gpg (gnupg.org) to encrypt our critical databases and code repository and send them to Amazon S3 every night. I started doing this after the last earthquake here in the bay area (which was centered near Freemont, where our colo is). I have tattooed the key to unencrypt them on my inner thigh.
To discourage the trick mentioned in the sixth paragraph:

http://ycombinator.com/newsguidelines.html
"Waste is wonderful, it's a resource", says Indian energy startup
I like the current iteration a lot.
Top-Down: A New Approach to the Semantic Web
false alarm. http://www.badastronomy.com/bablog/2008/01/16/no-alien-signa...
Entrepreneurship Takes Off in Ireland
Anyone want free tickets to the Xconomy meetup in boston?
If anyone is interested in going to this awesome Battle of the tech bands event, comment here with a way to get in touch with you. I have a 10 free tickets to give away.
Some Startups Use Digg -- FreshBooks Is Using an RV
Entrepreneurship: Trust your gut feeling
That'll learn ya.
Tips to become a successful entrepreneur
Perfect! I actually thought about doing it that way (root next to link | parent) while I was driving home. The only observation is that all of the Ask YC type discussions (not links to other sites) have news.ycombinator.com as the url (without the link to the specific conversation). Other than that, it's perfect (until the Arc customizations are released)!

This is certainly the best feature request I've ever been involved with! It reminds me of http://lib.store.yahoo.net/lib/paulgraham/bbnexcerpts.txt :

"When one of the customer support people came to me with a report of a bug in the editor, I would load the code into the Lisp interpreter and log into the user's account. If I was able to reproduce the bug I'd get an actual break loop, telling me exactly what was going wrong. Often I could fix the code and release a fix right away. And when I say right away, I mean while the user was still on the phone.

Such fast turnaround on bug fixes put us into an impossibly tempting position. If we could catch and fix a bug while the user was still on the phone, it was very tempting for us to give the user the impression that they were imagining it. And so we sometimes (to their delight) had the customer support people tell the user to just try logging in again and see if they still had the problem.
And of course when the user logged back in they'd get the newly released version of the software with the bug fixed, and everything would work fine. I realize this was a bit sneaky of us, but it was also a lot of fun."
AOL Rumored To Be In Talks To Buy UK Social Travel Site WAYN For $200 Million
That's a very interesting comment tx. I never quite thought about it that way, but that doesn't mean I'm convinced you're right.
Pretty standard stuff we've seen over and over again.

That being said, there was one thing we've all heard a million times that can never be said too much...

"Passion pays off"

He goes on...

"over 80 percent of millionaires say they never would have been successful if their vocation wasn't something they cared about."

A necessary (but not sufficient) condition.
This rant would be worth reading weren't it for the fact that programming language provability is but one of many areas of research in CS, which is something the author doesn't seem to realize in either of his two rants.

Almost no one cares about program provability, well gee golly, I'd never have known!

Why should anyone who needed years in grad school to figure that out even be taken seriously?
"... It was really pounding the server. Crawling for a search index is ok, but retrieving pages to dynamically update stuff generates a lot of requests. ..."

I hit the RSS feed every 15m and have so for months. I call the RSS file, extract the necessary data from this file, then re-hit user pages once [0] to get their contact info. Some of the things I looked at to try and reduce hits is to check some basic HTTP options [1]:

- 'Last-Modified/If-Modified-Since' to only take, if updated

- 'ETag/If-None-Match' checking admin defined time to reload

- 'gzip' to reduce size of downloads

But none of the above are implemented (probably for good reason - complicated & not really required) checking the HTTP spec implemented is pretty bare. I only found server status. I could also check the the robot.txt file, but like I said I only download the RSS file and user pages. A quick check for "robot.txt" reveals none.

So you could automate/enforce crawling rules using a combination of HTTP and/or robot.txt files. But this does require the client to correctly implement them. Having said that increasing the RSS feed from 25 to 30/35 entries to cover the front page [2] is about all I can really think of.

[0] Have to check this. Mainly get contact info & cache if found.

[1] The basics are found here: http://www.diveintopython.org/http_web_services/http_feature...

[2] Having only 25 entries means you might be tempted to grab the front page (and maybe the second page) instead of RSS because you miss the bottom 5 entries.
Gandalf's psuedo blog
OK. I bet I'm the only one on this board using this technology...

http://www-306.ibm.com/software/data/u2/universe/

EVERYTHING is stored as data on the u2 database. All javascript, css, html, php, and mysql is generated from this data through custom builds. No need to backup anything except the source data and the custom programs used to generate everything else. And all of that is within the IBM/u2/bin directory.

Have a disaster? Restore the environment, restore the u2 database & code, and rebuild everything else. I designed it for the development leverage. The ease of disaster recovery is just a bonus.
Executives vs Leaders
http://www.amazon.com/Presidential-Ambition-Gaining-Power-Co...

Any surprise that almost every single one "married up"?
You've picked well :-)
As a mouse shover and a yc news reader, you would come under the demographic that blocks ads probably even uses lynx. So, "they" would safely count you out of their target audience. Ha!
Well, h.264 is a fine codec, but just like aac, it offers a range of quality, and better quality means bigger files. I haven't played around with it to see how it handles motion, but I know with DivX/WMV that's always been a big problem. I can't imagine watching an action movie in compressed 720p is going to be anywhere near as good as uncompressed 1080p, but then I also can't imagine most people will know or care about the difference.
Couldn't offending sites simply use ssl?
Do you think I would have done something as heavy-handed as compile s-expressions at runtime if it weren't necessary?

We had a page description language called RTML that was Lisp underneath. Users created templates in it, which we then executed to generate their pages. It is these page templates that are still stored on disk as s-expressions in the C++ version.
server wise i use a very simple script to dump an archive to S3. http://paulstamatiou.com/2007/07/29/how-to-bulletproof-serve...

but for personal computer files I manually put important files in an S3 bucket or two. not the most efficient but it works. I'm considering writing some rsync to S3 type of thing. regardless i love having my stuff online (and safe). my MBP doesn't have much other than essential apps and a few files i'm working on at the moment. most other things are online.

the only exception being media which i sync to an external drive manually every few months.
This is a step beyond that. I'm not even doing releases; I've been pasting the changes into the running server's repl.

BTW, the urls of the ask-yc items are a bug. Will fix.

Edit: Should be fixed. I replaced

(link (ellipsize (s 'title) 50) (s 'url))

with

(link (ellipsize (s 'title) 50)
(if (empty (s 'url)) (item-url s) (s 'url)))
Ask YC: OfficeZilla.com at Twiistup right now, help me win.
The YC board helped me find the minor amount of angel funding I needed to buy new servers and better rackspace.

I signed up for Twiistup.com expecting not to get in but here I am now. If this article doesn't get removed I have an opportunity to get $5,000 in Yahoo Search Marketing which would be just very huge.

The instructions for voting are here
http://www.mozes.com/_/leaderboard/display?id=186&projected=1

It is free. It would be a huge thing!

Thanks

George
OfficeZilla.com
How to complete your PhD (or any large project): Hard and soft deadlines, and the Martini Method
Crunchies: Here's The Winners' "Award"
Tip #1 is a very interesting one...
SVN for code. Time Machine for OS X. Typical DB saves for the web, along with server image backups at Slicehost.
backup? what is that?
Facebook Up, But MySpace Still Leads By A Very, Very, Very Long Way
Surprise, Justin.TV Builds Own CDN To Cut Costs, Improve Performance
Steve Jobs praises a retiring Bill Gates
In Unix, what do some obscurely named commands stand for? - Knowledge Base
Finally  Instant Boot-Up on Windows Machines
Ask YC - comprehensive image gallery software
This is probably the wrong place to ask but I've been looking around for great image gallery software before I resort to building my own or hiring a fellow hacker with more time on their hands.

I want to expand my site with a place to put all the images from my reviews and so on. I love what weblogs inc has done with their recent image gallery revamp: http://www.autoblog.com/photos/detroit-2008-desktop-wallpaper-gallery/586793/

Particularly I'm looking for something that can auto-watermark - probably imagemagick-based. For clarification this would be hosted on my server and fully integrated within my setup.

would love any feedback
well .. some pieces of the article are not even wrong
pg, I think the current implementation has a problem. Most of the time, "on:" links go to NYC comment pages, but when the comments were done replying to the posted article (and not to other comments), "on:" links go directly to the posted article / website. I think all links should go to the comment pages always, to keep the "on:" behavior coherent.

Thanks a lot for implementing this feature, that was fast!

Edit: this happens only on the "threads" page I think.
I've been using the new /newcomments. One other thing I noticed that's missing: it's not stateful. There's no way for me to kill off some users/threads so I don't see them again. I think my query mechanisms were pretty elegant and lightweight. hystry permitted filtering conversations (rather than comments) involving specific people, and also easily blacklisting going-nowhere threads.

http://blog.hystry.com/post/20252078
http://news.ycombinator.com/item?id=83702
Intel and Apple: Best Friends 4-Ever
You may be interested in Duplicity, which can do encrypted backups to S3: http://duplicity.nongnu.org/ Duplicity uses the rsync algorithm for data transfer as well.

For those creating archives, you may be interested in xar: http://code.google.com/p/xar/ It preserves all the metadata such as SELinux information, ACLs, EAs, etc. On Mac OS X, it preserves all that, the resource fork and finder metadata as well. Two options are particularly relevant to backups:
--link-same makes identical file data into hardlinks, reducing space consumption both within the archive, and on disk when extracted.
--coalesce-heap when files with identical data are encountered, the data is only stored once in the archive, reducing the size of the archive.
> Do you think I would have done something as heavy-handed as compile s-expressions at runtime if it weren't necessary?

Why not, if the language makes it simple? I am not claiming that Lisp does not have more powerful abstractions than C, just that they are not usually essential to any given feature in the application.

If Lisp is actually part of the user interface, then clearly there is no way to avoid having a Lisp interpreter. This is begging the question though.
Will this be taken down? Summer 2008 Form?
Just Becase You Can, Doesn't Mean You Should
Tin Foil Hat Alert: Paypal Messes With Ron Paul Campaign
Ironic, given Peter Thiel's political leanings.
Ironic, given Peter Thiel's political leanings.
Why would it get taken down? I imagine that pg et al. are reviewing and tweaking it in advance of announcing that they're accepting applications.
I think that's a feature. It's pg's call as to which way is better, but right now it behaves like hystry did and how I prefer it.
I meant will pg be taking the link to it down.
Justin.TV seems to have a very interesting startup model. An apparently trivial "outer" application which needs a good deal of innovation "under the covers" to pull off, which is where the real competitive edge comes from. Pulls in a different direction from how most people parse "build what users want".

I wonder how they pitched it to YC (and other investors).
Self Destructing Palm Tree
Doesn't look like it's ready yet, note the (asterisks) instead of the due date.
Scrabble owners try to shut down enormously popular Scrabulous facebook app
http://www.parc.com/research/

I'm willing to accept that you are more than capable than predicting what's interesting. Let's try a zero knowledge proof. You point out the interesting ones, and we'll check back in a few years to see your accuracy.
they didn't pitch it to yc. IIRC their initial project was a web calendar that died when google did theirs. I think most of the current stuff they figured out when Justin did his initial stunt and it just picked up from there.
I've been trying out wuala - http://wua.la/en/home.html
Maybe we're so far away the only value of our planet and species is for alien anthropology majors writing theses. They'd look at how miserable life is for sentient beings to live in a time when object-oriented programming is in vogue.
>> I wonder how they pitched it to YC (and other investors).

AFAIK, Justin and a friend took part in the first YC session with a web calendar called Kiko, which they sold on eBay for 5 or 6 figures to a media company after Google came out with theirs. They started working on Justin.tv to have a product in a market which didn't exist at the time, mobile streaming video, to have a better chance of survival.

I believe Justin et al have been working on Justin.tv even before Google purchased YouTube, and are funded by investors closely linked to YCombinator who enjoyed his YCombinator calendar startup.
Wars are "expensive" in that they cost lots of money, but when america wages war the money that is spent in such great amounts is spent on American troops and government contractors, who in turn take the money they earned and spend it on other goods, so it does not "destroy" the money. In fact, war is generally an economy booster.

Dude, broken window fallacy: http://en.wikipedia.org/wiki/Parable_of_the_broken_window"
As a programmer who has a preferred Blub language myself, I saw it more as a flip comment than any sort of name calling. Until someone actually sits down and invents the "100 year language", it won't matter much if you're using a powerful language or one of the lesser blubs.
I'm pretty sure tx was being sarcastic.
Worry not. This article is probably a honeypot for people whose votes will never count again on Hacker News.
I like Ghost on Windows.
EnterpriseDB CEO on Sun buying MySQL
Absolutely Unbelievable: Richard Stallman Crankin' Dat Soulja Boy in front of the Green Building at MIT [video]
Boooring. I'll do it tomorrow.
Why schools should exclusively use free software
Eponym of ReiserFS is on trial for murder.  What the fsck?
"funded by investors closely linked to YCombinator"

And maybe a little from YC partners, too...
YC is the closest link to YC, smarty.
Illegal pictures of Britney Spears? I get the vague feeling that this guy doesn't know what he's talking about.

Also, any amount of traffic filtering can be done on the side without slowing down the traffic. You can always kill the rest of the packets after letting a few through.
I usually put the car into reverse, turn my head to look out the rear windshield then apply a little gas.
Wired did a feature on this, if you want a lot more info.

http://www.wired.com/techbiz/people/magazine/15-07/ff_hansre...
Wow that would be awesome! And if everyone can see and vote on each other's little Arc program, the best one(s) will win.
A praise bit late I guess ;)

"Being the richest man in the cemetery doesn't matter to me... Going to bed at night saying we've done something wonderful... that's what matters to me."
- Steve Jobs, On Gates and Microsoft, Wall Street Journal, Summer 1993

ref: http://en.wikiquote.org/wiki/Steve_Jobs
is it not hard to be a more advanced civilization than us?

There has been millions of species of creatures on Earth over the time it has been around. We are the most advanced, and frankly the other species aren't really close.

Perhaps we are pretty special.
xkcd - Compiler Complaint
Beauty of Ruby
I don't think running your own software on EC2 qualifies as building a CDN.
Um, there are lots of pictures of Britney that are copyrighted and therefore technically illegal to distribute over a network, except for purposes protected by fair use. Indeed, I'm sure that most of her pictures are under copyright. Do you know of any CC-licensed Britney videos?

Whether or not anyone still intentionally downloads B.S. photos is another matter... but, you know, law professors don't tend to be teenagers. They don't keep up with these things.
It just uses EC2/S3 to push off excess bandwidth.

Bandwidth from a company like Alpha Red works out to about 3 cents/gigabyte if you buy it in 50 megabit per second chunks, or 6 cents/gigabyte if you buy it in 4tb per month chunks.

S3 is 18 cents/gigabyte.

Megabit per second is cheaper if you can keep it at 50% utilization, so baseline bandwidth is actually quite cheap. You'd be crazy to be that bandwidth intensive and pay 18 cents/gig for all your traffic. It adds up quickly.
What is wrong with these python people?

Just because they are given no options by their language of choice and are forced into following strict and arbitrary rules, they think a language that provides options is 'unclean' and 'messy'.

What is wrong with a language that lets you express yourself in whichever way seems most intuitive to you? The only thing that comes to mind is future maintainability when the code changes owners.

tim-toady all the way.

</flamebait>
"Please tell us something surprising or amusing that one of you has discovered. (The answer need not be related to your project.)"
rdiff-backup locally; duplicity onto an rsync.net system.
P.S. Don't forget your towel
Wow, are techcrunch comments the new youtube comments?
YC Startup Loopt Launches Location Applications Platform
Are You This Agile? Paul Graham Changes Hacker News While You Wait
Monkey Brains In US Makes Robot Walk In Japan
My Friend Flickr: A Match Made in Photo Heaven (Library of Congress Blog)
I use rdiff also. It is an rsync mirroring system
http://www.nongnu.org/rdiff-backup/
"... let users write little Arc programs to control the way pages are generated ..."

Marc Andreessen has an article that describes how Ning is designed as a "Runtime Environment" just as you have described ~ http://blog.pmarca.com/2007/09/the-three-kinds.html

"... I believe that in the long run, all credible large-scale Internet companies will provide Level 3 platforms. Those that don't won't be competitive with those that do, because those that do will give their users the ability to so easily customize and program as to unleash supernovas of creativity. ..."

It would be a good way for hackers to develop, share, debug Arc code. Boost the language develpment using the pre-release code.
Microsoft patents process to allow remote monitoring of workers
One of the Best Presentations on Poverty and Development (video)
Yeah, I'm still waiting for some evidence for that particular claim. You might think it "bad", and bad as it might be, I'm perfectly willing to believe whatever the evidence points to. There just isn't much.

By the way, for anyone who missed it, the media ran its usual hit pieces against Ron Paul, accusing him of racism, etc, etc. Hence my (admittedly lame) joke.
I probably agree with half of these criticisms, but they are all pretty minor. I wish ruby 1.9 would get a cleanup, but I think it is just getting more features.
(defopr s2008form req (item-url (item 99665)))
Look like your boss
I'm reminded of Duff's Device
http://en.wikipedia.org/wiki/Duff%27s_device

send(to, from, count)
register short *to, *from;
register count;
{
register n = (count + 7) / 8;
switch (count % 8) {
case 0: do { *to = *from++;
case 7: *to = *from++;
case 6: *to = *from++;
case 5: *to = *from++;
case 4: *to = *from++;
case 3: *to = *from++;
case 2: *to = *from++;
case 1: *to = *from++;
} while (--n > 0);
}
}
The best confluence of hacking and hip hop since Monzy's "Drama in the PhD"

http://graphics.stanford.edu/~monzy/DramainthePhD.mp3
I need yoooouuu to put down the laptop before you start doing club dances.
More accurate title: Math books a Ph.D.-holding theoretical computer scientist can actually read at least 90% of, given several tries over 20 to 30 years, possibly being required to twice teach a course on the subject.
Upmod for clever Unix joke.
He's the worst dancer. I remember reading his bio about how he spent years training as a folk dancer. Guess it didn't do much good.
I keep any sort of file I really care about in a personal svn repository. Mostly this is either code, PDF's, Word Docs, Excel, text files, and so on - Not too much video or anything like that.

I then use DatedBackup (http://datedbackup.rubyforge.org/), to backup the repos, and a few servers, which is basically a wrapper around rsync. It more or less mimicks the functionality of a simplified time machine without the gui - keeping snapshots at certain intervals, and deleting them when the drive gets full. It uses a technique similar to rsnapshot, so extra space doesn't get wasted by every snapshot.
Makes sense from that perspective.
A decentralized Twitter?
Dash: Your Car Gets An API
Ok, that sounds like a more convincing argument than the original one.

Speculators are risking their own money re-aligning badly-allocated capital in the economy and correct prices. If they are right they earn a nice commission.
Special invite to news.yc'ers to apply--let me know if you want to build something