skysight-inc/HackerNews huggingface dataset:
Not necessarily. Let's think of some operations a crawler would need to do:

1. Spawn some threads (Python uses native threading)
2. Connect to and load some URLs (This isn't going to be slow anywhere.)
3. Run some regular expressions (Python's regexp engine is all C)
4. Write to the disk. (Python uses the standard system libraries for this)

It wouldn't be as fast, but it might be surprisingly close. There would be fewer lines of code to boot.

The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time. It sounds like the man only has 1 processor, so that wouldn't be the end of the world, but if he has more, you can just swap in the process module for the threading module. Of course, then you might have some RAM issues.
Well poop! If I wasn't neck deep in the middle of a project I don't want to quit, I'd apply.

I'll remember to check if you guys are still hiring 6 months to a year from now.
Ignorance, the Ultimate Level Boss (applications of the scientific method)
Since the author of this post is Jeremy Allison, a major Samba contributor, I also suggest reading the Samba code. I've been working with it, and have found it clean and well-structured. A great example of how to write a networked filesystem.
I actually learned a lot about coding good PHP by learning Rails. I'm not an expert by any means, but it forces tons of good programming practice on you which can carry over to other languages and frameworks.
The way I use it is by not logging in at all. I've set it to email me on events, and I click the links in the mails. Other than occasionally looking at event pictures, I don't even look at Facebook pages.
New iPOD classic for $249 with 120GB
I'll pick up the stable release from the last announcement.

I have bought five iPods and two Macs over the last five years. One of the Macs was new, but the rest were used or refurbished models that were at least a year old at the time of purchase.

It's a good way to save a little money and to watch the issues shake out. And Apple's refurb store is great.

Which is not to say I won't load up some Apple sites later today to find out what I'll be shopping for next year. :)
Apple announces new iTunes 8.0 with HDTV Shows
I think you didn't read this:

"Hmmmm - over the next four years of my life, I can make $600k a year plus bonuses/benefits working for ACME Corp or I can make $100k a year with the chance to take home a $3m-$5m payoff..."
It's true that utilities like grep, sort, and join are highly optimized. Many people have had wins replacing scripting logic with calls to these utilities. But I'm having some difficulty understanding the merits of putting the top level logic in C, using shell scripts, and the alleged advantages of flat text files.

Shell scripts are hard to make robust. I don't understand why you wouldn't just drive the unix utilities from perl.

Why use perl threads on unix? Everyone know they suck. Why not fork and use a transactional database for IPC?

I remain unconvinced of the alleged wins people have with flat file solutions. It's usually about replacing a toy like mysql or some convoluted berkelydb mess. You can use db2 for free up to 2 gigs of memory. Why waste even a minute replicating transactional RDBMS functionality by hand? As soon as you're dealing with flock and company and you COULD be using a DB, you should, as far as I'm concerned.
Congratulations. That is an impressive speed up.

I agree about the built-in Unix utils. You just do not see people taking advantage of these powerful and extremely optimized programs any more. I wonder how many times grep or sort have been unwittingly rewritten in Perl or Ruby because the programmer lacked familiarity with basic Unix tools?

As for your crawler, I think the significant thing here is that you rewrote something in C after you already had it working in another language. Not to bag on C, but writing the original in a higher level language first gives you a better shot at correcting any bugs in the actual solution domain. Then if you move to C you're only fighting against C, not against C and bugs in your solution at the same time.
From pictures and hearing about him I thought he would be more of a jock, but he was more of a ultimate geek.
Most startups that have scaling problems have them due to number of customers, in which case it's trivial to either monetize or raise some money.
This also seems to be true with Lua. I'm relatively new to the language, but in many ways it feels like a minimalistic Python, intended to be embedded in a C program as a library (though it's usable on its own as well).

Most of the language is closely tied to "tables", which are very similar to Python's dictionaries. Unlike Python, however, it has real closures and tail call optimization.
Okay, so it has a cost, and you're encouraged to ignore that cost. I don't think this is a good thing.
or you can live in california, where all these kinds of agreement are more likely to be un-enforcable. Maybe that's the reason that so many startups are here?
Thanks for the plug!

As a developer of Heritrix, I can't honestly say it's compact or Python, but it is well-behaved, highly customizable (both by settings and by many Java extension points), and capable of high-volume crawling for many purposes.

You could also embed Python code via Jython with a little work, if necessary.
What I do is this: model the part of the project you want with uml and write the function names. You solve the bulk of the design issues and all the coder has to do is fill in the functions. The skill lies in writing function names and descriptions that pretty much guarantee that he can hardly do wrong.
That's true. And I may know something about the stock that the market doesn't. It's pretty impractical to make your decisions by finding out that everyone else is wrong, rather than by ensuring that you're right (or as right as you can be). It doesn't scale, either -- AT&T used to have over a million shareholders, and interviewing every one of them about whether Ma Bell was going to raise their dividend next quarter sounds like a real chore.

Even if you're right, the market can stay irrational longer than you can stay solvent.

If you're leveraged, yes. If you aren't leveraged, you're going to be solvent forever. And anyway, arguing about the timing obscures the real decision -- I wouldn't discourage someone from pursuing a career because I didn't know whether or not every employer would recognize their talent after the first interview.
Download 30 Rock Episodes
30 Rock TV Show, a dramedy which moves around a comic who is stars in a late night comedy show called named ‘The Girlie Show’. 30 Rock is an Emmy award winning comedy series. Liz Lemon is the head writer who works hard to keep her job. She also wants to keep her sanity as she deals with the network and the stars.
Wonderful, but -- doesn't this violate the 'no runtimes/interpreters' Apple rule?
New multi-page HTTP compression proposal from Google
Yes, I love it when things can be speed up with old school techniques. Although I'm not sure C was a really great choice in case you need to upscale later.
A new iPod Touch is coming out for the holidays, it's thinner than the previous model
It has the same display, but integrated volume controls, which was the #1 request Apple got for an iPod Touch, Steve says. It has a built-in speaker, "it's for casual listening," he says, and has the Genius playlist creation and the App Store.
Download Aliens in America Episodes
You can download each and every episode of Aliens in America. Become a member and download anything you want. And server is amazingly fast. Downloading will take just a few minutes only.
Link (of a link) to the PDF: http://sdch.googlegroups.com/web/Shared_Dictionary_Compressi...
This article can be summed up by point 1 of George Leonard's keys to Mastery, or Getting Good at Anything. All 5 major points are as follows:

1. Find good instruction. (Read good code, find a good programmer)
2. Love to practice. (Write code.)
3. Have the beginner's mindset. (Don't get cocky, observe with new eyes.)
4. Have a vision / goal. (You want to be an architect? Graphics guru? AI master?)
5. Push the boundaries. (Code beyond your perceived "abilities".)

A more succint and general version of that quote is by TS Elliot: Talent copies; genius steals!

Anyway good read!
An interesting book on the same subject is Dan Ariely's Predictably Irrational.
I have done this a couple of times. None have been success so for. Some will never be, and some are still in the works.

http://www.tastesatlkr.com
* I wrote by myself, finsished it mostly and then it fizzeled. No intrest.

http://www.qwertykitchen.com
* Equal partnership, still working on it and it works. We just have to figure out how to bring in traffic.

http://www.loudfarm.com
* Not done yet.
* Minority equity holder.
* so far I feel has bee my most succsefull code wise. The most limited in scope, it is going to be bay area only.
* Feels like the best chance at working.

Things I have learned.
* The only projects that stuck, were ones that I had fun doing.
* The wife can be helpful, but respect her time.
* Don't spend money unless you absolutely have too.

Take away, only do something extra curricular if its fun, and you are learning something new. Otherwise you should invest your time in the company. Who knows you might be able to innovate there.
Absolutely. There's no such thing as a free lunch.
Or, briefly: Certain classes of Python programs can actually be faster than average C, because they consist of small amounts of Python code connecting libraries that are already written in very thoroughly tuned C.
Python has no advantage in this regard over any other language supported by SWIG. And perl almost certainly has more available library bindings than python.
I understand but the counter point is, how big is your ego? to think that the whole world would follow?
Ruby on Rails migrations version control issues
Sequential message logging from multiple threads to the same file
I had to read this twice before I realized it wasn't a joke.
In that case, I think I'm going to be the next Bill Gates.
The Honda Fit is tiny, underpowered and sells just fine. I think the article is correct on why it won't sell; size and power don't mean as much as they used to.
PC gaming is not on the way out any more than the cinema is on the way out.
Key to innovation: no funds, no Internet, a laptop and free time.
Isn't this something that goes without saying? I'd be surprised if anyone outside of Hacker News has even heard of TechCrunch, and the only reason people have heard of it here is because the site is spammed daily with their lame articles.

I think the average reader here deserves a little more credit than the original poster has given by submitting an article advising against such an obviously stupid strategy. How on earth did this make the front page?
I don't know if I speak for everyone, but can you stop spamming the front page every ten minutes with updates from the press release that we can all get from Gizmodo or Engadget?

I'm sure people will discuss that major points while it's over.
Why use threads when you can use an event-based IO system? (I think Twisted is the Python way of doing this.)
Congrats!

If you haven't yet, check out _The Unix Programming Environment_ and _The Practice of Programming_, both by Rob Pike and Brian Kernighan (K of K&R). They're concise, highly informative books about using the Unix toolset to their maximum potential. The former was written back when computers were slow and had little memory, the latter in from 1999 but very much in the same spirit. (It seems to include a lot of insights from developing Plan 9.)

Also, a dissenting opinion here: C's performance vs. higher level languages' development speed is not necessarily an either/or choice. Some languages (OCaml, the Chicken Scheme compiler, implementations of Common Lisp with type annotations or inference for optimizing, Haskell (under certain conditions...), others) can perform very favorably compared to C, but tend to be much, much easier to maintain and debug.

As a generalization, languages that let you pin down types are faster because they only need to determine casts once, at compile time, but if those decisions can be postponed until your program is already mostly worked out (or better still, automatically inferred and checked for internal consistency), you can keep the overall design flexible while you're experimenting with it. Win / win.

Also (as I note in a comment below), Python can perform quite well when the program is small amounts of Python tying together calls to its standard library, much of which (e.g. string processing) is written in heavily optimized C.

Alternately, you could embed Lua in your C and write the parts that don't need to be tuned (or the first draft of everything) in that.
But python is a pleasure to code. Also, I prefer Boost, and know the python bindings are solid.
Lua is semantically more similar to Scheme, according to the language designers. The resemblance is quite intentional. Ease of embedding into C is a first-priority of the language implementation. It's almost trivial to interface to C, and in many cases it's easier to write bindings to the Lua API than to massage SWIG generated interfaces.

"userdata" are first-class values in Lua. They are containers for pointers to arbitrary C structures, and by overloading their behavior it's extremely easy to interface between C and Lua.

Lua tables are extremely elegant, and are the basis of object and class systems, all data structures, namespaces, packaging, and libraries.

The Lua interpreter is quite fast by itself, and with the LuaJIT written by Mike Pall your speeds can be even greater. The rumor is that the first public release of LuaJIT 2.0 is coming this winter... that one is a tracing compiler based on the same concepts as Tamarin, SpiderMonkey, and V8.
Or on Twitter.
kingkongrevenge - I didn't mean to sound like I replaced all of my DB functionality with flat files. I have several stages that my data flows through (fetching, ripping, sanitizing, indexing, uploading) and I replaced the constant calls to mysql by writing things to a flat file, munging the data on-disk with the gnu tools before they get written to the database. With large datasets, DB calls (even with good indexing) can get expensive, so I tried to avoid them as much as possible for the crawling stage of the process. I certainly need a database for the other stages of the process.

A simplified example: If I crawl a website, the chances are good that I'll pull several thousand copies of the same url from the site over and over again. I want to insert all URLs that I find back into the database so I can can crawl them, but I don't want any dupes. I just want to crawl the URL once and then move on. If I insert a URL into a DB and let the DB check if it's a dupe, then I waste DB time/IO. My solution was to crawl thousands of pages, dump all of the urls into a text file, run the text file through 'sort | uniq' and then dump those URLs into the DB and let the DB ignore the dupes at that level. I still have to use a DB to do some of the work, but pre-processing the data up-front using 'sort | uniq' is mega-speedier in my special case.

Also, I did wrap most of the scripts with perl to talk to mysql and do other sanity-checking that's easier in perl than bash or tcsh. :)
Hmm, anyone want to do a 3d printing startup? email me.
ctypes, cython, Boost::python and pyrex are real advantages in this regard; SWIG is extremely painful in comparison.
> The problem would be the GIL (Global Interpreter Lock). You couldn't actually have more than 1 thread running at a time.

Not actually true; socket calls release the GIL when they're not active, as do many of python's C libraries. Especially the better-written standard ones.
I was just going off of what he had already done. I agree that async is a better way to do this.

Twisted is one way, but Python ships with asynchronous libraries http://docs.python.org/lib/module-asyncore.html.
all news headlines here are part of the press releases anyway - also there are HN readers who only read news here so its important for them to know the latest...
If it's cross-platform interfaces to WebKit that you're after, both Gtk+ and Qt offer widgets that integrate WebKit into their applications. However, I have little experience with either method of using WebKit, so I can't speak to how easily it would be to plug in a Javascript engine, or some other such manner of working directly with the WebKit DOM.
Steve Jobs: New Apple Nano iPods to be Greener
True, although I would argue that my statement is still correct. You cannot have more than 1 thread running simultaneously.

The GIL is not as big of a problem as most people see it, it only interferes with CPU bound, highly parallelizable tasks.
I enjoyed reading your comment. I am just about to start working part-time so that I can spend time on my own projects and so have found this whole thread very interesting.

I have to disagree with your idea that friends and family make good QA. The problem is that they love you (or at least like you). I think you would get much better feedback from strangers who can be honest without caring whether their opinion hurts you.
Why Apple Doesn't Need The iPod
perl threading was (and I'm sure still is) absolutely horrid.

I'd be interested in seeing the results of a rewrite not to C, but to python or ruby where the threading support is much much better. Then you could rewrite functions at a time in C as needed, but not have the extra burden of rewriting the whole thing.

I totally agree with the rest of the approach. Going low tech and using unix tools is a very good way to reduce overhead, increase parallelism, and delay calculations. One of the nice things about this approach is you can cobble up another $50 unix box to do some of the bulk processing via nfs or other means.

Congrats... It sounds like a very interesting project.
Computer Chronicles: Macintosh (1985) - 30 min video
For good code? Really?
I'd like to change my user id. Is that possible?
First, I really don't want to start a flame war. That being said, my opinion is that PHP as a language is not well structured or well written. You might want to check out Python or Ruby, both of which have great web frameworks that encourage structure.
No. It means that I generally don't like him, but some of his ideas are good. Like Ron Paul, or Michael Bloomberg or Jason Whitlock. That's all it means - nothing more, nothing less.

And because he's polarizing, which he admits and actively tries to do. Also he's a glorified scammy sounding pill pusher who is a master of PR and good at articulating ideas. The pill advertisements for BodyQuick and BrainQuicken, which he owns, are reminiscent of weight loss or penis pills.

Any time you praise someone who's done a lot of good and bad I think it's essential to qualify the parts you like about them.

But I do understand your point and should have elaborated in my original post. In addition, to the autoresponder (btw, what's hilarious is emailing him, getting an autoresponse and then a reply from him a minute later), here's a list of some of the other things he does or actively promotes that I think qualify him as an "epic dbag" or rather take a lot away from his successes, ideas and good qualities:

1. Shady pills making ridiculous claims

2. This is his world record:
http://www.youtube.com/watch?v=H9pWKB2D23k

3. He "won" Wired's greatest self promoter of all time

4. Read about the Chinese kick boxing championship

5. Self claimed "Advisor to more than 30 world record holders in professional and Olympic sports (not substantiated)."

6. Self claimed "Cage fighter in Japan, vanquisher of four world champions (MMA)" and a "National Chinese kickboxing champion" (not substantiated)."

7. Self claimed: "Actor on a hit TV series in mainland China and Hong Kong (not substantiated)"

8. Plagiarism? http://www.johnwinsor.com/my_weblog/2008/04/was-another-new....

Ivygate (yeah, I know) has a pretty hilarious write up as well:
http://www.ivygateblog.com/2007/05/timothy-ferriss-out-vayne...

http://is.gd/2pjW
One more reason why Yahoo SITG Tyroo SUCKS
Be careful just assuming they are unenforceable. Non-compete agreements are, but if your company can show that they have any "actual or anticipated" research, development or plans in this area things are up in the air.
It's the same problem in any language. Most of the code out there is mediocre or bad. The code you want to learn from is a few standard deviations to the right. When you are learning something always Remember Sturgeon's Law!

I would rather work with someone who produced excellent PHP than someone who wrote lots of weird hybrid procedural/functional Python full of esoteric array slicing hacks.

Being able to apply your intellect to revealing intention in code is a valuable skill in any language, and I suspect that it transcends any particular programming language.
Apple fans disappointed by the announcements 
No thrilling product launch, only more colored nano :-(
I wish I could find the link to the quote by him, but when he was at Paypal they actively tried to not hire jocks.
If you enjoyed learning/using sed, awk, grep and other Unix text processing utils, you'll love this:
http://borel.slu.edu/obair/ufp.pdf

Its called Unix For Poets and it'll show you just how far you can really push these tools.
Right, which is why I said "mature company" (who is hopefully, by then, issueing dividends :-D). At the same time, if a company issues dividends it can't afford to pay (cough Citi cough) this can be devastating to a Company's prospects.

At it's core, the whole reason stocks are worth anything to begin with is the anticipation of future dividends (DCF).
I don't know about hiring other people to work on it... good people will cost you and bad people will cost you even more. However, there is another alternative if you can split your design up into infrastructure and core technology; open source the infrastructure.

Obviously, the first step is to use as much existing open source code as possible, but a lot of companies will also need to do some custom infrastructure on top of that. These extensions can be a distraction, so if you can leverage the open source community to help develop/maintain them you will eliminate a huge time sink.
Why IT should get into the facilities business
Probably not. It's a walkthrough of how the Linux kernel is implemented, both in terms of design ideas and the code itself. They assume you already know operating systems basics.

I could be wrong, as I've also never read it cover-to-cover; I always dive into it when I specific questions about how things work. I've never read it from the perspective of how well it teaches OS concepts. But if you're interested, and you think you will remain interested in the future, then there's no harm in getting the book now. As you learn more, you'll be able to understand more.
Yes. It seems to have many of the same abilities as Scheme (and I forgot to mention it has coroutines.), though it doesn't have read / the sexp syntax or macros. It's extraordinarily portable, though, and also seems to also contain most of what I like about Python (though it lacks its remarkable libary).

I read The Implementation of Lua 5.0 (www.tecgraf.puc-rio.br/~lhf/ftp/doc/jucs05.pdf, 15ish page pdf) last week, experimented some with _Programming in Lua_ 2nd ed. since, and I've been very impressed with how tasteful the overall language design seems. It seems like a brilliant tiny language, much like Forth and its descendants. (I've read that the Lua source is good reading, too. See: http://www.reddit.com/comments/63hth/ask_reddit_which_oss_co...)

The Lua Emacs mode is nothing special, but I'll see if I can improve it. :)
Jumpin' Jehosaphat!
I've had that on my mind for a long time (ever since I started following RepRap), so I'm seriously interested. Can you email me, since your email is not in your profile?
Was the same for me, works now.
Electronics components choice for semiconductors applications
Semiconductor applications list: choose the right electronics components for your device!