skysight-inc/HackerNews huggingface dataset:
Gosh, I wish I got that kind of service from Bell Canada. It took me two weeks of calling them at least once per day before they sent a tech.
Other people profiting is a bad thing?

Actually picking up change increases inflation.
I actually tend to agree; Ruby's equivalent to Python's "Twisted" is EventMachine --- a respectable event loop library --- and there is a pernicious meme there that one of the things we should be doing with the event loop is yet-another-nonpreemptive-thread-scheduler.

There's nothing wrong with an app stack that is async up through the database client. Async is good. There are things that are wrong with app stacks that are entirely threaded, but we can have that argument somewhere else. But there is something wrong with the idea that we can have the best of both worlds, if we just try to make async code look like threaded code.
this is the first time google has made me doubt it's might
you need a believer in the ranks to champion that cause

Nah, you need an entire project that is demonstrably better executed than another and that attributes its success to unit testing (or whatever). This is where evangelists fall down. They shouldn't preach until they've gotten something tangible to show.
How makers work
"Temporary Error (502)

We’re sorry, but your Gmail account is currently experiencing errors. You won’t be able to use your account while these errors last, but don’t worry, your account data and messages are safe. Our engineers are working to resolve this issue.

Please try accessing your account again in a few minutes."

Wow! This is a first for me...
It's up for me, but I can't send any messages
wow. Uh, first, a 300W server is moderately heavy. I run core2quad boxes w/8G ram and they run about 120W. I soppose if you had dual quad core xeons, 300W isn't unheard of, but that's a beefy server. power(and cooling) in a datacenter in the bay area costs me around $400/month for 2000W (1500W usable if you are smart) so that is $400/month or $4800/year for 5 300W servers, or $960/year/box (for a massive 300W box.)

so I'm not seeing where he's getting $8000... Well, I guess it's more expensive to build a datacenter than to rent, I guess? then why not rent?
Looking for Start-Ups to Join Fantasy Football League; Prize is Blog Coverage for Company
If you have an iGoogle page with Gmail you can still get to your messages.
Yep, me too. I was just trying to send a bug report to one of my client's hosting providers about something they screwed up costing me an entire evening ... and now I can't even send the bug report!
How ironic :D
Is Customer Service The Business Model?
and here i thought it would flop. The e-paper style display is probably what makes the difference, it isn't annoying to read like traditional backlit LCD's.
If you're trying not to spawn threads at all, blocking database connections are also a dealbreaker.
Tom, you're not making any sense here. You stated that "threads weren't cheap". I said this really wasn't true anymore for this problem area (mostly-blocking network I/O) and mentioned an application I'd written that shows exactly this over a load regime very close to what you'd see with a heavily-contested web application database.

Now you're talking about stuff like synchronization overhead, which wasn't at issue: synchronous access to a few hundred separate file descriptors (one per thread) obviously doesn't need to synchronize anything. You're stating some stuff of questionable veracity (select/poll certainly do scale poorly once you get past a few hundred descriptors -- just check the kqueue/epoll justification documents for copious benchmarks to that effect).

And you're even making up some, er, interesting new terms: what on earth is an "I/O scheduler" as distinct from a "context scheduler"? Usually when you use the former term, you're talking about the block device request scheduler (elevator algorithm, etc...) which (1) isn't involved here as we're talking about balancing network I/O and (2) is invoked in the same way for local I/O regardless of whether you're doing I/O via an async request for 700 blocks or via 700 synchronous read() calls from separate threads.

And you capped it all off with a few ad hominems that I honestly don't think are appropriate on this site, at least in a technical context. Stop flaming.

I'll say it once more: threads are cheap in this regime. The linked article is a hack to get around the problems of database access from monolithic, poorly threaded language interpreters. It's not a "performance enhancement" in any meaningful way. Even a shared-nothing web app architecture a-la news.arc is likely to do better than an extra hop through this thing.
I meant - they do not even mention that one could be writing a compiler for a language that includes higher order functions.

By the way I found the book 'Modern Compiler Design' (http://www.cs.vu.nl/~dick/MCD.html) much more helpful.
I think you misunderstand me.

I don't think the article is great. But it -does- accomplish exactly what the author wants. People talk about it. It reaches the top of HN. And despite of our distaste of The Register... we still read it.

So when I say it's "Good enough" it just means it gets the job done. The purpose of the article is clearly attracting attention. Being ignored worse than being disliked, after all. And it does attract a lot of (negative) attention. The poor writing even contributes to the goal.
Just keep in mind that via the GIL (Global Interpreter Lock) Python limits the system to executing one bytecode instruction at a time - Threads execute in a more serial fashion, switching context every 100 bytecode instructions or so (e.g. let Thread 1 have 100 bytecodes, then switch to Thread 2).

This is a good thing and a bad thing - internally it keeps some non-threadsafe components from asploding but can be confusing and frustrating if you're looking for true multithreading.

There are some implementations of Python that don't use a GIL, such as stackless, if you're looking for true multithreading.
I have Google Apps and it is down too. One of the reasons I pay $50/year is for uptime....
Read it for entertainment purposes only and it becomes... better.

Along with the data processing framework, Doug Cutting also included a fault tolerant, replicated, distributed file system with Hadoop just because fuck you. -- I laughed!

Also, it makes fun of Ruby so another point for it.

It's unlikely this article is accurate or really serves any purpose whatsoever other than immature linkbaity sort of content, but that's fine with me because I never put it in the category of "real journalism."
That is awesome, this is the first time I have really used Twitter.
Encyclopedia Dramatica threatens shutdown
I was just thinking to myself, surely Gmail isn't down. Then this popped into my GoogleReader. Good thing they're not running all their services from the same box. :)
http://search.twitter.com/search?q=gmail - leave it open and watch the refresh count. Pretty interesting.
If you can be specific about how what I said was uncivil, I will apologize for offending you. I think you're wrong, but I don't think you're crazy or stupid.

You're right that we started talking about something very specific (the memory costs of stacks for 700 threads) and I quickly generalized (to the performance of threading versus async code). That's a fair critique. In my defense, the performance difference between threaded code and async code is very relevant to this article.

Here are my points:

* It is not a "myth", as you say, that async network code scales better than most threaded network code.

* Your anecdote about 700 concurrently served connections probably won't cause select(2) to break a sweat, let alone kqueue/epoll.

* It's horribly unfair to tar async code with select(2)'s performance, because performant applications use kqueue or epoll to replace it. Both resolve the scaling problem you're alluding to.

* It is perhaps weird that I see a similarity between a thread scheduler, which switches CPU contexts on a timer, and a select loop, which switches them nonpreemptively on I/O events. I retain the right to say that event loops are an instance of the "scheduler" problem; if you think that's crazy, read the papers on the MIT Click modular router. In fact, do that anyways; they're great.
Yeah I have been booted and had to log back in and such but it must be down pretty hard for the moment.

Sucks because I was in the middle of sending a message and don't know if it went out or was even saved as a draft... I really should always write my email in emacs.
It'd be awesome if the surge in tweets complaining about gmail going down, in itself caused twitter to go down... Domino rally anyone?
Top 50 Marketing Blogs To Watch In 2008
AH... back online!
Design matters because I need to be able to interpret the search engines results as quickly as possible. Google's lightweight design really shines because it loads instantly and I can usually look through the result page very quickly and figure out which link is the most relevant to my query and click on it.

However, that's not always true, and I sometimes have to click on 3 or 4 links to figure out if I need to refine my search or not. If someone found a better way to organize the search results, so that I can determine which ones are relevant faster that would be a big win.

I do agree with you that the quality of the search results are a lot more important though, but presentation definitely does matter.
They guarantee 99.9%, so they get almost 9 hours a year to screw up. ;)
Same. I won't buy one until the price is lower. I know you can get a lot of free content but the current price + price of buying ebooks is too much
Try the html view, it worked for me:

https://mail.google.com/mail/h/
Use HTML version for now:
https://mail.google.com/mail/h/
http://www.techcrunch.com/2008/08/11/systemwide-gmail-outage...
I am wondering, what caused this ?
That just got more and more arrogant as it went along.

I could have easily rewritten the whole thing in assembly language in another 4-6 months if really encouraged.

Well, why didn't you do it on the side, quit, and sell it yourself?
I'm still down and have been for at least the last 20 minutes.
As an agency programmer, I'll say this: if you're a startup, don't hire an agency unless your website is simply a marketing vehicle for your (non-web-based) product. An agency can give you top-notch graphic design, etc, but are much too slow to service a public-facing web startup. This is due to the project management process, and the fact that agencies have 30 other clients besides you. The article is largely right-on.
Both my accounts are down, too. My whole afternoon's agenda today was email. Guess I'll compose some messages offline...
https://mail.google.com/mail/h/ works
Systemwide GMail Outage
Meanwhile, the productivity of 20 million people worldwide just spiked upward.

This is taken straight out of Onion News.
All you can do is read, not send :\
We know :)
http://news.ycombinator.com/item?id=273338
I just sent one.
IMAP seems to be working fine, just the web interface is out.
Was talking to a coworker about the outage over GTalk and it suddenly went out as well, Pidgin is reporting 500 errors when trying to send messages.

Not a good sign.
The real surprise of the App Store isn’t number of downloads or revenue
I gathered that much just from the title. If you trained a Bayesian filter to recognize trolling, "grownups" would probably be one of the strongest troll tokens. adequacy.org even subtitled itself "news for grownups".
Hackerfeed for News.YC
http://ycfeeds.com/hackerfeed

Somewhat similar to friendfeed for News.YC. Its going to be slow because I need you guys to warm up the cache for me.

The AJAX features ("more", "replies") are currently disabled but should be working by this time tomorrow, also previous and next pages are in the works. I'm off to YC meetup in Cambridge, enjoy and don't break it.
Businesses approaching Green IT cautiously
2006 U.S. Census figures show 20.7 million non-payroll businesses
not for me.
I, on the other hand, have read other articles from The Register. This is par for the course.
I knew I should have stayed on my premier edition..
I googled the error and came up with this:

https://mail.google.com/support/bin/static.py?page=troublesh...

Heh.
totally agree... it is all-to-easy to convince yourself that BPMF is a phase you pass through - rather than a phase you have to obsessively monitor in order to work you way out from.
First time I've seen Gandhi quoted in support of pg. Really good reading summary.

August is always a good month for personal productivity, as so many other people are away on their holidays.
A first ever look inside the Defcon Network Operations Center
I got headache trying to read the content on this web site.
GFAIL, by Google.
Color schemes I can take or leave, but what really bugs me about my Kindle is the fact that both of the bottom corners are sharp. I have yet to be able to come up with a comfortable way of holding it that doesn't result in the corners digging into my hands. This seems like a pretty obvious mistake to me, so hopefully they will correct it.
They brought in no cash in 1999. (correct me if I'm wrong)
How's it a success?
OpenSolaris: Does it look familiar?
try: https://mail.google.com/mail/h/

Note the trailing slash!
Former creative agency social network programmer here.

Seems to be working out alright for former client Education.com :
http://venturebeat.com/2008/08/05/educationcom-scores-975-mi...

The product has changed quite a bit since we launched (design tweaks and additional features)

Also, they acquired one of the agency's ops guys post launch.

Also worth noting that Education.com has some phenomenal content curators.
Gestures of Pride and Shame Are Universal
it's back
And this addresses the subject how again?

Oh, I see. More personal style critique.

Still looking for substance, huh?
Hi,

I've setup a hadoop cluster for a start-up I am working in. Initial reason? The amount of incoming log data was getting too slow to process on a single machine and having had a single graduate course in distributed systems I knew a) what is required to go from a single script to a system distributed across nodes b) that I am not in the capacity (alone) to create this system in a time window that will meet business needs.

As a result, I saw hadoop as offering a solution to: implement a distribute process where by records are processes in parallel by multiple machines without having to worry about implementing locking and message passing; distribute disk seek times across a cluster of nodes, pool together disk space with ability to add disk space by simply adding new commodity machines to a cluster.

If you have these specific needs:
a) need to rapidly and robustly process more data than a single machine can handle (think of it this way -- if you do data processing with a Perl script on a single machine, will the data be simply too old by the time the Perl script is done?)
b) ability to add storage for new data without powering machines down
c) the data is non-relational (i.e. you aren't exporting data from MySQL and then re-importing it again - if your data can be stored effectively inside a MySQL cluster and queried in a relational manner, you don't need to use hadoop).

Now, if you still think you need hadoop (in most of the cases, answer is "you don't"):

To get hadoop going minimally to be useful right away is actually almost trivial.

However, as your dataset grows, the amount of jobs and the size of the cluster grow as well as the expectation of hadoop to also be a reliable data store grow, you will run into plateaus that you'll need to overcome.

To answer your questions directly:
- Getting it going is easy, trivial. Do it first on your desktop (if you run Linux directly, if not under VMWare). Process is straight forward.
- Peeking into the source is not required to get it going
- Pitfalls:
* Use enough memory, JVM is a memory hog. Always make sure you have sufficient swap (4x your memory)
* Find the optimal size of input data for your task (if you are using compressed files for input).

(Yes, I will write these learnings up eventually).
Yeah, it uses Gnome.
ah thank god. I've been waiting for an important (time sensitive) email, but have been unable to access my account for a while.

Thanks for the hint.
"... Somewhat similar to friendfeed for News.YC. Its going to be slow because I need you guys to warm up the cache for me ..."

Works fine for me, no delays. I'm still impressed with the personal RSS feed. With comments this makes it even better. Well done.
I caused it
Perhaps we will now see a slew of articles complaining about Google's inability to scale.

Isn't Gmail just like an IM system? What's so difficult about that?

;)
NewsCred is in.
never mind, IMAP is gone too.
...and in response, 300+ bloggers would whip an already dead horse about Twitter and their uptime.
I've definitely had this happen to me before on Gmail, this is the first time when it also seems to be happening to everyone.
Not gonna down mod you coz you've given thoughts to it.

Yet, your argument is inline with what the article says. It took Google 10 years for them to become a HUGE success. During this 10 years they probably have worked really hard everyday, and they're all brilliant people. For less talented people it probably takes even longer.

I guess your point is if you're just gonna do a declicious-scale success it doesn't take that long. YouTube even just took a year. But look at how much experience they have under their belt before that. It really depends on where you start counting that 10 years.

The article doesn't specify either, though. It's kind of vague.
So many people are focusing on the "10 years" thing. All he's actually saying is it's gonna take longer than you think it will.
I'm not sure what's worse, the fact that this is a stupid idea - or the background on http://gwei.org/pages/google/googleshare.php
A startup strategy - where I almost went wrong
egads! 728 in the last minute...
"your Gmail account is currently experiencing errors"

Is it just me or is this error message a bit annoying b/c it casts blame on my account, as if it's something I did.

how about "Gmail is having a problem".
Which agency built Education.com?
> The most interesting part of this is the explosive response on Twitter.

That's the definition of irony. ;)
Don't click through if you have epilepsy, or if you don't.
Nintendo Asks Miyamoto to Kindly Shut Up
Now if only I could also send that email I really need to send ...
Meanwhile, the productivity of 20 million people worldwide just spiked upward.

...Until it hit HN, where it seems to have displaced the stuff that actually makes HN a better place than those other link aggregators. Now our productivity has taken a nose dive...
hotmail users must be laughing at this.
Gmail is back already.
Will moving the BBC R&D centre damage innovation?
@ 3:30PM PDT

Looks like it's back up. There was a similar problem that affected my Gmail account last week. But it only occurred when viewing specific (older) emails. This problem seemed to affect everything.

It's disconcerting that it went down twice in the last week and with very little communication from Google as to what's happening.
Who owns your online networking contacts?
Well it IS in beta :)
Loader Crane | Lorry Crane
Loader Crane, Lorry Crane are HMF products which have the reputation of being one of the most durable and reliable brands in the market. Our aim is to make sure that our after sales service has an equally high regard. We understand that our customers have deadlines and busy schedules, and we provide a unique service that helps them fulfil their commitments.
Facts are Better than Dreams
I'm semi new to unit testing, but I really like it so far.

I'm not sure if language choice effects need to unit test. I think that good unit tests test a lot more than type errors and although they are related to design patterns, in that design patterns allow easily unit testable code, and having unit tests in place allows easy refactoring.. unit tests would be helpful even if no standard design patterns were in use.

One main barrier to Unit testing that I have personally seen is that a lot of my work is GUI based or simple database CRUD applications. I haven't found a good useful way to unit test those.
I've written three billing systems. None had any common code, and in retrospect it's doubtful that it would have made much sense. The business structure just had too big of an impact on the design. If all the businesses you're trying to serve have a similar business model (e.g. shipping physical stuff, or digital subscriptions, or selling software downloads) then you might get some leverage, but as soon as you throw in anything weird (refunds, credit card holds, recurring billing, staged billing, payment by something other than a credit card, etc. etc. etc.) it can get very, very complicated. Which is probably why there's not a one-size-fits-all solution out there.
>the fact that this is a stupid idea

It's a joke.
If necessity is the mother of internet inventions, who's the father?
seems to be back up now.....
Very interesting spatial thinking game - could be a winner on the iphone or Wii.
once you have paying customers it's hard to say those things :-)
If your customers have to download a big new framework before trying out your app, that is one more hurdle you've got to get past in order to sell. Customers who are not computer sophisticated may be scared away and others will consider it too much hassle (frex, if they are on dial-up).
Obama to Announce VP By SMS, Email
A Modern-Day Blackout: Gmail Goes Dark
They also seem to have ported a number of the major apps like gimp. I got it to play around with ZFS & this kinda put me off.
This is neat, not something I would actually use, but neat. Well designed and easy to use. I went to the page and instantaneously understood what it was for. Which, in my mind, is the sign of a well designed and good application. Nice work.
> It's a joke.

Worse, I think it's art.
Yeah, told someone at work about it and he told me I should switch back to hotmail ;-)
Likewise.

Their version of Active Record isn't the same as those from other frameworks, but we use the built-in one from Code Igniter. There's another floating around that's closer to the RoR style.
Why would this put you off? You get an open environment with access to things like ZFS and DTrace.
Apple can make kindle out of the Kindle with a free (or cheap) PDF/eBook reader for iPhone/iPod touch. Millions of people already have the devices, many millions more already have thousands of PDFs on their hard drives and downloading them can be as easy as using iTunes.
Primitive. Not real threads, more like co-routines. This will be a real limitation for python, especially when multi-core CPUs are the norm.
This ties in beautifully with Jeff Hawkins's theories in his book On Intelligence.

Interesting side note re: what other species blows itself up? Heard an interview with Carl Zimmer, author of "The Oracle in the Gut: E. Coli and the Meaning of Life"[http://zocalola.org/radio.html] about E. Coli. which seem to do exactly that: suicide, releasing toxins [which their clonemates can produce the antitoxins for] to kill off competing strains!
"The particular node of our server farm that your account is hosted on is currently experiencing errors" sounds a little too technical to me. Also, "GMail is having a problem" might lead them to believe that, say, if they send email from their hotmail account to somebody else's gmail, they wouldn't get it.
Nokia 5610 XpressMusic
The music slider 5610 XpressMusic falls in to the middle class music phones of the manufacturer, combining a youth design and a 3.2-megapixel camera with flash and autofocus, FM Radio with RDS, the ability to play videos with “quality close to DVD” and a 3G connection.
Three Ways (Times Three) for Entrepreneurs to Blow It
Gmail is up
I survived the Google Mail black out.

08/11/08

NEVER FORGET.

(Anyone wanna make shirts?)