skysight-inc/HackerNews huggingface dataset:
Sure thing.

We have a customized sphinxsearch engine powering bug.gd. We can't say enough good things about it. Actually, out of all the open source stack we use, I think I'm most pleased with how robust and solid that piece is specifically. Sphinx is one of those powerful little open source gems that people don't discuss enough. It feels very feature rich and is wicked fast. I don't understand why it gets so little attention overall.

Its reindexing speed is especially good-- I highly recommend it for any service that needs their search index updated quickly. A reindex of 100,000 documents takes about a second-- just fast enough that we could actually do a full reindex on every search if we were that silly.

Supposedly you'll also find it behind craigslist's new search backend and (supposedly) the new whitehouse.gov.

The selfish inkling in me doesn't like talking about Sphinx because it feels like one of those "best kept secrets" that you'd just want to keep to enjoy for yourself-- of course that's silly.
The world is overpopulated? News to me. The way I see it, it's under-producing food, largely because of irrational resistance to modern techniques.
I'm glad we've finally advanced to the point where computers can accurately simulate the efficiency and usefulness of a messy desk.

I really can't see why this is needed. I thought we had entered the era of search.
I'd be more interested to hear about what portion of this spam gets through modern filters.

It's much higher to me on any email service but Gmail. In a typical week on Gmail, I get no spams to my inbox. A bad week is getting two or three, among hundreds of good messages. But I also get one or two false positives (good emails sent to spam folder) in a typical week on Gmail. Those are not high-priority messages, typically, but of course as a user I expect Gmail to be perfect in winnowing good messages from spam.

I've given up using any other email service, mostly because even one that I subscribed to years ago because of its boasted spam-filtering capability shows me much more spam than good messages in my inbox. (That account was my primary email account for a long time, but now I have everything from that account forwarded to Gmail, to add one more layer of filtering before I read the messages.)
Our service actually allows you to push your code into the system rather than trying to pull back all of the page contents. So, you end up running your semantic analysis, image analysis, or whatever you want to do on our grid. Very specifically, you implement a processPage() function of the following form:

byte[] processPage ( String url, byte[] pageContents, Object userData); (EDIT: remove code tag that didn't work...)

We run your function on the contents of the pages/images/objects you want to analyze and give you back your results from the millions or billions of pages you want to analyze.

The results from the processPage() function are completely free form. You serialize your results into a byte array and that's what you get back (except you get it back for all of your urls).

Now, since the processPage() function is free form, you can just turn around and "return pageContents;" from your function. That will give you all of the page contents from your crawl. That's not an ideal case for us, but we can handle it. We might eventually charge a small bandwidth or storage cost for this type of usage, but we do not intend to do so for our normal use case.

The bigger charge to the customer if they try to pull back all of the contents will be their local bandwidth charge. They would need to pull all of these pages' contents to their own servers. That will cost them quite a lot of bandwidth assuming they don't have their own fat pipe.

In summary, $2/million-pages-crawled is our real price and is not just marketing.
I expect to see a few great submissions from the Hacker News community.
Ask HN: Need some advice
I Recently released my photo and video sharing site vivapixel.com into the wild and so far things are going ok.
Traffic has been growing thanks to folks posting stuff on reddit,stumbleupon,digg etc.
I've had a few new members signup but not that many folks are using the site ?
I've contacted some folks to get some feedback on useability, speed etc. and it's all thumbs up !

So I guess I have a few questions for the HN community and was hoping to get some insight from others who have been thrue this before:

1. How do you folks attract new members ?

2. How often (if at all) do you interact with your members?

3. I recently tried to monetize my site with some ads. I regret doing this, it wasn't my idea, a friend recommended I try... it was a bad idea. But how
do you folks monetize your sites ?

4. Should I Keep Going? With all the other photo sharing sites like google,flickr etc. Is it worth it for me to keep working on this ? Maybe I should focus
my efforts on a niche? (now what niche do I focus on?)

I am at a crossroads and have somewhat run out of ideas and inspiration, any advice/tips/suggestions would be greatly welcomed.

thanks
A lot of people maybe haven't heard Taleb's illustration (spoken in a conference of which the video was posted to HN) about what he means by avoiding harm from black swans. He points out that if you live in Florida, you can't be sure when the next hurricane will come or how strong it will be. But you can build a hurricane-proof house, resistant to any strength of hurricane. That's what he means by avoiding harm from black swans in general: not that black swan events won't happen, but that prudent investors invest in a way such that when they happen, they don't cause irreparable damage.
From my point of view, Gmail is still in beta because its contact list functionality is still broken. I have to switch to "older version" to add new names to a contact list successfully, using Firefox, and bizarrely when I use Chrome, I can't display my contact lists at all in "new version" mode. There are still some JavaScript mysteries to the implementation of Gmail that no one at Google fully understands.
its in beta because there is absolutely nothing for Google to gain by taking it out of beta
$15 for 24h access to the article on the Science website... It's just not right, but since I'm interested I'll have to pony up. I guess outrageous pricing like that is supposed to drive people to the $99/year subscription.
I thought it was a joke that these pitchmen would get their own "reality" TV show, but apparently it's true. I can't stand any of their commercials and can't hit the mute button fast enough on my TV, so I can't watch a half-hour of Billy Mays shouting at me. The psychology of selling is interesting, but I'll have to read about it rather than listen to these obnoxious people.
It's really amazing (though I guess it shouldn't be surprising) that the saturation there is approaching 100%.
Neat site,

I'll give my answers based on my own experience.

1. How do you folks attract new members ? - The best way to attract new members is for it to be an integral part of your service.So friends can show friends, without that even being the goal. User's trust the whole WOM (word of mouth) over anything else, so to see that their friends are using something will be the most powerful aspect. This means, focus on the users you currently have!

2. How often (if at all) do you interact with your members? As much as possible. Don't fill their inbox with spam. But make sure they know the site isn't just a rotting corpse. Make sure they can easily provide feedback, and make sure there is a blog (didnt check if you had one) so you can post about new updates and announcements.

3. I recently tried to monetize my site with some ads. I regret doing this, it wasn't my idea, a friend recommended I try... it was a bad idea. But how do you folks monetize your sites ? - There are only a few ways to monetize a site. You can post ads, sell a product, or sell services (such as a membership). Your site, as you're clearly aware, isn't the first of its kind. Look at how the other's monetize and follow lead. No point in reinventing the wheel.

4. Should I Keep Going? With all the other photo sharing sites like google,flickr etc. Is it worth it for me to keep working on this ? Maybe I should focus my efforts on a niche? (now what niche do I focus on?) - Of course you should keep going, even if the site doesn't become what you'd like it to, it'll always be nice to have in your portfolio. And yea, a niche would be great. You should consider the following when deciding your niche: "What AM I passionate about?" - and obviously, "what niche needs this, that doesn't yet have it." It's important for you to be passionate about it, because you want to know the niche & market better than anyone else, so you can really dominate it. There is nothing more unfortunate than when someone has a good idea for a business for a market they know nothing about. This is where execution comes into play.

Hope my comments helped! Good luck!
Delta Airlines Introduces iPhone Friendly Boarding Pass
Some Joe Users now associate beta with a sort of brand. I once worked with a guy (who had no clue technically what he was doing) if I could take advantage of the "beta technology". When I asked him what that meant, he said, "you know, all the cool sites nowadays are using the beta!" When I pressed him further and had him sit in front of a browser and point to this beta technology, he finally just pointed out all the beta badges that the Web 2.0 sites were sporting.

So... maybe Gmail keeps the beta because now it's a brand association with being cutting edge and cool or something.
Nambu is by far my favorite here. Can't stand Adobe Air. Just leaving an Air app running will consume enough extra CPU to kick my fan on and drain my batter 2x as fast. Also the lack of OS chrome sucks.
Because they dont have 100 million registered users testing the Gmail app for them everyday YET!
Microsoft Dynamics CRM for iPhone Now Available
Virus Bulletin has started testing the efficacity of spam filters and will be publishing the results. In the initial tests the best product caught 96.24% with a false positive rate of 0.06% (although there are some caveats---see the article below).

Using the ratios 97% is spam/3% is ham the ratio of spam to ham in your inbox using the best solution would be about 1.2. So you should see more spam in your inbox than ham, but not by much.

So it's probably manageable to manually delete the spam in the inbox (if annoying).

But the flip side is interesting. What's the ratio between false positives in the spam folder and captured spam messages? Using those figures above it's 1 in 50,000. i.e. almost impossible to manually look for false positives.

http://www.virusbtn.com/vbspam/trialresults.xml
It's true that much of the population has no idea what "beta" used to mean. I had to explain it to someone the other day.

Perhaps we've passed the point where "beta" changes meanings in mainstream usage. Oh, well, at least it didn't go the way of the word "hacker".
The growth in 1999 started after GKO default in 2nd half of 1998. There was only way up from that point.
reference: http://arxiv.org/abs/0904.0489
Write your Google App Engine applications in Groovy
I don't understand people's obsessions with version numbers.

When I release a project, I start at 0.01, then release 0.02, and so on. The version numbers mean nothing to me, other than that they are increasing.

I still have people email me, though, and say things like, "I saw your project Foo::Bar, when do you think it will be 1.0?" I don't really know what to say...
Customers just do not care about the things that keep developers up at night. See also: version numbers, license text, whether it is compatible with <insert any technology this question is ever asked about>, etc.
Companies Aren’t Progressive (But People Are...)
Startup #119: Why Startup Innovation Kicks Corporate Booty
If you look at Figure 4 in the actual article, you see that persistence DOES pay off on YouTube. Although the probability of a particular video becoming insanely popular declines, the overall probability that at least one video from your collection becomes popular continues to rise as you keep uploading them.

If your first lottery ticket has a 0.01 chance of winning, but your next lottery tickets have a declining chance of winning, the overall chance of winning the lottery can continue to rise. In their example, it rises from 0.01 to approaching 0.20. Doesn't sound all that irrational, does it? Persistence increases your chance of becoming popular by a factor of 20X. It just takes many more than 20 uploads to accomplish that increase.

In fact, ... when you think about it, ... it could not turn out any other way. If the probability of your next video getting you in the "top 1%" on YouTube stayed fixed, ... then EVERYONE could be in the "top 1%". ... sounds like everyone wanting their kids to be "above average" in school. It's just not possible.

-- dxjones
Wow that is really strange. Its not a bug. The store owner changed their entire product line. BIG change. My assumption is that they will probably change their domain name too.

This is almost equivalent to when I got a letter from my high school saying Mr. Reese will be coming back as Mrs. Reese after the summer.
If there's an award for deftly weaved wit, you are taking home the prize this year!
Yes, we need more "techniques" such as those that lead to an infection of spinach with some bacteria that is later traced to a gut of a wild boar used as a fertilizer.
>The way I see it, it's under-producing food, largely because of irrational resistance to modern techniques.

While I do agree that the planet can sustain a much higher population, modern farming practices are causing problems and not necessarily sustainable in the long term. (http://en.wikipedia.org/wiki/Intensive_farming#Disadvantages)

Properly thought out food production that utilizes under-used resources (vast swaths in Russia for instance!) is the key.
I agree. I had assumed there wasn't a volume control until I read this.
>There isn't yet a solution for that.

There isn't yet a pleasant solution for that.

Unfortunately, segments of population have been reduced before.
It actually kinda went the opposite way...
Same thing here, if you do search on your website you should have a look a sphinx. Just because it's so insanely fast (even if you don't do full text search). Once you start using it you'll never look back ;).
> British have problem with gin anymore.

The British don't have a problem? I thought it was the consensus that British cities are descending into alcoholic loutish hell holes.

http://ajanlo.kapu.hu/pics.php?d=cardiff
Of course it's not an end-user sort of thing. The market is technical, which doesn't respond well to marketing and early adopters tend to resent it. So why in the world would they want a Jobs co-founder? I don't know but my best guess is that it's because Paul & co. don't really know what they're doing. There are situations where it works, but certainly not here. You wouldn't want one here -- just a single technical founder (or two).

In fact, all the great, innovative startups were a Woz or Woz/Woz combination. Ok, except for Apple, but again, Jobs really only added value later by playing VC. The seed level was a pure Woz play.
Ah, good one! The images don't correctly relocate when you zoom in, which might explain why I didn't notice the glaringly obvious dashed line...
What do you mean take it out of beta? The name of product is "GMail beta" ;-)
The problem with spotify is that its only supported on Windows and Mac, and the only way to run it in Linux is via Wine. This is miles better and it just embeds into the browser which is nice.
I'm Sick of This
Not so much obsession as convention. Generally versions 0.x - 1.0 are the alpha and beta releases, with 1.0 being the first "stable" release. Obviously there are hundreds of projects that follow a different schema, but it's used enough that it has stuck. One solution if you want to keep your versioning style would be to annotate them as stable or development, possibly with a short paragraph explaining that version 0.0.3 is in fact bugfree/feature-complete/ready for use in production/etc.
Any chance you have JavaScript disabled?
Because version numbers are an implicit signal of maturity. The convention is that a 1.0 software release is complete. It's understood that it's not the final release, and it may still contain bugs, but it's a common signal to indicate "This is the first version of my software I feel is complete enough that I'm comfortable releasing in the wild."

Some applications have their own numbering conventions to indicate what kind of an update a new version is. If you're familiar with the software, it's useful to know at a glance what to expect. (The Linux kernel and gcc do this.)

You can ignore this convention, but you're going to encounter confusion from potential users.
It seems like a mistake to directly equate persistence and quantity.
Well, slowness is part of my issue. But both flash and slowness are fundamental usability issues. Anyone ought to be able to visit a landing page and quickly see some indication of what the site is about - and at least be informed in the case that disabling of flash or of javascript is preventing proper functioning.
The type of person who consistently pulls off these sorts of tasks, is someone I call "the firefighter". Every good organization has one, the person you call when all hope is nearly lost. The one who always manages to salvage the failing project, to fix that bug before the demo, etc.

The article is a great "how to" on becoming "the firefighter" in any situation. In my experience the big one that people miss is "take an active part". So many people when they hit the wall don't put themselves into the middle of the problem, and just hope that everyone else will follow through in time.

Great article!
If you pay for premium you could also use despotify: http://despotify.se/
Triumphing Over Long Odds to Succeed at School
The interesting thing is that this is based on some of the standard Java EE APIs, unlike the Python edition, which was nearly 100% proprietary.

Unfortunately, there's still very poor support for HTTPS. The limited HTTPS support is what keeps me from deploying any applications on it.
The only thing for them to gain now by leaving beta is the huge tech press/blog coverage the status change will get. Which they don't really need anyway.
Weird. I fully expected to read the article and be able to figure out what the FBI was trying to do, and see their side of the story. But I'm more baffled (and frustrated) than ever.
Or if you substitute "True" for "False" in the "Authorized=" part of the link.
EMG ( http://escapemg.com/ , makers of grooveshark) has other nice services like http://tinysong.com/
an interesting discussion from both sides. i just wanted to thank mattobrien for remaining politce and on-topic.
Working with better programmers is very similar to working on great code. In fact, I would say that working with solid code is more important than working with great programmers.

I learnt a ton of stuff about programming when modifying the code of the linux kernel for an Operating Systems course. Fixing bugs in a huge code base requires inherent understanding of where things are in the code and how they are connected. And it's during that process that you gain an understanding of how to handle complex things : by making them simple.
I hadn't seen it that way.. "the firefighter".. Thanks for sharing that angle, it's a good way to put it. And you're right, every organisation needs one of those.
Right, but don't you think Amazon has more resources/experience in layout/design which gets people to buy things? Or are you looking to target a niche of people who prefer a minimalist design?
it's funny how people get so excited about 3d printing, but haven't ever made something out of paper-mache or other re-usable components. i'm helping a friend with a desktop cnc miller and 3d printer, so i'm definitely gung-ho for the ubiquity aspect. i just wonder whether we shouldn't also be addressing a basic component of creativity and reuse.
Going into it, I was expecting to read about girls and meaningful relationships.
And that it is "not available in your country"
The 200 million number is monthly active users. Being "active" to Facebook means interacting with Facebook at least once in the past 30 days. That means logging in to Facebook, logging in to a Connect site, using a Facebook mobile app, or using a desktop application with Facebook access.
Ask HN: Does system administration dumb down the brain?
Some people here, a while ago, have said that good hackers consider whole system. E.g., one must consider Apache, MySQL, operating system, instead of only concentrating on code.

My question is: isn't it dumbifying to install various packages, configure servers, configure linux, etc, and hence not something hackers would enjoy?
> Angry people are always wrong, and they’re rarely worth helping or cooperating with.

I did technical support for a company that sold really cheap PCs to a regional department store. I felt sorry for people who bought those cheap PCs instead of spending a little more for a name-brand like Dell. A fair percentage of these calls dealt with fairly angry people (e.g. father calling on behalf of daughter who lost their term paper because computer crashed - for real). I learned to always keep my voice calm. And my manner upbeat. It soon became habit. The customers weren't very technically savvy; I tried semi-subliminal stuff like "I for Intelligent" when spelling out DOS commands. I did manage to solve a fair number of issues (in the arena of bringing modems back to life by resolving COM port issues). Sometimes, though, I had to tell them to get a RMA # (for returning their piece of junk).

Thanks for the well-written article. I need to apply these lessons more.
London Iyengar Yoga Courses
Yoga Classes in London, private tuition available, Iyengar Specialist.
for what it's worth, i've been using http://www.lala.com/ more and more, and have come to really enjoy it.