skysight-inc/HackerNews huggingface dataset:
IANAL, but...

To reuse copyrighted content, you have to consider fair use eligibility and such -- but fortunately, not all data is locked down by copyright.

One of my favorite ten USSC rulings:

http://en.wikipedia.org/wiki/Feist_Publications_v._Rural_Tel...

"""
It is a long-standing principle of United States copyright law that "information" is not copyrightable, O'Connor notes, but "collections" of information can be. Rural claimed a collection copyright in its directory. The court clarified that the intent of copyright law was not, as claimed by Rural and some lower courts, to reward the efforts of persons collecting information, but rather "to promote the Progress of Science and useful Arts" (U.S. Const. 1.8.8), that is, to encourage creative expression.
Since facts are purely copied from the world around us, O'Connor concludes, "the sine qua non of copyright is originality". However, the standard for creativity is extremely low. It need not be novel, rather it only needs to possess a "spark" or "minimal degree" of creativity to be protected by copyright.
...
In the late 1990s, Congress attempted to pass laws which would protect collections of data, but these measures failed. By contrast, the European Union has a sui generis (specific to that type of work) intellectual property protection for collections of data."""
Can I recommend the book "Programming Collective Inteligence" (http://www.amazon.com/Programming-Collective-Intelligence-Bu...)? It doesn't actually describe a Reddit-like algorithm but it does describe looks of recommendation algorithms along with practical Python code. Should get you thinking in the right direction.
30 Useful PHP Classes and Components
> Does the image approach mean that Smalltalk is not really suitable for quick and dirty sys admin type scripts that you would use Perl or Ruby for?

No, it's not, but that's what we have Ruby for. :-)

> would it be technically feasible to have a different Smalltalk image for every single account that is loaded only when that account is being used, with basically all the account data inside the image?

I think that that is how DabbleDB works today. They have a different image for every account.

Avi's post from March about how Gemstone works has a good overview of things: http://www.avibryant.com/2008/03/ive-had-a-numbe.html
I wasn't insulting the intelligence of the guy reaching out for help, I was insulting the intelligence of the fraction of the YC membership that thinks autodidactic is pretentious.
Google's Android demo shows app store, tweaks iPhone formulas
Umm...not sure on a specific "algorithm", but....

I would say just do something which takes into account how quickly items are getting voted on and how many comments vs. votes it has. Also take into account sometimes there will be more people on site than others; If you get a swarm of people around the same time (i.e. launch), some items submitted around that time will get tons more votes than things submitted an hour ago - those stories may not have been exactly "less popular", just had less users to make the votes count.
Vernor Vinge: Signs of the Singularity
The paper you linked to, which I just got around to looking at, has no bearing whatsoever on the security of SSL for password security. It was also never, so far as I can tell, published.
I think any similarities between the uncertainty principle in physics and limits of what we can know about our minds is poetic or formal.

There might be some rule which the uncertainty principle, the mind problem, and Godel's incompleteness fall out of, or that relates them eloquently and definitively at a higher order of abstraction, but they aren't the same.

I think there is a common kind of error equivocating the hard problem of consciousness (which is a formal way of understanding your "limit to what we can know in this aspect") with uncertainty principle or Godel's Incompleteness (another idea that looks like and is confused with the uncertainty principle or the hard problem of consciousness if you squint right). It is like recognizing y=2X^2 and y=2X^2+2 have the same derivative, but using this insight to argue that 2X^2=2X^2+2.
rapleaf have a free API, but they will create an account for each email address that passes through it.
Normally I'd agree. But without PG there'd be no Hacker News, and many of us would like some avenue to wish him well on a very significant occasion. So here's to the newlyweds :) Cheers.
If the brain uses bayesian inference, could emotions and intuitions perhaps be functionally equivalent to the approximations needed to do the impossible computations you are talking about?
Ugh, it was my birthday...
Twitter and the importance of architecture
The headline makes it sound worse than it was... no servers were damaged in the explosion, power was just knocked out.
Rm -r /* -- a lesson in policymaking
I think she owned a third to begin with.
Congrats PG!

Best wishes to the both of you.
I'm not sure if you actually read the post, but I re-iterate how TC is not an indicator of long term success. If was fun, and it put it on the map. I just wanted to share the experience. Its not the endgame, but rather, just the beginning.
Soocial has a working service (still private beta though) that uses Google Contact API.

They use OAuth authentication too, which means that you can grant access via GMail, so you never have to give out your password to a third party (Soocial in this case).

http://www.soocial.com

This widget is a terrible idea (you don't give the PIN code of your ATM card to your waitress do you?), but might be a great idea when implemented using OAuth.
Google caught “anonymously” bashing eBay
pg didn't post it here. this is a user generated news site. a user posted it. other users voted it up. that's how it's supposed to work.
You know, where I was growing up phone calls were expensive, so people didn't call in advance if they were dropping by for a chat. Cheap unlimited phone calls here have resulted in a culture whereby you absolutely, positively have to call in advance for the tiniest thing. My friends are always surprised when I do drop in without calling and Arrington thinks the practise is akin to breaking in. shakes head
I don't understand your concerns. Facebook, LinkedIn, Twitter, Loopt, Flickr, practically every single site with any type of social component implements this idea. What's the problem?
Well it was powerful enough to knock down 3 walls surrounding the power room!
Best Wishes!
Thanks! This is just the type of solution I was looking for.
i wasn't proposing it, i asked if thats what you meant
Ubiquitous implementation does not mean it's a good idea.
Thanks! Look forward to the launch.
i was implying, how come they didn't run into security problem when they are practising a non-safe method? why do they expose so many users to such danger without their knowledge?
When it works, it works well, and it's a division of RackSpace, which - until recently - had a fantastic record.

Unfortunately, Mosso has been plagued with problems. So if you require reliability, look elsewhere.
I've always thought of Noam Chomsky's "language acquisition device" as a kind of prior on the space of languages that's hardwired into the brain. The process of learning a language consists of fitting the parameters of this model to the sense data that each child is immersed in. Of course Bayes' rule provides one way to update our guess about the parameters of this model given the current data but who knows if that's what's really going on. I think the evidence is pretty strong that this flexibility in the space of priors over language only lasts a for small precious window until we become adults, and then we're all stuck with lame accents. So it does indeed have some time dependence.

When I was talking about the implementation of inference I was referring to the people doing experiments trying to figure out the "neural code" - Questions like: how much information is conveyed by a single spike? Is there any meaning in the rate of the spikes? How much information is there in a coincidence when two or more neurons fire simultaneously? What rules govern the plasticity of the activation or inhibition relationships among neurons? Does any of this have anything to do with probability? A great book on this is "Spikes" by de Ruyter and Bialek et. al.

I didn't mean to sound dismissive about the electricity thing - I think hodgkin and huxley were both given medicine nobels for their work on the nerves. I guess the electricity thing really goes back the 1700's when galvani made a dead frog move with electric current.
There's always the source to news.yc:
http://ycombinator.com/arc/arc2.tar
With the exception of a few close friends, having someone dropped by unannounced shakes my concentration for at least a couple hours.

The kind of work I do is like holding a house of cards in my head. I am glad to see the social conventions go that make people feel free to expect to come knock over my mental house of cards anytime they want, steal my valuable time with their petty small talk, and treat me like I'm rude because I'm not skilled at cheerfully pretending their intrusions are wanted while I inwardly curse the damage they did to my concentration.

Extroverts have had thousands of years to do things their way. I'm glad to see societal conventions shifting in a way that benefits introverts needs to have uninterrupted time to think.
Oh frabjous day! Callooh, callay!
I'm surprised to see the turks up there. Anyone here from Turkey care to offer an explanation?
I always had the thought in the back of my mind that there was something between pg and rtm. pg just speaks exceptionally warmly of rtm, so I considered it a possibility. I am not trying to be insulting or be understood in the wrong way, its just something that came to mind.

But anyway, enough with silly thoughts, congrats to pg and jl!
I'm not referring to cryptography, I'm referring to security in the general sense. Security is fundamentally about placing obstacles between your assets and those who you don't want to use/alter/control those assets.

When you design software, you have all sorts of assets you'd like to protect, even for the simplest cases. Administrative access, general data, CPU cycles, workflow, network bandwidth, and maybe even passwords, secrets, etc. Your job (with a security design hat on) is to place the firmest obstacles you can between those assets and those you don't want to use them (usually prioritized by a combination of probable risk, severity of loss, and cost to protect).

Thinking in terms of assets you need to protect is a highly recommended way to design securely, regardless of whether you're talking about military, bodyguards, buildings, software, cryptography, or systems. You then proceed to put up your best obstacles and then reshore/rebuild/redesign those obstacles when they are known to be compromised or compromisable. All the while ensuring that the system operates as efficiently as you can reasonably manage.

Fixing "a buffer overflow" is just a method for repairing the software obstacles you already had in place. Sanitizing inputs is, indeed, adding an obstacle for attackers. "Proving code paths" takes into account that your obstacles remain in place to protect your data with certain assumptions about the systems involved.

If you're interested in security engineering in general, I recommend picking up some of the latest (last 5-10 years) literature on Threat Modeling as it really can change one's perspective on what security is all about. A lot of engineers I've worked with think of it in terms of "using the best string library", when it's more about designing systems to protect your assets as best as you can and still get your job done.

There are all sorts of practices, tools, fixes, libraries, etc that help improve existing "bricks" in your obstacles, but they're absolutely no replacement for actually understanding what you're trying to protect and having a strategy for preventing those from being compromised. The best security is designing such that you never put your assets at risk than that you use the right libraries to, for example, manage them during transit.
It encourages people to give sites login details to other sites. That is a really bad idea, especially when it's only to import your contacts.

Would you give someone the keys to your house so they could rifle through your letters, address book, computer etc to find your contacts?

A far better solution would be a push from the user - email the website a list of contacts for example.
"I think the evidence is pretty strong that this flexibility in the space of priors over language only lasts a for small precious window until we become adults, and then we're all stuck with lame accents."

I think this is true for most people, not everyone.

If this is true for most people, we can locate these differences by comparing autistic people who quickly acquire languages as adults with normal people. It's interesting that autistic people skilled with language often don't have noticeable accents in that language, even with a language they learn in adulthood.

This flexibility in the space of languages involves a honing of perception. As you learn your first language, you become better at perceiving the sounds of that language, but your ability to perceive the sounds of another language go down. For example, a person who speaks exclusively english speaker cannot discern between sounds that would have different meanings that a chinese speaker would easily hear.

(Deferentially), I think you are wrong about where the hardwiring occurs. I think the hardwiring that limits most peoples ability to learn language arises is perception, not the space of priors. I believe if you could change the hardwiring that occurs in making sensory discriminations relevant to the language, the "space of priors" would still show the ability to learn language. It's true that there is a precious window for most people to learn languages, but that's because the perceptual paths get hardwired, not the memory and processing that make up what we seem to be calling the "space of priors"

But, that's not the scale of time dependance I was thinking. Even if the scale of time dependence was 5 seconds, and the new data was just seeing it was raining out, the way your brain understands the concept of rain involves accessing memories it destroys and recreates as part of recognizing "rain", as well as the particulars of your emotional state and what's going into your sense of smell, so much that the set of priors 5 seconds before would be different then the ones now, even if you took out the information about it raining.
That's why it's depressing, the fact that it was number 1 for a bit means that a fair amount of users didn't understand it.
So now she owns 2/3? :)
I enjoy your essays very much, especially the ones about "life" matters.
I spent only a week in London and it was enough to get that message. That city was very inspiring but also its message's pressure was a bit threatening to me. It wasn't a comfortable ambition to me, I think that it wasn't focused on the pleasure of the work it takes to achieve your ambitions.
I would've enjoyed reding more about this subject affecting european cities, I was raised and live in Spain. People not always put Spain and Ambition together, excepting Madrid, the capital, the message is "carpe diem", but in relaxed and quality-living (concerining the weather) way.
I believe that Europe's economical and political system influences in people's ambitions, I find Americans and those moving there (from what I read, at least) more ambitious, less frightened about things not turning as expecteted. In Europe we depend too much on the State.
You really need to calm the fuck down.

"if you have to explain to yourself what a "salt" is, or you can't spell "nonce", you shouldn't be designing security systems."

No, I don't have to explain to myself what a salt is, though my "Here's how I understand it" introduction probably mislead you into thinking that. The only reason I said "Here's how I understand it" is that initially I wanted to explain both what a salt is (a notion I certainly understand because I read about it and implemented it) and what a nonce is (a notion I probably don't understand because I didn't really read about it and didn't implement it).

I know I still have much to learn but at least I know what I don't know with respect to nonces, and I find it pretty lame that you're saying I shouldn't "design security systems" just because I can't spell a word properly for a technique I just admitted I "never used and can only make guesses [about]"

I've been working like crazy for the last 2 years to learn everything about Common Lisp and making websites and you're saying I should give up everything just because I still have things to learn?!

You can't judge someone's skills just by looking at a data point like that. The concepts of closures and macros are now completely automatic and obvious to me but I wouldn't call someone who never heard of it or has just a basic understanding of it "someone who should never program". Of course I'd point it out if they said they had a firm grasp of it while it was obvious that they didn't.
if bayesian probability is how the brain works, why is it so hard for most people to understand this:
http://en.wikipedia.org/wiki/Monty_Hall_problem

seriously, even the most clever people even have huge trouble grasping it.
Is there a viable market for this type of transportation? What happened to the concord? Lets not forget that was during a time when face-to-face was the only way to conduct business.
ahhh what a clever use for technology... it's certainly a sharp piece of equipment...
good points. more on the subject:

http://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_t...

Minds and machines

Main article: Mechanism (philosophy)#Gödelian arguments

Authors including J. R. Lucas have debated what, if anything, Gödel's incompleteness theorems imply about human intelligence. Much of the debate centers on whether the human mind is equivalent to a Turing machine, or by the Church-Turing thesis, any finite machine at all. If it is, and if the machine is consistent, then Gödel's incompleteness theorems would apply to it.

Hilary Putnam (1960) suggested that while Gödel's theorems cannot be applied to humans, since they make mistakes and are therefore inconsistent, it may be applied to the human faculty of science or mathematics in general. If we are to believe that it is consistent, then either we cannot prove its consistency, or it cannot be represented by a Turing machine.

http://en.wikipedia.org/wiki/Mechanism_%28philosophy%29#G.C3...
With a few bright exceptions like Robin Hanson and Paul Wallich, the IEEE Spectrum issue follows the usual pattern: People who have no engagement with the serious Singularity analysts, making up stuff at random and saying "This is what the Singularitarians must believe."

The term now has at least three major separate meanings, by the way. See http://www.singinst.org/blog/2007/09/30/three-major-singular... for preliminary disambiguation.
It's obvious that many of the existing Ruby implementations will continue to be adopted, each with their own strengths and weaknesses. It is also my understanding that they intend to adopt a business model in which they'd have a free version (that will suffice for most people's needs) as well, and they plan to make commercial versions affordable. But we don't know what they're going to charge yet, so it's really just speculation at this point.

They are going to release the Ruby parts as Open Source and it's not set in stone that they're going to keep the VM closed either. However, if they choose not to make it FOSS, it's very unlikely that MagLev will become the single most popular choice. That said, companies pay a lot of money in hardware and hosting fees, so if the speed claims were to be true, many companies would have no qualms about paying for a much faster implementation. It's not going to be everyone's cup of tea, but it's a choice, and choices are usually good.
Perhaps the servers were not near the explosion?
I've spoken to Avi and that's exactly how DabbleDB works. He front ends the system with Apache and uses a rewrite map to launch an external Ruby script that checks to see if the image for that request is running. If it is, it returns the port its running on, if it's not, it launches the image, records what port it's on and and proxies the request to it. Images time out after X amount of time and shut down on their own.

Each customer gets his own image, though the customers data is stored outside the image using an image segment. This lets them upgrade the Dabble image without messing with the customers data. When an image fires up, it loads up any image segments into it.

Basecamp is also exactly the kind of app that fits well with this architecture. Lot's of small independent databases.

It's not that different from how Hacker News uses load-table to store the customer data in files and loads everything into ram when the web server starts up. Both systems are basically homebrew object databases very well for the limited load placed on them. Gemstone is the granddaddy ODB that can handle anything you can throw at it.
Um, just read the linked article. "The population of smart machines would explode even faster than the economy. So even though total wealth would increase very rapidly, wealth per machine would fall rapidly." Robin Hanson is not a self-declared "Singularitarian", but I, his co-blogger, certainly am, and we had a discussion about the effects of future resource constraints on intelligence here:

http://www.overcomingbias.com/2008/05/biting-evolutio.html
Allow me to spare you all from reading this drivel by summarizing it.

"Using Macs will make how mediocre everything else in your business stand out the same way a clean spot on a grimy floor will make it obvious how bad the floor is. It's easier to shit on the clean spot then bring everything else to the same level of quality. So don't buy macs for your business."
I'm reading his book now, and it's excellent. And I rarely enjoy non-fiction 'meme of the day' type books.

His book is like conversation with an older, intelligent person who has the kind of wisdom you get from life experience. It's also very relevant to anyone working in a startup.
Military UAVs: Up in the Sky, an Unblinking Eye
Good for you two, Paul and Jessica. Congratulations.

Expecting a new essay on how this event change your life :)
Twitter definitely does not have millions of daily active users. There are only 824M total tweets. Comparing IDs from posts 24 hours apart shows there were ~500,000 tweets in the past 24 hours.
Me too, sorta -- we got married this Sunday a year ago, but the date is 2 days off
Yeah, congrats, Paul and Jessica.

Now, I have some questions about my startup, and I also have a few more questions about Arc, because it's been buggy for me lately. And, Hacker News has been really slow, and we really should ban such and such a site from submitting links here. And the quality of Hacker News... blah.... blah... blah...

Oh, wait, you're on your honeymoon... OOOOHhhhhhh, sorry, my bad.
And can I say, Jessica was planning startup school 6 weeks before her wedding! Holy Crap! How did they stay sane!?
Congrats PG and RTM!!
Because people don't necessarily have a conscience understanding of how the lower layers of their minds work. Being a Bayesian neural network doesn't imply you understand Bayesian neural networks in general.
Readbag - a shinier instapaper
It's not silly, that would be saying there's something wrong with it and there's not! CA just ruled same-sex marriage legal and it's just as morally valid regardless :)
...used the Internet in an attempt to have sex with a 13-year-old girl. The kicker, of course, was that the "girl" Naughton met in a chat room called "dads&daughterssex" was an FBI agent trawling for pedophiles, and even the decoy he met on the Santa Monica pier Sept. 16 was a sheriff's deputy in pigtails.

http://www.salon.com/media/col/elde/1999/12/22/naughtonwalks...