skysight-inc/HackerNews huggingface dataset:
The real problem with statistics is that people want an answer to an impossible problem. Namely they want to be told the probability of the world being a particular way. But anyone familiar with conditional probability can easily see that there is no way to come up with that answer, because the conditional probability of something after an observation depends on the assumed probability before the observation.

There are multiple approaches to this problem. What frequentist statistics does is answers a different question. Namely, "What is the probability of getting a result this strongly against the null hypothesis if the null hypothesis is true." This is appealing in that it is an objective probability that seems to say something about the problem under discussion. However people consistently read it as, "What is the probability that the null hypothesis is true?" Which is simply wrong.

There is a second problem with frequentist statistics, which is what this article is about. Which is that the objective-looking probability you get depends on the null hypothesis chosen in ways it shouldn't. Basic familiarity with Bayes' Theorem and conditional probability demonstrates that it is impossible for it to matter whether your intention was to flip a coin 6 times, flip it until you get both heads and tails, or flip it until you get heads. This factor cannot affect the conditional probability of the coin being biased. But those three different intentions translate into 3 different calculations, and 3 different answers in frequentist statistics.

So that's the frequentist approach. What is the Bayesian approach? It is to come up with statistics that inform us on how prior beliefs before observation turn into posterior beliefs after observation. The advantage of this method is that it is intellectually honest. The disadvantages are that it is complicated and people notice that it is confusing. (The frequentist approach is confusing as well, but people don't notice their confusion. Instead the confidently draw the wrong conclusion that the null hypothesis has been proven.)

There are other approaches. The article touched on my favorite when it pointed out that we should report likelyhood ratios rather than probabilities. This is absolutely right. The effect of an observation is to modify our prior beliefs, and likelyhood ratios concisely describe how we should modify them. Plus they have the great ability to stack - you can take 3 experiments and combine their likelyhood ratios to come up with the likelyhood ratio for having seen all three experiments.

Unfortunately, though, everyone knows frequentist methods, people accept them, and it is very hard to get people to see what is wrong with them. So alternate approaches, though theoretically superior, face an uphill battle towards acceptance.
Very, very true. Just as no battle plan survives contact with the enemy, no product idea survives contact with the customers. From the ground up you need to plan for change, keeping your development cycle flexible so that you can adapt and adjust to feedback from the customer.

It is like the difference between writing code with configuration variables embedded in hundreds of places in the code or building it with a user interface to allow the customer to adjust the settings.
The argument is that Bayesian statistics are less subjective because while we won't come to the same conclusion, we will perform the same update to our prior probabilities.

An example: consider a medical diagnostic test with a 1% false positive rate and no false negatives.

If I understand him correctly, Eliezer is arguing that the scientist (i.e. the lab running the test) should present an update rule, not the posterior probability. I.e., the lab report should say: "Whatever your prior estimate of P(sick) is, your posterior should now be P(sick)/(0.01 P(not sick) + P(sick))" instead of saying "the patient has an x% chance of being sick".

Suppose before the test I believe the odds the patient is sick are 1% and you think they are 10%. Once the test comes out positive, I've updated my probability estimate to 50%, you to 92%.

The update rule provided by Bayesian statistics is objective, even if the posterior is subjective. The frequentists have nothing comparable which is objective.

See also this link:

http://www.overcomingbias.com/2009/02/share-likelihood-ratio...
Can you please put a video of your product in action (being used by multiple users) on your site please?
Really nice article. Knowledge as an industry must mostly play a supportive role to the physical world of oil, food production, building and maintaining infrastructure, etc.

Most people probably don't like to think about this, but at least in the USA we are looking at a major transformation in our society. IMO, either our government will go bankrupt or inflate the value of the dollar towards zero.

This economic trend will certainly force us (eventually) to take a hard look on IT spending, military spending, etc. to evaluate what value we get for spent resources. I also expect our society to decentralize as unemployed/under-employed get involved in local food production, etc. The future does not frighten me, and will probably be very interesting to live through, but things will change a lot in the next ten years.
Simple Weight Loss Tips - 10
Welcome to HN.  Please, read 'Hacker News Guidelines.' We take them seriously.
Probably not today, but you try it out: http://collabedit.com/display?id=44959

You can also open multiple browser windows if no one else is there.
This isn't Reddit, or Digg, and it's one of the reasons why we don't like comments like the above. Please refrain from snarky one liners. See: http://ycombinator.com/newsguidelines.html
How web design goes straight to hell
Definitely! Agreed 100.
Thank you for repeating the party line, but it doesn't address the issue at all.

The most similar language in most similar situation - that is Ruby - took exactly the opposite road and was extremely successful at it.

Most syntax changes have minor impact on runtime - they increase expressiveness, but from interpreter's perspective just translate down to similar basic operations (look at decorators in particular). If alternative Python implementations couldn't handle even that much in 2 years, they'd be doomed anyway. I'm pretty sure they could do it just fine.

Python is extremely unlikely to introduce any difficult to implement features like callcc. It's just not that kind of language.
I'm 54 and come here first for the tech articles. But after lurking around for many months I've come to respect many HNers a great deal. So I also appreciate reading their thoughts and opinions on topics of the day. (healthcare reform, net neutrality etc.)
Jacob Kaplan-Moss recently wrote a series of really good articles about writing good documentation, I highly recommend reading them if you're about to embark on making technical docs:

http://jacobian.org/writing/great-documentation/

He subdivides documentation as a whole into more or less the following three sections:

- Tutorials

- Topical Guides

- Reference

Screencasts can work for Tutorials, and they can be used for marketing purposes, in which they're closer to topical guides but probably won't provide as much depth as a textual guide could.

Imho. a screencast can be great as a marketing tool, but not so much as a learning tool.
I would also like a fix. It sounds like someone simply encoded the video with way too strong audio-compression - the original material is probably fine.

I'm very interested in the talk but it's near impossible to understand him and the warbling noise gives me instant-headache. No way for me to consume it as it is.
NXDOMAIN = non-existent domain. In this case instead of the DNS server telling your resolver a given address doesn't resolve, it's telling it that they all resolve to OpenDNS's search service. There's a summary of some of the issues here: http://www.semicomplete.com/blog/geekery/comcast-dns-hijack-...
A few are, but in general I like two types of blogs:

1. how-to articles that solve specific development problems I have run into

2. thought-articles that expose me to a different point of view, even if I don't entirely agree with it

I have enjoyed screencasts on slime+emacs use and some rails stuff.
No navigation, no copy/paste, fixed speed, hard to do in parallel with actually monkeying with the product (the best way to learn anything :) ).

They might have a niche for things that are heavily GUI-oriented. Click there, drag this from here to there, slide this slider - this is where I think screencasts might be both faster and clearer than narration + screenshots. But that's rather user- than developer-oriented.
"This was largely because merb attempted to do much the same things that Rails was trying to do, but to do them in a much better way."

Or maybe because Merb was done by people already hooked into the Rails community, and so could more easily get the attention of developers who don't otherwise seem interested in exploring other frameworks?

Nitro, for example, predated Merb (it came out at the same time as Rails) and was doing many things better than Rails (multi-threaded, easy to swap out ORMs, no boinking of built-in classes, Rack-like app request processing pipeline before there was Rack; much faster overall), but got almost zero traction. And I don't think it was because people tried it and didn't care for it, but because all the noise was (rather deliberately) about Rails.

There is a "winner takes all" mentality among some Rubyists that leads them to dismiss anything that isn't the clear leader. Sort of like high school kids who are afraid to be seen talking to the nerds because people might think them uncool.
The difference is that bayesians acknowledge this explicitly. You publish the way to calculate the posterior given someones prior so that everyone can fill in their own priors to determine their posteriors. This calculation is objective. With frequentist statistics everything is subjective, including the results you publish.
Hehe, yeah, pretty impressive :)

I think the point I'm trying to make is that they can write high quality songs at the beginning of their career as well as at the end. They didn't need 10,000 hours to come up with a great song, an expert quality song. I bet many had only written 20-30 songs before getting famous, certainly nowhere near 10,000 hours of song writing. You even get bands which then fall into mediocrity after a fantastic debut.

I have no doubt they put in hours beforehand, but they've been at school, certainly not playing for 3 hours a day for 10 years.

Not only that they have some mastery of the instruments, song writing, stage charisma, all distinct skills. Many great guitarists write crap songs.

So where did they get the 30 years to get all 3 skills?

I think the Beatles example is bad because they had an extra skill, the ability to morph their music with the times that many other bands don't (Bowie is another example of someone who was able to do this). They are eternal because they constantly changed, put out great albums, which has nothing to do with the 10,000 hours of practice they put in in Germany.
I would suggest that age can bring focus. You may have various interests, but you get better and sorting one from another at deciding which gets attention when and where. You've seen more, and you've learned how to choose what gets attention. To be selective.

This is different from the absorption of youth. It may actually get a bit harder to be so fully absorbed. But a bit easier to set aside an intriguing novelty that does not pertain to your current endeavors.
Your sentence Tech starts to look the same after you have seen so much of it. resonates with me a bit (I'm 43 and have made my living as a developer for over 20 years). What happens a lot is some old idea is dusted off and becomes a current fad. Sometimes this is because the old idea was a good one, but sometime it's because a younger generation is ignorant of history and why that idea didn't work the last time it was tried.

So sometimes there a feeling of "What! That again??" when some topic such as "NoSQL" comes up, but that doesn't, to me, make tech in general less interesting.
Tell HN: I have 13 Wave invites for the asking
I'll happily send out all 13 to fellow HN'ers, since I've already invited my original core.

Oh, my email is cpr@emsoftware.com .

Anyone else have some to share in this thread, so we can make sure everyone's covered who might want one? Or does everyone already have an invite? ;-)
How To Find Those Red Balloons
It isn't the "xenophobes" who attached "we need founders visas" to the guys hanging out in front of Home Depot.

The fact that the US needs and can benefit from importing more brains does not imply that it should continue to import more brawn.

That said, H1B is pretty much a disaster. Look at the allocation - tech does all the fighting but doesn't get most of the recipients. It really should be an auction (to ensure that it does go to "high value") and then expanded.
Is the critique that they are using PHP or that they aren't using some kind of extensionless routing?

Either way it's silly to use superficial factors like that to judge large websites. Given the performance profile, data interconnectedness, complex public API, and interface richness of Facebook, the rate at which they push changes (small changes daily, weekly major pushes) and advance the product is astonishing.
Jacob's docs were fantastic. I read them after they were posted on HN a few weeks back.

The marketing aspect is pretty important. I was considering following his (my friend's) advice for that reason. Every week or so there seems to be a "Watch me build a blog in 5 minutes" screencast. I don't really think they are tutorials; they are closer to movie trailers.
To me, more interesting than Bayesianism versus Frequentism was the observation that scientific procedure as it currently stands allows a field like Parapsychology to survive.
Congratulations to them!

It's exciting that they were so impressed with Wave. It's easy to overlook the potential of Wave in its current incarnation, but this, to me, suggests there is some seriously awesome stuff coming to the Wave platform.

These guys created Etherpad, and they are excited by Wave... ergo, so am I!
I'm a 57-year old guy and whole-heartedly agree.

I'm also less tolerant of those who bring up old ideas as if they were newly discovered except if they happen to have been before their time. However, "stupid ideas" really get me.
They also claimed it wasn't proposed until the beginning of this year. I don't care who was dragging their feet, that is pathetic.
The speaker for http://www.stanford.edu/class/ee380/Abstracts/091202.html said that an interesting fraction of computer crime originated in Indonesia. It's not the US, Russia, China, the Baltics, or Korea, but ....
This is pretty sweet; it seems like a nicer way to do one-off pastebins than current sites.
I agree, I remember seeing some old RoR screencasts a couple years back, thinking: That editor is awesome. That's what made buy a copy of TextMate. (I still haven't ever used RoR tho)
You are not alone. I spent hours last night working on my own that's been abandoned for a long time. You are a little ahead of the game right now, and it looks nice!

With the leader out of the way for a while, I can't help but wonder how many of these will pop-up, and how many geeks did the same thing last night :)
> Novel for the early 80s, but in practice these systems seemed to be limited to very specific domains.

IIRC, Bacon's domain was discovery of certain kinds of relationshps, not lung disease or something like that.

I agree that something that is packaged for general use is more valuable.
It seems to me that many (most?) people I know just can not cope with the truth about our economy, but there is some real nervousness. I have had friends who have lost over 50% of their retirement funds in the last two years - they tend to be more realists than other friends and family members who get (or will get) government pensions. The pensioners tend to have faith in the US and local governments that seems to me to be outside the bounds of clear thought and common sense.

I may be over simplifying this, but I think that in addition to the obvious greed and avarice of the elites, that we have a basic problem with fear. Fear can keep a society from making sacrifices now rather than postponing the pain and making the future more difficult.

I think that the next ten years will be interesting to live through (in negative and in positive ways) but facing problems head on seems like a better approach to what our leaders are doing now.
I get lost in the argument with these simple coin flipping examples. The way it's presented is so confusing for the weird second case of how many trials it takes to get a tail.

In the first case, the probability of getting "five tails or more" from a fair coin is 11%, while in the second case, the probability of a fair coin requiring "at least five tails before seeing one heads" is 3%.

I didn't check the numbers here, but this seems perfectly reasonable to me in terms of hypothesis testing. He's playing on the fact that for cases like THXXXX you would get a result of 2 in the second experiment. You are taking the same results, but applying a different metric to them (position of first head versus count of tails). Of course p will be different.

I understand that the value chosen for significance of p is subjective, but the experiment itself makes perfect sense to me.
Your understanding is incomplete. Bayesian statistics are not subjective at all. Instead they objectively describe the correlation between prior belief and conclusion. People may draw different conclusions for subjective reasons, but that subjectivity is in the people, not the statistics.

Now it seems you are objecting to the fact that your initial state of mind affects your conclusion. But it is unavoidable that there is no way to make sense of observation except through the lens of prior beliefs. You can be explicit about it and draw conclusions with a correct methodology about it as in Bayesian statistics. You can be implicit about it and draw conclusions with an incorrect methodology as in frequentist statistics. (See my other post in this thread for an explanation of why frequentist methodology is incorrect.)

Let me offer a simple example. Suppose a pregnant woman you know gives birth to a boy. Is that evidence that babies are more likely to be boys than girls? Obviously it is. Is it strong evidence? Obviously not. Should we upon observing that conclude that boys are more likely than girls? Obviously not.

Now suppose that you have no prior knowledge other than we see about similar numbers of boys and girls. I know that of the last 100,000 babies born in the USA, 51,157 are boys. Suppose we are both told that at the local hospital, 92 of the last 200 babies born were boys. I submit that we both will and should wind up with different conclusions about the relative likelyhood of boys and girls. Why? Because different prior knowledge lead to different prior beliefs, and those prior beliefs when modified by identical evidence lead to different conclusions.
Agreed. Also, audio can be quite intrusive meaning that you then have to find an appropriate time to view them.
The Chutzpah of Israeli Startups
I must say, the value of the dollar is depressing. Take a gander at this comparison I did of the Yen and Yuan compared to the dollar (btw, falling is a bad thing in this chart!)...

> http://finance.yahoo.com/echarts?s=CNY=X#chart3:symbol=cny=x...

In my opinion as a US citizen, we need to enforce the currency law and enact the death sentence upon all who have perpetrated this upon our country and our fellow citizens. These persons have committed mass fraud, manipulated financial markets, and have almost irrevocably destroyed our prosperity. Off with their heads: we must make an example of them!

It's high time we return to the gold/silver standard.
Yes. I didn't want to jump the gun, but this is certainly the ways things are going.

I really hope you are right! :P
{less} CSS for .Net
I used to watch a lot of screencasts in the past years because of ~1 hour bus+train commute to work. It's a great tool to check out new things, or to get a fast intro. In this way, I enjoyed the New Relic screencasts - they were more about the general idea of how things work, not specific implementations.

For figuring out how to do something, I find Railscasts great. But that's partly because when I want copy/paste and search, I can always go to ASCIIcasts.

So, to for your problem: I'd like to be able to watch a screencast explaining the general idea of the app and a quick intro to developing with it, but for the specifics, stick to text.
any idea how the relationships are inferred?

[i've used last fm's artist tags to construct relationships between artists (artists with tags in common are related) and so generate playlists that of related tracks - the last fm api is really easy to use.]
Interesting article, but not terribly useful or pragmatic. I was also hoping for a little more self-awareness. The author is essentially railing against the US policy of printing money to get out of crisis (which is a valid point) by advocating a solution that would send the entire world economy into a recession that would make this one look like the 2001 dip. Because it's the WSJ, I guess I should have expected at least some conservative rhetoric, but I found it a little too naive.

What would have been a more interesting question to consider is what can be done to address the trade deficit and inflation problems with our current monetary system. Although it's valid to suggest that minimum viable product wouldn't work for macro and international economics, it has a lot less practical application.
Join us for a Django development sprint in Dallas, TX and Triangle, NC
ClutterHS: Create multi-platform, OpenGL-composited GUIs in Haskell
a much better approach would be to let users create a custom subject, to describe their idea

Have you ever looked at http://fairsoftware.net/publicProjects ?
Ask HN: Common Lisp vs Clojure for web apps?
I'm building a somewhat complex web app, but eventually what I need are 1) support for postgres, 2) libraries which make it easy to deal with data in json and xml, and 3) allow me to generate html and javascript.

Overall, I think both CL and clojure support this basic feature set. And I'm trying them both out on a trial app anyway to get a feel for them.

However, I'd appreciate if anyone can provide me with some tips/guidance/gotchas/"this might help" sort of advice to get me going. Thanks :)
Agreed. Screencasts are linear at the presenters pace. Documentation can be skimmed or probed randomly. Often I dive deep to see how a particular issue is handled.

All too often screencast production qualities are terrible: lighting sucks, the cameraman focuses on the wrong things, the pacing is wrong...

Developers are readers. It's mostly what we do: code, documentation, logs, stack traces. We are a pretty literate bunch. OTOH a video of a new tool would be more useful for a mechanic than the usual postcard description.

That being said, good graphics help a lot. I found the Wikipedia sort demos wonderful.
I'm with you. I like screencasts for getting started with a new tool but once I'm up and running I tend to reach for in-depth written documentation or reference materials.

But, this makes screencasts really valuable IMO. When you have an hour to evaluate something a screencast seems more useful than a massive book.
It's the opposite for me. As I get older I become more interested in tech stuff, and less interested in the general stuff. I used to spend almost all my time considering the meaning of life and all that.