skysight-inc/HackerNews huggingface dataset:
Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large. It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists. Maybe it is understood by scientists that peer review has flaws, but when they appear on morning talk shows the term "peer review" is wielded as a holy sword.

Regardless of the state of the current process, how could review become anything but better from increased openness?

>"And since reviewers know that they can't verify correctness, they tend to look for theoretical papers that very closely reproduce experimental results."

This seems to be part of the problem. "We'll use this data series for the 60 years that it agrees with everybody else, then throw it out for the 40 years where it doesn't, then not show anybody the 40 years we threw out".

But then again, I'm no climate scientist. Although, one would think that no harm would be done by releasing the data, even if there were legitimate statistical reasons for truncating it.

>"Ultimately, you have a system where a lot of genuinely smart, skeptical people are attacking ideas from all sides until either a consensus emerges, or the idea is destroyed."

If you're a smart, skeptical person attacking a paper that shows global warming to be severe, Michael Mann will get you fired. Or ignore your ideas. Mann just published a paper that appears to use an inverted data series after McIntyre already correctly pointed out that it was inverted, out of pride I suppose. Fine science, that.
Rachel Maddow doesn't even own a TV. Should that be a warning sign?
Depending on OP's history, I may disagree.

When I graduated in CS with an M.Sc. I said to myself that I didn't see myself in a cubicle coding along all day. I'm also more outgoing than the average hacker so I wanted to work with people more than with machines. So I applied for all the "wrong" jobs. (My thesis was about usability so I applied for usability engineering positions, competing with psych graduates etc. while the industry was desparately looking for programmers with my background.)

It was good that I couldn't get the job I wanted, forcing me to take on a programming job. I became a project manager within three years and a product manager in a larger company within five years. I didn't program anymore at my day job during that time.

But I realized I did love programming so I started all kinds of side projects and kept up with technology even though it was not needed much for my job.

What I'm saying is, you may not have discovered it yet. Get a programming job and if you really do hate it, get promoted to some management position (it's often easier than you would suspect because most hackers don't want to do management). I know quite a few people who did that. They're doing a good job and they're happy not having to code anymore.
Given the lack of consensus about what is or isn't happening, I'm really extremely skeptical that we could reach consensus on a way to prevent it.

Sure, it's nice to spend some money researching potential threats, but this one seems to have been blown way out of proportion. Although it's probably been overtaken now by swine flu.
(foo){2,5}
as mentioned before, only if he wants to be a programmer. Even if he eventually goes on to become a business person, I think being trained in CS will give him a far better perspective and trust from the programmers working for him.
I had a lecturer in CS who didn't really code and would freely admit it. A LOT of what she lectured stuck and I've used it everyday in my programming jobs.

CS != programming.
In CouchDB, unlike in MySQL, you come to understand what a join actually is - when it happens and what it costs performance wise.

What I like about CouchDB is that it's composed of a relatively small number of concepts. The more I code the more I prefer tools & libraries that utilize only a small number of concepts- the complexity, if it exists, is safely quarantined into the non-reusable application portion of the code and not in the tools I used to build it.
>In most universities, the computer science program is the best way for people to learn how to program. In the future, I think CS departments will split into software engineering (which focuses on teaching students how to program and engineer software) and computer science (which prepares students for careers researching in the field of computer science). But that's not how things are in most places.

Too right. I agree with this. I think that the "field" of software has generally gone in three directions.

1. How do you compute it (or Thinking about it)? That's CS
2. How do you plan to build it (or Planning it)? That's Software Engineering
3. Go build it (or Doing it)! That's IT

Sadly, at least at my alma mater, the trend of keeping these three fields separate has reversed. Instead, with budget cuts, the departments have merged under the CS umbrella. I think this continues to push forward the layman idea of these three things being equal because of how they are co-mingled -- but really they are as separate of disciplines as Physics is from Road Paving.

Ideally the number of students and practitioners in each fields should be distributed like a pyramid (a la any other engineering discipline). Most people doing, fewer people planning, and even fewer people thinking.

>Also, "theory of computation" is an area of computer science theory. There is much more to a CS curriculum than theory of computation.

I guess I also agree with this. I've always chafed at calling Computer Science the "Science of Computing" though since it's not a science in the classical sense. But everything I ever learned in my undergrad (not counting unrelated curriculum coursework) was definitely related to "computing" in even a loose sense of the term. I just don't think calling it the discipline of "computing" makes it any more clear since "computing" - the act of using a computer (like a Mac) and "computing" the act of performing computations, are easily confused. The "Theory of Computing" has always seemed to have the best balance while providing context for "computing". But I'm open to accepting other 2 word descriptions ;)
Well he says in the article "We jointly own the CrunchPad product intellectual property, and we solely own the CrunchPad trademark." So I'm thinking he probably has this covered, though it may be the other company sees a way around this or does not care about legal implications. It may be the product gets knocked-off by some overseas company before ever being produced itself (look out for the the iCrunchpod).
that is not logical because everyone wants their content found and there are only 10 spots on the first page (and adding more results per page does not fix the issue)

At some point you need a way to rank the sites
Top Words of 2009
This is from April, btw.
Hence the use of quotes around the word "advanced".
so I read a political message in your opposition (and focus on climatology): Well, here's the very first sentence of the article he's commenting on: "Recent revelations that the peer review system in climatology might have been compromised by the biases of corrupt reviewers miss a much bigger problem."

If focusing on climatology is evidence of a "political message", then I think you've fingered the wrong suspect.
The author covers this:

You can, if you are careful, achieve effects such as changing the appearance of all section headings with one command. But few users of Word exploit this consistently, and that is not surprising: the WYSIWYG approach does not encourage concern with structure.
Not to mention that the article skips over a couple important points.

Granted, the iPhone will never see any sophisticated downloadable porn-tagging apps like described in this article, it still has the internet at its disposal. If just looking at pictures of naked chicks on your phone in public places is enough for you to get your kicks, then I don't really see any phone manufacturer having any advantage greater than just the speed, stability, and ease of use of each of their respective mobile browsers (and they nearly all use Webkit so speed for that is pretty dead even).

If moving pictures are more your thing, then any phone that can display flash will ultimately have the iPhone beat, but not without a fight. Little known fact that the iPhone can stream h.264 from any site, not just YouTube. So any website that serves up h.264 through flash can make a mobile site using the same videos without the flash player. The only site I know of that does this is m.pornhub(dot)com (obviously very NSFW so no accidental clicking), but yes, it exists.
I work in a fairly large scientific collaboration. Our data sets are on the scale of a few hundred terabytes per year. So, there's a lot of software to convert from the bits that stream out of the detectors to something that end users (grad students) can analyze.

We have a software group devoted to maintaining the core code. Our code has a nightly autobuild for development code and libraries are released every couple months.

The code is based on the ROOT package (about one million lines of code) maintained by folks at CERN and is well established within the community and elsewhere. Our libraries probably come to about a million more lines of code, which is maintained scrupulously by our collaboration.

Now neither of these sets of code actually do any analysis, they just make it so that the data is usable. In my analysis, there is probably ~10k lines of code to do the analysis and make the figures. In total, there's probably 500k or more lines of analysis code that is not officially maintained.

That doesn't include all of the code loaded onto FPGAs and custom chips reading out the detectors or the code in the trigger system. Nor does it include code written into various simulators used to determine "expected" response of the detectors.

So for peer review to look at code at the level suggested, people would have to look over literally millions of lines of code. To even be able to make it run, a user would have to set up an environment that would take between a few days and a week.

Instead, when I go to publish something, I lay out both my method and how I verified that it works. Reviewers then check that my method is sound and that my verification looks right. They have to trust that I implemented the method as I said I did.

In my collaboration, there is actually an internal review process that verifies code runs and for which a much longer private note must be written to explain all of the details of the analysis. However, there is still a high level of trust that everything was implemented as stated. This review does not qualify as "peer" review because it's conducted by people who will be listed as authors.

While I agree that more review is better than less, I hope this comment illustrates why software peer review is not a reasonable expectation.
It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists.

Absolutely.

I think we're going to have to have some kind of new certification system for research that affects public policy. Scientists simply can not be activists -- the conflict of interest is too great. If you find a meteor is going to strike the earth next year, you'd better be spending your time on orbital calculations and mass estimates, not on the Today show. Leave that to the politicians. Mind your knitting.

In the code arena I'll be charitable and say that mistakes were made at CRU. Public-funded science should run from a public wiki or source control system where the scientists are the only authors but the public at large are readers. Yes, I know that will drive some scientists mad compared to the current secretive system but it's time to let some daylight in, folks. You'll never, ever win an argument with an honest skeptic if you're keeping secrets. As for the dishonest ones? They're not your job. Mind your knitting.
FSJ on The End of CrunchPad
What I'm saying is, dont double your materials cost, thats not nearly enough. You need to add margins TWICE - two 40% margins. Thats 3x material cost, 4x is even smarter. Skimping on margins kills, you can -always- lower the price later.
The way you are saying it, is implying that all executions are equal. They are not.

(think of the biggest companies you know, and their idea was not original at all, just the execution was different. (better).

Before Google there were a myriad of other search engines. Before Facebook there was Friendster (should be classical case study for poor execution), Myspace etc).

From what I can say it seems that idea doesn't matter (i.e. doesn't have to be original at all), as long as the execution is good/better than everybody's else.

Sure, Friendster validated the 'social networks' idea, but it was ultimately FB that gave a really good execution to that idea.

So the difference between a 100mil, to a 10bill company, is simply execution.

You can say execution is a multiplier of an idea, and if the idea is stupid, it doesn't matter. But that doesn't make idea the most important thing. Most ideas are not that great/original anyways.
You can be a mom and pop burger shop, (easy), or you can be mcdonalds (not that easy), even if the idea is the same (sell burgers), it is the execution and scale of operation that wins at the end.
I also think what the author describes would ring true for writers (a different kind of geek), thus the market viability of writing focused writing tools.
I'm not sure if I want to say you are in the wrong major, but I think there is a current disconnect between what people learn in high school and what people learn in college.

High-schools need to start informing students that what ever their major is in college, it doesn't necessarily affect what they will do out of college. I know a lot of great programmers who were history majors, and a lot of tech industry managers who were engineers.

What matters in college is that you are taking classes that you enjoy. If you are spending $20,000+ a year to just get a piece of paper, you are wasting your time. you need to find something that makes you happy and increases your knowledge of the world. for me, that happens to be CS, but if you aren't happy taking CS courses, get out now! Find something that you enjoy and do it, you are wasting your time and everyone else's if you are in something just for the money.
The state of social media analytics
What would be interesting is to take someone out of a non-technical and very 'foreign' culture, like the Maasai, and have them help derive a programmatic problem solving methodology.

For example, give them a set of known classical programming problems and see what methods they use to arrive at a solution.

I'm not sure this would bear much fruit, but it might still be interesting.
http://www.wowcool.org

boot.ugg.jacket.jeans.sweart. (UGG)
jordan air max oakland dunks $39;
Christan Audigier BIKINI JACKET $30;
gstar coogi evisu true jeans $35-39;
coach chanel gucci LV handbags $36;

http://www.wowcool.org
EDIT: I see some comments that are already talking about huge data volumes.

To those guys, simply take a look at Google, or Amazon, or the credit-card processors. Lots of places deal with huge datasets in a peer-reviewed, transparent manner. Of course,use common sense here.
You've tried to take my statement and substitute in things you consider similar to make a point... Different fields demand different levels of transparency...

Could you explain why you feel climate science deserves low levels of transparency, and why conspiracies to delete/withhold data here are acceptable?

Regarding medical trials, they do not delete the data they process.

A concrete example: consider an fMRI study. The MRI spits out analogue k-space data. The MRI converts this to x-space data (pictures) which are then processed by whoever is running the study. It's legit to dump the k-space data (it's not part of his input), but he can't dump the x-space data.

In contrast, I work directly on the k-space data, so I keep it. The rule of thumb is "keep your direct inputs".

This is why I say the data was important - it enabled Mann/Jones/etc to publish important results. When (part of) it was made available to Steve McIntyre, it also allowed him to correct mistakes in the results of Mann.

Even then, would that constitute evidence to you? What's to stop a conspiracy driven group of scientists from simply fabricating realistic-looking raw data?

Unfortunately, there is very little which stops fraudulent scientists. Peer review is not really set up to do this. The best check is openness (release of code and data) and independent attempts to replicate published results. Unfortunately, the folks at CRU conspired to prevent this from occurring.

Econometrics journals seem to be taking good steps in the right direction:

http://www.econometricsociety.org/submissions.asp#Replicatio...
I'd be curious to hear any real-life stories behind slide #3 ("Patterns in shoelaces can carry messages").

Do they mean can as in theoretically? Because I could also theoretically carry a message based on myriad other things as well.

Seems like you'd have to get really, really close to a person to be able to read their shoelaces for the patterns in the second and third images.
The article seems not to differentiate hearing with parsing language (or, I guess, really lexing the language). AIUI, these are handled in very different ways by the brain. So it may be that eyes and skin don't participate in the auditory experience at all, but later become input to language processing.
Ideas are personal. I just said I am not impressed. Maybe I will in the future when I see what others have built.
> If the probability of producing the desired result
> in any given attempt is zero, then the probability
> of producing the desired result given infinite
> attempts is still zero.

This is a common misconception. Pick a point uniformly at random in the interval from 0 to 1. The probability of any given point is zero, but the probability of being in the interval [a,b) is b-a (assuming 0<=a<=b<=1). This is a case of infinitely many zeroes giving a non-zero sum, and why you need to deal carefully with infinities.

Yes, I know about countable and uncountable infinities. This is a big subject, and small comments here won't cover it. There are rules and exceptions.
I'm not sure if the NoSql movement is confused, but they definitely chose a confusing name. I was glad to see this blog post, because I do think it's useful to categorize some of the reasons people want to say "no" to "sql".

I'd add "level of abstraction" to this list (ie., people say no because SQL provides the wrong level of abstraction for a problem). You know you're getting old when when the "high level" languages of your youth are now viewed as "low-level" building blocks, but this has clearly happened in the SQL world. I initially resisted hibernate, JPA, active record, and so forth, probably for the same reason some C/C++ programmers resisted Java's memory management - it felt strange to give up this control. Now that I've gotten used to these tools, I'd never want to go back to hand-written SQL for basic object persistence and relational mapping, and I'm more than happy to use JPA or ActiveRecord to automate my CRUD. In this sense, I'd say "NoSql" for the same reason I'd rather have garbage collection than malloc or the pointer asterisk.

But I do see huge and fundamental memory mistakes made by people who use ORM abstractions without understanding SQL (such as selecting all into an arraylist and filtering through code rather than using the where clause, or generating a list and using each element to generate a new list, rather than using a JOIN).

It could be that some of these folks never encounter a data set large enough or a relation complex enough to need SQL, or maybe SQL is actually not necessary to manage relations in huge, intertwined data sets, or maybe you can avoid huge, intertwined data sets through careful planning. But for me, this is where the "NoSQL" thing breaks down, and I definitely haven't heard a whole lot of discussion about it.
Cute, but wrong (as you probably know).

Even the size of the internet and all the drivel on it is infintesimally smaller than infinity.

Ron Graham once told me that the problems we have in mathematics are caused in part by the fact that we only ever deal with small numbers, and he's the one who used Graham's Number.

http://en.wikipedia.org/wiki/Grahams_number
"Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large."

There's clearly a disconnect between how science is performed, and how it is perceived. It's not clear that this is the fault of scientists. It's also very clear that certain political groups intentionally cultivate this misunderstanding, so that they can convince the public that an entire field of science is wrong.

"It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists."

I have unpleasant news for you, then: the "skeptical" side is composed almost exclusively of politicians and economists.

"This seems to be part of the problem. 'We'll use this data series for the 60 years that it agrees with everybody else, then throw it out for the 40 years where it doesn't, then not show anybody the 40 years we threw out'."

That's not actually what they did, and the fact that you don't understand the nuances of the subject goes back to what I was saying about spending more time looking for "smoking guns" than actually understanding the science. The tree-ring data wasn't omitted to make models fit better (that paper wasn't even about a model) -- it was omitted because it was unreliable data.

If you ask me, the tree-ring data should never have been included in that paper at all. And if it weren't, as far as I can tell, it wouldn't have affected the results -- except that there would be an obvious (but largely irrelevant) gap in modern temperature data -- and the "skeptics" would no doubt be attacking some other absurd corner of the climate change research.
Using WYSIWYG with predefined styles is more writing focused than messing around with TeX, I would think. I think the author seriously misrepresents the way Word is poised to be used today (could be that he's accurate about what Word was like back then, though).
The article's title is a bit unfortunate. The article is actually a long and very interesting summary of why the Java Virtual Machine is also a great platform for other languages. The title hints more in the direction of an anecdote.
Morphological computing
Why anyone would relocate to a country living in the dark ages with Medieval laws is beyond me. Commit adultry? Go to prison. Eat a poppy-seed bagel in another airport on your way to Dubai and leave a seed on your shirt? Same thing.

http://news.bbc.co.uk/2/hi/uk_news/7234786.stm
The world looks different if you're depressed
So I’m creating a list of things to accomplish in the hopes of defeating my professional burnout: Re-institute a workout and diet regimen

Yes, do this one. It works for me, at least. See also http://news.ycombinator.com/item?id=948481 for some possible biology behind this.

A rule of thumb I have learned to follow is that when I am in a crummy state, that is not the time to try to solve the problems I perceive. Better to try to make a transition to a better state, by whatever means, and then explicitly remember the problems and try to solve them when I'm in an energetically more capable place. The solutions that result this way are much better.
Google Website Optimizer will give you a plus minus rate for your estimated conversion rate, which helps you to at least estimate what the significance/confidence is.

For example, one of the pages in one of my current tests shows up as "Est. Conversion Rate: 17.0% +/- 1.4%".
Program Uses Interactive Genetic Algorithm to Help Witnesses Remember Criminals