skysight-inc/HackerNews huggingface dataset:
This got me thinking: The obvious way to do full-text search in the post body would be to build an index of {"word": [post_id, post_id, post_id]} of posts it appears in. However, this could be huge.

Does anyone know if there's a technique that uses a list of posts and a bloom filter that contains all the words in that post? I.e. you iterate over all the posts and check the bloom filter for membership of all the terms.

Since the number of words is (probably) much greater than the number of posts, you just need to loop a few hundred times at most, and you gain a lot in saved space. Also, since a Bloom filter has no false negatives, you are, at least, guaranteed to find all the posts that mention the specified words (with maybe a few "junk" ones in between, but which should be easy for the reader to filter out).

You can't do weighting with this technique, but it should at least be a quick way to figure out which post IDs you want to show.

Does anything like this exist currently?

EDIT: Here's a quick proof of concept: http://nbviewer.ipython.org/gist/skorokithakis/d115ab734d9ad...

It works fine, but the filters are a bit large (2 KB each), so I'm not sure how much space you save.

EDIT 2: This was so much fun that I wrote it up: http://www.stavros.io/posts/bloom-filter-search-engine/
Very good article! I remember I became obsessed with 3D rendering in high school 6 years ago and actually wrote a little photon mapper myself (I also remembering re-writing the Wikipedia article on photon mapping at one point).

I always had the goal to turn it into something really useful but once college started that project kind of ground to a halt. Nowadays if I were to do it again, I'd just do Metropolis light sampling because I much prefer the unbiased nature of that global illumination solution than photon mapping's approximation (meaning if you average thousands of results from a MLS render it will converge on the exact solution to the rendering equation; this is not the case with photon mapping).

I've always wondered though if someone has found a solution to the pinhole problem: imagine you have a scene that contains a box. Inside the box is a very bright light, and outside of the box it is pitch black. Your camera is pointing at the box. If you rendered the scene, your resulting image would be completely black. Now imagine you poke a small "pinhole" in the box. In real-life, your resulting image would be well-lit because of the very bright light now bleeding out of the box. But none of the rendering techniques I know of would be able to "find" this pinhole in a reasonable amount of time without manually provided additional information. Any rendering experts know the solution to this?
4 simple steps to vertical rhythm
Not really security, more like spam prevention and idiots clicking links.
> Prices going down meant I was able to buy more things at more shops.

Didn't your wages go down as well?
Well, anything that might result in fewer BTC posts in HN is fine by me, for one...
Except creating something that looks like HN does not automatically attract the same crowd as HN. There are plenty of clean, meme-free, well managed Reddits. This new site could just as well turn into /r/bitcoin.

Also, anything bitcoin related is going to turn into a "circle-jerk". That's the point isn't it? They exist to educate, market, hype-up, and encourage everyone to use BitCoin. No thanks.
"while for startups it is encouraged"

It's PG's advice, by no means universally shared in the investor community. Several investors I know will immediately go into a defensive shell as soon as they hear the "X for Y" construction. If you're a budding startup, please do your homework and know your audience before deciding the main tagline of your pitch.
Well I also disagree that the New York Times writes trash, either.

Yes we're all primates. Yes you can usefully analyze a lot of behavior in terms of tribes and regions, and who is the current or former alpha male, etc. At the same time, you can make an effort to transcend that, with at least partial success. An organization like the NYT exists to try. They can never be 100% successful. But I think it's missing the point to fling feces at them, saying, "Hey, they're just flinging feces." O_o

The article has a valid point. What some of y'all need is the Uber of Ubers -- an app that geolocates the nearest business, and spits out a business plan for Uber-izing it. Which you can run down to the average investor who wants to hide in the middle of the tribe/herd.

Obviously not all startups or investors are that unoriginal. pg has a great point that some are original, but need to seem less so -- to simplify their message -- for the average bear. Even so, many startups are yawningly copycat. And some folks doth protest too much when it's pointed out.
But that is generally what these large studies are finding.

Interphone study says no risks? The results are far from conclusive.

http://interphone.iarc.fr/interphone_results.php

http://www.ncbi.nlm.nih.gov/pubmed/20483835 - "CONCLUSIONS: Overall, no increase in risk of glioma or meningioma was observed with use of mobile phones. There were suggestions of an increased risk of glioma at the highest exposure levels, but biases and error prevent a causal interpretation. The possible effects of long-term heavy use of mobile phones require further investigation."

This conclusion has sentences that contradict each other. E.g. Heavy long-term use has higher negative numbers, compared to the "normal use" that causes "no increased risk" ?!
Luckily insurers have begun to find a way around this by selling bulk claim data to third parties who can analyze them and extrapolate the contracted rates on their own (a la Castlight, Change Healthcare, etc.).
What do you mean by things getting interresting ? Did I miss something after 2005 ? Do you have some reference on the web ?
On the same day, we get a filter that is by default HN without Bitcoin and another site that is HN only about Bitcoin.

It's like we're approaching some sort of binary tree data structure where the leaves are HN clones with their own indiosyncratic topic configurations.
I don't believe in unions as the ideal situation for workers or employers, I believe strongly when a company ends up with a union, it's because they did something stupid.

Didn't pay enough, treated employees unfairly, poor working environment, any number of things, but generally something that could have been mitigated by 'doing the right thing' in the first place.
#2: I happen to believe that people should be entitled to education, healthcare, safety and respect.

#3: There are certainly bad jobs. Jobs where the employer does not pay, abuses their workforce etc.

#5: Debt you cannot afford? Sure. But sometimes debt is useful. A mortgage or a business loan for example.

#6: Safety is everyone's responsibility. It's all well and good that you behave well but if Bill decides he doesn't need to follow safety precautions it can just as easily injure, maim or kill others as well as himself.

#7: Don't I regularly see "Work smarter, not harder" plastered over HN? If the only way you can stand out is by working more, you're either at the top of your game or you can do whatever you are doing better.

#8: "The machines here are not maintained correctly, but I guess complaining is bad so I'll either put up with it or leave."

#10 (ish): Car accident breaks both your legs, you can no longer perform manual labour. Tornado destroys your business, insurance shrugs and pays you a pittance. Someone else's bad decisions cause an equipment failure that costs you an arm.

#11: These two sentences are not equivalent. I'm okay saying I agree with the latter, but not the former. I'm not okay with the fact the world is not fair and feel it's a noble cause to try and correct this.

#12: This just reeks of the blame the poor mindset I see in a lot of US conservatives (and in our own UK conservatives to be fair).
"Here’s to the crazy ones. The rebels. The troublemakers. The ones who see things differently. Please do it somewhere else and just give us the benefits once you are established" - Steve Jobs (the quote usually truncates the last sentence)
HPV is also an oncovirus, responsible for cervical cancer and very likely many oral, throat, and esophageal cancers. And there's a vaccine for it.

[Edit: should be noted however that the vaccine is for the virus, not from the cancer that can arise from its infection.]
If an employer is dumb enough to not hire a great worker because of their gender or skin color, then that business is doomed in the long run anyways. Because somewhere out there is a competitor who will snatch up that talented person and crush them.

Capitalism is a brutal business, but it does have a way of sorting these things out.
From my "big ideas" evernote:

- drones - They will be like servers for the physical world. Business opportunity is "ec2 for drones" -- drones as a service, rent by the hour, remote controlled from a web interface, programmable with API.
- bitcoin
- privacy
- transparency
- online household (IoT)
- brain-computer interfaces
Ask HN: What to do & where to stay in San Francisco
Hey all, my girlfriend and I will be in San Fran for new years eve and about a week after that before heading home from our around the world trip (5 months).

Thought I would see if I could get some thoughts on what we should do and where we should go/stay while in the area from the HN community as I read here often.

I'm hugely interested in the start up space (I work for a small, growing company in Australia) and love chatting technologies and new businesses etc. so I like to hopefully meet some people around that, either randomly around or at a networking event if there are good ones?

Other than that we both love good food, boutique beers (mainly I love these) and tea.

Is it worth staying in the city area or in the valley? Or somewhere else?

Thanks in advance guys and girls

Cheers,
Luke
Most frameworks protect you against CSRF, that is not unique to Haskell. Other languages also allow you to check your work using tests and static checks.
Are you sure Carnivore and DCSNet are presently related? I had thought DCSNet and ISP-bound traffic monitors like Carnivore were separate.

That's interesting. Though I had always been stuck with the impression that FBI technology primarily made use of Windows as an OS and Perl as a scripting language.

I meant legitimate as in lawful.
It is not just you.
It needs a battery no matter what. It assuredly has a battery. How would it function at all without a battery small enough to power a wrist watch?

How much battery power do you need, in order to pipe readings from a sensor into a tiny 2GB cache of flash memory?

Then, the data syncs with your laptop, because how else will you find out what the readings are, without some kind of GUI interface showing you some charts and graphs that interpret the raw sensor data?

The sync software will come bundled with a "share your data, oh goody!" feature that defaults to constant sharing mode (dark pattern). You charge the battery whenever you sync. In the presence of an internet connection, it uplinks to MsSmartBra.com, where data is monitored and tracked so everyone knows your high score! Yay!
> the number of immigrants doing these same hard work in unpleasant conditions while native-born "modern Americans" moan about being unable to find work.

Are jobs that pay less than minimum wage even offered to citizens or documented immigrants?
I think people who make these arguments drastically underestimate the challenges faced in coming up with a single cure-all for cancer.

That said, I am very sorry to hear about your wife's condition and I hope that she comes out of it fine. That must be an extremely difficult situation and I'm not sure if I were in your situation I wouldn't have a similar opinion.
Quick, someone write MetaNews: A hacker news for "A hacker news for X" websites!
How To Write Unmaintainable Code (Ensure a job for life)
A lot of firsts on this for me, including the first time I've really used github other than just git clone.

I wrote this because I tend to write and rewrite queries a lot, and a lot of times ORMs get in the way.

Admittedly, I have different use cases, as I work in tables with 10 billion rows and, more frequently, tables with 100m rows, so sometimes I have to wrestle out all sorts of performance. In addition to that, I needed to come up with a good way of merging data from geographically and system-diverse databases at runtime for what is effectively a dynamic metadata file system (large physics sets). This isn't always easy to do, although I'm looking into possibly using Presto instead of something like this.
The 80s are still alive
Seriously? This is how many studies are done, unless you can have a highly controlled environment say with breeded rat populations. And yup, wouldn't it be great to keep track of absolutely everyone - and then see how they progress? The problem is most of us are bombarded with a lot of pollution (not just electrical), which all will affect our body's ability to cope with dis-ease and sickness.

I'm not sure where you're getting the sample size of 14,000 ... though different pieces of the study had upwards of 5,000 participants.
Have you read the paper? What that means is: no statistically significant links were found. But because of bias in the study, it's possible that we're wrong about the highest end of exposure, so this should be studied further.

Claims that interphone found any statistically significant risk increase due to the use of cell phones is not backed up by the data in any way, shape, or form. (But this, of course, doesn't mean there is no risk, but that we haven't seen it. Future studies could find such a risk, as unlikely as that may be.)
This is right up my alley. However, I don't think there are many developers creating applications that are mainly JS on the front end and .NET on the back end.

Most developers I know who use .NET use MVC for the full stack.
I think there are status pages for internal consumption and those for external advertising purposes ("See - we have 100% uptime over the past 30 days!"). I don't guess I understand why you thought all status pages would be public.
I think it's an equally valid technique in both industries. It's just been around longer in the film industry which has a similar pitch-for-play financing culture as silicon valley, except it's been that way for quite a long time (since the collapse of the studio system really) so there's been a lot longer of the ecosystem to develop a culture of fast-talking pitchmen who may or may not have any substance, but are certainly ripe for satire and mockery either way.
I've commented on this around here before. You read lots of junk about cancer "cures" in the popular press. They are almost always in mice or something and I or someone else with a similar background always feels compelled to weigh in and remind folks that it's a long way from curing lab rats to curing people.

This however, is the real deal. It's quite remarkable and there's likely more stories like this for other diseases on the way.

I went to a gene therapy session this fall at the American Society of Human Genetics conference in Boston and was blown away by some of the success people are having. I quipped to colleagues that I felt like I was in a science fiction movie. The most remarkable one was where they used an approach similar to the one here to cure a fatal metabolic disorder (relaying this from memory, so some of my recollection may be off). Kids with the disorder have a busted enzyme that causes slow degeneration of neurons. They don't live past 6 or 7 if I recall. The team showed how modifying a certain kind of stem cell found in the body normally to have the correct copy of the enzyme cured several patients. The corrected cells naturally move to the brain where they differentiate into glial cells and produce the correct copy of the enzyme. It turns out that because the neurons in the brain are starved for this enzyme, they express receptors that allow them to take it up from the environment. So the repaired glial cells supply enzyme to the entire brain (i.e. it's not necessary to modify every neuron in the brain to have a correct copy of the enzyme). They can completely cure kids with this approach. All of their muscular and neurological tests are 100% normal.

They had videos of these kids running around and playing just as if nothing was wrong. In one case, a younger brother lived but his older sister (who was too old when the therapy came out) had died. It was hard not to get choked up looking at their smiling, happy faces as they ran around, thinking that if this therapy hadn't existed, they would be in a nearly vegetative state.

Gene therapy had a rough start with the early setbacks, but I'm getting the sense that the tide is rapidly turing.
It'd be cool if some serious makers made a LFTR reactor :-)
No more “this is not available in your location.”! I have been waiting for that.

Just got picked up by TNW - http://tinyurl.com/bopfm1
That's reddit, really.
This is indeed how many studies are done. It's also why many studies are 1) completely worthless, and 2) impossible to reproduce.

This is simply not valid science. It's not blinded, it's not representative of the population at large (or these would've come out of a randomly selected population), and it's not properly controlled. It's junk science.
I'd argue against there being "only 1 heap". Only one OS provided heap, sure, but in C++ it is quite common to overload new and allocate into a separate statically declared buffer.

(See: Video games!"

And, as usual, no mention was made of static allocations! :(
I've read some criticism of bitmessage sometimes, but it's been quite scarce (as with all interesting feedback) as of now...

care to link to the bug report/forum post/blog post where you wrote the security issues you mentioned?
Not sure if upvote = tips is a fantastic idea or a way for trolls and dramatic people to make a living.
It's really not much like Lode Runner at all. Lode Runner itself was a "clone" of Space Panic much more so than this is a "clone" of Lode Runner. In both cases, there is clear inspiration drawn from the earlier game, but they are very much original works.

I wrote a shareware Lode Runner clone on the Amiga 20+ years ago. Mine was definitely a clone, no bones about it. Spelunky is barely related.
Do you know of an operating system to build personal robots?
Are there any robotic operating systems that make it easier to build personal robots. Search engine seems to think ROS is the way to go.

http://www.ros.org/
The two are, of course, motivated by the same plea: for HN to be only for discourse about technology/hacking/building/entrepreneur-ing itself, and to leave the discussion of the politics surrounding it to somewhere else.

I mean, we all want to discuss techpolitik sometimes... but generally not at the same time we're reading about new tech.

HN might be well-served doing something like MetaFilter's "blue/grey" split, but instead of just a content/meta division, a "hacks/sociopolitical-implications-of-hacks" division. HN is already a community, and it'd be nice if the account names and karma stayed synchronized.
How is this going to solve anything? All this is going to do is bring in a flood of applicants to the job that has no listing of a college degree as a requirement. Then HR is going to reject a bunch of applications. At the end you are just moving the barrier to an internal portal.
That would be a large reversal from the current situation. College degrees are not officially required for things like immigration or proving candidate qualifications in discrimination lawsuits, but they are generally accepted as a reliable standard. This means that if you don't require a college degree, you have to do a lot more work to define qualifications and prove they match.
This is incredible. Mike Rowe has a blog and he apparently understands economics (based on first answer.) once again HN has made my day.
Why Our Brains Make Us Click on Lists
You sound like you have a depression, which is completely understandable, since you can not in any way help your wife. There are good reasons why every cancer can not be completely and effectively cured in every case. But it will be cured, it's just a matter of time.
Huh? I've got plenty to spare. I was referring to the parent.
It would be helpful to know where you currently have citizenship. That may limit your options more than the lack of degree.
Freeman Dyson's book, Disturbing the Universe, which is quoted in this article, is a great read. Highly recommended.
> Because somewhere out there is a competitor who will snatch up that talented person and crush them.

Empirically, this is not always the case.
Ironically, there is already a HN subreddit.

But it's join-by-approval only and there's nothing there.
I shall call it a subreddit.
I don't think so. From my understanding he is talking about a HN clone that only allows posts about new HN clones.
The community has become more complex than the structure of Hacker News allows for.
This explanation is pure genius. Instead of explaining something very complicated, explain how to arrive at the complicated solution by evolving the simple solution and eliminating its flaws. Wish more explanations were like this. Someone do git.
Just one of the problems with kids these days, don't have any 'ppreciation for the skill and craft of building a hobbyist nuclear reactor in their parents basement.
Wait, I thought this was called "the HN front page". /snark
I guess the name is a reference to Trainsportting? Great domain name in any case.
Google’s $179 Moto G puts every single cheap Android phone to shame
Well. It's near the top of HN right now, the server's probably getting hammered.

Also.... maybe not:

Server: Apache/2.2.24 (FreeBSD) mod_ssl/2.2.24 OpenSSL/1.0.1e DAV/2
Server: Apache/2.2.25 (FreeBSD) mod_ssl/2.2.25 OpenSSL/1.0.1e Phusion_Passenger/4.0.23
OS X uses portions of FreeBSD including the userland so if you're used to that you probably won't find too many gotchas when using FreeBSD in that regard. I've never touched a straight *BSD install to know the practical differences in configuration and maintenance vs OS X and Linux.

I occasionally get caught by a difference between the BSD and GNU userlands so I've often contemplated giving one of the BSDs a go rather than my typical OS X/Debian/Ubuntu Dev/prod web server preference.
I'm not an expert, just an interested layperson, but from what I read, I am starting to wonder whether cancer cures might be more basic than we realize, we're just not looking at the right things.

The example that keeps coming to mind is HPV - it can cause cervical cancer etc. And, since there is an immunization for HPV, the disease is preventable, and to the extent it is the "only" cause of certain cancers, would also prevent those cancers.

So is HPV the "only" virus which causes cancer? Seems unlikely. I just don't know how much research is focused on bacterial or viral origins. Maybe it is not glamorous or mysterious enough to justify funding.

The 2005 nobel in physiology went to Barry Marshal for discovering that stomach ulcers, long a mystery, were caused by Helicobacter pylori bacteria. And in fairness to history, he wasn't even the first person to discover it, just the most recent person in western society.

http://westhunt.wordpress.com/2012/02/22/low-hanging-fruit/
How to make $800/mo from three lines of code

The title was just link bait then?
2006 is commonly cited as the year when "deep learning" started becoming practical. Pretty sure it was Hinton's group, but they used greedy unsupervised pre-training to get a good initialization of the weights, then followed by supervised finetuning of said weights. That result kicked off a lot of renewed interest in NNs, which then led to using GPUs for a 40x speedup, which then led to many more impressive results (and they just keep coming). It turns out the unsupervised pre-training isn't even necessary, go figure...