skysight-inc/HackerNews huggingface dataset:
Hey everybody, fauigerzigerk sort of gets into this, but I just downloaded the dump yesterday expecting there to be a relatively straightforward way to parse and search it with Python and extract and process articles of interest w/ NLTK.

I'm not sure what I was expecting exactly, but it sure wasn't a single 40gb XML file that I can't even open in Notepad++.

Is my only real option (for parsing and data mining this thing) to basically set up a clone of wikipedia's system, and then screen scrape localhost?
Virtual Users/Domains With Postfix, Courier, MySQL, SquirrelMail (CentOS 6.3)
I think so too. But simple pages in this configuration just measure nginx performance (check the breakdown)—that's a good sign for the rest of the process.
3 reasons why web developers need to learn online marketing
What's so insane about it? I ask as I've recently been working with curl for webdav backup scripts and haven't had too many problems beyond fat fingers.
How many times does it have to be said? Mirroring is NOT a backup strategy!

The number of times I've seen some sysadmin(s) base their entire organization on this faulty premise is absurd. Mostly it's because they have decided that RAID 1 or RAID 5 should be a decent "backup" strategy, but then there are those who believe mirroring systems is how to do backups.

They never, ever, take into consideration what happens when something corrupts/is deleted/is compromised. Without a way of going back in time (i.e. an actual backup) they are forever stuffed.

Sysadmins: MIRRORING IS NOT A BACKUP SOLUTION. STOP DOING THIS!!!
Project Management Hacks – Choose the right approach
Could you mandate signed commits to keep track of that?
Why is this even news? This was announced long time ago and I thought it was fairly evident. Use personal blocklist, instead.
That's true and I'm pretty stupid (and french) :(
I was using chrome for many years, almost from the start. But switched to firefox a week ago, because my chrome started to feel more and more laggy, both on home and work computer. I'm not disappointed yet.
you cannot combine sqlalchemy with Cassandra, Cassandra can scale so well because it isn't relational.
well in that case i guess you could always do

var _console = console;
console = undefined;
It's beginning to bother me that 'Starup' is becoming more conflated with 'Web business' (as the OP does). Software may be eating the world but the scope for disruptive innovation is far larger and should also be highlighted.
No, universities don't do this (though they can consider other factors, including their own admission exams, but this is rare). This is actually a problem, because the matriculation exams are not the same in all German states, so students in some states have an unfair advantage in those cases where grades matter. Supposedly, in 2014/2015 the difficulty level of the matriculation exam will be standardized.

The problem exists because traditionally, grades had not been much of a factor in admission (outside of a few restricted courses of study such as medicine). German universities (prior to Bologna) were used to running an open admission process and instead enforced selectiveness by failing those students who couldn't handle it early on.
Bedeutungsähnlich oder nicht?
Agreed, if you are in control of the host OS then you might as well just run some unprivileged processes in some lightweight containers (LXC, or just user/network namespaces + chroot).

There are two situations where running applications directly on the hypervisor might be beneficial though:

- If you have to run multiple untrusted (and possibly malicious) tasks on same host. In this case virtualization provides better isolation/security than just containers or processes (which would be vulnerable to kernel exploits, or other ways of gaining root inside a container allows to take control of the host)

- If the hypervisor is fixed and you can't touch it (i.e. Amazon EC2). In this case running applications directly on the hypervisor might eliminate some of the virtualization overhead.
If I can make a suggestion, build an official iphone and android app for this, and charge 99 cents.

You can then add the ability to do live annotations to presentations from the phone/tablet and keep that exclusive to the app.

This whole app is brilliant, I hope it really takes off for you and you make a metric fuck-ton of money out of it. Superb idea.
Idea is good, but realization is far from perfect.
you should blog about this... create a unique user and write about it.
No. Signed commits sign the commits, and because git is a DSCM, the person who created a commit doesn't need to be the person that put the commit into the repository (in fact, git further differentiates between author and committer, which is useful for "who wrote the patch vs. who put it in a commit object", but that's in the repo data).

This is in fact the main selling point of DSCMs - everything's a repo, repositories can (provided access) exchange data bidirectionally, and data can take all sorts of interesting routes from A to B involving as many repositories inbetween as you'd like.

Negotiating the actual write access to a repo happens entirely outside of git. That's really a good chunk of what software like Gitorious, GitHub or gitolite do, and then on top of that you get into the collaboration features and stuff.
It's not your only option. You can open the XML dump with a streaming XML parser (not a DOM parser) and use one of the existing wiki syntax parsers to extract what you need. If you just need a few specific items (for instance just the links to reconstruct the page graph or just the info boxes) that's a perfectly workable solution. There is a large number of small tools and scripts that extract various bits and pieces from the XML dump. You may well find a tool that suits your needs.

But there are two issues: The available parsers are not very robust and not very complete, because the wiki syntax is extremely convoluted and there is no formal spec. Second, the wiki syntax includes a kind of macro system. Without actually executing those macros you don't get the complete page as you see it online. The only way to get the complete and correct page content, to my knowledge, is to install the mediawiki site and import the data.

If you just want to look at the XML dump quickly you can use less or tail.
They were autosyncing. They were prepared for not being able to sync (restore main system and sync back), but not for the sync to replicate a corrupt state.

Any backup strategy with only one backup (instead of preserving historical snapshots) is vulnerable this way.
True, but the Erlang VM is not really written for fast startup. Module loading is serial and not parallel. The key here is that you can scale up in 0.6 seconds, which is awfully much faster than most other solutions, where scaling takes minutes.
>And then you get six different package managers on a single system for Ruby, Python, Node, Perl, X and Y?

I guess it would be better to have a cross-language VM - something that the CLR and the JVM are trying to do. Or it's possible to treat virtualization images (e.g. AMIs) as your execution environment and use a package manager inside that (although there are still no good package managers for C in terms of handling dependencies on conflicting versions of the same libraries). The point is that the "native" system is too inconsistent.

>Also, how would you specify a cross-language dependency, for example from a Python application to a Ruby application to a Ruby library?

I don't think that actually happens; you can only have a dependency where you have a common interface. If you were running the programs in Jython and JRuby you could have such a dependency, but in that case you would be able to use a JVM-oriented package manager for both.

You're right though; there's no reason a package manager should be language-specific. A virtualenv/bundler-like piece of software that managed the installation of libraries in any language would be a very good thing.
Isn't boredom just response of a body that what he is doing is not effective? So it encourages going doing something that's more effective, efficient and productive.
GSA breach highlights dangers of SSNs as IDs
Thanks for the advice. I'm taking a look at Selenium now.

I appreciate the plug for Feedity as well. Unfortunately, I don't think it's the right tool for my needs. A lot of the resources I'm looking for are not current, and once I've done the research, I don't generally have to research a company or person again.

In the end, I decided that (for my purposes), Google's custom search engine would suffice. It's not a perfect solution, but it seems like it will do the trick for now.

Thanks again for the comments.

Cheers,
-Sean
The git issue (if it can be reproduced and validated as true) is an interesting aspect of the story.

For admins there is another interesting one: they artificially generate an extremely vulnerable spof - and after this desaster hey are still doing it.

Can you see the mistake?

Hint: if you have many things, why do you make it one thing?
Separated by a common language
I recently was on a long solo hike and my smartphone ran out of batteries. With no music or anyone to talk to I had to find something to think about for the 2 hours it would take to complete the hike. I ended up going through each year of my life and thinking about where I lived and what I was doing at the time and memories associated with each time and place. It was actually quite entertaining.
It feels strange to me that so far only one commenter here (ok, he's at the top now) has identified where the main mistake seems to be:

A SYNCHED COPY IS NOT A BACKUP.

This includes:

- git

- svn

- Dropbox

- RAID (of any kind)

If you don't believe this, please reconcile everything you've learned about backups. More specifically, if you treat any VCS or dropbox as your only backup system, STOP RIGHT NOW and at least get something that's intended to be a backup, such as SpiderOak in backup mode.
Time Management Software - Efficient Tool to Eliminate Non-Productive Activities
Out of curiosity, does the QR app you used load webpages within the app?

Mine (RL Classic) asked me if I wanted to open the webpage and launched Safari.
Some countries invest in education others in armed forces and tax reductions for the rich.
I've only skimmed through the articles so far and they seem really interesting. And your comment ought to be the topmost comment :-) but people won't have time to read the three articles and upvote :-(

Anyway I updated the article I wrote with a section "This problem elsewhere, and solved?" that lists your links (and said thank you to you).

May I ask, how come you knew about how search engines function? (For example, do you work with developing search engines or have you studied them at University?)
That's good to hear - my discipline, biology, isn't quite so forward thinking.
This is of course well meant and I think it is good to remind people, that there is no need to use Ember Data in Ember, but I have one objection:

The example only fetches records from the API and never creates, updates or deletes any. This is actually something that can easily be achieved with Ember Data today and in fact it is the one thing that is well documented.
It doesn't sound like VAT to me.

Still, VAT is a taxation scheme that actually makes some sense. Is it really hated? Why would it be hated?
> edit: sorry, is that your post/do you have more insight? In that case I'm sure you know better than me speculating on what the author meant ;-)

Nah, I'm not Jeff. I have some general insight because I was involved with setting up our git infrastructure in its early days, but I haven't worked on the mirroring code, and I've been out of the loop on day-to-day admin operations for a while, so I can't comment on the backup schemes that may or may not be in action on the servers right now.
The number of commenters that think the KDE sysadmins were stupid enough to not know that "raid isn't a backup strategy" is depressing. You either didn't read the article or you haven't understood it.

Git repositories contain redundant information to perform consistency checks. If a bit flips in one of your repos, git fsck will immediately catch it. The KDE sysadmins thought these consistency checks were triggered when keeping a repository clone in sync and thus any FS level corruption would be caught at the first subsequent attempt to sync.

If you think "Gee, if I know this, surely they do?!" then perhaps the answer is: "They probably do and this issue is more subtle than you initially thought. Reread carefully before implying how dumb others were."
Shocking news: people keep on wanting to compare the two countries when public education is a state-matter, in which case Oregon and Finland is very comparable with regards to geographical and population size(I guess not comparable in temperature, though).
Show HN: Guitar Hero style game to be played with a real Guitar on iOS
EC2 + S3

Minimal charge to cover the bandwidth and hosting costs. Any profit donate to Wikipedia Foundation.

Solves that same problem for lots of people.
It might crash on iOS 5 for now. Update has been pushed, waiting for approval.
Donglegate: Why aren’t we questioning what Anonymous did?
Link to the demo: http://www.youtube.com/watch?v=yIq9wSGh9HU
The mirroring they were using is explicitly meant to be a fast local ad-hoc clone that doesn't do integrity checks.

They used the safe version before, because they were running into problems with the integrity checks, i.e. ref deletions and non-fast-forwards.

What they should have done was to write a hook or a script that did those non-safe updates manually (maybe only for some repositories, and some refs, don't want to rewind e.g. master).

But instead they completely bypassed the safety mechanisms and got screwed by corruption.
Now it's live, go ahead and try it for free here:
http://drawmusicwith.me
Let me give a little perspective on the issue. Linus lives in Dunthorpe, which is one of the Portland area's wealthiest neighborhoods. Back when I was in high school, of the Dunthorpe kids that went to public schools, half went to Wilson High School and half went to Lake Oswego, both of which good schools. Then the Oregon legislature passed some law that required that primary schools feed into secondary schools in the same district, so Dunthorpe renovated an old grade school and made Riverdale High School. It's tiny - I just checked the Wikipedia page [1] and it only has 131 students and Riverdale is the 3rd wealthiest school district in the country (who says Open Source doesn't pay?).

I mention all this because as far as public schools go, it's an outlier among outliers. So comparing it to schools in Minnesota or elsewhere goes beyond the normal case of poor school districts vs. wealthy ones.

Plus, in Oregon at least, school funding does not correlate well with student performance. The Portland School District has higher per pupil funding than most of the suburbs, and yet it consistently under performs many of them.

[1] http://en.wikipedia.org/wiki/Riverdale_High_School_%28Portla...
Sorry but I think you missed something. You should have let the emptiness and the boredom fill your self instead of proactively and orderly going through the years of your life.
There are several ways to deal with it.

Basic:

1. SSH - security related - modify sshd config - no password auth, only key based, no root login, other than that the default sshd config for Ubuntu 12.04 is pretty ok.

PermitRootLogin no

If you do not use IPv6, you can disable sshd to use it.

#ListenAddress ::

ListenAddress 0.0.0.0

2. SSH - ease of maintenance related - change default port from 22 to smth else. This will not help against targeted attacks, but this will reduce the noise in the logs and will allow better visibility of attack attempts.

#Port 22

Port 63777 (or whatever)

Don't forget to reload ssh for changes to take effect and check with netstat what's running where.

If you are concerned with getting locked out you can do this:

2.1. enable sshd to listen on multiple ports simultaneously

Port 22

Port 63777 (or whatever)

2.2. Login through Port 63777, and only then disable Port 22.

3. The PRC thing - fail2ban and hosts is simple but not the best tools for the job.

3.1. You can disable access to your site for all China (or any other country for this matter) using ipset. Much better speed and ease of maintenance, it's even better then using iptables itself [ipset is iptables module], one set can contain/block up to 65000+ IPs.

3.2. If you want smth more targeted, then consider either sshguard or psad. They can be configured to block inline and dynamically add rules [perm/temp] to iptables.

Edit: forgot to mention another cute recipe. see below

If you are on AWS and are using security groups (you should), after you are done on the server, you can go to the security group in AWS Mgt Console and remove ingress for ssh ports. Now your server is accessible only on 80/443 and ssh is NOT accessible to anyone. Later, when you need to access the server - enable ingress for the ssh ports, do your job and disable again.
https://news.ycombinator.com/item?id=5397629
For the first case, I'd rather use something like Google Native Client.

The attack surface of a VM is rather large, particularly when you consider the dozens of emulated devices and instructions. For example, there was a notable hole in SCSI emulation recently discovered which let any guest overwrite any part of a host's disk (by sending some obscure SCSI commands which weren't being filtered by the virtualized device properly).
Balzac is a better example, he had a frivolous mother and was left alone in a room most of the time. He started telling himself stories and became expert in imagination.
whathappenedto also suggested the article you linked, in this reply: https://news.ycombinator.com/item?id=5430495
(the "Dynamic bayesian network model" link).

The approximative solution at the end of the article seems really interesting :-) It'd feel better to implement something that is theoretically sound. I suppose the algorithm could need some tweaking, since a discussion is a tree or a graph, but a search result listing is... a list.

Generating a slightly different order for every reader also seems like a good idea, and fairly easy to implement. — So as not to make people confused when they reload the page (by shuffling comments around), perhaps one could use the user ID or IP number as random seed.

(May I ask, how come you know how search engines solve this problem?)
World’s smallest blood monitoring implant helps prevent heart attacks
yes, very soon we will start to give attention to writing application over this platform. there is some work still needs to be done in management stack, but it's manageable amount of work.
Work or start your own business?
Hi everybody,

I have studied computer science, and I have worked for 2 years. And now, I have changed the country I live as I get married. I'm in Qatar now, and I have been there for 2 months. I couldn't find job ( to be honest I didn't look for it seriously as I didn't know many people in Qatar, and unfortunately the only 2 interviews that I took were not successful because I haven't worked on the website too much) sometimes I think that to start my own business working in the internet rather spending too much time applying for jobs that will not come.
Really I don't know what should I do!
And where do you come from?
Wow. I really appreciate that!

I think Go, as a language, makes it much easier to write code that is both well-structured and readable. Of course, I have consumed a non-trivial quantity of the kool-aid, and I am convinced by the arguments that Rob Pike et al. made when outlining their design choices.
I had to deal with Meltwater once, and formed the view that it was a screamingly expensive news monitoring service that was easily replaced by a handful of free Google News alerts.

Sadly, management took the view that a subscription would be a set-and-forget way to tick the box on "staying abreast of the market" so I just had to suck it up and renew the sub...
Sorry gentlemen, we had used libvirt and it set some limits. http://erlangonxen.org/blog/glimpse-truly-elastic
> Linux runs everywhere at SpaceX,
> he said, on everything from desktops
> to spacecraft.

Yet every time I've seen them advertise for sysadmin sysadmin or programmer positions the indications are that at least all their internal network and website run heavily on Microsoft software, but their actual spacecraft use Linux/embedded.

It would be interesting to get an actual overview of what they use Linux for, and what they don't.
Hmm. Going from how commits are formatted to include signing, I think it should be possible to strip signatures and re-add them. This would create a new commit object of course, so if you had multiple commits that needed to be pushed you would have to actually need to re-write the rest of the commits to have different parents...

It may not be kosher, and I don't think the porcelain supports it, but I think you could write a remote that only accepts commits if they are all signed by someone on a trusted list. This would probably mess up a lot of workflows though, since if person A writes some commits, person B rewrites them to sign them and then pushes them to the repo, then when person A pulls the commits "they" created they would actually be different commits.

If we were willing to take that hit though, I think it should also be possible to modify git to permit multiple signatures on a single commit. Users could take a signed commit, append their signature too it (well, technically not append. Signatures are in the middle it seems) and then push it or pass it on to others to sign. This would only allow a single commit to go through the system at once (since commits after the first would need to be rewritten, invalidating previous signatures) but on the other hand it would let you create an integration server that only accepts commits that have been signed by some number of trusted code-reviewers, while still keeping all of that information in git. So long as trusted code-reviewers all signed off on it, who actually sent it to the integration server probably would not be particularly important. Such a modification to git would probably be extremely un-kosher of course...
An amazing thing about git (and other DVCSs as well) is that even if a much more serious catastrophe had happened (e.g., if a nuclear bomb had struck the KDE datacenter), it would probably still be possible to reconstruct (an approximation of) the master repo, simply due to the fact that it was fully cloned on hundreds of developers' machines worldwide.

Linus Torvalds once coined an adage that "real men don't make backups. They upload it via ftp and let the world mirror it." Well, the FTP bit isn't true anymore, but otherwise DVCSs have enabled this for mere mortals.
> I don't think that actually happens; you can only have a dependency where you have a common interface.

There is always the OS as a common interface. For example, I don’t see why some application should not require a mailserver to be installed, i.e. a ‘mail’ binary to be present, even if said binary might be a native C binary rather than a Python script.

I guess it all depends on what you want to do: If there is little to no interaction/overlap between applications, either because you want to keep them separate anyways[0] or because there is only one application running on a system, deploying a VM for each application makes perfect sense. On the other hand, if I had to install every library required by each application separately for said application[1], my system that currently fits nicely into 6 GB would probably require at least a few gigabytes more, and likely also more RAM.

[0] Because you don’t necessarily trust the authors, for example, as appears to be the case with all the Markets/Playstore/insert-fancy-name-here.

[1] This btw already exists, it’s called ‘static linking’.