skysight-inc/HackerNews huggingface dataset:
Ask HN: How do companies like Moz get search data without getting banned?
I know that Google bans scrapers outright and anyone trying to shadily get around the system gets caught at one point or another. So how is it that large companies like Moz, Ahrefs, and others are able to get all the data they need, quickly, without getting banned, and I swear that I've seen Matt Cutts even link back to Ahrefs in the past.
> I feel like I'm a different species when I hear someone excited about working 80+ hours a week.

Work smart, not hard. 80+ hours people are not the brightest.
I use xpath way more than I care to at my job but it get's the job done.
To remove all the non-porn content?
There's a Spanish initiative called Negobot that seems to rely on AI much more than Sweetie does — as its name suggests, it's a bot, not just a decoy. (Article in English: http://www.huffingtonpost.com/2013/07/11/negobot-virtual-lol... and paper: http://www.esi.uem.es/~jmgomez/papers/cisis12.pdf)
So, what are the chances the NSA doesn't already effectively own Bitcoin?
Glad to see they finally got an API, however I'm a bit disappointed that it doesn't return the oldest archived date for a site, only the newest. I often need to check how long ago a site was originally archived. The API would have been very helpful for that, but the closest they provide is an option to query whether or not it was archived on a specific date, which is nowhere near as helpful.
Chrome logos in CSS
Amazing technology in due in 3-5 years? Probably not.
Terre des Hommes has for years worked to combat child prostitution in Southeast Asia and staff members noticed in recent years that young prostitutes were disappearing from their usual haunts: cafes, restaurants and hotels frequented by sex tourists.

after reading this part, i too was asking myself if cracking down on 'webcam-sex' is the best idea, because if this phenomenon contributes to less girls and women being on the streets, that means that less people are going to be victims of related violence, sexually transmitted diseases and unwanted pregnancies.
Heck, the page has it half-written in the code for the "Save page now" button :)

document.location.href='//web.archive.org/save/'+$('#web_save_url').val();
Neat project and great post. One thing I'd like to point out is that there is another mechanism for implementing userspace filesystems in Linux: V9FS. FUSE has a lot of documentation -- much of it in various stages of bitrot -- and many of the comments here point to other FUSE projects as well!

The underlying mechanisms for FUSE are pretty convoluted and difficult, so much more than trivial examples can break the abstractions afforded by the libraries like the one used in the post.

V9FS is in contrast exceedingly simple and basically unchanged. And, yes, it's based on the Plan 9 filesystem protocols :)

http://v9fs.sourceforge.net/
For good ideas for async stuff you should try emacs. You don't have to keep using it, just look at what it does with its built-in asynchronous subprocess handling. The interactive python shell is particularly neat, but it also has grep and compile functionality that doesn't block the editor as it runs, and a gdb mode that (based on my very brief go with it) seemed to work pretty well.

One neat thing I use it for is TTY capture from a target system (usually games console or embedded device) over a serial or network cable. Run your terminal program in emacs at the start of the day, and emacs will dutifully capture all the terminal output as you work. Search, save and manipulate TTY output using all the commands you know and love.

Well - except for `.'. Or `*'. emacs doesn't do those two...
Oh nice, looks kind of like http://imageoptim.com/

We are mostly using OSX, but it is really a struggle to learn people to run their images through an optimizer.
Was IBM's Watson a Publicity Stunt From the Start?
What does everyone use for a payment plan?
Re: the last part about etched semiconductors:

Keep in mind that the size of a magnetic grain (which stores 1 bit, roughly) is about 8nm. Compare that to the current cutting edge lithography scale of 14nm, which is the size of the semiconductor 'wires'.

Thus, a bit is smaller than the size even of the wires in any semiconductor circuit we can build, which means a system as you describe would have much lower storage density.
I read 'old' as 'middle ages old'. There was a time when torture was openly acknowledged as acceptable practice. At least nowadays it's widely recognized as a Bad Thing, even if governments are willing to engage in it under the radar.
The fruiting body of a fungus is typically a reproductive mechanism. Bigger mushroom = more spores (ie. size does matter).
Interesting, (and this could just be a made up problem to illustrate the blog post), but wouldn't it have been much easier to just store the lyrics in another format (not HTML)?

For example, you could use TEI XML (http://www.tei-c.org/index.xml), and then use stanzas and lines. Then when you go to render your lyrics, you can capitalize the first letters in your presentation code.
At our company, we've found that a cohesive design throughout a funnel will always* drive better results. If you think about google as a step in your funnel, then you'll be improving conversion through that same principle.

Also, you've guaranteed familiarity by using Google to buy users. It might be interested to buy users via Bing or Yahoo or something, to see if that same design wins.

But, that's purely gravy. You've probably hit a pretty good balance of testing and results with your experiment already.

* "always" -- not really, but usually.
That's a point of view.

From my point of view it is a very dependable text editor with lots of room for customizability and an unparalleled text editing paradigm.

How many ifdefs it has, the length of its source files or its use of pointers don't matter to me because they have zero negative effect on my workflow. I like Vim as it is, I don't feel limited in any way.

The impossibility to make a clock in a text editor is not an indication that it's broken: it's a sign that you should buy a wristwatch.
A grim as it sounds, I prefer to see online underage prostitution rather than real prostitution.

I don't know.

On one side, this behaviour is wrong and should stop. On the other, it let those young girl make money without the risk of getting sick, pregnant or abused. The worst that can happen is her getting scammed.

But don't get me wrong. Child abuse should NEVER be encouraged.
Pandoc handles this pretty well in all the scenarios I've used it in.
I just started getting into xpaths pretty hardcore with my trivia generator for http://playhattrick.com ... I use it for identifying tables of data to scrape. It's not as fun as regex IMO but it is powerful.

Pro Tip: the chrome inspector lets you right-click on an element and get its xpath.

Pro Warning: sometimes the xpath generated by chrome doesn't work when scraping with Nokogiri. I'm not sure why yet, I've just learned not to rely on it.
> No one was applying this past that to then come to the claim the pixel size changed at different degrees, which is the implication that the author of this article seems so upset about. Everyone knows the CSS pixel stays the same size across the entire screen, we don't need to discuss that. We certainly don't need enormous articles like this written to correct a problem no one had.

Actually I think this article was (re?)posted in response to another article[1] that made the front page today[2] which did claim that pixels were angular measurements. So certainly some people have had that problem

To be honest, since this article is from January this year and the other is (seemingly) from 2009 I'm not sure why it's come up today, but that's the Internet for you!

[1]: http://inamidst.com/stuff/notes/csspx
[2]: https://news.ycombinator.com/item?id=6668395
He means that what the torturers/enablers of torture did is worse than the crimes the detainees are alleged to have committed.
"The room is full. Please try again later."

You're gonna need a bigger boat.
No, you have to be on the losing side of a war to be a war criminal.
Whereas xpath... does not. Which is a severe understatement considering the equivalent to the CSS selector you wrote up (or to `.foo`) in xpath 1 is something along the lines of:

[contains(concat(' ', normalize-space(@class), ' '), ' foo ')]

the normalize-space can be dropped IIF you're certain all spaces are normalized, the spaces around the needle not.

xpath 2 does quite a bit better through `tokenize`:

//*[tokenize(@class, '\s+')='foo']

but still not great. And god forbid you need to match multiple classes in the same selector.

[0] or xpath 1 + exslt if your xpath implementation provides it. exslt actually does slightly better as the pattern is optional and defaults to whitespace characters
It came close. Shortly thereafter, slavery was replaced with Sharecropping. I don't remember much about that from my US History classes, but I'm under the vague impression that it wasn't much of an improvement.

Cherry-picked, out-of-context quote from wikipedia: "Though the arrangement protected sharecroppers from the negative effects of a bad crop, many sharecroppers (both black and white) were economically confined to serf-like conditions of poverty."

http://en.wikipedia.org/wiki/Sharecropping#United_States
Tesla Model X will be AWD only, will cost 'slightly' more than Model S
Which is supported by the administrations' insistence on calling it "enhanced interrogation". Newspeak FTW.
New Snowden docs show more detail on how NSA broke into Google/Yahoo
There Are Probably Way More Earth-Like Exoplanets Than We Imagined
2014 Franklin Laureates announced
Also, he's not a US citizen.

(Edit: error on my part. His passport was revoked, which is what I was citing, but apparently he remains a citizen.)
Perceptor: An artificially intelligent guided navigation system for WPF (2009)
Bitcoin flaw could let group take control of currency
They all face the same problem: a captured FCC. I agree on opening up spectrum (spread the gospel!) and wish we could adopt policies against the duopoly like open access to the wires. Yochai Benkler has written on both if you're interested (you're probably aware)
One problem with XPath is that it can be a lot slower than native or JIT'd code depending on the implementation. Interestingly enough you can do xpath like things in Scala with native code using pattern matching:

http://ofps.oreilly.com/titles/9780596155957/HerdingXMLInSca...
Very interesting, thanks for the in-depth explanation.

I think that this does appear to be a fundamental difference between Windows devs and Unix devs. All of the things you've listed there, I'd do through the GUI on Windows.

Searching, obviously, I'd Windows+E (or Super + E in OS non-specific keyboard terms) to open up the explorer, then tab+tab to navigate to find, then type the query. I can't think of an easy way to pipe a set of files into some sort of function (e.g. for substitutions) outside of F#, maybe something exists for that in PowerShell, but I find PowerShell to be a bit of a poorly-documented mess.

Source control, again, I'd do through the GUI. With TortoiseHG, I've got my hg commands available for me with the right click of a file/folder under a repository. I see that it seems that this'd be significantly slower than doing it through command line, but the windows shortcuts actually make it pretty snappy to navigate through the file system in the GUI.

For kicking off a VM, I'd usually attempt to shortcut the command I'm looking to execute regularly and then Windows+D my way to the desktop to execute it.

Windows does seem to have made the decision early on (i.e. back in '95, when I first started using it) to provide a clean and concise GUI first and foremost, and then to add programmer/automation-friendly terminal APIs as an afterthought. I wonder if the issues that Unix users on windows have are caused by the fact that Unix systems seemed to go the other way, and that attempting to replicate a Unix experience on Windows would just lead to frustration?

This is interesting, because I rarely get a chance to use Unix at work, nor do I get much of a chance to speak to Unix devs (we're a Microsoft shop -- WPF, ASP.NET MVC, and SharePoint if we're feeling masochistic). Thanks again for an interesting snapshot into your workflow.
Back then, gravity was a lot less, so soft bodies can grow huge. Now they can't.
The case centered on the question of whether those companies would have to pay twice--once to buy bandwidth from their own ISP and then again to get through a toll to the end user, paying originating ISP then the terminating one.
Your comment may be intended ironically, but it's more or less true.

Some lower-level staff may get hung out to dry (e.g. Abu Ghraib) but if you were a senior-level commander or official, why would you finger one of your peers, when he probably has similar dirt on you? Better that everyone keeps his mouth shut.
My Heart
they need to fire their plastic surgeon if a font update and spacing is what is considered a facelift.
Pandoc does this well. It's what I use.
Show HN: Dotstorming – Real-time dot voting for crowdsourcing opinions
I know a lot of this post-PRISM fear of the US government is to be expected and is certainly popular on HN, but comments that single out the US like this really make little sense to me. The governments of Britain, Canada, Australia, and New Zealand all share tech-related intel with the United States and, potentially if not presumably, have similarly intrusive methods of surveillance. China goes so far as to ban social media sites that they don't have control over (e.g. Facebook, Twitter, and Youtube) and replaces them with sites that they do (renren, weibo, and youku/tudou). That, combined with general censorship of the web, is far scarier than what the NSA does, even assuming the NSA's scope is as wide as television shows make it out to be. And even in most other countries, the scope of domestic Internet surveillance is unknown, which is very different from "known to be minimal".

Edit: Ah if this was about intellectual property takedowns, then currently the US situation is a bit worse than most of the rest of the world. But still, China friggin censors their Internet, and comments like the above still seem to always single out the US.
I am pretty sure that torturing someone is worse than someone suspected of being part of a terrorist organization. Suspected is the key word, as is "part of".

We don't deliberately torture convicted murderers: we kill them in a way which we (correctly or not) believe to be humane. We shouldn't be torturing suspected (or convicted!) terrorists either.
Out of curiosity, why don't you use Google Docs?
Making Perks Work for Your Company
Oh baddox, you are right, (I forgot) and alas, it is young idealists whom we so often need to lead us into the future. Those Revolutionary lads founding our country were below presidential age bar and that is why they were so mighty--they were still idealists full of passion.
Pandoc! [1]

[1]: http://johnmacfarlane.net/pandoc/
> (it would be more correct to say trigonometry)?

No
Sir, not all of us our javascript/HTML5 people. I do mostly operations, and can barely drag myself through Javascript until I get the chance to take some vacation time to concentrate on learning the web side (JS/HTML5/etc). I admit that I don't know what I'm doing sometimes.

Does this look right?

javascript:void(open('//web.archive.org/save/'+encodeURI(document.location)))

I hacked it together from what you posted and what my archive.is bookmarklet specifies: javascript:void(open('http://archive.is/?run=1&url='+encodeURIComponent(document.l...)

EDIT: I can confirm that the bookmarklet I provided above does work. sp332, thanks for your help.
Show HN: Interesting for iPhone
Or they can just take the executable and change the extension. Extension-based blocking doesn't really work. However, these types of files generally have magic numbers in their headers that you could inspect to determine what they are.

Also, it'd be rather inconvenient if you couldn't upload a compressed archive to a file-sharing site.
Kik quietly opens its Cards content platform to third-party developers
I suggested it due to DMCA takedown notices, not NSA-related concerns.
Ousted BlackBerry CEO Had $56 Million Golden Parachute
Born in North Carolina. His passport has been revoked, but he is still a citizen.
Archive.org: Play Castle Wolfenstein in your browser
Making about 500EUR/mo from selling motivational posters at http://squaredo.com - primarily to startups
Their older CDX API provides that functionality:

https://github.com/internetarchive/wayback/tree/master/wayba...
So You Want To Quit Your Job (How To Do It)
It's assumed they grew that way because that's what the fossils look like.

If a fungus had a photosynthetic symbiont, it would have a good reason to compete for height. In fact, an algal symbiont is a leading theory, based on chemical traces that point to algae in the fossils.

Interestingly, lichens do this today: they are composite organisms of fungus and photosyth (algae or cyanobacteria). They don't currently get very high, but perhaps that's just because plants are so much better at height that all tall forms have gone extinct.

If there's no photosynthesis going on, it's harder to explain. Perhaps it's handy in a very low-growing world to lift most of your energy-storage mass or hard-won water above the ground, away from predators?
The southern economy did practically collapse after the Civil War.

Slaves were freed. Infrastructure was destroyed. A quarter of the white male population was dead. These factors together led to a large drop in per capita income, but it's impossible to try and assign levels of responsibility to each one. [1]

It's also worth noting that the formalized system of slavery was replaced with a de-facto slavery system called sharecropping [2]. Sharecropping involved forced borrowing of work necessities at a high interest rates. It led many farmers, both black and white, into the grip of wage slavery [3].

1. http://en.wikipedia.org/wiki/Reconstruction_Era#Material_dev...
2. http://en.wikipedia.org/wiki/Sharecropping#United_States
3. http://en.wikipedia.org/wiki/Wage_slavery
Build some community moderation tools and promote some trustworthy people.
Hey everyone, Interesting is a simple app for iOS 7 that shows links from hundreds of hand-picked sites across the Internet, including Reddit. I made it for myself, so I'd always have something to tap on and read at all times, but redesigned it for iOS 7 and released it. Would love to know what HN thinks.
In this case though, it does make sense. For this example we're not talking about surveillance, but about intellectual property takedowns, something which american companies find (slightly) harder to do in countries other than the USA.
People say things. Some of it is immoral. Some of it is stupid. Some of it is neither, maybe just throw-away. In this case, actually, the original question about having a family stuck in the author's head (not in a bad way) for decades. He thought of it as a good way to introduce the next part of his story.

It seems a shame to me that we're so knee-jerk politically correct that we can't allow people to not say things perfectly the first time, even when their intent was perfectly harmless.
I love the Wayback Machine (and all of Archive.org, really). I recently used it to reminisce about some old VRML-based chat communities that I frequented about 10 years ago. It had a record for every single of them.
they changed their interface. but I wasn't logged out.
That's a stupid example and it undermines everything sensible you said elsewhere.

What about the following useful usecases?

* on-the-fly syntax validation
* non-blocking as-you-type completion
* live results from shell commands
* fast syntax highlighting
* etc.
Plausible, but not apparently a problem from most fungi currently alive. I would guess that the trade-off in terms of marshaling resources vs. increase in spore distribution would not be favorable enough. I expect that's why the lichen argument appeared - we can easily see why being tall is a plus for photosynthesis.
"The trick is not to try and break the glass ceiling..."
"The trick isn't to try and break the glass ceiling. That's impossible. Instead, try and realize the truth: there is no ceiling." ~ The Matrix
The USA has taken possession of .com domains and raided physical premises based on copyright.

This need have nothing to do with the NSA and still be a perfectly valid and reasonable suggestion.

I think it's reasonable that any site that has user generated content should consider not being hosted on US soil or using a .tld that is under US control.
Fake perpetual motion machines you can build
Genuinely interested to know how Germany is a safe-haven for file-sharing since almost every other European country would probably give you trouble.
Given that Marxism _has_ changed over time, I'm not sure you can make this assertion.

Of course, ANY group has people that hold onto an 'original' form of an idea, but they don't represent the majority.
We allocate so many sprint storypoints for keeping technical debt down. That's a fixed amount, immune from interference. The value justification is thst it allows you implement features faster long term rather getting clunkly a codebase. So it does provide customer value.

For long tasks you use a thing called epics.
> There's still a non-zero chance that this one come true.

Best joke of the year.
A race condition is a set of (possibly very many) very well defined states - or at least defined as well as everything else in the program. Someone should discover some algebra for working with concurrent processes and the race conditions would go away!

(the second sentence is a joke.)
Chess 2: The Sequel [A street fightin' man fixed the world's most famous game]
What are your principles for great achievement?
Ask HN: Is Firefox causing me to be IP-banned from HN?
Hi Everyone. I have just been IP banned for the 2nd time in quick succession by HN. No complaint at all about that, in fact I want to thank those that have taken the time to show me logs of what is going on - appreciate your help!

I am bombarding HN with HEAD requests and yet from my point of view I am doing absolutely nothing differently in how I have been using HN for years. I use Firefox on Windows 7 64 and generally keep HN open in a tab. Every few minutes, or hours, I manually click on the title to see the latest updates to the front page and comments. Pretty normal useage I would say, and up to now never been a problem.

Clearly something is different and my browser, or something else is turning me into a rude user.

- Is it possible that keeping HN open in a tab is causing Firefox to send HEAD requests rapidly enough to get me banned?

- How can I test this?

- Aside from using another browser, how can I avoid this in future?

- Is anyone else noticing this sort of behaviour?
Yeah.

It's not unreasonable to expect your boss to treat you decently, at least if you treat her decently. She's a person, after all.

But the corporation as a whole? The corporation is a system. It responds to the environment around it. If it doesn't make the right response (as in one that maximizes its chance of survival), it is replaced by other corporations that do. That right response may be at odds with your interests as a person.

You, your boss, your boss's boss, and all your coworkers are part of that system. If they don't take the "right" action (as in, the one that maximizes their chance of survival within the organization), they will be replaced by others who do. If you feel you've been mistreated by this, your only option is to look for a different place in that system where you will be happier.

It's a two-way street. It's not just that people should refrain from treating you poorly, it's also that you shouldn't put yourself in a position where you will be treated poorly.
Are International Comparisons Unfair?