skysight-inc/HackerNews huggingface dataset:
Crawlers and spambots are the scourge of medium to small websites.

I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers.

I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic.

I feel like search engines should crawl websites in proportion to the traffic they send. For example, Yandex, Baidu, and Bing were all crawling my website hundreds or thousands of times a day, but never sending a single visitor (or in the case of Bing sending single-digit visitors). It's an absurd waste of resources, so I blocked them completely.
VCs aren't that insanely rich after all
Try disabling the Ad Blocker.
Yes, an obvious cue for something that is not news, but may be interesting seems wholly appreciate. I think HN is pretty good about this in general for articles not in the current calendar year. Such a modification is not editorializing or biasing the content in anyway, too. So this keeps a consistency in place on that front.
I expect to press the back button and land on the previous page, namely, this comments thread.
Your comment about maps is dead, fyi. No idea why. I agree.
You forget to take into account the tax bracket!
"Do the containers shield against strong EM fields?"

from article: Each EME has its own shock and vibration damping, power protection, water cooling systems, and electromagnetic shielding to prevent interference from the ship's radar and other big radio frequency emitters.

I would imagine that would include EMP given the modern threats and US Navy's general understatement of capabilities.
Being smart, we can also generally tell when a title is editorializing, and weigh that against the quality of the original source in deciding whether or not to click. If all titles are auto-reverted, then we're at the mercy of the crappy editorializing of the publisher; I feel I have to check out stuff with stupid titles because maybe there's a cogent article behind it.
Have a startup idea but don't know how to code well?
ah, yes. up is left. there's nothing counterintuitive about that.
"You didn't really give me a query to debug."

It's right there in quotes, at the top of the very discussion page you link to. I'd like to think you are really interested in these issues and not mounting a disingenuous public relations campaign, but you don't make it easy.
I actually did this exact robots.txt for my site this summer. The net effect? A massive loss of secondary market traffic - enough that, after a month, we probably lost 10-15% in revenue. It surprised us. We rolled the old robots.txt back into place and bam - life went back to normal. You can question my pithy 300-character explanation all you want but I'll leave you with this: we were 100% certain that the robots.txt change was the difference.

In the end, we switched to network blocks for the common bots/spiders. Much better.
Your project is great, Pascal is really underrated language.

I looked at parser (hand written are best), this part spotted my eyes https://github.com/lkesteloot/turbopascal/blob/master/Parser... - IMHO order of checking whatever node is identifier or not have to be reversed. Now, such simple program:

program Test;

procedure foo;
begin
end;

begin
foo := 10;
end.

produces error 'Error: can't cast from 70 to 76 ("10", line 8)' - whatever this means. :)
They've announced offline viewing for android recently which should help
I don't think that this is the best policy for creating useful and clear titles on the Hacker News site.

But I REALLY appreciate your making this post to explain the policy and the reasons behind it. I can go along with this even if I don't think it's the best policy, and now I can understand the reasoning behind it. The openness of explaining the policy and the reasons for it are a big improvement. Thank you.
Actually, there is such an obligation (or at least, submitters are strongly encouraged to do this in the guidelines). But this is widely ignored, and a distressingly high percentage of submissions are just blog churnalism.
The problem is that they have a simple policy that makes total sense based on their principles (first person to post a story doesn't have special privileges, and posts are community property) and their limitations (part time admins).

Unfortunately, that policy engages with an issue that nerds can debate endlessly --- which title is better? What constitutes editorializing? Are original author titles the optimal titles? Oh, look, there's that word "optimal" --- let's spend seventeen weeks debating it!

Therefore, it seems like there's something for us to discuss. But there really isn't a discussion to be had here. People who want titles to be managed should start their own HN alternatives. We could use more of them. Or, even simpler: if you have a story with a bad title and a new title you feel strongly about, instead of submitting and then writing a long comment, write a blog post about the story and submit that instead.

Of course, those two suggestions are much less fun to talk about than a debate about titling stories.
Hey, you are welcome. I'm glad you appreciate it especially since I pay $80 and don't even watch TV.
> The only way we can tell if a newly created title is accurate is to read the article, and we're not about to read every article submitted to HN. The only option is to revert to the original title, which is at least what the author intended.

So do it automatically then. It's ridiculous to say "we can't do this automatically because that would be too inflexible... so we'll get human moderators to blindly follow a process without thinking instead".

What happens now is that users put a lot of thought into a good title for the page they're submitting, and then a mod comes along and just trashes it. It should not be surprising that this upsets people.
I think this says a lot more about Business Insider than MobileWorks.
I've dealt with this same problem for many years as well. Recently I just completed a side project (http://www.randolunch.com) for the first time ever.

Three things really helped me get it to the finish line:

1) I listed out all the possible features of the project, then marked only the features I absolutely couldn't launch without as MVP. I used https://workflowy.com/ to do this because I liked the hierarchical structure and the ability to cross off features as I implemented them. Whenever I worked on my project I'd have workflowy open in one of the browser tabs.

2) If I didn't feel like working on the project I'd open up the feature list, pick one of the simplest items, and implement it. I found that if I started writing even just a few lines of code it was enough to motivate me to keep at it for hours. This was especially helpful as I got closer and closer to the finish line.

3) As others have mentioned here, launch your project as early as you can, even when you still feel "embarrassed" by it, and tell people about it! Looking on Google Analytics and seeing even just a handful of people using my site was incredibly motivating! It made me want to continue improving the site just so that those few people would have a better experience.
Modeling your App's User Session
UK orders WSJ to censor names from website, to no avail
Quick experiment says: Gzip can be written out as a stream ok. Can't comment on Apache but Node does it fine.

Firefox's xhr fires progress events for gzipped content but not Chrome's. Looking to see if I can find a way round it.
Okay I did it myself:
https://gist.github.com/javahead123/7045923
Yes, but with water on both sides of the windshield (it was a "wet" sub, no airtight compartment).
it's a wet submarine, so there's no pressure differential
I think there would be fewer complaints about this practice if the submit page explained this intent.
This is exactly the right approach, and has the added benefit of offering the submitter an opportunity to editorialize and summarize to their heart's content without taking something unfairly from the community.
It looks like a fake.
I believe they are transitioning to full time positions for exactly that reason.
Link is broken for me.

Yes, cost of living is differently something to consider when talking about salaries. I think a person living in the middle of no where not spending near nothing on rent making $40K probably saves more than a person living in San Francisco making $100K.

I would much rather live in the mountains getting paid $50K than live in San Francisco getting paid $100K+, but that's just my personal preference.
s/small business/small retail/
I prefer to fail as soon as possible. I use asserts in places where there is no good way to recover from a failed invariant.
What about a system like this:

You crawl your own site every day (or generate the same content into files)

Put the content into files with file names representing the URLs (or HAR format?)

Zip em up

Put them on bittorrent

Tell the search engines to look there for your content

Wouldn't that save everyone a lot of work?
> people need to stop reciting this compulsively and take the time to think each situation through.

I believe that thinking has already been done, and the reasoning published. If you can refute the well-publicized arguments against JS crypto, then of course that would be productive and appreciated. Given the discussions that have already taken place, I believe the burden of proof now rests with those who support JS crypto, not those who oppose it.

To make an analogy: We "compulsively" assert that the earth orbits the sun. But that's not a cargo cult. It's a conclusion which we've confidently accepted based on the weight of the evidence.

> Realize that the SecureDrop document submission client is a web application. The browser of the document submitter will run whatever the SecureDrop Source Server provides it barring the edge case of the submitter verifying the source page source with GitHub before allowing JS in NoScript.

If that's true, then SecureDrop might not be so secure. I'm not pointing to SecureDrop as a gold standard. There are very few cryptosystems I trust.

> The security of the document submitter is already prone to compromise by way of a malicious web app provided by malicious Source Server or MITM. Moving the project to something more JS heavy on the client side would in no way worsen the threat model.

That might be true, but then again it might not. We don't know much about JS crypto. We don't know what attacks are possible. (We know about compromising the JS source, but that's only one threat model. There could be others that are unstudied.) Thus, it's quite possible that there are attacks which depend on the application using a specific browser feature, such as drag-and-drop. Is this likely? Not terribly so. But is it possible? Absolutely.

But I'm sort of nitpicking. Like you said, JS can be MITMed, so there's not much point in debating whether a given JS crypto app is secure or not. The best strategy is to just not use JS crypto.
Kippt imports Github starred projects and makes them searchable. https://kippt.com
Odd. I'd say the same thing, except s/Walmart/Facebook.
If everybody does this then a new search engine will never come to be. You better hope Google is the best search engine anyone could have ever created.

As a search engine developer that has tried to compete with Google in the past, I find this disheartening.

I agree with blocking anyone doing bad. But you are cutting out the good with the bad.
HStore and especially JSON are useful because they allow you to store complex data structures fairly easily. Do you have something that contains a number of somethings that contain a number of somethings that contain a number of somethings? Normally you would need like, 4 tables to store this and carefully planned queries to work with them efficiently. However, if the object graph has no external references (besides the top-level object) you can just safely shove the whole thing into a JSON without losing referential integrity.

Your reasons are very valid however so obviously you have to think about the consequences before using non-standard features. However, not using advanced or novel features (which non-standard features tend to be) can also lead to stagnation and loss of competitiveness so you have to also be careful with being careful, if that makes sense.
If a Google Employee Dies His Spouse Gets Half His Salary For The Next 10 Years
They severely mistreat their employees when it comes to on-the-job injuries.

Friend of mine injured his knee while working in one of their warehouses. A perverse company policy gives bonuses if the crew is injury-free for a certain amount of time. This is supposed to improve safety, but what it does is cause management to put intense pressure on workers not to see a doctor when injured, because if they don't see a doctor then it doesn't count.

In my friend's case, this exacerbated the injury greatly. By the time he got medical attention, it was a $20,000 surgery. Wal-Mart then denied responsibility, saying it was an old football injury and thus a preexisting condition not covered by their health policy. This basically bankrupted him.

Beyond that, they achieve low prices by leveraging their massive size difference to completely screw producers. Examples:

http://webcache.googleusercontent.com/search?q=cache:MF1qnjm...

http://www.fastcompany.com/54763/man-who-said-no-wal-mart

In short, Wal-Mart will make big promises that generate big commitments from suppliers, then break their contracts in ways that are difficult to fight in order to save money and make the suppliers ever more dependent on them.

But, to be honest, the way they personally affected me by fucking over my friend is by far the bigger reason behind my judgment.

I'm not generally troubled by the way that companies act as amoral profit-maximizers. But they're supposed to do it within the law. Wal-Mart actively screws over people by playing outside the rules, far beyond what I see other companies doing.
For entry level it might be okay. Non-entry level or positions where bonus is a large percentage of total comp it is useless. Listed compensation for non-entry investment banking or trading is widely inaccurate.
Right, but that doesn't include the extremely valuable equity grants given to engineers at these high-growth tech companies.
Everything in the U.S. fleet except the large aircraft carriers and the submarines burns crappy low-grade diesel fuel. For the submarines, there are major advantages to not having oxygen-requiring engines, and for the aircraft carriers, they want to carry a lot of fuel for aircraft, not themselves. On the other hand nuclear reactors are too expensive to merit putting in ships which don't require them.
True. But that is only one alternative, not "the" alternative.

Take Australia. The government proposes a budget and the senate reviews it and after making any changes they decide are required and the government agrees, it passes.

In the situation where a deal can't be made and the senate blocks supply of money to the government the "politically neutral" governor general can dismiss the government and force an election.

This has happened once:

http://en.wikipedia.org/wiki/1975_Australian_constitutional_...
The original recommendation was phrased in a somewhat harsh manner. And while I could be wrong, I really don't think it was meant to be taken completely literally.

I think the literal intention of it was: "if you want to join this field, all you need to do is put in the time and effort to get really good at it". Not that you should literally lock yourself away in social isolation and speak to no other human beings for several years.

"Put in the time and effort" could also involve networking with other hackers, finding collaborators for open source project or startups, communicating on IRC channels, and learning from peers in person. Yes, there will be hours of solo grinding out code and debugging - but that's not the only thing that a programmer does.
I am very honestly interested in these issues, and in the past HN has been a great source of queries that fail in interesting ways, which is why I keep posting this request.

On "html title tag" what do you want to know about it that the w3schools result at 1 doesn't give you? I know you may not like the site in general, but is there more to a title tag than that?
Clearly, it's actually a flying car. The car will leave the water and the juice will at that point be able to manage lift off.
You're post is incoherent. I'll try to address what I can.

You ask what I would have controlling inflation / deflation. The free market is my answer. Study the history of inflation / deflation you will see that almost without exception inflationary shocks are caused by the control of money and its supply by a narrow self-serving elite. You ask what else I would favour when a much better question would be why, based on the historical record you would ever lend intellectual support to the status quo.

You mention "bullshit Fed conspiracies". Do you actually understand the Fed and its origins? I'm 100% sure that you don't. The robber barrons were never deposed. When the Rockefellers and Morgans were relieve of thier "monopolies" they gained the greatest cartel and monopoly that one can possess: The power to print the money. The won the central bank. That's no conspiracy. It's fact.

Your mention of mining is a bit confused, and I don't really understand what your getting at. You are aware that mining isn't cost-free activity?

Finally, before you repeat the asinine arguments as a peasant shill of their cartel how about asking yourself why the central banks of the world retain gold as their most prized asset while at the same time quite successfully employing you, for free to peddle the notion that only toilet paper has any utility in internet discussions.
Purely anecdotal here, but when I search Google on my mobile browser I'm only seeing 1 or 2 results per page. I've pretty much given up using it on my phone now.
> As others have mentioned they have gone all in as an exploiter of their demographic data

Oh, and exploit it they will. As expected, a future where "privacy" will become a lot more important, would be the "enemy" of what Google is trying to do, as long as ads are their main source of revenue.

> Lloyd made his pitch, proposing a quantum version of Google’s search engine whereby users could make queries and receive results without Google knowing which questions were asked. The men were intrigued. But after conferring with their business manager the next day, Brin and Page informed Lloyd that his scheme went against their business plan. “They want to know everything about everybody who uses their products and services,” he joked.

http://www.wired.com/wiredscience/2013/10/computers-big-data...

I'm hoping sooner rather than later, Google will find another cash cow, perhaps in self-driving cars, or Google Glass, or Calico or whatever, that will allow them to make a lot of money from that, and not be as desperate about tracking every single thing everyone wants does online, almost as much as the NSA.

Until that happens, we won't be seeing any true secure or private services from Google, because the desire to track will always come first, before anything.
Is Reposted Content the Secret to More Page Views?
> But don't ban everything just because a few (proximic and ADmantX) are hammering the site.

I can understand blocking somebody that has a long-term and clear pattern of disrupting your site, not following the robots.txt rules, and not providing any links or anything back to you. But I find the idea of somebody preemptively blocking everything but Google and maybe Bing extremely distasteful.

If everybody out there blocked everything but Google/Bing, it would make it very difficult for anybody to ever try and create a new search engine or create new types of web services or analyze data in new ways.

Possibly a better solution is making the common crawl initiative a better project - make it more frequently updated, make it easier to get started with it, provide better documentation, etc. If there was a way to get every web service out there that wants to crawl the web to contribute to this, it would lighten the load on everybody. http://commoncrawl.org
What is likely to be true is the pharmaceuticals will allow a conscious choice among a menu of tradeoffs, on a daily basis.

Such tradeoffs might be possible now, for those with sufficient discipline, but switching between very different ones is too hard.
VC firm discloses 'internal operating manual'
I never thought Elon Musk would try to disrupt the Duck Tours business.
Has anyone managed to load the page yet?
I've been trying on and off for ages.
But being a percentage of GDP and not a set amount of euros, the effect of this limit is quite different from the US. During a recession, the GDP obviously goes down (tautologically speaking), so the limit can be breached without borrowing a single euro more.

It's basically a mandate for governments to act in a procyclical way instead of a countercyclical one as reason would suggest. Fortunately, that makes the mandate so obviously stupid that no one really cares about it during a recession or in other special circumstances (like the German reunification)
The HN DDoS attack strikes again...
There's a Google Doc with a lot more detail as well:

https://docs.google.com/document/d/1xW9GeEpLOWPcddDg_hOPvK4o...

Your question is covered under "Bootstrapping"

> Bootstrapping
> Using ruby should be possible thanks to binary rubies, in worst case we can fallback to static build or use JRuby. The binary Ruby (or static or JRuby) will be installed along with RVM 2 using very simple bootstrapping script written in shell most likely pure SH to provide compatibility with as much systems as possible

I had similar thoughts. The fact that RVM is implemented (currently) in shell means you don't need Ruby installed. It does, however, expose your tool to all the idiosyncrasies of someone else's shell environment.

I sit in #rvm on Freenode from time to time and the vast majority of trouble pings are related to differences in environment. That is to say, if you run a standard (as in, not heavily customized) environment, you won't have much trouble. There are a large number of users who really like to tweak their shell though, so RVM has to face the issue head on.

I think this is what has lead the team to the "bundled" approach. It sounds like RVM 2.0 will be a full-blown application, rather than a collection of scripts. The choice to build the application in Ruby, as opposed to something like C, seems obvious when you consider what it is. Ruby folks want to write Ruby.
Some interesting background of the author:

http://www.teamten.com/lawrence/oscar/
You know, it seems like the solution here is to be able to distinguish the original title and the submitter's commentary about the title. To borrow from another comment in this post, something like:

A new beginning

...could become:

A new beginning (PHP elects new leader)

It's tempting to say "Titles shouldn't include commentary", but I think that there are valid times where the submitter should submit commentary. Otherwise, we just end up rewarding people for duplicating the same content with a more linkbaity title. You can imagine a techcrunch article that just quotes the original, but has the title "PHP Implodes as Leader Steps Down".
Also impressive is the market capitalization per permanent employee: $337B/46,421 = $7.3 M per employee.
Were you on a diesel boat? Because I was on a nuclear sub and it was pretty quiet and the sound powered phones worked great.
Oculus Rift's John Carmack says a new Rift dev kit is in the works...
Yes, apparently, but too late to delete.
... reservable parking spots featuring what look like manholes in the pavement, but are in fact wireless charging devices that will give the trucks a little more juice while they sit.

From the title I assumed that the cars were being charged as the passed over the manhole covers. That would have been much more interesting than what this service really is about.