skysight-inc/HackerNews huggingface dataset:
Ask HN: Advanced web crawling resources?
Does anyone know any good resources for advanced web scraping (scraping at scale, getting around various tricks to prevent crawling, etc.)?

I've looked around a lot but nearly all resources I find are the same. A short description, a small code snippet and that's it.

I'm really looking for more.
Not everyone wants that lifestyle.

Being a photographer and also someone who tends to do a lot of Maker and gardening projects I find SF near impossible to live for my lifestyle. It's really hard to load and unload photo gear when you have to make 4 trips to the truck that is a 5 minute walk away and go up 10 stories each time. Not to mention paying $40 for parking. I couldn't do what I do without a truck and SF is extremely unfriendly towards cars.

I do like being able to walk to lunch from an office though. When I was working in downtown Sunnyvale I found it really convenient. The parking back then was decent but now it is overcrowded and you're lucky if you can find a spot. You have to wait in long lines for lunch now too. It's gotten much worse IMO.

When I worked in Redwood City I tried to take Caltrain but it adds another 2 hours to your commute every day and the train schedules are extremely limited the further south you live. Ended up getting a motorcycle just for commuting.
The tracks were surprising until I thought about all the idiots I encounter on regulars roads. Then it made perfect sense.
It could be because religion is useful
I've read about some theoretical designs that would open the field to allow all negative charged particles out and use these electrons as power directly.

But first, designs would have to be able to keep the containment pressure/temperature much higher than it currently is.

https://en.wikipedia.org/wiki/Aneutronic_fusion#Energy_captu...
I do not think that Postgres is hard to use correctly and I also think that Uber did a poor job on trying to fix their problems. This PDF just outlines the details. Uber also has the tendency to switch databases every few years. It is hard for me to take their opionion seriously on databases.

https://www.yumpu.com/en/document/view/53683323/migrating-ub...
>I'm betting has more to do with it than...

I think you accidentally a word.
Honestly I think Uber switching was more of a business decision as their employees had trouble figuring out how to properly use Postgres. Instagram definitely has different data needs, but is probably much bigger than Uber wrt data. They use Postgres successfully via Django no less!

https://engineering.instagram.com/sharding-ids-at-instagram-...
Definitely bad. That's exactly the type of image I would use as a current home/land owner of a neighborhood to discourage urbanization and building more housing. "See what our neighborhood could become!!" /s

Even the residents don't like them:

>>“There isn’t much design inspiration. They’re just stacked up,” said Park In-seok, an architecture professor at Myongji University. He described a paradox in which the apartments are mocked for their appearance but coveted for their convenience. “Almost everybody hates the apartment,” Park said. “But everybody wants to live in one.”
Sorry I know that post quality is important and everyone should strive to make important points that move the conversation forward but at the same time I have to say "let's get ready for the aliens" really made me happy. Funny.
I think CEO pay is what it is because they're narcissistic personalities who are very good coddling the fragile emotional state of board of directors, and they all make each other feel important and useful.

The only solution I see is a syndicalism, or perhaps participatory economics (one of those is real, has existed, the other is an idea possibly with no potential in the real world). Basically the only thing people understand is go fuck yourself. Let's face it. It's a might makes right world. People don't negotiate win win if they can negotiate I win you lose and that's a win, so go fuck yourself. It's always adversarial when a single entity owns both capital and means of production.

But the very idea of worker unions brings about the open question of committing murder by certain capitalists. It's that much of a threat to their ability to say and get away with, go fuck yourself.
> you would think that it would at least be worth trying to save the world

Not if "saving the world" means spending trillions of dollars on things that (a) might well not fix the problem, and (b) could be better spent on other things with a much more certain payoff. For example, bringing people out of poverty and building up our infrastructure so it is more robust against any kind of disruption. Especially if doing those things makes it a lot easier to adapt to whatever happens to the climate anyway.
You don't have to be a bystander in this process... the incentives might be for cities to listen to their current homeowners and to add more parking and prefer office parks, but we can organize for change and more density.

Here are some steps you can take:

- The Brisbane City Council is deciding whether to build 4400 units of housing on 600 acres south of San Francisco, about the same number of units SF built in total last year. The Brisbane Planning Commission recommended building an office park instead. Contact them and ask them to build the housing version of the project.

- The Mountain View City Council is deciding whether to build 2000 units or 8000 units of housing next to Google. They are leaning toward the low end - 2000 units would be tough to support a grocery store or frequent transit. Contact them (or show up to their board meeting - tonight!) and ask them to build the high-housing version of the plan.

- The San Francisco Board of Supervisors is considering a plan to require 28% of all new developments to be below market rate. When you consider the unit costs to build in SF, that would make it _extremely_ difficult to justify new housing starts here. Please contact your Supervisor and ask them to oppose the Peskin/Kim Prop C plan.

- Call your CA State Assemblymember and ask them to oppose AB 915 (makes it harder to build affordable units)

- Call your CA State Assemblymember and ask them to support AB 71 (higher property taxes on second homes, money goes to affordable housing)

- Call your CA State Senator and ask them to support SB 167, which would put teeth in the state's Housing Accountability Act. (for more on this see carlaef.org)

- Call your CA State Senator and ask them to support SB 35 (would remove the ability of local government to block projects that meet certain criteria - near transit, have a high % of affordable units, use union construction labor)

- Email your VC's and C-level executives and tell them how hard it is to find housing in the area and how ridiculous the parking requirements are. Tell them about your awful commutes and the difficulty of finding good school districts for your kids. Ask them to get more involved politically in pro-housing causes. Ron Conway is a good example here.

NIMBY's are really well organized and things don't change unless we do something about them. All of these changes listed above will go a long way to support the development of housing in the Bay Area, which should help lower prices, and help keep families here and teachers in our neighborhoods.
https://kev.inburke.com/kevin/sf-housing-politics/
There is an established discipline of Requirements Engineering:

https://en.wikipedia.org/wiki/Requirements_engineering

Resources include:

IEEE: https://www.ieee.org/index.html

And the Software Engineering Institute: https://www.sei.cmu.edu/productlines/frame_report/req_eng.ht...
I only used Foursquare as an example because they are a "dream job" company. I've been trying for all kinds of companies, smaller start-ups included. I think I'd be great for a start-up due to my past work experience, but I've not had luck there.
On the downside, LA traffic sucks but at least you get the sun shining in. Spending your whole commute in tunnels seems a depressing way to live, even if its shorter.
Banking with Artificial Intelligence
In case somebody missed there is another article from Uber about justifying MySQL -> Postgres move a while back.

https://www.yumpu.com/en/document/view/53683323/migrating-ub...
A rocket is something with only an exhaust. A jet also has an intake. And it goes a little further than just not lifting the 02. Any unburnt gas (air) becomes a 'working fluid' that can be pushed. A rocket moves by throwing gas away at speed. Jets also grab gas from the air and accelerate it along with exhaust products. That's why a jet or turbofan can have a high specific impulse with such a low exhaust velocity, higher (>8k) than even an ion thruster in vacuum.
First of all, "greater responsibility" is not an advantage, it is a liability you accept in return for other advantages.

If the delta is worth "tens of thousands of dollars in salary", consider what your free time spent on other things is worth. It might not be.

But "freedom to design my own role"? Where do you get this?
The costs of tunneling are like FAR more expensive than most think. Breaking, excavating, and supporting rock is slow, time and cost heavy, and precarious work. While this is an interesting concept, unless there are serious advances in rock boring techniques (personal opinion: there are none coming) this will never approach fruition. I would suggest anyone interested in further research look into the "Big Dig" of Boston and the staggering costs and challenges it faced.

Good luck, Elon. It'll be another moonshot company if you can pull it off.
Good luck with your search! Hopefully you find something soon. I know from my search that React is a good way to go.
I think most people would find it ugly and would much prefer the high-rises of New York, or even certain parts of Hong Kong.
This seem to ignore something in my original comment: What about the wasted effort?
I'd scrap the MBP and get a 5K iMac instead with that amount. Personal opinion, though.
When I heard about Boring Company I kind of just assumed it would be used for underground hyper loops. This feels kind of wrong.
Separating Overlapping Chromosomes with Deep Learning Based Image Segmentation
Yes, I've worked with some. They have gotten me some of my interviews with companies where I have been a finalist. I will check out Mondo, thanks for the tip!
Blue Waters Tour
The rolling friction of rails is really really low. And couple that with removing driver variability out of the equation and it makes some sense.
r/legaladvice or see a lawyer.
That would satisfy "weight." It's conspicuously lacking in the ACM code of ethics (and basically every other code of ethics someone puts online with no industry buy-in).
This saved me at least 2 hours of awful fetchmail/getmail related work last month; Highly recommended.
Someone needs to edit the video so you see a pedestrian falling down the shaft and breaking their neck
Now what if we add an Uber-like component to this and let people share/carpool together to reduce the number of cars above and below ground? Instead of tires that wear out, we could use steel on rails! Aaaand, we just re-invented the subway.
> someone could take say $100K out of their 401K to start up a business. This money is sitting right now and doesn't help grow the economy.

Apart from being loaned to companies in exchange for bonds, loaned to various levels of government to support infrastructure projects, or capitalizing companies and decreasing corporate borrowing costs in exchange for equity.
That's your opinion. Getting stuck in a traffic jam on a hot sunny day is very displeasing to me. I'd rather move fast through a tunnel.

Edit: fixed typo
> without its axles

electric hub motors would work here quite easily.
Hi pseshadri, I would love the list. I will take any lead I can get. My email is in my profile. Thank you!
Streisand effect indeed but I don't mind. I got to see two posts and some technical aspects I wouldn't otherwise.
> unless there are serious advances in rock boring techniques

I'm pretty sure Elon Musk doesn't enter a market unless he intends to do just that.

Edit: I got downvoted a bit but it is his MO. He takes things that are expensive and makes them cheaper. For what it's worth though, I do think the tunnels in that video are ridiculous.
I dunno man, spending 2 hours in traffic versus 15 minutes is probably a pretty easy choice for most people. I get the sentiment, but if we are being honest, sitting in traffic is ALREADY a depressing way to live for most people.
I'm a little confused by the pdf. Is it implying that the program is opening and closing a connection for each query? Is that normal these days?
This isn't the only time something silly like that is in law. In New York, a loaded gun is any gun that you possess at the same time as ammunition suitable for it.
> I know PostgreSQL meant to defend themselves

[...]

> you don't see Microsoft defending MSSQL this way.

PostgreSQL Experts Inc. is a consultancy that specializes in PostgreSQL; they don't appear to be particularly linked to Postgres development organizationally (and none of their staff profiles highlight involvement in Postgres development). If this was the PostgreSQL Global Development Group, you'd be a bit more on point.
I wonder why some of these tips, like a statement timeout, aren't the default behavior of Rails. Some things can't be, like pg-bouncer, which makes sense. But why not have pg_stat_statements running on PG? Or have log_lock_waits in each statement?
Seattle's Bigger Dig is also worth studying: https://en.wikipedia.org/wiki/Alaskan_Way_Viaduct_replacemen...

Sending rockets to another solar system would be cheaper than the tunnels depicted in this video. I'm not even kidding.

Where is all that material going to go? There's so many tunnels there you could build a small mountain with it. Maybe he can team up with some sea-steading outfit and build a small continent off the shore of San Francisco.

You'd need three or four orders of magnitude reduction in tunnelling costs to make that anywhere near affordable, and even then you'd still have unbelievably complicated logistical issues. How much concrete do you need for those tunnels? What about ventilation? Safety procedures? Flood control? A single one of those could cost upwards of a billion dollars and I'm not sure there's a lot of cost savings by doing more of them, the complexities don't scale that way.

The more you dig, the more you're likely to hit something expensive you're going to have to pay to fix.
Might as well add Judge Judy and a scene in court when the lawsuit happens.
Well, since the Internet was once described as a series of tubes(tm), the Boring company is just extending this metaphor to create the Mole People Highway of The Future. That said, it seems like a great experiment to run somewhere in the middle of nowhere in China the next time they want to build another one of those nearly empty cities they wish to fill with future citizens some day.

But seriously, Elon Musk appears to be dating hot and crazy at the moment. Let's just let him get through this phase of his bucket list so he can get back to being visionary once he realizes why one should never marry hot and crazy.
Depends.

If it's a Robin Hood scenario it can be morally justifiable.

If the crime is stealing from anyone, rich or poor, so I can live easier, then no.
Yeah, that "trillions of dollars" is a baseless assumption that's just pulled out of somebody's behind. There's this assumption that it's going to cost so much, when in reality, renewable energy and storage are shaping up to be cheaper than fossil fuels in the coming decades.

Pretty much the worst case spending on mitigation plans is 3% of GDP. Less than we spend on the military. And that's worst case. Over the long run, most mitigation analyses have us saving money.

I think that these baseless assumptions about the cost of mitigation are just as much a problem as the denial. A good amount of the denial comes from people that assume that they'll have to give up driving and eating meat. Another good amount of the denalism comes from dark money that influences political thought. That dark money is hard to eliminate, but we need to stop the speculation that fighting climate change means a significantly lower quality of life.
I thought this too completely inefficient for your transport dollar. I actually thought maybe this was satire.
I think they reacted perfectly - they showed they are passionate about their software, while keeping it lighthearted and informative.
But where are you putting the waste heat from everybody's personal Mr. Fusion reactor?

Just imagine if everybody on Earth has a personal, nearly free 10 megawatt generator. Flying cars. Laser weapons. Direct desalination of seawater in coastal cities. Stomping around in powered exoskeleton suits.

Anyway, it would be awesome. But where do we put all the heat?
This doesn't seem feasible at all. Especially the overview shot of the underground network, like it's all open air. Definitely wouldn't be possible like that under a major city.
And concrete production? And underground hazard detection? And coordination with city infrastructure? And...
How about... mass transit. You know, not needing to dig giant big tunnels and build the infrastructure to ferry individual people in huge cars around underground, because the roads are too congested with individual people in huge cars.

Dig big tunnels to ferry trains with people underground. Works pretty well.
What if the tunnels themselves are congested? Wouldn't it mean a line of cars surface-side waiting for their turn?
It's about the last mile connectivity. People don't want to walk/uber from home to the train station and from the train station to work. Having a car on rails solves this problem.
> This comes across as a bit defensive and snarky

As somebody who works on postgres a good chunk of his time, I agree. I preferred the initial type of responses (where we just looked into the valid complaints and disregarded other angles). But everyone can speak about something like this, and that's good too.
Is there a resource in the vein of the linked review, but that explains the book (and its philosophical content) on a more basic level, for someone with no background in the subject? I was hoping that Pirsig would help me understand philosophy by placing it in a familiar and tangible setting -- that's what the title suggests. I was very disappointed.

Take the idea of Quality, for example. Pirsig talks about it for half the book; this is a concept that drives him insane and is central to his life and to the narrative. I still haven't the foggiest idea of what he is talking about, pages of strange analogies notwithstanding.

To me, this was a book about a troubled intellectual and his grappling with mental illness. About somebody that drives himself insane by obsessing over abstruse concepts arising from human experience. And it's a well-told story that made me think, but also made it very hard to relate to the protagonist simply because I do. not. understand. his intellectual struggle.
> a meal without bread is unthinkable in France.

It's an overstatement, it's totally thinkable (and in fact people often don't eat any bread depending on the kind of meal).
F-Droid: A new UX 6 years in the making
That's the entire point of The Boring Company. It's way more expensive than anyone expects. That's the problem they are trying to solve. The entire goal is "serious advances in rock boring techniques".

That may be a lofty goal, but at least the message I got is that it is the one they want to reach.
I would not call Cupertino, Santa Clara, or pretty much anywhere in the South Bay "high-density" except downtown San Jose, and there are a lot of employers and employees getting along just fine in this area. Could transit be better? Absolutely. For one, 85 could use a few more lanes and a rail line. But instead of trying to convince employers to move their office to denser areas that are currently better served by public transit, I would suggest that we invest in improving the infrastructure in the whole area.
The reason (and timing) for the somewhat snarky response is probably because Uber right now have a bit of a black mark against them PR wise.
So you raise a fence around it (stored on the sides, underground of course), or slide a roof in (once again, stored below the surface) to prevent anyone from going in even if they wanted to. This doesn't sound like a biggie to me.
Lily Allen – Hard Out Here by PeXiLeen
So if the neutrons are untouched by magnetic fields, don't they also heat the magnets? I can see them passing through walls to boil water, but any heat created in the magnets would be very difficult to harvest given the temperatures involved. How much energy would it take to keep the magnets cool in the presence of a sustained (ie days) fusion reaction nearby? Generating such a flux in a bottle surrounded by things that need to be kept so cool sounds a bit of a fool's errand... but I really want practical fusion to be a thing.
Except when stellarator starts doing longer runs and requires more hardened materials (like ITER) and when we need more instrumentation in the containment walls (like ITER) and when we need to actually plan to harvest energy from it (like ITER) and somewhere at that point someone proposes yet another concept for fusion reactor, modern and not dragged down by all that "practical" stuff. Also fusion is not 30 years away, it is likely more, after ITER we'll have DEMO, that would take many years again and only after that we may have first practical reactors running.
This is not a response from the PostgreSQL team. The author is a consultant specialising in PostgreSQL.
If there is no money involved, what is the benefit to you of moving your already-successful article to their publication (and removing it from Medium and your domain)? I don't see any.

You aren't necessarily going to get better exposure than you currently are... and what else is the possible upside for you?
If this is ever built someone will absolutely ghost-ride the whip in the tunnel.

And die.

And lawsuits.
2017: "On current trends, the Arctic ocean will be largely ice-free in summer by 2040."

2009: "[S]cientists at Cambridge University predict the Arctic ocean will be largely ice-free during the summer within the next ten years (i.e., by 2019)"

http://www.voanews.com/a/a-13-2009-10-15-voa41/414370.html

2007: Arctic summers ice-free 'by 2013'

http://news.bbc.co.uk/2/hi/7139797.stm

The date for the Arctic becoming "largely ice-free" appears to be receding about as fast as break-even fusion power. :-)
Startup Luminar claims it has the best LIDAR Tech in the world
Thanks for the awesome link!

I totally agree with writing slow. My handwriting is much worse than it shows on the post. I had to write slow to actually make it legible.
This is obviously just a marketing video, but that isn't a very hard problem to solve. The bigger issue is building an enormous underground highway network under an existing city.
The Boring Company isn't a jab at Boeing, it's a generic name for a company that bores holes in the ground... Ya know.. Tunnels?
Anonymous Open Source Projects
"unless there are serious advances" ...which is precisely what Musk is researching (bottleneck of boring machines): “We’re trying to dramatically increase the tunneling speed,” Musk says. “We want to know what it would take to get to a mile a week? Could it be possible?” See https://www.bloomberg.com/news/features/2017-02-16/elon-musk...
Always interesting to see right-wing parties attack universal government handouts on the basis that it'll go to rich people.

Do they assume that their wealthier voters know that it's a bluff?
One of the few things that benefit from pushed-back dates.
If its not PostgreSQL but instead a consultancy then the title should be changed to reflect that. The misunderstanding is likely to affect the comments (as above).