skysight-inc/HackerNews huggingface dataset:
Tricks like that don’t work with sites that are receiving a lot of traffic. Also, the exact solution you’ve described is a liability—IP addresses leak when people send each other links, and having unique URLs like that can cause issues with caching. Sure, we could store tokens in a database, but then you’ve just moved the bottleneck to the database.

We do have various ways to combat these issues; like any website of sufficient size, we have pretty complex methods of detecting problematic traffic and assessing the risk of any given request or session. However, no solution is perfect, and with the number of broken crawlers we see, some will inevitably cause problems.

To be clear, we can adjust our code and block them—that’s not an issue. The issue is that I have to wake up at 3 AM to do it, and even if it’s blocked, dealing with that traffic can be expensive. This guy got his $72k bill forgiven, but don’t expect the websites on the other end to be so lucky. (Yes, yes, ingress bandwidth is often free, but it’s never that simple. Scaling up? Bezos takes a cut. More database traffic? Pay the Bezos tax. Replication of enormous logs to other providers? Bezos hungry!)

Negligence is negligence. If you get in a car and drive recklessly without proper training, even if you didn’t intend to hurt anyone, you’re not going to get a lot of sympathy when you mow down a pedestrian. Likewise, I have little sympathy for people who face enormous bills for abusing powerful tools.

That’s not to say cloud providers don’t have billing problems. The delays are unacceptable, and the budgeting tools are often unintuitive or, as was likely the case here, outright inadequate. But in no universe was deploying code that spun up a container for every URL encountered a good idea.

Should such a mistake result in a $72k bill? Eh, probably not. I doubt this person will make the same mistake again, even with the bill forgiven. Or maybe they’ll just blame Google and attempt the same thing on AWS.
I can just never get past the arg/flag inconsistency/complexity across commands. Perhaps if the docs started with a simple example of what has been seen over time to be the most common incantation for each command it might be tolerable. But as it is, I spend more time dealing with idiosyncracies of a command than I do expressively piping stuff.

Perhaps if Rust had five mutually incompatible borrow-checkers which get called based on who wrote the code for a particular language construction-- that might get across the frustration I feel when using the shell. (Well, honestly I just StackOverflow for the shell incantation I need and that seems to work well enough.)
I’m not arguing that there’s not a “case surge”. If you test everyone who walks in a hospital door (which we are), you’re going to find “cases”. If you did this with influenza, you’d find the same thing.

People go in to the ER for a broken leg, have a positive covid test, and get counted as a covid admission. This isn’t fiction: multiple state health departments have admitted it is happening. It’s too difficult to try to determine if Covid is “the reason” for an admission, so nobody tries.

Therefore, the only important question is: how are hospitals faring relative to any other year? And so far, the answer is that it looks like a moderately bad flu season. To hype the narrative that “hospitals are overwhelmed”, the news media routinely cherry-picks one or two hospitals, fixates on them for a week, then moves on to the next.

https://mobile.twitter.com/justin_hart/status/13277409970783...
DeepMind should re-read their own policies: https://deepmind.com/about/ethics-and-society
Also - as someone with a ton of experience on the other side of this coin:

Puppeteer etc. are nice and all but if you can get away with raw HTTP requests grabbing and parsing the HTML without pulling down stylesheets, JS, etc. do it. It is WAY more efficient than requesting the full overhead for the user experience from these folks and threading out 5-10 workers to gracefully crawl a site this way doesn't typically cause things to melt down on your target's end.

You may be saying "well I need a browser-stack or evaluated JS to do my work" and you may be right... but honestly though 90% of this stuff is reverse-engineer-able with Charles Proxy and some basic webdev experience. Heck - I've even sandboxed JS from a target's site to generate tokens/etc to cut down on repeat requests. Even CAPTCHA stuff can easily be done without having to pull down full UIX overhead these days.

---

"Save a sysadmin: don’t snowball."

Implement thread limits, rate limiting, throttling, intelligent caching, and try to fit within your target's hosting capabilities without being disrespectful. Often I will "smear" large jobs over weeks worth of time so that it's only a trickle of traffic here and there (and to also fly under the radar... sorry).

Also - on the custom UAS: Unless you're trying to make it easy to get blocked/identified then don't take this advice. Let's face it - this is a gray area for most. The best way is to not "snowball" and to make your scrapers indistinguishable from a reasonable stream of real users from real networks. I would never expect a sysadmin to contact me because frankly they aren't paid to.

---

One last thought - the people who are out there writing these bots/crawlers/etc. are often the lowest common denominator. They're the type that will get something "working" and hurry onto the next job because the nature of the work tends to be a ton of low-paid contract stuff. Also, at almost every place I've worked at in ecommerce that has scraping involved it's the bottom-rung dev talent that's assigned to the work.

Sucks, but near-100% I attribute your "snowball" situation to that.
Tuition, room and board at stanford isn't $120k
If you really intend to parse the output of "git status" (or any command, really) you want the --porcelain option which is guaranteed to stay somewhat stable and parsable by scripts.

Many other commands have such an option. As an example, iproute2 has -json.
"Some sources cite [James] Cameron as the brains behind the glider navigation sequence..."

https://wearethemutants.com/2017/02/14/the-glider-navigation...

Clip from the film:

https://youtu.be/xxYmMRxnEic
Well, here is an article by a Nobel prize winner, reviewing a large amount of empirical evidence that personality traits predict success. See particularly section 4.

https://www.dphu.org/uploads/attachements/books/books_4871_0...
> But after that it is pretty much a crap-shoot, and all this theorizing are just post-hoc rationalizations to satisfy your ego.

On the other hand, all these baby boomers with money seeking to reinvent themselves as trend-predicting geniuses... at least they write checks.

Whom are you going to get money from instead, mom and dad? Someone has to take risks.
This is a dumb idea that would presumably cause other problems but ... what if they put the patient upside down? It's certainly make using some sort of lubricant a lot less dangerous. Now presumably it would introduce a host of other problems that would offset this but who knows.
Red Hat Goes Full IBM and Says Farewell to CentOS
This is a worry that almost came true to me last week. My sons Laptop charge cord stopped working so went to Facebook and found someone selling a charger for it. The charger was from amazon they said. It was an obvious fake and looked nothing like the company provided one but tested it and the voltage output was correct so used it. Fast forward several months and last week I noticed the casing around the cords power supply was deformed and had at one point almost melted. Scary to say the least.
Speaking as someone who feels very alone and plays mmorpgs: its an extremely poor replacement. Nothing more than an unhealthy coping mechanism, like drugs or alcohol.

Sure its possible to have a healthy relationship with such a thing if you have fulfillment in other areas of your life, but if you don't its not good for you.

I think perhaps it would be possible to design something that pushed people together a lot more, but if you were going to do that it would probably be better to work on actual real life social engineering, making meetup groups etc.
I've programmed for a long time, since before the days of personal computers and graphical terminals (unless you count oscilloscope like vector graphics output devices). So, I'm quite comfortable at the command line and use it all the time. I think it's interesting that there are nevertheless a few tools that I find indispensable which don't run in a terminal: obviously, the "big" graphical apps (like those from Adobe) and web browsers, but even my apple track pad for scrolling works so well that there is nothing quite like it.
> I wish that shell would be more sane.

Absolutely agreed, shell is in many respects terrible.

>I don't think that a shell should be a complete programming language.

On the contrary, I think shell should be a more complete programming language! Drop the stringly typing and add actual types (hence eliminating 80% of bothersome awksedgrep magic; yes, no need to tell me it's a real tall order), add proper error handling...

I truly value simplicity where possible, but I think that the primary interface that I use to communicate with my computer should be as powerful as possible. Numerous times I've built up a shell pipeline only to realize near the end that I need to do something that shell is horrible at, and had to redo the whole thing in Python from scratch. I cannot really see a reason to limit it.

>Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions.

UNIX "philosophy" got a lot of things wrong, but not this one. These common utilities you mention are good especially when they're separate, because it's not the shell's job to improve performance, provide consistency, etc... of a dozen and a half different utilities, a number sure only to grow in time as people discover what commonly used thing they want in their shell. Though I'll give you that `awk '{ print $2 }'` should definitely be its own utility or a built-in.
I think people are used to terms being used when they don’t really mean anything.
While that’s true, not everything can be cached, and many websites run expensive code to assign a session to each new “user.” Larger sites generally learn to avoid that or have the infrastructure to accommodate it, but even moderate-sized blogs and forums probably can’t cope with that scenario all too well.
I have read PG essays for the last 4-5 something years. YC was never under my radar before that. The first time I saw ‘PG essays’, I was excited because I thought they meant PG Wodehouse who is my favourite author ever and I was excited that YC reads Wodehouse together.

Alas..that wasn’t the case. But I don’t regret the PG essays and have come to appreciate them over the years. I think it’s because I didn’t have any expectation about this person’s writing or prior knowledge about PG. But not knowing who Paul Graham was and reading them for the first time, I couldn’t figure out the enthusiasm around it. Maybe it was my disappointment that they weren’t by my PG. But over time, I have come to appreciate it.

These are what I call ‘through my lens’ writings. The words derive weight from the cult of personality. And the lens they see the world through..
In unreal and unity, most of the third party code I’ve tried to use ended up unusable because of either bugs, performance issues, or limitations. There are great exceptions but it’s been frustrating overall. The ones that do work, well, the documentation is often terrible and it takes longer to learn and integrate than it would to make your own. Finally, unity has a history of retroactively changing their terms of service to squeeze more money out of you for asset store buys, so it’s scary to trust them. I write a lot more of my own systems now.
> at the same time, it is a bit of a hack. what is the minimal non-hack version of this technique? seems abstractly like liking a file-descriptor.

Using a "proper" media manager and adding the file to a play queue?
I am probably being naive; but because the Zodiac uses multiple ciphers and eventually a very complex and detailed one, couldn’t we infer he was probably either a computer science or math major?

Clearly this should have been able to filter out potentials suspects and reduce the pool of possible suspects? Just a run of the mill person is not gonna come up with these codes right?
This person used a new IP address for every single request, so that won’t work. And that’s a growing trend.
In case you aren't aware Eagle Mode [0] (a self-described advanced zoomable interface) exists.

[0]http://eaglemode.sourceforge.net/screenshots.html
As a general rule, when engaging with thoughtful people, if your presumption is that it's obvious you are correct and a phrase like "big deal" enters ones repertoire, it's probably a sign you are not arguing the same point, since if it was so obvious then thoughtful people would have come to the same conclusion.

One reason this issue is worthy of debate is because it is novel - we have not been a position before where there are a few global scale 'public squares' under centralized corporate control, through which it seems inevitable the whole world's discourse will flow through in some form or another. Nevermind the fact that these corporations' business models are directly tied to influencing human behavior on those platforms - the incentives align in a direction that seems incredibly harmful both in theory and practice.

As society evolves and advances, particularly an advanced society like ours with many exponentially advancing elements, new emergent phenomenona appear. We ought to not appeal to tradition to guide us in all cases: we could be dealing with something genuinely new that our generation needs to address directly, because our ancestors could not foresee it, even though they tried to create generalized systems. I would argue that is the case now, much like it was the case in the response to large changes in society like the industrial revolution and the creation of nuclear weapons. In fact, your mention of monopolies is an exact example of this: we created this concept in a legal sense, in response to unexpected damages that come from having monopolistic actors. If anything, leveraging the argument of monopolies existing or not in your argument is a tacit concession that such structures can be created and have moral weight in determining how we should act.

In our era, arguing society needs to create new legal and policy structures to ensure social media and these global public squares best serve us all seems reasonable to propose. We seem less likely to do that insofar as people reject the idea that such a discussion is ridiculous, because we already have all the existing mental models and tools we need to address the problems they bring.
The article, as is typical in my experience, loads fine with javascript disabled. Browsing with javascript off by default is the only way to stay sane on the modern web I think. Most sites work better with javascript disabled.
> How frequently are vaccine side effects measured 1-3 years out? How frequently do vaccines present side effects on the 1-3 year timeframe?

Basically never: negative effects are overwhelmingly weighted to close to administration. This could, of course, always be the first vaccine to really do this. Indeed, most childhood vaccine trials have been about a year long.

Much more interesting / troublesome is what immunogenicity is at 1 years / 3 years / 5 years. That's why vaccines have failed at the end of their trials: efficacy isn't durable enough, not that too many people get side effects 2.5 years in.

There's a fair chance we'll find that these vaccines don't offer great long term protection. So far protection seems to be holding up OK... But that's next year's problem.

> Had that judgement call not been made in one of the least competent governments in US history, I think it might carry more weight, no?

We have plenty of other competent regulators making the same choice, and the review process followed seems reasonable. The pressure from the top to speed it up further is yucky, I admit.

> I would challenge this statement in that I don't believe it possible to fully understand the risks yet.

Well, sure. We can't know exactly the probability that it will turn us all into vampires a year from now. We can't exactly know the risks from letting COVID-19 doing its thing unimpeded, either. But the former risk looks pretty small and the latter risk looks pretty big, and it's not like we've not studied both pretty extensively.
You mean like any vaccine? They’re CYAing because no vaccine is 100% effective. It’s that simple.
Thanks for writing this! I enjoyed this article.

The "make" tool is really powerful, besides its use for calling compilers. You can write really nifty scripts with make that have automatically optimal parallel scheduling.
The whole “unalienable rights” of individuals suggests that this isn’t a democratic notion, it’s a moral philosophy and conviction.
I don't think Javascript-the-language specifies how implementations implement them, so there is no clear answer to the question. Actually many languages don't. And runtimes on real hardware have another pile of weirdness on top, so thinking too detailed about O-notation for real code quickly runs into limitations, especially for small-ish n.
Re: Flex Dance: open-source dance pad
If you think the unix shell sucks, try fucking around with the Windows shell - either of them. You're either in a world of legacy and inconsistency dating to DOS and the days of 8.3 filenames, or you're in the also wildly inconsistent, but very verbose, world of PowersHell.
AWS launched in 2006. The cloud was very much a part of and a driver of the earlier boom. It's not something that's arrived since. The cloud, too, is old news.

I'm not taking a position on whether or not YC is "necessary". I'm saying that fifteen years ago it was clear to most of us that YC was making society better. Now? It's really not clear.
[EDIT: tl;dr is if a bully cop targets someone black because they know they can get away with it because the person is black, not white, that's still an example of white supremacy / systemic racism in action.]

No. Fuck off, seriously.

Do you actually believe that about the murder of George Floyd? Do you actually think that cop would have behaved that way if George Floyd was white?

What you said only makes any sense in a fantasy world where you simply ignore the facts of police violence over the last several decades.

Or maybe you’re ignorant, i.e. uneducated, in which case I would encourage you to read more and better material on the subject.
It depends on what you mean by "the field". Do you mean machine learning? That might be the case given the corporate dominance of that technique. But in bioinformatics (the domain to which AlphaFold2 is applying ML to) it is generally expected that all data and code be made available for a tool. Otherwise what's the point? If people can't build upon it, science hasn't been done.
Yeah the way I think of this is that some people write a text or Markdown file with shell commands for documentation.

I INVERT this, and write a shell script with comments :) That way someone else can reuse your knowledge more easily (and your future self as well).

Examples:

Constructing a big curl command to use the Zulip API, and also using jq for the first time. I used this to easily make a blog post out of a long Zulip thread [1]

https://github.com/oilshell/oil/blob/master/services/zulip.s... (oops some tabs snuck in here)

Figuring out how to use uftrace (and successfully optimizing Oil with it [2]):

https://github.com/oilshell/oil/blob/master/benchmarks/uftra...

Though one issue is that shell scripts don't really specify their environment, but there is a large number of tools growing around containers that can solve this problem. (Basically Docker is being refactored into something more sane; thank you to OCI and others.)

So I hope to integrate the Oil shell and containers more so shell scripts are more reproducible. I mean most of the container tools are already command line tools so in some sense it's already done, but you can imagine closer integration (e.g. not just passing code as text from outside the container to inside the container).

----

I wrote some notes about the documentation issue here: http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#...

And one thing I've wondered is if Oil should literally run shell out of markdown, so you can create executable docs. I can see it being useful, but it might be something you should do with a separate tool that converts markdown code blocks to a shell script...

[1] http://www.oilshell.org/blog/2020/11/more-syntax.html

[2] http://www.oilshell.org/blog/2020/01/parser-benchmarks.html
Well, I just showed you, per the CDC’s own numbers, that a third of those had nothing to do with Covid.

A reasonable person might then ask your question, but look for answers that involve something a little more intellectually rigorous than morally scolding strangers. For example: it’s seeming quite likely that our extreme, panicked reaction to this illness has killed a huge number of people.

How many more will die due to postponed cancer screenings, surgeries, and so on? How many children will be set back in their education, permanently? How many minority and low-income families will
be carelessly tossed into generational poverty? How does that number weigh against the people who have died from this virus? Have you even given it a second thought?

But sure, go be “sad” about people sharing documented facts, instead. It’s much easier than critical thought, and it lets you feel morally superior. I can see why you prefer it.
Peak Hacker News right here ^.
> Perhaps if the docs started with a simple example of what has been seen over time to be the most common incantation for each command it might be tolerable.

You might enjoy https://cheat.sh which is usable via curl: `curl cheat.sh/grep`
The inconsistency sometimes bothers me too, but it makes me feel a little better to remember that so many of the CLI commands we take for granted are part of an old historical heritage. The inconsistency is part of that heritage--today's CLI wasn't designed all at once by one group, but rather evolved over 50+ years from many contributors, back when nobody expected that people would still be using `sed` in 2020.

For me, reflecting on that history helps dull the annoyance of having to type `grep --extended-regexp` but `sed --regexp-extended`.
This Johns Hopkins doc claims they took off four days for Thanksgiving:
https://thedispatch.com/p/fda-career-staff-are-delaying-the
I'd argue that it's rarely been decided by a public debate. It's traditionally been decided by (for example) the small number of companies who are in the business of publishing textbooks. Or the small number of people who teach law at prestigious universities. Or the small number of people who decide which textbooks will be bought in Texas, particularly since Texas is a large enough market so that this decision affects other states as well. Or religious figures, laying down a decision about what doctrine can be taught in churches.

Obviously in the past these small groups of people have made the wrong decision, and sometimes that's had very bad consequences. I am not arguing that this is a good methodology. I am arguing that we can't assume that public debate will generate a better set of decisions.
> what is the minimal non-hack version of this technique

Rename the file 00WATCHME
Is there an implication that the bills appoached $20k/month? Whether he owed $1/month or $20k/month, 20k/month is only being asserted as at least equal to the amount of his unspecified bills.