skysight-inc/HackerNews huggingface dataset:
Around a decade ago, I was part of the team responsible for msnbot (a web crawler for bing). There used to be robot.txt (forgot the extension now). Most of the website was giving 10-20x higher limits to googlebot than rest other crawler.

Google definitely has unfair advantage there.

Bing and duckduckgo still provide very reasonable result with 10-20x less resources but not at par of google.
Force americans to pay more taxes, take no responsibility, kiss China.

Who matches this description?
Mentioned some guidance in a separate thread, but encoding/sanitization of dynamic content is crucial. Also, dig into the options for the tool and disable features that aren’t needed. As an example, wkhtmltopdf enables you to disable local file inclusion except for predetermined locations.
Contracts are actually quite necessary for modularity. In OOP we have data types (interfaces, classes).

As long as other paradigms provide contracts/abstractions, modularity can be achieved there also.

Faked modularity doesn't count. Leaky abstractions neither.
> Perhaps there could be some kind of 'Crawler consortium'?

An industry-wide agreement not to compete for commercially valuable access to suppliers of data?

Comprised of companies that are current (and in some cases perennial) focusses of antitrust attention?

I think there might be a problem with that plan.
A quote I stole from some conference talk that i think of all the time as a manager: "You become the manager you wish you had, you need to become the manager your reports wish you had."
I think they’re going to lose this. One of the most basic concepts we covered in contract law in the crappy undergraduate business school was the tort responsibility through the entire lifecycle of a product.

If I have a lemonade stand and sell defective lemonade, I’m liable for it. Even the defect was caused by the lemonade mix, even if the defect was caused by the red dye #5 manufacturer. We’re all liable, not just the root.
Back when I started uni, they had a whole room full of old Apollo Domain/OS machines. I think they were previously used for CAD or some EE stuff, but even than (late 90s) pretty much only used for people to do their email (pine, mostly).

Anyone got some actual war stories about the soft- and hardware?
That won't make a difference unless the underlying algorithms can be perfectly parallelized. That's the whole point of Amdahl's Law.
> Ars seems to be calling all open source software insecure

> The article starts as a gotcha piece

Haha, that’s the problem. You, Netapp, value saving face so very much that you are incapable of constructive response to well-meaning, fair, and honest criticism.
What a weird reasoning. In most contexts I've seen, "professionalism" doesn't really do anything except strip away all the human factor that goes into every interaction. Personally, I don't care whether the person I'm talking to is being "professional" or not. What I care is that they're respectful and can properly communicate their thoughts.

With that mindset, emojis (and emoticon) can actually add context to interactions, considering how much context is lost when communicating over text. A simple smiley face at the end of a sentence can go a long way in my experience :)
Did you notice how your budgeting / estimation guideline converges to "multiply the estimate by Pi" advice?
What is a string an array of?

If you ask the user, it's an array of characters (aka extended grapheme clusters in Unicode speak).

If you ask the machine it's an array of integers (how many bytes make up an integer depends on the encoding used).

Nothing really considers them an array of code points. Code points are only useful as intermediary values when converting between encodings or interpreting an encoded string as grapheme clusters.
Potatoes will keep for weeks as long as you keep them dark. They don't need to be particularly cold. Do people really keep potatoes in the fridge?
Good word, what a title. This is the guide to compensation as practiced by Whimsical, Inc. Not a light-hearted explanation of the hijinks that your employer keeps your wages down with.
It's sorta not surprising that unicorns lose a lot of money. You only get a huge valuation by being a company of global scale, and that's always going to be something that spends a lot of money on expansion.

I do think that there's way too much focus on unicorns, and a lot of good but not earth-shattering ideas have a hard time connecting with seed money. But if you do want to look at the biggest profitability for investors, that's not going to be reflected in the P&L statements this year or even five years from now.
Then someone can start a competitor up again, right? Assuming there's actually a market for it.
Sometimes I wonder if the world would be better or worse if there was a max size for companies.

It’s also interesting how many chat apps are acquired and exist. Just how large is that market?
Year 2025: "I work for Uber. My job is to sit in the self-driving car the whole day. The car can drive itself, but the law doesn't allow that, so I have to sit there. My title is 'legal compliance associate'."
This is an argument against centralization more than it is against government.

"One index to rule them all" seems more fraught with difficulty than, "large cloud providers are unhappy that crawlers on the open web are crawling the open web".
Ha! I picked up Starting Strength from HN too. It transformed my workout routine and changed my understand of strength building.
In this case, they bought it, bought it, bought it.
Hah, no, but thank you, new account created to comment on intel related threads. This question of how technologists should deal with pressure of all kinds would benefit from peoples experiences and more mature frameworks of ethical considerations. It's not unique to spies.
This is exactly why I don't play games like factorio or tis-100. It's too much like work for me.
>> While it might be a failure from the taxpayer's and military's point of view, it's a raving success for Congress and the contractors. The metrics for the latter are jobs, grandstanding, campaign donations, kickbacks, crony benefits, and mountains of cash, plus guaranteed more mountains ongoing until it's killed.

> I wish this was the cynical view but it's more like reality of government procurement.

> I'd argue that three distinct aircraft could have accomplished that goal better too (as they could be assembled in different places by different companies, with different US-based suppliers).

Which I think is proof that the GP is wrong. If all Congress wanted to do was spend cash to create jobs, they could have done better than the F-35.

I think Congress and the military were honestly trying to save money by building the F-35, by getting more bang for their buck by filling more roles with it. It's just that they failed at that. Some people can't help but assuming malice when incompetence is a far better explanation.

Lockheed spread the work around to make it more politically difficult to kill, but that was for their own profit-seeking reasons.
I am a proponent of Certified B Corporations [0]. You can look up their directory to see the companies certified for a particular industry [1]. From their main page:

"Certified B Corporations are a new kind of business that balances purpose and profit. They are legally required to consider the impact of their decisions on their workers, customers, suppliers, community, and the environment. This is a community of leaders, driving a global movement of people using business as a force for good."

Take a look!

[0] https://bcorporation.net/
[1] https://bcorporation.net/directory
Small nit: in table 2 with the columns labeled "Key, Hash, Hash modulo 16", the thing called "hash" is actually known as a "prehash", while the "hash modulo 16" is the hash. In the context of hash tables, pre-hashing is when you take something like a string or a data structure and turn it into a number. Hashing is the mapping from that number to one of the hash table bins. The distinction is often lost in practice, but if we're talking about hash table design and implementation it's important to make this clear at least once.
Use an object?
You're wrong on a lot of facts here. Google Flights doesn't get its data just by crawling, they get it from Sabre, the FAA, Eurocontrol, etc. Airlines are, obviously, extremely pleased to disseminate this information. Google Flights "gives back" in the exact same way as any other travel outlet: they book passengers.

As for Wikipedia, the WMF is quite happy that most of their traffic is now served by Google. WMF is in the business of distributing knowledge, not in the eyeballs business. Serving traffic is just a cost for them. The main problem has been that the average cost for Wikipedia to serve a page has gone up, because many readers read it via Google, and more people who visit Wikipedia are logged-in authors, which costs them more to serve. I'm sure there's an easy solution to this problem (for example, beneficiaries of Wikipedia can donate compute facilities and services, or something along those lines).
I see this argument being raised a lot, but there's not a ton of evidence suggesting the problems with the F-35 is that it has to have these three different variants. The lion's share of the issues with the F-35 have to do with the acquisition strategy and the avionics. The acquisition strategy of procuring the planes before they were fully operational, like CI/CD but with fighter jets instead of aircraft, has made everything much more expensive and has really hurt availability because you have a dozen slightly different builds of the same plane, and across three variants. Any time an issue is fixed they have to roll it out to all of the different builds, which adds to costs. The second major issue is the advanced avionics and the software for associated systems, like the MDL and ALIS. In short, it's a software problem. Neither or these are really related to having three different variants.

It's not really fair to compare this to the F-111 debacle, because there we genuinely saw incompatible intended uses for the same plane. The Air Force wanted a tactical bomber, while the Navy wanted an interceptor, these are very different designs. But for the F-35 everyone wants the same thing: a multirole fighter.
Wow! What an amazing write-up and find!

It’s also amazing that they noticed the subtle difference in the NFS packet capture.

I can’t wait for the rest to be published.

Bookmarked
Rectitude : a procedural generation editor for pixel art and generative art. https://lbarret.itch.io/rectitude

Starting to make cool stuff with it :
- https://twitter.com/LBdN/status/1345931367738175488?s=20
- https://imgur.com/a/BJwZFdU
Tried and convicted of crimes or bullied by police officers into accepting some minimal sentence via plea bargain, even if they had no relation to the crime at hand
from Igor:

>I think he had some suspicions, but he is denying that vehemently ;)

https://twitter.com/IgorBog61650384/status/13753134251323146...
Get a linkedin login and start looking. Seems like recruiting either are focused on other teams, or don't have enough context. Both of those can be helped by you doing some lead generation.
We are already there. Current day communist China has very little to offer in terms of universally appealing culture but they are happy to ride deconstructionist woke to nullify western culture(s)
WTF Is Transfer Learning?
Thierry Henry leaving social media until they start fighting racism, bullying
Todays microservices (with containers and kubernetes) are so complicated that entire fleets of engineers are required to keep them from just falling apart. They make tools to automate a bit, that tool fails. So now, more tooling is needed to detect and failover this in-house tool. Now we're dependent on many other services simply to maintain another in-house tool. Then a new hire comes along and brings more tools.

The analogy is civil engineers have started building a structure with new and shiny tools/materials. With new materials, the buildings collapse in many new ways so they build tools to stop them from collapsing. Only now, these new tools need constant management and handholding by thousands more civil engineers. If one of these tools fails, the structure collapses.

While I'm not opposed to tooling and automation, the constant patchwork over patchwork makes enterprise Java look more robust and humane.
Wikipedia isn't monetized. Doesn't it benefit them if Google is serving their content for free and people are finding the information they want without having to hit Wikipedia??

And also, isn't Google the largest sponsor for Wikipedia already? In 2019 - Google donated $2M [1]. In 2010, Google also donated $2m [2].

[1] https://techcrunch.com/2019/01/22/google-org-donates-2-milli...

[2] https://en.wikipedia.org/wiki/Wikimedia_Foundation
I'm guessing that rm is trying to update the metadata for the file which is what is actually failing because of a lack of disk space.
It's really not a different question. Either way if my other son also had Down Syndrome, I would accept him and love him for who is. That's our job as parents, right? BTW, our odds of having our first son with Down Syndrome and autism were about 1 in a million (maybe higher). Our odds of having a second child with Down Syndrome were 1 in 100. That didn't stop us.

I don't mind answering questions about our experience. Hopefully this line of questions isn't trying to prove a point.
I tried DSP, very fun game but the performance is terrible. i7-9700K and RTX2080 and I get 20 FPS before using even 1/10th of the first planet. Can't recommend it currently.
> I've noticed that when you put a bag of potatoes in the fridge, they will still keep far shorter than if you had them in a dedicated potato cellar.

That's because fridges are entirely the wrong place to store potatoes, potatoes react to light, you're better off keeping them at ambient temperature in the dark (and regularly trimming the sprouts) than you are keeping them in a fridge which gets lit up 15 times a day.

For other veggies, it might be that they're e.g. ethylene-reactive, a fridge is a sealed environment so any gas is going to stay around. Humidity is also a factor.
An interesting figure! For easy comparison with the figures in the article it is around £10,000 / acre year - makes you wonder what some of the landowners listed would do in such a situation (and makes it obvious why they would fight very hard to prevent it coming about).
Bill Clinton? Most American presidents starting from Nixon and going until Trump?
context: https://www.youtube.com/watch?v=SaVTHG-Ev4k
Amazon's won these cases (where the defective product came from a seller on amazon) before on the premise that they're not your lemonade stand, they're the flea market where you set up your lemonade stand.
Google makes $150+ billion from Google Search per year. Running Google Search could be operated for likely (much less than) $10 billion per year.

So, Google is in effect taxing us all $140 billion per year.

It's not dissimilar from how Wall Street effectively taxes us all for an even larger amount.

In both cases, we could use some kind of non-profit open system to facilitate web search and stock trading.

The Great Lie that Google is doing a good thing by charging money to insert "relevant ads" above the search results is totally wrong. If those ads are the most relevant, they should just be the top organic results, obviously.

Google mostly solved search 20 years ago. There's really nothing that impressive about Google Search in 2021. It should be relatively easy to replace it with something open, leveraging the massive improvements in hardware and software. It could operate like Wikipedia or Archive.org. The hard part is probably getting the right team and funding assembled.
It's their job to sex up the headline and ours to deflate it to size.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I've replaced the title with representative language from the article body.

https://hn.algolia.com/?dateRange=all&page=5&prefix=true&sor...
The trouble is that the factory costs something too. If you're only running half the time, now you need to spend twice as much on factory to make the same amount of stuff each month.

So if we build a wind/solar grid without enough storage, this sort of demand management isn't a free lunch. It's just that instead of the power company paying the cost of intermittency, it'd be externalizing that cost to everybody else, who pay in higher capital costs.

(For aluminum in particular you have another problem: if you shut down more than about six hours, the molten aluminum freezes and you spend a lot of money on repairs.)
There are definitely crappy things about Unicode that are separate from language.

- Several writing systems are widely scattered across multiple ‘Supplement’/‘Extended’/‘Extensions’ blocks.

- Operators (e.g. combining forms, joiners) are a mishmash of postfix, infix, and halffix. They should have been (a) in an easily tested reserved block (e.g. 0xF0nn for binary operators, 0xFmnn for unary), so that you could parse over a sequence even if it contains specific operators from a later version — i.e. separate syntax from semantics, and (b) uniformly prefix, so that read-ahead isn't required to find the end of a sequence (and dead keys become just like normal characters).
Because controlling timing of signals is easier than controlling level. You can take advantage of the fast and low-jitter clocks already made for digital electronics.

If you have a multi-bit DAC, it's difficult to made it linear. With a 1-bit DAC, it's much easier, because you only have two possible output values. And if you run it fast enough, you can apply a simple analog low-pass filter to the output and convert speed to bit-depth.
Why Is Ethical Hacking Important?
I just don't see this working out legally. How would it even work?

From the "learn more"

> Sometime soon we will be publishing what we think should happen and what we think will happen. These two futures diverge and we believe that, while the gap between them exists, it will entrench Google’s control over the internet further. We believe that nothing short of socialization of these resources will work to remove Google’s control over the internet. Our hope is that in publishing this work right now we will let the genie out of the bottle and start a process towards socialization that cannot be undone.

Sorry, but I deeply skeptical of this. This sounds like the first step towards a non-free internet. At the end of the day, it is your box on the web, and if you want or don't want someone/something to crawl it, that is your call to make.
Right. My biggest fear during the pandemic was a supply chain collapse, which would be FAR more damaging than the disease itself. Pleasantly, it hasn't yet materialized.

I think the chain was fragile, in that we immediately saw disruptions and shortages... but also adaptable, in that people responded and corrected them. Few problems can stand against a million competent businesses all working to fix things.

I want to see a detailed, process-oriented documentary about how the toilet paper manufacturers and distributors responded during the first few weeks of the pandemic. I bet there were damn near heroic efforts put out by some individuals, just to keep the supply flowing.
I have an idea: remove the art of web crawling from the domain of a single company and instead create a international group of interested parties to run it instead. I'm thinking broadly along the lines of the Bluetooth SIG. Maybe it will be a bit more complicated, and require international political efforts, but it will make the search engine market way more democratic.
I became a software engineer because I am a Bay Area native that got priced out of the market by software engineers.
Nope.

Types can completely replace tests. The tests described in the article are ATDD tests, not TDD or unit tests. They're the result of a specific business workflow that has a goal of making sure you're building things that are useful. If you don't have that workflow, you can just use types to do the same work.

And nope, no magic tool required. Sure, tooling and automation would be really cool, much the same as having a C compiler beats the heck out of writing assembly, but it's not directly relevant to the points in the essay.
So what's the actual number for S1?
The low end estimate for the cost of completing ITER is $22 billion, and it’s a 500 MW (thermal) output reactor.

The cost estimate for batteries was about $137/kWh last year.

Therefore, ITER costs about as much as ~160 GWh of storage, which is enough for the entire UK for 4 hours 40 minutes… or the thermal output of ITER for 13 days, 9 hours. Which is enough to cover even the least productive production gaps in combined wind and solar output. Also, those batteries are rechargeable and last at least 1000 cycles, which would be a bit more than 36.5 years of drain (even with negligible recharge time), which is 16.5 years longer than the planned lifetime of ITER.

And remember, the batteries are made at a profit, even including the cost of building the factories.
You don't have farmers filtering the red out and selling it - its also whey a lot of UK farmers love Diesel Landy's
The kind of eyes that can spot the hinky pattern while watching that monitor are the vital ingredient, and thats not something i can quantify. Or even articulate well.
this only takes care of the client, the account must be scuttled as well.

>>The most common cause behind uninstalling discord is to fix any kind of issue. But if you are planning to switch to any other application and want to quit discord then you need to delete your discord account.<<

https://support.discord.com/hc/en-us/articles/212500837-How-...
I remember when rich snippets (one type of those widgets) came out there were a lot of funny examples. One for a common query about cancer treatments that pulled data from a dodgy holistic site saying that "carrots cured most types of cancer" (or something like that).

There was a similar one where Google emphatically claimed a US quarter was worth five cents in a pretty and large snippet graphic.
Unfortunately the smaller companies don't have a choice. Many acquisitions from Big Tech come with a threat - sell your company or we will move into your space and tank your valuation.
My freedom to be protected by the police includes being able to not be protected by the police when I don't want to.

I don't think this freedom to be protected by the police is a thing
Oof. The number of 4b's I've had to deal with in my career...

What are some techniques to use when you see this happening? I frequently try to get stakeholders to simplify and really identify the core problem / proposed solution, but I run into a lot of "all or nothing" or "design for every possible use case" folks in startups.
Try to think beyond Trump? You're almost there.
That depends, would Google let us know?
>You can get off an airplane just about anywhere and get a taxi at the taxi stand, but that never enabled a global company.

Because it didn't streamline the process as much as Uber/Lyft (and the use of internet). It's a big difference when you don't even need to talk to anyone.

>but you do it once per trip, just like the in flight entertainment app.

And that is enough friction to make Uber/Lyft worth it. The in flight entertainment app has a monopoly. If there was sufficient broadband access on the plane, no one would get the in flight entertainment app either.

However, I might download a new local app if I found out the local app's fare were sufficiently cheaper, since the friction of downloading a new app is relatively low.
Yes, but they lack enforcement.