skysight-inc/HackerNews huggingface dataset:
So first things first, PySpark is not a dead, dying, or a dead-end. When Databricks and Spark die, then we'll see the end of PySpark. Adoption of Spark and Databricks is growing. I actually see Arrow errors in PySpark jobs in Databricks, so Arrow/Ursa/VoltronData is already being used in the guts of Databricks/Spark.

We actually see a lot of what you are describing at my current company. All our data is stored in a database that was once very popular, but is old now and not cloud based.

We have a couple of challenges.

1) The data engineering team typically loads data (typically 100s of millions of rows) into the database before anybody really decides how it will be used. Typically we (Data Scientists) only get access to views. The views and underlying tables are not indexed or partitioned as we need, so almost any query takes forever to run since it almost always results in a full table scan of the raw source table.

2) The database team is caught in a budgeting trap. There is a long term-migration to another cloud based database, so both people and financial resources are focused on that. The end result is that no further scaling of the current database provider is possible, which means that every complex query we make on this database creates more load, which makes every other query run slower. This database has a lot of users, tables, and queries, which means that a lot of the available people spend their time just making sure the database maintains some basic standard of performance.

3) Since any complex query (e.g. even a basic date based aggregation) increases the load on the server, user queries should only fetch and filter data. Aggregations and anything complex (e.g. a string operation) should be done downstream.

Based on these constraints, we basically have adopted the following.

1) Once the users (e.g. us) have defined a use case for the data, the Data Engineers/Data Platform team will index and partition the data properly. This typically results in a huge performance increase. Recently we had one view that went from 5+ hours for a basic query to less than 5 minutes.

2) All aggregations and data transformations are done in Databricks/PySpark.

3) Typically, after 2) if we convert the Spark Dataframe to a Pandas dataframe, data dataset is small enough to run anywhere.

One of the things to keep in mind is the people supporting it after your contract is over. I think Databricks/PySpark and Dask are fairly common and well known in the data community. Snowflake can probably help speed up the SQL queries once the data is moved into that, but I don't think it can cover some of the analytical things you can do in Spark.

Arrow/VoltronData doesn't seem like a fit for the use case you are describing unless you have bunch of developers trying to develop their own data engine with Arrow behind it.
> What you're suggesting here is go catch a very serious case of covid and nearly die

Can you please point to where I made such a suggestion? You are arguing against a straw man. My comment was in response to your attempt at scapegoating:

> But I hope we agree that anti-vaxxers suck. And they are providing a pool for the virus to evolve past the vaccines

To repeat myself, unvaccinated people will eventually catch the virus and in the overwhelming majority of the cases recover and develop potent and robust immunity against future infection. Apart from possibly putting strain on hospitals, they are not a threat to your health.

> All of that is pre-omicron which seems evolved to evade vaccines.

This is speculation bordering on fearmongering. There is hardly any data on efficacy of vaccines against Omicron. Although there is some preliminary evidence that reinfection rate due to Omicron is higher: A whopping 1.27%. Which is again, much lower than the rate of breakthrough infections.

https://www.medrxiv.org/content/10.1101/2021.11.11.21266068v...
We’ve got the beets: Food consumers may see red, naturally
Swedish researcher faces prosecution for finding immigrants commit most rapes
Embedded Software Development Services
Embedded software is a binary file written to the device’s EPROM (Erasable Programmable Read-Only Memory) managed by its embedded controller. Without embedded software , a PCB-based device is a “brick” with which the user cannot do anything, since it is not operational. Using this type of software, you can optimize apps installed on the flash memory of the device, as well as expand and reprogram the built-in device functionality (For example, you could change settings of one sort or another).
https://archive.md/gMdLn
It feels like gradually things are getting 'less fun'. We can't have nice things, people can't do jokes, everything is like "your service must be completely professional even if it is just this thing you needed and built and offered to others for free". Everything is SO SERIOUS now, when one thing isnt how a small group feels it should be, suddenly everyone mobs up and more fun disappears.

Is there a generation thing that made this happen? It seems now that no matter what you do, if it deviates from what a paying user would expect, even if the thing is free, that as a developer you can get called up in front of these ad-hoc tribunals of the common man to answer and atone for your sins.

It's so dumb the way the internet is going, I thought when we invented it that things would be so much better. I have nothing but spite for the last 20 years of the way we have treated this golden resource.
Requirements and Test Management for Jira – consistent testing for best quality
Viral social network justice
"Or the real estate market, where basically every single price you see is a lie"

"these rules are stupid"

What never ceases to amaze me about free market extremists, is how much they take living in a civilised society for granted.

Try living in Russia where no-one is keeping fraud in check, you will quickly appreciate why we have these rules. A productive economy is imposible where every transaction could mean your savings dissapear into a black hole
The medium was supposed a soothing massage, and nowadays in some regions/channels the elicited emotion is,

panic

(Consistently with the text: «Innumerable confusions and a profound feeling of despair invariably emerge in periods of great technological and cultural transitions. Our "Age of Anxiety" is, in great part, the result of trying to do today's job with yesterday's tools - with yesterday's concepts.»)
Really not following the HN commenting guidelines here lol. The above commenter was just stating a point on their responsibility to local businesses. From that you assumed way too much about them and their community connections.
Would not that have massive CO2 footprint? It would be very expensive to pay for it. Vulcanic eruption is natural and does not count, but man-made emmisions require permit.
You don't have to go all the way in to be able to obtain energy. Hawaii and Iceland both have geothermal power generation facilities; it's very doable.
Better.com CEO fires 900 employees on Zoom call
Wasm is defined to trap when it accesses memory outside the sandbox (the embedder can decide how to handle that trap, say by shutting down that particular sandbox).

With wasm2c the trapping can be implemented in a variety of ways, for example using the signal handler trick like wasm VMs do (~14% overhead) or manual bounds checks (~42% overhead, but fully portable).
True but you might surprised what that dislike button is actually correlated to.I don't know but I can guess. Often "content creator experience" is way less important than preventing some companies having their campaigns downvoted to death. So there might be a strong commercial incentive which also youtubers don't pause and think who is benefiting from this.
Lol! Weed is barely imported anymore. Seizures at the border have been on a continuous downward trend since 2000. Imports are now close to miniscule. It’s illegally grown domestically, whether that be guerrilla grows in the national forest, or massive warehouses in plain sight.

https://www.princetonpolicy.com/ppa-blog/2018/5/15/we-are-wi...

https://sgp.fas.org/crs/misc/R45812.pdf

https://abcnews.go.com/US/wireStory/oregon-state-police-seiz...

https://www.cbsnews.com/news/marijuana-seized-california-bil...
I get that you're arguing a systemic position and it's one that I mostly agree with but I do think you take it too far - great man don't drive history alone, but they do contribute incrementally to it. The Wright brothers weren't the only ones trying to invent the plane, many had come before and were attempting simultaneously, and it was all based on existing work from those who came before, but they were successful where others were not. That counts for something.

Where did I say that either the great men of history model, or national pride, were good things? My whole point that I explicitly laid out was that attacking symbols that people think are good is not good praxis. That doesn't mean that perpetuating myths is good, but if you want to win people over to your side, taking a shit on thinga they think are good is not the way to do it. It just makes you feel good in your righteous indignation, which is basically the image a lot of moderates have of progressivism in general.
The closest thing is probably companies assuring they can interact with the transaction formats mandated for electronic transactions under the HIPAA transactions and code sets rule, or EHR standards also mandated/incentivized by the federal government.
I discovered Zig a few days ago. I'm evaluating it for use with my STM32 microcontroller.

A few observations:
* it cross-compiles to other architectures without any extra libraries required. The chances are that you've already installed the GNU cross-compiler tools, but it's a nice touch to have an all-in-one solution.
* you can split memory into bitfields. This is useful for microcontrollers, where different parts of a 32-bit register do different things. The compiler can check the field widths, too. C++ does have bitfields, but AFAIK, it doesn't give guarantees as to exactly how the bits might be packed.
* I can have a single registers file, that lays out the memory map of the microcontroller, somewhere in my path. In C, vendor-supplied libraries tend to have definitions all over the place, so a single file is a bonus.
* a module system. You write the build script in Zig itself.
* no macros, but it can perform compile-time execution. I've been wondering if compile-time execution actually provides a whole new way of thinking about programming.
* async/await/suspend/resume. Ooo, a way of doing cooperative programming baked into the language itself. Seems very handy for microcontrollers
* a library system that is part of the language. This means that you could write bare-metal systems, and it will give you things like maps and list. You don't have to try to integrate third-party libraries like newlib. Presumably it doesn't all work out of the box, though. I've not delved into the mechanics of it yet.

There are plenty of other things of well, but those are the ones that jumped out at me if you're interested in system-level work.

Zig is squarely aimed at C, a kind of "C done right". The syntax is fairly intuitive to understand. In terms of size, there's virtually a one-to-one translation between C and Zig. Zig aims for correctness and transparency, although it isn't as strict as Rust.

To give perhaps one example, C99 introduced flexible array members, so you could do stuff like

```
void func(int n)
{
int arr[n];
}
```

And the question is: where/how is the memory allocated for arr? The answer is: probably on the heap. If you're using a microcontroller, though, heap allocation might be a big no-no. You need a memory allocator for starters, and you have to worry about memory fragmentation on resource-constrained systems.

Zig takes the philosophy, if I can put words in the developers mouths, of having a strong dislike of "stuff going on under the hood" that you don't know about. They take and ability to reason about what the code does seriously.
Ah, the classic. In order for your issue to be resolved you have to make the news. Really sad how this applies to many/most online services.
Who knows what they're optimising for?

Other than the start page, Spotify is perfectly fine though.
With the recompilation back to C I fail to see how RLbox prevents memory corruption due to lack of bounds checking or UB being explored by the optimizer.
Was settled https://www.greentechmedia.com/articles/read/walmart-settlem...
I think the most helpful thing with five whys is that it counter acts blame-games within an organisation. Normally when something happens the natural reaction is to blame someone that did something which is counterproductive, it feels good for everybody else but it doesn’t change anything. But five whys is just a method, for it to work there needs to be a true will to actually change and improve things.
Bitcoin's plunge is another sign of market angst
It's better for Main Street if the money from the community is spent in the community and remains in the community, improving the community.

Buying from Amazon just shunts money to Seattle. Not everyone lives in Seattle.
But it's not a lawn. It's a critical tool. Do you want your car's brake system to have easter eggs? What about your smoke detectors?
Requirements and Test Management for Jira
In pizza's defense it isn't often delivered in a "bake at home" state.
It's an advocacy piece. It's meant to push a certain point - not to educate the reader on anything.
Because if these are trading actions that lead to these fines then a rational actor might decide against taking then in the future lest trading become a net negative.
I've never understood where to stop a root cause analysis, and the "Five Whys" has never helped at all. What's the appropriate level of abstraction? Eventually you get down to:

...

"Because the sales targets for the enterprise are unrealistic"

"Why?"

"Because the Board of Directors can't attract investors with reasonable sales targets"

"Why?"

"Because investors are only interested in a company scaling at a certain rate"

"Why?"

"Because investors are greedy"

"Why?"

"Because when God first created Man, He also created Temptation..."

The concept of a "root" cause is silly, because the roots of a plant spread far and wide and have many different ends. In the analogy, if the plant is the "problem", and the root is a "cause", then there are hundreds of different roots (causes) that contribute to the plant (problem). People are always trying to assign a single "root" cause as if there is one. Many causes contribute to systemic failures, and often times fixing any one of the root causes would have delayed the failure for another cycle, without actually addressing the procedural risk. Your technician accidentally skipped a step, sure, but he skipped a step because he was in a rush, and he was in a rush because he was overworked, and he was overworked because of those damn sales targets, and...

If we're interested in "root" causes, then we need to end up with a list of several causes that combine to result in the problem. If we only want a single cause, we should call it a "seed" cause, as defined by "this problem (plant) is guaranteed to happen (grow) every time this cause (seed) occurs (is planted)".
Iceland generates 25% of their energy geothermically https://nea.is/geothermal/
Biden administration expected to announce diplomatic boycott of Beijing Olympics
Has the cmd on windows an equivalent to .bashrc? If yes - TIL
"Testing Firefox more efficiently with machine learning": https://hacks.mozilla.org/2020/07/testing-firefox-more-effic...
They are showing your argument is invalid in the first place.
There is plenty of geothermal. Where there’s not, it’s usually because another local form of generation is more attractive (e.g., hydro for volcanos near the sea) or the volcanic system is environmentally or culturally protected.
Works great, I just hope they figure out GPU passthrough and potentially Looking Glass integration in the future to make things easier for Linux gamers who need a VM.
Coal-fired power is on the rise in America for the first time since 2014
It's wild hearing that in other countries drug dealers still accept cash and meet people in person. Although prices for practically everything are way higher here (10g would cost you about 200$ or more), the convenience of using bitcoin and not having to meet a single soul to make a purchase at any hour of the day still outweighs it for me.
> falls apart when disassembled

Funny, in the 30+ years that I've had IKEA furniture, I've moved several times. That you can disassemble IKEA furniture is a benefit when moving, and I've had no issues disassembling and reassembling multiple times.
This is what my dissertation was about, in a different field: Everyone using the same datasets, even though they are severely flawed.

As it turned out, differences between datasets proved significantly larger (by a big margin) than differences between algorithms. And the most popular datasets in fact included biases and eccentricities that were bound to cause such problems.

https://bastibe.github.io/Dissertation-Website/ (figures 12.9-12.11, if you're interested)
LOL amazing metaphor.

The one thing that I've learned recently which surprised me- if you want to cluster MySQL (via partitioned tables or PlanetScale, etc) you cannot use foreign keys.

Some of the Postgres compatible tooling (Citus, Cockroach, etc) appears to support foreign keys across partitions / shards.

I was quite surprised to see this. Makes sense that not allowing FKs makes the data easier to shard but I never imagined that would be a thing you would have to contend with.
It's almost incredible that for something as "simple" as an air quality sensor, particularly in relation to PM2.5 (the most important metric for most buyers, IMHO), there's no simple answer as: product X costs $50 and provides 99.X% accuracy; product Y costs $200 and provides 99.9X% accuracy.

Two years ago, during one of the most intense fire seasons in California, I spent a few hours studying different solutions, and ended up buying a refurbished PurpleAir sensor. It seemed to work great, but I felt it was a bit too expensive for what it was doing, too bulky, and in general not as easy to use for the ordinary person as it should be.

Same problem with home air filters - if you want to purify the air in your house, there's too many snake-oil solutions out there, and few that are worth the spending.
Henry Cavill says his dog, Kal, has saved his mental health
Stuff like this makes me appreciate that most of the software I use manages to innovate without throwing out the UX every 6 months. I use Ableton Live and Reaper a lot. Both look more or less like they did at launch 20 and 15 years ago. You could follow a manual for the first version and only be a little lost with newer features since they all follow the same UX conventions.

I had to crack open Google Analytics to grab the tag to paste into something. It took me 2 minutes to find it. WordPress is more or less the same if you go to /wp-admin/, but the infiltration of the new UX into it tells me that's on the way out.
Can't wait for Firefox XP.
I have copies of Newton's PM, W&R's Pricipia and The Elements for post-apocalyptic reading ;)

For someone with little computational theorem proving, but not new to real analysis, would it be best to start with Lean vs. Coq? I have heard that HoTT has put Coq up front for this, but that Lean is much easier to get started with as someone with a mathematical background vs. a computational mathematics background, and that Lean tops out after you really dig in. Any tips would be much appreciated!

Thanks!
I'm already vaccinated. I'm not getting the booster. My age group has like a .0007% chance of dying from COVID. A booster isn't necessary. Stop with this anti-science cultish behavior. It's weird. COVID is not going away, getting boosters indefinitely is not a sound plan. I'll take my chances and get the natural immunity which is better and longer lasting.

I don't want advice from Germany. You'd think after what your country did they'd be less inclined to go down the authoritarian route....
I'm willing to bet the guy who did that, did it to humour this crowd rather than for Air France and his support ticket. Otherwise it wouldn't make any sense for it to be written in English, considering he's quite obviously French [1].

[1]

- *one hour* is written as « 1h ».
- He uses one space after ever punctuation mark, including commas.
- He knows what CNIL is.
- He uses Air France.
You would be surprised how much particulate certain cooking methods produce, your home can go from 0 AQI to forest fire levels for a few hours without a filter.

With the right filter and sensor, you can keep your AQI in the single digits consistently and react appropriately to particle buildup. If you have a CO2 sensor as well you can react to CO2 buildup which is something I have experienced plenty of times in a modern apartment (not necessarily dangerous levels but levels high enough that cognitive effects should be expected). It is hard to keep indoor CO2 levels anywhere near outdoor levels without forced ventilation with the outside or living in a large house.
Fresh milk delivery seems like something that could definitely make a comeback though!
I hope most of them, 91 million lines of code is insane
It is when you are probably wrong and in disagreement with most economists, and don't post any citations or evidence. Like, argue for a minority position, but argue, don't just insult.
> the researchers present evidence that minimum wage changes led area restaurants to raise prices, change menu items, obtain lower rents in the high wage areas and, in some cases, caused eateries to shut down.

I wonder if the units made empty by closures may have reduced price pressure on the ones that survived.
Run any desktop application in the browser
Companies raise prices to what the customers are willing to pay. It's not related to costs unless the cost exceeds price.
> you can’t trust that the thing you ordered is what will actually be delivered

I see this all over HN, news articles, and reddit ... yet I've never experienced it. Nor has anyone in friend/family circle. I don't doubt it happens, but it makes me wonder what kind of purchase bubble I'm in that prevents such an observance. Is this more prevalent with certain types of goods?
There is that risk, yes. For wasm2c to be correct it must emit C code without undefined behavior. As best we know it does that properly today, and we've tested and fuzzed it quite a lot, but whenever you use an optimizing C compiler on the output there is no 100% guarantee.
Yeah, next time I'm on the highway and go into my foot-brake's "About" menu and click on the word "2020" twenty times I might be in serious danger of crashing my car!
It’s funny how Google’s automated system will flag a small video like yours but I see tons of crypto scams on SpaceX imposter live feeds with thousands of viewers
Does the consumer even really win? Once Amazon/whoever has a monopoly, they get to decide when and how to innovate, and the prices. Is the product now as good and as cheap as it could have been with more competition? Their PR and lawyers will say “yes” and who can prove otherwise?
Triggering a fire inside the castle might be enough to change the output of calls being done into the sandbox, it doesn't need to escape it to be exploitable.
I believe the fundamental problem with BuzzFeed is that it is built on a foundation of clickbait. Does clickbait work? Absolutely, but only for a while before people adapt to it. The reason is that clickbait is inherently disappointing. Truly interesting content does not need to be fluffed up, while most content on BuzzFeed is made to look more intriguing than it actually is.

I believe they are conditioning their users to be interested and then let-down after each article, listicle, video, piece of content, etc. They seem to be doing well in the short term, but I do not think their business model / hotness will last into the future.