skysight-inc/HackerNews huggingface dataset:
Indeed, those are both great options when possible. But easy access to parallel computing is still quite useful.

For interactive analysis or building statistical models, you probably do indeed still probably want your data fit in memory. But often it's most useful to make your data smaller by calculating some sort of summary statistics instead of subsampling. For example, if you're interested in climate change, you might want to work with monthly means instead of the original daily or sub-daily data. Currently, climate scientists usually do this sort of thing with command line tools.

As for machines where datasets fit into memory -- that's also great, if you have access to them. But even then, for most operations numpy will be limited to a single core. Calculating the mean of 51GB of data is still pretty slow, even if it already is in memory. Your machine with 256 GB of memory almost assuredly has 32+ cores to go along with it, and it's a shame to let them sit idle.

This post by Nikolay Koldunov gives some more context about the value of dask.array for weather data: http://earthpy.org/dask.html
I know it's cool to bash MongoDB but it is really nicely integrated with Spark, extremely quick for a write workloads (3000 writes/second on our slow drives and that's inside the RDD map) and doesn't flinch even when getting it to write 1 billion rows in quick succession. One thing that is really nice is that being schemaless you don't have to worry about setting up tables structures beforehand.

You can look at Cassandra which is historically known for exceptional write performance.
Nice attitude! The fact that your curiousity drove you to figure out the problem long after its solution had immediate utility...that's always a good feature for a programmer.
Hmm, are any of the handset manufacturers making profit? I think Samsung's phone division is losing money.
"You guys" is gender neutral to me.
I wonder if they could create their own android app market that had a more rigorous audit process.
Fact is, it just isn't true. As a Googler, I am surrounded by enormously talented engineers, the likes of which I rarely encountered on the outside. They love what they do, and it shows.

I have no doubt that there are false negatives/positives, but I am convinced that Google is doing something right.
I don't know why, but I really don't like saying or hearing "y'all"
JDBC works great if you have a large RDD that you want to persist in one go.

But if you are writing row by row you will need to implement your own batching algorithm and connection pooling to get any decent performance.
How was that snippet an improvement to your coding style? The original was extremely clear, and now someone who reads your code has to understand what (,) <$> and <*> do. IMO this tendency for code golf one-upmanship is a huge detractor for anyone wanting to learn the language.
I faced similar experience but here in India , since good interview questions are rare , most of the times questions are repeated from forums such as Career cup , I was asked questions irrespective of my role . I do take interviews at my current organization and ask simple practical problems with very simple solutions but most of the times candidates think in terms of complex data structures instead of simply trying to solve a problem, they do premature optimization and what not .
A slightly...biased perspective?
Could someone explain 'applicative'? To me it seems like throwing some magic syntax at a simple usecase to make it more terse and less readable - but maybe I misunderstand and it's something every Haskell programmer knows.
Your kids were without their parents for 90 minutes, so we're going to take them for 30 days.
I agree. The real question is why did even European cars go boxy in the second half of the 1970s? A good example is the Ford Cortina Mk 3:
http://en.wikipedia.org/wiki/Ford_Cortina#TC_Mark_III_.28197...
vs Mk 4:
http://en.wikipedia.org/wiki/Ford_Cortina#Mark_IV_.281976.E2...

I think late 70s/early 80s boxy cars were the anomaly - all other eras were curvy.
I like this idea for small firms, but how does this scale to the level that the parent comment is asking about? If we want to interview a few hundred candidates this month, who finds the few hundred open-source project tasks and audits them for appropriateness?

And, for what it's worth, the peg game problem sounds to me like standard interview fare: an interesting puzzle that bears little resemblance to real-world software challenges. Why is it a better alternative?
I always try to find out if the candidate could - with enough information - solve the puzzle.

Can you, given time, figure out a good solution to the problem I am asking? How would you go about solving the problem? What is your methodology?

I agree with you. These sorts of problems are really to understand how someone reacts to a new idea that he or she does not understand. A lot of times people tell me that they don't know they answer to a problem, which is fine. Knowing that there are things that we don't know if a sign of maturity, and the curiosity to figure out how to solve a problem is what we should be looking for.
Not sure if you have used it but Spark is exceptionally good at data movement.

In fact that is what a lot of people initially started using it for (as a replacement for Hive/Pig). You can write SQL against HCatalog tables, do some transformation work then write the results out to a different system. We have hundreds of jobs that do just this.
I don't think inverting binary trees is the problem per se. I mean, algorithms are the bread and butter of programming. You should be able to implement simple ones like that even if you'd never actually do that in practice. The real problem in my opinion is the whiteboard and time pressure.
As someone who also recently joined an all haskell team, but hadn't coded professionally in haskell before, I'll add the biggest thing I've noticed to the "how it feels" list:

On a biweekly basis I'm shown something new causing my brain to melt, leak out of my ear & pool onto the floor near my desk. It's awesome[0]!

Today's was using impredicative types + `$` has a special case for dealing with it[1].

[0]: ...some number of hours later when I've collected my thoughts and think "OHHH... duh"
[1]: https://ghc.haskell.org/trac/ghc/wiki/ImpredicativePolymorph...
I'll be the contrarian and claim that inverting a binary tree is a perfectly reasonable interview question. (Although not anymore now that it's famous). It combines basic background knowledge (what is a binary tree and how is it structured?), ability to reason logically (binary tree algorithms are often recursive, "inverting" a tree with no children is a no-op, otherwise you want to swap the children, and the children's children, and...), and enough familiarity in one language to write simple constructs without an IDE or stackoverflow holding your hand. None of this requires you to be Jeff Dean.

I will agree that the guy who kicked all this off has a legitimate complaint if the recruiter failed to adequately explain the interview process, and I'm sure there are lots of bad interviewers that focus on meaningless details or obscure trivia. But I don't agree that the concept of having candidates write code is fundamentally awful.
I'm most excited about the collection of documents in their GitHub repo. I've casually tried to build my own collection, but most media organizations aren't very good about consistently providing the source documents in an easily downloadable format.

https://github.com/transparencytoolkit/nsa-data
Arf, it was a blog post, and IIRC in sexp syntax. Good find anyway.
So instead of giving them an algorithm problem that they might have seen in the past (or seen something similar) you give them a random open source project -- along with all the idiosycracies it certainly contains -- that they are 100% unfamiliar with and ask them within a few minutes to make sense of the entire codebase and produce a meaningful contribution? Sorry, but that sounds even more cruel and unusual than asking a very obscure algorithm question.
Work sample tests, of course. Basically everybody who read the literature knows what is the better alternative.
Wow, I did the exact same thing, for exactly the same reason. My dad called me a few hours later and said, "I finished the 38000 word article you sent.". I checked and he wasn't far off; it's around 29000 words.
Time Lapse- Pregnant to Baby in 90 seconds in 2015
Have you encountered anyone outside?
Sure there are talented engineers there but so are engineers in startups and other companies. I'm afraid that's a very myopic statement.
There are even more subtle issues of updating than were touched on in the post. A lot of meats (especially chicken and pork) are much different than their equivalents of even a few decades ago. As a result, recipes in even relatively recent cookbooks like Julia Child's classics will "work" but they're probably not ideal for today's leaner pork and more tender chicken.
The last interview I did was for a node.js position. I was told to bring my laptop. When I got there, the put me in an office and handed me a packet with instructions on checking out a nearly empty git repo. Then they said "Write an API that meets as many of the following requirements as possible and push the code. We'll be back in 3 hours."

After 3 hours, I was taken to a conference room where I demonstrated the API. I also talked about a bug I had that had hung me up, but I fixed it right there in the conference room.

Got the offer. It was a great experience. I enjoyed the challenge, but it felt useful and related to the work rather than just esoteric problem that would not likely ever be required by the job, and if it was I'd have time to research it and figure it out.
There seem to be a lot of 'minor' effects. However, the most convincing argument I have seen was simply increased thermal conductivity by partially melting the surface below the container. A good test of this would be to start with a warm container and then add hot or cold water to it.
I love them too. Lots of brilliant people have trouble solving problems while a group of strangers are staring at them.

Part of the interview process is trying to find out what type of person you are dealing with.
I was told by someone many years ago to always take a copy of my resume with me to an interview. That advice has paid off a couple of times. These days I also keep an email draft ready to send.
CPS are disgusting scum. This kind of story is reported every week. The government is criminalizing parenthood. Kids can't play in their backyard, they can't walk to the playground, they can't ride their bikes. Do people remember their childhood? When did Americans become so socially retarded?
Ask HN: Ideal language for running the systems of an Orion class spaceship?
In science fiction tv, movies and books we often see and hear about systems and sub-systems working together to run the entire ship. And almost always there are scenarios where the "OS" for the ship allows quick hacks to re-purpose modules or re-route data/power across areas.

Being a developer, I have always wondered whether there is a language currently that can withstand the rigours of deep space? Or it is something yet to be invented?

Such a programming language should have certain features - fast execution times, versatility, good driver support for hardware components, ability to script in, baked in support for messaging,etc.
Their recruiting for foreign agents. Which means what the agent could be doing might be illegal in their home county. That's why they don't want applicants to use their home or mobile phone.
I recently interviewed for (and got) a new job.

The interview process took about two weeks and on the whole was pretty reasonable for both sides.

There was a short phone screen (~30 minutes)

Then I had two technical exercises to do. For each, I was given a reasonable time period (4 hours) that started when I visited a special link to the get the problem description, and then used whatever tools I wanted to get it done, then email in the completed exercise.

The two exercises were not completely trivial, but were relatively straightforward and actually involved skills relevant to the position and not arcane puzzles or comp-sci theory. (Basically, the first was to write a Postgres schema from scratch (total 5 or 6 tables and maybe 30 columns across all the tables). The second task was to write a CSV parser (including reasonable error checking/recovery) to load data into the schema from the first problem.

Finally, there was the main interview (conducted via Facetime, this is a remote position). That ran for close to 3 hours.

This worked well, since it gave me a feel for what the work would actually be like, and I was given, as the candidate, every oppurtunity to put my best foot forward - no whiteboard exercises, no surprises.
I'm not the parent poster, but it is hard enough to find a similar Android app, let alone build your own ecosystem. Pairing with Google should have been an early move—it is what I would have done immediately—announce that new RIM tech will be more secure than ever, but with more apps than ever. (Yes, tech folks would shake their heads, but that doesn't matter)
Nonsense. I'm saying that people live in the real world with real contexts, not in Context-Free-Abstraction-Land.

Real contexts, where abstract concepts have been concrete realities, put natural and rational limits on the ability to treat those concepts as abstract ideas to be debated solely on their hypothesized merits rather than their hard realities. There is a very rational difference between arguing in favor of communism on an American college campus, where communism is a lovely abstract ideal, and arguing in favor of it in Romania or Ukraine where communism was a brutal reality.

Moldbug is an American, being read by Americans, writing about his support for the institution of slavery, and explicitly saying that some races are genetically better suited for slavery than others. It is far more rational to treat that as racist than to bend over backwards and pretend that it is not in the name of free debate.
I'm not sure if 'new sales' vs 'upgrades' is a useful metric for a company like oracle. They're already prevalent in the enterprise infrastructure of the Global 2000. Oracle's new in-memory database with columnar compression is considered an 'upgrade' to their flagship DB product.
Yeah, definitely. I worked for eight years in the Toronto startup scene. Several coworkers were unquestionably awesome, but they were few and far between. The bar here is much higher.
It's clearly a technical screening challenge - first the applicant must find a public payphone in Australia... and it must also be in working order.
An Applicative functor is a type constructor which lets you lift functions of arbitrary numbers of arguments to functions whose arguments are wrapped with that type constructor.

Concretely, if you think about promises in JavaScript, for example, if you have a function A -> B -> C -> D, say, and you have three promises of types Promise A, Promise B and Promise C, you can construct a Promise of type Promise D by running the three in parallel, and applying your three-argument function when they are all done. So you've turned a function of type A -> B -> C -> D into one of type Promise A -> Promise B -> Promise C -> Promise D. If you can do that sensibly for any number of function arguments, you have an Applicative. ("sensibly" here means that there are type class laws which have to hold)

Every Monad is also an Applicative functor, since you could use do notation to compose your promises instead, but there are other interesting Applicatives which do not come from Monads.
I don't get the obsession with binary trees questions on interviews. I've had several in a couple different interviews. You know how many times I've coded a binary tree algorithmn, in 15 years as a professional programmer? Zeeeeero. They just aren't that common in actual code.

Why not ask questions about, say, working with a general tree data structure, like XML or an HTML DOM? That's something that comes up all the time in many fields. Why the obsession with low-level pointer juggling?
This might be viable if you do it remotely: let them take their time to understand the project, understand the task, find the relevant section of the project, find the relevant section of code, understand the change they need to make, set up the development environment, and make the change.

That's at minimum a day's work for most tasks, and sometimes two if setup or debugging turns out to be more difficult than expected. It definitely doesn't scale as an in-person interview, but by doing it asynchronously you lose a lot of the live communication and thought-process data that current interviews provide…

…then again, you could take this opportunity to measure their ability to give asynchronous status updates instead: can they clearly document their progress, the challenges they faced, and how they overcame them?
> One thing that might work along these lines- Bring in a hard problem that the interviewer doesn't know how to solve and spend some time trying to solve it with the candidate. As a team, discussing different options and problems with those options. You know, like people do in actual work situations...

^This guy gets it.

I've never understood why interviews are set up to put the interviewee on edge and make him or her feel out of place, while simultaneously putting the interviewer in a temporary position of ultimate power. Said position does one of two things based on their personality: Makes the interviewer feel superior and aggressive, or makes them uncomfortable and ready to get it over with. Neither situation is good for the people involved nor the process itself. You're basically testing an employee on non-work-related problems under artificial pressure, which is pointless.

And I say all of that as both an interviewee and interviewer in the past.
I also drew a complete blank at an interview in a coding problem... and then went home an implemented it (taking quite a bit more time than I'm comfortable admitting ;-) ).

Interviewing is hard (from both sides). Sometimes it just isn't your day. Sometimes it just isn't your job. I was quite depressed when I screwed up that interview because it was the first one I had done after taking 5 years away from programming professionally. I was definitely rusty and I was secretly worried that I wasn't going to recover.

But, in retrospect, I'm glad I didn't get that job. It's a bit like the situation of being asked to TDD building Pascal's Triangle. I am an avid TDDer and I would have a hard time doing justice to that problem. Of course it is easy to get started, but then at some point there is going to be a "and then there is magic" bit because you are trying to test a function that is outputting an infinite sequence. It could make a fascinating conversation, but I think it says alot more about the interviewer than the interviewee.

Unfortunately, just like there a people who look like they would be amazing an amazing hire, but turn out to be so-so (at best), employers are the same. The the 10x employer is probably more mythical than the 10x employee.
>Sometimes, you just can't win.

folks? people? etc?

"Alright folks, let's get back on topic."

"Alright people, let's get back on topic."

"You folks going to happy hour?"
Writing software is not what (many) companies do. Companies exist to create things of value for their clients. Software is one aspect of the secondary functions that enhance this value creation.

In the article, the VP worked at a company that sold things on the internet. The things they produced were their primary function; internet platform development is a secondary function, much like marketing, hiring, business development etc etc.

So the theoretical VP's core competency should not have been software dev, or even technology - that's what CTOs and technical leads are for. In fact his core competency probably wasn't product development anymore, if it ever was. He was a manager, and his role was managing resources within the company to optimise their primary function.

Hence the disconnect, and hence why articles like this (and audiences for them) exist.
It doesn't (yet) seem particularly comfortable, but this was an impressive demo of someone coding their environment in the Rift.

https://www.youtube.com/watch?v=db-7J5OaSag
I say this literally every time a story like this pops up but, as ever, I'll say it again: Have the (adults) in charge completely lost their minds? Is the government so out of touch with reality that this makes sense to them? I loved going on walks or riding my bike to the shops or sleeping in/playing at home when i was 10 or 11, and yet now this seems all but impossible to consider? How the hell did this happen?
Funny, I thought this was going to be "inverting binary trees considered harmful" because the sensible way to do it is to change your traversal algorithm to be right-before-left, and leave the data alone. Silly me!
Yeah I don't think OP was defending Twitter I think it was just more of a story about how crappy the interview process is at most tech companies.

My experience with Twitter was really odd. I knew someone internal and had them directly submit my resume with a recommendation. The resume was tailored for exactly the position they wanted showing me besting all of their requirements. I got an email 4 months later telling me my resume was not good enough for even a phone screening. I've never had that happen before and it's still perplexing.
In your daily programming activities, how many times do you have to implement a binary tree? I'm genuinely interested.
I also think it's an extremely simple question, but only if you've ever taken a data structures class.

Turns out a lot of people in our industry don't! I have a Computer Engineering degree, and know this stuff mainly due to this being my hobby (and having tried out for the IOI in high school). But there are former classmates who are only vaguely aware of these data structures, because they never got any exposure to them despite writing real programs.

I don't think there's a super strong argument that most computer engineers need to know much of anything about base data structures. There are so many other concepts that are more important to writing and deploying useful software (especially in the enterprise world) that at no point involve writing complex data structures. Data structures (or rather, knowledge of the internals) are much more domain specific than some of us would like to admit, especially in the age of RoR and Unity. There are many more design patters more important to know about than being able to implement quicksort (it took 6 years for researchers to write the first bug-free implementation!)

Google does it because they can. They're probably filtering out a lot of people who treat code as prose rather than a technical manual though. Which is probably why Google libraries look the way they look.

The biggest surprise for me is how many game programmers don't know about this stuff. It seems like the sort of stuff you'd run into pretty quickly
I love interviewing and being interviewed. Don't know why but I do. However, I hate these types of interviews. I mean, sure if I am a new grad or you suspect I might not know my shit, ask me some probing questions. However, almost nobody these days needs to invert a binary tree or compute the hash of a string by hand.

Here is my secret go to question when interviewing someone: "can you describe how an AJAX request works, from start to finish?". The answer involves knowing that AJAX works over HTTP, same as regular page requests, knowing what IP, TCP, and HTTP are, and how client and server interact using them. It is a question most competent people can answer, yet it has lots of room for depth of detail. By discussing technical subjects, challenges and solutions I think you get a much better idea of how good someone is vs some predefined set of puzzles.