skysight-inc/HackerNews huggingface dataset:
Key points:

1. Dropbox moved from AWS to its own datacenters after 8 months of rigourous testing. They didn't exactly build a S3 clone, but something tailored to their needs, they named it Magic Pocket.

2. Dropbox still uses AWS for its European customers.

3. Dropbox hired a bunch of engineers from Facebook to build its own hardware heavily customised for data-storage and IOPS (naturally) viz. Diskotech. Some 8 Diskotech servers can store everything that humanity has ever written down.

4. Dropbox rewrote Magic Pocket in Golang, and then rewrote it again in Rust, to fit on their custom built machines.

5. No word on perf improvements, cost savings, stability, total number of servers, amount of data stored, or how the data was moved. (Edit: Dropbox has a blog post up: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr... )

6. Reminds people of Zynga... They did the same, and when the business plummeted, they went back to AWS.

7. Not a political move (in response to AWS' WorkDocs or CloudDrive), but purely an engineering one: Google and then Facebook succeeded by building their own data centers.
Get your startup funded: Breed Reply is selecting Europe's best IoT startups
> Perhaps I am bias as I have enough savings/income from an existing small business...

Yes.
Good questions, let me try to tackle them one by one.

> The article makes a brief mention of Go causing issues with RAM usage. Was this due to large heap usage, or was it a problem of GC pressure/throughput/latency? If the former, what were some of the core problems that could not be further optimized in Go?

The reasons for using rust were many, but memory was one of them.

Primarily, for this particular project, the heap size is the issue. One of the games in this project is optimizing how little memory and compute you can use to manage 1GB (or 1PB) of data. We utilize lots of tricks like perfect hash tables, extensive bit-packing, etc. Lots of odd, custom, inline and cache-friendly data structures. We also keeps lots of things on the stack when we can to take pressure off the VM system. We do some lockfree object pooling stuff for big byte vectors, which are common allocations in a block storage system.

It's much easier to do these particular kinds of optimizations using C++ or Rust.

In addition to basic memory reasons, saving a bit of CPU was a useful secondary goal, and that goal has been achieved. The project also has a fair amount of FFI work with various C libraries, and a kernel component. Rust makes it very easy and zero-cost to work closely with those libraries/environments.

For this project, pause times were not an issue. This isn't a particularly latency-sensitive service. We do have some other services where latency does matter, though, and we're considering Rust for those in the future.

> Could you comment more generally on what advantages Rust offered and where your team would like to see improvement?

The advantages of Rust are many. Really powerful abstractions, no null, no segfaults, no leaks, yet C-like performance and control over memory and you can use that whole C/C++ bag of optimization tricks.

On the improvements side, we're in close contact with the Rust core team--they visit the office regularly and keep tabs on what we're doing. So no, we don't have a ton of things we need. They've been really great about helping us out when those things have sprung up.

Our big ask right now is the same as everyone else's--improve compile times!

> Are there portions where the decision to use Rust caused complications or problems?

Well, Dropbox is mostly a golang shop on the backend, so Rust is a pretty different animal than everyone was used to. We also have a huge number of good libraries in golang that our small team had to create minimal equivalents for in Rust. So, the biggest challenge in using Rust at Dropbox has been that we were the first project! So we had a lot to do just to get started...

The other complication is that there is a ton of good stuff that we want to use that's still being debated by the Rust team, and therefore marked unstable. As each release goes on, they stabilize these APIs, but it's sometimes a pain working around useful APIs that are marked unstable just because the dust hasn't settled yet within the core team. Having said that, we totally understand that they're being thoughtful about all this, because backwards compatibility implies a very serious long-term commitment to these decisions.
It is very early so we are not yet incorporated. The issues you mention will definitely be taken into account!

At scale, we'd probably have multiple legal entities in different countries anyway, like Wikimedia.
But numerals "10" in bases other than decimal aren't usually called ten, I think.
Same here. I say I'd use !g on duckduckgo about 1/3 of the time. If I'm getting really frustrated on a problem and am hacking away, I sometimes just default to the !g.

Duckduckgo is now doing localized results (you can choose your region, so it's transparent, unlike Google's). The thing about Google is that, even if you're not signed in, it still tried to present you with personalized results (based on previous searches for that session, your IP, your region ... if you're searching from work; it probably factors that in as well).

When people talk about getting to the top of Google results, my response has always been, "Well you need to be more popular and relevant. Also you may be at the top..for some people, but not everyone."
Couple useful speedups for your loop: you only need the xrange to go up to floor of sqrt(p), since any divisor would have been found already by that point. Also you can save a lot of divisions by working as a sieve with the first couple thousand primes (so check divisibility against a store list of primes, instead of all numbers, for as long as your memory allows).
(1) Adversarial learning is unsupervised and works great.
Most of language modeling is unsupervised (you predict next word, but it's not real supervision because it's self-supervision).
There're many works in computer vision which are unsupervised and still give more or less reasonable performance.
See f.e. http://arxiv.org/pdf/1511.05045v2.pdf for unsupervised learning in action recognition, also http://arxiv.org/pdf/1511.06434v2.pdf and http://www.arxiv-sanity.com/search?q=unsupervised

(2) ImageNet supervision gives you much information to solve other computer vision tasks. So perhaps we don't need to learn everything in unsupervised manner, we might learn most features relevant for most tasks using several supervision tasks. It is kind of cheating but very reasonable one.

Moreover,

(3) We observe now just fantastic decrease of perplexity (btw, it's all unsupervised = self-supervised). It's quite probable that in the very near future neural chat bots write reasonable stories, answer intelligibly with common sense, discuss things. All of this would be just a mere consequence of low enough perplexity. If neural net says smth inconsistent it means that it gives too much probability to some inappropriate words i.e, it's perplexity isn't optimized yet.

(4) It's quite probable that it would open a finish line for human-level AI. AI would be able to learn from textbooks, scientific articles, video lectures. Btw, http://arxiv.org/pdf/1602.03218.pdf gives a potential to synthesize IBM Watson with deep learning. May be, the finish line to human level AI has been opened already.
Good point! I've been meaning to make Kimchi but unfortunately my gf hates the smell so much that she's banned me from making it...

Another (banned) item on my list is homemade natto... (https://en.wikipedia.org/wiki/Natt%C5%8D)
Microsoft SQL Server for Linux is a brilliant and logical idea
WhatsApp reportedly plans to expand encryption soon
> Primes have no inherent relation to decimal, so all prime patterns in decimal are mathematically meaningful.

No. There's all sorts of random nonsense in all base expansions of anything that we assign meaning to. Most base-related stuff is numerology.

> P.S. never say "base 10", it's a tautology. Say "decimal"

No, it's not. You need to specify your base number.
Meet the Silicon Valley CEO Opening Doors for People of Color
Show HN: gotests – Generate table driven tests in Go
You answered my question!
The perfect blend of content and design to improve UX
Primes seem to me to be more of an information theoretic concept than a number concept.

Primes are the simplest way to encode specific kinds of graphs that unambiguously encodes all sub-graphs.

If you try to come up with a bit-representation that is equivalently rich it becomes difficult to think of one that is as simple yet preserves the semantics of the factorization tree.

So I guess my point is that the factorization tree of numbers is the fundamental concept, and it's information theoretic. Primes happen to be an encoding of that fundamental concept into integers, but if we found an equivalently rich representation using a different encoding, we might understand primes better. I doubt that the quirks of the encoding has anything to do with the fundamental concept however.
Duckduckgo added localization, but it's transparent and configurable.
Frankly? That's not really I question I should be answering. That's a question you should be a/b testing.

For example:
Option 1: sign up with all permisions
Option 2: sign up with 5-6 most common permission, request additional for least common
Option 3: sign up without permissions, request every permission as needed.

Then check what matches your conversion goals.
If you have a background in hardware design / chip architecture, things get even more interesting. Figuring out how to balance memory bandwidth, power and optimal datatypes (think integer v. floating point) is fascinating stuff.
The importance of Deep Work for software developers
Brewing Kombucha is insanely easy. Way, way easier than beer or anything else.

1. Buy a large glass dispenser with a stainless steel spigot at the bottom (this is important and will make life WAY easier).

2. Buy a Kombucha starter kit online or locally. I usually use green tea as my base. I buy my kits from Amazon and they come with the scoby (mother) and tea bases. Some come with glass jars, etc. I'm looking at them now and there are a LOT more options than when I got into it a few years ago.

3. Cover the top with cheesecloth so it can breathe. Make sure mold stays out. If you get mold toss the batch and start over. As the scobys grow and break apart and you drink (using the spigot at the bottom) they float to the top, with the bottom spigot you can easily pour without having to move the scoby out of the way.
Speculative discussion of AlphaGo's failure in Match 4
Cops often let off hook for civil rights complaints
How my Kickstarter blew up my life
Then it would seem that AlphaGo on a single machine isn't equivalent to an Amateur. Presumably it's really good even when running in a single machine, and the marginal improvement for each additional machine tails off quite quickly.

But when you're pitching it at a top class opponent in a match getting global coverage, you want all the incremental improvement you can get.
I'm a mathematician and I'd say their ideas only become interesting if that conjecture is true. Otherwise it's just numerology.
My experience is pretty much the same. I sleep and function just as well (or badly) without caffeine now as I used to do with several cups of coffee, before I quit (2 years ago). The big bonus for me is in the mornings. I don't need several cups of coffee to wake me up anymore. When I get up I am as awake as I used to be after breakfast.

Be careful when you try to drink it again. One cup of coffee is enough to get me from a hard-to-focus-after-lunch-dip to fully awake and focused. If I drink another cup I become all jittery and have symptoms resembling a minor panic attack.

Good luck!
Only in bases that are itself prime ;)
> People need money to live, people make money from proprietary games. People also invest that money and make even better games. It's not practical to only limit yourself to OSS.

Of course it is. The whole point of the free software movement is to demand freedom for all users. If you are implying it is not possible to go through life without using exclusively free software for your personal computing, then that's simply false---many, including myself, reject proprietary software entirely.

Using proprietary software on behalf of your employer is different. You're not being robbed of your freedoms because you're not doing your own computing---your employer is being robbed of those freedoms.

The argument that it's not possible to make money developing free software is an often used one and is simply false:

https://en.wikipedia.org/wiki/Business_models_for_open-sourc...

(I don't endorse a number of the methods on that page.)
> That was your answer to the question I posed.

It most certainly was not. I said you could tell your friend that you're sorry and you hope the people responsible are caught and punished. I said nothing about giving a speech at the wake on the dangers of giving up privacy. You said it wouldn't play well, which is of course true; but that's precisely why you don't want to have that discussion at a wake. You want to have it when there's no immediate issue at hand, so people can be reasonably objective.

> The hard part is what do you say when you have to look someone in the eye and explain yourself.

If your friend comes and asks you point blank why you're against giving up privacy, given what happened to his family, then you have to answer him, yes. But that is worlds different from making a speech, unprompted, at a wake.

If he asks you, your honest answer would have to be that, while you are sorry about his family and hope that the people responsible are caught and punished, you still believe that, in the long run, the dangers of excessive government power are greater than the dangers of terrorism. (That's assuming, of course, that you actually do believe that. But if not, what's the point of the hypothetical?) What else can you say? Your friend probably won't like it, but he's your friend; you've got to be honest with him.

> That's why we have to be willing to discuss it also.

You're missing the point. We can't discuss it because we don't know the facts. And we can't be told the facts because, if there is any benefit to the surveillance, making public how much benefit we get would destroy the benefit. That's why all of these activities are kept secret in the first place.

> Otherwise they have to make the determination and we all get to sit back and throw rocks at them.

We can, but nothing forces us to. Anyone who believes it's worth it for the government to do mass surveillance, and any other covert activities for that matter, has to also be willing to not criticize decisions made based on data that can't be made public. Otherwise they're just a hypocrite who wants the benefits without the costs. I would say that the main source of hypocrisy about that is not individual citizens, but the press.
How Running a Modern Hospital Can Be Like Playing a 90’s Video Game
Anecdotally, a lot of the people taking these classes totally lack the background or intuition needed and aren't getting any real training in machine learning. They're learning some very rudimentary bits of data cleaning and how to use basic machine learning libraries.

I recently interviewed someone taking a (reputable) online masters in machine learning, and they couldn't describe how or why any of the models they were using worked, nor could they answer most basic questions about the problems / data they were working on.
I definitely share your usage of "!g", which is why Common Search already supports it ;)
Look at base 11: there are a lot of rarefied diagonals (which correspond to any prime + the base (11) + 2, or prime + 13). I wonder if prime + other_prime is rarefied in general.

https://play.golang.org/p/t3o00iEQgF
Thanks for your concern. However I was under the impression that Wikimedia was a US organisation, with some local chapters. Perhaps incorporate in a EU country from the start?
I'm not the original commenter but there could be a big huge case in research. Lots of researchers work on UIs for search, interactive search systems, and query refinement algorithms that are really just abstract layers over an existing search engine. It used to be that we could just overlay stuff over Google, but most search engines nowadays are a pain to work with.
You can just increase the length of the suffix since that is equivalent to talking about the last digit in a power-of-two base. E.g. if the prime ends in 11 in binary then the following prime is less likely to end in 11, since this is equivalent to saying that a prime ending in 3 in quaternary is less likely to be followed by another prime ending in 3.
Mobile UX Design: User-Friendly Search
Please do a technical blog post on some of these:

1. Go vs Rust at Dropbox's scale and requirements.

2. Maintaining availability whilst moving from AWS to Diskotech and Magic Pocket.

3. Internals of Magic Pocket (file-system, storage engine, availability guarantees, scaling up and scaling out, compression details, load balancing, cloning etc)

4. Improvements in perf, stability, security, and cost.

Thanks.
I am surprised this took so long to discover -- wouldn't this be one of the first things to examine when looking for non-randomness?
Side note: Zen and the Art of Motorcycle Maintenance is a classic read about "The hypothesis has to come from somewhere".
Realistically, what are the chances of bad bacteria being generated with these made at home ?

I understand the benefits, but when cultivating any bacteria, how do you know what you've made without lab testing ?
Here's the full list of 97 launches for Proton M series. Failures, 9 in total, denoted with red border or green background: https://ru.wikipedia.org/wiki/%D0%A1%D0%BF%D0%B8%D1%81%D0%BE...
Yes, chrome extension so that next to the +1 button is a way to chip in money, and adds a comment to the issue like "seibelj pledged $10 to this issue".
So among the less populated set of same final digit-ed sequential primes is the next-digit-more-significant anti-correlated (to mitigate its 'misbehavior' ;)? In a sense, how likely does this unusual homo-digitalism rise once our focus on the final digit is shifted?
Happy Pi Day 2016
Went to website, noticed scroll override, closed site. If you're inexperienced enough to think scroll overrides are ever a good idea, your product probably sucks way more.
between dwavesys and http://home.cern i think the "scientific method" is a little out of date. No longer is it just what we can observe in this universe that's true science. Very related to AI.
Totally on point, if privacy is a concern the company should not be inc. on American soil. I'm also having a hard time trying to find the roadmap for infrastructure, how many machines are running or will be expected to run. Scaling is not trivial and I don't see how that would be addressed.
without spending the time to make a better prime generator, you can get your test going much faster by only testing factors up till sqrt(p).

With just that change to your program, and asking for 100k primes:

{'9-1': 8829, '1-1': 4104, '9-7': 5671, '3-9': 8387, '3-7': 7419, '7-1': 6438, '1-3': 7961, '3-3': 3604, '7-9': 8022, '1-7': 8297, '7-3': 6928, '1-9': 4605, '3-1': 5596, '7-7': 3627, '9-3': 6513, '9-9': 3994}
Someone woke up the sleeping giant. Curious to see how this compares with Sketch. Simple interactive wireframes is huge.
Sure, but the segment that mentioned other encryption apps was talking about what would happen under a US government mandate. Open Whisper Systems would have to comply, move or cease operating (Just like Apple).

Apple also improved the latest iPhones, the OS replace bypass at question here will no longer work. So Signal has an advantage over older iPhones, but not all of them.
> On the improvements side, we're in close contact with the Rust core team

One small note here: this is something that we (Rust core team) are interested in doing generally, not just for Dropbox. If you use Rust in production, we want to hear from you! We're very interested in supporting production users.
Ridiculously fast MongoDB replica recovery
> Supervised learning may be how it looks from the outside, but consider that out of the >6,570,0000 waking seconds of a child's life up to age 5, there maybe only a few dozen instances of supervised adult instruction per day. Besides those, what do neurons do the remaining 99.99% of the time?

This seems like a really facile analysis. For example, if I read a child a storybook, I'm deliberately providing several signals every second. That's a "single instance" but I've effectively provided a lot of training information. At least enough to keep a child's mind busy for 3600 seconds.
> it's not necessarily the best decision to exclude kids from what can be a significant social interaction.

And I struggle with this. Ultimately, I do not force my philosophy on others. I will do my best to raise my kids to value their freedoms, but I'm not going to go out of my way to prohibit them from playing those games, because that would be detrimental to their social development.

But those social pressures are a huge perpetrator of proprietary software. They need to be addressed, but forcing a single member of the social circle to invoke those beliefs is not an effective strategy.

I have long made the point that some have priorities that are higher than software freedom. The happiness and social well being (to a degree that I find reasonable, as any parent must balance) is higher on that list for me, because the well being of my sons are more important to me than my ideals. My wife uses proprietary software. I'm the sole earner right now in my household; we share money. She uses that money to buy proprietary software/devices. I'm not going to prohibit her from doing that, because that would be forcing my ideals.

But when it comes to me, I would sooner sacrifice my social well-being than use proprietary software.
Chill out. I didn't claim that ADHD doesn't exist. I asserted that there is a lack of diagnostic rigor in diagnostics and implied that the incentives are aligned to provide over diagnosis.

We've moved to a model where "raising standards" in education and increased evaluation of teachers equates to formulaic, inflexible school curriculum that younger kids cannot handle. So they act out and get referred for evaluation. My public school district begins assessing children using computerized testing starting in November of the kindergarten year. There's a big difference between a 5 year old born on January 1 and a five year old born on August 1, and those later born kids are going to get labeled with mental health conditions for life and drugged because they are square pegs that don't fit in the prescribed round hole.

The article didn't claim that ADHD doesn't exist either. A key quote:

> Zoëga says the only country studied so far where the relative age of young children doesn't seem to have an effect on ADHD diagnosis is Denmark, where there's more flexibility for when children enter school

Bullshit ADHD diagnosis are bad for people like you who are genuinely affected, as the resources available to help you end up getting shared and stretched to cover disciplinary and structural educational problems. A non-ADHD child isn't helped by ADHD prescription and unnecessary services, and an ADHD child isn't helped by not getting the attention he needs.
Why gun violence barely rates on Silicon Valley's long world-changing to-do list
Quick thumb mashing game for Android Phones
Good points. I'm coming from a computational physics research background and applied for a few data science positions at startups, a large social network and a private ML research group, so not that many overall, beware of the small sample size.

The smaller startups seemed to want more "data engineering" experience.

What do you mean by "what if any bias was there in your job search"?