Hi, I am a research engineer in Yann LeCun's group at Facebook.

Really? Let's chat offline. I'm michael.o.church at gmail.

I'm not looking for a new job right now but I've been in this business for long enough to know that that can change at any time. If nothing else, I'd love to have lunch the next time I'm in New York (am I correct in assuming that you are in NYC?) and get the kind of intellectual ass-kicking you get when you meet someone who actually knows this sort of field at a deep level.

I'm also getting to the point where I have to decide whether I want to go into "real AI"-- and be a small fish again-- or take the big-fish/smaller-pond path of management (I'm in very early discussions about the MD/Data-Sci position at a fast-growing HK/Sing hedge fund, which probably scares the shit out of guys like you who actually know this stuff, as opposed to traders who read two papers, think they understand them better than they do, and build trading systems.) I'm afraid that if I take the executive/finance track, I might get even farther away from the deep-knowledge/R&D space.

I'm only 31, so I'm not afraid to take the Real AI route and be a small fish in a large/badass pond again. In fact, I'd prefer it, even though the winds seem to be taking me the other way. (Finance/management would be the big-fish/small-pond route, since my data science/ML understanding is well into the top 1% in that world and, again, I'm not offended in the least if you say that that scares you. It scares me. R&D people like you get deep knowledge; guys like me in the private sector-- "private sector", here, meaning startups and finance but not R&D labs-- spend about 85% of our time fighting political battles and self-promoting and rarely have the time to learn anything as deeply as we should.)

I hate to seem to be picking a fight

Don't worry. You're not. It's great to hear from someone who actually gets to use this stuff at work. Thanks for taking the time.

Unfortunately almost every sentence you've written betrays a subtle misunderstanding of the space

I understand the space theoretically, but I'll readily admit that I have, compared to you, almost no real-world experience. I've built neural nets for a few small problems, but nothing at the scale you have.

Perhaps the issues I'm raising are completely theoretical and pose no problem in practice.

Starting at a random (no scare quotes needed) point in weight space

I put "random" in quotes because it's not always clear how to sample a useful "random" point for initialization. There's no such thing as a uniformly random point in R^n, of course, so you need to choose a distribution a priori like U[0,1]^n or N(0, I_n). This seems to pose no problem (even while it's nowhere near the "correct" weights, and we both know that individual weights have no independent meaning in neural nets) if there's a heterogeneity in the scales of the inputs, but can be a problem if you have large scale variations.

If one of your inputs ranges from 0 to 1000 and another ranges from 0 to 0.001, then those "random" (scale-agnostic) weight-initialization distributions actually begin with a 10^6:1 bias favoring the former input. Of course, this is a trivial example and scale normalization is as old as dirt, but I think the point (that useful "random" initialization is not so easily defined, especially when you have deep and messy network topologies in which signals tend to vanish or amplify) is sound.

When you transform the space (e.g. feature extraction, scale normalization, adjustment for multicollinearity) a scale-agnostic distribution like U[0,1]^n or N(0, I_n) becomes dramatically different in terms of what it actually means, relative to the data. The fact that these pre-training techniques seem to be effective if not necessary (at least, the people who I read swear by them) seems to indicate that, at least for some problems, this is a real issue.

With a small number of layers, you still need some randomness to not arrive at the (trivial and useless) stationary point you get from w = 0-- because the hidden nodes don't differentiate-- and then SGD with momentum is enough to get you to a good local minimum. However, it doesn't seem that initialization (beyond "random enough to differentiate the hidden nodes") becomes a major concern for shallower nets.

If I understand correctly, it's when you have 6+ layers (and certain categories of neural nets, like recurrent neural nets, are effectively much deeper) that you start to have these initialization issues, because activation values vanish or grow (to saturation) exponentially in the depth of the network and a bad initial point can leave the network in a borked state (e.g. saturation) where the training performs very badly.

Putting "random" in quotes was an attempt to say, "hey, picking a 'random' point in a useful way is not always trivial, because you still have to choose a sampling distribution a priori" but it was late, I am jet-lagged from a trip to Asia, etc., so maybe I didn't express it well.

Aside: even single-layer neural networks do not have convex error surfaces; convexity, and funky error surface geometry

It's correct that single-layer neural nets are non-convex. My understanding, and correct me if I'm wrong, is that with shallower nets, the "broad == deep" bet (that the best local minima will have the largest basins of attraction) is usually correct, and that this breaks down when the nets are very deep (as with recurrent nets, since BPTT is effectively "unrolling" an RNN into a very deep BPNN). I can't even begin to visualize the error surface of a 10+ layer deep neural net, so if that understanding is wrong, please correct me.

(Broad == deep, the contention that the better local minima are more likely to have larger basins, implies that you're likely to get the optimal local minima with a few initial samples. What you wouldn't want is for all the good local minima to have tiny basins-- to be narrow but deep-- because you'd be unlikely to hit with your initial sampling.)

The problem with single-layer neural nets is not that they converge slowly; the problem is that the layer size needs to grow exponentially with the task size. Single-layer neural nets' universal approximation power is thus not of great practical consequence. The power of deep nets is the power of composition: f(g(h(x))) is a strictly more powerful model than f(x) holding the number of parameters constant.

Thanks. This makes a lot of sense. Yay for composition. (I still have a ways to go in ML; my expertise is in functional programming/language design.)

You just initialize with a Gaussian ball around zero and explore whatever valley in the error surface you happen to be in. Works 100% dandy.

Here's the paper I had in mind when I wrote that comment: http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pd... . If I'm misunderstanding the lessons of it, or if the paper is just wrong, please correct me. What I've taken from it is that deep neural network training is quite sensitive to initial starting point, hence the successes of pre-training. To pre-train is, effectively, to change the meaning of "Gaussian ball" (or the like) relative to the data.

That also gets to why I put "random" in quotes. If you do pre-training, feature extraction, et al (which seem to be necessary for many problems but, again, correct me if I'm wrong) then the Gaussian unit ball in the weight space for the (transformed) data is an entirely different set. Even with linear transforms (e.g. scale normalization) this is true.

But again, you've actually used this stuff in your day job and I haven't yet (though I hope to, in my next gig) so I'll defer to your judgment as to whether this is actually an issue. Am I making sense, at least?

It doesn't really make sense to talk about a "single layer convolutional net"

That was my sense, too. It was 11 at night and I didn't want to commit to saying "single-layer convolutional nets are never useful", so I reduced my certainty in what I was saying to "I would hazard the guess ... a lot more useful ...". Generally, when I'm fighting Pacific levels of jet lag and it's after dark, "I can't see how it would work" does not justify "It cannot work".