  We summarize the usual implementations of the large $N$ limit of $O(N)$
models and show in detail why and how they can miss some physically important
fixed points when they become singular in the limit $N\to\infty$. Using
Wilson's renormalization group in its functional nonperturbative versions, we
show how the singularities build up as $N$ increases. In the Wilson-Polchinski
version of the nonperturbative renormalization group, we show that the
singularities are cusps, which become boundary layers for finite but large
values of $N$. The corresponding fixed points being never Gaussian, are out of
reach of the usual perturbative approaches. We find four new fixed points and
study them in all dimensions and for all $N>0$ and show that they play an
important role for the tricritical physics of $O(N)$ models in $d=3$. Finally,
we show that some of these fixed points are bi-valued when they are considered
as functions of $d$ and $N$ thus revealing important and nontrivial homotopy
structures. The Bardeen-Moshe-Bander phenomenon that occurs at $N=\infty$ and
$d=3$ is shown to play a crucial role for the internal consistency of all our
results.
