  We discuss fermion mass hierarchies within modular invariant flavour models.
We analyse the neighbourhood of the self-dual point $\tau=i$, where modular
invariant theories possess a residual $Z_4$ invariance. In this region the
breaking of $Z_4$ can be fully described by the spurion $\epsilon \approx \tau
- i$, that flips its sign under $Z_4$. Degeneracies or vanishing eigenvalues of
fermion mass matrices, forced by the $Z_4$ symmetry at $\tau=i$, are removed by
slightly deviating from the self-dual point. Relevant mass ratios are
controlled by powers of $|\epsilon|$. We present examples where this mechanism
is a key ingredient to successfully implement an hierarchical spectrum in the
lepton sector, even in the presence of a non-minimal K\"ahler potential.
