  In modular-invariant models of flavour, hierarchical fermion mass matrices
may arise solely due to the proximity of the modulus $\tau$ to a point of
residual symmetry. This mechanism does not require flavon fields, and modular
weights are not analogous to Froggatt-Nielsen charges. Instead, we show that
hierarchies depend on the decomposition of field representations under the
residual symmetry group. We systematically go through the possible fermion
field representation choices which may yield hierarchical structures in the
vicinity of symmetric points, for the four smallest finite modular groups,
isomorphic to $S_3$, $A_4$, $S_4$, and $A_5$, as well as for their double
covers. We find a restricted set of pairs of representations for which the
discussed mechanism may produce viable fermion (charged-lepton and quark) mass
hierarchies. We present two lepton flavour models in which the charged-lepton
mass hierarchies are naturally obtained, while lepton mixing is somewhat
fine-tuned. After formulating the conditions for obtaining a viable lepton
mixing matrix in the symmetric limit, we construct a model in which both the
charged-lepton and neutrino sectors are free from fine-tuning.
