  We present out-of-plane dielectric and magnetodielectric measurements of
single crystallines $\alpha$-RuCl$_3$ with various degrees of stack faults. A
frequency dependent, but field independent, dielectric anomaly appears at
$T_{A}\:(f=100\:\mathrm{kHz})\sim$ 4 K once both magnetic transitions at
$T_{N1}\sim$ 7 K and $T_{N2}\sim$ 14 K set in. The observed dielectric anomaly
is attributed to the emergency of possible local electric polarizations whose
inversion symmetry is broken by inhomogeneously distributed stacking faults. A
field-induced intermediate phase is only observed when a magnetic field is
applied perpendicular to the Ru-Ru bonds for samples with minimal stacking
faults. Less pronounced in-plane anisotropy is found in samples with sizable
contribution from stacking imperfections. Our findings suggest that dielectric
measurement is a sensitive probe in detecting the structural and magnetic
properties, which may be a promising tool especially in studying
$\alpha$-RuCl$_3$ thin film devices. Moreover, the stacking details of RuCl$_3$
layers strongly affect the ground state both in the magnetic and electric
channels. Such a fragile ground state against stacking faults needs to be
overcome for realistic applications utilizing the magnetic and/or electric
properties of Kitaev based physics in $\alpha$-RuCl$_3$.
