  We perform a systematical analysis of the $A_4$ modular models with
generalized CP for the masses and flavor mixing of quarks and leptons, and the
most general form of the quark and lepton mass matrices is given. The CP
invariance requires all couplings real in the chosen basis and thus the vacuum
expectation value of the modulus $\tau$ uniquely breaks both the modular
symmetry and CP symmetry. The phenomenologically viable models with minimal
number of free parameters and the results of fit are presented. We find 20
models with 7 real free parameters that can accommodate the experimental data
of lepton sector. We then apply $A_4$ modular symmetry to the quark sector to
explain quark masses and CKM mixing matrix, the minimal viable quark model is
found to contain 10 free real parameters. Finally, we give two predictive
quark-lepton unification models which use only 16 real free parameters to
explain the flavor patterns of both quarks and leptons.
