I'd argue that probabilistic programming is a language or framework of programming that lets you easily build and then fit probabilistic models to estimate or predict things of interest.

If you're taking a Bayesian approach to statistical modelling and inference, then they're probably a fairly good tool to consider. With the Bayesian approach you're trying to compute some posterior probability distribution that summarises your prior information (this might capture domain knowledge, information from related studies) and information from observations.

There are different ways to compute a posterior distribution. In very simple or contrived cases you might be able to manually grind out an answer analytically with pen and paper and lots of algebra and integrals. But that isn't very efficient or scalable. Also, nice algebraic structure is very easily broken by small perturbations to the problem statement -- need to add a weird bit onto the model to capture some real world behaviour? Good chance that ruins your algebraic structure and previous analytic "attack".

MCMC can be used to estimate the integrals you need when computing a posterior distribution. MCMC isn't the only way to estimate or approximate these calculations -- e.g. another approach is variational inference where a bunch of approximations are introduced to replace the original calculation with an approximation that is easier to compute -- this likely introduces bias into the results but can give you something that can then be solved analytically or semi analytically (e.g. approximate everything as Gaussian distributions and a lot of integration collapses to efficiently computable algebraic identities).

Some probabilistic programming platforms like Stan let you define your probabilistic model and parameters and decouple it from the computational backend used to estimate the posterior distribution. E.g. in Stan you can switch the computational backend between MCMC (https://mc-stan.org/docs/2_18/stan-users-guide/sampling-diff...) and ADVI (auto-differentiation variational inference).

MCMC has practical problems in that it is only guaranteed to give you the correct (unbiased) estimate asymptotically, in the limit if you run it for an infinite amount of time. If you're trying to approximate the integral of a function that is very multi-modal -- where it would be difficult for a global optimisation algorithm to locate the global optima -- then MCMC will likely also struggle to produce a good estimate. MCMC is difficult to parallelise effectively as the algorithm is inherently like an iterative local search procedure -- the next state in the chain is some mutation of the previous state. You can run n MCMC chains in parallel from n different initial configurations, but it's not obvious that you'll get a better estimate from n short chains vs a single long chain -- the longer a chain runs, the more chance it has of being able to discover and explore higher probability (more realistic, more plausible) configurations of the parameter space.

MCMC isn't only used for probabilistic programming, you can apply it for other things. E.g. it gets used in material science to study statistical properties of molecular dynamics simulations etc.