This is mostly relegated to Latin and some Greek plurals:

Latin:

stimulus/stimuli, fungus/fungi (2nd declension masculine)
medium/media (2nd declension neuter)
alga/algae, larva/larvae (1st declension)
index/indices (3rd declension)

Greek:

phenomenon/phenomena, criterion/criteria (2nd declension neuter)
analysis/analyses (3rd declension, I think proper Greek and Latin would actually be 'analyseis')

Latin and Greek were the classical languages that were expected to be spoken by the educated upper class in 19th century Anglo-Saxon world (the British Empire and the US). Most of the present day language conventions we have were gradually developed throughout the 17th to 19th centuries, as a result of print becoming established. Before that, English spelling varied greatly from text to text and the even the English plural themselves were determined by the writer's dialect, as in this example by William Caxton (England's first printer) in 1490 (in modernized spelling):

In modern spelling:

"And specially he asked after 'egges'. And the good wife answered that she could speak no French. And the merchant was angry for he also could speak no French, but would have had 'egges' and she understood him not. And then at least another said that he would have 'eyren'. Then the good ife said she understood him well.

The original splling was[1]:

"And specyally he axyed after eggys. And the good wyf answerde that she coude speke no frenshe. And the marchaunt was angry for he also coude speke no frenshe but wold haue hadde egges and she understode hym not. And thenne at laste a nother sayd that he wolde haue eyren. Then the good wyf sayd that she understood hym we."

During the period of standardization, it was the educated class I mentioned before that influenced spelling the most. Up to the 19th century they were expected to have a good grasp of Latin (and sometimes Greek) grammar, which is why "Grammar Schools" existed (originally meant to teach Latin grammar). Latin remained a requirement in some Elite schools and universities (like Oxford and Cambridge) well into the mid 20th century. This explains why the standard setters were quite pedantic about pluralizing Latin (and sometimes Greek) nouns correctly, but gave themselves a free hand in mutilating plurals (and even the words themselves) in other languages.

Since most of the educated upper class didn't know Ancient Greek as well as Latin, you find that Greek plural usage is not as systemic as Latin. That's why we're missing the first declension (we do not pluralize catastrophe as catastrophai, we say dogmas and not dogmata, atlases and not atlantes) and second declension masculine (cosmos is pluralized as cosmoses, not cosmoi). On the other hand, the high classes in the 19th century often spoke French and Italian, which is why it used to be fashionable to have some pluralizations such as -eau/eaux (bureau/bureaux) and -o/i (virtuoso/virtuosi). I feel like this is not so faddish anymore.

It's also important to note that the words we're covering here are _learned words_ - which are more influenced by elitist tendencies, since they first appear in print in academic publications and newspapers, before finding their way into everyday speech (if at all). Early borrowed words from Latin that have long become colloquial by then like wine (vinum/vina), pillow (pulivnus/pulvini) dish (discus/disci) and pound (pondus/pondera) don't even much the Latin singular form and are sometimes barely recognizable.

tl;dr: The explanation has to do mostly with elitist tendencies and the particular languages the elite during the time of standardization knew well and happened to care about.
There are many exceptions.

The best rule of thumb is that learned Latin words are mostly pluralized as in Latin, Greek words are pluralized as in Greek only with certain suffixes (-on/a and -sis/ses) and very rarely with any other language.

The second rule of thumb to remember is that once a Latin or Greek word becomes colloquial, the foreign plural is often used in conjunction with a native English plural or becomes abandoned completely. To use medium as an example:

Academic usage:

* means of communication (media) - 'mass medium' is almost always pluralized 'mass media'
* material conductor (e.g. air as medium) - usually 'media'

Colloquial usage:

* average, in the middle - usually mediums
* someone who communicates with spirits and ghosts - always mediums

[1] https://www.bl.uk/learning/timeline/item126611.html