Could you give some examples of notation that you struggled to understand, and explain what about it you find confusing?

One difficulty of notation is that the hierarchy of abstraction builds dizzyingly quickly, and soon you're manipulating symbols that generalise a whole classes of structures that were themselves originally defined in terms of lower-level abstractions. When this becomes overwhelming, it usually means that I didn't give my understanding of the lower levels long enough to settle and mature.

Concise notation and terminology is only useful if the underlying ideas are organised neatly in your mind, and the best way I've found of achieving this is to study a subject obsessively for some time, then put it away for a few weeks, and then go back and try to see the big picture and find out where it doesn't fit together by trying to derive the main results from scratch. Then I start again and fill in the blanks. After a few years things begin to make sense, but this process takes time and it's difficult and tiring (or at least that has been my experience of it).

In order to read research papers fruitfully it's crucial that you understand the basics well, and the best way to do that is to work through books aimed at undergraduates or young graduates. People don't read foreign literature by jumping straight in and looking up every word and every grammatical construction as they go. They become familiar enough with the language by reading easier texts until the language is no longer an obstruction - then they're free to appreciate what's happening at a higher level. The same for driving: you wait until you're comfortable operating the car mechanically before you drive on busy roads. The same, also, for mathematics.

It is not at all unusual to to find notation and technical terminology tiring. Everyone does to some extent. I hate it. But it's necessary.

Some resources I found useful:

Naive Set Theory by Paul Halmos. One of the great mathematical expositors, Paul Halmos here describes the fundamental language of mathematics: set theory. This is a book for people who want to understand enough set theory to do other parts of mathematics without obstacle.

How to Prove It by Daniel Vellemen. A nice introduction to logical notation and common proof structures, aimed at helping incoming maths students to become comfortable with the basics of formal language and notation.

Anything by John Stillwell. Stillwell is an inspiring teacher who insists on including the practical and historical motivations for the abstractions we use (this is, sadly, rare for modern teachers of mathematics). If you find yourself wondering why people cared about a problem enough to solve it, Stillwell might be able to help.

I suspect you'll also need resources on linear algebra (Halmos has 'Finite Dimensional Vector Spaces') and analysis but I'm not sure as I don't work in machine learning. I just sort of learnt linear algebra as I went and I avoid analysis as much as a supposed mathematician can. (context: my undergraduate degree was in economics and didn't carry much mathematical content other than some basic linear algebra - now I'm a graduate student in mathematics and computer science who uses a lot of category theory and abstract algebra. The transition was painful. Really painful.)