Publishing papers is not just about getting work out there, it is also about making other people understand the work done. As a 2nd year PhD, you are already part of the target audience of papers in your field.
As a reviewer, it is perfectly fine to come to the conclusion that a paper is not properly presented. Do not hide such a conclusion just because of the conclusion itself.
If your general impression is good, then a weak accept may be appropriate. If you are basically guessing the content, then a (weak) reject is no shame either.
- My impostor syndrome when seeing overly mathy papers.
This is really where you should put your focus on. Is the paper hard to understand because you generally struggle with mathy papers? Is the paper hard to understand compared to the usual mathy papers?
In a proper review, you can rate you degree of confidence. If you generally struggle with such papers, then a low confidence score is appropriate. This allows other, more thorough reviews to take precedence.
- Their math formulation is not the traditional one and they do naming and formulation variations. Even though they can be properly defined.
This seems like a clear shortcoming of the paper. Using obscure formulations – be it for math or prose – can hide veritable errors or blunders. At the very least, it makes it needlessly difficult to understand and/or reproduce the findings – one of the key goals of publications.
If you are feeling unsure about this point, check the paper's references. Do those use the same formulations? Is there a consistent formulation for key points that the reviewed paper deviates from?
- Over relies on the appendix to understand the content of the main body of the paper.
This depends strongly on the field and publication formats. In some it is perfectly fine to use the appendix for background and explanations that is already known to scientists in the field. In others an appendix is more of a technical addition to thoroughly reproduce a paper.
If you are unsure about this, note it as a comment to the editor. There may be formal rules about the appendix that have not been communicated to you.
- The paper content is very dense.
This ties in with some of the above points. Is the paper very dense compared to usual papers? Does the denseness make the paper needlessly hard to understand?
Try and identify specific points where content seems either superfluous or too short. It is fine for a review to suggest that some parts should be dropped in favour of expanding others. Take into account the appendix as well; it might be acceptable in the field to have a dense paper and supply lengthier explanations extra.