Typesetting is everything that relates to the visual aspects of the paper's formatting, from spacing between individual letters / words / formulas to overall distribution of text / figures / tables etc.
LateX will take care of some that, but it's essential that you use the template provided by the journal your submitting to! If for no other reason then because the styles might be subject to copyright.
In general, rejecting a paper because of a poor writing or typesetting is perfectly valid. Your job as a researcher is to 1) produce a novel idea, 2) explore the heck out of it and 3) communicate it to the community succinctly and clearly. If #3 is done poorly, then #1 and #2 are pointless.
I recommend you to:
- read at least a basic guide on visual design and typesetting,
- observe the style of the existing papers published in your target journal, and
- once you have a draft, ask one of your peers to proofread it to weed out as many issues as possible before you submit.