What are some factors that enable some papers to be published in top-tier journals while others (apparently similar) cannot?
For example, in the field of control theory, there are so many papers with complicated mathematics. Some paper is related to a very similar topic, but some get published in a top-tier journal while others not.
This question is important because knowing this, then I know which journal my paper might be submitted to that is possible to be accepted in the end.
To be accepted and published it has to be submitted. Some papers that might be accepted just aren't ever submitted.
Some of it is just luck. The editor was looking for something. Even something as simple or stupid as s/he needed to fill an 8 page gap in an issue and yours was the best available candidate at the moment.
Some of it is just the writing itself. Good journals want, and try to get, well written, understandable, papers. If the reviewers have trouble understanding you, it will be hard to get accepted.
But most of it is that a paper answers a question (or two) that seems important at the time the paper arrives. It is the science/mathematics/whatever behind the paper that really matters. The members of a scientific community are fairly often on the lookout for an answer to a perplexing problem. If you can provide that, and submit a well written paper, you are more likely to get published.
For many fields, especially in the natural sciences, the "top-tier" are not just looking for well-done, well-written research reports. Instead, they have an explicit editorial goal of finding papers that are impactful and interesting to a wide, interdisciplinary audience. For example, Nature wants manuscripts that
In practice, this means that they often select for 'hot' or controversial topics, the use of exciting new techniques, and surprising, counterintuitive results. As a result, the zeitgeist of the field is probably the major determinant of whether your paper gets accepted in a 'glamorous' journal like Science, Nature, or PNAS. Quality is obviously important too: even the trendiest paper won't get in if the experiments are obviously flawed or the writing is impenetrable (usually!). However, it is manifestly not true that the best quality papers are published in the best quality journals, or conversely, that everything published in a well-regarded journal is gold.
To add some specific examples, Physical Reviws rejected Theodore Maiman's description of the laser, Physics Letters didn't want the Higgs model, and Nature declined the first reports on MRI, the Krebs Cycle, and(!) the cell cycle. Paul Lauterbur, whose MRI paper was rejected from Nature--but nevertheless won him a Nobel Prize (2003)--has quipped that "You could write the entire history of science in the last 50 years in terms of papers rejected by Science or Nature."
On the hand, these journals are also full of papers that seemed exciting at the time, but either went nowhere or turned out to be fatally flawed (e.g., arsenic-based DNA, water memory, and so on).
If you already have a target venue in mind, you can often submit a presubmission enquiry. Each journal handles them slightly differently, but they almost always contain the manuscript title and abstract, and often include a short discussion of the results and why the journal's audience may care about them. The editor will reply, often in a few days, to tell you if the proposed article is a good fit for the journal. While you often don't get much feedback at this stage, it can help you avoid journal-specific scutwork, like reformatting the text or redrawing figures. Occasionally, editors will also mention concerns related to a specific topic: Current Biology told us, for example, that while they were interested in the topic generally, they also felt that much of the existing literature was not very good, and so they would be expecting very rigorous controls.
If you don't have a "target journal", you need to think about the potential audience for your paper. There are many journals which specialize in particular techniques (e.g., J Neurophysiology, for neurophysiological experiments), topics (Attention and Perception for studies of, well, attention and perception), and application (J. Neural Engineering). Browsing your own reference list may help: your paper is likely on-topic at the journal you cite most!
Finally, I would quickly browse through a few complete issues of your candidates. For example, while Journal of Vision and Vision Research are both nominally interested in anything pertaining to seeing, Journal of Vision skews very strongly towards human behavioral experiments. Although you could submit other kinds of work there--and people occasionally do--there are probably better venues for experiments with other techniques or animal subjects.
This question should actually ask about something different. As it stands, the obvious answer is good papers are accepted in good journals.
The thing here is that knowing a good paper when seeing it comes with experience. There are lots of questions on this site asking "how to know in which journals publish?". This comes from knowing the field, so also comes from experience. At first a researcher has no idea if his research is good or bad – that's why advisors exist (ideally). Then one knows the scope and quality of the major journals in the relevant field. Ultimately, one also knows the quality of one's research, so the question "in which journals should I publish?"/"which journals are worth publishing in?" changes to "what research is worth publishing (at all)?" – and then one aims at the best journals.
Whenever I read through a paper in a top-tier journal, I will usually notice the paper has top-tier results. However, that is not enough. You also need top-tier presentation, including top-tier figures and top-tier handling of data. For example, if it's possible to use a statistical method and obtain useful results at a 99.9% confidence level, go for it!
To help you getting a paper published in top-tier journal, you can:
However, I would say that you should go through this list only if you have top-tier results in the first place. A paper having mediocre results, but top-tier presentation, top-tier figures and top-tier handling of data will get published only by sheer luck if you're targeting the very best of the journals.
Actually, my personal experience has been that top-tier journals tend to publish articles that are more like summaries of a certain field. Take Nature, the papers you will find in there are either groundbreaking discoveries from CERN or similar institutes, or papers that offer a kind of Big Picture of a certain field.
So I would partially disagree with some of the other answers - it's not all about writing high-quality papers. The content also needs to be what the journals are looking for, and what they want is often not a brilliant, but highly technical paper. Instead they want the paper that summarizes/reviews your brilliant technical paper, together with a dozen others, and offers some general/accessible insight.
And it feels wrong to omit that of course, your standing in the scientific community has some influence. There are exceptions to the rule, but I would imagine that even someone with Einstein-level brilliance would have trouble getting his groundbreaking theoretical paper published in Nature if he is only just starting his PhD. Such is the way of the world.
A lot of it is about audience: this is obvious in places like Nature and Science, which are general science journals, and are therefore looking for papers that will appeal to a general science audience - for your example of control thoery: what is a paper in control theory that a biologist or materials scientists (for example) might think was cool. This also applies further down the pile:
Would a mathematician or engineer from a different, but related field find it interesting?
Step down again: Would anybody in control theory find it interesting, or only people studying that particular problem?
