First let me say that I work in a field (pure math) that is closely related intellectually but for which approximately zero percent of papers are published in conferences. In this case I don't see how it matters that it is a conference rather than a journal except possibly to reduce the timeline until the decision (though some journals in my field are starting to ask for reports within a few weeks!), so I will answer based on my own experience.
The main problem here is that your expert opinion has been called for on something that you do not feel like a fully fledged expert and that (as is most often the case!) it is not practical for you to fix this by substantially increasing your expertise.
So I think your first task must be to decide whether to referee the paper at all. If you for instance you didn't understand the paper at all then certainly you should not be a referee. If you've already agreed to do it, that makes backing out of it more socially awkward, but it could still be the right decision, as it may take some reading of the paper to find the stuff you don't understand. If you vaguely understand it but really not enough to evaluate it in any meaningful way, again I think you should not referee it. If you like, you can explicitly offer to referee something else instead, and I'll bet the chances are good that a different paper could be found.
If you feel that you can or must referee the paper but really feel shaky about evaluating it, then you can write a report in such a way that your evaluation will be minimized, e.g. by explicitly writing something like
I was asked to review this paper, and so I will. But I want to be clear that it lies outside of my core area of expertise, so my recommendations about its suitability for the conference are tentative. I hope that more weight will be put on the other recommendations.
And then you can give a "weak" recommendation, which should then be drowned out by the others.
However, your question suggests that you are sufficiently qualified to referee the paper that the first alternative is not appropriate and the second one may not be either: you write
The paper is so so, not too strong, not too weak. It would be a straight reject in some top-tier conferences, but might be accepted/weak accepted at some other lower rank conferences. How do I make a decision about this paper for this particular conference?
[Point of order: a paper can't be "weak accepted" by a conference. It must either be accepted or rejected. So here you are doing a bit of what other people have brought up: conflating the referee's recommendation with the editor's decision.]
Rather importantly, you don't say why the paper would be a clear reject at top conferences. (Not the assertion is surprising: presumably the vast majority of submitted papers in your field fall into this category.) Knowing that means that you have some information and insight about the paper (and more than you've told us). In fact it might be helpful for you to tell us how you came to this conclusion.
Probably I don't need to tell you that if the paper does not contain any correct, novel work that is of interest to someone, it should not be published anywhere. Conversely, if it does meet these requirements it should be published somewhere. If you don't know enough about the standards of this particular conference, you can work around that by explaining you would recommend rejection in venue A but recommend acceptance in venue B (this is rather common information in referee reports, at least in my field). Then the editor can decide where the conference lies with respect to the data points you've provided.
If you feel qualified enough, then at a certain point you do have to impose whatever standard you feel is most reasonable. If you don't know the field very well and the paper is not interesting to you, then [assuming you've decided to go ahead with the referee job] you should recommend it for rejection: what else? On the other hand, if you find the paper to be at least somewhat interesting you should act so as to leave the door open for the paper to be accepted, in particular by writing that you would recommend it for acceptance at journal or conference X. Ultimately you're leaving a decision to the editorial board that they would have anyway: among papers that are publishable in absolute terms, do they want to publish these papers or those papers? If they get a whole bunch of reports of the form "The paper is okay; it could be published somewhere in between heaven and hell; I don't really have strong feelings about it" then they're going to have a problem....but who is to blame for the problem? Them, of course: they did not find the right referees and were not clear enough in their standards.
Let me end by saying that I have found myself in similar situations more than once: namely, I get asked to referee a paper by a journal I've never heard of, in a subfield of mathematics different from any of the ones I've thought deeply about. And in fact I have usually done more or less the above: sometimes I turn it down (and I have learned to be decisive about this; if I gave some choice to the editor, it always turns out that they want me to do it anyway), but if I can do it I often take the job. A paper should certainly look novel to the relative outsider if it will look so to the expert, and in some cases I have rejected the paper for not making clear progress over (even) its own citations. More often the papers have been a bit interesting. Sometimes I have found significant mistakes: I can't remember a situation where the mistake was so bad that I outright recommended rejection, but there have been situations in which revisions have been necessary in order for the principal results to look correct. (I don't know how this plays out for the shorter timeframe of a conference: presumably outright rejection becomes more likely.) In fact the most common outcome is that I understand the paper well enough and think it's somewhat interesting and novel, though certainly not the kind of breakthrough to be published in a higher tier journal. In these cases I have recommended the paper for acceptance and included in my referee report an honest depiction of the situation: e.g. if I am unfamiliar with the journal, I say so. I believe in every such case the paper has been accepted. This has been fine for me, since having been an author many times and an editor never, fundamentally I am more sympathetic to the situation of a solid paper being rejected than to the plight of a journal that publishes a good paper rather than a great one. "A tie goes to the author," I feel. Given the number of papers I've refereed, this attitude seems to be okay with the editors, who do in fact once in a (great) while cheerfully reverse my recommendations.
