I have on several occasions given students assignments (in math classes I was teaching) to contribute to Wikipedia, either for bonus credit or in lieu of a traditional final paper. Both I and the students were very happy with the results and with the fact that our efforts (which included fairly substantial involvement on my part, see below) resulted in the creation of a valuable resource for the community. I am therefore inclined to view this idea very positively.
That being said, in my experience this type of project would definitely not be suitable for all students and in all circumstances, so one should consider carefully (as you are indeed doing by asking the question) the various implications and factors involved.
For which types of courses does it work and for which it doesn't?
The idea is suitable for advanced classes where you cover topics that are not currently well-covered on Wikipedia. E.g., something like a calculus class is a bad idea, but a graduate class on differentiable manifolds would almost certainly involve several topics that Wikipedia needs help with.
For which types of assignments does it work and for which it doesn't?
The first rule should be "do no harm". What I mean is that some students do not have the writing skills to make a positive contribution to Wikipedia with a reasonable effort of the sort that a course assignment should involve, and we certainly don't want them making negative contributions. Thus, my philosophy is that a Wikipedia assignment should be elective - i.e. either being for extra credit or being an option the student can choose instead of a traditional paper/essay.
Another reason for this rule is that some students would feel self-conscious about writing material for public consumption, and I think it's wrong to force them to do it, even if the contribution is anonymous. Finally, submission to Wikipedia requires agreeing to its Creative Commons license which means giving up certain rights to your creative work. For legal and ethical reasons I think it's untenable to make this a grade requirement unless specifically agreed to by the student who prefers this over alternative assignments.
Will it improve students' motivation to learn?
I've definitely seen students who became very enthusiastic about the writing project and after creating a page on a new topic continued to expand it, which ended with them adding quite a bit more material than was the minimum I required. I'd also like to hope (but don't know if it's the case) that some of them might have caught the Wikipedia bug and continued contributing "for free" later.
On the other hand, some students were clearly just doing it for the grade and didn't do any more than the minimum, so I guess for them there wasn't any improvement to their motivation.
How can one define a rubric for this type of assignment?
Good question. I was using my own subjective judgment to assign grades. Those were small graduate classes and in practice almost everyone got an A or A-, so that wasn't too much of an issue, but for a larger-scale project one might have to give this question some more careful thought.
Let me add a few thoughts about things you didn't ask about. A key thing to keep in mind is that writing for Wikipedia is very different than writing other kinds of content, both technically, stylistically and philosophically (in particular the collaborative aspects). Good Wikipedia content has to be extremely neutral, unopinionated, well-referenced, and written in clear and error-free language. Frankly, in my experience very few students would be capable of producing by themselves a draft for a new Wikipedia article that would not be either outright deleted or heavily modified (possibly to the point of becoming unrecognizable) very soon afterwards by other Wikipedia users/editors -- needless to say that would be quite bad for motivation... For this reason, in the projects I assigned I ended up taking on a fairly substantial role of reviewing and helping polish up the original article drafts before they were officially submitted. This was done in a sandbox page. It was practical for me to help out in this way, first of all because I was happy to do it, and second of all because only a small number of students were involved (4-5 in each of the courses I tried this at). At the same time, there is an obvious problem of scalability here, and I would be very reluctant to attempt such a project in a large class. The bottom line is be prepared to put in quite a bit of work yourself if you want the project to be a success.
A final thought (sorry for the long answer) to consider is that it would be wise for you as the instructor to propose to the students a list of possible contributions they can make (in the form of either new articles or new sections in existing articles). Some students can also come up with their own ideas for what to write about, but I feel that most would prefer being offered a list to choose from. Good luck!
Edit: as @Thunderforge helpfully pointed out in the comments, Wikipedia has a dedicated page with guidelines to students and instructors for Wikipedia-editing course assignments.