How can I assess the quality of a paper outside my field?

Question

As per the title, I would like to determine whether a paper cited by a student comes from a reputable source or not, given that the paper's content lies outside my field. My first thought was to check the journal's website, as well as doing a Google Scholar search to see whether papers in other reputable journals cite papers from this one.

The particular case I am concerned with is a Psychology paper, appearing in AllPsych Journal [sic]. All the journal's webpage says is that:

The educational articles and papers published in AllPsych Journal are written by mental health professionals, psychology students, psychology instructors, and other authors outside the field of psychology. * They represent problems, solutions, and suggestions about everyday life and its relation to psychology. For submission consideration, please contact us via our feedback form.

This seems dubious to me, but nonetheless, a Google Scholar search suggests that some reputable looking publications have cited works appearing in this journal.

I suppose I have two questions then:

1. Are there any other techniques I can use to assess the quality of journals outside my field for this purpose. Ideally these should take relatively little time.
2. Is this specific journal considered reputable in psychology?

Answer 1

This question concerns assessing the quality of papers outside one's field.

The question seems to be more leaning towards the quality of journals outside of one's field. Here's a brief list of pointers (some of which also appear in this question):

• check if the journal has an impact factor (some good journal's don't and vice versa, so beware of false positives and false negatives)
• find out the most prominent literature indexes in the area and see if it's in there (e.g., DBLP for Comp. Sci., PubMed for life sciences, etc.)
• check the Australian ranking tables (don't take them too seriously: there is no such thing as a total ordering of journals)
• check for journal citations in Scholar/Academic Search/Scopus
• otherwise check paper citations in Scholar/Academic Search/Scopus
• find out if the publisher is reputable (Elsevier, Springer, a good university press, etc.)
• check Beall's list to make sure the publisher is not "predatory"
• find out if the editors-in-chief are established academics (e.g., high h-indexes in Scholar)
• talk to experts in the area
• apply common sense.

The last point is particularly important. Though it is easy to discover if a journal is excellent or terrible, in between those extremes, there's a lot of grey. For example, new journals may be much-needed and drawing quality work from quality people, but they won't have a high h-index or an impact factor, etc.

For those grey areas, talking to an expert who knows the field is important.

AllPsych Journal

This doesn't seem to be a formal journal, but rather an informal way for authors to publish educational articles online. All papers are published on the web-site, there's no mention of an editorial board, a publisher, a means of submission, etc.

Of course, this says nothing of the quality/utility of the papers published there: just that it would probably not count in academic circles as a "traditional journal".

For the quality of a paper itself independent of the venue, check how many citations it has in Scholar versus it's age, check the nature of those citations, check the h-index and background of the authors, etc. (Of course these methods are only an approximation for reading it and making up your own mind as to its quality, but since it is outside your area ...)

Answer 2

@badroit already gave a very good answer on the general question.

As to the specific question about AllPsych: this is not a journal per se, it is only a website with submissions from different people run by one person, apparently without any peer review, see here. I would rate this as only marginally better than Wikipedia. Whether it is admissible as a source in a student paper could be treated the way you treat Wikipedia entries.