Where can I get help with this?
You could try asking his colleagues or other professional acquaintances (including past coauthors of his), or former graduate students. Even if they can't help, they may be able to direct you to more appropriate people.
If you can't track down any relevant individuals, you could try asking the current chair of his former department or trying to contact someone at any relevant professional societies he was a member of. They probably couldn't help directly, but they could try to put you in touch with those who could. It's not really their job to do this, so there are no guarantees, but I bet they would be sympathetic.
Would something like American Journal Experts be a good idea?
Almost certainly not. This seems to be a company that offers editing and formatting assistance for papers, which isn't really the issue here. Rather, you need scientific assistance from someone who knows the research area well (to help flesh out the ideas and connect them with the literature), and it's not likely that any company can offer that.
I contacted IOP publishing but they said that, while Dad’s writings were interesting, they were not ready for publishing in any of their journals. For example, they need a reference list citing other current published research on the same topic.
It's hard to know what to make of this. One possibility is that the papers were unfinished but just need a little more work, and someone could complete them without great difficulty. Another possibility is that the papers were only fragmentary, with interesting ideas that still require major work. In that case, they are unlikely to be published in their present form but might still inspire further work (which should credit your father). A third possibility is that these writings are not really suitable for publication, and the editor was just being polite and focusing on the most objective reason.
If you are unable to get the papers formally published, you may still be able to make them available and permanently archive them through the arXiv. This is a web site that offers open access to research papers in physics, among other fields. It would ensure that your father's work is not forgotten.
Assuming your father's name is also Krogdahl, he has two papers on the arXiv. If you decide to try posting his more recent writings, I'd strongly recommend first consulting with someone in the field (such as a former colleague) about what is worth posting and how to do so, as well as getting in touch with the arXiv administrators. The arXiv is a little finicky about various aspects of submissions, so it's best to handle things carefully.
