Normalising across fields is hard, but it's worth attempting. I would recommend reading this PLOS Biology paper as a good introduction.
This proposed measure is not appropriate because papers do not have an impact factor. Journals do, but there is a very wide distribution of papers within a journal - an impact factor can, in some cases, jump an order of magnitude because of a single paper.
"Mean impact factors" also take no account of publication volume. If you have a journal with an IF of 2 and one with an IF of 3, it's tempting to say that the mean is 2.5. But what if the first journal publishes ten times as many papers? It gets complicated quite fast.
The approach of normalising against the field/year makes sense, but ideally what you want to do is compare the actual citations for the researcher's papers in year X against the actual citations for the field as a whole in year X. The hard part here is defining the "field"; most approaches do what you planned to do here, treat all journals in a field as equivalent to the field as a whole. This is the basis of the source-normalized impact per paper (SNIP) and mean [log-]normalized citation score (M[L]NCS) measures; a more complex approach involving defining the field by citation networks is the basis for the relative citation ratio (RCR). All the usual caveats about counting citations apply, and numbers should be assumed to have fairly wide error bars for single-person data, but the results will make a lot more sense than one purely based on journal-level averages.
These approaches are a bit more labour intensive, but should be practical if you have access to a tool like Web of Science or Scopus.
This will effectively normalise for age of the paper and for the different citation practices in different fields. What it won't normalise for is fractional authorship - these measures can give somewhat confusing results if you're comparing someone who's routinely fifteenth author on large collaborative papers with someone who works in fields where single-authored papers are normal - or with someone who's usually the lead author on the large collaborations. Likewise, it will give misleading results if either person works in a field with substantial non-journal activity (eg the humanities).