I am replicating a model to report on how well it does on a new evaluation metric I am proposing. The method/hyperparameters of the model are given in a paper about it, where it breaks all benchmarks on the tasks on which it is being evaluated.
After that paper came out, several people tried to replicate the results, and found that they could not. Their questions of “Am I doing this right?” can be found on a online message board for the project. They got several responses from the second author of the paper (very well-known and respected in the field), saying that he didn’t carry out the experiments and that they would have to wait for a response from the first author.
Eventually the first author responded with just a few lines saying: “The hyperparameters to use are ....” People ran with those parameters and replicated the results.
However, the parameters he gave disagree with those reported in the paper. Quite likely this is through no malicious intent: possibly the model was rerun and the changes were not re-incorporated into the draft.
Now, I would like my model when I am evaluating to be as similar as possible to the one in their paper. That way, people can compare my new proposed metric to the metric that that paper uses and see that this model is good at X, but less good at Y, without having to correct for the fact that the models are different.
I have three options:
- Use what I think are good hyperparameters for their model (this is what I am currently doing).
- Use the hyperparameters given in their paper.
- Use the hyperparameters given in the online message board post, which actually achieve the results given in the paper.
Which of these three options is best?