5

I am replicating a model to report on how well it does on a new evaluation metric I am proposing. The method/hyperparameters of the model are given in a paper about it, where it breaks all benchmarks on the tasks on which it is being evaluated.

After that paper came out, several people tried to replicate the results, and found that they could not. Their questions of “Am I doing this right?” can be found on a online message board for the project. They got several responses from the second author of the paper (very well-known and respected in the field), saying that he didn’t carry out the experiments and that they would have to wait for a response from the first author.

Eventually the first author responded with just a few lines saying: “The hyperparameters to use are ....” People ran with those parameters and replicated the results.

However, the parameters he gave disagree with those reported in the paper. Quite likely this is through no malicious intent: possibly the model was rerun and the changes were not re-incorporated into the draft.

Now, I would like my model when I am evaluating to be as similar as possible to the one in their paper. That way, people can compare my new proposed metric to the metric that that paper uses and see that this model is good at X, but less good at Y, without having to correct for the fact that the models are different.

I have three options:

  1. Use what I think are good hyperparameters for their model (this is what I am currently doing).
  2. Use the hyperparameters given in their paper.
  3. Use the hyperparameters given in the online message board post, which actually achieve the results given in the paper.

Which of these three options is best?

Improve this question
  • What are you asking? – user6726 Aug 21 '15 at 5:01
  • 1
    @user6726 Which of the 3 options presented at the bottom is best. – Lyndon White Aug 21 '15 at 5:06
  • Considering that you have limited space, can the results of their model be compared directly across papers, or are they quite sensitive to the particular implementation? – user38309 Aug 21 '15 at 5:35
  • 4
    How much time and efforts would cost you if you do all three options? – scaaahu Aug 21 '15 at 5:53
  • 4
    While this is not your question, something that you should definetly do is to see that the paper gets amended or there is an erratum. – Wrzlprmft♦ Aug 21 '15 at 6:55
15

Given that you don't have space enough to present all three sets of results, I would suggest that you choose the parameters that best illustrate the scientific point that you are aiming to make.

When you state the parameters, however, you should add a parenthetical note or footnote that says something along the lines of:

Note that we have chosen these parameters because [REASON]. Alternative parameters given [in the original paper / by the author] produce results that are [better / worse].

This balances the openness of making sure the reader is aware of the choice you have made and its consequences, while at the same time restricting your presentation space budget to only the most relevant set of results.

Improve this answer
5

I would propose that you include all three parameter sets and their respective results in your paper. In that way it will be more insightful for your readers to understand the influence of the parameters, and it will be easier for others to replicate the results that have been reported by the previous authors.

Improve this answer
  • It is a conference paper (this is CS where conferences are a big deal) and has a tight page limit. I have many other models to talk about beyond this one. Good advise that I may take in an extended form though. – Lyndon White Aug 21 '15 at 5:11
  • Of course you will have to balance page limits with the importance to include all parameters. I can understand that. Yet, you might put the results in a table (just three rows), or even in a single sentence. – Danny Ruijters Aug 21 '15 at 5:17
  • But I would have to explain the different parameters (1 sentence per parameter set). And then more if I wanted to explain why these parameters were being used. And it would add to all tables and all graphs . – Lyndon White Aug 21 '15 at 5:20
  • Ok, I agree that it might be more suited for an extended version, as you already have written. – Danny Ruijters Aug 21 '15 at 5:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.