0

I am writing a master's thesis in a Machine Learning domain. I created a system and I am now testing it.

I designed most of my system while validating on benchmark set A. The results are good and realistic and I am confident that I can explain the behaviour of the system scientifically. Now I have one month left and I still have to do a lot of writing which I plan to do full-time until the end of the submission time.

Two weeks before, my supervisor asked me to also test my system on benchmark set B to have a better picture of the performance. Adding this took very little time but running evaluations on benchmark set B takes ages: I have to do around 6-10 experiments and now one experiment takes around 1.5 days. I don't have to do anything during these experiments but the latency of getting results is really high.

This is where my problem begins. After a week of evaluating on both benchmark sets, I noticed that the performance on set B is a bit worse, but the really bad thing is: I cannot explain why and I don't think that I have the time to find the explanation.

The easiest solution would be to just drop set B, but this would of course be really bad scientific practice and is not acceptable. The other option is to live with not being able to explain what is going on. This would be unfortunate and I wouldn't know what to write in my discussion section.

Do you have an idea how I can still save the thesis now?

Improve this question
3
  • 3
    What does your supervisor say? You need to make them happy, not random strangers on the internet...
    – Stephan Kolassa
    Commented Nov 11, 2023 at 11:54
  • I didn't aim to make anyone happy. I aim to write a thesis with scientific standards. My supervisor has told me to evaluate it further but is not completely informed about my current status as they do not work on the weekend.
    – Rasi
    Commented Nov 11, 2023 at 13:27
  • Then the solution to your problem is to wait until Monday. No-one here can offer you advice remotely comparable to that from your advisor.
    – Adam Přenosil
    Commented Nov 11, 2023 at 16:44

1 Answer 1

Reset to default
2

You shouldn't expect that the results are the same on different data sets. Any given set is only a subset of all possible results and there are probabilistic effects leading to different outcomes. You say the new results are "a bit different". That should be no concern unless one of the data sets seems to negate the hypothesis. It is only in that case that you have to do the deeper analysis. One of the data sets contradicting the other doesn't tell you which gives the more valid evidence. And in that case you have little evidence for the correctness of your work.

I don't recommend dropping either data set. If they are both broadly consistent with hypotheses or expected outcomes then there should be no issue. Dropping one set would probably overstate the validity of the other.

I agree that further analysis can be valuable as it may lead to deeper understanding.

Since you tagged this with "ethics" let me add that it is usually unethical to hide results, especially unfavorable ones.

Improve this answer
1
  • Thanks for your answer! @Buffy
    – Rasi
    Commented Nov 11, 2023 at 15:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .