Is it necessary to justify every single choice in a proposed method with concrete data?

Question

I'm writing a master's thesis about a novel method. It is a multilayered algorithm with very large number of configurable parts and parameters. Furthermore, it involves machine learning and big data, which means each testing run is quite a substantial resource and time-consuming process. That's why researching the effect of every single parameter of this method (or worse, the combinations of possible parameters) would have been an enormous task for a single person with a regular, unrelated job.

Is it acceptable then to omit providing data/results for some less significant parts of the algorithm, and just justify these choices with a more or less vague description of why they seemed promising or "good enough"? Should I mention limited time as a reason for omitting that data?

Answer 1

Omitting anything is rarely a good idea; consciously not doing some of the experiments and giving some reasoning for that is perfectly fine and happens all the time in research.

Citing limited time as a (personal) reason is likewise a terrible idea, IMHO.

Good:

• With 37 hyperparameters, the search space is enormous. Parameters $\alpha$, $\kappa_1$ and $\kappa_2$ are of special interest, as indicated by prior research (Claus, 2019). In addition, $\lambda$ is pertinent to the problem because of the Moon phase constantly changing. Other parameters were left at their default values as they were deemed less significant (see Table 42 for details).

Bad:

• I am working a full-time job and don't have the time to tune all 37 parameters so I toyed with just 4 hoping it's good enough.

The difference here, of course, is that in the first case you refer to some objective truth: experiment would take a long while for anyone and at the current state of research, it is more fruitful to focus on a handful of important parameters and only concern yourself with other, less significant ones, later.

Answer 2

There are a couple of interesting ways of dealing with this, but mainly consider that you need to enclose or delimit your experiment's reach and expectations.

Since most of the hyperparameters are tuned by hand, stating that your p1=0.5 for empirical reasons is perfectly fine.

As @Lodinn mentioned, focusing on those parameters deemed as important is also a good idea. Since some of them are not that impactful, their analysis is not in the scope of this work. You could also consider them as future work :)

Edit: Clarifying: Yes, it is OK to left experimentation settings unexplored, but you need to provide a reasonable explanation on why you decided to do that.