Some aspects of computer science research requires some form of implementation and an according evaluation. I have not yet seen where the minimum level of implementation expected for an academic paper is defined. For example, if a "novel" work requires a persistence layer and a researcher uses a file based method e.g. csv. Will that be wrong? Will his architecture point to a database or a file system ?

  • I think you need to say more to be understood. Every paper is different. Every paper is evaluated by (hopefully) competent referees. If you are lacking you will be told. But if you have an advisor, he or she should be able to give you explicit advice. – Buffy Jul 30 '18 at 21:00
  • @Buffy I agree papers differ, that's why I give a simplistic example in the question about the database implementation. – SyCode Jul 30 '18 at 21:03
  • Still, too esoteric to judge or give advice. Things can be novel in some ways and pedestrian in others. But the paper has to make an advance on some aspect if it is to have much value. – Buffy Jul 30 '18 at 21:06
  • 1
    The general nature of this question is frustrating. I'm not sure that there's a single requirement for all of computer science research and a "minimum level of implementation" is hard to define for all academic papers. I'm not sure that the "ethics" tag is particularly relevant. – St. Inkbug Jul 31 '18 at 6:31

Your implementation needs to be good enough to evaluate the concept you want to evaluate - no more, but certainly no less. What this means in practice is largely dictated by what kind of evaluation or experiments you want to do. I have basically seen four broad classes in Computer Science:

  • Proof-of-Concept (PoC): you only want to show that your idea can be implemented, so you only implement your idea (often in a fairly naive or bare-bones manner). Note that this is often good enough for a quick demo, but not good enough for a paper (because reviewers usually feel that the conclusion that something can be implemented is not satisfactory by itself).
  • Evaluation Prototype: in addition to the PoC, you want to evaluate how well your idea works in some quantitative manner. For instance, if you are building some novel file system, you probably want to not only show that it can be implemented, but also that it works better in some dimension (e.g., faster) than existing systems. Your prototype can still be bare-bones, but you presumably need to build the core parts well because otherwise your evaluation may end up showing primarily that your prototype sucks. In a way, your prototype results show a sort of lower bound on how your idea works - that is, if your results are good your idea is validated, but if your results are not good, it may be because they idea does not work or your code is too bad of an implementation.
  • User Study Prototype: in addition to quantitatively evaluating your prototype, you may want to run one or more case or user studies with it, or conduct a controlled experiment. Now you need to improve your evaluation prototype with at least a limited amount of bells and whistles, because it is hard to run an expressive user study if your system is almost unusable to anybody but yourself. In this stage you may be adding a GUI and a bunch of features that are unrelated to your science, but which may be needed even in trivial practical usage.
  • (Simple) Product: the final stage is that you plan to actually release your implementation in some manner to the public. Maybe as part of an open source project, as a consulting delivery to a company, or even in the context of a spin-off company. You will now need to add many more bells and whistles unrelated to your science. Further, now you need to start caring about documentation, a release cycle, not doing breaking changes, etc., all of which are basically irrelevant as long as you are just building a prototype for your science.

So what does this mean for your project?

For example, if a "novel" work requires a persistence layer and a researcher uses a file based method e.g. csv. Will that be wrong? Will his architecture point to a database or a file system ?

I generally suggest my students to follow a lean approach: what kind of experiment do you plan to do? Will it matter what kind of persistence layer the prototype has? If no, then build whatever you can do quickest and where you are most confident that it will work correctly - but keep in mind that even the simplest prototype can sometimes run through all the phases above, so at least plan for the possibility that you may need to swap out your persistence layer for something a little more state-of-the-art than a flat text file later on.

  • awesome response. Thank you for comprehensively explaining this to me, it is much clearer. I have a related question, in the case of a demo paper, will the implementation fall into category 2 above ? – SyCode Jul 31 '18 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.