10

I wasted spent like a month or so trying to get two software libraries to work. They partly worked, but at some point produced only unplausible results. That might have very well been my fault.

The third one worked right out of the box and it's the one I should have chosen to begin with. The lack of problems with this library suggests that there was either something wrong with the other two, the setup, the system or whatever.

Either way, the conclusion is that I would not suggest to try the first two libraries and go with the third one, no matter how tempting it is to try the two.

I would like to include this in my thesis, because it appears to be valuable information. On the other hand, I'm not a software expert and I might simply have done things totally wrong, which is why those libraries didn't work for me. But despite some effort, I could not find documentation on what could be the problem.

Basically, what I have to say is this:

Tried X and Y. Didn't work. Don't know why. Not recommended. Use Z instead. Works perfectly.

Is it a good idea to include such information? It's not backed up by any reference and I cannot include all the related information about the details to reproduce the problem. Partly because I don't know them and partly because that's not what my work is about. I'm also afraid that including this information will make me look incompetent.

How should I include this information? I think the best thing to do is to state that under the overall time constraints getting either of the two libraries to work was not explored any further, which is what actually happened. But then this might look like I didn't spent my time efficiently.

Maybe I'm just too much concerned that there will be only glass-half-empty people reading this, trying to see the problem in it.

  • X and Y are independent substitutes for Z. Either one of the two working would be sufficient. They are written in a specific programming language (different from Z) which would make them a good choice for the project. X and Y are poorly written and/or documented, but then I do not have the know-how to make that judgement call and it looks like I'm looking for cheap excuses. – clueless Oct 16 '16 at 22:54
  • More in my answer, but in "what I have to say", I would not include "don't know why". I would say everything else. After "didn't work", I would say "this is what did not work about it. this is what I did to try to resolve it. In the end, not recommended and recommend to use z instead. Works perfectly right out of the box." – user51808 Oct 17 '16 at 15:33
10

I would have to disagree with the previous answers. I would consider documenting what did not work. I am not saying that I would definitely do it, but I am saying that ruling out the idea, out of hand, is not good advice. I have 27 years of experience in software development and project management, and I would very seriously think about at least a short section where I would lay out what I tried. Sounds like there are some serious holes in your knowledge of those libraries and your attempts to use them, which might make it difficult to document what you did, but I encourage you to give it a shot and run it by your advisor.

In the end, you could take the section out if it ends up detracting from the finished thesis. But I find that kind of honest appraisal of failed attempts to be often more informative than the success stories. I find it to be counterproductive to the community and the researcher when academia is reluctant or completely unable to admit when something did not work or the researcher had a knowledge gap. I find the disarming honesty of "I tried and it did not work" elevates my opinion of a researcher. But there is so little of it, that I find myself going to non-academic sources much more often than not, when I want to find answers that are real and practical.

Be careful in how you write it, and how you make the final decision to include it or not. If you can say, I tried this library. This is what it gave me. This is why it was not suitable. I tried to find relevant documentation to resolve the issues, but no such documentation could be found. And in the end, the ability of this project to integrate a proper library is demonstrated in the use of X library, which worked. If you can frame it like that, informative, honest, yet demonstrating your competence by highlighting, once again, that you did successfully integrate a library that worked, you could add some real value to your thesis.

The main reason for not including such an admission, in my opinion, would only be if your ability to integrate the third library was more luck than real skill, if it did not demonstrate a concerted effort aided by much better features and documentation. If that is the case, then forget what I have said and go with the other answers' advice.

  • From the domain specific point of view I felt that the input I provided was reasonable, but the output of the first libraries was not. Throwing the same input at the same functions in the third library immediately produced a plausible result. I will write something up. You're right, I can always check with advisor and remove it later. Maybe I'm overthinking this. – clueless Oct 17 '16 at 15:47
  • 3
    Overthinking things seems to be a common human condition. I tend to go there a lot myself. Just remember, don't frame it as "I don't know why these failed". Rather, tell us what you do know about those failures. Nobody should expect you to know everything, and no one can ever document everything they don't know - self defeating task. Tell me what you do know about your misses, and that is valuable to me. – user51808 Oct 17 '16 at 16:10
8

Report what did work. Forget the rest. Anyone looking to reproduce or expand on your results needs to know the software you did use.

Edit: I note with some chagrin that I answered other than what OP asked, namely how to include unfruitful attempts to use software X and Y.

At the point where you describe your use of package Z, include a footnote that you first attempted to use X and Y, but received implausible results. One sentence is enough. That's enough to warn anyone who cares that you had difficulty, but does not belabor problems that you cannot explain. It might even prompt someone to ask you for more information some day.

  • 3
    In the course of expanding on the results, they might try to replace Z with another (potentially better) library, thereby replicating the initial fruitless efforts. I would report these efforts. – lighthouse keeper Oct 17 '16 at 5:51
  • Since OP isn't clear why his efforts were fruitless, someone who does that might very well succeed and learn something new. Reporting specific errors: Good. Reporting general and unexplained difficulty: Bad. – Bob Brown Oct 17 '16 at 11:44
  • 2
    But then it's unclear why it took me so long to do this. Also, "someone who does that might very well succeed and learn something new." but what prevents them from doing that if I mention that it didn't work for me? Isn't the opposite true: if somebody mentions failure and couldn't investigate fully, this is a possible target for future research? By ignoring those results entirely, the obvious question that it will cause is: Why were X or Y not tried? I doubt that not mentioning it at all is a good idea. – clueless Oct 17 '16 at 12:30
  • 1
    It would really (at least in computer science and related fields) help a lot of people if everyone finally would start to acknowledge and publish what did NOT work. If you really have to omit this from your main body of work, please include it as a supplementary or appendix. Many PhD students and other people following after you will appreciate and potentially even cite this if asked why they did not try method/package X, but instantly used Z. – skymningen Oct 17 '16 at 13:08
  • Keeping it minimal like in a footnote sounds like a good idea. +1 – clueless Oct 17 '16 at 15:58
1

I think in order to answer this, you have to find out why those other libraries didn't work. Maybe talk to someone who knows them well.

  • If they didn't work because you didn't do them right, then you don't include them because it doesn't help anyone else.

  • If they didn't work because there's something wrong with them, then that's worth at least a footnote. I would probably write a single sentence about what library you did use, then in the linked footnote mention that you tried these other two libraries but found they behaved wrong due to X.

If there's one thing I've learned as the rare social scientist with a wide range of programming skills, it's that nearly everyone thinks the software they know best, is the best. If there's an actual problem with one of the packages, you need to mention that because those who prefer it will just assume it's right and then fail to replicate your results. But I wouldn't pay more attention to it than a footnote.

1

As Bob Brown said earlier, report in the thesis what did work (Z), and omit everything else (X and Y). Church spent 1 year until he found a 1-liner for subtraction. We know little about his failed attempts. Certain excellent dissertations are 10 pages long proving a theorem. We know nothing about failed proofs.

In some sense, doing research means getting dirty. Very dirty. And after you made the point, you clean off all the dirt, and the main result remains.

However, you were probably financed by someone during thesis writing or received some other kind of help. (John Donne: "No man is an island entire of itself; every man is a piece of the continent, a part of the main".) So do report about the failures, bad experiences, and good experiences to your stakeholders. This is a different kind of report and there, you should mention X, Y, and Z by all means. How you do it (in a weekly meeting / as a technical report / in a research report / as a phone call / during dinner / in bed before going to bed with your spouse ;-) / ...) is very, very project-specific and stakeholder-specific.

In fact, the more I think about it, the more I am likely to encourage you to say that X and Y (in versions xxx and yyy) are poorly written/documented publicly if it is the case. You have a chance of making enemies this way, but, in general, the research community would benefit from it. Boldy (but, unfortunately, truthfully) stated, we would benefit from it at your cost if you do that.

  • 2
    "you were probably financed" no I wasn't. I just got a task for my thesis and that's what I'm doing. I find it odd that the people with the money should be informed but the research community should not. – clueless Oct 17 '16 at 13:04
  • I obviously told my supervisor that it didn't work. But I doubt that just telling somebody orally at one point in time is a proper way to document results. And no, there is no such thing as a "separate report". Why would I want to split this information off from the thesis anyway? What's the motivation behind this idea? – clueless Oct 17 '16 at 15:14
  • "you'd make some enemies" how so? Why would I make enemies? I'm new to all this and am not aware that I can make enemies by including information. Can you elaborate? Also, have an upvote for your explanations so far. – clueless Oct 17 '16 at 15:33
  • The authors of libraries X and Y would not be happy to see this published. – user7019377 Oct 17 '16 at 15:43
  • 1
    @user7019377 huh? Are results taken that personally? If it didn't work, it didn't work. It's not an attack or something. – clueless Oct 17 '16 at 16:13
0

You can give the information about technically failed attempts upon request, e.g. in talks or conference presentations. Someone may ask: "You said you used library Z. Why not X or Y?" If nobody asks, don't bother. They are obviously fine with Z working, so should you be.

However, the issue about "unplausible results" is in fact an issue for a closer look. A computational routine may never give flawed results that look correctly, i.e. without at least a warning like "error bounds not met". So if you have found a bug in the libraries, you should contact the developer or support or at least ask some colleagues to reproduce this bug. If you are not sure that it is a bug, you should be careful with badmouthing other people's work in your publications.

  • Again, I think this is valuable information. In a way worth a monthly salary. Why would I limit this kind of information to only be shared at talks or presentations. ? This audience is limited to maybe 5 people at max in my case. Is academia like an ancient culture where some information is only passed orally between a few people? And you only get the information you might find useful when you happen to be at that specific talk? – clueless Oct 17 '16 at 15:08
  • I think your distaste at academia's secrecy on failure is exactly right. I respect a person more when I hear them say "I tried this and it failed. It did not give me the results I needed and I could not find useful documentation to resolve the issues, so I tried a different approach (library), and was able to get the job done right." But academia tends to want to hide such information, too often in the false belief that it represents weakness, when failure is part and parcel of life, especially when you are trying new things, and research should be very much about trying out the new. – user51808 Oct 17 '16 at 15:25
  • Updated because I didn't give enough attention to the issue of "unplausible results". – Horst Grünbusch Oct 17 '16 at 15:31
  • Just to be clear here: I'm not intending to badmouth anybody. And pointing out that it didn't work for me is not badmouthing either. – clueless Oct 17 '16 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.