69

I have an example demonstrating the solution produced by a popularly used software package is not the optimal solution, despite the software claiming that it is. Several recent papers published from my lab have used this software in their evaluations, and I know it is extensively used in the research community to solve optimization problems.

So I am unsure how to report this bug; it is a proprietary, closed-source software. As a first step, I could contact the software developer about the bug, although they don't seem to have a straightforward method to report bugs.

But a larger issue is that papers published with results based upon a buggy software could be compromised. What if the eventual fix causes the software to take much longer to execute, so that running on large instances is now infeasible?

I have told my advisor about the issue, but he does not seem very interested, possibly because of the potential ramifications.

I am currently working on a project where I could have used this software; I now don't trust it and have switched to a much less powerful open-source alternative. So I guess my question is: how should I handle this situation?

  • 118
    Solve the problem with the open-source software, demonstrate the results are better than those given by proprietary software, publish. What can possibly go wrong? – Dmitry Savostyanov May 16 '17 at 22:06
  • 45
    There are certain branches of optimization that focus on producing a "sufficiently good" solution, rather than the globally optimal one. Are you sure that the problem in your case absolutely requires the globally optimal solution? – lighthouse keeper May 16 '17 at 22:26
  • 38
    "I have an example demonstrating the solution produced by a popularly used software package is not the optimal solution, despite the software claiming that it is." Does the package documentation state whether or not there are specific conditions that must be met/assumptions that must be made for the reported result to be correct? Does your example meet all those conditions? (Necessary vs. sufficient conditions and all that.) – JAB May 17 '17 at 1:56
  • 16
    Without naming the commercial software can you at least name the open source alternative that you have switched to now? – curious_cat May 17 '17 at 3:14
  • 7
    @JAB and lighthousekeeper make excellent points that you need to consider. I would add to it: is their definition of optimal the same as yours? Are there tweakable options that modify the optimization process? – Bryan Krause May 17 '17 at 4:39
140

You should create a reproducible test case that demonstrates the flaw conclusively. If you are that confident and the software is that widely used, it warrants publication in a technical journal. They key is to involve the company. Show them your test case and mention you have interest in publishing the result. Offer to collaborate with them on the publication. That way, they will be forced to either demonstrate you are wrong or admit you are right. In the latter case, they save face by admitting the problem. If they ignore you, press ahead.

  • 42
    This is a good answer. Take care in crafting your letter/email to the company, bearing in mind that you might want to make the letter public in the future. Also, the company might quickly fix the problem, which might make your planned publication less impressive or less likely to be accepted for publication. – user72102 May 16 '17 at 22:36
  • 23
    A colleague of mine did this -- successfully. – HEITZ May 17 '17 at 2:00
  • 9
    @Mehrdad the fact that the bug exists does not warrant a paper. The reasons for the bug to exist + a modified method that would be bug-free likely warrant a paper. Applies to your case as well as to the OP's. – svavil May 17 '17 at 11:46
  • 12
    @HEITZ: Calling a solution x optimal implies there's an f(x) such that no y exists for which f(y)>f(x). If software reports x is optimal, and yet there exists an y such that f(y)>f(x), then that assertion is incorrect, i.e. a bug. – MSalters May 17 '17 at 12:59
  • 47
    @Mehrdad The fact the bug exists is not publishable. The fact that the bug potentially affects the conclusions of previously published papers is. If you can demonstrate that the bug is relevant to the results of previously published papers, that's worth publishing. Ideally, you'd have at least one example of a previously published paper where the conclusions are changed because of the bug, but depending on journal even a "we can no longer be certain of the strength of these conclusions" works. -- It's not so much publishing the bug, but publishing "these other papers may be wrong". – R.M. May 17 '17 at 16:15
34

The technical answer to the question seems obvious: if there is a clear definition of what "optimum" means for your problems, run both programs and see which one wins, then publish the results.

Any advice on the politics of how to proceed is merely speculation, until you have done the technical work to discover the facts of the case.

These things happen. We once discovered a "schoolboy error" in one of the industry-standard software packages in its field. In the meeting where we presented the results, we had three senior members of the vendor's development team (which totaled about 300 people) sitting with their heads in their hands, knowing full well that if we wanted to we could destroy maybe 75% of their international user base. (Some of the other 25% wouldn't understand what the fuss was about, and ironically their biggest individual customers were likely to be using computer systems where the error didn't occur anyway - which might explain why nobody else had spotted it in the previous 10 years or so).

But there was much more benefit to us in getting the problem fixed rather than nuking the software house, so we went for the "just fix it ASAP" option - and "ASAP" actually took about 2 weeks, to get a beta-test version.

  • 1
    What journal is interested in publishing "I ran the same optimization on two systems. They gave different answers."? – David Richerby May 19 '17 at 10:26
  • 10
    You seem to grossly overestimate the impact of bugs on the popularity of software products. If a single bug was sufficient to destroy 75% of a product's user base, Mathworks would have been long out of business, never mind Microsoft. – Dmitry Grigoryev May 19 '17 at 12:45
  • @DmitryGrigoryev this is fairly similar: Intel Pentium rounding errors – Stephen S May 19 '17 at 20:40
  • 3
    @DmitryGrigoryev It depends on the field, surely? i.e. whether to the specific customers the bug represented you goofed, and that is really very annoying, but I guess that's just how it goes - or you failed so fundamentally that you posed a risk to our credibility/safety/etc. If a bug is serious enough to fall into the latter category, then maybe 75% isn't such a wild estimate... well, especially if the software only has 4 users. :P – underscore_d May 20 '17 at 17:43
  • @StephenS Again, really depends on the field and who's judging. Specialist users of very specific software that might serve an extremely important function are very different from, e.g. a lay user who doesn't even know floating-point exists, never mind how Intel fudged it or how it threw their budget off by 2p that one time. Plus, huge companies like Intel typically don't suddenly fall from grace and get swept away, but specialist software houses might easily. – underscore_d May 20 '17 at 17:47
19

I have an example demonstrating the solution produced by a popularly used software package is not the optimal solution, despite the software claiming that it is.

This seems to be a lot of fuss about possibly nothing.

Read the software documentation, if any. It should explain in what cases it declares "solution found". It is usually the case that this indicates that several parameters associated with a necessary condition for optimality are sufficiently small, which indicates that the solution was probably found. No guarantees here.

If you can't find a reason for this behaviour in the documentation, write to the developer with a minimum working example where the unexpected behaviour occurs and ask for clarification.

7

I have an example demonstrating the solution produced by a popularly used software package is not the optimal solution, despite the software claiming that it is.

So the software may claim too much. This is not that uncommon.

So I am unsure how to report this bug; it is a proprietary, closed-source software. As a first step, I could contact the software developer about the bug, although they don't seem to have a straightforward method to report bugs.

Contacting the manufacturer in any suitable way, if no special way is indicated, would be the recommened procedure if you want to help the manufacturer (and thereby indirectly also yourself and others). Just make sure the bug does really exist and you explain it thoroughly so it can be reproduced.

But a larger issue is that papers published with results based upon a buggy software could be compromised.

Probably all software is somewhat buggy. Without knowing details it's difficult to judge the severeness of this case. However, it's always a good idea not to rely too much on one software package but test concurrent products as well. You could check these papers and see if they are affected by the bug and if so, how much they are affected.

What if the eventual fix causes the software to take much longer to execute, so that running on large instances is now infeasible?

A fixed software taking longer to solve a problem correctly is always preferred to a buggy, quick software. It would just mean that correct solutions to large instances were never feasible so far.

I have told my advisor about the issue, but he does not seem very interested, possibly because of the potential ramifications.

Maybe he doesn't care so much about using bug-free software or is not yet convinced of the existence of the bug or is just too lazy to bother the company or doesn't want any conflict.

I am currently working on a project where I could have used this software; I now don't trust it and have switched to a much less powerful open-source alternative.

It's natural to lose some trust but maybe not all. You could carefully evaluate the old software (keeping in mind that bugs occur occasionally) if it still can be trusted in the special circumstances of your project, then decide if you want to use it or if you want to use both software projects or if you want to use only the alternative software (which also could be buggy, so don't trust that one completely either).

So I guess my question is: how should I handle this situation?

  • You should contact the company and write them a short message telling them about the error. They may quickly fix it, restoring trust.
  • You might also publish the bug (on your blog, on a mailing list, even as a technical report if the bug is important enough and a journal is interested in that) so others are aware of it, especially if the company does not quickly fix it.
  • You should use the two software packages that you have and compare them. Assume that both may have bugs you don't know anything about them yet.
3

I think the first question you need to answer is just how much of an issue this actually is. Is this only an edge case? Is the "optimal" solution only marginally better? It is hard to say much without more details about the actual use case and solutions. With some of the optimization/fitting problems I worked on for my PhD, you could dig around forever finding "more" optimal solutions, but depending on how noisy the data was, that didn't actually mean that the new solutions were "better". With a good understanding of your problem set and the underlying optimization algorithms, you should be able to figure out what exactly is going wrong with the closed source software, and whether or not there is an actual problem.

For the rest of this discussion I'm going to presume that you are trying to find the optimum fit to some sort of empirical data set. If so, you are effectively searching some sort of chi^2-parameter space, you finding a more optimum fit can mean very different things, e.g.

  1. Your chi^2 space is noisy and a "better" solution doesn't actually mean anything
  2. The proprietary system is getting stuck in a local minimum, and isn't finding the global minimum you found
  3. You are both probing the same global minimum but your solution is just slightly closer

Effectively, the process of finding the optimum solution can be very tricky and there are a number of different algorithms that can be used, each with their own benefits and drawbacks. If you have a smoothly varying relationship between your fitting parameters and chi^2, just about any optimization method can find the minimum, but life isn't always that simple. All this to say that even if you have found a better optimum in one particular problem, this doesn't mean that the software is trash and should be thrown away. In particular, how did you find the better optimum? Can you apply that same method to these kinds of optimization problems in general? Will it always give a better fit? If the answer to either of those last two questions is no, I would say that there really isn't much of a big deal here at all. Most likely the only real problem is with an over-zealous marketer with the software company in question, who has promised things that no software system can realistically deliver.

In summary, I would say that before you really run with this you need to really understand in detail why their algorithm is not finding the most optimum solution, and if that even matters in practice.

2

I have seen well-known commercial optimisation packages such as CPLEX producing wrong result under a stress, so to speak---solving problems which are on paper solvable with the said software, but in practice sufficiently dissimilar from what they are tested on, and targeted at.

By wrong I mean as wrong as it gets, i.e. a linear optimisation problem is reported infeasible, while a request to produce an infeasibility certificate fails (and if solved with a higher precision on a slower open-source solver, the problem turns out to be feasible).

Surely sooner or later the error, if there was an error, will be found and exposed, this is how science works. Incidentally, in the past couple of years I published papers, which, among other things, correct errors in a monograph and in textbooks (sic!); the corresponding erroneous papers were published over 20 years ago. Oh well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.