Popular software used in research is incorrectly reporting solution is optimal: what to do?

Question

I have an example demonstrating the solution produced by a popularly used software package is not the optimal solution, despite the software claiming that it is. Several recent papers published from my lab have used this software in their evaluations, and I know it is extensively used in the research community to solve optimization problems.

So I am unsure how to report this bug; it is a proprietary, closed-source software. As a first step, I could contact the software developer about the bug, although they don't seem to have a straightforward method to report bugs.

But a larger issue is that papers published with results based upon a buggy software could be compromised. What if the eventual fix causes the software to take much longer to execute, so that running on large instances is now infeasible?

I have told my advisor about the issue, but he does not seem very interested, possibly because of the potential ramifications.

I am currently working on a project where I could have used this software; I now don't trust it and have switched to a much less powerful open-source alternative. So I guess my question is: how should I handle this situation?

Answer 1

You should create a reproducible test case that demonstrates the flaw conclusively. If you are that confident and the software is that widely used, it warrants publication in a technical journal. They key is to involve the company. Show them your test case and mention you have interest in publishing the result. Offer to collaborate with them on the publication. That way, they will be forced to either demonstrate you are wrong or admit you are right. In the latter case, they save face by admitting the problem. If they ignore you, press ahead.

Answer 2

The technical answer to the question seems obvious: if there is a clear definition of what "optimum" means for your problems, run both programs and see which one wins, then publish the results.

Any advice on the politics of how to proceed is merely speculation, until you have done the technical work to discover the facts of the case.

These things happen. We once discovered a "schoolboy error" in one of the industry-standard software packages in its field. In the meeting where we presented the results, we had three senior members of the vendor's development team (which totaled about 300 people) sitting with their heads in their hands, knowing full well that if we wanted to we could destroy maybe 75% of their international user base. (Some of the other 25% wouldn't understand what the fuss was about, and ironically their biggest individual customers were likely to be using computer systems where the error didn't occur anyway - which might explain why nobody else had spotted it in the previous 10 years or so).

But there was much more benefit to us in getting the problem fixed rather than nuking the software house, so we went for the "just fix it ASAP" option - and "ASAP" actually took about 2 weeks, to get a beta-test version.

Answer 3

I have an example demonstrating the solution produced by a popularly used software package is not the optimal solution, despite the software claiming that it is.

This seems to be a lot of fuss about possibly nothing.

Read the software documentation, if any. It should explain in what cases it declares "solution found". It is usually the case that this indicates that several parameters associated with a necessary condition for optimality are sufficiently small, which indicates that the solution was probably found. No guarantees here.

If you can't find a reason for this behaviour in the documentation, write to the developer with a minimum working example where the unexpected behaviour occurs and ask for clarification.

Answer 4

I have an example demonstrating the solution produced by a popularly used software package is not the optimal solution, despite the software claiming that it is.

So the software may claim too much. This is not that uncommon.

So I am unsure how to report this bug; it is a proprietary, closed-source software. As a first step, I could contact the software developer about the bug, although they don't seem to have a straightforward method to report bugs.

Contacting the manufacturer in any suitable way, if no special way is indicated, would be the recommened procedure if you want to help the manufacturer (and thereby indirectly also yourself and others). Just make sure the bug does really exist and you explain it thoroughly so it can be reproduced.

But a larger issue is that papers published with results based upon a buggy software could be compromised.

Probably all software is somewhat buggy. Without knowing details it's difficult to judge the severeness of this case. However, it's always a good idea not to rely too much on one software package but test concurrent products as well. You could check these papers and see if they are affected by the bug and if so, how much they are affected.

What if the eventual fix causes the software to take much longer to execute, so that running on large instances is now infeasible?

A fixed software taking longer to solve a problem correctly is always preferred to a buggy, quick software. It would just mean that correct solutions to large instances were never feasible so far.

I have told my advisor about the issue, but he does not seem very interested, possibly because of the potential ramifications.

Maybe he doesn't care so much about using bug-free software or is not yet convinced of the existence of the bug or is just too lazy to bother the company or doesn't want any conflict.

I am currently working on a project where I could have used this software; I now don't trust it and have switched to a much less powerful open-source alternative.

It's natural to lose some trust but maybe not all. You could carefully evaluate the old software (keeping in mind that bugs occur occasionally) if it still can be trusted in the special circumstances of your project, then decide if you want to use it or if you want to use both software projects or if you want to use only the alternative software (which also could be buggy, so don't trust that one completely either).

So I guess my question is: how should I handle this situation?

• You should contact the company and write them a short message telling them about the error. They may quickly fix it, restoring trust.
• You might also publish the bug (on your blog, on a mailing list, even as a technical report if the bug is important enough and a journal is interested in that) so others are aware of it, especially if the company does not quickly fix it.
• You should use the two software packages that you have and compare them. Assume that both may have bugs you don't know anything about them yet.

Answer 5

I think the first question you need to answer is just how much of an issue this actually is. Is this only an edge case? Is the "optimal" solution only marginally better? It is hard to say much without more details about the actual use case and solutions. With some of the optimization/fitting problems I worked on for my PhD, you could dig around forever finding "more" optimal solutions, but depending on how noisy the data was, that didn't actually mean that the new solutions were "better". With a good understanding of your problem set and the underlying optimization algorithms, you should be able to figure out what exactly is going wrong with the closed source software, and whether or not there is an actual problem.

For the rest of this discussion I'm going to presume that you are trying to find the optimum fit to some sort of empirical data set. If so, you are effectively searching some sort of chi^2-parameter space, you finding a more optimum fit can mean very different things, e.g.

1. Your chi^2 space is noisy and a "better" solution doesn't actually mean anything
2. The proprietary system is getting stuck in a local minimum, and isn't finding the global minimum you found
3. You are both probing the same global minimum but your solution is just slightly closer

Effectively, the process of finding the optimum solution can be very tricky and there are a number of different algorithms that can be used, each with their own benefits and drawbacks. If you have a smoothly varying relationship between your fitting parameters and chi^2, just about any optimization method can find the minimum, but life isn't always that simple. All this to say that even if you have found a better optimum in one particular problem, this doesn't mean that the software is trash and should be thrown away. In particular, how did you find the better optimum? Can you apply that same method to these kinds of optimization problems in general? Will it always give a better fit? If the answer to either of those last two questions is no, I would say that there really isn't much of a big deal here at all. Most likely the only real problem is with an over-zealous marketer with the software company in question, who has promised things that no software system can realistically deliver.

In summary, I would say that before you really run with this you need to really understand in detail why their algorithm is not finding the most optimum solution, and if that even matters in practice.

Answer 6

I have seen well-known commercial optimisation packages such as CPLEX producing wrong result under a stress, so to speak---solving problems which are on paper solvable with the said software, but in practice sufficiently dissimilar from what they are tested on, and targeted at.

By wrong I mean as wrong as it gets, i.e. a linear optimisation problem is reported infeasible, while a request to produce an infeasibility certificate fails (and if solved with a higher precision on a slower open-source solver, the problem turns out to be feasible).

Surely sooner or later the error, if there was an error, will be found and exposed, this is how science works. Incidentally, in the past couple of years I published papers, which, among other things, correct errors in a monograph and in textbooks (sic!); the corresponding erroneous papers were published over 20 years ago. Oh well.