I have an example demonstrating the solution produced by a popularly used software package is not the optimal solution, despite the software claiming that it is.
So the software may claim too much. This is not that uncommon.
So I am unsure how to report this bug; it is a proprietary, closed-source software. As a first step, I could contact the software developer about the bug, although they don't seem to have a straightforward method to report bugs.
Contacting the manufacturer in any suitable way, if no special way is indicated, would be the recommened procedure if you want to help the manufacturer (and thereby indirectly also yourself and others). Just make sure the bug does really exist and you explain it thoroughly so it can be reproduced.
But a larger issue is that papers published with results based upon a buggy software could be compromised.
Probably all software is somewhat buggy. Without knowing details it's difficult to judge the severeness of this case. However, it's always a good idea not to rely too much on one software package but test concurrent products as well. You could check these papers and see if they are affected by the bug and if so, how much they are affected.
What if the eventual fix causes the software to take much longer to execute, so that running on large instances is now infeasible?
A fixed software taking longer to solve a problem correctly is always preferred to a buggy, quick software. It would just mean that correct solutions to large instances were never feasible so far.
I have told my advisor about the issue, but he does not seem very interested, possibly because of the potential ramifications.
Maybe he doesn't care so much about using bug-free software or is not yet convinced of the existence of the bug or is just too lazy to bother the company or doesn't want any conflict.
I am currently working on a project where I could have used this software; I now don't trust it and have switched to a much less powerful open-source alternative.
It's natural to lose some trust but maybe not all. You could carefully evaluate the old software (keeping in mind that bugs occur occasionally) if it still can be trusted in the special circumstances of your project, then decide if you want to use it or if you want to use both software projects or if you want to use only the alternative software (which also could be buggy, so don't trust that one completely either).
So I guess my question is: how should I handle this situation?
- You should contact the company and write them a short message telling them about the error. They may quickly fix it, restoring trust.
- You might also publish the bug (on your blog, on a mailing list, even as a technical report if the bug is important enough and a journal is interested in that) so others are aware of it, especially if the company does not quickly fix it.
- You should use the two software packages that you have and compare them. Assume that both may have bugs you don't know anything about them yet.