This is a style question, so there is not going to be a uniform answer that everyone agrees on. Some people use (and some journals seem to prefer) nested parentheses just like in your examples. I personally find this both esthetically unappealing and often hard to parse.
My practice (and this seems to be followed by a fair number of journals in the mathematical sciences) is to use delimiters in running text the same way that I would in mathematical expressions. For a expression like the one given in the question,* that would mean
The space O(X) [O(Y)] has functions defined on it as g(x) – f(x) [g(x) – f(x)]....
I find that more readable than text that involves the same delimiter repeated (even if the text and math parentheses are in different fonts). Moreover, this continues with further delimiters in the pattern
{ [ ( { [ ( ) ] } ) ] }
so that, for example, if a parenthetical includes a mathematical expression with square brackets, it would be marked off with French brackets, e.g.
This is the largest the function can grow {assuming, as before, that Δ[sin(x)/x] remains within the unitarity domain}, so we can conclude...
*Actually, this expression, even with the square brackets has the potential to be confusing. As a general rule, in any situation where you are using multiple delimiters like this, it is good to look to see whether you can tweak what you have written to make it more readable. For example, I think your statement would be still easier to follow with the following added clarifications:
The space O(X) [respectively, O(Y)] has functions defined on it as g(x) – f(x) [respectively, g(x) – f(x)]....
(Maybe the second "respectively" is superfluous, but it probably doesn't hurt.)