Is this allowed in academic writing or do I have to formulate it differently?
There is no Central Committee for Academic Writing issuing decisions on which things are Allowed and which are Not Allowed, so it does not make a lot of sense to formulate the question this way. There are just people with various opinions on what constitutes good writing, with near consensus on some things and violent disagreement on others. So it is really up to you to figure out which principles you want to follow when it comes to parentheses etc.
Accordingly, the question is not can I do this? (yes, you can), it is should I do this? As you yourself have noticed, two consecutive parentheses look a bit awkward, so if there is an obvious way to avoid them, I would say it is probably for the best to do so. For example, you can instead write
This leads to a function f(x), which then ...
This leads to a function f(x). The function f(x) then ...
This leads to a function, which we will call f(x). The function f(x) then ...
This leads to a function, f(x), which then ...
This leads to a function – f(x) – which then ...
or any of a number of other possibilities. If you want to write mathematics well, I recommend that you invest some time into thinking about various different ways you can formulate sentences like this. After some practice your brain will automatically start coming up with a number of alternative ways of phrasing any given sentence, which will pay off handsomely when it comes to avoiding issues of this sort.