> it is impossible to minimize three different functions over the same parameters simultaneously.

There is a topic called "multi-objective" optimization. Once I took a course in it from a guy, later President at CMU, who was one of my Ph.D. dissertation advisors.

There is a geometric concept, the non-inferior set where it is impossible to do better on all the functions at once.

There is a term, a condition, a situation called Pareto optimality based on the non-inferior set.

Sometimes finding a non-inferior set is a tweak of ordinary sorting.

I used some of that once: The main costs in flying an airplane are due to (i) fuel and (ii) maintenance based on flight time (in some cases also landings). So, buried deep in the math and code, I was able to make the code run faster by keeping just the non-inferior set -- it was easy to prove that this was sufficient, and I found that set via a tweak of sorting. E.g., in part I was looking for flight plans, so I considered only ones non-inferior on both fuel cost and flight time cost.

Also the OP is talking about false positives and false negatives. Commonly it is relatively easy to be able to adjust, set, and know the rate of false positives. Then the challenge is, for that rate of false positives, find the lowest rate of false negatives. Sure, for a given detector, there may be a cost for a false positive and also a cost for a false negative. Then we would want, say, to pick the false positive to minimize the sum of the two costs.

E.g., just now some travelers may be put into quarantine for two weeks because a false positive where we would conclude they are sick when they are well and a false negative would be that we conclude that they are well when they are sick. So, the cost of the false positive would be the quarantine but the cost of the false negative would be the risk of a quarantine of a city or a pandemic for the world.

I got into this material due to a paper I wrote on anomaly detection for server farms and networks: There maybe a false positive has the system management staff waste some time/effort while a false negative might risk a security breach that might bankrupt the company.

Lesson: The costs of (i) false positives and (ii) false negatives might be very different.

Given a detection problem and a rate of false positives, how do we design the detector that will give the lowest rate of false negatives? There is an answer in the classic Neyman-Pearson theorem. Once I worked out a proof based on the Hahn decomposition based on the Radon-Nikodym theorem in measure theory. The problem can be a knapsack problem in optimization, known to be in NP-complete.

In optimization we have to keep in mind that commonly finding a feasible solution can be about as difficult as finding an optimal solution -- in particular, just finding a feasible solution can be very difficult.

For optimization, it is true that even if the feasible region is discrete -- e.g., with some positive integer n, the set of real numbers R, and the set of all points, vectors, in R^n where each vector has components only 0 or 1 -- commonly we can set up a dual problem that is continuous, differentiable, and convex and that will help finding an optimal solution or at least a good approximation to one. E.g., once I had a 0-1 integer linear programming problem with 40,000 constraints and 600,000 0-1 variables and using this duality approach found a feasible solution with objective function within 0.025% of optimality -- this duality method can provide such bounds. So here I may be helping with the remark in the OP about "non-differentiable".