Have you got a flat neck to start with? It sounds like you have a lot of relief in it unless I'm misinterpreting you.
Basically, you have two shapes to consider here. There's the section between bridge and the neck fixings furthest away from it (the neck bolts closest to the unheadstock), and the very long thin triangle between those neck fixings and the nut. The height of the bridge at its lowest should correspond with the height of the neck and fretboard (so the strings would rest flat on the fretboard if the bridge height was minimised). The long thin triangle needs to be angled such that the strings would lie flat on the fretboard if you hold them down at the first fret.
If you need to work out the thickness of a shim, you can do that by considering the positions of the shim S and the fixings about which the neck will effectively pivot P (the bolts furthest from the bridge if you're putting a shim in at the bridge end to lower the nut end, the bolts closest to the bridge if you're putting a shim in at the other end to raise the nut). Then given the nut position N, the ratio NP:PS is also the ratio of thickness of the shim to the height by which the nut is to move (similar triangles). I think, anyway, my own pure maths O-levels were over 50 years ago.