Logic latency for VI

I am wondering if I am expecting logic to do the impossible?

I often try to do soft synth parts near the end of projects, so have lots of plugs running , may have raised buffer etc..

I tend to use higher sample rates 96k.

So, I Experience too long latency and find levels hard to get right, mix too loud.

I often do a stereo bounce and import into new session do midi parts, with negligible latency snd export to main session for mixing: I know assbackwards.

Was surprised over weekend to experience all this with my new m1 mini which is supposed to have way more processing grunt ?

I know about low latency mode, input monitoring, reducing buffers, sampling rate, shutting off plugs, seems like a lot of bother?

Am using a new apollo X6.

Thought/ suggestions ?

THX!