If we review this Apple T/N Install memory in an iMac
We’ll find this statement “If you have mixed capacity DIMMs, see the install memory section for installation recommendations.” And we jump down to that section: “iMac (Retina 5K, 27-inch, 2020) DIMMs have a notch on the bottom, slightly left of the middle. If your DIMMs are mixed in capacity, minimize the capacity difference between Channel A (slots 1 and 2) and Channel B (slots 3 and 4) when possible.”
So you really can’t mix within the channel only across channels. So you’ll need to install your different RAM sizes in Slot 1 and Slot 3. Don’t forget to allow the system to initialize which can take awhile!
Given the cost and the performance benefit I would just go with a matched pair within each channel pair.