Thermal expansion alters aluminium's dimensions that is for sure. I don't turn all day long or take large bits so my chuck and base plate don't really get hot, but just in case that does happen that is why I made the base plate so it needs some heat to even fit. After that it is just a spacer so aluminium should do fine as my machine does not spin real fast (max is 1600rpm).
I have never had a 4 jaw chuck so did not think about holding it with the 3 jaw, great suggestion. My 3 jaw and 4 jaw weigh a bit, combined that is a fare old weight being supported by the bearings without the job weight spinning force or tool pressure. Have you ever had yours fly off or just fall onto the bed? What size 3 jaw chuck do you have in comparison to your 4 jaw?
I have several lathes of different sizes and they all attach via cam lock so spinning off isn't a problem. You can turn your 3 jaws around to grip the outside or the 4 jaw. I some times turn for hours at a time.