Answer:
Your model is unstable because the Compression member type is assigned to the columns. In RFEM, a compression member means that the element is only subjected to compression, and has hinges at the ends, so it does not transfer any moments. The hinges cause the instability. These elements are used in steel structures, for example. For this reason, we recommend using the Beam member type that can transfer all types of loads and is generally used for columns.