Answer:
The effective length factors that you can change in the "Effective Lengths" tab are not considered in the determination of internal forces in RFEM 5 or RSTAB 8. These values are presettings for add-on modules used to perform stability analyses; for example, RF‑/STEEL EC3 or RF‑/TIMBER Pro. The values will be considered in the modules only.
However, the "Check exceeding of critical buckling load" option has an influence on the calculation: If the critical load is reached, the member fails. In this case, the program shows a message about the instability.