I guess this happens because you moved from the .append to pd.concat. In that case you have to rewrite the logic of the check_state_exist function and the choose_action functions.