Abstract characterizations of arithmetic groups with the congruence property