Affine and cyclotomic Schur categories