Limits on weak magnetic confinement of neutral atoms