Band insulators appear in a crystalline system only when the filling—the number of electrons per unit cell and spin projection—is an integer. At fractional filling, an insulating phase that preserves all symmetries is a Mott insulator; that is, it is either gapless or, if gapped, exhibits fractionalized excitations and topological order. We raise the inverse question—at an integer filling is a band insulator always possible? Here we show that lattice symmetries may forbid a band insulator even at certain integer fillings, if the crystal is non-symmorphic—a property shared by most three-dimensional crystal structures. In these cases, one may infer the existence of topological order if the ground state is gapped and fully symmetric. This is demonstrated using a non-perturbative flux-threading argument, which has immediate applications to quantum spin systems and bosonic insulators in addition to electronic band structures in the absence of spin–orbit interactions.