The origin of the Moon’s large-scale topography is important for understanding lunar geology1, lunar orbital evolution2 and the Moon’s orientation in the sky3. Previous hypotheses for its origin have included late accretion events4, large impacts5, tidal effects6 and convection processes7. However, testing these hypotheses and quantifying the Moon’s topography is complicated by the large basins that have formed since the crust crystallized. Here we estimate the large-scale lunar topography and gravity spherical harmonics outside these basins and show that the bulk of the spherical harmonic degree-2 topography is consistent with a crust-building process controlled by early tidal heating throughout the Moon. The remainder of the degree-2 topography is consistent with a frozen tidal–rotational bulge that formed later, at a semi-major axis of about 32 Earth radii. The probability of the degree-2 shape having both tidal-heating and frozen shape characteristics by chance is less than 1%. We also infer that internal density contrasts eventually reoriented the Moon’s polar axis by 36 ± 4°, to the configuration we observe today. Together, these results link the geology of the near and far sides, and resolve long-standing questions about the Moon’s large-scale shape, gravity and history of polar wander.