Predation acts as a selective pressure, driving prey adaptation. Predators can also evolve counter-defenses to increase the likelihood of successful attack. Investment in either trait can be costly, leading to a trade-off between the traits and other fitness components. Costs for defense have been shown experimentally to depend on environmental factors such as resource availability. This suggests that costs can increase with population size, rather than remaining constant as models often assume. Using a quantitative trait model with predator–prey coevolution, we investigate how both population and trait dynamics are affected by density-dependent prey defense cost (“variable cost”) vs. density-independent cost (“fixed cost”). We assume predator counter-defense cost is always density independent. We also investigate the effect of relative speeds of prey and predator evolution on population and trait dynamics, by varying a parameter that determines each population's additive genetic variance. For both models, increasing the speed of predator evolution always has a stabilizing effect on the dynamics, while increasing the speed of prey evolution (within a biologically reasonable range) is largely destabilizing. Within the plausible range of prey evolution speed, variable cost of prey defense is more stabilizing than fixed cost. Our results suggest that density-dependent costs of defense can have important effects on predator–prey dynamics, even when evolution is relatively slow. Our results might also help to explain why many real populations do not display predator–prey cycles.