International policy is focused on increasing the proportion of the Earth’s surface that is protected for nature^1,2. Although studies show that protected areas prevent habitat loss^3–6, there is a lack of evidence for their effect on species’ populations: existing studies are at local scale or use simple designs that lack appropriate controls^7–13. Here we explore how 1,506 protected areas have affected the trajectories of 27,055 waterbird populations across the globe using a robust before–after control–intervention study design, which compares protected and unprotected populations in the years before and after protection. We show that the simpler study designs typically used to assess protected area effectiveness (before–after or control–intervention) incorrectly estimate effects for 37–50% of populations—for instance misclassifying positively impacted populations as negatively impacted, and vice versa. Using our robust study design, we find that protected areas have a mixed impact on waterbirds, with a strong signal that areas managed for waterbirds or their habitat are more likely to benefit populations, and a weak signal that larger areas are more beneficial than smaller ones. Calls to conserve 30% of the Earth’s surface by 2030 are gathering pace¹⁴, but we show that protection alone does not guarantee good biodiversity outcomes. As countries gather to agree the new Global Biodiversity Framework, targets must focus on creating and supporting well-managed protected and conserved areas that measurably benefit populations.