Understanding how temperature determines the distribution of life is necessary to assess species’ sensitivities to contemporary climate change. Here, we test the importance of temperature in limiting the geographic ranges of ectotherms by comparing the temperatures and areas that species occupy to the temperatures and areas species could potentially occupy on the basis of their physiological thermal tolerances. We find that marine species across all latitudes and terrestrial species from the tropics occupy temperatures that closely match their thermal tolerances. However, terrestrial species from temperate and polar latitudes are absent from warm, thermally tolerable areas that they could potentially occupy beyond their equatorward range limits, indicating that extreme temperature is often not the factor limiting their distributions at lower latitudes. This matches predictions from the hypothesis that adaptation to cold environments that facilitates survival in temperate and polar regions is associated with a performance trade-off that reduces species’ abilities to contend in the tropics, possibly due to biotic exclusion. Our findings predict more direct responses to climate warming of marine ranges and cool range edges of terrestrial species.