The influences of prenatal and postnatal growth on intellectual performance are unclear. We examined the associations of birth size and gestational age with intellectual performance and explored whether these associations were influenced by adult body size and social factors. In this nationwide cohort study, the records of 317,761 male infants registered in the Medical Birth Registry of Norway (1967-1979) were linked to the Norwegian Conscript Service (1984-1999). The variation in intelligence test score at age 18 due to birth weight and birth length was evaluated using absolute and standardized (z scores) values. Mean intelligence score increased by gestational age, birth weight, and birth length. However, a decline in intellectual performance was observed for gestational age >41 wk and birth weight >4500 g. There was a strong interaction on intellectual performance between birth size and gestational age (p < 0.0005). Adjusting for adult size strongly attenuated the association of birth size with intellectual performance. The overall R of intellectual performance explained by birth size was <1%; however, adding adult body size and social factors to the model increased R to 12%. In conclusion, the association of birth size with intellectual performance was weak, but still present after adjustment for adult body size and social factors.