Reminds me of the birth interval in hunter-gatherers vs. farming people in prehistory. Hunter-gatherers were supposedly constrained in their birth spacing by the need to carry their infants when roaming around, while farmers had no such constraint and were thus able to out-reproduce hunter-gatherers.
I would imagine that the amount of available calories is overall the larger influence on which society grows and which doesn't. Remember that in most hunter-gatherer and farming societies, population stayed constant despite high birth numbers.
This is kind of mentioned in the interview I posted in the sibling reply, but I could imagine that this mainly drives down the density, but not so much the rate of growth. The reason is that grown individuals can actually usually feed themselves by roaming. What is hard is achieving a caloric surplus, which is why there is little specialisation and social stratification.
