Systemic signals provided by nutrients and hormones are known to coordinate the growth and proliferation of different organs during development. However, within the brain, it is unclear how these signals influence neural progenitor divisions and neuronal diversity. Here, in the Drosophila visual system, we identify two developmental phases with different sensitivities to dietary nutrients. During early larval stages, nutrients regulate the size of the neural progenitor pool via insulin/PI3K/TOR-dependent symmetric neuroepithelial divisions. During late larval stages, neural proliferation becomes insensitive to dietary nutrients, and the steroid hormone ecdysone acts on Delta/Notch signaling to promote the switch from symmetric mitoses to asymmetric neurogenic divisions. This mechanism accounts for why sustained undernourishment during visual system development restricts neuronal numbers while protecting neuronal diversity. These studies reveal an adaptive mechanism that helps to retain a functional visual system over a range of different brain sizes in the face of suboptimal nutrition.