Leydig cells are the steroidogenic lineage of the mammalian testis that produces testosterone, a key hormone required throughout male fetal and adult life for virilization and spermatogenesis. Both fetal and adult Leydig cells arise from a progenitor population in the testis interstitium but are thought to be lineage-independent of one another. Genetic evidence indicates that Notch signaling is required during fetal life to maintain a balance between differentiated Leydig cells and their progenitors, but the elusive progenitor cell type and ligands involved have not been identified. In this study, we show that the Notch pathway signals through the ligand JAG1 in perivascular interstitial cells during fetal life. In the early postnatal testis, we show that circulating levels of testosterone directly affect Notch signaling, implicating a feedback role for systemic circulating factors in the regulation of progenitor cells. Between Postnatal Days 3 and 21, as fetal Leydig cells disappear from the testis and are replaced by adult Leydig cells, the perivascular population of interstitial cells active for Notch signaling declines, consistent with distinct regulation of adult Leydig progenitors.