Aromatase-expressing neuroendocrine neurons in the vertebrate male brain synthesize estradiol from circulating testosterone. This locally produced estradiol controls neural circuits underlying courtship vocalization, mating, aggression, and territory marking in male mice. How aromatase-expressing neuronal populations control these diverse estrogen-dependent male behaviors is poorly understood, and the function, if any, of aromatase-expressing neurons in females is unclear. Using targeted genetic approaches, we show that aromatase-expressing neurons within the male posterodorsal medial amygdala (MeApd) regulate components of aggression, but not other estrogen-dependent male-typical behaviors. Remarkably, aromatase-expressing MeApd neurons in females are specifically required for components of maternal aggression, which we show is distinct from intermale aggression in pattern and execution. Thus, aromatase-expressing MeApd neurons control distinct forms of aggression in the two sexes. Moreover, our findings indicate that complex social behaviors are separable in a modular manner at the level of genetically identified neuronal populations.