In the developing brain, neural progenitor cells switch differentiation competency by changing gene expression profiles that are governed partly by epigenetic control, such as histone modification, although the precise mechanism is unknown. Here we found that ESET (Setdb1), a histone H3 Lys9 (H3K9) methyltransferase, is highly expressed at early stages of mouse brain development but downregulated over time, and that ablation of ESET leads to decreased H3K9 trimethylation and the misregulation of genes, resulting in severe brain defects and early lethality. In the mutant brain, endogenous retrotransposons were derepressed and non-neural gene expression was activated. Furthermore, early neurogenesis was severely impaired, whereas astrocyte formation was enhanced. We conclude that there is an epigenetic role of ESET in the temporal and tissue-specific gene expression that results in proper control of brain development.