Decline in working memory (WM) functions during aging has been associated with hippocampal dysfunction mediated by age-related changes to the corticotropin-releasing hormone (CRH) system. Recent reports suggest that GG-homozygous individuals of single nucleotide polymorphisms (rs110402 and rs242924) in the CRH receptor 1 (CRHR1) gene show increased stress vulnerability and decreased BOLD responses in WM relevant regions. However, until now, no study investigated the interaction effects of variation in the CRHR1 gene and age on individual differences in WM. Here, young, middle-aged and old subjects (N = 466) were genotyped for rs110402 and rs242924 within the CRHR1 gene and an n-back task was used to investigate the hypothesis that vulnerable genotypes (GG-homozygotes) would show impaired WM functions that might be magnified by increased CRH production with advancing age. Our results show an impact of genotype already in middle-age with significantly better performance in AT-carriers. Working memory performance in AT-carriers did not differ between young and middle-aged subjects, but was significantly impaired in old age. In GG-homozygotes, severe working memory dysfunction occurred already in middle age. Our data indicate that GG-homozygotes of CRHR1 rs110402 and rs242924 represent a genetically driven subtype of early WM impairments due to alterations in hippocampal CRHR1 activation. Early interventions that have proven effective in delaying cognitive decline appear to be particularly important for these subjects at risk for premature memory decline, who are in the prime of their personal and professional lives.