Apolipoprotein E4 (APOE4) is a genetic risk factor for developing Alzheimer's disease (AD). Once AD manifests clinically, however, the effects of APOE4 are less clear. Therefore, we investigated the longitudinal effects of APOE4 on topographical changes in AD patient brain atrophy. We prospectively recruited 35 patients with AD (19 APOE4 carriers and 16 non-carriers), and 14 normal controls, then followed them for five years. We measured hippocampal deformities and cortical thickness. Hippocampal comparison between APOE4 carriers and non-carriers with AD showed carriers had rapid changes in the head and body, while non-carriers had rapid changes in a small portion of the body. Cortical thickness comparison between APOE4 carriers and non-carriers with AD dementia showed carriers had rapid thinning in the lateral frontal, temporal, and parietal regions, while no region showed more rapid cortical thinning in non-carriers than in carriers. These findings underlined the importance of the APOE4 allele for designing and interpreting future treatment trials in patients with AD dementia.