Exposure histories to SARS-CoV-2 variants and vaccinations will shape the specificity of antibody responses. To understand the specificity of Delta-elicited antibody immunity, we characterize the polyclonal antibody response elicited by primary or mRNA vaccine-breakthrough Delta infections. Both types of infection elicit a neutralizing antibody response focused heavily on the receptor-binding domain (RBD). We use deep mutational scanning to show that mutations to the RBD’s class 1 and class 2 epitopes, including sites 417, 478, and 484–486 often reduce binding of these Delta-elicited antibodies. The anti-Delta antibody response is more similar to that elicited by early 2020 viruses than the Beta variant, with mutations to the class 1 and 2, but not class 3 epitopes, having the largest effects on polyclonal antibody binding. In addition, mutations to the class 1 epitope (e.g., K417N) tend to have larger effects on antibody binding and neutralization in the Delta spike than in the D614G spike, both for vaccine- and Delta-infection-elicited antibodies. These results help elucidate how the antigenic impacts of SARS-CoV-2 mutations depend on exposure history.