The high-mannose patch on the HIV-1 envelope (Env) glycoprotein is the epicenter for binding of the potent broadly neutralizing PGT121 family of antibodies, but strategies for generating such antibodies by vaccination have not been defined. We generated structures of inferred antibody intermediates by X-ray crystallography and electron microscopy to elucidate the molecular events that occurred during evolution of this family. Binding analyses revealed that affinity maturation was primarily focused on avoiding, accommodating, or binding the N137 glycan. The overall antibody approach angle to Env was defined very early in the maturation process, yet some variation evolved in the PGT121 family branches that led to differences in glycan specificities in their respective epitopes. Furthermore, we determined a crystal structure of the recombinant BG505 SOSIP.664 HIV-1 trimer with a PGT121 family member at 3.0 Å that, in concert with these antibody intermediate structures, provides insights to advance design of HIV vaccine candidates.