Understanding affinity maturation of antibodies that can target many variants of HIV-1 is important for vaccine development. While the antigen-binding site of antibodies is known to mutate throughout the co-evolution of antibodies and viruses in infected individuals, the roles of the mutations in the antibody framework region are not well understood. Throughout affinity maturation, the CH103 broadly neutralizing antibody lineage, from an individual designated CH505, altered the orientation of one of its antibody variable domains. The change in orientation was a response to insertions in the variable loop 5 (V5) of the HIV envelope. In this study, we generated CH103 lineage antibody variants in which residues in the variable domain interface were mutated, and measured the binding to both autologous and heterologous HIV-1 envelopes. Our data show that very few mutations in an early intermediate antibody of the lineage can improve binding toward both autologous and heterologous HIV-1 envelopes. We also crystallized an antibody mutant to show that framework mutations alone can result in a shift in relative orientations of the variable domains. Taken together, our results demonstrate the functional importance of residues located outside the antigen-binding site in affinity maturation.