Phthalonitrile‐based composites containing sodium montmorillonite (MMT0) and modified montmorillonites (MMT1 and MMT2) were prepared, respectively. The effect of functionalization, content and ball milling of the clay on the morphology, thermal mechanical, and thermal properties of the composites was thoroughly investigated by such methods as small angle X‐ray scattering, scanning electron microscopy, and dynamic, mechanical, and thermogravimetric analyses. FTIR and dynamic rheology measurements confirmed that the polymerization reaction of the PAEK‐CN prepolymer between individual clay layers had occurred and that intercalated tactoids and exfoliated/delaminated structures might have formed during the curing process. Small angle X‐ray scattering patterns reflected the increase of d‐spacing after clay modification and the subsequent possible exfoliation of the clay platelets within the phthalonitrile composites. Elevated heat distortion temperature and improved dynamic mechanical properties were observed for the PAEK‐CN/clay nanocomposites containing milled clay. Ball milling pristine clay improved the composite properties resulting from homogeneous dispersion to a certain extent, similar to the modified clay. In addition, composite properties depended on the content and modifier structures of the organoclays. POLYM. COMPOS., 37:3003–3014, 2016. © 2015 Society of Plastics Engineers