The dynamics of exciton spin relaxation in CdSe nanorods of various sizes and shapes are measured by an ultrafast transient polarization grating technique. The measurement of the third-order transient grating (3-TG) signal utilizing linear cross-polarized pump pulses enables us to monitor the history of spin relaxation among the bright exciton states with a total angular momentum of F = +/-1. From the measured exciton spin relaxation dynamics, it is found that the effective mechanism of exciton spin relaxation is sensitive to the size of the nanorod. Most of the measured cross-polarized 3-TG signals show single-exponential spin relaxation dynamics, while biexponential spin relaxation dynamics are observed in the nanorod of the largest diameter. This analysis suggests that a direct exciton spin flip process between the bright exciton states with F = +/-1 is the dominant spin relaxation mechanism in small nanocrystals, and an indirect spin flip via the dark states with F = +/-2 contributes as the size of the nanocrystal increases. This idea is examined by simulations of 3-TG signals with a kinetic model for exciton spin relaxation considering the states in the exciton fine structure. Also, it is revealed that the rate of exciton spin relaxation has a strong correlation with the diameter, d, of the nanorod, scaled by the power law of 1/d4, rather than other shape parameters such as length, volume, or aspect ratio.