Rotor position detection is a prerequisite for achieving good control performance of PMSM. For a PMSM control system based on an optical encoder, it is a difficulty to detect rotor position and achieve R-Signal zero-setting. To solve the problem, a hybrid optical encoder is used in the paper by which a scheme for rotor position detection and R-Signal zero-setting is proposed. This encoder can do absolute and incremental rotor position detection simultaneously; here, the former is used for acquiring imprecise rotor position and the latter is for precise rotor position. Firstly, two detection methods of the encoder are analyzed, and a scheme for rotor position detection is proposed: absolute rotor position is used for motor starting before achieving R-Signal zero-setting; once achieving R-Signal zero-setting, incremental rotor position detection that has high precision is adopted. Then a novel scheme for R-Signal zero-setting is emphatically proposed. Finally, the simulation is conducted. Results show that rotor position detection and R-Signal zero-setting can be achieved by the proposed scheme.