Abstract: In array signal processing,mutual coupling between sensors has an adverse effect on the estimation of parameters. A new direction of arrival (DOA) estimation algorithm for unknown mutual coupling model was proposed to reduce the effect of mutual coupling. Utilizing the idea of iteration,the algorithm was arranged firstly to preliminary estimate DOA in the whole array. Then the angularly dependent coefficients (ADCs) were estimated based on the well-known subspace theory to further improve the accuracy of DOA estimation. The mutual coupling coefficients were finally determined by solving the least squares problem,and starting the next iteration with the new DOA as the preliminary value. Simulation results show that the proposed method can achieve better performance and more robust than other DOA estimation algorithm for unknown mutual coupling model,in particular at a low signal-to-noise ratio (SNR) or a small-sized array.