Abstract: Due to the great difference of the images from imaging apparatuses, the existing state-of-the-art algorithms are hard to obtain corresponding features for matching processing or the instance recognition on multi-sensor images (such as RGB and infrared images). To solve this problem, a new feature matching algorithm was proposed based on a self-labeling technique, being able deep learning and extracting feature. Firstly, an immature detector was designed and trained with synthetic images to be competent for feature extraction on different types of images. Then, a self-labeling method was proposed to obtain the corresponding feature points on the multi sensors images, and the self-labeling results were used for detector and descriptor training to achieving the instance recognition on multi-sensor images based on the matching feature points. Finally, hundreds of pairs RGB and Infrared images were collected from different scenarios and condition, and some experiments were carried out to compare the proposed algorithm with 6 different state-of-the-art algorithms. The experiment results show that the proposed algorithm can provide much and accurate corresponding feature points than other state-of-the-art algorithms, improving the average precision significantly.