Vulnerability Mining of Deep Learning Framework for Model Generation Guided by Reinforcement Learning

In the existing methods, the vulnerability mining is randomly generating the structural information of the model according to application model, generating easily a large number of low-quality test cases, and seriously affecting the efficiency and effect of vulnerability mining. To solve this problem, a vulnerability mining method of deep learning framework was proposed based on a guiding model generation method with reinforcement learning. Firstly, frame state information during model running was extracted, including Softmax distance and program execution results, etc. Then the extracted frame running state information was taken as a reward variable to guide the generation of model structure and hyper-parameters, so as to improve the quality and efficiency of test case generation. Experimental results show that this method can find more vulnerability of deep learning frameworks under the same number of generated test cases, possessing high practical value.