Abstract: To solve the problem of a free-hand interaction method in 3D scene modeling, some possible solutions were put forward from three levels for the key problems of scene modeling. On the macroscopic level, an interaction frame was established based on user activities and system tasks. On the middle level, a method was proposed to construct a complete and practical gesture set, and then a gesture set was designed for the system based on the proposed method. On the microcosmic level, to solve the "Midas touch" problem, an optimal method was put forward from the aspects of time, space, and gesture assistance to improve the recognition rate of gestures. Finally, experiment was carried out to verify the availability of proposed methods, providing a future research direction.