Abstract: High efficiency video coding is the latest video coding standard. It adopts a quad-tree structure and 35 direction modes to improve coding performance, however, is at the expense of high coding complexity. To speed up the encoder, a fast CU depth decision method was proposed in this paper. Firstly, a spatial correlation between the current CU and its neighboring CUs was utilized to predict the depth for the current CU, accordingly a quick decision could be made for the division mode of CU. And then, the direction mode decision algorithm was implemented by a rate-distortion optimization process, skipping some unimportant direction modes. Test results show that the proposed algorithm can achieve 53% coding time saving on average, and can keep a better coding quality, while BDBR is only 1.17%.