In this research, we present an AR framework that allows natural hand and tangible AR interaction for physically-based interaction and environmental awareness to support face-to-face collaboration using Microsoft Kinect. Our framework comprises six major components: (1) marker tracking (2) depth acquisition (3) image processing (4) physics simulation (5) communication and (6) rendering. The resulting augmented environment supports occlusion, shadows, and physically-based interaction of real and virtual objects. We propose three methods of natural hand representations including mesh-based, direct sphere substitution and variational optical flow.