Human action understanding and analysis for various applications are still in infancy due to various factors. In this paper, for recognizing various complex activities, a combined cue for motion representation and later recognition is demonstrated based on the optical flow-based four directional motion history and basic energy images. Optical flow between consecutive frames are computed to create the update function and to segment the moving regions. These motion vectors are split into four different channels. From these channels, the corresponding four directional history templates are computed. These along with frame-subtracted energy motion templates represent the final motion information of an action sequence. From these templates, feature vectors are calculated according to the seven Hu invariants. We develop a 35-dimensional feature vector for each action. For classification, k-nearest neighbor classification scheme is employed. For partitioning scheme, we employ leave-one-out cross-validation method. Both indoor and outdoor dataset provide satisfactory recognition results. These analysis, representation can be used for robot vision, interactive systems, computer games, behavior understanding, etc.