Ph.D. Thesis

Title: On the Identification and Detection of Hidden Malware Behaviors


The malware analysis process is one of the most difficult, complex, tedious, and time-consuming steps of the malware detection process. The governments have supported and funded for identifying the malware automatically. Several automated techniques have been developed. Despite lots of supports and funds, the number of cyber-attacks is increasing. Hence, signature-based malware detection techniques have non-solved problems to clarify polymorphism and metamorphism malware, researchers and antimalware companies have used malicious behavioral patterns. However, it does not solve the problem of a daily increase in the number of cyber-attacks. We clarified that the identification of non-correct and non-accurate behavior is the main problem of behavioral analysis. Every behavior is the reflex of an event. Hidden behaviors observe when the conditions that trigger the behavior be provided. There is a question that what the conditions are. A behavior can be modeled by a graph that has vertexes and edges. Vertexes describe system calls and edges illustrate the relation of system calls. Another problem is the merge of graphs in different conditions that malware is analyzed. The point is a set of truth behavior can throw a malicious behavior. A problem is the fusion of behaviors. Therefore, conducting a layered architecture of benign behavior that describes malicious behavior can be helpful. Consequently, a layered graph-based model for malware can be generated that involves environmental conditions. These conditions can be applied to the malware in an isolated or limited sandbox. Another problem is generating a non-detectable sandbox.


Malware Detection, Dynamic program analysis, Behavior analysis, Behavioral patterns, Malicious behaviors.


Detection of malware through behavioral signatures is more successful than detection based on structural signatures. However, a major problem with behavior-based identification methods is that malicious behavior is often detected after malignant surgery. In practice, they have not been able to accurately predict malicious behaviors. Context-aware malware mainly conceals its behavior and changes in response to environmental conditions.

So here is the main question:

How can the hidden behaviors of context-aware malware be detected on time?

To answer this question we have classified the behavior of malicious activities published in paper [J1]. In this paper, we determined the main requirements of malware to show their malicious activities. We have proposed VECG to identify and supply the malware requirements. You can read the paper for the details.