Evaluation

We evaluated CCMOP for 100 real-world C/C++ programs. We collected the compilation time of each project for different properties. For runtime overhead, we evaluated CCMOP on the programs using Use-afer-free property. All the experiments were carried out on a laptop with a 2.60GHz CPU and 32G memory, and the operating system is Ubuntu 20.04. The result values are the averaged values of three runs within -O3.

1. Benchmark

The 50 real-world C++ programs are as follows.

The 50 real-world C programs and the programs in movec-minibench are as follows.

The properties used in the evaluation are as follws.

There are two properties, i.e., use-after-free and memory-leak, are used for both C and C++ programs. However, the event definations of them are different. We weave monitors when calling malloc and free in C programs, and we weave monitors to the new and delete statements in C++ programs.

2. Evaluation results

The evaluation results contain the compilation time for 100 real-world C/C++ programs and the runtime overhead only for C/C++ programs with valid test drivers.

(1).Compilation Time

We evaluate each C/C++ program for three properties. For C programs, the properties are Use-after-free, Memory-leak, and Read-after-close. For C++ programs, the properties are Use-after-free, Memory-leak, and Safe-Iterator. In addition, the common properties have different pointcuts for different languages. The results of compilation time are shown as follows.

The columns labeled Average-O, Average-U, Average-M, and Average-R/U represent the average of three test values of Original compilation, RV for Use-after-free, RV for Memory-leak and RV for Read-afer-close or Safe-Iterator respectively. The rows labeled Overhead-* means the average compilation time with RV (denoted C_RV) overhead compared with the average original compilation time (denoted C_O); that is (C_RV-C_O)/C_O. In addition, the former 50 programs are C programs, and the latter 50 are C++ programs.

The following figure shows the above data, where the X-axis shows the project names, and the Y-axis shows the value of compilation overhead.

The above figure or table shows that the compilation overhead of the majority (78%) of the C/C++ programs is below 300%. There are some programs whose compilation overhead exceeds 500%, with most of the overall compilation time occupied by the monitor library building stage. You can download the original data of the three runs from the following link.
Download

(2).Runtime Overhead

We evaluate the runtime overhead for the Use-after-free property. We did experiments on AddressSanitizer within -O3 for comparison. We use the below compilation options of AddressSanitizer; We expect these options to suppress other checkers.

 -O3 -fsanitize=address  -fsanitize-recover=address  -mllvm -asan-stack=0 -mllvm -asan-globals=0 -fno-sanitize-address-use-after-scope 
export ASAN_OPTIONS=detect_container_overflow=0:detect_leaks=0:leak_check_at_exit=0:check_printf=0:detect_container_overflow=0:buffer_overflow=0:report_globals=0:detect_stack_use_after_return=0:new_delete_type_mismatch=0:alloc_dealloc_mismatch=0:poison_heap=0:poison_partial=0:protect_shadow_gap=0:check_malloc_usable_size=0:allow_user_poisoning=0:replace_intrin=0:replace_str=0

Out of the 100 real-world programs, those without drivers or a runtime overhead of AddressSanitizer or CCMOP lower than 5% are not shown. In addition, the programs on which Sanitizer fails have not been listed. The evaluation results of valid C++ programs are shown below.

We draw a figure to represent the above data as follows. The X-axis is the project name, and the Y-axis is log₁₀^{Overhead-RV+1}.

If you are interested in the data from the three test runs, you can download it by clicking the following link.
Download

The evaluation results of the C programs are shown below.

Some programs listed above frequently involve memory operations and have a higher overhead for CCMOP, which increases the average value. Besides, the figure is as follows.

If you are interested in the data from the three test runs, you can download it by clicking the following link.
Download

3.Evaluation reproduction

We provide the scripts for reproducing our evaluation. Before running the following command, you need to confirm that wac, CCMOP/bin in your PATH and export the <CCMOP/lib> to ASPECT_LIB. For the evaluation of the compilation test, you can run the following command:

python3 test_compileTime.py

If the above command succeeds, you can see compilelist.csv and compilelist.pdf in your current directory. For the evaluation of the runtime overhead test, you can run the following command:

python3 test_runtimeOverhead.py

If the above command succeeds, you could see c_runtime.csv,cxx_runtime.csc,c_runtime.pdf and cxx_runtime.pdf in your current directory. In addition, the runtime PDF is a figure drawn with the data in the CSV file and filters out some invalid data.

Contacts

Please feel free to contact us if you have any questions about CCMOP.

• Yongchao Xing (xingyc0979@nudt.edu.cn)
• Zhenbang Chen (zbchen@nudt.edu.cn)