Preprocessor option combiner
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 3 of 3

Thread: Preprocessor option combiner

  1. #1
    Join Date
    Jan 2009
    Posts
    1,689

    Cool Preprocessor option combiner

    If you are like me, you occasionally have big projects that have tons of preprocessor in it (usually debugging stuff, library options...) and you have a unit test suite, but there is no way to sync the test suite up so that it automatically tries every combination of options that alter the code.

    Say you have this

    test.h
    Code:
    #ifdef OPTION1
    	#ifdef OPTION2 
    		#if (defined (OPTION3) || !defined(OPTION4))
    			//code
    		#endif
    	#elif !defined(OPTION5) && define(I_DONT_CARE_ABOUT_THIS_OPTION)
    		//code
    	#else
    		//code
    	#endif
    #endif
    With a few files it's easy to figure out all of the combinations that you need to test (and if you only have a few options you can brute force it and just do all possible combinations.) But if you have lots of files, and/or lots of options, you need something better.

    I scoured the internet looking for something to perform this task and came up empty handed. So I wrote my own, and it case anyone else is interested in this, I'm posting the script.

    TestAllOptions.sh
    Code:
    #!/usr/bin/php -f
    
    #########################################################################################
    #
    #	About:
    #		This script will iterate through a folder or list of folders and finds all of
    #		the preprocessor options, and creates all possible combinations used in the
    #		code.  This allows you to test every combination that alters the code automatically
    #	Features:
    #		Only looks for preprocessor you care about
    #		Able to go through multiple folders recursively
    #		Able to go through individual files
    #		Can output to both commandline arguments or to a set of files
    #		Able to not only define, but define to something
    #	Known Issues:
    #		Almost no comments / documentation
    #		Multiline preprocessor (with \s) are not supported
    #		Super nested definitions may not be recognized based on how it starts
    #
    #########################################################################################
    
    <?php
    	class config {
    		const OUTPUT_HEADER = true;
    		const OUTPUT_COMMANDLINE = false;
    	
    		static public function isDebugging(){
    			return false;
    		}
    		static public function getProductPath(){
    			return dirname(__FILE__) . '/test.h';
    		}
    		static public function getOutputPath(){
    			return dirname(__FILE__) . '/Output';
    		}
    		static public function getOutputStyle(){
    			return self::OUTPUT_COMMANDLINE;
    		}
    		static private function _getDefinitions(){
    			return array (
    				'OPTION1',
    				'OPTION2' => 15,
    				'OPTION3',
    				'OPTION4',
    				'OPTION5'
    			);
    		}	
    				
    		static public function getDefinitions(){
    			static $defs;
    			if (!isset($defs)){
    				$temp = self::_getDefinitions();
    				$defs = array();
    				$i = 1;
    				foreach($temp as $key => $item){
    					if (!is_string($key)){
    						$key = $item;
    						$item = '';
    					}
    					$defs[$i] = array($key, $item);
    					$i *= 2;
    				}
    			}
    			return $defs;
    		}
    	}
    
    
    	class funcs {
    		public static function fail($msg, $object = NULL, $exitter = true){
    			if (is_null($object)){
    				$msg = $msg . PHP_EOL;
    			} else {
    				$msg = $msg . PHP_EOL . $object -> getFileName() . ' : ' . $object -> getFileLine() . PHP_EOL;
    			}
    			
    			if ($exitter){
    				exit($msg);
    			} else {
    				echo($msg);
    			}	
    		}
    		
    		public static function println($msg){
    			if (config::isDebugging()){
    				echo $msg . PHP_EOL;
    			}
    		}
    	}
    
    
    	class fileFetcher {
    		private $_options = array();
    		private $_stack = array();
    		private $_filename;
    		private $_linenum;
    		static $_ignores = array();		
    
    		public function getFileName(){
    			return $this -> _filename;
    		}
    		public function getFileLine(){
    			return $this -> _linenum;
    		}
    	
    		public function __construct($filename){
    			$this -> _filename = $filename;
    		}
    		
    		public function findOptions(& $options){
    			$this -> _options = $options;
    			$cont = file_get_contents($this -> _filename);
    			if ($cont === false) funcs::fail($this -> _filename . ' not gettable' . PHP_EOL);
    			
    			$pos = 0;
    			$param = NULL;
    			while ($this -> _getExtractionParameter($cont, $pos, $param)){
    				$this -> _extractOption($cont, $pos, $param);
    			}
    			$options = $this -> _options;
    		}
    		
    		public function dump(){
    			if (!empty($this -> _stack)) funcs::fail('Something is still on the stack for ' . $this -> _filename . PHP_EOL);
    			echo $this -> _filename . PHP_EOL;
    			var_dump($this -> _options);
    		}
    		
    		private function _getExtractionParameter(& $cont, & $pos, & $param){
    			static $searchers = array (
    				'#ifdef',
    				'#ifndef',
    				'#if defined',
    				'#if (defined',
    				'#if(defined',
    				'#if !defined',
    				'#if !(defined',
    				'#if !(!defined',
    				'#if(!defined',
    				'#elif defined',
    				'#elif !defined',
    				'#else',
    				'#endif',
    				'#if'
    			);
    			
    			$first = false;
    			foreach($searchers as $searcher){
    				$ind = strpos($cont, $searcher, $pos);
    				if ($ind !== false){
    					if (($first === false) || ($ind < $first)){
    						$param = $searcher;
    						$first = $ind;
    					}
    				}
    			}
    			if ($pos = $first){
    				$this -> _linenum = substr_count(substr($cont, 0, $pos), PHP_EOL) + 1;
    			} else {
    				$this -> _linenum = 0;
    			}
    			return $pos !== false;
    		}
    		
    		private function _extractOption(& $cont, & $pos, $param){
    			$endl = $this -> _endOfLine($cont, $pos);
    			if ($endl === false) funcs::fail($this -> _filename . ' has an unterminated option, or is missing a newline' . PHP_EOL, $this);
    			$paramlen = strlen($param);
    			$orig = $option = trim(substr($cont, $pos + $paramlen, $endl - $pos - $paramlen));
    			$orig_count = count($this -> _stack);
    			$combo = '';
    			
    			switch($param){
    				case '#if(defined':
    				case '#if (defined':
    					$option = '(' . $option;
    				case '#if defined':
    					$option = str_replace('defined', '', $option);
    					$combo = $this -> _pushOnStack($option, false, '!(' . $option . ')');
    					break;
    				case '#ifdef':
    					$combo = $this -> _pushOnStack($option, false, '!(' . $option . ')');
    					break;
    				case '#if (!defined':
    				case '#if(!defined':
    					$option = str_replace('defined', '', $option);
    					$combo = $this -> _pushOnStack('(!' . $option, false, '!(' . $option . '))');
    					break;
    				case '#if !(!defined':
    					$option = str_replace('defined', '', $option);
    					$combo = $this -> _pushOnStack('!(!' . $option, false, '(' . $option . '))');
    					break;
    				case '#if !(defined':
    					$option = str_replace('defined', '', $option);
    					$combo = $this -> _pushOnStack('!(' . $option, false, '(' . $option);
    					break;
    				case '#if !defined':
    					$option = str_replace('defined', '', $option);
    					$combo = $this -> _pushOnStack('!' . $option, false, '!(!' . $option . ')');
    					break;
    				case '#ifndef':
    					$combo = $this -> _pushOnStack('!' . $option, false, '!(!' . $option . ')');
    					break;
    				case '#else':
    					$combo = $this -> _popOffStack('');
    					--$orig_count;
    					break;
    				case '#endif':
    					$combo = $this -> _popOffStack();
    					--$orig_count;
    					break;
    				case '#elif !defined':
    					$option = str_replace('defined', '', $option);
    					$combo = $this -> _popOffStack('!' . $option);
    					--$orig_count;
    					break;
    				case '#elif defined':
    					$option = str_replace('defined', '', $option);
    					$combo = $this -> _popOffStack($option);
    					--$orig_count;
    					break;
    				case '#if':
    					$combo = $this -> _pushOnStack('', false, '');
    					break;
    			}
    			
    			funcs::println(str_repeat('   ', $orig_count) . $param . ' ' . $option . ' -- Line:' . $this -> _linenum . ' -- ' . $combo);
    			
    			$pos = $endl;
    		}
    		
    		private function _endOfLine(& $cont, $pos){
    			static $searchers = array (
    				PHP_EOL,
    				'//',
    				'/*'
    			);
    	
    			$first = false;
    			foreach($searchers as $searcher){
    				$ind = strpos($cont, $searcher, $pos);
    				if ($ind !== false){
    					if (($first === false) || ($ind < $first)){
    						$first = $ind;
    						$found = $searcher;
    					}
    				}
    			}
    			
    			return $first;
    		}
    		
    		private function _pushOption(){
    			$option = '';
    			foreach($this -> _stack as $stackitem){
    				if (!empty($stackitem)){
    					$item = $stackitem[0];
    					if (empty($option)){
    						$option = $item;
    					} else if ((strpos($item, ' ') === false) && 
    							   (strpos($item, '|') === false) && 
    							   (strpos($item, '!') === false)){
    						$option .= ' && ' . $item;
    					} else {
    						$option .= ' && (' . $item . ')';
    					}
    				}
    			}
    			
    			if (empty($option)) return;
    			$optionvaltxt = $this -> _cleanOption($option);
    			if (empty($optionvaltxt)) return;
    			$code = ' return ' . $optionvaltxt . ';';
    			$optionval = eval($code);
    			if ($optionval == 0){
    				funcs::fail('something went wrong: ' . $option . ' : ' . $code, $this, false);
    				return;
    			}
    			if (!isset($this -> _options[$optionval])){
    				$this -> _options[$optionval] = array(
    					'willbeused' => $this -> _getSimplifiedOptions($optionval),
    					'locations' => array(
    						array (
    							'file' => $this -> _filename,
    							'line' => $this -> _linenum,
    							'options' => $option
    						)
    					)
    				);
    			} else {
    				$this -> _options[$optionval]['locations'][] = array (
    					'file' => $this -> _filename,
    					'line' => $this -> _linenum,
    					'options' => $option
    				);
    			}
    			return $optionvaltxt;
    		}
    		
    		private function _getSimplifiedOptions($optionval){
    			$defs = config::getDefinitions();
    			$option = '';
    			$optioncode = '';
    			foreach($defs as $key => $opt){
    				if ($optionval & $key){
    					if (!empty($opt[1])){
    						$option .= ' -D' . $opt[0] . '=' . $opt[1];
    					} else {
    						$option .= ' -D' . $opt[0];
    					}
    					$optioncode .= '#define ' . $opt[0] . ' ' . $opt[1] . PHP_EOL;
    				}
    			}
    			$optioncode .= PHP_EOL . '#endif' . PHP_EOL;
    			return array ($option, $optioncode);
    		}
    		
    		private function _cleanOption($option){
    			$options = config::getDefinitions();
    			//convert names of options to their numeric value
    			foreach($options as $keyvalue => $value){
    				$option = str_replace($value[0], $keyvalue, $option);
    				$option = str_replace('&&', '&', $option);
    				$option = str_replace('||', '|', $option);
    				$option = str_replace('!', '~', $option);
    				$option = str_replace(' ', '', $option);
    			}
    			$this -> _removeIgnoredOptions($option);
    			return $option;			
    		}
    		
    		private function _removeIgnoredOptions(& $option){
    			static $chars;
    			if (!isset($chars)){
    				$chars = str_split('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_');
    			}
    			
    			//There should currently be no text in the option, only numeric values
    			while (1){
    				$first_instance = 0xFFFFFFF;
    				foreach($chars as $char){
    					if (($pos = strpos($option, $char)) !== false){
    						if ($pos < $first_instance) $first_instance	= $pos;
    					}
    				}
    				if ($first_instance === 0xFFFFFFF) goto finish_clean;
    				
    				
    				$lparen = strpos($option, '(', $first_instance);
    				$rparen = strpos($option, ')', $first_instance);
    				$and = strpos($option, '&', $first_instance);
    				$or = strpos($option, '|', $first_instance);
    				
    				if ($lparen === false) $lparen = 0xFFFFFFF;
    				if ($rparen === false) $rparen = 0xFFFFFFF;
    				if ($or === false) $or = 0xFFFFFFF;
    				if ($and === false) $and = 0xFFFFFFF;
    				
    				if (($rparen == $lparen) && ($lparen == $or) && ($or == $and) && ($and == 0xFFFFFFF)){
    					$remove = substr($option, $first_instance);
    				} else {
    					$remove = substr($option, $first_instance, min($rparen, $lparen, $or, $and) - $first_instance);
    				}
    				if (!isset(self::$_ignores[$remove])){
    					self::$_ignores[$remove] = $remove . ' -- ' . $this -> _filename . ' : ' . $this -> _linenum;
    				}
    				$option = str_replace($remove, '', $option);
    			}
    			
    			finish_clean:
    			//Now there will probably be lots of empty parens and stuff
    			$strlen = strlen($option);
    			$oldlen = $strlen + 1;
    			while($strlen != $oldlen){
    				$oldlen = $strlen;
    				$option = str_replace('||', '|', $option);
    				$option = str_replace('&&', '&', $option);
    				$option = str_replace('()', '', $option);
    				$option = str_replace('&~)', ')', $option);
    				$option = str_replace('|~)', ')', $option);
    				$option = str_replace('~~', '~', $option);
    				$option = str_replace('~&', '&', $option);
    				$option = str_replace('~|', '|', $option);
    				$option = str_replace('(~)', '', $option);
    				$option = str_replace('(&~)', '', $option);
    				$option = str_replace('(|~)', '', $option);
    				$option = str_replace('(&)', '', $option);
    				$option = str_replace('(|)', '', $option);
    				$option = str_replace('(|', '(', $option);
    				$option = str_replace('(&', '(', $option);
    				$option = str_replace('|)', ')', $option);
    				$option = str_replace('&)', ')', $option);
    				$strlen = strlen($option);
    			}
    			while(  ($option{$strlen - 1} == '&') ||
    					($option{$strlen - 1} == '|') ||
    					($option{$strlen - 1} == '~')){
    						$option = substr($option, 0, -1);
    						$strlen = strlen($option);
    			}
    			
    			while(  ($option{0} == '&') ||
    					($option{0} == '|')){
    						$option = substr($option, 1);
    			}
    			
    			$option = str_replace('&', '|', $option);
    		}
    
    		private function _pushOnStack($item, $elsed, $elsetext){
    			array_push($this -> _stack, array($item, $elsed, $elsetext));
    			return $this -> _pushOption();
    		}
    		
    		private function _popOffStack($elsetext = NULL){
    			if (empty($this -> _stack)){
    				funcs::fail($this -> _filename . ' has mismatched push/pops', $this);
    			} 
    			
    			$option = array_pop($this -> _stack);
    			if (!$option[1]){
    				if ($elsetext === NULL){
    					$this -> _pushOnStack($option[2], true, '');
    					$option = array_pop($this -> _stack);  //its not supposed to be there permanently
    				} else if (!empty($elsetext)){
    					$this -> _pushOnStack($elsetext, false, $option[2] . '&&!(' . $elsetext . ')');
    				} else {
    					$this -> _pushOnStack($option[2], true, '');
    				}
    			}
    			return $option;
    		}
    	}
    	
    	class analyzer {
    		public static function handleFile($file, array & $options){
    			$ext = substr(strrchr($file, '.'), 1);
    			if (($ext == 'h') ||
    				($ext == 'c') ||
    				($ext == 'hxx') ||
    				($ext == 'hpp') ||
    				($ext == 'cpp')){
    				echo 'Analyzing ' . $main . '/' . $file . PHP_EOL;
    				$f = new fileFetcher($main . '/' . $file);
    				$f -> findOptions($options);
    			}
    		}
    		
    		public static function readDirs($main, array & $options){
    			if (is_file($main)){
    				self::handleFile($main, $options);
    			} else {
    				$dirHandle = opendir($main);
    				while($file = readdir($dirHandle)){
    					if(is_dir($main . '/' . $file) && $file != '.' && $file != '..'){
    						self::readDirs($main . '/' . $file, $options);
    					} else {
    						self::handleFile($main . '/' . $file, $options);
    					}
    				} 
    			}
    		}
    		
    		public static function go(){
    			$options = array();
    			if (is_array(config::getProductPath())){
    				foreach(config::getProductPath() as $folder){
    					self::readDirs($folder, $options);
    				}
    			} else {
    				self::readDirs(config::getProductPath(), $options);
    			}
    			self::_finish($options);
    		}
    		
    		private static function _finish(&$options){
    			mkdir(config::getOutputPath(), 0777, true);
    			if (config::getOutputStyle() == config::OUTPUT_HEADER){
    				$counter = 0;
    				foreach($options as $option){
    					$output = '#ifndef AUTOGENERATED_OPTION_COMBOS_H' . PHP_EOL . '#define AUTOGENERATED_OPTION_COMBOS_H' . PHP_EOL . PHP_EOL;
    					$output .= '/* This option combo satisfies the following locations: ' . PHP_EOL;
    					foreach($option['locations'] as $location){
    						$output .= '     ' . $location['file'] . ' : ' . $location['line'] . PHP_EOL;
    						$output .= '          Options: ' . $location['options'] . PHP_EOL; 
    					}
    					$output .= '*/' . PHP_EOL . PHP_EOL . $option['willbeused'][1];
    
    					file_put_contents(config::getOutputPath() . '/output' . ++$counter . '.h', $output); 
    				}
    				$outputto = config::getOutputPath() . '/output*.h';
    			} else {
    				$output =  '# This is the combinations created by the combo generator' . PHP_EOL;
    				$output .= '# You can iterate this file using a script, any non-empty line not starting with # is a combo' . PHP_EOL . PHP_EOL . PHP_EOL;
    				foreach($options as $option){
    					$output .= '# This option combo satisfies the following locations: ' . PHP_EOL;
    					foreach($option['locations'] as $location){
    						$output .= '#     ' . $location['file'] . ' : ' . $location['line'] . PHP_EOL;
    						$output .= '#          Options: ' . $location['options'] . PHP_EOL; 
    					}
    					$output .= $option['willbeused'][0] . PHP_EOL . PHP_EOL . PHP_EOL;
    				}
    				$outputto = config::getOutputPath() . '/output.txt';
    				file_put_contents($outputto, $output); 
    			}
    			echo 'Outputting ' . count($options) . ' combinations of options to ' . $outputto . PHP_EOL;
    			file_put_contents(config::getOutputPath() . '/ignored.txt', implode(PHP_EOL, fileFetcher::$_ignores));
    		}
    	}
    		
    	analyzer::go();
    It will output all combinations of options to a file, and tell you what preprocessor option set goes to what piece of code. It's also optimized to remove duplicate option sets. It may merge sets too if they are not mutually exclusive. You have to tell it which options you care about, for most preprocessor I use is for detecting compiler version, which we don't want to alter when testing.

    It spits out a file like this:
    output.txt
    Code:
     # This is the combinations created by the combo generator
    # You can iterate this file using a script, any non-empty line not starting with # is a combo
    
    
    # This option combo satisfies the following locations: 
    #     //Users/ninja9578/Documents/Acceptance Tests/test.h : 2
    #          Options: OPTION1
     -DOPTION1
    
    
    # This option combo satisfies the following locations: 
    #     //Users/ninja9578/Documents/Acceptance Tests/test.h : 3
    #          Options: OPTION1 && OPTION2
     -DOPTION1 -DOPTION2=15
    
    
    # This option combo satisfies the following locations: 
    #     //Users/ninja9578/Documents/Acceptance Tests/test.h : 4
    #          Options: OPTION1 && OPTION2 && (((OPTION3) || !(OPTION4)))
     -DOPTION1 -DOPTION2=15 -DOPTION3 -DOPTION5
    
    
    # This option combo satisfies the following locations: 
    #     //Users/ninja9578/Documents/Acceptance Tests/test.h : 6
    #          Options: OPTION1 && OPTION2 && (!(((OPTION3) || !(OPTION4))))
     -DOPTION1 -DOPTION2=15 -DOPTION4
    
    
    # This option combo satisfies the following locations: 
    #     //Users/ninja9578/Documents/Acceptance Tests/test.h : 7
    #          Options: OPTION1 && (!(OPTION5) && define(I_DONT_CARE_ABOUT_THIS_OPTION))
     -DOPTION1 -DOPTION2=15 -DOPTION3 -DOPTION4
    
    
    # This option combo satisfies the following locations: 
    #     //Users/ninja9578/Documents/Acceptance Tests/test.h : 9
    #          Options: OPTION1 && (!(OPTION2)&&!(!(OPTION5) && define(I_DONT_CARE_ABOUT_THIS_OPTION)))
     -DOPTION1 -DOPTION3 -DOPTION4 -DOPTION5
    
    
    # This option combo satisfies the following locations: 
    #     //Users/ninja9578/Documents/Acceptance Tests/test.h : 12
    #          Options: !(OPTION1)
     -DOPTION2=15 -DOPTION3 -DOPTION4 -DOPTION5
    It can be configured to output a set of header files instead if you wish. It also prints what it ignored to a file so that you can be sure you don't have any other options you want to check.

    Enjoy. Let me know if you have any problems with it, I wrote it pretty quickly as I'm sure you can tell.
    Last edited by ninja9578; April 3rd, 2012 at 03:16 PM.

  2. #2
    Join Date
    Oct 2006
    Location
    Sweden
    Posts
    3,634

    Re: Preprocessor option combiner


    That could be handy even though I tend to keep the number of #define as low as possible in my code. Quite frankly I have some love/hate relation to them.
    Debugging is twice as hard as writing the code in the first place.
    Therefore, if you write the code as cleverly as possible, you are, by
    definition, not smart enough to debug it.
    - Brian W. Kernighan

    To enhance your chance's of getting an answer be sure to read
    http://www.codeguru.com/forum/announ...nouncementid=6
    and http://www.codeguru.com/forum/showthread.php?t=366302 before posting

    Refresh your memory on formatting tags here
    http://www.codeguru.com/forum/misc.php?do=bbcode

    Get your free MS compiler here
    http://www.microsoft.com/visualstudio/eng/downloads

  3. #3
    Join Date
    Jan 2009
    Posts
    1,689

    Re: Preprocessor option combiner

    Quote Originally Posted by S_M_A View Post

    That could be handy even though I tend to keep the number of #define as low as possible in my code. Quite frankly I have some love/hate relation to them.
    Me too, I usually just use them to do things like check for NDEBUG, _GNU_C_, or __cplusplus, but they have there uses from time to time, especially when you maintain libraries.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center