{"meta":{"title":"test run","intro":"Run unit tests for QL queries.","product":"Security and code quality","breadcrumbs":[{"href":"/en/code-security","title":"Security and code quality"},{"href":"/en/code-security/reference","title":"Reference"},{"href":"/en/code-security/reference/code-scanning","title":"Code scanning"},{"href":"/en/code-security/reference/code-scanning/codeql","title":"CodeQL"},{"href":"/en/code-security/reference/code-scanning/codeql/codeql-cli-manual","title":"CodeQL CLI manual"},{"href":"/en/code-security/reference/code-scanning/codeql/codeql-cli-manual/test-run","title":"test run"}],"documentType":"article"},"body":"# test run\n\nRun unit tests for QL queries.\n\n> \\[!NOTE]\n> This content describes the most recent release of the CodeQL CLI. For more information about this release, see <https://github.com/github/codeql-cli-binaries/releases>.\n>\n> To see details of the options available for this command in an earlier release, run the command with the <span style=\"white-space: nowrap;\">`--help`</span> option in your terminal.\n\n## Synopsis\n\n```shell copy\ncodeql test run [--threads=<num>] [--ram=<MB>] <options>... -- <test|dir>...\n```\n\n## Description\n\nRun unit tests for QL queries.\n\n## Options\n\n### Primary Options\n\n#### `<test|dir>...`\n\nEach argument is one of:\n\n* A `.ql` or `.qlref` file that defines a test to run.\n* A directory which will be searched recursively for tests to run.\n\n#### `--failing-exitcode=<code>`\n\n\\[Advanced] Set the exit code to produce if any failures are\nencountered. Usually 1, but tooling that parses the output may find it\nuseful to set it to 0.\n\n#### `--format=<fmt>`\n\nSelect output format. Possible choices:\n\n`text` *(default)*: A human-readable textual rendering.\n\n`json`: A streamed JSON array of test result objects.\n\n`betterjson`: A streamed JSON array of event objects.\n\n`jsonz`: A stream of zero-terminated JSON test result objects.\n\n`betterjsonz`: A stream of zero-terminated JSON event objects.\n\nFor the `betterjson` and `betterjsonz` formats, each event has a `type`\nproperty specifying the type of the event. New event types may be added\nin the future, so consumers should ignore any event with an unrecognized\n`kind` property.\n\n#### `--[no-]keep-databases`\n\n\\[Advanced] Preserve the databases extracted to run the test queries,\neven where all tests in a directory pass. (The database will always be\nleft present when there are tests that *fail*).\n\n#### `--[no-]fast-compilation`\n\n\\[Deprecated] \\[Advanced] Omit particularly slow optimization steps\nwhen compiling test queries.\n\n#### `--[no-]learn`\n\n\\[Advanced] When a test produces unexpected output, instead of failing\nit, update its `.expected` file to match the actual output, such that it\npasses. Tests can still fail in this mode, for example if creation of a\ntest database to query does not succeed.\n\n#### `--consistency-queries=<dir>`\n\n\\[Advanced] A directory with consistency queries that will be run for\neach test database. These queries should not produce any output (except\nwhen they find a problem) unless the test directory includes a\n`CONSISTENCY` subdirectory with a `.expected` file. This is mostly\nuseful for testing extractors.\n\n#### `--[no-]check-databases`\n\n\\[Advanced] Run\n[codeql dataset check](/en/code-security/reference/code-scanning/codeql/codeql-cli-manual/dataset-check) over each test database created and report a failure if it detects inconsistencies. This is useful when testing extractors. If the check is (temporarily!) expected to fail for a particular database, place a `DB-CHECK.expected` file in the test directory.\n\n#### `--[no-]show-extractor-output`\n\n\\[Advanced] Show the output from extractor scripts that create test\ndatabases. This can be useful while developing or editing test cases.\nBeware that it can cause duplicated or malformed output if you use this\nwith multiple threads!\n\n#### `-M, --ram=<MB>`\n\nSet total amount of RAM the test runner should be allowed to use.\n\n#### `--slice=<N/M>`\n\n\\[Advanced] Divide the test cases into *M* roughly equal-sized slices\nand process only the *N*th of them. This can be used for manual\nparallelization of the testing process.\n\n#### `--[no-]strict-test-discovery`\n\n\\[Advanced] Only use queries that can be strongly identified as tests.\nThis mode tries to distinguish between `.ql` files that define unit\ntests and `.ql` files that are meant to be useful queries. This option\nis used by tools, such as IDEs, that need to identify all unit tests in\na directory tree without depending on previous knowledge of how the\nfiles in it are arranged.\n\nWithin a QL pack whose `qlpack.yml` declares a `tests` directory, all\n`.ql` files in that directory are considered tests, and `.ql` files\noutside it are ignored. In a QL pack that doesn't declare a `tests`\ndirectory, a `.ql` file is identified as a test only if it has a\ncorresponding `.expected` file.\n\nFor consistency, `.qlref` files are limited by the same rules as `.ql`\nfiles even though a `.qlref` file cannot really be a non-test.\n\n### Options to find libraries and extractors used by the tests\n\n#### `--search-path=<dir>[:<dir>...]`\n\nA list of directories under which QL packs may be found. Each directory\ncan either be a QL pack (or bundle of packs containing a\n`.codeqlmanifest.json` file at the root) or the immediate parent of one\nor more such directories.\n\nIf the path contains more than one directory, their order defines\nprecedence between them: when a pack name that must be resolved is\nmatched in more than one of the directory trees, the one given first\nwins.\n\nPointing this at a checkout of the open-source CodeQL repository ought\nto work when querying one of the languages that live there.\n\nIf you have checked out the CodeQL repository as a sibling of the\nunpacked CodeQL toolchain, you don't need to give this option; such\nsibling directories will always be searched for QL packs that cannot be\nfound otherwise. (If this default does not work, it is strongly\nrecommended to set up `--search-path` once and for all in a per-user\nconfiguration file).\n\n(Note: On Windows the path separator is `;`).\n\n#### `--additional-packs=<dir>[:<dir>...]`\n\nIf this list of directories is given, they will be searched for packs\nbefore the ones in `--search-path`. The order between these doesn't\nmatter; it is an error if a pack name is found in two different places\nthrough this list.\n\nThis is useful if you're temporarily developing a new version of a pack\nthat also appears in the default path. On the other hand, it is *not\nrecommended* to override this option in a config file; some internal\nactions will add this option on the fly, overriding any configured\nvalue.\n\n(Note: On Windows the path separator is `;`).\n\n#### `--library-path=<dir>[:<dir>...]`\n\n\\[Advanced] An optional list of directories that will be added to the\nraw import search path for QL libraries. This should only be used if\nyou're using QL libraries that have not been packaged as QL packs.\n\n(Note: On Windows the path separator is `;`).\n\n#### `--dbscheme=<file>`\n\n\\[Advanced] Explicitly define which dbscheme queries should be compiled\nagainst. This should only be given by callers that are extremely sure\nwhat they're doing.\n\n#### `--compilation-cache=<dir>`\n\n\\[Advanced] Specify an additional directory to use as a compilation\ncache.\n\n#### `--no-default-compilation-cache`\n\n\\[Advanced] Don't use compilation caches in standard locations such as\nin the QL pack containing the query or in the CodeQL toolchain\ndirectory.\n\n### Options for configuring the CodeQL package manager\n\n#### `--registries-auth-stdin`\n\nAuthenticate to GitHub Enterprise Server Container registries by passing\na comma-separated list of \\<registry\\_url>=\\<token> pairs.\n\nFor example, you can pass\n`https://containers.GHEHOSTNAME1/v2/=TOKEN1,https://containers.GHEHOSTNAME2/v2/=TOKEN2`\nto authenticate to two GitHub Enterprise Server instances.\n\nThis overrides the CODEQL\\_REGISTRIES\\_AUTH and GITHUB\\_TOKEN environment\nvariables. If you only need to authenticate to the github.com Container\nregistry, you can instead authenticate using the simpler\n`--github-auth-stdin` option.\n\n#### `--github-auth-stdin`\n\nAuthenticate to the github.com Container registry by passing a\ngithub.com GitHub Apps token or personal access token via standard\ninput.\n\nTo authenticate to GitHub Enterprise Server Container registries, pass\n`--registries-auth-stdin` or use the CODEQL\\_REGISTRIES\\_AUTH environment\nvariable.\n\nThis overrides the GITHUB\\_TOKEN environment variable.\n\n### Options to control query compilation\n\n#### `--no-release-compatibility`\n\n\\[Advanced] Use the newest compiler features, at the cost of\nportability.\n\nFrom time to time, new QL language features and evaluator optimizations\nwill be supported by the QL evaluator a few releases before they are\nenabled by default in the QL compiler. This helps ensure that the\nperformance you experience when developing queries in the newest CodeQL\nrelease can be matched by slightly older releases that may still be in\nuse for Code Scanning or CI integrations.\n\nIf you do not care about your queries being compatible with other\n(earlier or later) CodeQL releases, you can sometimes achieve a small\namount of extra performance by using this flag to enable recent\nimprovements in the compiler early.\n\nIn releases where there are no recent improvements to enable, this\noption silently does nothing. Thus it is safe to set it once and for all\nin your global CodeQL config file.\n\nAvailable since `v2.11.1`.\n\n### Options that control the evaluation of test queries\n\n#### `--[no-]tuple-counting`\n\n\\[Advanced] Display tuple counts for each evaluation step in the query\nevaluator logs. If the `--evaluator-log` option is provided, tuple\ncounts will be included in both the text-based and structured JSON logs\nproduced by the command. (This can be useful for performance\noptimization of complex QL code).\n\n#### `--timeout=<seconds>`\n\n\\[Advanced] Set the timeout length for query evaluation, in seconds.\n\nThe timeout feature is intended to catch cases where a complex query\nwould take \"forever\" to evaluate. It is not an effective way to limit\nthe total amount of time the query evaluation can take. The evaluation\nwill be allowed to continue as long as each separately timed part of the\ncomputation completes within the timeout. Currently these separately\ntimed parts are \"RA layers\" of the optimized query, but that might\nchange in the future.\n\nIf no timeout is specified, or is given as 0, no timeout will be set\n(except for codeql test run, where the default timeout is 5 minutes).\n\n#### `-j, --threads=<num>`\n\nUse this many threads to evaluate queries.\n\nDefaults to 1. You can pass 0 to use one thread per core on the machine,\nor -*N* to leave *N* cores unused (except still use at least one\nthread).\n\n### Options for controlling outputting of structured evaluator logs\n\n#### `--evaluator-log=<file>`\n\n\\[Advanced] Output structured logs about evaluator performance to the\ngiven file. The format of this log file is subject to change with no\nnotice, but will be a stream of JSON objects separated by either two\nnewline characters (by default) or one if the `--evaluator-log-minify`\noption is passed. Please use `codeql generate log-summary <file>` to\nproduce a more stable summary of this file, and avoid parsing the file\ndirectly. The file will be overwritten if it already exists.\n\n#### `--evaluator-log-minify`\n\n\\[Advanced] If the `--evaluator-log` option is passed, also passing\nthis option will minimize the size of the JSON log produced, at the\nexpense of making it much less human readable.\n\n### Options for checking imported TRAP\n\n#### `--[no-]check-undefined-labels`\n\n\\[Advanced] Report errors for undefined labels.\n\n#### `--[no-]check-unused-labels`\n\n\\[Advanced] Report errors for unused labels.\n\n#### `--[no-]check-repeated-labels`\n\n\\[Advanced] Report errors for repeated labels.\n\n#### `--[no-]check-redefined-labels`\n\n\\[Advanced] Report errors for redefined labels.\n\n#### `--[no-]check-use-before-definition`\n\n\\[Advanced] Report errors for labels used before they're defined.\n\n#### `--[no-]fail-on-trap-errors`\n\n\\[Advanced] Exit non-zero if an error occurs during trap import.\n\n#### `--[no-]include-location-in-star`\n\n\\[Advanced] Construct entity IDs that encode the location in the TRAP\nfile they came from. Can be useful for debugging of TRAP generators, but\ntakes up a lot of space in the dataset.\n\n#### `--[no-]linkage-aware-import`\n\n\\[Advanced] Controls whether [codeql dataset import](/en/code-security/reference/code-scanning/codeql/codeql-cli-manual/dataset-import) is linkage-aware *(default)* or not. On projects where this part of database creation\nconsumes too much memory, disabling this option may help them progress\nat the expense of database completeness.\n\nAvailable since `v2.15.3`.\n\n### Common options\n\n#### `-h, --help`\n\nShow this help text.\n\n#### `-J=<opt>`\n\n\\[Advanced] Give option to the JVM running the command.\n\n(Beware that options containing spaces will not be handled correctly.)\n\n#### `-v, --verbose`\n\nIncrementally increase the number of progress messages printed.\n\n#### `-q, --quiet`\n\nIncrementally decrease the number of progress messages printed.\n\n#### `--verbosity=<level>`\n\n\\[Advanced] Explicitly set the verbosity level to one of errors,\nwarnings, progress, progress+, progress++, progress+++. Overrides `-v`\nand `-q`.\n\n#### `--logdir=<dir>`\n\n\\[Advanced] Write detailed logs to one or more files in the given\ndirectory, with generated names that include timestamps and the name of\nthe running subcommand.\n\n(To write a log file with a name you have full control over, instead\ngive `--log-to-stderr` and redirect stderr as desired.)\n\n#### `--common-caches=<dir>`\n\n\\[Advanced] Controls the location of cached data on disk that will\npersist between several runs of the CLI, such as downloaded QL packs and\ncompiled query plans. If not set explicitly, this defaults to a\ndirectory named `.codeql` in the user's home directory; it will be\ncreated if it doesn't already exist.\n\nAvailable since `v2.15.2`."}