mirror of
https://github.com/hedge-dev/XenonRecomp.git
synced 2025-06-06 18:31:03 +00:00
160 lines
4.6 KiB
Markdown
160 lines
4.6 KiB
Markdown
# Testing in Capstone
|
|
|
|
## Running tests
|
|
|
|
### Types of test and their location
|
|
|
|
_YAML test files_
|
|
|
|
These test files are consumed by the various `cstest` tools.
|
|
They contain all detail tests. As well as the LLVM regression tests (`MC` tests).
|
|
|
|
Directories group tests by the category they intent to test.
|
|
|
|
_Legacy (integration)_
|
|
|
|
Legacy tests which only printed to `stdout`. In practice they only test if the code segfaults.
|
|
Checking the produced output was not implemented.
|
|
|
|
### Testing tools and usage
|
|
|
|
#### `cstest`
|
|
|
|
`cstest` is the testing tool written in C. It is implemented in `suite/cstest/`
|
|
It consumes the `yaml` files and reports errors or mismatches for disassembled instructions and their details.
|
|
|
|
**Building**
|
|
|
|
> _Dependencies:_ `cstest` requires the `libyaml` library.
|
|
|
|
You build `cstest` by adding the `-DCAPSTONE_BUILD_CSTEST=1` option during configuration of the Capstone build.
|
|
|
|
If you build and install Capstone `cstest` gets installed as well.
|
|
Otherwise you find it in the build directory.
|
|
|
|
```bash
|
|
# Install libyaml
|
|
# sudo apt install libyaml-dev
|
|
# or
|
|
# sudo dnf install libyaml-devel
|
|
cd "<capstone-repo-root>"
|
|
# Optionally add the `-DENABLE_ASAN=1` flag.
|
|
cmake -B build -DCMAKE_BUILD_TYPE=Debug -DCAPSTONE_BUILD_CSTEST=ON
|
|
cmake --build build --config Debug
|
|
cmake --install build --prefix "<install-prefix>"
|
|
```
|
|
|
|
Run the integration tests for `cstest` itself
|
|
|
|
```bash
|
|
./suite/cstest/test/integration_tests.py cstest
|
|
```
|
|
|
|
**Run the tests**
|
|
|
|
```bash
|
|
# Check supported options
|
|
cstest -h
|
|
# Run all
|
|
cstest tests/
|
|
```
|
|
|
|
Alternatively, you can use the `CMake` test manager.
|
|
|
|
```bash
|
|
# List available tests
|
|
ctest --test-dir build -N
|
|
# Run a specific test
|
|
ctest --test-dir build -R "<name>"
|
|
```
|
|
|
|
#### `cstest_py`
|
|
|
|
`cstest_py` is the testing tool written in Python. It is implemented in `bindings/python/cstest_py`
|
|
It consumes the `yaml` files and reports errors or mismatches for disassembled instructions and their details.
|
|
|
|
**Installing**
|
|
|
|
You need to install the Capstone Python bindings first and afterwards the `cstest_py`.
|
|
|
|
```bash
|
|
# Optionally, create a new virtual environment
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
|
|
cd bindings/python
|
|
pip install -e .
|
|
pip install -e cstest_py
|
|
cd ../..
|
|
```
|
|
|
|
Run the integration tests for `cstest_py` itself
|
|
|
|
```bash
|
|
./suite/cstest/test/integration_tests.py cstest_py
|
|
```
|
|
|
|
And run the tests
|
|
|
|
```bash
|
|
# Check supported options
|
|
cstest_py -h
|
|
# Run all
|
|
cstest_py tests/
|
|
```
|
|
|
|
## Add new tests
|
|
|
|
### Unit and integration tests
|
|
|
|
Add the source into `test/integration` or `test/unit` respectively and update the `CMakeLists.txt` file.
|
|
|
|
### YAML
|
|
|
|
There are very few fields which are mandatory to fill.
|
|
Check `suite/cstest/test/min_valid_test_file.yaml` to see which one.
|
|
|
|
- In general it is useful to just copy a previous test file and rewrite it accordingly.
|
|
- If you assign C enumeration identifiers to some fields (to check enumeration values),
|
|
ensure they are added on the `suite/cstest/include/test_mapping.h`. Otherwise, `cstest` cannot map the strings
|
|
to the values for comparison.
|
|
- Rarely used, but useful fields are: `name`, `skip`, `skip_reason`.
|
|
|
|
#### MC regression tests
|
|
|
|
The `MCUpdater` translates most test files of the LLVM MC regression tests into our YAML files.
|
|
|
|
The LLVM regression tests, check the bytes and assembly for all instructions of an architecture.
|
|
They do it by passing bytes or assembly to the `llvm-mc` and `FileCheck` tool and compare the output.
|
|
We capture this output and process it into YAML.
|
|
So you need to install `llvm-mc` and `FileCheck` for our updater to work.
|
|
|
|
To update the YAML MC regression tests, you need to install `Auto-Sync` and run the `MCUpdater`.
|
|
|
|
```bash
|
|
cd suite/auto-sync/
|
|
# Follow install instructions of Auto-Sync described in the README
|
|
# And run the updater:
|
|
./src/autosync/MCUpdater.py -a ARCH
|
|
ls build/mc_out/
|
|
# The produce yaml files. Copy them manually to tests/MC/ARCH
|
|
```
|
|
|
|
**Please note:**
|
|
|
|
Each of the LLVM test files can contain several `llvm-mc` commands to run on the same file.
|
|
This is done to test the same file with different CPU features enabled.
|
|
So it can test different assembly flavors etc.
|
|
|
|
In Capstone all modules enable always all CPU features (even if this is not
|
|
possible in reality).
|
|
Due to this, we always parse all `llvm-mc` commands but only write the last version of them to disk.
|
|
So if the same test file is tested with three different features enables, once with `FeatureA`, `FeatureB` and `FeatureC`
|
|
we only save the output with `FeatureC` enabled.
|
|
|
|
This might give you MC test files which fail due to valid but mismatching disassembly.
|
|
You can set the `skip` field for those tests and add a `skip_reason`.
|
|
|
|
Once https://github.com/capstone-engine/capstone/issues/1992 is resolved, we can
|
|
test all variants.
|