README.md

# BASE: Benchmarking Autonomous Scattering Experiments

An implementation of the benchmarking procedure presented in Teixeira Parente et al. (2021), _Benchmarking autonomous scattering experiments illustrated on TAS_.

## License
The contents of this repository are put under the GNU General Public License (GPL) in version 3, see `LICENSE`.

The file `tas/_tas.py` contains code from _NICOS_, the instrument control
system at the Heinz Maier-Leibnitz Zentrum (MLZ), see
<https://forge.frm2.tum.de/wiki/projects:nicos:index>.


## Introductory remarks
The mentioned article proposes a benchmarking procedure for scattering experiments in general and illustrates its application on three-axes spectrometry (TAS).
The key components of the procedure are _cost measures_, _benefit measures_, and _test cases_ as defined in the article.

The file structure in this repository follows the logic of the article, i.e., it provides implementations of these components and makes them precise for TAS.


## Idea
The idea is to run the benchmarking procedure and store the results as CSV file in order to provide it to the community for comparison.


## Requirements
The code requires _Python 3.7+_ and has no special dependencies except for the usual scientific stack (_numpy_, _scipy_, _matplotlib_).


## Library
Directory: `lib/`

### General
Implementation of the mentioned components, independent of the scattering method used, as well as the procedure itself.

Directory: `lib/general/`

`lib/general/benchmarking.py`: Classes and functions for components and the procedure itself.

`lib/general/diagnostics.py`: Functions for diagnostics of components.

`lib/general/utils.py`: General utilities.

### TAS
Implementation of TAS-specific components following the description in the article.

Directory: `lib/tas/`

`lib/tas/_tas.py`: Classes for TAS-specific components as `Instrument`, `InstrumentConfiguration`, etc.
It is imported by `lib/tas/__init__.py`.

`lib/tas/benchmarking.py`: Classes for benchmark components concretized for the TAS setting.

`lib/tas/diagnostics.py`: Functions for diagnostics of TAS test cases, TAS experiments, and TAS benchmark results.

`lib/tas/utils.py`: Utilities for TAS-specific components.


## Execution
### General
Directory: `general/`

`general/run_diagnostics.py`: Script for running diagnostics of benchmark results.

### TAS
Directory: `tas/`

`tas/experiments/`: Directory for storing JSON files of conducted experiments.
It is empty after cloning.
Each approach the benchmarking procedure is run for has its own subdirectory.

`tas/results/`: Directory for storing CSV files of benchmark results.
It is empty after cloning.
Each approach the benchmarking procedure is run for has its own subdirectory.

`tas/test_cases/`: Directory with TAS test cases as JSON files.

`tas/dummies.py`: Implementation of dummy approaches.
`RandomnessApproach` chooses measurement locations uniformly at random in the valid part of the region of interest.
`GridApproach` chooses measurement locations on a grid structure in a specific order.

`tas/run_benchmark.py`: Script for running the benchmark procedure with experiments from the approach's subdirectory in `tas/experiments/`.

`tas/run_diagnostics.py`: Script for running diagnostics of the components in the TAS setting, i.e., test cases, experiments, and benchmark results.
Lines for components that are not needed to be diagnosed can be commented out.

`tas/run_experiments.py`: Script for running experiments in the context of test cases from `tas/test_cases/`.

`tas/setting.py`: File with the environment of the TAS setting.
It contains instances of TAS-specific components and is supposed to be imported by the scripts.
It is the only file that needs to be adjusted for running a custom benchmark.

#### Running a benchmark
1. Implement your approach by deriving from `TASAutonomousApproach` in `lib.tas.benchmarking`.
2. Setup benchmark components (your approach, cost measures, benefit measure, cost milestone values for each test case) in `tas/setting.py`.
The components mentioned in the article are already instantiated by default.
3. Run experiments (run `tas/run_experiments.py` from the root directory).
Experiments are stored in the approach's subdirectory in `tas/experiments/`.
Filenames are aligned to the name of the corresponding test case.
4. If necessary, run diagnostics (run `tas/run_diagnostics.py`) to investigate outcomes of the experiments.
5. Run benchmark (run `tas/run_benchmark.py` from the root directory).
Results are stored in the approach's subdirectory in `tas/results/`.
6. If necessary, run diagnostics (run `general/run_diagnostics.py`) to investigate benchmark results.
7. Share results with the community.
8. If applicable, get results from the community, store them in a subdirectory in `tas/results` and make a comparison by running the second part of `general/run_diagnostics.py`.

#### Testing with a dummy approach in the TAS setting
The procedure can be tested in the TAS setting without modifying a fresh clone of this repository with the given test cases in `tas/test_cases` and the dummy approach based on random sampling from `tas/dummies.py`.
1. Run `tas/run_experiments.py`.
2. Run `tas/run_diagnostics.py` to diagnose test cases and experiments.
3. Run `tas/run_benchmark.py`.
4. Run `general/run_diagnostics.py` to diagnose the results.