README.md 5.25 KB
Newer Older
1
# BASE: Benchmarking Autonomous Scattering Experiments
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
2

3
An implementation of the benchmarking procedure presented in Teixeira Parente et al. (2021), _Benchmarking autonomous scattering experiments illustrated on TAS_.
4

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
5
6
## License
The contents of this repository are put under the GNU General Public License (GPL) in version 3, see `LICENSE`.
7

8
9
The file `tas/_tas.py` contains code from _NICOS_, the instrument control
system at the Heinz Maier-Leibnitz Zentrum (MLZ), see
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
10
11
12
13
<https://forge.frm2.tum.de/wiki/projects:nicos:index>.


## Introductory remarks
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
14
15
16
17
The mentioned article proposes a benchmarking procedure for scattering experiments in general and illustrates its application on three-axes spectrometry (TAS).
The key components of the procedure are _cost measures_, _benefit measures_, and _test cases_ as defined in the article.

The file structure in this repository follows the logic of the article, i.e., it provides implementations of these components and makes them precise for TAS.
18
19
20


## Idea
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
21
22
The idea is to run the benchmarking procedure and store the results as CSV file in order to provide it to the community for comparison.

23

24
## Requirements
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
25
The code requires _Python 3.7+_ and has no special dependencies except for the usual scientific stack (_numpy_, _scipy_, _matplotlib_).
26
27


Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
28
29
## Library
Directory: `lib/`
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
30

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
31
### General
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
32
Implementation of the mentioned components, independent of the scattering method used, as well as the procedure itself.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
33

34
35
Directory: `lib/general/`

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
36
`lib/general/benchmarking.py`: Classes and functions for components and the procedure itself.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
37

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
38
`lib/general/diagnostics.py`: Functions for diagnostics of components.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
39

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
40
`lib/general/utils.py`: General utilities.
41

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
42
### TAS
43
44
Implementation of TAS-specific components following the description in the article.

45
Directory: `lib/tas/`
46

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
47
`lib/tas/_tas.py`: Classes for TAS-specific components as `Instrument`, `InstrumentConfiguration`, etc.
48
It is imported by `lib/tas/__init__.py`.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
49
50
51
52

`lib/tas/benchmarking.py`: Classes for benchmark components concretized for the TAS setting.

`lib/tas/diagnostics.py`: Functions for diagnostics of TAS test cases, TAS experiments, and TAS benchmark results.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
53

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
54
55
56
`lib/tas/utils.py`: Utilities for TAS-specific components.


57
58
## Execution
### General
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
59
60
61
62
Directory: `general/`

`general/run_diagnostics.py`: Script for running diagnostics of benchmark results.

63
64
### TAS
Directory: `tas/`
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
65

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
66
67
`tas/experiments/`: Directory for storing JSON files of conducted experiments.
It is empty after cloning.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
68
Each approach the benchmarking procedure is run for has its own subdirectory.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
69
70
71

`tas/results/`: Directory for storing CSV files of benchmark results.
It is empty after cloning.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
72
Each approach the benchmarking procedure is run for has its own subdirectory.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
73
74
75

`tas/test_cases/`: Directory with TAS test cases as JSON files.

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
76
77
78
`tas/dummies.py`: Implementation of dummy approaches.
`RandomnessApproach` chooses measurement locations uniformly at random in the valid part of the region of interest.
`GridApproach` chooses measurement locations on a grid structure in a specific order.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
79

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
80
`tas/run_benchmark.py`: Script for running the benchmark procedure with experiments from the approach's subdirectory in `tas/experiments/`.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
81

82
83
`tas/run_diagnostics.py`: Script for running diagnostics of the components in the TAS setting, i.e., test cases, experiments, and benchmark results.
Lines for components that are not needed to be diagnosed can be commented out.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
84

Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
85
`tas/run_experiments.py`: Script for running experiments in the context of test cases from `tas/test_cases/`.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
86
87
88

`tas/setting.py`: File with the environment of the TAS setting.
It contains instances of TAS-specific components and is supposed to be imported by the scripts.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
89
It is the only file that needs to be adjusted for running a custom benchmark.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
90

91
#### Running a benchmark
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
92
93
1. Implement your approach by deriving from `TASAutonomousApproach` in `lib.tas.benchmarking`.
2. Setup benchmark components (your approach, cost measures, benefit measure, cost milestone values for each test case) in `tas/setting.py`.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
94
95
The components mentioned in the article are already instantiated by default.
3. Run experiments (run `tas/run_experiments.py` from the root directory).
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
96
Experiments are stored in the approach's subdirectory in `tas/experiments/`.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
97
Filenames are aligned to the name of the corresponding test case.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
98
99
100
101
102
103
104
105
106
4. If necessary, run diagnostics (run `tas/run_diagnostics.py`) to investigate outcomes of the experiments.
5. Run benchmark (run `tas/run_benchmark.py` from the root directory).
Results are stored in the approach's subdirectory in `tas/results/`.
6. If necessary, run diagnostics (run `general/run_diagnostics.py`) to investigate benchmark results.
7. Share results with the community.
8. If applicable, get results from the community, store them in a subdirectory in `tas/results` and make a comparison by running the second part of `general/run_diagnostics.py`.

#### Testing with a dummy approach in the TAS setting
The procedure can be tested in the TAS setting without modifying a fresh clone of this repository with the given test cases in `tas/test_cases` and the dummy approach based on random sampling from `tas/dummies.py`.
107
1. Run `tas/run_experiments.py`.
Mario Teixeira Parente's avatar
Mario Teixeira Parente committed
108
109
110
2. Run `tas/run_diagnostics.py` to diagnose test cases and experiments.
3. Run `tas/run_benchmark.py`.
4. Run `general/run_diagnostics.py` to diagnose the results.