Skip to content

Introduce the new embedded Python and produce Python packages (Major change)

Ammar Nejati requested to merge newPythonAPI into main

This is the last part of the major issue described in #96 (closed). This major change is performed in several steps; for detailed descriptions of the previous steps, see MRs !509 (merged), !593 (merged) and !601 (merged).

In this last step, the old Python API is completely removed and Python API is provided only via the Python packages ('wheels'). The corresponding CMake modules are merged into the new Python-related modules in cmake/BornAgain/multipython. That means that the build mechanism yields basically two products: The main GUI version, and the Python packages (for as many versions of Python as given). Hence, the main GUI version has no Python API, and the Python API is provided only via installing a Python package (via the standard Python tool pip).

These changes has led to a considerable complication of the build mechanism, esp. due to the shortcomings of the current CMake language.

New Embedded Python Interpreter

Besides the aforementioned changes, a new implementation of the embedded Python interpreter is introduced, with all the implementation strictly confined to the new PyTools module. The new implementation provides the Python and Numpy C-API to the rest of the codebase, so that the other modules remain completely Python-free. This is of crucial importance for the soundness and robustness of the code as a whole, as the embedded Python introduces a score of difficult issues to deal with (e.g., memory management via reference counting, Python's garbage collection, or error-checking the results from the Python interpreter).

Previously, the parts of the codebase which used Python C-API explicitly, were Base/Py, Device/Data/OutputData, and Sample/Multilayer/PyImport.

The 'stable' part of the Python C-API is marked explicitly in the embedded Python interpreter (see PyTools/Embed/PythonInterpreter.h). The compatibility across Python 3.7–3.9 is ensured; see https://docs.python.org/3/c-api/stable.html. Note that the Numpy C-API has no official guarantee for stability; see https://numpy.org/doc/stable/reference/c-api and https://numpy.org/doc/stable/reference/c-api/array.html.

Furthermore, the embedded Python functionality can be turned off completely by setting BORNAGAIN_PYTHON=OFF (provided the SWIG wrappers are produced with the same flag).

Moreover, as the first step, the embedded Python can apply the fabio Python package to read different file formats (see issue #156). This is yet to be tested thoroughly.

As a proposal, the class Result is introduced as a container for returned results which may fail (see PyTools/Embed/Result). A Result<T> instance contains a value of type T (or none) and a Status which denotes the status of the value (OK/Warning/Error). This is used to avoid throwing exceptions in a core module (the problems of which are addressed in detail before), leaving the decision to the callers (or the higher interface) whether an exception should be raised or not. The class Result is conceived after Rust's Result type; see https://doc.rust-lang.org/std/result.

Edited by Ammar Nejati

Merge request reports

Loading