Cppyy insights
This issue is referencing branch cppyy5.
How it works:
With cppyy you include headerfiles (cppyy.include) and load libraries (cppyy.load_library).
The symbols are not accessible directly after calling cppppyy.include, because cppyy is implemented in a lazy loading style.Therefore each symbol which is wanted first needs to manually be adressed by the developer. This can be done e.g by just typing
from cppyy.gbl import a_bornagain_symbol
or by calling
getattr(cppyy.gbl,"a_bornagain_symbol")
After this step it is possible to use the symbol by calling:
my_instance = cppyy.gbl.a_bornagain_symbol()
my_instance.do_sth()
Migrating from SWIG:
SWIG provides interface files where every header needed is declared in.
// libBornAgainBase.i
...
%{
#include <heinz/Complex.h>
#include "Base/Types/ICloneable.h"
#include "Base/Types/Span.h"
#include "Base/Const/Units.h"
#include "Base/Util/ThreadInfo.h"
#include "Base/Axis/Bin.h"
#include "Base/Axis/VariableBinAxis.h"
#include "Base/Axis/ConstKBinAxis.h"
#include "Base/Axis/CustomBinAxis.h"
#include "Base/Axis/FixedBinAxis.h"
#include "Base/Axis/PointwiseAxis.h"
#include "Base/Axis/Frame.h"
#include "Base/Vector/RotMatrix.h"
%}
...
We don't need these files anymore. Instead: We add a new cmake variable named api_files
for every libBornagain Module that has a cmake library target (Base,Device,Param,...). This variable is a list of file paths to the header files listed in the .i files.
This variable is used in the add_library_to_wheel
cmake function. Here we use a tool provided by cppyy. The cppyy-generator parses all header files provided by the api_files
and outputs the structure as a JSON file. Every file is getting created in the root directory of the python package (e.g libBornAgainbase.map). Additionally right now every header file is getting copied into a include directory into the python package.
(This tool also depends on libclang provided by pip, please before running, change the path to libclang.so in MakePythonWheel.cmake line 162 to the path of the installed libclang pip package (or add cmake script that automates finding it))
Using the JSON files prevents the need for importing every needed symbol manually. On package loading all JSON files get parsed and every found symbol is getting mapped from cppyy.gbl
to the module bornagain
This is done in function map_symbols_to_module(pkg) listed below:
#initializer.py
...
def map_symbols_to_module(pkg):
api_files = [
'libBornAgainBase.map',
'libBornAgainDevice.map',
'libBornAgainFit.map',
'libBornAgainParam.map',
'libBornAgainResample.map',
'libBornAgainSample.map',
'libBornAgainSim.map'
]
for file in api_files:
with open(os.path.join(ROOT_PATH,file),'r') as fp:
header_files = json.load(fp)
lib_name = file.split("BornAgain")[1].split(".")[0]
for header_file in header_files:
if header_file["kind"] != "file":
raise RuntimeError(f"Cannot load ${pkg} doe to broken python mapping.")
cppyy.include(os.path.join(ROOT_PATH,'include',lib_name,header_file["name"]))
for symbol in header_file["children"]:
name = symbol["name"]
entity = getattr(cppyy.gbl, name)
if getattr(entity, "__module__", None) == "cppyy.gbl":
setattr(entity, "__module__", pkg)
setattr(sys.modules[pkg], name, entity)
Because of this it is now possible for users of the library bornagain
to write
my_instance = bornagain.a_bornagain_symbol()
my_instance.do_sth()
instead of this
my_instance = cppyy.gbl.a_bornagain_symbol()
my_instance.do_sth()
Things which still would need to be done:
- API consistency: Cppyy respects the C++ structure including namespaces. There might be some symbols which were mapped to the global python package space using SWIG, while with Cppyy they are still inside a namespace.
- Lazy-Loading: Check in detail which header files need to be included to the python package? If header files in
api_files
depend on on other headers, these need to be available on the user system. - 3rd-party headers: Check what would be the best way of including ROOT,heinz,formfactor headers for the user system.
- Pythonization: In the current state special functions implemented in SWIGs .i files were ignored, these can be reimplemented in either C++ or Python. (See https://cppyy.readthedocs.io/en/latest/pythonizations.html)
- Import from Python: The BornAgain UI uses SWIG, investigate the best way to update this part
Building using cppyy:
pip install cppyy libclang
- Edit MakePythonWheel.cmake line 162 change the path to libclang.so to your python package path
cmake .. -DBA_PY_PACKAGE=On
make
bash var/mk_py_package.sh
pip install PyPackage/py310
Performance Testing(n=10;upper table:cppyy, lower table: swig):
Additionally here are some performance tests. Here we test the performance of every Examples/ff Python example and run each test 10 times for SWIG and for Cppyy. The upper table displays the results taken from testing with Cppyy. The lower table displays the result from testing with SWIG.
5182ee | Dodecahedron.py | PlatonicTetrahedron.py | Pyramid4.py | Pyramid2.py | Icosahedron.py | PlatonicOctahedron.py | sim_det_box.py | Sphere.py | Cone.py | TruncatedSphere.py | Box.py | SawtoothRippleBox.py | HemiEllipsoid.py | Spheroid.py | Prism3.py | HorizontalCylinder.py | TruncatedCube.py | Pyramid6.py | Bipyramid4.py | sim_demo_1quadrants.py | CantellatedCube.py | Pyramid3.py | sim_demo_4quadrants.py | Prism6.py | SasCosineRipple.py | Cylinder.py | EllipsoidalCylinder.py | TruncatedSpheroid.py |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
mean | 3.34276207099997 | 2.79965125049998 | 2.9494933856 | 3.2128743100001 | 3.42447132090001 | 2.9514600080001 | 1.94268009860002 | 2.86845716739999 | 3.52410304689988 | 3.02792742360007 | 2.66751606119997 | 2.75923403730003 | 3.60219025469996 | 2.6577893274 | 2.71643187529994 | 2.89672972909998 | 3.21509325319998 | 3.32063746380004 | 3.71075087849995 | 2.12205279120003 | 3.17704765809995 | 3.12622730379999 | 1.94553996479999 | 2.84665271119998 | 1.96892243560005 | 2.7138803368 | 2.7722023324 | 2.99571711659996 |
stddev | 0.183479613661811 | 0.0157522524407603 | 0.0754508130795409 | 0.0644149973903588 | 0.0421313359430174 | 0.0170348802772594 | 0.0759302817753841 | 0.0288650934989848 | 0.096161792682904 | 0.0254099561863507 | 0.0157398345254834 | 0.071215110665438 | 0.129192283547371 | 0.0181602897734817 | 0.0116492576666151 | 0.0112433320359328 | 0.0231683681251344 | 0.0524262838836981 | 0.721763731810366 | 0.259458725188499 | 0.0207627682444578 | 0.107578590917981 | 0.0585982695537285 | 0.09192908264215 | 0.0684311845397714 | 0.0619145027507019 | 0.0653505360339707 | 0.019881574249568 |
min | 3.25147535999986 | 2.77798475000009 | 2.89483780299997 | 3.10936834800032 | 3.38180723699998 | 2.93210847400042 | 1.89370528900008 | 2.82383491600012 | 3.4336364589999 | 2.98819764400014 | 2.64380353499973 | 2.68312021700012 | 3.41236439399972 | 2.63060792700026 | 2.69781963300011 | 2.87450098299996 | 3.17756714200004 | 3.27686823800013 | 3.41070553000009 | 1.93292018600005 | 3.13636459000008 | 3.04572813499999 | 1.88416838299963 | 2.72446736900019 | 1.89590959200041 | 2.65678512300019 | 2.66136586399989 | 2.9718713530001 |
median | 3.27634443750003 | 2.79822265200005 | 2.93523093649992 | 3.20551380699999 | 3.40516495249994 | 2.94852265000009 | 1.92023844200003 | 2.86518547749984 | 3.49389550599994 | 3.03417437550024 | 2.66610279349993 | 2.75110228999984 | 3.59259246850024 | 2.65648042199996 | 2.71892888050002 | 2.89783980750008 | 3.21604763999994 | 3.30581252399998 | 3.425347504 | 2.06241059550007 | 3.17461127999991 | 3.0667604570001 | 1.92938834350025 | 2.85115747750001 | 1.95446966949999 | 2.69905524299998 | 2.77945494299979 | 2.98969381200027 |
max | 3.85353895700018 | 2.82168345900027 | 3.15749878099996 | 3.30661556700034 | 3.49671947500019 | 2.97726379000005 | 2.1465252869998 | 2.9233807999999 | 3.77731765099998 | 3.05855282499988 | 2.69935877000034 | 2.89184001000012 | 3.83685331799961 | 2.68301155300014 | 2.73776733800014 | 2.91465403699976 | 3.25482642899988 | 3.46527290199992 | 5.70077428500008 | 2.80785408800011 | 3.21478926000009 | 3.30033949900007 | 2.07688465899992 | 3.00379815899987 | 2.0772680340001 | 2.87453830100003 | 2.86163482199981 | 3.02449532199989 |
Return | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | FAILURE | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | FAILURE | SUCCESS | SUCCESS | FAILURE | SUCCESS | FAILURE | SUCCESS | SUCCESS | SUCCESS |
5f3e18 | Dodecahedron.py | PlatonicTetrahedron.py | Pyramid4.py | Pyramid2.py | Icosahedron.py | PlatonicOctahedron.py | sim_det_box.py | Sphere.py | Cone.py | TruncatedSphere.py | Box.py | SawtoothRippleBox.py | HemiEllipsoid.py | Spheroid.py | Prism3.py | HorizontalCylinder.py | TruncatedCube.py | Pyramid6.py | Bipyramid4.py | sim_demo_1quadrants.py | CantellatedCube.py | Pyramid3.py | sim_demo_4quadrants.py | Prism6.py | SasCosineRipple.py | Cylinder.py | EllipsoidalCylinder.py | TruncatedSpheroid.py |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
mean | 1.33910755699999 | 0.88913060409991 | 0.97804873630007 | 1.18590325140003 | 1.43971411640009 | 0.984567696800013 | 0.215120034300026 | 0.897078821799914 | 1.50119466080005 | 1.19519759189993 | 0.976996539700031 | 0.805872025100007 | 1.62873919459998 | 0.7833775393 | 0.773625292600082 | 0.956954904500026 | 1.3206725606 | 1.34510699779989 | 1.50885546939999 | 0.229150938500015 | 1.21842722850001 | 1.14006286999993 | 0.219661601399866 | 0.837591796100105 | 0.257127258499986 | 0.789909253199994 | 0.760700090100045 | 1.08353628489999 |
stddev | 0.0657868978195177 | 0.0409571095476205 | 0.026486323085047 | 0.0226635140628365 | 0.0290509182583103 | 0.00466116685949051 | 0.00214356420992682 | 0.0197516007750506 | 0.0232197812898962 | 0.167820328095021 | 0.113021022629153 | 0.0438235491533582 | 0.107940913725801 | 0.0455375676308709 | 0.0221928215159933 | 0.0185721750980553 | 0.0736947907411737 | 0.0239421299534376 | 0.0735165791927896 | 0.00770748973247399 | 0.0244223547974128 | 0.0595726198787645 | 0.00570838150373798 | 0.0908829112436658 | 0.0258726909223686 | 0.0467979702431285 | 0.0951260752253945 | 0.0352262815394364 |
min | 1.29730209900026 | 0.83759829399969 | 0.944563796000239 | 1.14614947100017 | 1.4078203009999 | 0.976636436000263 | 0.212385473000268 | 0.886041432999718 | 1.462825694 | 1.05291545599994 | 0.771590670000023 | 0.757785057000092 | 1.49128805700002 | 0.702586600999894 | 0.754246425000019 | 0.9361728519998 | 1.23760222200008 | 1.32537343600006 | 1.45611311199991 | 0.2218253369997 | 1.18483498000023 | 1.07453590500018 | 0.214044098999693 | 0.760331080000015 | 0.228234089000125 | 0.737394703000064 | 0.689436440999998 | 1.03258319299994 |
median | 1.3085305269999 | 0.883955139499904 | 0.972828197500121 | 1.18542926349983 | 1.43261282200001 | 0.984181313000136 | 0.215173004000008 | 0.891574006499923 | 1.5101589840001 | 1.14589764849984 | 0.993287979000343 | 0.811826828999983 | 1.60946032799984 | 0.786317079500122 | 0.763643775500214 | 0.95951596000009 | 1.29645137800003 | 1.3375826864999 | 1.48231149599997 | 0.228345743500086 | 1.21988593899982 | 1.14492504550003 | 0.218358169999874 | 0.798540487000082 | 0.25178545599988 | 0.776457526500053 | 0.715087202999939 | 1.08081771550019 |
max | 1.46816353500026 | 0.94710312899997 | 1.03392899699975 | 1.22076283200022 | 1.4995120829999 | 0.993908022000142 | 0.218156477000321 | 0.952401965999798 | 1.52724493400001 | 1.53960662099962 | 1.116568981 | 0.883826251000301 | 1.86206379300029 | 0.841355298000053 | 0.817448339000293 | 0.999137444000098 | 1.46788753500005 | 1.40791563199991 | 1.70147421699994 | 0.249385637000159 | 1.26458005099994 | 1.24753483799987 | 0.233237967999685 | 1.01253183600011 | 0.305922982000084 | 0.864672011000039 | 0.976842465000118 | 1.15262895499973 |
Return | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | FAILURE | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | FAILURE | SUCCESS | SUCCESS | FAILURE | SUCCESS | FAILURE | SUCCESS | SUCCESS | SUCCESS |
Diff | -2.00365451399998 | -1.91052064640007 | -1.97144464929993 | -2.02697105860007 | -1.98475720449992 | -1.96689231120008 | -1.7275600643 | -1.97137834560008 | -2.02290838609983 | -1.83272983170014 | -1.69051952149994 | -1.95336201220002 | -1.97345106009998 | -1.8744117881 | -1.94280658269986 | -1.93977482459995 | -1.89442069259999 | -1.97553046600015 | -2.20189540909996 | -1.89290185270002 | -1.95862042959993 | -1.98616443380006 | -1.72587836340012 | -2.00906091509987 | -1.71179517710007 | -1.92397108360001 | -2.01150224229996 | -1.91218083169997 |
Conclusion: The overall execution time using cppyy increases by about 2 second. This may be preventable using precompiled header files instead of raw headers.
This can be confirmed by just timing (n=10) the following snippet:
import bornagain as ba
from bornagain import ba_plot as bp, deg, nm, std_samples, std_simulations
The average execution time here is: 2.281655096600025s for only importing and initializing the package
After optimizing the includes the average execution time of an example running with cppyy dropped to a difference of 0.5-1.2 seconds
ad4815 | Dodecahedron.py | PlatonicTetrahedron.py | Pyramid4.py | Pyramid2.py | Icosahedron.py | PlatonicOctahedron.py | sim_det_box.py | Sphere.py | Cone.py | TruncatedSphere.py | Box.py | SawtoothRippleBox.py | HemiEllipsoid.py | Spheroid.py | Prism3.py | HorizontalCylinder.py | TruncatedCube.py | Pyramid6.py | Bipyramid4.py | sim_demo_1quadrants.py | CantellatedCube.py | Pyramid3.py | sim_demo_4quadrants.py | Prism6.py | SasCosineRipple.py | Cylinder.py | EllipsoidalCylinder.py | TruncatedSpheroid.py |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
mean | 2.7432242713999586 | 2.2942829109997547 | 2.212913983400358 | 2.158841763399869 | 2.4482925276999596 | 2.0143637971003043 | 0.9436566752996441 | 1.9345536543003619 | 2.57817558650022 | 2.1418041516000814 | 1.7328445611998178 | 1.7251811526997698 | 2.483882754200022 | 1.7113314901003833 | 1.764314533900142 | 1.955152703400381 | 2.3516601445997365 | 2.53405819089985 | 2.542381649600247 | 0.958024651300002 | 2.2179335812999854 | 2.119817284700184 | 0.9846616708000511 | 1.8152433689998362 | 0.9621647766998649 | 1.7726254217997848 | 1.7386188380996828 | 2.074690428699796 |
stddev | 0.09772338081821835 | 0.1305187071699955 | 0.17208703616454193 | 0.008340797222271793 | 0.013240390893544557 | 0.017826647319886067 | 0.007026990349737903 | 0.07782495660926973 | 0.0846964704349635 | 0.07191854562455996 | 0.0252134057633042 | 0.020153887590855946 | 0.010444989540119435 | 0.012228664253190835 | 0.013059259899296266 | 0.019237531799044274 | 0.08988192323110519 | 0.123089812296123 | 0.07520776348905402 | 0.014701060431923178 | 0.010376916189618798 | 0.03382659107541675 | 0.05646214131987946 | 0.03251544118773943 | 0.011910564018835822 | 0.04081371683260542 | 0.02352384164071929 | 0.027802253212069354 |
min | 2.5925702109998383 | 2.181972809999934 | 1.9937848720001057 | 2.14497561799908 | 2.42678908400012 | 1.994971076001093 | 0.9363064449989906 | 1.8678805610015843 | 2.470006072000615 | 2.057979470999271 | 1.7123730429993884 | 1.7027911429995584 | 2.4733432730008644 | 1.6912163790002523 | 1.7487849649987766 | 1.936145146999479 | 2.250936939000894 | 2.375713180001185 | 2.4782671590000973 | 0.9424734610001906 | 2.199907707999955 | 2.0909683509999013 | 0.9451685809999617 | 1.766386830000556 | 0.9451708370015695 | 1.7181745469988527 | 1.7015240929995343 | 2.0356048710000323 |
median | 2.732213489500282 | 2.2323821694999424 | 2.1942466145001163 | 2.1616675394989215 | 2.4482395249997353 | 2.007381235000139 | 0.9427481414995782 | 1.8999355850000939 | 2.5516317279998475 | 2.1152195835002203 | 1.7242460809993645 | 1.7214851335002095 | 2.480652154999916 | 1.7115339760002826 | 1.7664641840001423 | 1.9485987620000742 | 2.340649215999292 | 2.537840595500711 | 2.508009766500436 | 0.9530009684995093 | 2.2212865319997945 | 2.113158977999774 | 0.971683494500212 | 1.8237247124989153 | 0.9672634009993999 | 1.7763549009996495 | 1.7381522329997097 | 2.0693590839991884 |
max | 2.86522587699983 | 2.5902267830006167 | 2.5045696170000156 | 2.1690352850000636 | 2.472951366999041 | 2.0409056409989716 | 0.9615123019993916 | 2.0847997090004355 | 2.7089876840000215 | 2.2573730849999265 | 1.7890444149998075 | 1.7659327490000578 | 2.506958127998587 | 1.7267154210003355 | 1.7852245430003677 | 1.997139844999765 | 2.519646131000627 | 2.785485782998876 | 2.716632544999811 | 0.9835127089991147 | 2.234965348001424 | 2.208626780000486 | 1.141757367999162 | 1.8551722379997955 | 0.9776715649986727 | 1.8388769660014077 | 1.772564135999346 | 2.1164182760003314 |
Return | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | FAILURE | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | SUCCESS | FAILURE | SUCCESS | SUCCESS | FAILURE | SUCCESS | FAILURE | SUCCESS | SUCCESS | SUCCESS |
- The examples failing still use an older python api which provides the module 'bornplot' instead of 'ba_plot'