mirrors/llvm-project
0
0
Fork 0
mirror of https://github.com/llvm/llvm-project.git synced 2025-04-17 01:06:33 +00:00
Code Issues Projects Releases Wiki Activity
llvm-project/mlir/cmake/modules/MLIRDetectPythonEnv.cmake

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

127 lines
5.9 KiB
CMake
Raw Permalink Normal View History

[mlir][Python] Support finding pybind11 from the python environment. * Makes `pip install pybind11` do the right thing with no further config. * Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable. * Also adds the version spec to find_package so it will skip older versions that may be lying around. * Tested the full matrix of old system install, no system install, pip install and no pip install. Differential Revision: https://reviews.llvm.org/D91903
2020-11-20 17:57:46 -08:00
# Macros and functions related to detecting details of the Python environment.
[mlir][python] Add a warning for cmake version < 3.15. As discussed on discord, we have never actually been able to build with the project-wide published min version of 3.14.3. The buildbot that tests the Python configuration is currently pinned to 3.19.1, and there are a number of non-version/policy controlled features that Python building relies on that makes it unreliable with older versions. Some of the issues are pretty fundamental and I don't know how to do them on the older version. I think that, as an optional feature, at least advertising the PSA as in this patch is a good middle ground until the next project-wide CMake version bump. Also moves setup logic to a macro so that everyone can use it.
2021-10-12 19:32:48 -07:00
# Finds and configures python packages needed to build MLIR Python bindings.
macro(mlir_configure_python_dev_packages)
[mlir] Add option to disable MLIR Python dev package configuration. (#117934) Adds a CMake option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES which gates doing package discovery and configuration for Python dev packages by MLIR (this was made opt-out to preserve compatibility with find_package(MLIR) based uses which do not set the standard options). The default Python setup that MLIR does has been a problem for super-projects that include LLVM for a long time because it forces a very specific package discovery mechanism that is not uniform in all uses. When reviewing #117922, I noted that this would effectively be a break the world event for downstreams, forcing them to adapt their nanobind dep to the exact way that MLIR does it. Adding the option to just wholesale skip the built-in configuration heuristics at least gives us a mechanism to tell downstreams to migrate to, giving them complete control and not requiring packaging workarounds. This seemed a better option than (once again) creating a situation where downstreams could not integrate the dep change without doing tricky infra upgrades, and it removes the burden from the author of that patch from needing to think about how this affects super-projects that include MLIR (i.e. they can just be told to do it themselves as needed vs being in a wedged state and unable to upgrade).
2024-11-27 17:11:32 -08:00
if(NOT MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES)
if(MLIR_DETECT_PYTHON_ENV_PRIME_SEARCH)
# Prime the search for python to see if there is a full development
# package. This seems to work around cmake bugs searching only for
# Development.Module in some environments. However, in other environments
# it may interfere with the subsequent search for Development.Module.
find_package(Python3 ${LLVM_MINIMUM_PYTHON_VERSION}
COMPONENTS Interpreter Development)
endif()
[mlir] Update Python CMake version requirement. The minimum required version is now 3.19 due to the usage of some more recent features. Update the version check and error message accordingly. Also remove some logic that behaved differently before 3.18, since we can assume we are now on version 3.19+. Reviewed By: stella.stamenova Differential Revision: https://reviews.llvm.org/D130171
2022-07-20 08:21:36 -06:00
[mlir] Add option to disable MLIR Python dev package configuration. (#117934) Adds a CMake option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES which gates doing package discovery and configuration for Python dev packages by MLIR (this was made opt-out to preserve compatibility with find_package(MLIR) based uses which do not set the standard options). The default Python setup that MLIR does has been a problem for super-projects that include LLVM for a long time because it forces a very specific package discovery mechanism that is not uniform in all uses. When reviewing #117922, I noted that this would effectively be a break the world event for downstreams, forcing them to adapt their nanobind dep to the exact way that MLIR does it. Adding the option to just wholesale skip the built-in configuration heuristics at least gives us a mechanism to tell downstreams to migrate to, giving them complete control and not requiring packaging workarounds. This seemed a better option than (once again) creating a situation where downstreams could not integrate the dep change without doing tricky infra upgrades, and it removes the burden from the author of that patch from needing to think about how this affects super-projects that include MLIR (i.e. they can just be told to do it themselves as needed vs being in a wedged state and unable to upgrade).
2024-11-27 17:11:32 -08:00
# After CMake 3.18, we are able to limit the scope of the search to just
# Development.Module. Searching for Development will fail in situations where
# the Python libraries are not available. When possible, limit to just
# Development.Module.
# See https://pybind11.readthedocs.io/en/stable/compiling.html#findpython-mode
set(_python_development_component Development.Module)
[mlir] Update Python CMake version requirement. The minimum required version is now 3.19 due to the usage of some more recent features. Update the version check and error message accordingly. Also remove some logic that behaved differently before 3.18, since we can assume we are now on version 3.19+. Reviewed By: stella.stamenova Differential Revision: https://reviews.llvm.org/D130171
2022-07-20 08:21:36 -06:00
[mlir] Add option to disable MLIR Python dev package configuration. (#117934) Adds a CMake option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES which gates doing package discovery and configuration for Python dev packages by MLIR (this was made opt-out to preserve compatibility with find_package(MLIR) based uses which do not set the standard options). The default Python setup that MLIR does has been a problem for super-projects that include LLVM for a long time because it forces a very specific package discovery mechanism that is not uniform in all uses. When reviewing #117922, I noted that this would effectively be a break the world event for downstreams, forcing them to adapt their nanobind dep to the exact way that MLIR does it. Adding the option to just wholesale skip the built-in configuration heuristics at least gives us a mechanism to tell downstreams to migrate to, giving them complete control and not requiring packaging workarounds. This seemed a better option than (once again) creating a situation where downstreams could not integrate the dep change without doing tricky infra upgrades, and it removes the burden from the author of that patch from needing to think about how this affects super-projects that include MLIR (i.e. they can just be told to do it themselves as needed vs being in a wedged state and unable to upgrade).
2024-11-27 17:11:32 -08:00
find_package(Python3 ${LLVM_MINIMUM_PYTHON_VERSION}
COMPONENTS Interpreter ${_python_development_component} REQUIRED)
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 16:37:43 -05:00
Fix for issue #126162: warn for inconsistent Python search options (#126329) Fixes #126162 I check locally that it works without warning for: - neither options are defined - both defined to the same value And I checked that it warns if: - only one is defined - they defined to different values
2025-02-18 11:32:26 -08:00
# We look for both Python3 and Python, the search algorithm should be
# consistent, otherwise disastrous result is almost guaranteed.
# Warn if the policies for treating virtual environment are not defined
# consistently.
# For more details check issue #126162.
if(((DEFINED Python_FIND_VIRTUALENV) AND (NOT DEFINED Python3_FIND_VIRTUALENV)) OR
((NOT DEFINED Python_FIND_VIRTUALENV) AND (DEFINED Python3_FIND_VIRTUALENV)))
message(WARNING "Only one of Python3_FIND_VIRTUALENV and Python_FIND_VIRTUALENV variables is defined. "
"Make sure that both variables are defined and have the same value.")
elseif((DEFINED Python_FIND_VIRTUALENV) AND (DEFINED Python3_FIND_VIRTUALENV) AND
(NOT Python_FIND_VIRTUALENV STREQUAL Python3_FIND_VIRTUALENV))
message(WARNING "Python3_FIND_VIRTUALENV and Python_FIND_VIRTUALENV are defined differently. "
"Make sure that the variables have the same values.")
endif()
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 16:37:43 -05:00
# It's a little silly to detect Python a second time, but nanobind's cmake
# code looks for Python_ not Python3_.
find_package(Python ${LLVM_MINIMUM_PYTHON_VERSION}
COMPONENTS Interpreter ${_python_development_component} REQUIRED)
[mlir] Add option to disable MLIR Python dev package configuration. (#117934) Adds a CMake option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES which gates doing package discovery and configuration for Python dev packages by MLIR (this was made opt-out to preserve compatibility with find_package(MLIR) based uses which do not set the standard options). The default Python setup that MLIR does has been a problem for super-projects that include LLVM for a long time because it forces a very specific package discovery mechanism that is not uniform in all uses. When reviewing #117922, I noted that this would effectively be a break the world event for downstreams, forcing them to adapt their nanobind dep to the exact way that MLIR does it. Adding the option to just wholesale skip the built-in configuration heuristics at least gives us a mechanism to tell downstreams to migrate to, giving them complete control and not requiring packaging workarounds. This seemed a better option than (once again) creating a situation where downstreams could not integrate the dep change without doing tricky infra upgrades, and it removes the burden from the author of that patch from needing to think about how this affects super-projects that include MLIR (i.e. they can just be told to do it themselves as needed vs being in a wedged state and unable to upgrade).
2024-11-27 17:11:32 -08:00
unset(_python_development_component)
message(STATUS "Found python include dirs: ${Python3_INCLUDE_DIRS}")
message(STATUS "Found python libraries: ${Python3_LIBRARIES}")
message(STATUS "Found numpy v${Python3_NumPy_VERSION}: ${Python3_NumPy_INCLUDE_DIRS}")
mlir_detect_pybind11_install()
find_package(pybind11 2.10 CONFIG REQUIRED)
message(STATUS "Found pybind11 v${pybind11_VERSION}: ${pybind11_INCLUDE_DIR}")
message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
"suffix = '${PYTHON_MODULE_SUFFIX}', "
"extension = '${PYTHON_MODULE_EXTENSION}")
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 16:37:43 -05:00
mlir_detect_nanobind_install()
[mlir python] Port Python core code to nanobind. (#120473) Relands #118583, with a fix for Python 3.8 compatibility. It was not possible to set the buffer protocol accessers via slots in Python 3.8. Why? https://nanobind.readthedocs.io/en/latest/why.html says it better than I can, but my primary motivation for this change is to improve MLIR IR construction time from JAX. For a complicated Google-internal LLM model in JAX, this change improves the MLIR lowering time by around 5s (out of around 30s), which is a significant speedup for simply switching binding frameworks. To a large extent, this is a mechanical change, for instance changing `pybind11::` to `nanobind::`. Notes: * this PR needs Nanobind 2.4.0, because it needs a bug fix (https://github.com/wjakob/nanobind/pull/806) that landed in that release. * this PR does not port the in-tree dialect extension modules. They can be ported in a future PR. * I removed the py::sibling() annotations from def_static and def_class in `PybindAdapters.h`. These ask pybind11 to try to form an overload with an existing method, but it's not possible to form mixed pybind11/nanobind overloads this ways and the parent class is now defined in nanobind. Better solutions may be possible here. * nanobind does not contain an exact equivalent of pybind11's buffer protocol support. It was not hard to add a nanobind implementation of a similar API. * nanobind is pickier about casting to std::vector<bool>, expecting that the input is a sequence of bool types, not truthy values. In a couple of places I added code to support truthy values during casting. * nanobind distinguishes bytes (`nb::bytes`) from strings (e.g., `std::string`). This required nb::bytes overloads in a few places.
2024-12-18 21:55:42 -05:00
find_package(nanobind 2.4 CONFIG REQUIRED)
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 16:37:43 -05:00
message(STATUS "Found nanobind v${nanobind_VERSION}: ${nanobind_INCLUDE_DIR}")
message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
"suffix = '${PYTHON_MODULE_SUFFIX}', "
"extension = '${PYTHON_MODULE_EXTENSION}")
[mlir] Add option to disable MLIR Python dev package configuration. (#117934) Adds a CMake option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES which gates doing package discovery and configuration for Python dev packages by MLIR (this was made opt-out to preserve compatibility with find_package(MLIR) based uses which do not set the standard options). The default Python setup that MLIR does has been a problem for super-projects that include LLVM for a long time because it forces a very specific package discovery mechanism that is not uniform in all uses. When reviewing #117922, I noted that this would effectively be a break the world event for downstreams, forcing them to adapt their nanobind dep to the exact way that MLIR does it. Adding the option to just wholesale skip the built-in configuration heuristics at least gives us a mechanism to tell downstreams to migrate to, giving them complete control and not requiring packaging workarounds. This seemed a better option than (once again) creating a situation where downstreams could not integrate the dep change without doing tricky infra upgrades, and it removes the burden from the author of that patch from needing to think about how this affects super-projects that include MLIR (i.e. they can just be told to do it themselves as needed vs being in a wedged state and unable to upgrade).
2024-11-27 17:11:32 -08:00
endif()
[mlir][python] Add a warning for cmake version < 3.15. As discussed on discord, we have never actually been able to build with the project-wide published min version of 3.14.3. The buildbot that tests the Python configuration is currently pinned to 3.19.1, and there are a number of non-version/policy controlled features that Python building relies on that makes it unreliable with older versions. Some of the issues are pretty fundamental and I don't know how to do them on the older version. I think that, as an optional feature, at least advertising the PSA as in this patch is a good middle ground until the next project-wide CMake version bump. Also moves setup logic to a macro so that everyone can use it.
2021-10-12 19:32:48 -07:00
endmacro()
[mlir][Python] Support finding pybind11 from the python environment. * Makes `pip install pybind11` do the right thing with no further config. * Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable. * Also adds the version spec to find_package so it will skip older versions that may be lying around. * Tested the full matrix of old system install, no system install, pip install and no pip install. Differential Revision: https://reviews.llvm.org/D91903
2020-11-20 17:57:46 -08:00
# Detects a pybind11 package installed in the current python environment
# and sets variables to allow it to be found. This allows pybind11 to be
# installed via pip, which typically yields a much more recent version than
# the OS install, which will be available otherwise.
function(mlir_detect_pybind11_install)
if(pybind11_DIR)
message(STATUS "Using explicit pybind11 cmake directory: ${pybind11_DIR} (-Dpybind11_DIR to change)")
else()
[mlir] use STATUS instead of CHECK_* in MLIRDetectPythonEnv.cmake CHECK_* directives for message() where added in Cmake 3.17, LLVM requires 3.14 as minimum so they may not be intepreted correctly and just print "CHECK_*" into the message stream. Replace them with STATUS. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D91959
2020-11-23 13:58:03 +01:00
message(STATUS "Checking for pybind11 in python path...")
[mlir][Python] Support finding pybind11 from the python environment. * Makes `pip install pybind11` do the right thing with no further config. * Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable. * Also adds the version spec to find_package so it will skip older versions that may be lying around. * Tested the full matrix of old system install, no system install, pip install and no pip install. Differential Revision: https://reviews.llvm.org/D91903
2020-11-20 17:57:46 -08:00
execute_process(
[mlir][Python] Fix the last remaining instance of PYTHON_EXECUTABLE. * Was causing auto-detect of pybind11 to fail on clean configure. Differential Revision: https://reviews.llvm.org/D92043
2020-11-24 17:50:18 +00:00
COMMAND "${Python3_EXECUTABLE}"
[mlir][Python] Support finding pybind11 from the python environment. * Makes `pip install pybind11` do the right thing with no further config. * Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable. * Also adds the version spec to find_package so it will skip older versions that may be lying around. * Tested the full matrix of old system install, no system install, pip install and no pip install. Differential Revision: https://reviews.llvm.org/D91903
2020-11-20 17:57:46 -08:00
-c "import pybind11;print(pybind11.get_cmake_dir(), end='')"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
RESULT_VARIABLE STATUS
OUTPUT_VARIABLE PACKAGE_DIR
ERROR_QUIET)
if(NOT STATUS EQUAL "0")
[mlir] use STATUS instead of CHECK_* in MLIRDetectPythonEnv.cmake CHECK_* directives for message() where added in Cmake 3.17, LLVM requires 3.14 as minimum so they may not be intepreted correctly and just print "CHECK_*" into the message stream. Replace them with STATUS. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D91959
2020-11-23 13:58:03 +01:00
message(STATUS "not found (install via 'pip install pybind11' or set pybind11_DIR)")
[mlir][Python] Support finding pybind11 from the python environment. * Makes `pip install pybind11` do the right thing with no further config. * Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable. * Also adds the version spec to find_package so it will skip older versions that may be lying around. * Tested the full matrix of old system install, no system install, pip install and no pip install. Differential Revision: https://reviews.llvm.org/D91903
2020-11-20 17:57:46 -08:00
return()
endif()
[mlir] use STATUS instead of CHECK_* in MLIRDetectPythonEnv.cmake CHECK_* directives for message() where added in Cmake 3.17, LLVM requires 3.14 as minimum so they may not be intepreted correctly and just print "CHECK_*" into the message stream. Replace them with STATUS. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D91959
2020-11-23 13:58:03 +01:00
message(STATUS "found (${PACKAGE_DIR})")
[mlir][Python] Support finding pybind11 from the python environment. * Makes `pip install pybind11` do the right thing with no further config. * Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable. * Also adds the version spec to find_package so it will skip older versions that may be lying around. * Tested the full matrix of old system install, no system install, pip install and no pip install. Differential Revision: https://reviews.llvm.org/D91903
2020-11-20 17:57:46 -08:00
set(pybind11_DIR "${PACKAGE_DIR}" PARENT_SCOPE)
endif()
endfunction()
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 16:37:43 -05:00
# Detects a nanobind package installed in the current python environment
# and sets variables to allow it to be found. This allows nanobind to be
# installed via pip, which typically yields a much more recent version than
# the OS install, which will be available otherwise.
function(mlir_detect_nanobind_install)
if(nanobind_DIR)
message(STATUS "Using explicit nanobind cmake directory: ${nanobind_DIR} (-Dnanobind_DIR to change)")
else()
message(STATUS "Checking for nanobind in python path...")
execute_process(
COMMAND "${Python3_EXECUTABLE}"
-c "import nanobind;print(nanobind.cmake_dir(), end='')"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
RESULT_VARIABLE STATUS
OUTPUT_VARIABLE PACKAGE_DIR
ERROR_QUIET)
if(NOT STATUS EQUAL "0")
message(STATUS "not found (install via 'pip install nanobind' or set nanobind_DIR)")
return()
endif()
message(STATUS "found (${PACKAGE_DIR})")
set(nanobind_DIR "${PACKAGE_DIR}" PARENT_SCOPE)
[mlir python] Port in-tree dialects to nanobind. (#119924) This is a companion to #118583, although it can be landed independently because since #117922 dialects do not have to use the same Python binding framework as the Python core code. This PR ports all of the in-tree dialect and pass extensions to nanobind, with the exception of those that remain for testing pybind11 support. This PR also: * removes CollectDiagnosticsToStringScope from NanobindAdaptors.h. This was overlooked in a previous PR and it is duplicated in Diagnostics.h. --------- Co-authored-by: Jacques Pienaar <jpienaar@google.com>
2024-12-20 23:32:32 -05:00
execute_process(
COMMAND "${Python3_EXECUTABLE}"
-c "import nanobind;print(nanobind.include_dir(), end='')"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
RESULT_VARIABLE STATUS
OUTPUT_VARIABLE PACKAGE_DIR
ERROR_QUIET)
if(NOT STATUS EQUAL "0")
message(STATUS "not found (install via 'pip install nanobind' or set nanobind_DIR)")
return()
endif()
set(nanobind_INCLUDE_DIR "${PACKAGE_DIR}" PARENT_SCOPE)
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 16:37:43 -05:00
endif()
endfunction()
Reference in New Issue Copy Permalink