Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[python-package] cannot import name '_LGBMBaseCrossValidator' from 'lightgbm.compat' #5631

Closed
JisongXie opened this issue Dec 13, 2022 · 4 comments · Fixed by #5632
Closed

[python-package] cannot import name '_LGBMBaseCrossValidator' from 'lightgbm.compat' #5631

JisongXie opened this issue Dec 13, 2022 · 4 comments · Fixed by #5632
Labels
bug

Comments

@JisongXie
Copy link

Description

After I build lightgbm gpu version, import lightgbm gets error

In [1]: import lightgbm as lgb
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[1], line 1
----> 1 import lightgbm as lgb

File /usr/local/LightGBM/python-package/lightgbm/__init__.py:10
      8 from .basic import Booster, Dataset, Sequence, register_logger
      9 from .callback import early_stopping, log_evaluation, record_evaluation, reset_parameter
---> 10 from .engine import CVBooster, cv, train
     12 try:
     13     from .sklearn import LGBMClassifier, LGBMModel, LGBMRanker, LGBMRegressor

File /usr/local/LightGBM/python-package/lightgbm/engine.py:15
     12 from . import callback
     13 from .basic import (Booster, Dataset, LightGBMError, _choose_param_value, _ConfigAliases, _InnerPredictor,
     14                     _LGBM_CustomObjectiveFunction, _log_warning)
---> 15 from .compat import SKLEARN_INSTALLED, _LGBMBaseCrossValidator, _LGBMGroupKFold, _LGBMStratifiedKFold
     17 _LGBM_CustomMetricFunction = Callable[
     18     [np.ndarray, Dataset],
     19     Tuple[str, float, bool]
     20 ]
     22 _LGBM_PreprocFunction = Callable[
     23     [Dataset, Dataset, Dict[str, Any]],
     24     Tuple[Dataset, Dataset, Dict[str, Any]]
     25 ]

ImportError: cannot import name '_LGBMBaseCrossValidator' from 'lightgbm.compat' (/usr/local/LightGBM/python-package/lightgbm/compat.py)

Reproducible example

I build lightgbm gpu version in my dockerfile with codes as follow:

FROM nvidia/cuda:11.2.0-cudnn8-devel-ubuntu18.04

RUN apt-get update && \
    apt-get install -y wget

RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
    bash ~/miniconda.sh -b -p /root/miniconda

RUN /root/miniconda/bin/conda init bash

RUN apt-get update && \
    apt-get install --no-install-recommends -y git build-essential libboost-dev libboost-system-dev libboost-filesystem-dev && \
    apt-get install --no-install-recommends -y ocl-icd-libopencl1 ocl-icd-opencl-dev

RUN wget https://github.com/Kitware/CMake/releases/download/v3.22.1/cmake-3.22.1-linux-x86_64.sh -O cmake.sh
RUN sh cmake.sh --prefix=/usr/local/ --exclude-subdir

RUN cd /usr/local && \
    git clone --recursive https://github.com/microsoft/LightGBM

RUN cd /usr/local/LightGBM && \
    mkdir build && \
    cd build && \
    /usr/local/bin/cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ .. && \
    make -j8

RUN cd ../python-package/ && \
    /root/miniconda/bin/python setup.py install --precompile

It seems that make is succeessful.

-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - found
-- Found OpenCL: /usr/local/cuda/lib64/libOpenCL.so (found version "1.2")
-- OpenCL include directory: /usr/local/cuda/include
-- Found Boost: /usr/include (found suitable version "1.65.1", minimum required is "1.56.0") found components: filesystem system
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/local/LightGBM/build


[  1%] Building CXX object CMakeFiles/lightgbm_capi_objs.dir/src/c_api.cpp.o
[  3%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/boosting.cpp.o
[  5%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_model_text.cpp.o
[  7%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt.cpp.o
[  9%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/cuda/cuda_score_updater.cpp.o
[ 11%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_metadata.cpp.o
[ 13%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/config.cpp.o
[ 17%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o
[ 17%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_tree.cpp.o
[ 21%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/bin.cpp.o
[ 21%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_prediction.cpp.o
[ 23%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_row_data.cpp.o
[ 25%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/dataset.cpp.o
[ 26%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/config_auto.cpp.o
[ 28%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/cuda/cuda_utils.cpp.o
[ 30%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_column_data.cpp.o
[ 32%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/file_io.cpp.o
[ 34%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/dataset_loader.cpp.o
[ 36%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/json11.cpp.o
[ 38%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/metadata.cpp.o
[ 42%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/train_share_states.cpp.o
[ 42%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/parser.cpp.o
[ 44%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/tree.cpp.o
[ 46%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_regression_metric.cpp.o
[ 48%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/dcg_calculator.cpp.o
[ 50%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/metric.cpp.o
[ 51%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linker_topo.cpp.o
[ 53%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linkers_mpi.cpp.o
[ 55%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linkers_socket.cpp.o
[ 57%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/network.cpp.o
[ 59%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_binary_objective.cpp.o
[ 61%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_multiclass_objective.cpp.o
[ 63%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_rank_objective.cpp.o
[ 65%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_regression_objective.cpp.o
[ 67%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/objective_function.cpp.o
[ 69%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_best_split_finder.cpp.o
[ 71%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_data_partition.cpp.o
[ 73%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_histogram_constructor.cpp.o
[ 75%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_leaf_splits.cpp.o
[ 76%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_single_gpu_tree_learner.cpp.o
[ 78%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda_tree_learner.cpp.o
[ 80%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 82%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 84%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 86%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/linear_tree_learner.cpp.o
[ 88%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/serial_tree_learner.cpp.o
[ 90%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/tree_learner.cpp.o
[ 92%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 92%] Built target lightgbm_capi_objs
[ 92%] Built target lightgbm_objs
[ 94%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
[ 96%] Linking CXX shared library ../lib_lightgbm.so
[ 98%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
[ 98%] Built target _lightgbm
[100%] Linking CXX executable ../lightgbm
[100%] Built target lightgbm

However, python setup.py install outputs some warnings,

running install
/root/miniconda/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running build
running build_py
creating build
creating build/lib
creating build/lib/lightgbm
copying lightgbm/plotting.py -> build/lib/lightgbm
copying lightgbm/dask.py -> build/lib/lightgbm
copying lightgbm/basic.py -> build/lib/lightgbm
copying lightgbm/__init__.py -> build/lib/lightgbm
copying lightgbm/libpath.py -> build/lib/lightgbm
copying lightgbm/engine.py -> build/lib/lightgbm
copying lightgbm/compat.py -> build/lib/lightgbm
copying lightgbm/sklearn.py -> build/lib/lightgbm
copying lightgbm/callback.py -> build/lib/lightgbm
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
warning: no files found matching 'LICENSE'
warning: no files found matching '*.txt'
warning: no files found matching '*.so' under directory 'lightgbm'
warning: no files found matching 'compile/CMakeLists.txt'
warning: no files found matching 'compile/cmake/IntegratedOpenCL.cmake'
warning: no files found matching '*.so' under directory 'compile'
warning: no files found matching '*.dll' under directory 'compile/Release'
warning: no files found matching 'compile/external_libs/compute/CMakeLists.txt'
warning: no files found matching '*' under directory 'compile/external_libs/compute/cmake'
warning: no files found matching '*' under directory 'compile/external_libs/compute/include'
warning: no files found matching '*' under directory 'compile/external_libs/compute/meta'
warning: no files found matching 'compile/external_libs/eigen/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Cholesky'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Core'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Dense'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Eigenvalues'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Geometry'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Householder'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Jacobi'
warning: no files found matching 'compile/external_libs/eigen/Eigen/LU'
warning: no files found matching 'compile/external_libs/eigen/Eigen/QR'
warning: no files found matching 'compile/external_libs/eigen/Eigen/SVD'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Cholesky'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Core'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Eigenvalues'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Geometry'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Householder'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Jacobi'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/LU'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/misc'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/plugins'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/QR'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/SVD'
warning: no files found matching 'compile/external_libs/fast_double_parser/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE'
warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE.BSL'
warning: no files found matching '*' under directory 'compile/external_libs/fast_double_parser/include'
warning: no files found matching 'compile/external_libs/fmt/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/fmt/LICENSE.rst'
warning: no files found matching '*' under directory 'compile/external_libs/fmt/include'
warning: no files found matching '*' under directory 'compile/include'
warning: no files found matching '*' under directory 'compile/src'
warning: no files found matching 'LightGBM.sln' under directory 'compile/windows'
warning: no files found matching 'LightGBM.vcxproj' under directory 'compile/windows'
warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
warning: no previously-included files found matching 'compile/external_libs/compute/.git'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
copying lightgbm/VERSION.txt -> build/lib/lightgbm
running install_lib
creating /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/plotting.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/dask.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/basic.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/__init__.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/libpath.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/VERSION.txt -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/engine.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/compat.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/sklearn.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/callback.py -> /root/miniconda/lib/python3.9/site-packages/lightgbm
INFO:LightGBM:Installing lib_lightgbm from: ['/usr/local/LightGBM/lib_lightgbm.so']
copying /usr/local/LightGBM/lib_lightgbm.so -> /root/miniconda/lib/python3.9/site-packages/lightgbm
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/plotting.py to plotting.cpython-39.pyc
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/dask.py to dask.cpython-39.pyc
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/basic.py to basic.cpython-39.pyc
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/__init__.py to __init__.cpython-39.pyc
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/libpath.py to libpath.cpython-39.pyc
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/engine.py to engine.cpython-39.pyc
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/compat.py to compat.cpython-39.pyc
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/sklearn.py to sklearn.cpython-39.pyc
byte-compiling /root/miniconda/lib/python3.9/site-packages/lightgbm/callback.py to callback.cpython-39.pyc
running install_egg_info
Copying lightgbm.egg-info to /root/miniconda/lib/python3.9/site-packages/lightgbm-3.3.3.99-py3.9.egg-info
running install_scripts

And then I try import lightgbm, it occurs error,

In [1]: import lightgbm as lgb
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[1], line 1
----> 1 import lightgbm as lgb

File /usr/local/LightGBM/python-package/lightgbm/__init__.py:10
      8 from .basic import Booster, Dataset, Sequence, register_logger
      9 from .callback import early_stopping, log_evaluation, record_evaluation, reset_parameter
---> 10 from .engine import CVBooster, cv, train
     12 try:
     13     from .sklearn import LGBMClassifier, LGBMModel, LGBMRanker, LGBMRegressor

File /usr/local/LightGBM/python-package/lightgbm/engine.py:15
     12 from . import callback
     13 from .basic import (Booster, Dataset, LightGBMError, _choose_param_value, _ConfigAliases, _InnerPredictor,
     14                     _LGBM_CustomObjectiveFunction, _log_warning)
---> 15 from .compat import SKLEARN_INSTALLED, _LGBMBaseCrossValidator, _LGBMGroupKFold, _LGBMStratifiedKFold
     17 _LGBM_CustomMetricFunction = Callable[
     18     [np.ndarray, Dataset],
     19     Tuple[str, float, bool]
     20 ]
     22 _LGBM_PreprocFunction = Callable[
     23     [Dataset, Dataset, Dict[str, Any]],
     24     Tuple[Dataset, Dataset, Dict[str, Any]]
     25 ]

ImportError: cannot import name '_LGBMBaseCrossValidator' from 'lightgbm.compat' (/usr/local/LightGBM/python-package/lightgbm/compat.py)

Additional Comments

I try apt-get install cmake which cmake's version is 3.10.x.
And the newest version of cmake which is 3.25.1.
But both are failing.
@jameslamb
Copy link
Collaborator

Thanks very much for the excellent report!

I believe I see the issue. Can you please try installing scikit-learn (no need to re-install lightgbm)?

conda install \
    --yes \
    -c conda-forge \
    scikit-learn

In this project compat.py is used to handle optional dependencies possibly not being available.

Looks like #5271 added an import of sklearn.model_selection.BaseCrossValidator but not a corresponding condition to account for the case where scikit-learn isn't installed! I'll put up a fix for that.

@jameslamb jameslamb changed the title lightgbm build [python-package] cannot import name '_LGBMBaseCrossValidator' from 'lightgbm.compat' Dec 13, 2022
@jameslamb jameslamb added the bug label Dec 13, 2022
@jameslamb jameslamb mentioned this issue Dec 13, 2022
Merged
@JisongXie
Copy link
Author

Well, I follow this tutorial Install Python Interface (optional), however I wrongly think that I have installed these packages. Because I have installed some other packages like torch, xgboost and so on, which is omitted in dockerfile above.

sudo apt-get -y install python-pip
sudo -H pip install setuptools numpy scipy scikit-learn -U
cd python-package/
sudo python setup.py install --precompile
cd ..

After I install scikit-learn as you suggest, it works. Thanks @jameslamb !
I should have checked whether these packages are installed already.

@jameslamb jameslamb reopened this Dec 13, 2022
@jameslamb
Copy link
Collaborator

Thanks for that @JisongXie . But I'm re-opening this. It is a bug...scikit-learn shouldn't be required to use lightgbm.

I've opened #5632 to address it, and we'll close this once that's merged.

Thanks for helping us catch this bug!

jameslamb added a commit that referenced this issue Dec 29, 2022
@jameslamb
[ci] [python-package] fix missing import, test that lightgbm can be i…
7f4dbc8 
…mported with only required dependencies (fixes #5631) (#5632)
@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 19, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
2 participants
@jameslamb @JisongXie