Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ask for Help in Solving Bazel Build Problem #471

Closed
NicolausLiu opened this issue Feb 19, 2025 · 16 comments
Closed

Ask for Help in Solving Bazel Build Problem #471

NicolausLiu opened this issue Feb 19, 2025 · 16 comments
Assignees

Comments

@NicolausLiu
Copy link

I'm not sure if anyone else has encountered the same issue as me. I'm trying to build the YACL library in a Docker container on CentOS, following the "Getting Started" documentation and installing all the required libraries. I'm using Bazelisk for this process. However, I keep running into the following issue with bazel build. I've managed to resolve the dependencies for @bazel_skylib, @rules_jni, and @YacL, but the problem persists. I don't think the YACL library should be this difficult to configure, but I'm stuck with these initial issues and can't seem to find a solution. I'm reaching out for help here.

ERROR: Skipping '//yacl/...': error loading package under directory 'yacl': error loading package 'yacl/yacl/base': Every .bzl file must have a corresponding package, but '//bazel:yacl.bzl' does not have one. Please create a BUILD file in the same or any parent directory. Note that this BUILD file does not need to do anything except exist.
ERROR: error loading package under directory 'yacl': error loading package 'yacl/yacl/base': Every .bzl file must have a corresponding package, but '//bazel:yacl.bzl' does not have one. Please create a BUILD file in the same or any parent directory. Note that this BUILD file does not need to do anything except exist.

@tongke6
Copy link
Collaborator

tongke6 commented Feb 19, 2025

@NicolausLiu Could you provide the commit ID of YACL, bazel build command, and the container image information.

@Candicepan Candicepan assigned changtong9 and unassigned changtong9 Feb 19, 2025
@NicolausLiu
Copy link
Author

The container image information: Centos:latest, sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6.

I just git clone the yacl code without noticing the commit ID of YACL. The bazel build command is "bazel build //yacl/...".
The output of "git log":
commit cd5d377 (HEAD -> main, origin/main, origin/HEAD)
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date: Wed Feb 12 10:56:36 2025 +0800

chore(deps): update dependency rules_foreign_cc to v0.14.0 (#437)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

commit 2f5f453
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date: Tue Feb 11 10:07:11 2025 +0800

chore(deps): update dependency apple_support to v1.18.0 (#464)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

commit 109d334
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date: Tue Feb 11 10:03:43 2025 +0800

chore(deps): update dependency platforms to v0.0.11 (#430)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

commit a493712
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date: Sat Jan 25 09:32:58 2025 +0800

chore(deps): update dependency bazel_features to v1.24.0 (#431)

commit d06a8de
Author: tongke [email protected]
Date: Thu Jan 23 22:09:46 2025 +0800

Fix .bcr templates (#461)

commit b07fcac
Author: Maths [email protected]
Date: Tue Jan 14 16:01:22 2025 +0800

Fix oprf.h (#459)

* Update native_factory.cc

* Update native_factory.cc

* Update native_factory.cc

* Update aead.h

* Update oprf.h

commit f724366 (tag: 0.4.5b10-nightly-20250110)
Author: tongke [email protected]
Date: Fri Jan 10 15:15:20 2025 +0800

Bump version to 0.4.5b10-nightly-20250110 (#458)

commit dad0433
Author: tongke [email protected]
Date: Fri Jan 10 15:10:35 2025 +0800

Add .bcr files to automate publishing yacl rulesets to bazel registry (#457)

commit 25db0bf (tag: 0.4.5b10-nightly-20241224)
Author: tongke6 [email protected]
Date: Tue Dec 24 18:46:45 2024 +0800

Add update_version.py and Bump version to 0.4.5b10-nightly-20241224 (#455)

* tweak MODULE.bazel file

* Add update_version.py and Bump version to 0.4.5b10_nightly_20241224

* fix invalid version error

commit 3fd1cb6
Author: shaojian-ant [email protected]
Date: Thu Dec 19 16:01:58 2024 +0800

repo-sync-2024-12-19T15:23:27+0800 (#454)

commit b91d7e6
Author: tongke6 [email protected]
Date: Wed Dec 18 20:32:25 2024 +0800

Fix upstream app psi build failures on macos m1 (#453)

* Fix upstream app psi build failures on macos m1

* remove version field

commit 871832a
Author: tongke6 [email protected]
Date: Thu Dec 12 19:31:27 2024 +0800

Fix brpc randomly print stack issue (#447)

* see https://github.com/apache/brpc/commit/4035c8020e2ec01e2dcbda8a9e4eaf3feb492c01

commit 3df391a
Author: tongke6 [email protected]
Date: Tue Dec 10 18:59:25 2024 +0800

doc: fix readme (#445)

commit 9c314d2
Author: tongke6 [email protected]
Date: Tue Dec 10 18:46:10 2024 +0800

Add USE_BAZEL_VERSION instruction (#444)

* fix license check

* add USE_BAZEL_VERSION instructions

commit fc0fe17
Author: tongke6 [email protected]
Date: Tue Dec 10 17:44:46 2024 +0800

fix build on arm64 (#443)

* fix build on arm64

* update .bazelrc

* fix --cxxopt=-Wno-mismatched-new-delete not works on linux_aarch64

commit b91787c
Author: tongke6 [email protected]
Date: Tue Dec 10 14:19:01 2024 +0800

specify bazel version in .bazeliskrc since .bazelversion not working … (#441)

* specify bazel version in .bazeliskrc since .bazelversion not working in bzlmod

* update circleci config

* ignore license check for .bazeliskrc

commit b2e3d2f
Author: huocun [email protected]
Date: Tue Dec 10 12:48:38 2024 +0800

fix test (#442)

commit 3f5b224
Author: tongke6 [email protected]
Date: Mon Dec 9 23:02:08 2024 +0800

chore: update circleci config (#440)

commit c1de028
Author: tongke6 [email protected]
Date: Mon Dec 9 22:43:55 2024 +0800

fix: fix m1 macos build (#439)

* fix: fix m1 macos build

* format MODULE.bazel

commit b9dc6c3
Author: tongke6 [email protected]
Date: Fri Dec 6 17:36:38 2024 +0800

fix: add version in bazel_dep (#432)

commit ecbe46b
Author: Candicepan [email protected]
Date: Fri Dec 6 17:01:41 2024 +0800

Revert "chore(deps): update dependency openssl to v3.3.2.bcr.1 (#427)" (#429)

This reverts commit 13f9763aa295c036e897136620fcd4ca9522f9c5.

commit 13f9763
----------------------------------------

@tongke6
Copy link
Collaborator

tongke6 commented Feb 19, 2025

If you'd like to build inside a container, we recommend using our pre-built image: secretflow/ubuntu-base-ci:latest https://hub.docker.com/r/secretflow/ubuntu-base-ci/tags

@NicolausLiu
Copy link
Author

Thank you very much. Unfortunately, I just operated according to Getting Started in yacl within a container, but the build error still appeared. Is there a document for this image?

@tongke6
Copy link
Collaborator

tongke6 commented Feb 20, 2025

Can you list your operating steps?

The image secretflow/ubuntu-base-ci already has the dependencies installed. You just need to mount the code into the image and build it directly.

steps

# 1. clone yacl & cd yacl
$ git clone https://github.com/secretflow/yacl.git && cd yacl

# 2. start container
$ docker run -d -it --name yacl-dev-$(whoami) \
         --mount type=bind,source="$(pwd)",target=/home/admin/dev/ \
         -w /home/admin/dev \
         --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
         --cap-add=NET_ADMIN \
         --privileged=true \
         secretflow/ubuntu-base-ci:latest \
         bash
$ docker exec -it yacl-dev-$(whoami) bash
# now, you are in container

# 3. start build

$ bazel build //yacl/... 

@Jamie-Cui
Copy link
Collaborator

@NicolausLiu could you please tell us your bazel version?

@NicolausLiu
Copy link
Author

@NicolausLiu could you please tell us your bazel version?
Sure. The Bazel version is (8.1.0). The last log in the container is shown below.

(base)root@da7f4017cdbd:/experiments/yacl_t# git clone https://github.com/secretflow/yacl.git
Cloning into 'yacl'...
remote: Enumerating objects: 6202, done.
remote: Counting objects: 100% (766/766), done.
remote: Compressing objects: 100% (186/186), done.
remote: Total 6202 (delta 658), reused 584 (delta 579), pack-reused 5436 (from 2)
Receiving objects: 100% (6202/6202), 5.08 MiB | 993.00 KiB/s, done.
Resolving deltas: 100% (4351/4351), done.

(base)root@da7f4017cdbd:/experiments/yacl_t# apt install gcc wget cmake ninja-build nasm automake libtool libomp-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
automake is already the newest version (1:1.16.5-1.3).
automake set to manually installed.
gcc is already the newest version (4:11.2.0-1ubuntu1).
gcc set to manually installed.
nasm is already the newest version (2.15.05-1).
ninja-build is already the newest version (1.10.1-1).
cmake is already the newest version (3.22.1-1ubuntu1.22.04.2).
wget is already the newest version (1.21.2-2ubuntu1.1).
The following additional packages will be installed:
file libllvm14 libltdl-dev libltdl7 libmagic-mgc libmagic1 libomp-14-dev libomp5-14
Suggested packages:
libtool-doc libomp-14-doc gfortran | fortran95-compiler gcj-jdk
The following NEW packages will be installed:
file libllvm14 libltdl-dev libltdl7 libmagic-mgc libmagic1 libomp-14-dev libomp-dev libomp5-14
libtool
0 upgraded, 10 newly installed, 0 to remove and 0 not upgraded.
Need to get 25.4 MB of archives.
After this operation, 129 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmagic-mgc amd64 1:5.41-3ubuntu0.1 [257 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmagic1 amd64 1:5.41-3ubuntu0.1 [87.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 file amd64 1:5.41-3ubuntu0.1 [21.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB]
Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libltdl7 amd64 2.4.6-15build2 [39.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 libltdl-dev amd64 2.4.6-15build2 [169 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libomp5-14 amd64 1:14.0.0-1ubuntu1.1 [389 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libomp-14-dev amd64 1:14.0.0-1ubuntu1.1 [347 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libtool all 2.4.6-15build2 [164 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libomp-dev amd64 1:14.0-55exp2 [3074 B]
Fetched 25.4 MB in 19s (1327 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libmagic-mgc.
(Reading database ... 22874 files and directories currently installed.)
Preparing to unpack .../0-libmagic-mgc_1%3a5.41-3ubuntu0.1_amd64.deb ...
Unpacking libmagic-mgc (1:5.41-3ubuntu0.1) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../1-libmagic1_1%3a5.41-3ubuntu0.1_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.41-3ubuntu0.1) ...
Selecting previously unselected package file.
Preparing to unpack .../2-file_1%3a5.41-3ubuntu0.1_amd64.deb ...
Unpacking file (1:5.41-3ubuntu0.1) ...
Selecting previously unselected package libllvm14:amd64.
Preparing to unpack .../3-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ...
Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ...
Selecting previously unselected package libltdl7:amd64.
Preparing to unpack .../4-libltdl7_2.4.6-15build2_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.6-15build2) ...
Selecting previously unselected package libltdl-dev:amd64.
Preparing to unpack .../5-libltdl-dev_2.4.6-15build2_amd64.deb ...
Unpacking libltdl-dev:amd64 (2.4.6-15build2) ...
Selecting previously unselected package libomp5-14:amd64.
Preparing to unpack .../6-libomp5-14_1%3a14.0.0-1ubuntu1.1_amd64.deb ...
Unpacking libomp5-14:amd64 (1:14.0.0-1ubuntu1.1) ...
Selecting previously unselected package libomp-14-dev.
Preparing to unpack .../7-libomp-14-dev_1%3a14.0.0-1ubuntu1.1_amd64.deb ...
Unpacking libomp-14-dev (1:14.0.0-1ubuntu1.1) ...
Selecting previously unselected package libtool.
Preparing to unpack .../8-libtool_2.4.6-15build2_all.deb ...
Unpacking libtool (2.4.6-15build2) ...
Selecting previously unselected package libomp-dev:amd64.
Preparing to unpack .../9-libomp-dev_1%3a14.0-55
exp2_amd64.deb ...
Unpacking libomp-dev:amd64 (1:14.0-55exp2) ...
Setting up libomp5-14:amd64 (1:14.0.0-1ubuntu1.1) ...
Setting up libmagic-mgc (1:5.41-3ubuntu0.1) ...
Setting up libmagic1:amd64 (1:5.41-3ubuntu0.1) ...
Setting up file (1:5.41-3ubuntu0.1) ...
Setting up libltdl7:amd64 (2.4.6-15build2) ...
Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ...
Setting up libomp-14-dev (1:14.0.0-1ubuntu1.1) ...
Setting up libomp-dev:amd64 (1:14.0-55
exp2) ...
Setting up libtool (2.4.6-15build2) ...
Setting up libltdl-dev:amd64 (2.4.6-15build2) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...

(base) root@da7f4017cdbd:/experiments/yacl_t# bazel build //yacl/...
WARNING: Invoking Bazel in batch mode since it is not invoked from within a workspace (below a directory having a MODULE.bazel file).
Extracting Bazel installation...
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
ERROR: The 'build' command is only supported from within a workspace (below a directory having a MODULE.bazel file).
See documentation at https://bazel.build/concepts/build-ref#workspace

(base) root@da7f4017cdbd:/experiments/yacl_t# touch WORKSPACE

(base)root@da7f4017cdbd:/experiments/yacl_t# bazel build //yacl/...
Starting local Bazel server (8.1.0) and connecting to it...
WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to bazelbuild/bazel#18958.
WARNING: Target pattern parsing failed.
ERROR: Skipping '//yacl/...': error loading package under directory 'yacl': error loading package 'yacl/examples/zkp': Unable to find package for @@[unknown repo 'yacl' requested from @@]//bazel:yacl.bzl: The repository '@@[unknown repo 'yacl' requested from @@]' could not be resolved: No repository visible as '@YacL' from main repository. Was the repository introduced in WORKSPACE? The WORKSPACE file is disabled by default in Bazel 8 (late 2024) and will be removed in Bazel 9 (late 2025), please migrate to Bzlmod. See https://bazel.build/external/migration.
ERROR: error loading package under directory 'yacl': error loading package 'yacl/examples/zkp': Unable to find package for @@[unknown repo 'yacl' requested from @@]//bazel:yacl.bzl: The repository '@@[unknown repo 'yacl' requested from @@]' could not be resolved: No repository visible as '@YacL' from main repository. Was the repository introduced in WORKSPACE? The WORKSPACE file is disabled by default in Bazel 8 (late 2024) and will be removed in Bazel 9 (late 2025), please migrate to Bzlmod. See https://bazel.build/external/migration.
INFO: Elapsed time: 36.243s
INFO: 0 processes.
ERROR: Build did NOT complete successfully

@Jamie-Cui
Copy link
Collaborator

What is the output of bazel version?

@NicolausLiu
Copy link
Author

Can you list your operating steps?

The image secretflow/ubuntu-base-ci already has the dependencies installed. You just need to mount the code into the image and build it directly.

steps

1. clone yacl & cd yacl

$ git clone https://github.com/secretflow/yacl.git && cd yacl

2. start container

$ docker run -d -it --name yacl-dev-$(whoami)
--mount type=bind,source="$(pwd)",target=/home/admin/dev/
-w /home/admin/dev
--cap-add=SYS_PTRACE --security-opt seccomp=unconfined
--cap-add=NET_ADMIN
--privileged=true
secretflow/ubuntu-base-ci:latest
bash
$ docker exec -it yacl-dev-$(whoami) bash

now, you are in container

3. start build

$ bazel build //yacl/...

按照你的步骤进行了,但是出现如下报错:
(base) root@755ab79ac288:/home/admin/dev# bazel build //yacl/...
ERROR: Error computing the main repository mapping: Error accessing registry https://raw.githubusercontent.com/secretflow/bazel-registry/main: Failed to fetch registry file https://raw.githubusercontent.com/secretflow/bazel-registry/main/modules/abseil-cpp/20240722.0/MODULE.bazel: Connection refused
Computing main repo mapping:

两个网址尝试打开后,现实是invalid request。

@NicolausLiu
Copy link
Author

NicolausLiu commented Feb 20, 2025

What is the output of bazel version?

The output is bazel 8.1.0.
(base) root@da7f4017cdbd:/home/admin# bazel --version
bazel 8.1.0

@Jamie-Cui
Copy link
Collaborator

简单来看的话,你可以在你自己镜像里面,把 yacl/exmaples 删除之后再重新编译。

你的 bazel log 确实有点奇怪,我这里在 mac m1 上使用 bazel 8.1.0 编译现在的 master 是没有出现你的问题的,如果你使用的现在不是 bazelisk 是一个裸 bazel 的话,可以尝试用一下 bazelisk 试试。记得在重试之前 bazel clean --expunge 清理一下 bazel 的缓存

@anakinxc
Copy link
Collaborator

@NicolausLiu
Copy link
Author

https://raw.githubusercontent.com/secretflow/bazel-registry/main/modules/abseil-cpp/20240722.0/MODULE.bazel: Connection refused

Looks like network issue

But there seems to be no abseil-cpp folder in the bazel-registry.

Image

@anakinxc
Copy link
Collaborator

https://raw.githubusercontent.com/secretflow/bazel-registry/main/modules/abseil-cpp/20240722.0/MODULE.bazel: Connection refused

Looks like network issue

But there seems to be no abseil-cpp folder in the bazel-registry.

Image

abseil-cpp should provided by official bcr here

@tongke6
Copy link
Collaborator

tongke6 commented Feb 21, 2025

https://raw.githubusercontent.com/secretflow/bazel-registry/main/modules/abseil-cpp/20240722.0/MODULE.bazel: Connection refused

Looks like network issue

But there seems to be no abseil-cpp folder in the bazel-registry.

Image

@NicolausLiu Yes, there is no abseil-cpp in secretflow/bazel-registry, bazel supports multiple bazel registries, it will search these registries in the order they are defined.

For this case, it would resolve abseill-cpp in offical bcr.

Please make sure that you have unrestricted access to Github and raw.githubusercontent.com.

@NicolausLiu
Copy link
Author

@NicolausLiu Yes, there is no abseil-cpp in secretflow/bazel-registry, bazel supports multiple bazel registries, it will search these registries in the order they are defined.

For this case, it would resolve abseill-cpp in offical bcr.

Please make sure that you have unrestricted access to Github and raw.githubusercontent.com.

Thank you. I have fixed the build error by manually adding bazel-dep from offical bcr and libtommath release from github in a new MODULE.bazel. The bazel test yacl has also passed. Thank you guys, very much.

@tongke6 tongke6 closed this as completed Feb 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants