[QUESTIONS] Some Questions about maintenance, compilation, static linking & more #26

Azathothas · 2024-05-06T05:48:06Z

Hi, I recently came across your great repo. It's truly awesome.
I maintain a similar repo for compiling android binaries at: https://github.com/Azathothas/Toolpacks
Though, I dropped support for Android due to these reasons: https://github.com/Azathothas/Toolpacks/tree/main/Docs#why-not-many-android-binaries
After having seen this awesome project and the great formulas

❯ ndk-pkg ls-available | wc -l
712

That are already available for me to use, I wanted to ask some questions.

Maintenance/Future

It's great to see that the first commit was all the way on Jun 25, 2020
And since then this repo has received over a thousand commits and is very well maintained. Though, no one can predict the future, and assuming you still have time & passion for this project, will this project (& the related adjacent ones) be continued to be maintained/updated?

Compiling

As I stated, I maintain a repo at https://github.com/Azathothas/Toolpacks with the specific aim to compile binaries and make them readily available at: https://bin.ajam.dev

Do you already have something similar?
There would be no point in me using your tool to compile & release binaries if you already release/publish them somewhere.
This is a snipped version of how I intend to use your tool to compile & extract only the binaries. Is there a better way, if I am only interested in the /bin?

!# Assuming I already installed/setup ndk-pkg in docker
##https://apilevels.com/
#android-34 --> Android 14
#android-33 --> Android 13
#android-31,32 --> Android 12
#android-30 --> Android 11
#android-29 --> Android 10
#android-28 --> Android 9
#android-26,27 --> Android 8
#android-24,25 --> Android 7
#android-23 --> Android 6
#android-21,22 --> Android 5
export TOOLPACKS_ANDROID_APILEVEL_DYNAMIC="android-29"
export TOOLPACKS_ANDROID_APILEVEL_STATIC="android-34"
export TOOLPACKS_ANDROID_ABI="arm64-v8a"
export TOOLPACKS_ANDROID_BUILD_DYNAMIC="${TOOLPACKS_ANDROID_APILEVEL_DYNAMIC}-${TOOLPACKS_ANDROID_ABI}"
export TOOLPACKS_ANDROID_BUILD_STATIC="${TOOLPACKS_ANDROID_APILEVEL_STATIC}-${TOOLPACKS_ANDROID_ABI}"

!# Build Curl
❯ docker exec -it "ndk-pkg" ndk-pkg install "${TOOLPACKS_ANDROID_BUILD_DYNAMIC}/curl" --profile="release" --jobs="$(($(nproc)+1))"

!# Get Install Dir
❯ TOOLPACKS_ANDROID_BUILDIR="$(docker exec -it "ndk-pkg" ndk-pkg tree "${TOOLPACKS_ANDROID_BUILD_DYNAMIC}/curl" --dirsfirst -L 1 | grep -o '/.*/.*' | tail -n1 | tr -d '[:space:]')" && export TOOLPACKS_ANDROID_BUILDIR="${TOOLPACKS_ANDROID_BUILDIR}"
❯ docker exec -it "ndk-pkg" ls "${TOOLPACKS_ANDROID_BUILDIR}/bin"
curl  curl-config

!#Copy 
❯ docker cp "ndk-pkg:/${TOOLPACKS_ANDROID_BUILDIR}/bin/." "./"
Successfully copied 5.15MB to /tmp/tmp.RhSYTZRvL6/./

!#Meta 
❯ file "./curl" && du -sh "./curl"
./curl: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, stripped
5.0M    ./curl

!#Cleanup
❯ docker exec -it "ndk-pkg" ndk-pkg uninstall "${TOOLPACKS_ANDROID_BUILD_DYNAMIC}/curl"
❯ docker exec -it "ndk-pkg" ndk-pkg cleanup
==> rm -ff /root/.ndk-pkg/installed/android-29-arm64-v8a/curl
==> rm -rf /root/.ndk-pkg/installed/android-29-arm64-v8a/b8179338859a8b8c7494ff61912bb44ce311c75ba576647558f835ee1956ba77
[✔] Done.

Static Linking

It's great to see that there's a simple switch to toggle fully static builds --fsle
But if the binary makes heavy use of network/dns, there are some known issues:

https://github.com/Zackptg5/Cross-Compiled-Binaries-Android/tree/master/build_script#dns-issues

https://github.com/termux/termux-packages/wiki/Common-porting-problems

Does your script/tooling already accounts for these? If not, Should a warning/caution be written in the README.md?
What do you think of these?

Testing

Do you know of a way I could test these binaries by running them in a container/vm directly on the builder machine (it's GH Actions x86_64 Linux headless) ?

Once again, this is truly amazing work. Thank you so much for making this open source.
Once I have some time, I will probably contribute a lot to the https://github.com/leleliu008/ndk-pkg-formula-repository-official-core repo.

The text was updated successfully, but these errors were encountered:

leleliu008 · 2024-05-06T10:39:27Z

This project is being actively developed.

I also release binaries via GitHub actions. I am working on it right now.

https://github.com/fpliu1214/uppm-package-repository-android-aarch64
https://github.com/fpliu1214/uppm-package-repository-android-29-aarch64

I will release more packages.

Due to this project is being actively developed, you are highly recommended using ndk-pkg via GitHub Actions, otherwise you should frequently update it.

https://github.com/leleliu008/ndk-pkg-package-manually-build

It is meaningless to build as fully statically linked executables for some packages (e.g. curl, wget, etc) due to the dns resolver issue, so do not use --fsle option when building these packages. ndk-pkg will not give you a warning.

for fully statically linked executables packages, you could do this:

ndk-pkg pack  android-34-arm64-v8a/${pkg}} -o . --exclude=include --exclude=lib

About testing, If you want to use GitHub actions, you need to pay for them. I test these packages on my phone.

leleliu008 · 2024-05-06T10:51:05Z

mkdir ndk-pkg-home

docker create -it --name ndk-pkg -v "$PWD/ndk-pkg-home:/root/.ndk-pkg"  fpliu/ndk-pkg
docker start ndk-pkg
docker exec -it ndk-pkg ndk-pkg setup
docker exec -it ndk-pkg ndk-pkg update

docker exec -it ndk-pkg ndk-pkg install android-34-arm64-v8a/darkhttpd --fsle
docker exec -it ndk-pkg ndk-pkg pack   android-34-arm64-v8a/darkhttpd -o /root/.ndk-pkg/ --exclude=include --exclude=lib

packed packages is in $PWD/ndk-pkg-home

no packed packages tree is in $PWD/ndk-pkg-home/installed

you do not need to use docker cp command

leleliu008 · 2024-05-06T11:44:58Z

I am working on running ndk-pkg on Android devices directly. This feature may be published in the next release.

Azathothas · 2024-05-07T00:59:37Z

mkdir ndk-pkg-home

docker create -it --name ndk-pkg -v "$PWD/ndk-pkg-home:/root/.ndk-pkg"  fpliu/ndk-pkg
docker start ndk-pkg
docker exec -it ndk-pkg ndk-pkg setup
docker exec -it ndk-pkg ndk-pkg update

docker exec -it ndk-pkg ndk-pkg install android-34-arm64-v8a/darkhttpd --fsle
docker exec -it ndk-pkg ndk-pkg pack   android-34-arm64-v8a/darkhttpd -o /root/.ndk-pkg/ --exclude=include --exclude=lib

packed packages is in $PWD/ndk-pkg-home

no packed packages tree is in $PWD/ndk-pkg-home/installed

you do not need to use docker cp command

I don't mount any volumes because I use an ephemeral container and it will all get purged anyway.
Currently I install ndk-pkg like this:

##https://github.com/leleliu008/ndk-pkg#using-ndk-pkg-via-docker-or-podman
 #Container: https://hub.docker.com/r/fpliu/ndk-pkg
  sudo docker stop "$(sudo docker ps -aqf name=ndk-pkg)" 2>/dev/null && sleep 5
  sudo docker rm "$(sudo docker ps -aqf name=ndk-pkg)" 2>/dev/null && sleep 5
  docker create -it --name "ndk-pkg" "fpliu/ndk-pkg:latest"
  docker start "ndk-pkg"

 #Setup & Config
  docker exec -it "ndk-pkg" ndk-pkg upgrade-self
  docker exec -it "ndk-pkg" ndk-pkg setup
  docker exec -it "ndk-pkg" ndk-pkg update
  docker exec -it "ndk-pkg" ndk-pkg sysinfo
  #https://github.com/leleliu008/ndk-pkg-formula-repository-official-core
  docker exec -it "ndk-pkg" ndk-pkg formula-repo-list
  docker exec -it "ndk-pkg" ndk-pkg formula-repo-sync "official-core"

So in this scenario I have found docker cp to be cleaner.
I asked because I was thinking maybe there was an option to get the binary directly in my $CWD similar to how nix-build creates a ./result dir with all the bins inside.

Azathothas · 2024-05-07T01:12:15Z

This project is being actively developed.

That's great to hear, I have decided to use your tool in my repo.

I also release binaries via GitHub actions. I am working on it right now.

https://github.com/fpliu1214/uppm-package-repository-android-aarch64 https://github.com/fpliu1214/uppm-package-repository-android-29-aarch64

I will release more packages.

I see, thanks! Maybe I can directly pull the bins from there and save time.

Due to this project is being actively developed, you are highly recommended using ndk-pkg via GitHub Actions, otherwise you should frequently update it.

https://github.com/leleliu008/ndk-pkg-package-manually-build

It is meaningless to build as fully statically linked executables for some packages (e.g. curl, wget, etc) due to the dns resolver issue, so do not use --fsle option when building these packages. ndk-pkg will not give you a warning.

I think curl still needs some patches, as the dynamically linked curl, complains about certs:

!#Built using
❯ docker exec -it "ndk-pkg" ndk-pkg install "android-29-arm64-v8a/curl" --profile="release" --jobs="$(($(nproc)+1))"

!# file
❯ file curl
curl: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, stripped

!# It will work if I add -k | --insecure flag
❯ ./curl https://bin.ajam.dev -v
* Host bin.ajam.dev:443 was resolved.
* IPv6: 2606:4700:3033::ac43:8db2, 2606:4700:3033::6815:2702
* IPv4: 104.21.39.2, 172.67.141.178
*   Trying [2606:4700:3033::ac43:8db2]:443...
* Connected to bin.ajam.dev (2606:4700:3033::ac43:8db2) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

!# Hints: https://github.com/stunnel/static-curl/issues/43 suggests using
--cacert /data/data/com.termux/files/usr/etc/tls/cert.pem
This works, but requires termux.

About testing, If you want to use GitHub actions, you need to pay for them. I test these packages on my phone.
I ended up using the offical termux-docker image

docker run --privileged -it --rm --platform="linux/arm64" --network="host" -v "$BINDIR:/mnt" "termux/termux-docker:aarch64" "/mnt/curl" --version

But this still runs with the dns issues, perhaps I need to directly ssh into a real android phone and test the binaries.

leleliu008 · 2024-05-07T01:19:52Z

I think curl still needs some patches, as the dynamically linked curl, complains about certs:

curl -LO https://curl.se/ca/cacert.pem
export SSL_CERT_FILE="$PWD/cacert.pem"

Azathothas · 2024-05-07T01:31:10Z

I think curl still needs some patches, as the dynamically linked curl, complains about certs:
curl -LO https://curl.se/ca/cacert.pem
export SSL_CERT_FILE="$PWD/cacert.pem"

Ahh great.
For anyone who runs into similar issues:

!# certs:
curl -kqfsSLO "https://curl.se/ca/cacert.pem"  || wget -q --no-check-certificate "https://curl.se/ca/cacert.pem"
export SSL_CERT_FILE="$PWD/cacert.pem"

!# Manually specify in curl :
 --cacert="$PWD/cacert.pem"
 
!# Manually specify in wget :
 --ca-certificate="$PWD/cacert.pem"

Azathothas · 2024-05-07T07:00:14Z

Hi again,
regarding the official repo at: https://github.com/leleliu008/ndk-pkg-formula-repository-official-core
there's a subdirectory inside ./formula labeled 1, is it labeled that way because it contains still-testing | broken packages?

Also, is there a way to directly specify a formula.yml file using the cli?

leleliu008 · 2024-05-07T07:05:14Z

formula/1/*.yml are broken packages, we need more patches.

no way to specify a formula file via cli options. You could add a your own formula repository.

leleliu008 · 2024-05-07T07:26:44Z

Also, is there a way to directly specify a formula.yml file using the cli?

I have considered it for a log time. I'm still considering.

leleliu008 · 2024-05-07T07:37:42Z

This idea is still on the table. It is a low priority task at least for now.

Azathothas · 2024-05-07T07:38:30Z

Also, is there a way to directly specify a formula.yml file using the cli?

I have considered it for a log time. I'm still considering.

That would be really convenient for testing if the formula file is correct + builds successfully, all without the pains of setting up a repo and configuring ndk-pkg to use it...

leleliu008 · 2024-05-07T07:43:28Z

add a formula repository is very easy, just one command ./ndk-pkg formula-repo-add

Azathothas · 2024-05-07T07:48:00Z

I meant, creating the dirs and copying the files and tracking changes. Git is insane sometimes...
Not to mention, if I am using docker without mounts, it will be extra steps.
Whereas, if we could directly point to a file and have ndk-pkg take care of it, it would be too easy/time saving.

But this will probably encourage people to fork and submit more patches, so I guess relying on repo only is one way to get pull requests...

leleliu008 · 2024-05-07T07:59:35Z

I don't known why you use docker like that.

I myself always mount my host machine directory to container, and open two windows, one for editing formula files, another for running ndk-pkg.

leleliu008 · 2024-05-07T08:02:24Z

Azathothas · 2024-05-15T01:26:56Z

Hello again,

Is it possible to be a bit less verbose in the default setting?

!# This is too verbose
docker exec -it "ndk-pkg" ndk-pkg install "android-34-arm64-v8a/aria2" --profile="release" --jobs="$(($(nproc)+1))"

!# This is too quiet
docker exec -it "ndk-pkg" ndk-pkg install "android-34-arm64-v8a/aria2" --profile="release" --jobs="$(($(nproc)+1))" -q
==> uppm package 'android-ndk-r26d' to be installed.
/root/.ndk-pkg/uppm/downloads/eefeafe7ccf177de7cc57158da585e7af119bb7504a63604ad719e4b2a328b54.zip already have been fetched.

uppm package 'android-ndk-r26d' was successfully installed.
ANDROID_NDK_HOME='/root/.ndk-pkg/uppm/installed/android-ndk-r26d'
ANDROID_NDK_ROOT='/root/.ndk-pkg/uppm/installed/android-ndk-r26d'
ANDROID_NDK_VERSION='26.3.11579264'
ANDROID_NDK_VERSION_MAJOR='26'
ANDROID_NDK_TOOLCHAIN_ROOT='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64'
ANDROID_NDK_TOOLCHAIN_BIND='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin'
ANDROID_NDK_SYSROOT='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot'
ANDROID_NDK_CC='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/clang'
ANDROID_NDK_CXX='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++'
ANDROID_NDK_CPP='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -E'
ANDROID_NDK_LD='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/ld.lld'
ANDROID_NDK_AS='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-as'
ANDROID_NDK_AR='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar'
ANDROID_NDK_NM='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-nm'
ANDROID_NDK_SIZE='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-size'
ANDROID_NDK_STRIP='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip'
ANDROID_NDK_RANLIB='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ranlib'
ANDROID_NDK_STRINGS='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strings'
ANDROID_NDK_OBJDUMP='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-objdump'
ANDROID_NDK_OBJCOPY='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-objcopy'
ANDROID_NDK_READELF='/root/.ndk-pkg/uppm/installed/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-readelf'
ANDROID_NDK_SUPPORTED_MIN_SDK_API_LEVEL='21'
ANDROID_NDK_SUPPORTED_MAX_SDK_API_LEVEL='34'

!# It didn't even say that it built successfully


!# In the help flag:
        -v
            verbose mode. many messages will be output to terminal.

        -vv
            very verbose mode. many many messages will be output to terminal.

!# But it seems like even without specifying the -v flag, the default is automatically verbose (-v) ?
I tested with and wthout -v flag: 
(Even after doing:   ndk-pkg uninstall  && ndk-pkg cleanup , it seems like cache is preserved as the first time I ran this, the log file was huge 145,166 Lines (9.5 MB) )

-v enabled   --> wc -l < build-v.log : 10,428 (421.2 KB)
-vv enabled --> wc -l < build-vv.log : 20,002 (1.84 MB)
without -v   --> wc -l < build-default.log : 10,428 (421.2 KB) 

!# I would like it to be quiet but also show progress etc, maybe like nix-build?
Right now each build creates about 8-10 MB log file

build-default.log
build-vv.log
build-v.log
build-default-first-run.log

leleliu008 · 2024-05-15T01:29:35Z

I will change this behave in next release.

Azathothas · 2024-05-15T02:19:22Z

Another question, from the readme:

show information of the given available package

ndk-pkg info-available curl
ndk-pkg info-available curl --yaml
ndk-pkg info-available curl --json
ndk-pkg info-available curl version
ndk-pkg info-available curl license
ndk-pkg info-available curl summary
ndk-pkg info-available curl web-url
ndk-pkg info-available curl git-url
ndk-pkg info-available curl git-sha
ndk-pkg info-available curl git-ref
ndk-pkg info-available curl src-url
ndk-pkg info-available curl src-sha
ndk-pkg info-available curl src-ft
ndk-pkg info-available curl src-fp

is there a way to show information of all available packages?

currently, I am using something like this:

docker exec "ndk-pkg" ndk-pkg ls-available | while read -r formula; do
    echo -n "$formula: "
    docker exec -e "formula=$formula" "ndk-pkg" ndk-pkg info-available "$formula" summary | sed 's/^/ /'
done
❯
ARM_NEON_2_x86_SSE:  The platform independent header allowing to compile any C/C++ code containing ARM NEON intrinsic functions for x86 target systems using SIMD up to SSE4 intrinsic functions
aalib:  A portable ASCII art graphics library
abseil:  C++ Common Libraries
acl:  C library and utilities for Manipulating POSIX Access Control Lists
act:  Run your GitHub Actions locally
actionlint:  Static checker for GitHub Actions workflow files
adig:  A command-line tool that allows you to perform DNS lookups from the command line
ag:  A fast code-searching tool similar to ack
age:  Simple, modern, secure file encryption
alass:  Automatic Language-Agnostic Subtitle Synchronization
algernon:  Pure Go web server with Lua, Markdown, HTTP/2 and template support
antibody:  Shell plugin manager
aom:  Codec library for encoding and decoding AV1 video streams
apkeep:  A command-line tool for downloading APK files from various sources
aptly:  Swiss army knife for Debian repository management
archiver:  Cross-platform, multi-format archive utility
args:  A simple header-only C++ argument parser library
aria2:  Download with resuming and segmented downloading
arping:  Utility to check whether MAC addresses are already taken on a LAN
attr:  C library and utilities for Manipulating Filesystem Extended Attributes
awk:  Text processing scripting language
axel:  A lightweight download accelerator
azcopy:  Azure Storage data transfer utility
b3sum:  A command line utility for calculating BLAKE3 hashes
base16:  base16 encoder and decoder
base64:  Encode and decode base64 files
bash:  Bourne-Again SHell, a UNIX command interpreter
basis_universal:  Basis Universal GPU texture codec command-line compression tool
bat:  Clone of cat(1) with syntax highlighting and Git integration
bc:  Arbitrary precision numeric processing language
bcrypt:  Cross platform file encryption utility using blowfish
berkeley-db:  A high-performance key/value database
bgrep:  Like grep but for binary strings
binaryen:  Compiler infrastructure and toolchain library for WebAssembly
bind:  Implementation of the DNS protocols
binocle:  A graphical tool to visualize binary data
bison:  Yacc-compatible Parser generator
bk:  Terminal Epub reader
blockhash:  A perceptual image hash calculation tool based on algorithm descibed in Block Mean Value Based Image Perceptual Hashing by Bian Yang, Fan Gu and Xiamu Niu
boost:  A collection of portable C++ source libraries
boringssl:  A fork of OpenSSL that is designed to meet Google needs
boxes:  Draw boxes around text
brook:  Cross-platform strong encryption and not detectable proxy. Zero-Configuration
broot:  New way to see and navigate directory trees
brotli:  A generic-purpose lossless compression algorithm by Google
bsdtar:  BSD tar
bullet:  Physics SDK
byacc:  (Arguably) the best yacc variant
bzip2:  Burrows–Wheeler-based data compression utilities with high compression ratio
caddy:  A powerful, enterprise-ready, open source web server with automatic HTTPS
cairo:  Vector graphics library with cross-device output support
cargo-c:  A helper program to build and install C-ABI compatible dynamic and static libraries
catch2:  Modern, C++-native, header-only, test framework
ccache:  Object-file caching compiler wrapper
cereal:  C++11 library for serialization
cfitsio:  C access to FITS data files with optional Fortran wrappers
cflow:  A command-line tool to generate call graphs from C code
cgal:  Computational Geometry Algorithms Library
cheat:  Create and view interactive cheat sheets for *nix commands
check:  A unit testing framework for C
chezmoi:  Manage your dotfiles across multiple diverse machines, securely
chinese-calendar:  chinese festival/jieqi algorithm
chisel:  A fast TCP/UDP tunnel over HTTP
choose:  Human-friendly and fast alternative to cut and (sometimes) awk
cjson:  Ultralightweight JSON parser in ANSI C
cli11:  Simple and intuitive command-line parser for C++11
clog:  Colorized pattern-matching log tail utility
cmake:  Cross-platform make
cmark:  The C reference implementation of CommonMark
cmatrix:  A command-line tool for producing a Matrix-style animation
cmocka:  An elegant unit testing framework for C with support for mock objects
coreutils:  GNU File, Shell, and Text utilities
cotp:  TOTP/HOTP authenticator app with import functionality
cpio:  Copies files into or out of a cpio or tar archive
cpprestsdk:  A C++ library for cloud-based client-server communication
cpptoml:  Header-only library for parsing TOML
cppunit:  Unit testing framework for C++
cpu_features:  Cross platform C99 library to get cpu features at runtime
cpuid:  CPU feature identification for Go
cpuinfo:  CPU INFOrmation library
...

I am trying to filter library packages and only print binary/cli packages

I have found you already have one at: https://github.com/fpliu1214/uppm-package-repository-android-aarch64/blob/master/.github/workflows/publish.yml#L11

pkg: [aria2, axel, base16, base64, bash, bat, bison, bc, bsdtar, bzip2,...

It would be handy to provide a way to list all packages but also show a brief summary...

maybe something like this:

ndk-pkg info-available all #or another placeholder if it conflicts with a package name
ndk-pkg info-available all  --yaml #yaml dump of all , pipeable to yq
ndk-pkg info-available all  --json #json dump of all , pipeable to jq
ndk-pkg info-available all  version #$PKG: $OUTPUT format
ndk-pkg info-available all  license #$PKG: $OUTPUT format
ndk-pkg info-available all  summary #$PKG: $OUTPUT format
ndk-pkg info-available all  web-url #$PKG: $OUTPUT format
ndk-pkg info-available all  git-url #$PKG: $OUTPUT format
ndk-pkg info-available all  git-sha #$PKG: $OUTPUT format
ndk-pkg info-available all  git-ref #$PKG: $OUTPUT format
ndk-pkg info-available all  src-url #$PKG: $OUTPUT format
ndk-pkg info-available all  src-sha #$PKG: $OUTPUT format
ndk-pkg info-available all  src-ft #$PKG: $OUTPUT format
ndk-pkg info-available all  src-fp #$PKG: $OUTPUT format

leleliu008 · 2024-05-15T02:25:40Z

I will add ndk-pkg ls-available -v in next release.

leleliu008 · 2024-05-15T02:28:10Z

I used to support ndk-pkg info-available @all, but I removed it again. ndk-pkg ls-available -v would be a good solution.

Azathothas · 2024-05-15T03:14:39Z

A json dump of your https://github.com/leleliu008/ndk-pkg-formula-repository-official-core formulas would be greatly appreciated.
Perhaps I can set it up...

leleliu008 · 2024-05-15T03:20:55Z

Yes, you could do it by yourself. I have no plan to add this feature, at least for now.

leleliu008 · 2024-05-15T04:26:47Z

./ndk-pkg ls-available -v --json

Azathothas · 2024-05-15T04:50:53Z

I see, thanks!

Will this format remain consistent now?

# ndk-pkg ls-available -v
===
pkgname: ARM_NEON_2_x86_SSE
pkgtype: lib
version: 2024.05.15
summary: The platform independent header allowing to compile any C/C++ code containing ARM NEON intrinsic functions for x86 target systems using SIMD up to SSE4 intrinsic functions
web-url: https://github.com/intel/ARM_NEON_2_x86_SSE
git-url: https://github.com/intel/ARM_NEON_2_x86_SSE
dep-upp: git cmake ninja
bsystem: cmake
binbstd: 0
parallel: 1
installed: no
===
pkgname: aalib
pkgtype: lib
version: 1.4rc5
license: GPL-2.0-or-later
summary: A portable ASCII art graphics library
web-url: https://aa-project.sourceforge.io/aalib/
src-url: https://downloads.sourceforge.net/project/aa-project/aa-lib/1.4rc5/aalib-1.4rc5.tar.gz
src-sha: fbddda9230cf6ee2a4f5706b4b11e2190ae45f5eda1f0409dc4f99b35e0a70ee
dep-pkg: ncurses
dep-upp: libtool curl bsdtar gmake gm4 perl autoconf automake gmake
bsystem: autotools
binbstd: 0
ppflags: -include stdlib.h -include string.h
parallel: 1
installed: no
===

# ndk-pkg ls-available -v --yaml
===
pkgname: ARM_NEON_2_x86_SSE
pkgtype: lib
version: 2024.05.15
summary: The platform independent header allowing to compile any C/C++ code containing ARM NEON intrinsic functions for x86 target systems using SIMD up to SSE4 intrinsic functions
web-url: https://github.com/intel/ARM_NEON_2_x86_SSE
git-url: https://github.com/intel/ARM_NEON_2_x86_SSE
dep-upp: git cmake ninja
bsystem: cmake
binbstd: 0
parallel: 1
installed: no
===
pkgname: aalib
pkgtype: lib
version: 1.4rc5
license: GPL-2.0-or-later
summary: A portable ASCII art graphics library
web-url: https://aa-project.sourceforge.io/aalib/
src-url: https://downloads.sourceforge.net/project/aa-project/aa-lib/1.4rc5/aalib-1.4rc5.tar.gz
src-sha: fbddda9230cf6ee2a4f5706b4b11e2190ae45f5eda1f0409dc4f99b35e0a70ee
dep-pkg: ncurses
dep-upp: libtool curl bsdtar gmake gm4 perl autoconf automake gmake
bsystem: autotools
binbstd: 0
ppflags: -include stdlib.h -include string.h
parallel: 1
installed: no
===

!# ndk-pkg ls-available -v --json
# There seems to be extra trailing comma (jq: parse error: Expected value before ',' at line 2, column 1)
[
,
{
  "pkgname": "ARM_NEON_2_x86_SSE",
  "pkgtype": "lib",
  "version": "2024.05.15",
  "summary": "The platform independent header allowing to compile any C/C++ code containing ARM NEON intrinsic functions for x86 target systems using SIMD up to SSE4 intrinsic functions",
  "web-url": "https://github.com/intel/ARM_NEON_2_x86_SSE",
  "git-url": "https://github.com/intel/ARM_NEON_2_x86_SSE",
  "dep-upp": "git cmake ninja",
  "bsystem": "cmake",
  "binbstd": "0",
  "parallel": "1",
  "install": "cmakew"
}
,
{
  "pkgname": "aalib",
  "pkgtype": "lib",
  "version": "1.4rc5",
  "license": "GPL-2.0-or-later",
  "summary": "A portable ASCII art graphics library",
  "web-url": "https://aa-project.sourceforge.io/aalib/",
  "src-url": "https://downloads.sourceforge.net/project/aa-project/aa-lib/1.4rc5/aalib-1.4rc5.tar.gz",
  "src-sha": "fbddda9230cf6ee2a4f5706b4b11e2190ae45f5eda1f0409dc4f99b35e0a70ee",
  "dep-pkg": "ncurses",
  "dep-upp": "libtool curl bsdtar gmake gm4 perl autoconf automake gmake",
  "bsystem": "autotools",
  "binbstd": "0",
  "ppflags": "-include stdlib.h -include string.h",
  "parallel": "1",
  "dopatch": "sed_in_place '203c return 0;' src/aalinuxkbd.c",
  "install": "configure --without-x --with-ncurses=\"$ncurses_INSTALL_DIR\""
}
,

leleliu008 · 2024-05-15T05:10:38Z

Azathothas · 2024-05-15T05:57:51Z

Great!

I tested and it all works! Thanks!

I have setup auto updating workflow to generate these json/yaml/txt dumps along with a prettified readme at:
https://github.com/Azathothas/ndk-pkg-formula-repository-fork-core

Now, it's time to figure out what bins can be compiled statically and work.
go lang based ones are out already, thanks to your issue at: golang/go#59942
Probably bins that use any form of networking are also out due to dns issues.
Let's see which one will actually compile + work.

Azathothas · 2024-05-17T04:22:26Z

I have been testing ndk-packages on a bunch of different os & arch
As per https://github.com/leleliu008/ndk-pkg?tab=readme-ov-file#caveats , it's true musl-based system aren't compatible, however using something like https://github.com/sgerrand/alpine-pkg-glibc , I was able to successfully use ndk-pkg on alpine.

But testing ndk-pkg on aarch64/arm64 host (os: alpine with glibc), I got a bunch of 404 Not found when doing dk-pkg setup, and it failed to setup NDK Toolchain.
Is it because there's no official ndk toolchain for arm64 Linux?

leleliu008 · 2024-05-17T04:29:37Z

Could you tell me why you want to use Alpine linux (musl-based GNU/Linux) ?

I never tested ndk-pkg on Alpine Linux.

There is no official Android NDK prebuild binary for Linux-aarch64, You could build it from source by yourself if you want.

Azathothas · 2024-05-17T04:36:32Z

Could you tell me why you want to use Alpine linux (musl-based GNU/Linux) ?

I never tested ndk-pkg on Alpine Linux.

It was just a curiosity, but also I have found that if I want to quickly setup the ndk-toolchain, ndk-pkg is very, very useful.
I have been using an alpine docker image to compile packages that aren't in your formula repo or are broken etc.

There is no official Android NDK prebuild binary for Linux-aarch64, You could build it from source by yourself if you want.

So Linux-aarch64 isn't supported by ndk-pkg?
So how do you plan to support android?

I looked into building it from source but it seems like i's too much effort and requires lot's of manual patching.
It's very likely you already build it from source or intend to do at some time in the future, in that case I had rather wait for you to release an aarch64 version of ndk-pkg

leleliu008 · 2024-05-17T04:41:30Z

So Linux-aarch64 isn't supported by ndk-pkg?

Yes, ndk-pkg does not support linux-aarch64 at this monment due to no available Android NDK prebuild binary provided by official. I might support it at some point in the future.

Azathothas · 2024-05-17T04:46:22Z

I found it
https://github.com/leleliu008/ndk-pkg/blob/5d9341050fee9ff05997f2c63a0ec97666a39512/ndk-pkg#L8451C1-L8460C11

# setup_android_ndk_env [ANDROID_NDK_HOME]
  setup_android_ndk_env() {
    if [ -z "$1" ] ; then
        if [ "$NATIVE_OS_KIND" = android ] ; then
            # use comunity compiled Android NDK for aarch64-linux right now, I might compile my own at some point.
            # https://github.com/lzhiyong/termux-ndk/releases/tag/android-ndk
            ANDROID_NDK_REVISION='r26b'
        else
            ANDROID_NDK_REVISION='r26d'
        fi

leleliu008 · 2024-05-17T04:47:31Z

Although linux-aarch64 based clould server is quite popular, but It's rare to see in pc desktop. That is the mainly reason why Android NDK Team do not provide linux-aarch64 based prebuild binary.

leleliu008 · 2024-05-17T04:50:27Z

I found it https://github.com/leleliu008/ndk-pkg/blob/5d9341050fee9ff05997f2c63a0ec97666a39512/ndk-pkg#L8451C1-L8460C11

# setup_android_ndk_env [ANDROID_NDK_HOME]
  setup_android_ndk_env() {
    if [ -z "$1" ] ; then
        if [ "$NATIVE_OS_KIND" = android ] ; then
            # use comunity compiled Android NDK for aarch64-linux right now, I might compile my own at some point.
            # https://github.com/lzhiyong/termux-ndk/releases/tag/android-ndk
            ANDROID_NDK_REVISION='r26b'
        else
            ANDROID_NDK_REVISION='r26d'
        fi

It just doing some tests. I have no time to test all formulas right now.

Azathothas · 2024-05-17T04:55:51Z

It just doing some tests. I have no time to test all formulas right now.

No worries, I have been testing (at least the binary ones), and will periodically update leleliu008/ndk-pkg-formula-repository-official-core#10

Although, it seems sort of pointless to have static binaries but then need termux to make it work.
I had given up on android compilaion due to the existence of termux-packages :
https://github.com/Azathothas/Toolpacks/tree/main/Docs#why-not-many-android-binaries

I have found that android bins will almost always have to be compiled dynamically to be able to work in the mksh real shell in android without termux or other layers.
The only exclusions being simple cli tools like coreutils etc.

I have a fairly high spec android device which has no use, so aarch64 support would make it useful.

leleliu008 · 2024-05-17T04:59:44Z

No worries, I have been testing (at least the binary ones), and will periodically update leleliu008/ndk-pkg-formula-repository-official-core#10

Thank you.

leleliu008 · 2024-05-17T05:14:54Z

I have a fairly high spec android device which has no use, so aarch64 support would make it useful.

You want to run ndk-pkg on linux-aarch64 or android-aarch64 or both?

Azathothas · 2024-05-17T05:20:03Z

I have a fairly high spec android device which has no use, so aarch64 support would make it useful.

You want to run ndk-pkg on linux-aarch64 or android-aarch64 or both?

both would be ideal, as I currently run a few distros on my phone using proot
But if we are to prioritize , then supporting linux-aarch64 would be a higher one.

leleliu008 · 2024-05-17T05:28:51Z

I'd be curious to know what specifically you're using proot? Just because you have a fairly high spec android device?

leleliu008 · 2024-05-17T05:31:14Z

I thought you bought a linux-aarch64 based clould server.

Azathothas · 2024-05-17T05:38:59Z

I'd be curious to know what specifically you're using proot? Just because you have a fairly high spec android device?

I run a debian & an alpine distro on it.
Can't run docker etc, but it serves well just using in userspace.

I have a few projects: https://github.com/Azathothas/CertStream-Domains
I run the tooling in those proot distros.

I thought you bought a linux-aarch64 based clould server.

I want to, but it's been very difficult.
I am from Nepal, so it's very hard to get things like paypal, international visa card etc working.
I managed to get a card, but Hetzner requires passport and instantly blocks my account whenever I try to create one.
Netcup requires ery strict kyc (your selfie holding a paper with order id written)
Things like aws,gcp,azure are too expensive.
I have found another: https://www.layerstack.com/pricing-cloud-servers?cid=arm
I will probably buy one from them if I don't have any options.
If you know any other arm server provider, do let me know

leleliu008 · 2024-05-17T05:48:21Z

If you know any other arm server provider, do let me know

I have never bought a linux-aarch64 based clould server. I just saw some youtubers introduced linux-aarch64 based clould server in Chinese.

Azathothas · 2024-05-17T06:02:13Z

I have never bought a linux-aarch64 based clould server. I just saw some youtubers introduced linux-aarch64 based clould server in Chinese.

arm servers are quite cheap compared to x86_64.
I have noticed 50-70% less price than x86_64 based ones for similar specs.

Also, this issue has gotten too cluttered, I should probably close it.

If you have other means to reach you, for example telegram (we chat won't work in Nepal I think) etc, do let me know:
https://t.me/Azathothas

leleliu008 · 2024-05-17T06:10:25Z

If you have other means to reach you, for example telegram (we chat won't work in Nepal I think) etc, do let me know: https://t.me/Azathothas

I don't use any social media excpet github.com

leleliu008 · 2024-06-19T17:53:22Z

Also, is there a way to directly specify a formula.yml file using the cli?

I have considered it for a log time. I'm still considering.

That would be really convenient for testing if the formula file is correct + builds successfully, all without the pains of setting up a repo and configuring ndk-pkg to use it...

@Azathothas

you could specify a formula search directory like below:

ndk-pkg install <PACKAGE> -I /somewhere/formula/

/somewhere/formula/ is the directory where your formula files are located in.

This feature is in the latest release.

Azathothas closed this as completed May 17, 2024

[QUESTIONS] Some Questions about maintenance, compilation, static linking & more #26

[QUESTIONS] Some Questions about maintenance, compilation, static linking & more #26

Comments

Azathothas commented May 6, 2024

Maintenance/Future

Compiling

Static Linking

Testing

leleliu008 commented May 6, 2024 • edited Loading

leleliu008 commented May 6, 2024 • edited Loading

leleliu008 commented May 6, 2024

Azathothas commented May 7, 2024

Azathothas commented May 7, 2024

leleliu008 commented May 7, 2024

Azathothas commented May 7, 2024

Azathothas commented May 7, 2024

leleliu008 commented May 7, 2024

leleliu008 commented May 7, 2024

leleliu008 commented May 7, 2024

Azathothas commented May 7, 2024

leleliu008 commented May 7, 2024

Azathothas commented May 7, 2024

leleliu008 commented May 7, 2024 • edited Loading

leleliu008 commented May 7, 2024

Azathothas commented May 15, 2024 • edited Loading

leleliu008 commented May 15, 2024

Azathothas commented May 15, 2024

leleliu008 commented May 15, 2024

leleliu008 commented May 15, 2024 • edited Loading

Azathothas commented May 15, 2024

leleliu008 commented May 15, 2024

leleliu008 commented May 15, 2024

Azathothas commented May 15, 2024 • edited Loading

leleliu008 commented May 15, 2024

Azathothas commented May 15, 2024

Azathothas commented May 17, 2024

leleliu008 commented May 17, 2024 • edited Loading

Azathothas commented May 17, 2024

leleliu008 commented May 17, 2024

Azathothas commented May 17, 2024

leleliu008 commented May 17, 2024

leleliu008 commented May 17, 2024

Azathothas commented May 17, 2024

leleliu008 commented May 17, 2024

leleliu008 commented May 17, 2024

Azathothas commented May 17, 2024

leleliu008 commented May 17, 2024

leleliu008 commented May 17, 2024

Azathothas commented May 17, 2024

leleliu008 commented May 17, 2024

Azathothas commented May 17, 2024

leleliu008 commented May 17, 2024

leleliu008 commented Jun 19, 2024 • edited Loading

leleliu008 commented May 6, 2024 •

edited

Loading

leleliu008 commented May 6, 2024 •

edited

Loading

leleliu008 commented May 7, 2024 •

edited

Loading

Azathothas commented May 15, 2024 •

edited

Loading

leleliu008 commented May 15, 2024 •

edited

Loading

Azathothas commented May 15, 2024 •

edited

Loading

leleliu008 commented May 17, 2024 •

edited

Loading

leleliu008 commented Jun 19, 2024 •

edited

Loading