Sharing dependencies across environments

[robsonos]

What kind of issue is this?

• [*] Feature Request.

---

Configuration

Operating system: Windows 11 Pro

PlatformIO Version (platformio --version): PlatformIO Core, version 6.1.4

Description of problem

Hi, I was wondering if it is possible to create a feature to allow sharing dependencies across environments. I have a project with several dependencies with the same version and they end up getting downloaded multiple times.

Steps to Reproduce

pio run

Actual Results

The above command will create:

.pio
'-libdeps
  '-env1
     `-ArduinoJson
  '-env2
     `-ArduinoJson

Expected Results

Something like:

.pio
'-libdeps
  '-env-shared
     `-ArduinoJson

If problems with PlatformIO Build System:

The content of platformio.ini:

[env]
platform = nordicnrf52
board = adafruit_feather_nrf52840
framework = arduino
lib_deps = 
    bblanchon/ArduinoJson@^6.19.4

[env:env1]

[env:env2]

Source file to reproduce issue:

#include <Arduino.h>

void setup() {
  // put your setup code here, to run once:
}

void loop() {
  // put your main code here, to run repeatedly:
}

Additional info

[ivankravets]

with several dependencies with the same version and they end up getting downloaded multiple times.

PlatformIO does not download dependencies multiple times, it uses a cache for HTTP and binary requests. So, don't worry about this.

---

Regarding common dependencies. You can put them in the lib folder. Libraries located here are sharable between environments.

[robsonos]

Hi @ivankravets,

I see. However, PlatformIO doesn't seem to cache dependencies from git (private) repositories (I apologise for not mentioning that before), which makes sense but it drastically slows down continuous integration. I am aware of the lib folder alternative, but I still wanna treat the libraries as dependencies as they are versioned per release (but not per environment).

Would it be able to achieve dependency sharing across environments with Advanced Scripting?

[ivankravets]

Yes, https://docs.platformio.org/en/latest/scripting/index.html is the right solution. Just use the PRE script and pre-install dependencies to the lib folder. Also, you can keep declarations in the config and re-use them, see example https://docs.platformio.org/en/latest/scripting/examples/platformio_ini_custom_options.html

Another idea is to use the LibraryPackageManager API directly but I don't know how you are good in Python. See part of the code https://github.com/platformio/platformio-core/blob/develop/platformio/package/commands/install.py#L220

[mcspr]

^ I believe something like this was previously mentioned after lib_extra_dirs stopped working with lib_deps in #4206, so in our (espurna) case the only real option now is to manually install everything through the API

• https://github.com/xoseperez/espurna/blob/cc90554b0e7e1c2923d3f08fd2ce1edeaec393f1/code/scripts/pio_pre.py#L62-L103 (...with a small addition of migrating .pio/ $env libs...)
• https://github.com/xoseperez/espurna/blob/cc90554b0e7e1c2923d3f08fd2ce1edeaec393f1/code/scripts/pio_pre.py#L139-L144 (there is still a lib_deps-like variable, but installation happens on our explicit request. old versions removal included)
• https://github.com/xoseperez/espurna/blob/cc90554b0e7e1c2923d3f08fd2ce1edeaec393f1/code/platformio.ini#L139-L140

[robsonos]

Thanks, guys. I was considering using LibraryPackageManager though was not sure how to approach this in extra_scripts as my understanding is that extra_scripts run after the Library Manager. What you sent looks promising @mcspr.

[robsonos]

Hi guys. I ended up going with something like this:

platformio.ini:

[env]
...
extra_scripts = pre:pre_script.py
lib_extra_dirs = .pio/libdeps/shared/
shared_lib_dir = .pio/libdeps/shared/
shared_lib_deps =
  https://github.com/...
  https://github.com/...
  https://github.com/...

pre_script.py:

from SCons.Script import ARGUMENTS
import subprocess
from os import makedirs
from os.path import isdir
Import("env")


def install_shared_dependencies(env, verbose):
    # Get shared_lib_dir
    shared_lib_dir = env.GetProjectOption('shared_lib_dir')

    # Create shared_lib_dirif it does not exist
    if (not isdir(shared_lib_dir)):
        makedirs(shared_lib_dir)

    # Get lib_deps
    config = env.GetProjectConfig()
    raw_lib_deps = env.GetProjectOption('shared_lib_deps')
    lib_deps = config.parse_multi_values(raw_lib_deps)

    # Build dependency installation command
    cmd = [env.subst("$PYTHONEXE"), "-m", "platformio", "pkg", "install"]
    cmd.extend(["--storage-dir", shared_lib_dir])

    # Add verbose to command
    if int(verbose) < 1:
        cmd.append("-s")

    # Add dependencies to command
    for lib in lib_deps:
        cmd.append("-l")
        cmd.append(lib)

    # Run command
    subprocess.check_call(cmd)


# Get verbose level
VERBOSE = ARGUMENTS.get("PIOVERBOSE", 0)

# Intall dependencies listed in env shared_lib_deps to shared_lib_dir
install_shared_dependencies(env, VERBOSE)

I am happy for this issue to be closed but I still see value in having a feature in platformio to do this.

Cheers.

[shennan]

PlatformIO does seem to want to install multiple versions per environment. With a platformio.ini like this:

[env:Build1]
platform = atmelsam
board = adafruit_itsybitsy_m4
framework = arduino
src_filter = +<Build1>
lib_deps =
  bblanchon/ArduinoJson @ ^6.21.0

[env:Build2]
platform = atmelsam
board = adafruit_itsybitsy_m4
framework = arduino
src_filter = +<Build2>
lib_deps =
  adafruit/Adafruit NeoPixel@^1.12.3
  bblanchon/ArduinoJson @ ^6.21.0
build_flags =
  -D USE_TINYUSB  ; Enable TinyUSB stack

And directories like this:

src/
├─ Build1/
│  ├─ main.ccp
├─ Build2/
│  ├─ main.cpp

I get an error about a non-shared NeoPixel dependency in src/Build2/main.cpp:

#include errors detected. Please update your includePath. Squiggles are disabled for this translation unit (/path/to.my/project/src/RecyclingSensor/main.cpp).C/C++(1696)
cannot open source file "Adafruit_NeoPixel.h"C/C++(1696)

This is after I pio lib install, and looking at the .vscode/c_cpp_properties.json I see that only includePaths for Build1 are explicitly included. As soon as I add "path/to/my/project/.pio/libdeps/Build2/Adafruit NeoPixel" to the includePaths, the error goes away.

But c_cpp_properties.json is an auto-generated file that I don't feel comfortable editing. Shouldn't installing dependencies from my platformio.ini simply add these paths automatically?