.env file referenced underneath env_file in a compose file is being parsed when file is not used.

[beyondmeat]

Description

In Compose V2 -2.0.0.0, .env files that are underneath other folders are being picked up and parsed by compose. In my case these .env files are stored underneath compose-stack/.config/.myapp/.env and don't have anything to do with compose.

Steps to reproduce the issue:

1. use compose file (mine uses nested compose file and profiles)
2. make .env file in a deep nested folder
3. run compose up

Describe the results you received:

unexpected character "/" in variable name near "rclone/drive\nRCLONE_LOG_LEVEL=ERROR\nRCLONE_LOG_FILE=/home/user/drive.log\n\nRCLONE_DRIVE_PACER_MIN_SLEEP=10ms\nRCLONE_DRIVE_PACER_BURST=1000\nRCLONE_BWLIMIT_FILE=100M"

Describe the results you expected:
Not to include the .env file since it's not in the top working dir or in the same folder as the compose file.

Additional information you deem important (e.g. issue happens only occasionally):

I could rename the non-compose .env files but that feels like a workaround.

Output of docker compose version:

Docker Compose version v2.0.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 16
  Running: 16
  Paused: 0
  Stopped: 0
 Images: 35
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-84-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.843GiB

Additional environment details:

I store my compose file in a folder, in that folder I have .config, .appdata, .secrets, etc. I use a 1 folder nested approach, so everything that has to deal with the compose stack is all nested in 1 folder structure. Compose v1 and 2.0RC3 didn't have this issue.

[page-down]

I can't reproduce it with the same OS and compose version. The .env file in the subfolder is not loaded, and no error is reported when invalid characters are added.

mkdir -p .config/.myapp/
printf 'DEBUG=debug' > .config/.myapp/.env
printf 'services: {debug: {image: alpine:3.14, network_mode: none, command: ["echo", "DEBUG=$DEBUG"], init: true}}' > docker-compose.yaml
docker compose run --rm debug

WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string. 
DEBUG=

[ulyssessouza]

Hello @beyondmeat! Could you please add a complete minimal testcase so we could try to reproduce it?

[beyondmeat]

docker-compose.yml

version: "3.9"

services:
  whoami:
    extends:
      file: ./docker-compose.whoami.yml
      service: whoami
    profiles:
      - whoami

  watchtower:
    command: "--no-startup-message --cleanup --schedule '0 30 4 * * *'"
    image: containrrr/watchtower
    labels:
      traefik.enable: "false"
    restart: always
    security_opt:
      - "no-new-privileges=true"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

docker-compose.whoami.yml

version: "3.9"

services:
  whoami:
    image: "traefik/whoami"
    env_file: 
      - ./.config/whoami/.env
  

./.config/whoami/.env

MY_VAR1=mydrive/v1
MY_VAR2=pass myapp/mykey
DOESNT_MATTER=ddfd/fdf/dfdf/dfdsfd

DO NOT use the "whoami" profile. In this situation, even though whoami isn't going to be pulled or started, it's reading the env_file linked in a (unused) compose file and using it like it was a env file for compose. env_file in a compose service is env vars for the container, not used by compose it. Which causes the entire pull/up command to fail complaining about invalid env vars it shouldn't even be looking at.

I was able to test this out by simply commenting out the env_file section in the compose files (that aren't being currently used unless activated with a profile).

Watchtower is the only thing that should be pulled and started in this case (since the whoami profile isn't active) but yet, it's reading the files underneath docker-compose.whoami.yml env_file section and parsing them like it's the compose .env file

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[page-down]

I can confirm that this issue still exists in docker compose v2.3.4.

# debug.compose.yaml
services:
  debug:
    image: alpine
    env_file:
      - ./debug/.env

# debug/.env
this/should/not/be/parsed=debug

# compose.yaml
services:
  debug-only:
    extends:
      file: ./debug.compose.yaml
      service: debug
    profiles:
      - debug
  hello:
    image: alpine

docker compose up
# unexpected character "/" in variable name near "this/should/not/be/parsed=debug\n"

The debug profile is not used.

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[m3adow]

Retested the use-case from @page-down and I can confirm this bug is still there with Docker Compose version v2.12.2.

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[PatrykKuniczak]

Anybody can fix it?
That's really frustrating

I using: v2.13.0

@milas

[ndeloof]

@beyondmeat while this bug seems relevant, I wonder: if env_file is invalid, even service is disabled by profile, how do you use this service? We could prevent compose to parse this file if service is disabled, but still this means this service config is basically "dead code" (actually, "dead config") that you will never be able to use for concrete actions.

to prevent this we would need to delay call to resolveEnvironment after profiles have been applied and some services excluded from configuration.

[beyondmeat]

@ndeloof in my case, I use the profiles keyword to use it, but in this case the point is that I'm not using it. It's more of a dynamic service that I use sometimes but not always. Basically, I have a git repo that has a main docker-compose that has individual service compose files that are used across multiple servers but each only use a portion of the services that is set by using the profile feature. I basically use it like a library if that makes sense.

[ndeloof]

re-opening this issue, as compose-spec/compose-go#338 allows to fix it, but still need to update compose codebase accordingly

[iTrooz]

It seems like this issue is still present with docker compose 2.33.1

Example docker compose:

services:
  serviceA:
    image: nginx:latest
  
  serviceB:
    image: nginx:latest
    env_file:
      - /doesnotexist/.env

Command:

$ docker compose up -d serviceA
env file /doesnotexist/.env not found: stat /doesnotexist/.env: no such file or directory
exit status 1

[iTrooz]

Could you re-open this please ?

[ndeloof]

@iTrooz `