AttributeError when merging env_from in pod_generator

[cris-b]

Apache Airflow version: 1.10.14

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.19.3

Environment:

• Cloud provider or hardware configuration: Azure
• OS (e.g. from /etc/os-release): RHEL8

What happened:

In my executor_config I had an env_from. I also wanted to pass some env as secrets but cant do that directly in executor_config so had to use the AIRFLOW__KUBERNETES__ENV_FROM_SECRET_REF variable.

This led to a crash when trying to create the pod:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1382, in _execute
self._execute_helper()
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1453, in _execute_helper
if not self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1515, in _validate_and_run_task_instances
self.executor.heartbeat()
File "/usr/local/lib/python3.7/site-packages/airflow/executors/base_executor.py", line 134, in heartbeat
self.sync()
File "/usr/local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 832, in sync
self.kube_scheduler.run_next(task)
File "/usr/local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 447, in run_next
base_worker_pod=self.worker_configuration_pod
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 564, in construct_pod
return reduce(PodGenerator.reconcile_pods, pod_list)
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 429, in reconcile_pods
client_pod_cp.spec = PodGenerator.reconcile_specs(base_pod.spec, client_pod_cp.spec)
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 476, in reconcile_specs
base_spec.containers, client_spec.containers
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 509, in reconcile_containers
client_container = extend_object_field(base_container, client_container, 'env_from')
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 647, in extend_object_field
base_obj_set = _get_dict_from_list(base_obj_field, field_to_merge)
File "/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_generator.py", line 676, in _get_dict_from_list
result[getattr(obj, field_to_merge)] = obj
AttributeError: 'V1EnvFromSource' object has no attribute 'name'

What you expected to happen:
Pod created successfully

How to reproduce it:

Define a pod in executor_config (or pod_template) and additionally use AIRFLOW__KUBERNETES__ENV_FROM_SECRET_REF or AIRFLOW__KUBERNETES__ENV_FROM_CONFIGMAP_REF

As long as both the base config and the template contain envFrom/env_from.

Anything else we need to know:

This is caused by PodGenerator.extend_object_field trying to merge on 'name' but V1EnvFromSource don't have those, they either specify config_map_ref or secret_ref.

[cris-b]

Broken by [ #10366 ]

[dimberman]

@cris-b would you be interested in merging your fix? I'm glad to help you get setup with breeze/review your PR if you have cycles :)

[kaxil]

closed by #14090