KubernetesExecutor: let each team connect to its own cluster #74844

Description

KubernetesExecutor reads the settings it uses to connect to a cluster (in_cluster, config_file, cluster_context, verify_ssl, ssl_ca_cert) from the global [kubernetes_executor] section, even in multi-team mode. Namespace, pod template and the other pod settings are team-scoped, but every team connects to the same cluster. The client_factory and async_client_factory settings (#73014) can be set per team, but the factory is called with no arguments, so it can't tell which team it's building a client for.

This also affects KubernetesExecutor subclasses that bring their own client factory, such as AwsEksExecutor (#74399). The only way for a subclass to install its factory is through the [kubernetes_executor] client_factory setting, so a plain KubernetesExecutor in the same scope (both global, or the same team) picks it up too and connects to the subclass's cluster.

Proposed change: build the executor's clients with the team name, so the connection settings resolve per team and a factory is told which team it's for. Let a subclass declare its client factory on the class rather than through config, and have the scheduler and the pod watcher use it.

Use case/motivation

Multi-team deployments where teams run tasks on different Kubernetes clusters, and deployments that run KubernetesExecutor next to a subclass with its own client factory, for example while moving from one to the other.

Related issues

#73014, #74399 (review discussion: #74399 (comment), #74399 (comment))

Are you willing to submit a PR?

Yes