How to work with multiple azure kubernetes contexts

Question

I have a azure function app in powershell, dedicated to kubernetes functions.

There are multiple functions but on each of them, the first thing we do is something like this.

 az login --service-principal -u $usr -p $pwd --tenant $tenant
 
 $config = $HOME + "/.kube/$tenant";
 
 az account set --subscription $subscription
 az aks get-credentials --overwrite-existing --name $cluster --resource-group $rg --admin --file $config
 kubectl config set-context --current --namespace=$tenant

This sets the contexts we will be working with on our next kubectl commands.

We do it, because we have multiple clusters on multiple regions, so everytime we get a call on the function app, we need to connect to the given cluster to work on it.

The problem with this is when we get 2 requests at exactly the same time for 2 different clusters and the earliest call will get its context overriden and be executing kubectl commands on the incorrect cluster. Is there a way to isolate each call or a solution to this?

Answer 1

You should be able to just use kubectl's global --context parameter (and --kubeconfig if needed), example:

az aks get-credentials -n $cluster -g $rg
kubectl get pod -n $tenant --context $cluster

In case you have two different clusters that are named the same, you can use az aks get-credentials own --context command param, and just pick a discriminator for your concurrent script executions:

$id = "$(new-guid)"
$contextName = "$cluster-$id"
az aks get-credentials -n $cluster -g $rg --context $contextName
kubectl get pod -n $tenant --context $contextName

Note that adding --admin command param when fetching credentials will also append -admin to the names in the kubeconfig; thus forcing you to add "-admin" to the $contextName:

$contextName = "$cluster-$id-admin"

There is a limitation to this, in that az aks get-credentials doesn't allow us to override the parameter name for the generated user object (at least not to my knowledge, nor is it mentioned in the docs). Its format is "static" and will always be clusterUser_{resourceGroupName}_{clusterName} or clusterAdmin_{resourceGroupName}_{clusterName} if using --admin.

This is really only a concern if you have 2 different clusters that have the exact same name and resource group name (I'd question why this would ever be the case, but alas).

You'd most likely see 2 different outcomes here:

• "An object of same name already exist"-ish error, or
• The 2nd invocation of az aks get-credentials will override the existing object with the new values if you use --overwrite-existing command param, causing all API calls to the 1st invocation's cluster to fail with either 401 or 403 related errors (is my guess at least)

To counter this you can write to a specific kubeconfig file, again unique to the script's current context/execution, and then use kubectl's global --kubeconfig command param:

$id = "$(new-guid)"
$contextName = $kubeconfig = "$cluster-$id"
az aks get-credentials -n $cluster -g $rg --context $contextName --file $kubeconfig
kubectl get pod -n $tenant --context $contextName --kubeconfig $kubeconfig

Remember to add "-admin" to your $contextName, if you use --admin (and separate the $contextName and $kubeconfig assignments, obviously).

Answer 2

I think you are referring to concurrent powershell commands execution?

You can wrap those az login/k8s context setting/kubectl commands stuff into a script, and use powershell's builtin concurrency mechanism to get your job done.

PowerShell concurrently runs commands and scripts through jobs.

These docs or tutorial may be helpful:

1. PowerShell: about_Thread_Jobs
2. PowerShell Multithreading: A Deep Dive
3. SO: Can Powershell Run Commands in Parallel?