Fix anx/provider/sync crash and possible deadlock

[LittleFox94]

No idea if this fixes all issues, but at least it's not crashing anymore on startup (see attached log).

We should really get into a code review mode in this repository, too.

crashlog

{"level":"info","time":"2022-02-07T12:17:44.716Z","logger":"http-prober","caller":"http-prober/main.go:125","msg":"Probing","attempt":1,"max-attempts":100,"target":"https://apiserver-external.cluster-lvm8gwn7fj.svc.cluster.local./healthz"}
{"level":"info","time":"2022-02-07T12:17:44.721Z","logger":"http-prober","caller":"http-prober/main.go:114","msg":"Hostname resolved","hostname":"apiserver-external.cluster-lvm8gwn7fj.svc.cluster.local.","address":"10.202.31.1:443"}
{"level":"info","time":"2022-02-07T12:17:44.724Z","logger":"http-prober","caller":"http-prober/main.go:138","msg":"Endpoint is available"}
I0207 12:17:45.609083       1 serving.go:348] Generated self-signed cert in-memory
W0207 12:17:46.381555       1 authentication.go:419] failed to read in-cluster kubeconfig for delegated authentication: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W0207 12:17:46.381633       1 authentication.go:316] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W0207 12:17:46.381646       1 authentication.go:340] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W0207 12:17:46.381674       1 authorization.go:225] failed to read in-cluster kubeconfig for delegated authorization: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W0207 12:17:46.381708       1 authorization.go:193] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I0207 12:17:46.383851       1 controllermanager.go:143] Version: v0.0.0-master+$Format:%H$
I0207 12:17:46.384891       1 secure_serving.go:200] Serving securely on [::]:8080
I0207 12:17:46.384985       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0207 12:17:46.385368       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cloud-controller-manager...
I0207 12:18:04.398830       1 leaderelection.go:258] successfully acquired lease kube-system/cloud-controller-manager
I0207 12:18:04.400050       1 event.go:294] "Event occurred" object="kube-system/cloud-controller-manager" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="anx-cloud-controller-manager-6879bfb7b8-nt92h_08ca2462-795e-40fa-ab1b-69b0c28651aa became leader"
I0207 12:18:05.896636       1 provider.go:71] discovered load balancer '7bc62185ddd243f18029ffd107397b8a'
I0207 12:18:05.896655       1 provider.go:73] discovered load balancers for replication [c7ddd180ea524231940d0523ea0aa6b0]
I0207 12:18:05.896662       1 provider.go:81] Running with customer prefix ''
I0207 12:18:05.897016       1 controllermanager.go:297] Started "anx-lbaas-sync"
I0207 12:18:05.897308       1 node_controller.go:115] Sending events to api server.
I0207 12:18:05.897503       1 controllermanager.go:297] Started "cloud-node"
I0207 12:18:05.897652       1 node_controller.go:154] Waiting for informer caches to sync
I0207 12:18:05.897794       1 node_lifecycle_controller.go:76] Sending events to api server
I0207 12:18:05.897829       1 controllermanager.go:297] Started "cloud-node-lifecycle"
I0207 12:18:05.898213       1 controllermanager.go:297] Started "service"
W0207 12:18:05.898245       1 core.go:110] --configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0207 12:18:05.898255       1 controllermanager.go:285] Skipping "route"
I0207 12:18:05.898268       1 controller.go:230] Starting service controller
I0207 12:18:05.898322       1 shared_informer.go:240] Waiting for caches to sync for service
I0207 12:18:05.998567       1 shared_informer.go:247] Caches are synced for service 
I0207 12:18:05.998805       1 event.go:294] "Event occurred" object="hello-kubernetes/hello-kubernetes-hello-world" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0207 12:18:05.998824       1 event.go:294] "Event occurred" object="hello-kubernetes/hello-kubernetes-hello-world" kind="Service" apiVersion="v1" type="Warning" reason="UnAvailableLoadBalancer" message="There are no available nodes for LoadBalancer"
E0207 12:18:05.998830       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 365 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x19a8860, 0x2ddf700})
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x7d
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0001a05b0})
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x75
panic({0x19a8860, 0x2ddf700})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/anexia-it/anxcloud-cloud-controller-manager/anx/provider/sync.(*SubjectLock).Lock(0x0, {0xc0009ec000, 0x4e})
	/go/src/github.com/github.com/anexia-it/anxcloud-cloud-controller-manager/anx/provider/sync/sync.go:16 +0x4c
github.com/anexia-it/anxcloud-cloud-controller-manager/anx/provider.loadBalancerManager.EnsureLoadBalancer({{0x1ef3ed0, 0xc000136d20}, 0xc00089bf80, 0x0}, {0x1ece948, 0xc000128008}, {0x7fff05d8b75c, 0x20}, 0xc0004ee4b0, {0x0, ...})
	/go/src/github.com/github.com/anexia-it/anxcloud-cloud-controller-manager/anx/provider/load_balancer.go:75 +0x1b5
k8s.io/cloud-provider/controllers/service.(*Controller).ensureLoadBalancer(0xc0000d6000, 0xc0004ee4b0)
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:452 +0x103
k8s.io/cloud-provider/controllers/service.(*Controller).syncLoadBalancerIfNeeded(0xc0000d6000, 0xc0004ee4b0, {0xc0009241b0, 0x2d})
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:407 +0x5ed
k8s.io/cloud-provider/controllers/service.(*Controller).processServiceCreateOrUpdate(0xc0000d6000, 0xc0004ee4b0, {0xc0009241b0, 0x2d})
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:343 +0x110
k8s.io/cloud-provider/controllers/service.(*Controller).syncService(0xc0000d6000, {0xc0009241b0, 0x2d})
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:854 +0x213
k8s.io/cloud-provider/controllers/service.(*Controller).processNextWorkItem(0xc0000d6000)
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:301 +0x105
k8s.io/cloud-provider/controllers/service.(*Controller).worker(...)
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:280
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f3dc4aa4178)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x189d080, {0x1e95900, 0xc00090a1e0}, 0x1, 0xc0002cade0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x43f5c5)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0xc0002cade0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x25
created by k8s.io/cloud-provider/controllers/service.(*Controller).Run
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:238 +0x267
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17a53ec]

goroutine 365 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0001a05b0})
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x19a8860, 0x2ddf700})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/anexia-it/anxcloud-cloud-controller-manager/anx/provider/sync.(*SubjectLock).Lock(0x0, {0xc0009ec000, 0x4e})
	/go/src/github.com/github.com/anexia-it/anxcloud-cloud-controller-manager/anx/provider/sync/sync.go:16 +0x4c
github.com/anexia-it/anxcloud-cloud-controller-manager/anx/provider.loadBalancerManager.EnsureLoadBalancer({{0x1ef3ed0, 0xc000136d20}, 0xc00089bf80, 0x0}, {0x1ece948, 0xc000128008}, {0x7fff05d8b75c, 0x20}, 0xc0004ee4b0, {0x0, ...})
	/go/src/github.com/github.com/anexia-it/anxcloud-cloud-controller-manager/anx/provider/load_balancer.go:75 +0x1b5
k8s.io/cloud-provider/controllers/service.(*Controller).ensureLoadBalancer(0xc0000d6000, 0xc0004ee4b0)
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:452 +0x103
k8s.io/cloud-provider/controllers/service.(*Controller).syncLoadBalancerIfNeeded(0xc0000d6000, 0xc0004ee4b0, {0xc0009241b0, 0x2d})
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:407 +0x5ed
k8s.io/cloud-provider/controllers/service.(*Controller).processServiceCreateOrUpdate(0xc0000d6000, 0xc0004ee4b0, {0xc0009241b0, 0x2d})
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:343 +0x110
k8s.io/cloud-provider/controllers/service.(*Controller).syncService(0xc0000d6000, {0xc0009241b0, 0x2d})
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:854 +0x213
k8s.io/cloud-provider/controllers/service.(*Controller).processNextWorkItem(0xc0000d6000)
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:301 +0x105
k8s.io/cloud-provider/controllers/service.(*Controller).worker(...)
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:280
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f3dc4aa4178)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x189d080, {0x1e95900, 0xc00090a1e0}, 0x1, 0xc0002cade0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x43f5c5)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0xc0002cade0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x25
created by k8s.io/cloud-provider/controllers/service.(*Controller).Run
	/go/pkg/mod/k8s.io/[email protected]/controllers/service/controller.go:238 +0x267