Aug 26 11:09:55 groomlake k3s[20510]: time="2023-08-26T11:09:55-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:00 groomlake k3s[20510]: time="2023-08-26T11:10:00-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:00 groomlake k3s[20510]: time="2023-08-26T11:10:00-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:05 groomlake k3s[20510]: time="2023-08-26T11:10:05-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:05 groomlake k3s[20510]: time="2023-08-26T11:10:05-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:10 groomlake k3s[20510]: time="2023-08-26T11:10:10-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:10 groomlake k3s[20510]: time="2023-08-26T11:10:10-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:15 groomlake k3s[20510]: time="2023-08-26T11:10:15-04:00" level=info msg="Waiting for API server to become available" Aug 26 11:10:15 groomlake k3s[20510]: time="2023-08-26T11:10:15-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:15 groomlake k3s[20510]: time="2023-08-26T11:10:15-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:20 groomlake k3s[20510]: time="2023-08-26T11:10:20-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:20 groomlake k3s[20510]: {"level":"warn","ts":"2023-08-26T11:10:20.425-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00053c1c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Aug 26 11:10:20 groomlake k3s[20510]: time="2023-08-26T11:10:20-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:25 groomlake k3s[20510]: time="2023-08-26T11:10:25-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:25 groomlake k3s[20510]: time="2023-08-26T11:10:25-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:26 groomlake k3s[20510]: time="2023-08-26T11:10:26-04:00" level=info msg="Waiting for API server to become available" Aug 26 11:10:28 groomlake k3s[20510]: time="2023-08-26T11:10:28-04:00" level=fatal msg="cri-dockerd exited: " Aug 26 11:10:28 groomlake systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE Aug 26 11:10:28 groomlake systemd[1]: k3s.service: Failed with result 'exit-code'. Aug 26 11:10:28 groomlake systemd[1]: Stopped k3s.service - Lightweight Kubernetes. Aug 26 11:10:28 groomlake systemd[1]: k3s.service: Consumed 34min 25.547s CPU time. Aug 26 11:10:28 groomlake systemd[1]: Starting k3s.service - Lightweight Kubernetes... Aug 26 11:10:29 groomlake k3s[24836]: time="2023-08-26T11:10:29-04:00" level=info msg="Starting k3s v1.25.3+k3s-9afcd6b9-dirty (9afcd6b9)" Aug 26 11:10:29 groomlake k3s[24836]: time="2023-08-26T11:10:29-04:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s" Aug 26 11:10:29 groomlake k3s[24836]: time="2023-08-26T11:10:29-04:00" level=info msg="Configuring database table schema and indexes, this may take a moment..." Aug 26 11:10:29 groomlake k3s[24836]: time="2023-08-26T11:10:29-04:00" level=info msg="Database tables and indexes are up to date" Aug 26 11:10:32 groomlake k3s[24836]: time="2023-08-26T11:10:32-04:00" level=info msg="Kine available at unix://kine.sock" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Reconciling bootstrap data between datastore and disk" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Running kube-apiserver --advertise-address=10.0.0.2 --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/log/k3s_server_audit.log --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/mnt/data/ix-applications/k3s/server/tls/temporary-certs --client-ca-file=/mnt/data/ix-applications/k3s/server/tls/client-ca.crt --egress-selector-config-file=/mnt/data/ix-applications/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,NamespaceLifecycle,ServiceAccount --enable-aggregator-routing=true --etcd-servers=unix://kine.sock --feature-gates=MixedProtocolLBService=true --kubelet-certificate-authority=/mnt/data/ix-applications/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/mnt/data/ix-applications/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/mnt/data/ix-applications/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/mnt/data/ix-applications/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/mnt/data/ix-applications/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/mnt/data/ix-applications/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/mnt/data/ix-applications/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/mnt/data/ix-applications/k3s/server/tls/service.key --service-cluster-ip-range=172.17.0.0/16 --service-node-port-range=9000-65535 --storage-backend=etcd3 --tls-cert-file=/mnt/data/ix-applications/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/mnt/data/ix-applications/k3s/server/tls/serving-kube-apiserver.key" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Tunnel server egress proxy mode: agent" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/mnt/data/ix-applications/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/mnt/data/ix-applications/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/mnt/data/ix-applications/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/mnt/data/ix-applications/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/mnt/data/ix-applications/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=172.16.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/mnt/data/ix-applications/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/mnt/data/ix-applications/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/mnt/data/ix-applications/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/mnt/data/ix-applications/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/mnt/data/ix-applications/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/mnt/data/ix-applications/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/mnt/data/ix-applications/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/mnt/data/ix-applications/k3s/server/tls/server-ca.key --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/mnt/data/ix-applications/k3s/server/cred/controller.kubeconfig --leader-elect=false --node-cidr-mask-size=16 --profiling=false --root-ca-file=/mnt/data/ix-applications/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/mnt/data/ix-applications/k3s/server/tls/service.key --service-cluster-ip-range=172.17.0.0/16 --terminated-pod-gc-threshold=5 --use-service-account-credentials=true" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/mnt/data/ix-applications/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/mnt/data/ix-applications/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/mnt/data/ix-applications/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=172.16.0.0/16 --configure-cloud-routes=false --controllers=*,-route,-cloud-node,-cloud-node-lifecycle --kubeconfig=/mnt/data/ix-applications/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Waiting for API server to become available" Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.204062 24836 server.go:581] external host was not specified, using 10.0.0.2 Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Server node token is available at /mnt/data/ix-applications/k3s/server/token" Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.204267 24836 server.go:171] Version: v1.25.3+k3s-9afcd6b9-dirty Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.204280 24836 server.go:173] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="To join server node to cluster: k3s server -s https://10.0.0.2:6443 -t ${SERVER_NODE_TOKEN}" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Agent node token is available at /mnt/data/ix-applications/k3s/server/agent-token" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="To join agent node to cluster: k3s agent -s https://10.0.0.2:6443 -t ${AGENT_NODE_TOKEN}" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Run: k3s kubectl" Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.218554 24836 shared_informer.go:255] Waiting for caches to sync for node_authorizer Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.219475 24836 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.219491 24836 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.220532 24836 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.220545 24836 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. Aug 26 11:10:34 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:10:34.223-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c936c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Aug 26 11:10:34 groomlake k3s[24836]: W0826 11:10:34.245536 24836 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.246436 24836 instance.go:261] Using reconciler: lease Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="certificate CN=ix-truenas signed by CN=k3s-server-ca@1673211341: notBefore=2023-01-08 20:55:41 +0000 UTC notAfter=2024-08-25 15:10:34 +0000 UTC" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="certificate CN=system:node:ix-truenas,O=system:nodes signed by CN=k3s-client-ca@1673211341: notBefore=2023-01-08 20:55:41 +0000 UTC notAfter=2024-08-25 15:10:34 +0000 UTC" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Module overlay was already loaded" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Module nf_conntrack was already loaded" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Module br_netfilter was already loaded" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Module iptable_nat was already loaded" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Running cri-dockerd --container-runtime-endpoint=unix:///run/k3s/cri-dockerd/cri-dockerd.sock --cri-dockerd-root-directory=/mnt/data/ix-applications/k3s/agent/cri-dockerd --network-plugin=cni --pod-infra-container-image=rancher/mirrored-pause:3.6" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/mnt/data/ix-applications/k3s/agent/client-ca.crt --cluster-dns=172.17.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/cri-dockerd/cri-dockerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ix-truenas --kubeconfig=/mnt/data/ix-applications/k3s/agent/kubelet.kubeconfig --max-pods=250 --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/mnt/data/ix-applications/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/mnt/data/ix-applications/k3s/agent/serving-kubelet.crt --tls-private-key-file=/mnt/data/ix-applications/k3s/agent/serving-kubelet.key" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Connecting to docker on the Endpoint unix:///var/run/docker.sock" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Start docker client with request timeout 0s" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Handling backend connection request [ix-truenas]" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Hairpin mode is set to none" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Loaded network plugin cni" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Docker cri networking managed by network plugin cni" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Docker Info: &{ID:3DFC:3DRY:7CTQ:OODJ:AG6Z:VS2F:NVUQ:G5DX:ZZRI:FKDO:EWPX:LQNG Containers:118 ContainersRunning:0 ContainersPaused:0 ContainersStopped:118 Images:55 Driver:overlay2 DriverStatus:[[Backing Filesystem zfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:27 OomKillDisable:false NGoroutines:40 SystemTime:2023-08-26T11:10:34.373871071-04:00 LoggingDriver:json-file CgroupDriver:cgroupfs CgroupVersion:2 NEventsListener:0 KernelVersion:5.15.107+truenas OperatingSystem:Debian GNU/Linux 11 (bullseye) OSVersion:11 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc000778070 NCPU:4 MemTotal:50292404224 GenericResources:[] DockerRootDir:/mnt/data/ix-applications/docker HTTPProxy: HTTPSProxy: NoProxy: Name:groomlake Labels:[] ExperimentalBuild:false ServerVersion:dev ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:} runc:{Path:runc Args:[] Shim:}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster: Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6 Expected:9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6} RuncCommit:{ID:v1.1.4-0-g5fd4c4d Expected:v1.1.4-0-g5fd4c4d} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=cgroupns] ProductLicense: DefaultAddressPools:[] Warnings:[]}" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Setting cgroupDriver cgroupfs" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:,},}" Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Starting the GRPC backend for the Docker CRI interface." Aug 26 11:10:34 groomlake k3s[24836]: time="2023-08-26T11:10:34-04:00" level=info msg="Start cri-dockerd grpc backend" Aug 26 11:10:34 groomlake k3s[24836]: I0826 11:10:34.755518 24836 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.030874 24836 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.046713 24836 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.087594 24836 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.167619 24836 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.199403 24836 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.213552 24836 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.213622 24836 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.361929 24836 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.361961 24836 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.417750 24836 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.417778 24836 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.421365 24836 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.425893 24836 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.425925 24836 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.427486 24836 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.427513 24836 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.835267 24836 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources. Aug 26 11:10:37 groomlake k3s[24836]: W0826 11:10:37.974442 24836 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. Aug 26 11:10:38 groomlake k3s[24836]: W0826 11:10:38.102982 24836 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources. Aug 26 11:10:38 groomlake k3s[24836]: W0826 11:10:38.103015 24836 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources. Aug 26 11:10:38 groomlake k3s[24836]: W0826 11:10:38.132286 24836 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:38 groomlake k3s[24836]: W0826 11:10:38.294105 24836 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:38 groomlake k3s[24836]: I0826 11:10:38.296060 24836 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. Aug 26 11:10:38 groomlake k3s[24836]: I0826 11:10:38.296283 24836 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. Aug 26 11:10:38 groomlake k3s[24836]: W0826 11:10:38.496919 24836 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources. Aug 26 11:10:39 groomlake k3s[24836]: time="2023-08-26T11:10:39-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:39 groomlake k3s[24836]: time="2023-08-26T11:10:39-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:44 groomlake k3s[24836]: time="2023-08-26T11:10:44-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:44 groomlake k3s[24836]: time="2023-08-26T11:10:44-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:49 groomlake k3s[24836]: time="2023-08-26T11:10:49-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:49 groomlake k3s[24836]: time="2023-08-26T11:10:49-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:52 groomlake k3s[24836]: I0826 11:10:52.689530 24836 trace.go:205] Trace[2077530441]: "List(recursive=true) etcd3" audit-id:,key:/persistentvolumeclaims,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.277) (total time: 18412ms): Aug 26 11:10:52 groomlake k3s[24836]: Trace[2077530441]: [18.412150144s] [18.412150144s] END Aug 26 11:10:52 groomlake k3s[24836]: I0826 11:10:52.689736 24836 trace.go:205] Trace[280406337]: "Reflector ListAndWatch" name:storage/cacher.go:/persistentvolumeclaims (26-Aug-2023 11:10:34.277) (total time: 18412ms): Aug 26 11:10:52 groomlake k3s[24836]: Trace[280406337]: ---"Objects listed" error: 18412ms (11:10:52.689) Aug 26 11:10:52 groomlake k3s[24836]: Trace[280406337]: [18.412391359s] [18.412391359s] END Aug 26 11:10:52 groomlake k3s[24836]: I0826 11:10:52.690979 24836 trace.go:205] Trace[477227429]: "List(recursive=true) etcd3" audit-id:,key:/flowschemas,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.688) (total time: 17001ms): Aug 26 11:10:52 groomlake k3s[24836]: Trace[477227429]: [17.001983238s] [17.001983238s] END Aug 26 11:10:52 groomlake k3s[24836]: I0826 11:10:52.691304 24836 trace.go:205] Trace[1175305183]: "Reflector ListAndWatch" name:storage/cacher.go:/flowschemas (26-Aug-2023 11:10:35.688) (total time: 17002ms): Aug 26 11:10:52 groomlake k3s[24836]: Trace[1175305183]: ---"Objects listed" error: 17002ms (11:10:52.691) Aug 26 11:10:52 groomlake k3s[24836]: Trace[1175305183]: [17.002351973s] [17.002351973s] END Aug 26 11:10:52 groomlake k3s[24836]: I0826 11:10:52.787376 24836 trace.go:205] Trace[519906557]: "List(recursive=true) etcd3" audit-id:,key:/services/specs,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.373) (total time: 18413ms): Aug 26 11:10:52 groomlake k3s[24836]: Trace[519906557]: [18.413553733s] [18.413553733s] END Aug 26 11:10:52 groomlake k3s[24836]: I0826 11:10:52.787495 24836 trace.go:205] Trace[1189953814]: "Reflector ListAndWatch" name:storage/cacher.go:/services/specs (26-Aug-2023 11:10:34.373) (total time: 18413ms): Aug 26 11:10:52 groomlake k3s[24836]: Trace[1189953814]: ---"Objects listed" error: 18413ms (11:10:52.787) Aug 26 11:10:52 groomlake k3s[24836]: Trace[1189953814]: [18.413716121s] [18.413716121s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.294188 24836 trace.go:205] Trace[1108766874]: "List(recursive=true) etcd3" audit-id:,key:/pods,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.354) (total time: 18939ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[1108766874]: [18.939822544s] [18.939822544s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.294402 24836 trace.go:205] Trace[1882592071]: "Reflector ListAndWatch" name:storage/cacher.go:/pods (26-Aug-2023 11:10:34.354) (total time: 18940ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[1882592071]: ---"Objects listed" error: 18939ms (11:10:53.294) Aug 26 11:10:53 groomlake k3s[24836]: Trace[1882592071]: [18.940078452s] [18.940078452s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.565097 24836 trace.go:205] Trace[1881542403]: "List(recursive=true) etcd3" audit-id:,key:/minions,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.303) (total time: 19261ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[1881542403]: [19.261906931s] [19.261906931s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.565191 24836 trace.go:205] Trace[1544777671]: "Reflector ListAndWatch" name:storage/cacher.go:/minions (26-Aug-2023 11:10:34.303) (total time: 19262ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[1544777671]: ---"Objects listed" error: 19262ms (11:10:53.565) Aug 26 11:10:53 groomlake k3s[24836]: Trace[1544777671]: [19.262044347s] [19.262044347s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.572415 24836 trace.go:205] Trace[149591074]: "List(recursive=true) etcd3" audit-id:,key:/persistentvolumes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.271) (total time: 19300ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[149591074]: [19.300827959s] [19.300827959s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.572554 24836 trace.go:205] Trace[804958354]: "Reflector ListAndWatch" name:storage/cacher.go:/persistentvolumes (26-Aug-2023 11:10:34.271) (total time: 19301ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[804958354]: ---"Objects listed" error: 19300ms (11:10:53.572) Aug 26 11:10:53 groomlake k3s[24836]: Trace[804958354]: [19.30102119s] [19.30102119s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.657854 24836 trace.go:205] Trace[35914629]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.269) (total time: 19388ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[35914629]: [19.388636059s] [19.388636059s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.712704 24836 trace.go:205] Trace[267443030]: "Reflector ListAndWatch" name:storage/cacher.go:/secrets (26-Aug-2023 11:10:34.269) (total time: 19443ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[267443030]: ---"Objects listed" error: 19443ms (11:10:53.712) Aug 26 11:10:53 groomlake k3s[24836]: Trace[267443030]: [19.443500478s] [19.443500478s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.669186 24836 trace.go:205] Trace[250466758]: "List(recursive=true) etcd3" audit-id:,key:/configmaps,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.278) (total time: 19390ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[250466758]: [19.390229172s] [19.390229172s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.670221 24836 trace.go:205] Trace[1596170792]: "List(recursive=true) etcd3" audit-id:,key:/namespaces,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.293) (total time: 19376ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[1596170792]: [19.376457813s] [19.376457813s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.711449 24836 trace.go:205] Trace[65139546]: "List(recursive=true) etcd3" audit-id:,key:/ingress,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.025) (total time: 18685ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[65139546]: [18.685570343s] [18.685570343s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.720474 24836 trace.go:205] Trace[2106289140]: "Reflector ListAndWatch" name:storage/cacher.go:/ingress (26-Aug-2023 11:10:35.025) (total time: 18694ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[2106289140]: ---"Objects listed" error: 18694ms (11:10:53.720) Aug 26 11:10:53 groomlake k3s[24836]: Trace[2106289140]: [18.694629013s] [18.694629013s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.720332 24836 trace.go:205] Trace[1799591688]: "Reflector ListAndWatch" name:storage/cacher.go:/configmaps (26-Aug-2023 11:10:34.278) (total time: 19441ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[1799591688]: ---"Objects listed" error: 19441ms (11:10:53.720) Aug 26 11:10:53 groomlake k3s[24836]: Trace[1799591688]: [19.441391886s] [19.441391886s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.722145 24836 trace.go:205] Trace[332621045]: "Reflector ListAndWatch" name:storage/cacher.go:/namespaces (26-Aug-2023 11:10:34.293) (total time: 19428ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[332621045]: ---"Objects listed" error: 19426ms (11:10:53.720) Aug 26 11:10:53 groomlake k3s[24836]: Trace[332621045]: [19.428418346s] [19.428418346s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.887353 24836 trace.go:205] Trace[215339456]: "List(recursive=true) etcd3" audit-id:,key:/serviceaccounts,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.364) (total time: 19522ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[215339456]: [19.522765002s] [19.522765002s] END Aug 26 11:10:53 groomlake k3s[24836]: I0826 11:10:53.894699 24836 trace.go:205] Trace[600321419]: "Reflector ListAndWatch" name:storage/cacher.go:/serviceaccounts (26-Aug-2023 11:10:34.364) (total time: 19530ms): Aug 26 11:10:53 groomlake k3s[24836]: Trace[600321419]: ---"Objects listed" error: 19522ms (11:10:53.887) Aug 26 11:10:53 groomlake k3s[24836]: Trace[600321419]: [19.530117486s] [19.530117486s] END Aug 26 11:10:54 groomlake k3s[24836]: time="2023-08-26T11:10:54-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:54 groomlake k3s[24836]: time="2023-08-26T11:10:54-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:54 groomlake k3s[24836]: I0826 11:10:54.589046 24836 trace.go:205] Trace[1549951258]: "List(recursive=true) etcd3" audit-id:,key:/jobs,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.881) (total time: 19707ms): Aug 26 11:10:54 groomlake k3s[24836]: Trace[1549951258]: [19.707765369s] [19.707765369s] END Aug 26 11:10:54 groomlake k3s[24836]: I0826 11:10:54.732355 24836 trace.go:205] Trace[2062707552]: "Reflector ListAndWatch" name:storage/cacher.go:/jobs (26-Aug-2023 11:10:34.881) (total time: 19851ms): Aug 26 11:10:54 groomlake k3s[24836]: Trace[2062707552]: ---"Objects listed" error: 19851ms (11:10:54.732) Aug 26 11:10:54 groomlake k3s[24836]: Trace[2062707552]: [19.851103508s] [19.851103508s] END Aug 26 11:10:54 groomlake k3s[24836]: I0826 11:10:54.612643 24836 trace.go:205] Trace[1456937483]: "List(recursive=true) etcd3" audit-id:,key:/priorityclasses,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.373) (total time: 19238ms): Aug 26 11:10:54 groomlake k3s[24836]: Trace[1456937483]: [19.238942741s] [19.238942741s] END Aug 26 11:10:54 groomlake k3s[24836]: I0826 11:10:54.732977 24836 trace.go:205] Trace[739137541]: "Reflector ListAndWatch" name:storage/cacher.go:/priorityclasses (26-Aug-2023 11:10:35.373) (total time: 19359ms): Aug 26 11:10:54 groomlake k3s[24836]: Trace[739137541]: ---"Objects listed" error: 19359ms (11:10:54.732) Aug 26 11:10:54 groomlake k3s[24836]: Trace[739137541]: [19.359368833s] [19.359368833s] END Aug 26 11:10:55 groomlake k3s[24836]: I0826 11:10:55.546362 24836 trace.go:205] Trace[1390570096]: "List(recursive=true) etcd3" audit-id:,key:/validatingwebhookconfigurations,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:36.086) (total time: 19459ms): Aug 26 11:10:55 groomlake k3s[24836]: Trace[1390570096]: [19.459718265s] [19.459718265s] END Aug 26 11:10:55 groomlake k3s[24836]: I0826 11:10:55.546470 24836 trace.go:205] Trace[1458158264]: "Reflector ListAndWatch" name:storage/cacher.go:/validatingwebhookconfigurations (26-Aug-2023 11:10:36.086) (total time: 19459ms): Aug 26 11:10:55 groomlake k3s[24836]: Trace[1458158264]: ---"Objects listed" error: 19459ms (11:10:55.546) Aug 26 11:10:55 groomlake k3s[24836]: Trace[1458158264]: [19.45986533s] [19.45986533s] END Aug 26 11:10:55 groomlake k3s[24836]: I0826 11:10:55.547076 24836 trace.go:205] Trace[1809413542]: "List(recursive=true) etcd3" audit-id:,key:/endpointslices,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.984) (total time: 20562ms): Aug 26 11:10:55 groomlake k3s[24836]: Trace[1809413542]: [20.562728386s] [20.562728386s] END Aug 26 11:10:55 groomlake k3s[24836]: I0826 11:10:55.558305 24836 trace.go:205] Trace[1910058332]: "Reflector ListAndWatch" name:storage/cacher.go:/endpointslices (26-Aug-2023 11:10:34.984) (total time: 20573ms): Aug 26 11:10:55 groomlake k3s[24836]: Trace[1910058332]: ---"Objects listed" error: 20573ms (11:10:55.558) Aug 26 11:10:55 groomlake k3s[24836]: Trace[1910058332]: [20.573949134s] [20.573949134s] END Aug 26 11:10:56 groomlake k3s[24836]: I0826 11:10:56.219997 24836 trace.go:205] Trace[131706808]: "List(recursive=true) etcd3" audit-id:,key:/services/endpoints,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.301) (total time: 21918ms): Aug 26 11:10:56 groomlake k3s[24836]: Trace[131706808]: [21.918050047s] [21.918050047s] END Aug 26 11:10:56 groomlake k3s[24836]: I0826 11:10:56.282221 24836 trace.go:205] Trace[328327752]: "Reflector ListAndWatch" name:storage/cacher.go:/services/endpoints (26-Aug-2023 11:10:34.301) (total time: 21980ms): Aug 26 11:10:56 groomlake k3s[24836]: Trace[328327752]: ---"Objects listed" error: 21980ms (11:10:56.282) Aug 26 11:10:56 groomlake k3s[24836]: Trace[328327752]: [21.980316403s] [21.980316403s] END Aug 26 11:10:56 groomlake k3s[24836]: I0826 11:10:56.671393 24836 trace.go:205] Trace[145442376]: "List(recursive=true) etcd3" audit-id:,key:/apiextensions.k8s.io/customresourcedefinitions,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.223) (total time: 22447ms): Aug 26 11:10:56 groomlake k3s[24836]: Trace[145442376]: [22.447707316s] [22.447707316s] END Aug 26 11:10:56 groomlake k3s[24836]: I0826 11:10:56.952664 24836 trace.go:205] Trace[1363300113]: "Reflector ListAndWatch" name:storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions (26-Aug-2023 11:10:34.223) (total time: 22728ms): Aug 26 11:10:56 groomlake k3s[24836]: Trace[1363300113]: ---"Objects listed" error: 22728ms (11:10:56.952) Aug 26 11:10:56 groomlake k3s[24836]: Trace[1363300113]: [22.728974326s] [22.728974326s] END Aug 26 11:10:56 groomlake k3s[24836]: I0826 11:10:56.966550 24836 trace.go:205] Trace[298437913]: "List(recursive=true) etcd3" audit-id:,key:/flowschemas,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.785) (total time: 21180ms): Aug 26 11:10:56 groomlake k3s[24836]: Trace[298437913]: [21.180778901s] [21.180778901s] END Aug 26 11:10:56 groomlake k3s[24836]: I0826 11:10:56.966847 24836 trace.go:205] Trace[1848546655]: "Reflector ListAndWatch" name:storage/cacher.go:/flowschemas (26-Aug-2023 11:10:35.785) (total time: 21181ms): Aug 26 11:10:56 groomlake k3s[24836]: Trace[1848546655]: ---"Objects listed" error: 21181ms (11:10:56.966) Aug 26 11:10:56 groomlake k3s[24836]: Trace[1848546655]: [21.181124388s] [21.181124388s] END Aug 26 11:10:56 groomlake k3s[24836]: I0826 11:10:56.967504 24836 trace.go:205] Trace[966071879]: "List(recursive=true) etcd3" audit-id:,key:/storageclasses,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.460) (total time: 21506ms): Aug 26 11:10:56 groomlake k3s[24836]: Trace[966071879]: [21.506795388s] [21.506795388s] END Aug 26 11:10:56 groomlake k3s[24836]: I0826 11:10:56.967745 24836 trace.go:205] Trace[383676445]: "Reflector ListAndWatch" name:storage/cacher.go:/storageclasses (26-Aug-2023 11:10:35.460) (total time: 21507ms): Aug 26 11:10:56 groomlake k3s[24836]: Trace[383676445]: ---"Objects listed" error: 21507ms (11:10:56.967) Aug 26 11:10:56 groomlake k3s[24836]: Trace[383676445]: [21.507069924s] [21.507069924s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.297613 24836 trace.go:205] Trace[926412496]: "List(recursive=true) etcd3" audit-id:,key:/prioritylevelconfigurations,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.720) (total time: 21576ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[926412496]: [21.576801702s] [21.576801702s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.298143 24836 trace.go:205] Trace[1838440048]: "Reflector ListAndWatch" name:storage/cacher.go:/prioritylevelconfigurations (26-Aug-2023 11:10:35.720) (total time: 21577ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[1838440048]: ---"Objects listed" error: 21577ms (11:10:57.298) Aug 26 11:10:57 groomlake k3s[24836]: Trace[1838440048]: [21.577354815s] [21.577354815s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.299861 24836 trace.go:205] Trace[972924603]: "List(recursive=true) etcd3" audit-id:,key:/roles,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.253) (total time: 22046ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[972924603]: [22.04676275s] [22.04676275s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.300157 24836 trace.go:205] Trace[573547426]: "Reflector ListAndWatch" name:storage/cacher.go:/roles (26-Aug-2023 11:10:35.253) (total time: 22047ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[573547426]: ---"Objects listed" error: 22047ms (11:10:57.300) Aug 26 11:10:57 groomlake k3s[24836]: Trace[573547426]: [22.047102668s] [22.047102668s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.303530 24836 trace.go:205] Trace[1561159223]: "List(recursive=true) etcd3" audit-id:,key:/daemonsets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.847) (total time: 21455ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[1561159223]: [21.455878352s] [21.455878352s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.303670 24836 trace.go:205] Trace[13997105]: "Reflector ListAndWatch" name:storage/cacher.go:/daemonsets (26-Aug-2023 11:10:35.847) (total time: 21456ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[13997105]: ---"Objects listed" error: 21455ms (11:10:57.303) Aug 26 11:10:57 groomlake k3s[24836]: Trace[13997105]: [21.456059218s] [21.456059218s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.344910 24836 secure_serving.go:210] Serving securely on 127.0.0.1:6444 Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.345039 24836 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/mnt/data/ix-applications/k3s/server/tls/request-header-ca.crt" Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.345636 24836 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/mnt/data/ix-applications/k3s/server/tls/serving-kube-apiserver.crt::/mnt/data/ix-applications/k3s/server/tls/serving-kube-apiserver.key" Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.347887 24836 customresource_discovery_controller.go:209] Starting DiscoveryController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.348306 24836 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/mnt/data/ix-applications/k3s/server/tls/client-auth-proxy.crt::/mnt/data/ix-applications/k3s/server/tls/client-auth-proxy.key" Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.352096 24836 tlsconfig.go:240] "Starting DynamicServingCertificateController" Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.352332 24836 controller.go:80] Starting OpenAPI V3 AggregationController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.352367 24836 apf_controller.go:300] Starting API Priority and Fairness config controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.352385 24836 available_controller.go:491] Starting AvailableConditionController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.352390 24836 cache.go:32] Waiting for caches to sync for AvailableConditionController controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.372495 24836 autoregister_controller.go:141] Starting autoregister controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.372532 24836 cache.go:32] Waiting for caches to sync for autoregister controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.372547 24836 controller.go:83] Starting OpenAPI AggregationController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.372583 24836 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.364840 24836 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.379400 24836 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.379439 24836 crdregistration_controller.go:111] Starting crd-autoregister controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.379446 24836 shared_informer.go:255] Waiting for caches to sync for crd-autoregister Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.364885 24836 apiservice_controller.go:97] Starting APIServiceRegistrationController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.379526 24836 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.364910 24836 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/mnt/data/ix-applications/k3s/server/tls/client-ca.crt" Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.387035 24836 trace.go:205] Trace[1229988281]: "List(recursive=true) etcd3" audit-id:,key:/clusterroles,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.285) (total time: 22101ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[1229988281]: [22.101810122s] [22.101810122s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.364972 24836 controller.go:85] Starting OpenAPI controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.364993 24836 controller.go:85] Starting OpenAPI V3 controller Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.365004 24836 naming_controller.go:291] Starting NamingConditionController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.365017 24836 establishing_controller.go:76] Starting EstablishingController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.365026 24836 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.365034 24836 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.365043 24836 crd_finalizer.go:266] Starting CRDFinalizer Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.365559 24836 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/mnt/data/ix-applications/k3s/server/tls/client-ca.crt" Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.365574 24836 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/mnt/data/ix-applications/k3s/server/tls/request-header-ca.crt" Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.366489 24836 trace.go:205] Trace[1927448130]: "List(recursive=true) etcd3" audit-id:,key:/csidrivers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.620) (total time: 21746ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[1927448130]: [21.746315509s] [21.746315509s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.367087 24836 trace.go:205] Trace[1701683851]: "List(recursive=true) etcd3" audit-id:,key:/clusterrolebindings,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.342) (total time: 22024ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[1701683851]: [22.02458653s] [22.02458653s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.532646 24836 trace.go:205] Trace[728327374]: "Reflector ListAndWatch" name:storage/cacher.go:/clusterroles (26-Aug-2023 11:10:35.285) (total time: 22247ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[728327374]: ---"Objects listed" error: 22247ms (11:10:57.532) Aug 26 11:10:57 groomlake k3s[24836]: Trace[728327374]: [22.247452597s] [22.247452597s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.584534 24836 trace.go:205] Trace[828682827]: "Reflector ListAndWatch" name:storage/cacher.go:/csidrivers (26-Aug-2023 11:10:35.620) (total time: 21964ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[828682827]: ---"Objects listed" error: 21964ms (11:10:57.584) Aug 26 11:10:57 groomlake k3s[24836]: Trace[828682827]: [21.964382156s] [21.964382156s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.586182 24836 trace.go:205] Trace[949302945]: "Reflector ListAndWatch" name:storage/cacher.go:/clusterrolebindings (26-Aug-2023 11:10:35.342) (total time: 22243ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[949302945]: ---"Objects listed" error: 22243ms (11:10:57.586) Aug 26 11:10:57 groomlake k3s[24836]: Trace[949302945]: [22.243718127s] [22.243718127s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.668228 24836 trace.go:205] Trace[644119000]: "List(recursive=true) etcd3" audit-id:,key:/replicasets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.858) (total time: 21809ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[644119000]: [21.809977798s] [21.809977798s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.670666 24836 trace.go:205] Trace[1735183058]: "Reflector ListAndWatch" name:storage/cacher.go:/replicasets (26-Aug-2023 11:10:35.858) (total time: 21812ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[1735183058]: ---"Objects listed" error: 21810ms (11:10:57.668) Aug 26 11:10:57 groomlake k3s[24836]: Trace[1735183058]: [21.812442329s] [21.812442329s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.676610 24836 trace.go:205] Trace[217409607]: "List(recursive=true) etcd3" audit-id:,key:/deployments,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.845) (total time: 21831ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[217409607]: [21.831405854s] [21.831405854s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.676730 24836 trace.go:205] Trace[1638515359]: "Reflector ListAndWatch" name:storage/cacher.go:/deployments (26-Aug-2023 11:10:35.845) (total time: 21831ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[1638515359]: ---"Objects listed" error: 21831ms (11:10:57.676) Aug 26 11:10:57 groomlake k3s[24836]: Trace[1638515359]: [21.831566681s] [21.831566681s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.677016 24836 trace.go:205] Trace[263521894]: "List(recursive=true) etcd3" audit-id:,key:/csinodes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.526) (total time: 22150ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[263521894]: [22.1505039s] [22.1505039s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.677149 24836 trace.go:205] Trace[2097266420]: "Reflector ListAndWatch" name:storage/cacher.go:/csinodes (26-Aug-2023 11:10:35.526) (total time: 22150ms): Aug 26 11:10:57 groomlake k3s[24836]: Trace[2097266420]: ---"Objects listed" error: 22150ms (11:10:57.677) Aug 26 11:10:57 groomlake k3s[24836]: Trace[2097266420]: [22.15065957s] [22.15065957s] END Aug 26 11:10:57 groomlake k3s[24836]: I0826 11:10:57.685451 24836 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.124254 24836 shared_informer.go:262] Caches are synced for crd-autoregister Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.787465 24836 trace.go:205] Trace[1448844093]: "List(recursive=true) etcd3" audit-id:,key:/controllerrevisions,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.898) (total time: 22889ms): Aug 26 11:10:58 groomlake k3s[24836]: Trace[1448844093]: [22.889017254s] [22.889017254s] END Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.787574 24836 trace.go:205] Trace[67073254]: "Reflector ListAndWatch" name:storage/cacher.go:/controllerrevisions (26-Aug-2023 11:10:35.898) (total time: 22889ms): Aug 26 11:10:58 groomlake k3s[24836]: Trace[67073254]: ---"Objects listed" error: 22889ms (11:10:58.787) Aug 26 11:10:58 groomlake k3s[24836]: Trace[67073254]: [22.889160012s] [22.889160012s] END Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.858694 24836 trace.go:205] Trace[497900530]: "List(recursive=true) etcd3" audit-id:,key:/prioritylevelconfigurations,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.823) (total time: 23035ms): Aug 26 11:10:58 groomlake k3s[24836]: Trace[497900530]: [23.035430429s] [23.035430429s] END Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.858798 24836 trace.go:205] Trace[1459779736]: "Reflector ListAndWatch" name:storage/cacher.go:/prioritylevelconfigurations (26-Aug-2023 11:10:35.823) (total time: 23035ms): Aug 26 11:10:58 groomlake k3s[24836]: Trace[1459779736]: ---"Objects listed" error: 23035ms (11:10:58.858) Aug 26 11:10:58 groomlake k3s[24836]: Trace[1459779736]: [23.035569668s] [23.035569668s] END Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.959921 24836 trace.go:205] Trace[731989647]: "List(recursive=true) etcd3" audit-id:,key:/leases,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.945) (total time: 24014ms): Aug 26 11:10:58 groomlake k3s[24836]: Trace[731989647]: [24.014484396s] [24.014484396s] END Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.960236 24836 trace.go:205] Trace[479397324]: "Reflector ListAndWatch" name:storage/cacher.go:/leases (26-Aug-2023 11:10:34.945) (total time: 24014ms): Aug 26 11:10:58 groomlake k3s[24836]: Trace[479397324]: ---"Objects listed" error: 24014ms (11:10:58.960) Aug 26 11:10:58 groomlake k3s[24836]: Trace[479397324]: [24.014842542s] [24.014842542s] END Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.961223 24836 trace.go:205] Trace[761931217]: "List(recursive=true) etcd3" audit-id:,key:/rolebindings,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.273) (total time: 23687ms): Aug 26 11:10:58 groomlake k3s[24836]: Trace[761931217]: [23.687365008s] [23.687365008s] END Aug 26 11:10:58 groomlake k3s[24836]: I0826 11:10:58.973343 24836 trace.go:205] Trace[256695279]: "Reflector ListAndWatch" name:storage/cacher.go:/rolebindings (26-Aug-2023 11:10:35.273) (total time: 23699ms): Aug 26 11:10:58 groomlake k3s[24836]: Trace[256695279]: ---"Objects listed" error: 23699ms (11:10:58.973) Aug 26 11:10:58 groomlake k3s[24836]: Trace[256695279]: [23.699501198s] [23.699501198s] END Aug 26 11:10:59 groomlake k3s[24836]: time="2023-08-26T11:10:59-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:10:59 groomlake k3s[24836]: time="2023-08-26T11:10:59-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:10:59 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:10:59.545-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0044c8fc0/kine.sock","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Aug 26 11:11:01 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:11:01.604-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0044c9180/kine.sock","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Aug 26 11:11:02 groomlake k3s[24836]: I0826 11:11:02.247012 24836 trace.go:205] Trace[1770927702]: "List(recursive=true) etcd3" audit-id:,key:/statefulsets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.846) (total time: 26400ms): Aug 26 11:11:02 groomlake k3s[24836]: Trace[1770927702]: [26.400879198s] [26.400879198s] END Aug 26 11:11:02 groomlake k3s[24836]: I0826 11:11:02.247146 24836 trace.go:205] Trace[300814788]: "Reflector ListAndWatch" name:storage/cacher.go:/statefulsets (26-Aug-2023 11:10:35.846) (total time: 26401ms): Aug 26 11:11:02 groomlake k3s[24836]: Trace[300814788]: ---"Objects listed" error: 26400ms (11:11:02.247) Aug 26 11:11:02 groomlake k3s[24836]: Trace[300814788]: [26.401052436s] [26.401052436s] END Aug 26 11:11:03 groomlake k3s[24836]: I0826 11:11:03.999118 24836 trace.go:205] Trace[1586129319]: "List(recursive=true) etcd3" audit-id:,key:/apiregistration.k8s.io/apiservices,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:38.303) (total time: 25695ms): Aug 26 11:11:03 groomlake k3s[24836]: Trace[1586129319]: [25.695847695s] [25.695847695s] END Aug 26 11:11:03 groomlake k3s[24836]: I0826 11:11:03.999258 24836 trace.go:205] Trace[1246147008]: "Reflector ListAndWatch" name:storage/cacher.go:/apiregistration.k8s.io/apiservices (26-Aug-2023 11:10:38.303) (total time: 25696ms): Aug 26 11:11:03 groomlake k3s[24836]: Trace[1246147008]: ---"Objects listed" error: 25695ms (11:11:03.999) Aug 26 11:11:03 groomlake k3s[24836]: Trace[1246147008]: [25.696034131s] [25.696034131s] END Aug 26 11:11:04 groomlake k3s[24836]: time="2023-08-26T11:11:04-04:00" level=info msg="Waiting for API server to become available" Aug 26 11:11:04 groomlake k3s[24836]: time="2023-08-26T11:11:04-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:11:04 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:11:04.539-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0044c8fc0/kine.sock","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Aug 26 11:11:04 groomlake k3s[24836]: I0826 11:11:04.585630 24836 trace.go:205] Trace[229217091]: "List(recursive=true) etcd3" audit-id:,key:/mutatingwebhookconfigurations,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:36.098) (total time: 28486ms): Aug 26 11:11:04 groomlake k3s[24836]: Trace[229217091]: [28.4868736s] [28.4868736s] END Aug 26 11:11:04 groomlake k3s[24836]: I0826 11:11:04.585725 24836 trace.go:205] Trace[748367288]: "Reflector ListAndWatch" name:storage/cacher.go:/mutatingwebhookconfigurations (26-Aug-2023 11:10:36.098) (total time: 28487ms): Aug 26 11:11:04 groomlake k3s[24836]: Trace[748367288]: ---"Objects listed" error: 28486ms (11:11:04.585) Aug 26 11:11:04 groomlake k3s[24836]: Trace[748367288]: [28.487014112s] [28.487014112s] END Aug 26 11:11:04 groomlake k3s[24836]: time="2023-08-26T11:11:04-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:11:06 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:11:06.539-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0044c9180/kine.sock","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Aug 26 11:11:06 groomlake k3s[24836]: I0826 11:11:06.956995 24836 trace.go:205] Trace[1485116800]: "List(recursive=true) etcd3" audit-id:,key:/horizontalpodautoscalers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.860) (total time: 32096ms): Aug 26 11:11:06 groomlake k3s[24836]: Trace[1485116800]: [32.096050181s] [32.096050181s] END Aug 26 11:11:06 groomlake k3s[24836]: I0826 11:11:06.957088 24836 trace.go:205] Trace[1032040646]: "Reflector ListAndWatch" name:storage/cacher.go:/horizontalpodautoscalers (26-Aug-2023 11:10:34.860) (total time: 32096ms): Aug 26 11:11:06 groomlake k3s[24836]: Trace[1032040646]: ---"Objects listed" error: 32096ms (11:11:06.957) Aug 26 11:11:06 groomlake k3s[24836]: Trace[1032040646]: [32.09618218s] [32.09618218s] END Aug 26 11:11:07 groomlake k3s[24836]: I0826 11:11:07.132635 24836 trace.go:205] Trace[105700804]: "List(recursive=true) etcd3" audit-id:,key:/podtemplates,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.250) (total time: 32882ms): Aug 26 11:11:07 groomlake k3s[24836]: Trace[105700804]: [32.8822233s] [32.8822233s] END Aug 26 11:11:07 groomlake k3s[24836]: I0826 11:11:07.136578 24836 trace.go:205] Trace[1293071828]: "Reflector ListAndWatch" name:storage/cacher.go:/podtemplates (26-Aug-2023 11:10:34.250) (total time: 32886ms): Aug 26 11:11:07 groomlake k3s[24836]: Trace[1293071828]: ---"Objects listed" error: 32886ms (11:11:07.136) Aug 26 11:11:07 groomlake k3s[24836]: Trace[1293071828]: [32.886180686s] [32.886180686s] END Aug 26 11:11:07 groomlake k3s[24836]: I0826 11:11:07.321695 24836 trace.go:205] Trace[1392279297]: "Get" url:/api/v1/namespaces/default,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:1131d44a-c99a-43c0-b4c2-e5b09f0fca90,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:57.520) (total time: 9801ms): Aug 26 11:11:07 groomlake k3s[24836]: Trace[1392279297]: ---"About to write a response" 9801ms (11:11:07.321) Aug 26 11:11:07 groomlake k3s[24836]: Trace[1392279297]: [9.801161497s] [9.801161497s] END Aug 26 11:11:07 groomlake k3s[24836]: I0826 11:11:07.589202 24836 trace.go:205] Trace[1451781986]: "List(recursive=true) etcd3" audit-id:,key:/horizontalpodautoscalers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.861) (total time: 32727ms): Aug 26 11:11:07 groomlake k3s[24836]: Trace[1451781986]: [32.727552556s] [32.727552556s] END Aug 26 11:11:07 groomlake k3s[24836]: I0826 11:11:07.590932 24836 trace.go:205] Trace[1186157591]: "Reflector ListAndWatch" name:storage/cacher.go:/horizontalpodautoscalers (26-Aug-2023 11:10:34.861) (total time: 32729ms): Aug 26 11:11:07 groomlake k3s[24836]: Trace[1186157591]: ---"Objects listed" error: 32729ms (11:11:07.590) Aug 26 11:11:07 groomlake k3s[24836]: Trace[1186157591]: [32.729289085s] [32.729289085s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.068702 24836 trace.go:205] Trace[789311124]: "List(recursive=true) etcd3" audit-id:,key:/resourcequotas,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.261) (total time: 33807ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[789311124]: [33.807115233s] [33.807115233s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.068800 24836 trace.go:205] Trace[1518278528]: "Reflector ListAndWatch" name:storage/cacher.go:/resourcequotas (26-Aug-2023 11:10:34.261) (total time: 33807ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[1518278528]: ---"Objects listed" error: 33807ms (11:11:08.068) Aug 26 11:11:08 groomlake k3s[24836]: Trace[1518278528]: [33.807307357s] [33.807307357s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.185636 24836 trace.go:205] Trace[1589597757]: "List(recursive=true) etcd3" audit-id:57c7517c-73e2-4976-9dc3-83d599a3ecec,key:/clusterroles,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (26-Aug-2023 11:10:57.522) (total time: 10663ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[1589597757]: [10.663504372s] [10.663504372s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.193147 24836 trace.go:205] Trace[282573116]: "List" url:/apis/rbac.authorization.k8s.io/v1/clusterroles,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:57c7517c-73e2-4976-9dc3-83d599a3ecec,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:57.522) (total time: 10671ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[282573116]: ---"Listing from storage done" 10670ms (11:11:08.192) Aug 26 11:11:08 groomlake k3s[24836]: Trace[282573116]: [10.671025326s] [10.671025326s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.199340 24836 trace.go:205] Trace[475543782]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.25.3-k3s1/tools/cache/reflector.go:169 (26-Aug-2023 11:10:57.373) (total time: 10826ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[475543782]: ---"Objects listed" error: 10825ms (11:11:08.199) Aug 26 11:11:08 groomlake k3s[24836]: Trace[475543782]: [10.826061681s] [10.826061681s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.587387 24836 trace.go:205] Trace[716187826]: "List(recursive=true) etcd3" audit-id:,key:/horizontalpodautoscalers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.844) (total time: 33742ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[716187826]: [33.742498488s] [33.742498488s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.587626 24836 trace.go:205] Trace[1377302956]: "Reflector ListAndWatch" name:storage/cacher.go:/horizontalpodautoscalers (26-Aug-2023 11:10:34.844) (total time: 33742ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[1377302956]: ---"Objects listed" error: 33742ms (11:11:08.587) Aug 26 11:11:08 groomlake k3s[24836]: Trace[1377302956]: [33.742774144s] [33.742774144s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.772370 24836 trace.go:205] Trace[57912441]: "List(recursive=true) etcd3" audit-id:,key:/csistoragecapacities,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.433) (total time: 33339ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[57912441]: [33.339236619s] [33.339236619s] END Aug 26 11:11:08 groomlake k3s[24836]: I0826 11:11:08.772993 24836 trace.go:205] Trace[1429503644]: "Reflector ListAndWatch" name:storage/cacher.go:/csistoragecapacities (26-Aug-2023 11:10:35.433) (total time: 33339ms): Aug 26 11:11:08 groomlake k3s[24836]: Trace[1429503644]: ---"Objects listed" error: 33339ms (11:11:08.772) Aug 26 11:11:08 groomlake k3s[24836]: Trace[1429503644]: [33.339872668s] [33.339872668s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.166364 24836 trace.go:205] Trace[1537587198]: "List(recursive=true) etcd3" audit-id:,key:/controllers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.369) (total time: 34796ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1537587198]: [34.796335038s] [34.796335038s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.166639 24836 trace.go:205] Trace[933160173]: "Reflector ListAndWatch" name:storage/cacher.go:/controllers (26-Aug-2023 11:10:34.369) (total time: 34796ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[933160173]: ---"Objects listed" error: 34796ms (11:11:09.166) Aug 26 11:11:09 groomlake k3s[24836]: Trace[933160173]: [34.796645769s] [34.796645769s] END Aug 26 11:11:09 groomlake k3s[24836]: time="2023-08-26T11:11:09-04:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.262080 24836 trace.go:205] Trace[53445675]: "List(recursive=true) etcd3" audit-id:,key:/certificatesigningrequests,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.904) (total time: 34357ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[53445675]: [34.357195761s] [34.357195761s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.262347 24836 trace.go:205] Trace[1774328108]: "Reflector ListAndWatch" name:storage/cacher.go:/certificatesigningrequests (26-Aug-2023 11:10:34.904) (total time: 34357ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1774328108]: ---"Objects listed" error: 34357ms (11:11:09.262) Aug 26 11:11:09 groomlake k3s[24836]: Trace[1774328108]: [34.357500446s] [34.357500446s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.265244 24836 trace.go:205] Trace[59728306]: "List(recursive=true) etcd3" audit-id:,key:/runtimeclasses,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.113) (total time: 34151ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[59728306]: [34.151409939s] [34.151409939s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.265318 24836 trace.go:205] Trace[2026394218]: "Reflector ListAndWatch" name:storage/cacher.go:/runtimeclasses (26-Aug-2023 11:10:35.113) (total time: 34151ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[2026394218]: ---"Objects listed" error: 34151ms (11:11:09.265) Aug 26 11:11:09 groomlake k3s[24836]: Trace[2026394218]: [34.151545828s] [34.151545828s] END Aug 26 11:11:09 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:11:09.271-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0044c8fc0/kine.sock","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Aug 26 11:11:09 groomlake k3s[24836]: time="2023-08-26T11:11:09-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error" Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.464866 24836 trace.go:205] Trace[5100790]: "List(recursive=true) etcd3" audit-id:,key:/limitranges,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.253) (total time: 35211ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[5100790]: [35.211312841s] [35.211312841s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.465846 24836 trace.go:205] Trace[320546264]: "Reflector ListAndWatch" name:storage/cacher.go:/limitranges (26-Aug-2023 11:10:34.253) (total time: 35212ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[320546264]: ---"Objects listed" error: 35212ms (11:11:09.465) Aug 26 11:11:09 groomlake k3s[24836]: Trace[320546264]: [35.212471431s] [35.212471431s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.466151 24836 trace.go:205] Trace[1419693497]: "List(recursive=true) etcd3" audit-id:,key:/volumeattachments,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.482) (total time: 33983ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1419693497]: [33.983160523s] [33.983160523s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.466316 24836 trace.go:205] Trace[2039892646]: "Reflector ListAndWatch" name:storage/cacher.go:/volumeattachments (26-Aug-2023 11:10:35.482) (total time: 33983ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[2039892646]: ---"Objects listed" error: 33983ms (11:11:09.466) Aug 26 11:11:09 groomlake k3s[24836]: Trace[2039892646]: [33.983353283s] [33.983353283s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.466494 24836 trace.go:205] Trace[2125942606]: "List(recursive=true) etcd3" audit-id:,key:/networkpolicies,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.988) (total time: 34477ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[2125942606]: [34.477613477s] [34.477613477s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.466636 24836 trace.go:205] Trace[1806970912]: "Reflector ListAndWatch" name:storage/cacher.go:/networkpolicies (26-Aug-2023 11:10:34.988) (total time: 34477ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1806970912]: ---"Objects listed" error: 34477ms (11:11:09.466) Aug 26 11:11:09 groomlake k3s[24836]: Trace[1806970912]: [34.477780181s] [34.477780181s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.569832 24836 trace.go:205] Trace[1649561741]: "List(recursive=true) etcd3" audit-id:11360ef8-81f7-4df5-a96c-7903a35db1e3,key:/mutatingwebhookconfigurations,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (26-Aug-2023 11:10:57.587) (total time: 11981ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1649561741]: [11.981882374s] [11.981882374s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.570321 24836 trace.go:205] Trace[484537968]: "List" url:/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:11360ef8-81f7-4df5-a96c-7903a35db1e3,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:57.587) (total time: 11982ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[484537968]: ---"Listing from storage done" 11982ms (11:11:09.570) Aug 26 11:11:09 groomlake k3s[24836]: Trace[484537968]: [11.982391849s] [11.982391849s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.570946 24836 trace.go:205] Trace[1131170007]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.25.3-k3s1/tools/cache/reflector.go:169 (26-Aug-2023 11:10:57.421) (total time: 12149ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1131170007]: ---"Objects listed" error: 12149ms (11:11:09.570) Aug 26 11:11:09 groomlake k3s[24836]: Trace[1131170007]: [12.149874109s] [12.149874109s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.978748 24836 trace.go:205] Trace[2075114751]: "List(recursive=true) etcd3" audit-id:,key:/csistoragecapacities,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.633) (total time: 34345ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[2075114751]: [34.345057369s] [34.345057369s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.988504 24836 trace.go:205] Trace[1463243699]: "Reflector ListAndWatch" name:storage/cacher.go:/csistoragecapacities (26-Aug-2023 11:10:35.633) (total time: 34354ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1463243699]: ---"Objects listed" error: 34354ms (11:11:09.988) Aug 26 11:11:09 groomlake k3s[24836]: Trace[1463243699]: [34.354831563s] [34.354831563s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.988806 24836 trace.go:205] Trace[668846238]: "List(recursive=true) etcd3" audit-id:,key:/cronjobs,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:34.898) (total time: 35090ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[668846238]: [35.090053846s] [35.090053846s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.988936 24836 trace.go:205] Trace[1546583415]: "Reflector ListAndWatch" name:storage/cacher.go:/cronjobs (26-Aug-2023 11:10:34.898) (total time: 35090ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1546583415]: ---"Objects listed" error: 35090ms (11:11:09.988) Aug 26 11:11:09 groomlake k3s[24836]: Trace[1546583415]: [35.090216107s] [35.090216107s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.989196 24836 trace.go:205] Trace[12711856]: "List(recursive=true) etcd3" audit-id:1432c87a-5391-4cb5-a3f4-14de310bab8c,key:/prioritylevelconfigurations,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (26-Aug-2023 11:10:57.588) (total time: 12401ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[12711856]: [12.401058043s] [12.401058043s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.989734 24836 trace.go:205] Trace[1728412783]: "List" url:/apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:1432c87a-5391-4cb5-a3f4-14de310bab8c,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:57.588) (total time: 12401ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[1728412783]: ---"Listing from storage done" 12401ms (11:11:09.989) Aug 26 11:11:09 groomlake k3s[24836]: Trace[1728412783]: [12.401602498s] [12.401602498s] END Aug 26 11:11:09 groomlake k3s[24836]: I0826 11:11:09.991611 24836 trace.go:205] Trace[348627790]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.25.3-k3s1/tools/cache/reflector.go:169 (26-Aug-2023 11:10:57.421) (total time: 12570ms): Aug 26 11:11:09 groomlake k3s[24836]: Trace[348627790]: ---"Objects listed" error: 12570ms (11:11:09.991) Aug 26 11:11:09 groomlake k3s[24836]: Trace[348627790]: [12.570445365s] [12.570445365s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.067947 24836 apf_controller.go:305] Running API Priority and Fairness config worker Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.288874 24836 trace.go:205] Trace[342751127]: "List(recursive=true) etcd3" audit-id:,key:/ingressclasses,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.039) (total time: 35249ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[342751127]: [35.249173017s] [35.249173017s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.288946 24836 trace.go:205] Trace[1383402253]: "Reflector ListAndWatch" name:storage/cacher.go:/ingressclasses (26-Aug-2023 11:10:35.039) (total time: 35249ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[1383402253]: ---"Objects listed" error: 35249ms (11:11:10.288) Aug 26 11:11:10 groomlake k3s[24836]: Trace[1383402253]: [35.249281787s] [35.249281787s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.308691 24836 trace.go:205] Trace[1683337264]: "List(recursive=true) etcd3" audit-id:f51e3aca-8b71-47ea-95dd-00315c0c7032,key:/clusterrolebindings,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (26-Aug-2023 11:10:57.522) (total time: 12786ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[1683337264]: [12.786342483s] [12.786342483s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.309052 24836 trace.go:205] Trace[1374391092]: "List" url:/apis/rbac.authorization.k8s.io/v1/clusterrolebindings,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:f51e3aca-8b71-47ea-95dd-00315c0c7032,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:57.522) (total time: 12786ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[1374391092]: ---"Listing from storage done" 12786ms (11:11:10.308) Aug 26 11:11:10 groomlake k3s[24836]: Trace[1374391092]: [12.786712836s] [12.786712836s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.309634 24836 trace.go:205] Trace[513826250]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.25.3-k3s1/tools/cache/reflector.go:169 (26-Aug-2023 11:10:57.373) (total time: 12936ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[513826250]: ---"Objects listed" error: 12936ms (11:11:10.309) Aug 26 11:11:10 groomlake k3s[24836]: Trace[513826250]: [12.936289979s] [12.936289979s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.310390 24836 trace.go:205] Trace[862779299]: "List(recursive=true) etcd3" audit-id:33e6fbb8-2694-4bfe-b861-38dbe6d3d21e,key:/apiregistration.k8s.io/apiservices,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (26-Aug-2023 11:10:57.520) (total time: 12789ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[862779299]: [12.789463945s] [12.789463945s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.310714 24836 trace.go:205] Trace[1438585530]: "List" url:/apis/apiregistration.k8s.io/v1/apiservices,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:33e6fbb8-2694-4bfe-b861-38dbe6d3d21e,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:57.520) (total time: 12789ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[1438585530]: ---"Listing from storage done" 12789ms (11:11:10.310) Aug 26 11:11:10 groomlake k3s[24836]: Trace[1438585530]: [12.789803886s] [12.789803886s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.319911 24836 trace.go:205] Trace[1413604084]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.25.3-k3s1/tools/cache/reflector.go:169 (26-Aug-2023 11:10:57.372) (total time: 12947ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[1413604084]: ---"Objects listed" error: 12947ms (11:11:10.319) Aug 26 11:11:10 groomlake k3s[24836]: Trace[1413604084]: [12.947115502s] [12.947115502s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.374108 24836 cache.go:39] Caches are synced for autoregister controller Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.380445 24836 cache.go:39] Caches are synced for APIServiceRegistrationController controller Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.384440 24836 cache.go:39] Caches are synced for AvailableConditionController controller Aug 26 11:11:10 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:11:10.637-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c936c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"} Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.686362 24836 trace.go:205] Trace[707815903]: "List(recursive=true) etcd3" audit-id:,key:/poddisruptionbudgets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (26-Aug-2023 11:10:35.115) (total time: 35571ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[707815903]: [35.571159028s] [35.571159028s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.686440 24836 trace.go:205] Trace[918676646]: "Reflector ListAndWatch" name:storage/cacher.go:/poddisruptionbudgets (26-Aug-2023 11:10:35.115) (total time: 35571ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[918676646]: ---"Objects listed" error: 35571ms (11:11:10.686) Aug 26 11:11:10 groomlake k3s[24836]: Trace[918676646]: [35.571319972s] [35.571319972s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.686608 24836 trace.go:205] Trace[1305384369]: "List(recursive=true) etcd3" audit-id:c1236816-f3a8-4a28-95e2-63d64559b1d1,key:/rolebindings,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (26-Aug-2023 11:10:57.773) (total time: 12912ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[1305384369]: [12.912902703s] [12.912902703s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.686973 24836 trace.go:205] Trace[667515770]: "List" url:/apis/rbac.authorization.k8s.io/v1/rolebindings,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:c1236816-f3a8-4a28-95e2-63d64559b1d1,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:57.773) (total time: 12913ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[667515770]: ---"Listing from storage done" 12912ms (11:11:10.686) Aug 26 11:11:10 groomlake k3s[24836]: Trace[667515770]: [12.91330918s] [12.91330918s] END Aug 26 11:11:10 groomlake k3s[24836]: I0826 11:11:10.687776 24836 trace.go:205] Trace[1798865651]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.25.3-k3s1/tools/cache/reflector.go:169 (26-Aug-2023 11:10:57.421) (total time: 13265ms): Aug 26 11:11:10 groomlake k3s[24836]: Trace[1798865651]: ---"Objects listed" error: 13265ms (11:11:10.687) Aug 26 11:11:10 groomlake k3s[24836]: Trace[1798865651]: [13.265927458s] [13.265927458s] END Aug 26 11:11:11 groomlake k3s[24836]: I0826 11:11:11.121487 24836 trace.go:205] Trace[986776781]: "Get" url:/apis/scheduling.k8s.io/v1/priorityclasses/system-node-critical,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:f83ec16c-2b05-4ce1-9ab7-73aea4e3d8e0,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:58.556) (total time: 12564ms): Aug 26 11:11:11 groomlake k3s[24836]: Trace[986776781]: ---"About to write a response" 12564ms (11:11:11.121) Aug 26 11:11:11 groomlake k3s[24836]: Trace[986776781]: [12.564444288s] [12.564444288s] END Aug 26 11:11:11 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:11:11.280-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0044c9180/kine.sock","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Aug 26 11:11:11 groomlake k3s[24836]: I0826 11:11:11.658309 24836 trace.go:205] Trace[1823867793]: "List(recursive=true) etcd3" audit-id:ce8b9095-8367-4598-9faa-c9f12102a354,key:/clusterroles,resourceVersion:,resourceVersionMatch:,limit:0,continue: (26-Aug-2023 11:10:58.557) (total time: 13100ms): Aug 26 11:11:11 groomlake k3s[24836]: Trace[1823867793]: [13.100476641s] [13.100476641s] END Aug 26 11:11:11 groomlake k3s[24836]: I0826 11:11:11.659155 24836 trace.go:205] Trace[1907796247]: "List" url:/apis/rbac.authorization.k8s.io/v1/clusterroles,user-agent:k3s/v1.25.3+k3s (linux/amd64) kubernetes/9afcd6b,audit-id:ce8b9095-8367-4598-9faa-c9f12102a354,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (26-Aug-2023 11:10:58.557) (total time: 13101ms): Aug 26 11:11:11 groomlake k3s[24836]: Trace[1907796247]: ---"Listing from storage done" 13100ms (11:11:11.658) Aug 26 11:11:11 groomlake k3s[24836]: Trace[1907796247]: [13.101336289s] [13.101336289s] END Aug 26 11:11:12 groomlake k3s[24836]: {"level":"warn","ts":"2023-08-26T11:11:12.360-0400","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0044c8fc0/kine.sock","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}