+--------------------------------------------------------------------------------+
+                      Kubernetes Configuration @1693898877                      +
+--------------------------------------------------------------------------------+
{
  "id": 1,
  "pool": "VLAN10FC8Storage",
  "cluster_cidr": "172.16.0.0/16",
  "service_cidr": "172.17.0.0/16",
  "cluster_dns_ip": "172.17.0.10",
  "route_v4_interface": null,
  "route_v4_gateway": null,
  "route_v6_interface": null,
  "route_v6_gateway": null,
  "node_ip": "0.0.0.0",
  "configure_gpus": true,
  "servicelb": true,
  "validate_host_path": true,
  "passthrough_mode": false,
  "dataset": "VLAN10FC8Storage/ix-applications"
}
debug finished in 1 seconds for Kubernetes Configuration


+--------------------------------------------------------------------------------+
+                     k3s kubectl describe nodes @1693898878                     +
+--------------------------------------------------------------------------------+
Name:               ix-truenas
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    egress.k3s.io/cluster=true
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ix-truenas
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
Annotations:        k3s.io/node-args:
                      ["server","--cluster-cidr","172.16.0.0/16","--cluster-dns","172.17.0.10","--data-dir","/mnt/VLAN10FC8Storage/ix-applications/k3s","--kube-...
                    k3s.io/node-config-hash: 5G7SHGFLPBJITP5ASTZEHSQRWZNMSTJL5WM2DHRTLIQ64DVURXUA====
                    k3s.io/node-env:
                      {"K3S_DATA_DIR":"/mnt/VLAN10FC8Storage/ix-applications/k3s/data/814fb91b7bfff921102404a4b77e94cfa15ada3c1c4735f6560f515889994d38"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 11 Aug 2023 09:16:44 +0800
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  ix-truenas
  AcquireTime:     <unset>
  RenewTime:       Tue, 05 Sep 2023 15:27:51 +0800
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 05 Sep 2023 15:23:50 +0800   Mon, 04 Sep 2023 18:21:39 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 05 Sep 2023 15:23:50 +0800   Mon, 04 Sep 2023 18:21:39 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Tue, 05 Sep 2023 15:23:50 +0800   Mon, 04 Sep 2023 18:21:39 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Tue, 05 Sep 2023 15:23:50 +0800   Mon, 04 Sep 2023 18:21:39 +0800   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  10.100.1.10
  Hostname:    ix-truenas
Capacity:
  cpu:                16
  ephemeral-storage:  6970353408Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             98509208Ki
  pods:               250
Allocatable:
  cpu:                16
  ephemeral-storage:  6780759789985
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             98509208Ki
  pods:               250
System Info:
  Machine ID:                 07aa430959da4b1d89d903408cedd316
  System UUID:                6a8eba76-050d-11e9-9842-0a94ef717d35
  Boot ID:                    98ac90cc-6b9c-4afa-809a-332c059cc4b4
  Kernel Version:             5.15.107+truenas
  OS Image:                   Debian GNU/Linux 11 (bullseye)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://Unknown
  Kubelet Version:            v1.25.3+k3s-9afcd6b9-dirty
  Kube-Proxy Version:         v1.25.3+k3s-9afcd6b9-dirty
PodCIDR:                      172.16.0.0/16
PodCIDRs:                     172.16.0.0/16
Non-terminated Pods:          (3 in total)
  Namespace                   Name                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                        ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-75fc8f8fff-n768f    100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     25d
  kube-system                 openebs-zfs-node-58727      0 (0%)        0 (0%)      0 (0%)           0 (0%)         25d
  kube-system                 openebs-zfs-controller-0    0 (0%)        0 (0%)      0 (0%)           0 (0%)         25d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (0%)  0 (0%)
  memory             70Mi (0%)  170Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:              <none>
debug finished in 1 seconds for k3s kubectl describe nodes


+--------------------------------------------------------------------------------+
+k3s kubectl get pods,svc,daemonsets,deployments,statefulset,sc,pvc,ns,job --all-namespaces -o wide @1693898879+
+--------------------------------------------------------------------------------+
NAMESPACE     NAME                           READY   STATUS             RESTARTS       AGE   IP            NODE         NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-75fc8f8fff-n768f   1/1     Running            3 (4d1h ago)   25d   172.16.0.9    ix-truenas   <none>           <none>
kube-system   pod/openebs-zfs-node-58727     1/2     ImagePullBackOff   3 (4d1h ago)   25d   10.100.1.10   ix-truenas   <none>           <none>
kube-system   pod/openebs-zfs-controller-0   1/5     ImagePullBackOff   3 (4d1h ago)   25d   172.16.0.8    ix-truenas   <none>           <none>

NAMESPACE     NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
default       service/kubernetes   ClusterIP   172.17.0.1    <none>        443/TCP                  25d   <none>
kube-system   service/kube-dns     ClusterIP   172.17.0.10   <none>        53/UDP,53/TCP,9153/TCP   25d   k8s-app=kube-dns

NAMESPACE     NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE   CONTAINERS                                     IMAGES                                                                             SELECTOR
kube-system   daemonset.apps/openebs-zfs-node   1         1         0       1            0           <none>          25d   csi-node-driver-registrar,openebs-zfs-plugin   k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,openebs/zfs-driver:2.0.0   app=openebs-zfs-node

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                   SELECTOR
kube-system   deployment.apps/coredns   1/1     1            1           25d   coredns      rancher/mirrored-coredns-coredns:1.9.1   k8s-app=kube-dns

NAMESPACE     NAME                                      READY   AGE   CONTAINERS                                                                           IMAGES
kube-system   statefulset.apps/openebs-zfs-controller   0/1     25d   csi-resizer,csi-snapshotter,snapshot-controller,csi-provisioner,openebs-zfs-plugin   k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0,k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0,k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,openebs/zfs-driver:2.0.0

NAMESPACE   NAME                                                          PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
            storageclass.storage.k8s.io/openebs-zfspv-default (default)   zfs.csi.openebs.io   Delete          Immediate           true                   4d

NAMESPACE   NAME                        STATUS   AGE
            namespace/default           Active   25d
            namespace/kube-system       Active   25d
            namespace/kube-public       Active   25d
            namespace/kube-node-lease   Active   25d
            namespace/openebs           Active   25d
debug finished in 0 seconds for k3s kubectl get pods,svc,daemonsets,deployments,statefulset,sc,pvc,ns,job --all-namespaces -o wide


+--------------------------------------------------------------------------------+
+         k3s kubectl describe deployments --all-namespaces @1693898879          +
+--------------------------------------------------------------------------------+
Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Fri, 11 Aug 2023 09:16:47 +0800
Labels:                 k8s-app=kube-dns
                        kubernetes.io/name=CoreDNS
                        objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:            deployment.kubernetes.io/revision: 1
                        objectset.rio.cattle.io/applied:
                          H4sIAAAAAAAA/6xVUW/bNhD+K8M9S7GVtI0rYA+dna1FV8+ok74URkFTJ4szxePIkxMj0H8fTrIdu02TdtiTZfK743cfvzveg/LmE4ZoyEEOyvs42GSQwNq4AnKYoLe0rdExJFAjq0...
                        objectset.rio.cattle.io/id: 
                        objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                        objectset.rio.cattle.io/owner-name: coredns
                        objectset.rio.cattle.io/owner-namespace: kube-system
Selector:               k8s-app=kube-dns
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       rancher/mirrored-coredns-coredns:1.9.1
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /etc/coredns/custom from custom-config-volume (ro)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
   custom-config-volume:
    Type:                       ConfigMap (a volume populated by a ConfigMap)
    Name:                       coredns-custom
    Optional:                   true
  Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector k8s-app=kube-dns
  Priority Class Name:          system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   coredns-75fc8f8fff (1/1 replicas created)
Events:          <none>
debug finished in 0 seconds for k3s kubectl describe deployments --all-namespaces


+--------------------------------------------------------------------------------+
+             k3s kubectl describe pods --all-namespaces @1693898879             +
+--------------------------------------------------------------------------------+
Name:                 coredns-75fc8f8fff-n768f
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      coredns
Node:                 ix-truenas/10.100.1.10
Start Time:           Fri, 11 Aug 2023 09:16:59 +0800
Labels:               k8s-app=kube-dns
                      pod-template-hash=75fc8f8fff
Annotations:          k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.9"
                            ],
                            "mac": "ea:26:50:9c:40:f1",
                            "default": true,
                            "dns": {}
                        }]
Status:               Running
IP:                   172.16.0.9
IPs:
  IP:           172.16.0.9
Controlled By:  ReplicaSet/coredns-75fc8f8fff
Containers:
  coredns:
    Container ID:  docker://6d73e9c7b4bf2da912955b58925c0bc90bff8ba4a538a9502c9191167d9d4ab2
    Image:         rancher/mirrored-coredns-coredns:1.9.1
    Image ID:      docker://sha256:99376d8f35e0abb6ff9d66b50a7c81df6e6dfdb649becc5df84a691a7b4beca4
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Fri, 01 Sep 2023 14:36:25 +0800
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 29 Aug 2023 16:09:14 +0800
      Finished:     Fri, 01 Sep 2023 14:21:03 +0800
    Ready:          True
    Restart Count:  3
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /etc/coredns/custom from custom-config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9fd7g (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  custom-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns-custom
    Optional:  true
  kube-api-access-9fd7g:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    Burstable
Node-Selectors:               beta.kubernetes.io/os=linux
Tolerations:                  CriticalAddonsOnly op=Exists
                              node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                              node-role.kubernetes.io/master:NoSchedule op=Exists
                              node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector k8s-app=kube-dns
Events:                       <none>


Name:                 openebs-zfs-node-58727
Namespace:            kube-system
Priority:             900001000
Priority Class Name:  openebs-zfs-csi-node-critical
Service Account:      openebs-zfs-node-sa
Node:                 ix-truenas/10.100.1.10
Start Time:           Fri, 11 Aug 2023 09:16:59 +0800
Labels:               app=openebs-zfs-node
                      controller-revision-hash=66bcbb6975
                      openebs.io/component-name=openebs-zfs-node
                      openebs.io/version=ci
                      pod-template-generation=1
                      role=openebs-zfs
Annotations:          <none>
Status:               Pending
IP:                   10.100.1.10
IPs:
  IP:           10.100.1.10
Controlled By:  DaemonSet/openebs-zfs-node
Containers:
  csi-node-driver-registrar:
    Container ID:  
    Image:         k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      ADDRESS:               /plugin/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/zfs-localpv/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
      NODE_DRIVER:           openebs-zfs
    Mounts:
      /plugin from plugin-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m69jt (ro)
  openebs-zfs-plugin:
    Container ID:  docker://1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62
    Image:         openebs/zfs-driver:2.0.0
    Image ID:      docker-pullable://openebs/zfs-driver@sha256:472dc7ca43f9780246cc6d2d478ff4b08ead904d757dbb263f796fb613dbe04d
    Port:          <none>
    Host Port:     <none>
    Args:
      --nodename=$(OPENEBS_NODE_NAME)
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_NODE_DRIVER)
    State:          Running
      Started:      Fri, 01 Sep 2023 14:36:26 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 29 Aug 2023 16:09:15 +0800
      Finished:     Fri, 01 Sep 2023 14:21:13 +0800
    Ready:          True
    Restart Count:  3
    Environment:
      OPENEBS_NODE_NAME:      (v1:spec.nodeName)
      OPENEBS_CSI_ENDPOINT:  unix:///plugin/csi.sock
      OPENEBS_NODE_DRIVER:   agent
      OPENEBS_NAMESPACE:     openebs
      ALLOWED_TOPOLOGIES:    All
    Mounts:
      /dev from device-dir (rw)
      /home/keys from encr-keys (rw)
      /host from host-root (ro)
      /plugin from plugin-dir (rw)
      /sbin/zfs from chroot-zfs (rw,path="zfs")
      /var/lib/kubelet/ from pods-mount-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m69jt (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  device-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
  encr-keys:
    Type:          HostPath (bare host directory volume)
    Path:          /home/keys
    HostPathType:  DirectoryOrCreate
  chroot-zfs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openebs-zfspv-bin
    Optional:  false
  host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
  plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/zfs-localpv/
    HostPathType:  DirectoryOrCreate
  pods-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/
    HostPathType:  Directory
  kube-api-access-m69jt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type    Reason   Age                    From     Message
  ----    ------   ----                   ----     -------
  Normal  Pulling  21m (x1082 over 4d)    kubelet  Pulling image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
  Normal  BackOff  101s (x24444 over 4d)  kubelet  Back-off pulling image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"


Name:                 openebs-zfs-controller-0
Namespace:            kube-system
Priority:             900000000
Priority Class Name:  openebs-zfs-csi-controller-critical
Service Account:      openebs-zfs-controller-sa
Node:                 ix-truenas/10.100.1.10
Start Time:           Fri, 11 Aug 2023 09:16:59 +0800
Labels:               app=openebs-zfs-controller
                      controller-revision-hash=openebs-zfs-controller-698698d48b
                      openebs.io/component-name=openebs-zfs-controller
                      openebs.io/version=ci
                      role=openebs-zfs
                      statefulset.kubernetes.io/pod-name=openebs-zfs-controller-0
Annotations:          k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ix-net",
                            "interface": "eth0",
                            "ips": [
                                "172.16.0.8"
                            ],
                            "mac": "22:7e:bd:17:07:b6",
                            "default": true,
                            "dns": {}
                        }]
Status:               Pending
IP:                   172.16.0.8
IPs:
  IP:           172.16.0.8
Controlled By:  StatefulSet/openebs-zfs-controller
Containers:
  csi-resizer:
    Container ID:  
    Image:         k8s.gcr.io/sig-storage/csi-resizer:v1.2.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --leader-election
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7js7 (ro)
  csi-snapshotter:
    Container ID:  
    Image:         k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --leader-election
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7js7 (ro)
  snapshot-controller:
    Container ID:  
    Image:         k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --leader-election=true
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7js7 (ro)
  csi-provisioner:
    Container ID:  
    Image:         k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
      --feature-gates=Topology=true
      --strict-topology
      --leader-election
      --extra-create-metadata=true
      --enable-capacity=true
      --default-fstype=ext4
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      ADDRESS:    /var/lib/csi/sockets/pluginproxy/csi.sock
      NAMESPACE:  kube-system (v1:metadata.namespace)
      POD_NAME:   openebs-zfs-controller-0 (v1:metadata.name)
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7js7 (ro)
  openebs-zfs-plugin:
    Container ID:  docker://703931cd2f3cb0ae3c95a62df02cc20fea965041d6806608cf9de81772348bfa
    Image:         openebs/zfs-driver:2.0.0
    Image ID:      docker-pullable://openebs/zfs-driver@sha256:472dc7ca43f9780246cc6d2d478ff4b08ead904d757dbb263f796fb613dbe04d
    Port:          <none>
    Host Port:     <none>
    Args:
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_CONTROLLER_DRIVER)
    State:          Running
      Started:      Fri, 01 Sep 2023 14:37:20 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 29 Aug 2023 16:10:08 +0800
      Finished:     Fri, 01 Sep 2023 14:21:13 +0800
    Ready:          True
    Restart Count:  3
    Environment:
      OPENEBS_CONTROLLER_DRIVER:    controller
      OPENEBS_CSI_ENDPOINT:         unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      OPENEBS_NAMESPACE:            openebs
      OPENEBS_IO_INSTALLER_TYPE:    zfs-operator
      OPENEBS_IO_ENABLE_ANALYTICS:  true
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7js7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  socket-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-s7js7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Warning  Failed   56m (x439 over 4d)     kubelet  Failed to pull image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   BackOff  112s (x20617 over 4d)  kubelet  Back-off pulling image "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
debug finished in 1 seconds for k3s kubectl describe pods --all-namespaces


+--------------------------------------------------------------------------------+
+           k3s kubectl describe services --all-namespaces @1693898880           +
+--------------------------------------------------------------------------------+
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.17.0.1
IPs:               172.17.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.100.1.10:6443
Session Affinity:  None
Events:            <none>


Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=CoreDNS
                   objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:       objectset.rio.cattle.io/applied:
                     H4sIAAAAAAAA/4yRwW7UMBCGXwXN2Qlxstukljig9oKQUKUCF8TBcWZZk8S2PJMgtMq7I++mYqEK9JZoPn/6558T6GA/YyTrHSiYJQjoretAwSPG2RoEASOy7jRrUCfQznnWbL2j9O...
                   objectset.rio.cattle.io/id: 
                   objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                   objectset.rio.cattle.io/owner-name: coredns
                   objectset.rio.cattle.io/owner-namespace: kube-system
                   prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.17.0.10
IPs:               172.17.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         172.16.0.9:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         172.16.0.9:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         172.16.0.9:9153
Session Affinity:  None
Events:            <none>
debug finished in 0 seconds for k3s kubectl describe services --all-namespaces


+--------------------------------------------------------------------------------+
+         k3s kubectl describe statefulset --all-namespaces @1693898880          +
+--------------------------------------------------------------------------------+
Name:               openebs-zfs-controller
Namespace:          kube-system
CreationTimestamp:  Fri, 11 Aug 2023 09:16:48 +0800
Selector:           app=openebs-zfs-controller,role=openebs-zfs
Labels:             objectset.rio.cattle.io/hash=4b1efd47d7f37713706abe9f8b0de3f0adf02e2c
                    openebs.io/component-name=openebs-zfs-controller
                    openebs.io/version=ci
Annotations:        objectset.rio.cattle.io/applied:
                      H4sIAAAAAAAA/8xWXY/aOBT9KyurD60U8zEzu9ONxAMdshLqFBCgSlWFkHFughfH9to3Wego/33lBJh0Zpgvdau+ge17fM7xuY5vCDPiM1gntCIhYca4dtElAdkIFZOQzJAhJLmcAZ...
                    objectset.rio.cattle.io/id: 
                    objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                    objectset.rio.cattle.io/owner-name: zfs-operator
                    objectset.rio.cattle.io/owner-namespace: kube-system
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=openebs-zfs-controller
                    openebs.io/component-name=openebs-zfs-controller
                    openebs.io/version=ci
                    role=openebs-zfs
  Service Account:  openebs-zfs-controller-sa
  Containers:
   csi-resizer:
    Image:      k8s.gcr.io/sig-storage/csi-resizer:v1.2.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --leader-election
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   csi-snapshotter:
    Image:      k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=$(ADDRESS)
      --leader-election
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   snapshot-controller:
    Image:      k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --leader-election=true
    Environment:  <none>
    Mounts:       <none>
   csi-provisioner:
    Image:      k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
      --feature-gates=Topology=true
      --strict-topology
      --leader-election
      --extra-create-metadata=true
      --enable-capacity=true
      --default-fstype=ext4
    Environment:
      ADDRESS:    /var/lib/csi/sockets/pluginproxy/csi.sock
      NAMESPACE:   (v1:metadata.namespace)
      POD_NAME:    (v1:metadata.name)
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
   openebs-zfs-plugin:
    Image:      openebs/zfs-driver:2.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_CONTROLLER_DRIVER)
    Environment:
      OPENEBS_CONTROLLER_DRIVER:    controller
      OPENEBS_CSI_ENDPOINT:         unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      OPENEBS_NAMESPACE:            openebs
      OPENEBS_IO_INSTALLER_TYPE:    zfs-operator
      OPENEBS_IO_ENABLE_ANALYTICS:  true
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
  Volumes:
   socket-dir:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:             
    SizeLimit:          <unset>
  Priority Class Name:  openebs-zfs-csi-controller-critical
Volume Claims:          <none>
Events:                 <none>
debug finished in 0 seconds for k3s kubectl describe statefulset --all-namespaces


+--------------------------------------------------------------------------------+
+             k3s kubectl describe job --all-namespaces @1693898880              +
+--------------------------------------------------------------------------------+
No resources found
debug finished in 0 seconds for k3s kubectl describe job --all-namespaces


+--------------------------------------------------------------------------------+
+           k3s kubectl describe cronjob --all-namespaces @1693898880            +
+--------------------------------------------------------------------------------+
No resources found
debug finished in 1 seconds for k3s kubectl describe cronjob --all-namespaces


+--------------------------------------------------------------------------------+
+          k3s kubectl describe daemonsets --all-namespaces @1693898881          +
+--------------------------------------------------------------------------------+
Name:           openebs-zfs-node
Selector:       app=openebs-zfs-node
Node-Selector:  <none>
Labels:         objectset.rio.cattle.io/hash=4b1efd47d7f37713706abe9f8b0de3f0adf02e2c
                openebs.io/component-name=openebs-zfs-node
                openebs.io/version=ci
Annotations:    deprecated.daemonset.template.generation: 1
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/6xWUY/aOBD+K5XVk1opIezSdnuR9oFCrl11FxBsew/VCjn2JPhw7Mh20uVW+e8nOwEChO2e1Lcknvlm5ptvJn5COGffQWkmBQoRznMdlBfIQ2smKArRGEMmxQIM8l...
                objectset.rio.cattle.io/id: 
                objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
                objectset.rio.cattle.io/owner-name: zfs-operator
                objectset.rio.cattle.io/owner-namespace: kube-system
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=openebs-zfs-node
                    openebs.io/component-name=openebs-zfs-node
                    openebs.io/version=ci
                    role=openebs-zfs
  Service Account:  openebs-zfs-node-sa
  Containers:
   csi-node-driver-registrar:
    Image:      k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    Environment:
      ADDRESS:               /plugin/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/zfs-localpv/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
      NODE_DRIVER:           openebs-zfs
    Mounts:
      /plugin from plugin-dir (rw)
      /registration from registration-dir (rw)
   openebs-zfs-plugin:
    Image:      openebs/zfs-driver:2.0.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --nodename=$(OPENEBS_NODE_NAME)
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_NODE_DRIVER)
    Environment:
      OPENEBS_NODE_NAME:      (v1:spec.nodeName)
      OPENEBS_CSI_ENDPOINT:  unix:///plugin/csi.sock
      OPENEBS_NODE_DRIVER:   agent
      OPENEBS_NAMESPACE:     openebs
      ALLOWED_TOPOLOGIES:    All
    Mounts:
      /dev from device-dir (rw)
      /home/keys from encr-keys (rw)
      /host from host-root (ro)
      /plugin from plugin-dir (rw)
      /sbin/zfs from chroot-zfs (rw,path="zfs")
      /var/lib/kubelet/ from pods-mount-dir (rw)
  Volumes:
   device-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
   encr-keys:
    Type:          HostPath (bare host directory volume)
    Path:          /home/keys
    HostPathType:  DirectoryOrCreate
   chroot-zfs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openebs-zfspv-bin
    Optional:  false
   host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
   registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
   plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/zfs-localpv/
    HostPathType:  DirectoryOrCreate
   pods-mount-dir:
    Type:               HostPath (bare host directory volume)
    Path:               /var/lib/kubelet/
    HostPathType:       Directory
  Priority Class Name:  openebs-zfs-csi-node-critical
Events:                 <none>
debug finished in 0 seconds for k3s kubectl describe daemonsets --all-namespaces


+--------------------------------------------------------------------------------+
+                            docker ps -a @1693898881                            +
+--------------------------------------------------------------------------------+
CONTAINER ID   IMAGE                        COMMAND                  CREATED      STATUS                    PORTS     NAMES
703931cd2f3c   2e17748d7c44                 "/usr/local/bin/zfs-…"   4 days ago   Up 4 days                           k8s_openebs-zfs-plugin_openebs-zfs-controller-0_kube-system_2728fa87-0c11-42bb-b5a8-9950ab76f184_3
6d73e9c7b4bf   99376d8f35e0                 "/coredns -conf /etc…"   4 days ago   Up 4 days                           k8s_coredns_coredns-75fc8f8fff-n768f_kube-system_75d37ab7-4508-4cfb-8811-e666f2c7796a_3
1db75a7a7b3b   2e17748d7c44                 "/usr/local/bin/zfs-…"   4 days ago   Up 4 days                           k8s_openebs-zfs-plugin_openebs-zfs-node-58727_kube-system_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3_3
675bc1020f93   rancher/mirrored-pause:3.6   "/pause"                 4 days ago   Up 4 days                           k8s_POD_coredns-75fc8f8fff-n768f_kube-system_75d37ab7-4508-4cfb-8811-e666f2c7796a_3
34d3a734efa8   rancher/mirrored-pause:3.6   "/pause"                 4 days ago   Up 4 days                           k8s_POD_openebs-zfs-controller-0_kube-system_2728fa87-0c11-42bb-b5a8-9950ab76f184_3
b1ac60da927a   rancher/mirrored-pause:3.6   "/pause"                 4 days ago   Up 4 days                           k8s_POD_openebs-zfs-node-58727_kube-system_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3_3
20c51d8148b9   2e17748d7c44                 "/usr/local/bin/zfs-…"   6 days ago   Exited (137) 4 days ago             k8s_openebs-zfs-plugin_openebs-zfs-controller-0_kube-system_2728fa87-0c11-42bb-b5a8-9950ab76f184_2
751f943a7191   99376d8f35e0                 "/coredns -conf /etc…"   6 days ago   Exited (0) 4 days ago               k8s_coredns_coredns-75fc8f8fff-n768f_kube-system_75d37ab7-4508-4cfb-8811-e666f2c7796a_2
6cce9cf7b0fa   2e17748d7c44                 "/usr/local/bin/zfs-…"   6 days ago   Exited (137) 4 days ago             k8s_openebs-zfs-plugin_openebs-zfs-node-58727_kube-system_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3_2
f699c67911aa   rancher/mirrored-pause:3.6   "/pause"                 6 days ago   Exited (0) 4 days ago               k8s_POD_coredns-75fc8f8fff-n768f_kube-system_75d37ab7-4508-4cfb-8811-e666f2c7796a_2
468a798a4414   rancher/mirrored-pause:3.6   "/pause"                 6 days ago   Exited (0) 4 days ago               k8s_POD_openebs-zfs-controller-0_kube-system_2728fa87-0c11-42bb-b5a8-9950ab76f184_2
5d25e0524217   rancher/mirrored-pause:3.6   "/pause"                 6 days ago   Exited (0) 4 days ago               k8s_POD_openebs-zfs-node-58727_kube-system_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3_2
debug finished in 0 seconds for docker ps -a


+--------------------------------------------------------------------------------+
+                          docker images -a @1693898881                          +
+--------------------------------------------------------------------------------+
REPOSITORY                         TAG       IMAGE ID       CREATED         SIZE
rancher/mirrored-library-busybox   1.34.1    827365c7baf1   8 months ago    4.86MB
rancher/mirrored-coredns-coredns   1.9.1     99376d8f35e0   18 months ago   49.5MB
rancher/klipper-lb                 v0.3.5    dbd43b6716a0   19 months ago   8.09MB
openebs/zfs-driver                 2.0.0     2e17748d7c44   20 months ago   176MB
rancher/mirrored-pause             3.6       6270bb605e12   2 years ago     683kB
debug finished in 0 seconds for docker images -a


+--------------------------------------------------------------------------------+
+                     K3s Logs (last 1000 lines) @1693898881                     +
+--------------------------------------------------------------------------------+
Sep 05 15:00:40 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:00:40+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:00:44 vlan10fc8truenas k3s[7084]: E0905 15:00:44.135627    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:00:44 vlan10fc8truenas k3s[7084]: E0905 15:00:44.136005    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:00:45 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:00:45+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:00:46 vlan10fc8truenas k3s[7084]: E0905 15:00:46.707665    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:00:50 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:00:50+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:00:54 vlan10fc8truenas k3s[7084]: E0905 15:00:54.153561    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:00:54 vlan10fc8truenas k3s[7084]: E0905 15:00:54.153840    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:00:55 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:00:55+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:00 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:00+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:01 vlan10fc8truenas k3s[7084]: E0905 15:01:01.707840    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:01:02 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:01:02.014+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:01:04 vlan10fc8truenas k3s[7084]: E0905 15:01:04.171840    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:04 vlan10fc8truenas k3s[7084]: E0905 15:01:04.172231    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:05 vlan10fc8truenas k3s[7084]: E0905 15:01:05.769378    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:01:05 vlan10fc8truenas k3s[7084]: E0905 15:01:05.769632    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:01:05 vlan10fc8truenas k3s[7084]: E0905 15:01:05.769886    7084 kuberuntime_manager.go:862] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-m69jt,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-58727_kube-system(f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:01:05 vlan10fc8truenas k3s[7084]: E0905 15:01:05.769987    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:01:05 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:05+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:10 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:10+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:14 vlan10fc8truenas k3s[7084]: E0905 15:01:14.189623    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:14 vlan10fc8truenas k3s[7084]: E0905 15:01:14.189988    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:15 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:15+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:16 vlan10fc8truenas k3s[7084]: E0905 15:01:16.703111    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:01:16 vlan10fc8truenas k3s[7084]: E0905 15:01:16.708208    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:01:20 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:20+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:24 vlan10fc8truenas k3s[7084]: E0905 15:01:24.206684    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:24 vlan10fc8truenas k3s[7084]: E0905 15:01:24.207107    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:25 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:25+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:28 vlan10fc8truenas k3s[7084]: E0905 15:01:28.702860    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:01:29 vlan10fc8truenas k3s[7084]: E0905 15:01:29.707329    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:01:30 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:30+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:34 vlan10fc8truenas k3s[7084]: E0905 15:01:34.224999    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:34 vlan10fc8truenas k3s[7084]: E0905 15:01:34.225397    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:35 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:35+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:40 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:40+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:41 vlan10fc8truenas k3s[7084]: E0905 15:01:41.707117    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:01:42 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:01:42.890+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:01:43 vlan10fc8truenas k3s[7084]: E0905 15:01:43.702367    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:01:44 vlan10fc8truenas k3s[7084]: E0905 15:01:44.242829    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:44 vlan10fc8truenas k3s[7084]: E0905 15:01:44.243114    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:46 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:46+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:51 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:51+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:54 vlan10fc8truenas k3s[7084]: E0905 15:01:54.260832    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:54 vlan10fc8truenas k3s[7084]: E0905 15:01:54.261215    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:01:54 vlan10fc8truenas k3s[7084]: E0905 15:01:54.707860    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:01:56 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:01:56+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:01:56 vlan10fc8truenas k3s[7084]: E0905 15:01:56.702474    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:02:01 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:01+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:04 vlan10fc8truenas k3s[7084]: E0905 15:02:04.278595    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:04 vlan10fc8truenas k3s[7084]: E0905 15:02:04.278971    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:06 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:06+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:09 vlan10fc8truenas k3s[7084]: E0905 15:02:09.702127    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:02:09 vlan10fc8truenas k3s[7084]: E0905 15:02:09.707392    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:02:11 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:11+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:14 vlan10fc8truenas k3s[7084]: E0905 15:02:14.297477    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:14 vlan10fc8truenas k3s[7084]: E0905 15:02:14.297788    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:16 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:16+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:20 vlan10fc8truenas k3s[7084]: E0905 15:02:20.707736    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:02:20 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:02:20.981+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:02:21 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:21+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:21 vlan10fc8truenas k3s[7084]: E0905 15:02:21.701751    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:02:24 vlan10fc8truenas k3s[7084]: E0905 15:02:24.316149    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:24 vlan10fc8truenas k3s[7084]: E0905 15:02:24.316430    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:26 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:26+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:31 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:31+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:32 vlan10fc8truenas k3s[7084]: E0905 15:02:32.702874    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:02:34 vlan10fc8truenas k3s[7084]: E0905 15:02:34.333556    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:34 vlan10fc8truenas k3s[7084]: E0905 15:02:34.333854    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:35 vlan10fc8truenas k3s[7084]: E0905 15:02:35.707949    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:02:36 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:36+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:41 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:41+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:44 vlan10fc8truenas k3s[7084]: E0905 15:02:44.351601    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:44 vlan10fc8truenas k3s[7084]: E0905 15:02:44.351983    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:44 vlan10fc8truenas k3s[7084]: E0905 15:02:44.702541    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:02:46 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:46+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:49 vlan10fc8truenas k3s[7084]: E0905 15:02:49.707331    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:02:51 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:02:51.039+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:02:51 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:51+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:54 vlan10fc8truenas k3s[7084]: E0905 15:02:54.368864    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:54 vlan10fc8truenas k3s[7084]: E0905 15:02:54.369225    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:02:56 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:02:56+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:02:59 vlan10fc8truenas k3s[7084]: E0905 15:02:59.702454    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:03:00 vlan10fc8truenas k3s[7084]: E0905 15:03:00.707751    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:03:01 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:01+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:04 vlan10fc8truenas k3s[7084]: E0905 15:03:04.387973    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:04 vlan10fc8truenas k3s[7084]: E0905 15:03:04.388268    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:06 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:06+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:11 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:11+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:11 vlan10fc8truenas k3s[7084]: E0905 15:03:11.702199    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:03:14 vlan10fc8truenas k3s[7084]: E0905 15:03:14.406512    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:14 vlan10fc8truenas k3s[7084]: E0905 15:03:14.406911    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:16 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:16+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:21 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:21+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:23 vlan10fc8truenas k3s[7084]: E0905 15:03:23.702299    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:03:24 vlan10fc8truenas k3s[7084]: E0905 15:03:24.425054    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:24 vlan10fc8truenas k3s[7084]: E0905 15:03:24.425423    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:26 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:26+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:26 vlan10fc8truenas k3s[7084]: E0905 15:03:26.825395    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:03:26 vlan10fc8truenas k3s[7084]: E0905 15:03:26.825467    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:03:26 vlan10fc8truenas k3s[7084]: E0905 15:03:26.825682    7084 kuberuntime_manager.go:862] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:03:27 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:03:27.844+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:03:31 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:31+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:34 vlan10fc8truenas k3s[7084]: E0905 15:03:34.442616    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:34 vlan10fc8truenas k3s[7084]: E0905 15:03:34.442897    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:36 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:36+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:38 vlan10fc8truenas k3s[7084]: E0905 15:03:38.702674    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:03:41 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:41+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:41 vlan10fc8truenas k3s[7084]: E0905 15:03:41.872653    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:03:41 vlan10fc8truenas k3s[7084]: E0905 15:03:41.872725    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:03:41 vlan10fc8truenas k3s[7084]: E0905 15:03:41.872903    7084 kuberuntime_manager.go:862] container &Container{Name:csi-snapshotter,Image:k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:03:44 vlan10fc8truenas k3s[7084]: E0905 15:03:44.460352    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:44 vlan10fc8truenas k3s[7084]: E0905 15:03:44.460724    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:46 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:46+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:50 vlan10fc8truenas k3s[7084]: E0905 15:03:50.702766    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:03:51 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:51+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:54 vlan10fc8truenas k3s[7084]: E0905 15:03:54.477501    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:54 vlan10fc8truenas k3s[7084]: E0905 15:03:54.477856    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:03:56 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:03:56+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:03:56 vlan10fc8truenas k3s[7084]: E0905 15:03:56.970804    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:03:56 vlan10fc8truenas k3s[7084]: E0905 15:03:56.970880    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:03:56 vlan10fc8truenas k3s[7084]: E0905 15:03:56.971051    7084 kuberuntime_manager.go:862] container &Container{Name:snapshot-controller,Image:k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0,Command:[],Args:[--v=5 --leader-election=true],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:03:56 vlan10fc8truenas k3s[7084]: E0905 15:03:56.973320    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-snapshotter\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"snapshot-controller\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:04:01 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:01+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:03 vlan10fc8truenas k3s[7084]: E0905 15:04:03.706866    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:04:04 vlan10fc8truenas k3s[7084]: E0905 15:04:04.495536    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:04 vlan10fc8truenas k3s[7084]: E0905 15:04:04.495818    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:06 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:06+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:11 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:11+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:12 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:04:12.209+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:04:12 vlan10fc8truenas k3s[7084]: E0905 15:04:12.707337    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:04:14 vlan10fc8truenas k3s[7084]: E0905 15:04:14.513959    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:14 vlan10fc8truenas k3s[7084]: E0905 15:04:14.514322    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:14 vlan10fc8truenas k3s[7084]: E0905 15:04:14.701933    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:04:16 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:16+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:21 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:21+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:23 vlan10fc8truenas k3s[7084]: E0905 15:04:23.707129    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:04:24 vlan10fc8truenas k3s[7084]: E0905 15:04:24.532921    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:24 vlan10fc8truenas k3s[7084]: E0905 15:04:24.533295    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:26 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:26+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:27 vlan10fc8truenas k3s[7084]: E0905 15:04:27.702777    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:04:31 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:31+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:34 vlan10fc8truenas k3s[7084]: E0905 15:04:34.551514    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:34 vlan10fc8truenas k3s[7084]: E0905 15:04:34.551800    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:36 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:36+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:38 vlan10fc8truenas k3s[7084]: E0905 15:04:38.702475    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:04:41 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:41+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:44 vlan10fc8truenas k3s[7084]: E0905 15:04:44.570182    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:44 vlan10fc8truenas k3s[7084]: E0905 15:04:44.570541    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:46 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:46+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:50 vlan10fc8truenas k3s[7084]: E0905 15:04:50.747653    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:04:50 vlan10fc8truenas k3s[7084]: E0905 15:04:50.747724    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:04:50 vlan10fc8truenas k3s[7084]: E0905 15:04:50.747938    7084 kuberuntime_manager.go:862] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:04:50 vlan10fc8truenas k3s[7084]: E0905 15:04:50.748089    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:04:51 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:51+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:04:52 vlan10fc8truenas k3s[7084]: E0905 15:04:52.702547    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:04:54 vlan10fc8truenas k3s[7084]: E0905 15:04:54.588198    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:54 vlan10fc8truenas k3s[7084]: E0905 15:04:54.588557    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:04:56 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:04:56.595+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:04:56 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:04:56+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:01 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:01+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:02 vlan10fc8truenas k3s[7084]: E0905 15:05:02.707294    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:05:04 vlan10fc8truenas k3s[7084]: E0905 15:05:04.606855    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:04 vlan10fc8truenas k3s[7084]: E0905 15:05:04.607232    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:06 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:06+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:07 vlan10fc8truenas k3s[7084]: E0905 15:05:07.702414    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:05:11 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:11+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:14 vlan10fc8truenas k3s[7084]: E0905 15:05:14.625392    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:14 vlan10fc8truenas k3s[7084]: E0905 15:05:14.625788    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:16 vlan10fc8truenas k3s[7084]: E0905 15:05:16.707903    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:05:16 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:16+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:18 vlan10fc8truenas k3s[7084]: E0905 15:05:18.702678    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:05:21 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:21+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:24 vlan10fc8truenas k3s[7084]: E0905 15:05:24.643073    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:24 vlan10fc8truenas k3s[7084]: E0905 15:05:24.643418    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:27 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:27+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:30 vlan10fc8truenas k3s[7084]: E0905 15:05:30.707598    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:05:32 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:32+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:32 vlan10fc8truenas k3s[7084]: E0905 15:05:32.702604    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:05:34 vlan10fc8truenas k3s[7084]: E0905 15:05:34.660715    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:34 vlan10fc8truenas k3s[7084]: E0905 15:05:34.661008    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:37 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:37+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:39 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:05:39.633+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:05:42 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:42+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:43 vlan10fc8truenas k3s[7084]: E0905 15:05:43.706944    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:05:44 vlan10fc8truenas k3s[7084]: E0905 15:05:44.679029    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:44 vlan10fc8truenas k3s[7084]: E0905 15:05:44.679409    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:44 vlan10fc8truenas k3s[7084]: E0905 15:05:44.702340    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:05:47 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:47+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:52 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:52+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:54 vlan10fc8truenas k3s[7084]: E0905 15:05:54.696428    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:54 vlan10fc8truenas k3s[7084]: E0905 15:05:54.696783    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:05:56 vlan10fc8truenas k3s[7084]: E0905 15:05:56.707190    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:05:57 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:05:57+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:05:58 vlan10fc8truenas k3s[7084]: E0905 15:05:58.701563    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:06:02 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:02+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:04 vlan10fc8truenas k3s[7084]: E0905 15:06:04.714550    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:04 vlan10fc8truenas k3s[7084]: E0905 15:06:04.714915    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:07 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:07+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:09 vlan10fc8truenas k3s[7084]: E0905 15:06:09.708025    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:06:12 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:12+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:14 vlan10fc8truenas k3s[7084]: E0905 15:06:14.731737    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:14 vlan10fc8truenas k3s[7084]: E0905 15:06:14.732109    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:17 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:17+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:22 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:22+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:23 vlan10fc8truenas k3s[7084]: E0905 15:06:23.708512    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:06:23 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:06:23.946+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:06:24 vlan10fc8truenas k3s[7084]: E0905 15:06:24.750472    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:24 vlan10fc8truenas k3s[7084]: E0905 15:06:24.750843    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:25 vlan10fc8truenas k3s[7084]: E0905 15:06:25.855846    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:06:25 vlan10fc8truenas k3s[7084]: E0905 15:06:25.855918    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:06:25 vlan10fc8truenas k3s[7084]: E0905 15:06:25.856134    7084 kuberuntime_manager.go:862] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-m69jt,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-58727_kube-system(f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Sep 05 15:06:25 vlan10fc8truenas k3s[7084]: E0905 15:06:25.856224    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:06:27 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:27+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:32 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:32+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:34 vlan10fc8truenas k3s[7084]: E0905 15:06:34.768688    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:34 vlan10fc8truenas k3s[7084]: E0905 15:06:34.769044    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:36 vlan10fc8truenas k3s[7084]: E0905 15:06:36.707749    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:06:37 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:37+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:40 vlan10fc8truenas k3s[7084]: E0905 15:06:40.702766    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:06:42 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:42+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:44 vlan10fc8truenas k3s[7084]: E0905 15:06:44.786882    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:44 vlan10fc8truenas k3s[7084]: E0905 15:06:44.787259    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:47 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:47+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:50 vlan10fc8truenas k3s[7084]: E0905 15:06:50.708246    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:06:52 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:52+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:06:54 vlan10fc8truenas k3s[7084]: E0905 15:06:54.702735    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:06:54 vlan10fc8truenas k3s[7084]: E0905 15:06:54.804795    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:54 vlan10fc8truenas k3s[7084]: E0905 15:06:54.805160    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:06:57 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:06:57+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:02 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:02+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:02 vlan10fc8truenas k3s[7084]: E0905 15:07:02.707338    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:07:04 vlan10fc8truenas k3s[7084]: E0905 15:07:04.816618    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:04 vlan10fc8truenas k3s[7084]: E0905 15:07:04.816850    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:04 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:07:04.818+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:07:06 vlan10fc8truenas k3s[7084]: E0905 15:07:06.702811    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:07:07 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:07+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:12 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:12+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:14 vlan10fc8truenas k3s[7084]: E0905 15:07:14.707902    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:07:14 vlan10fc8truenas k3s[7084]: E0905 15:07:14.833770    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:14 vlan10fc8truenas k3s[7084]: E0905 15:07:14.834183    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:17 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:17+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:17 vlan10fc8truenas k3s[7084]: E0905 15:07:17.702716    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:07:22 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:22+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:24 vlan10fc8truenas k3s[7084]: E0905 15:07:24.851815    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:24 vlan10fc8truenas k3s[7084]: E0905 15:07:24.852172    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:27 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:27+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:29 vlan10fc8truenas k3s[7084]: E0905 15:07:29.707120    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:07:30 vlan10fc8truenas k3s[7084]: E0905 15:07:30.702582    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:07:32 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:32+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:34 vlan10fc8truenas k3s[7084]: E0905 15:07:34.865438    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:34 vlan10fc8truenas k3s[7084]: E0905 15:07:34.865777    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:37 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:37+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:41 vlan10fc8truenas k3s[7084]: E0905 15:07:41.706980    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:07:42 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:42+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:43 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:07:43.273+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:07:44 vlan10fc8truenas k3s[7084]: E0905 15:07:44.701990    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:07:44 vlan10fc8truenas k3s[7084]: E0905 15:07:44.876816    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:44 vlan10fc8truenas k3s[7084]: E0905 15:07:44.877154    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:47 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:47+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:52 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:52+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:54 vlan10fc8truenas k3s[7084]: E0905 15:07:54.894038    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:54 vlan10fc8truenas k3s[7084]: E0905 15:07:54.894430    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:07:55 vlan10fc8truenas k3s[7084]: E0905 15:07:55.707523    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:07:57 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:07:57+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:07:58 vlan10fc8truenas k3s[7084]: E0905 15:07:58.702029    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:08:02 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:02+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:04 vlan10fc8truenas k3s[7084]: E0905 15:08:04.911533    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:04 vlan10fc8truenas k3s[7084]: E0905 15:08:04.911927    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:07 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:07+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:09 vlan10fc8truenas k3s[7084]: E0905 15:08:09.706680    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:08:12 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:12+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:13 vlan10fc8truenas k3s[7084]: E0905 15:08:13.702054    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:08:14 vlan10fc8truenas k3s[7084]: E0905 15:08:14.930278    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:14 vlan10fc8truenas k3s[7084]: E0905 15:08:14.930567    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:17 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:17+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:20 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:08:20.971+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:08:22 vlan10fc8truenas k3s[7084]: E0905 15:08:22.708320    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:08:22 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:22+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:24 vlan10fc8truenas k3s[7084]: E0905 15:08:24.949923    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:24 vlan10fc8truenas k3s[7084]: E0905 15:08:24.950431    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:26 vlan10fc8truenas k3s[7084]: E0905 15:08:26.701899    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:08:27 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:27+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:32 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:32+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:34 vlan10fc8truenas k3s[7084]: E0905 15:08:34.967198    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:34 vlan10fc8truenas k3s[7084]: E0905 15:08:34.967568    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:37 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:37+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:38 vlan10fc8truenas k3s[7084]: E0905 15:08:38.701544    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:08:42 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:42+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:44 vlan10fc8truenas k3s[7084]: E0905 15:08:44.984471    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:44 vlan10fc8truenas k3s[7084]: E0905 15:08:44.984750    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:47 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:47+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:50 vlan10fc8truenas k3s[7084]: E0905 15:08:50.702872    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:08:50 vlan10fc8truenas k3s[7084]: E0905 15:08:50.773788    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:08:50 vlan10fc8truenas k3s[7084]: E0905 15:08:50.773864    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:08:50 vlan10fc8truenas k3s[7084]: E0905 15:08:50.774035    7084 kuberuntime_manager.go:862] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:08:52 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:52+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:08:54 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:08:54.523+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:08:55 vlan10fc8truenas k3s[7084]: E0905 15:08:55.002308    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:55 vlan10fc8truenas k3s[7084]: E0905 15:08:55.002592    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:08:57 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:08:57+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:02 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:02+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:05 vlan10fc8truenas k3s[7084]: E0905 15:09:05.019974    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:05 vlan10fc8truenas k3s[7084]: E0905 15:09:05.020265    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:05 vlan10fc8truenas k3s[7084]: E0905 15:09:05.701809    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:09:05 vlan10fc8truenas k3s[7084]: E0905 15:09:05.792740    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:09:05 vlan10fc8truenas k3s[7084]: E0905 15:09:05.792804    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:09:05 vlan10fc8truenas k3s[7084]: E0905 15:09:05.792963    7084 kuberuntime_manager.go:862] container &Container{Name:csi-snapshotter,Image:k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:09:08 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:08+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:13 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:13+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:15 vlan10fc8truenas k3s[7084]: E0905 15:09:15.037583    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:15 vlan10fc8truenas k3s[7084]: E0905 15:09:15.037952    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:17 vlan10fc8truenas k3s[7084]: E0905 15:09:17.702695    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:09:18 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:18+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:20 vlan10fc8truenas k3s[7084]: E0905 15:09:20.829794    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:09:20 vlan10fc8truenas k3s[7084]: E0905 15:09:20.829889    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:09:20 vlan10fc8truenas k3s[7084]: E0905 15:09:20.830059    7084 kuberuntime_manager.go:862] container &Container{Name:snapshot-controller,Image:k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0,Command:[],Args:[--v=5 --leader-election=true],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:09:20 vlan10fc8truenas k3s[7084]: E0905 15:09:20.832428    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-snapshotter\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"snapshot-controller\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:09:23 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:23+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:25 vlan10fc8truenas k3s[7084]: E0905 15:09:25.055626    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:25 vlan10fc8truenas k3s[7084]: E0905 15:09:25.055999    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:26 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:09:26.391+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:09:28 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:28+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:28 vlan10fc8truenas k3s[7084]: E0905 15:09:28.701930    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:09:33 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:33+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:35 vlan10fc8truenas k3s[7084]: E0905 15:09:35.073326    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:35 vlan10fc8truenas k3s[7084]: E0905 15:09:35.073632    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:35 vlan10fc8truenas k3s[7084]: E0905 15:09:35.707734    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:09:38 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:38+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:42 vlan10fc8truenas k3s[7084]: E0905 15:09:42.702092    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:09:43 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:43+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:45 vlan10fc8truenas k3s[7084]: E0905 15:09:45.091568    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:45 vlan10fc8truenas k3s[7084]: E0905 15:09:45.091931    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:46 vlan10fc8truenas k3s[7084]: E0905 15:09:46.707744    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:09:48 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:48+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:53 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:53+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:09:55 vlan10fc8truenas k3s[7084]: E0905 15:09:55.110423    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:55 vlan10fc8truenas k3s[7084]: E0905 15:09:55.110707    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:09:57 vlan10fc8truenas k3s[7084]: E0905 15:09:57.701941    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:09:58 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:09:58+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:03 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:03+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:05 vlan10fc8truenas k3s[7084]: E0905 15:10:05.127707    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:05 vlan10fc8truenas k3s[7084]: E0905 15:10:05.128080    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:08 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:08+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:09 vlan10fc8truenas k3s[7084]: E0905 15:10:09.702167    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:10:10 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:10:10.506+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:10:13 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:13+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:14 vlan10fc8truenas k3s[7084]: E0905 15:10:14.772664    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": dial tcp 108.177.97.82:443: i/o timeout" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:10:14 vlan10fc8truenas k3s[7084]: E0905 15:10:14.772731    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": dial tcp 108.177.97.82:443: i/o timeout" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:10:14 vlan10fc8truenas k3s[7084]: E0905 15:10:14.772983    7084 kuberuntime_manager.go:862] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 108.177.97.82:443: i/o timeout
Sep 05 15:10:14 vlan10fc8truenas k3s[7084]: E0905 15:10:14.773132    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": dial tcp 108.177.97.82:443: i/o timeout\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:10:15 vlan10fc8truenas k3s[7084]: E0905 15:10:15.146343    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:15 vlan10fc8truenas k3s[7084]: E0905 15:10:15.146722    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:18 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:18+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:22 vlan10fc8truenas k3s[7084]: E0905 15:10:22.702354    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:10:23 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:23+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:25 vlan10fc8truenas k3s[7084]: E0905 15:10:25.163795    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:25 vlan10fc8truenas k3s[7084]: E0905 15:10:25.164079    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:25 vlan10fc8truenas k3s[7084]: E0905 15:10:25.707577    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:10:28 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:28+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:33 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:33+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:35 vlan10fc8truenas k3s[7084]: E0905 15:10:35.181578    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:35 vlan10fc8truenas k3s[7084]: E0905 15:10:35.181938    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:35 vlan10fc8truenas k3s[7084]: E0905 15:10:35.702374    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:10:38 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:38+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:38 vlan10fc8truenas k3s[7084]: E0905 15:10:38.707298    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:10:43 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:43+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:45 vlan10fc8truenas k3s[7084]: E0905 15:10:45.199407    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:45 vlan10fc8truenas k3s[7084]: E0905 15:10:45.199577    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:47 vlan10fc8truenas k3s[7084]: E0905 15:10:47.702521    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:10:48 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:48+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:50 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:10:50.608+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:10:50 vlan10fc8truenas k3s[7084]: E0905 15:10:50.707834    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:10:53 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:53+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:10:55 vlan10fc8truenas k3s[7084]: E0905 15:10:55.216815    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:55 vlan10fc8truenas k3s[7084]: E0905 15:10:55.217093    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:10:58 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:10:58+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:01 vlan10fc8truenas k3s[7084]: E0905 15:11:01.702431    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:11:01 vlan10fc8truenas k3s[7084]: E0905 15:11:01.707731    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:11:03 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:03+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:05 vlan10fc8truenas k3s[7084]: E0905 15:11:05.234666    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:05 vlan10fc8truenas k3s[7084]: E0905 15:11:05.234960    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:08 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:08+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:13 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:13+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:15 vlan10fc8truenas k3s[7084]: E0905 15:11:15.252093    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:15 vlan10fc8truenas k3s[7084]: E0905 15:11:15.252453    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:15 vlan10fc8truenas k3s[7084]: E0905 15:11:15.707412    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:11:16 vlan10fc8truenas k3s[7084]: E0905 15:11:16.702728    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:11:18 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:18+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:23 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:23+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:24 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:11:24.958+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:11:25 vlan10fc8truenas k3s[7084]: E0905 15:11:25.270301    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:25 vlan10fc8truenas k3s[7084]: E0905 15:11:25.270585    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:28 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:28+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:29 vlan10fc8truenas k3s[7084]: E0905 15:11:29.706817    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:11:33 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:33+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:35 vlan10fc8truenas k3s[7084]: E0905 15:11:35.287425    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:35 vlan10fc8truenas k3s[7084]: E0905 15:11:35.287711    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:38 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:38+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:43 vlan10fc8truenas k3s[7084]: E0905 15:11:43.707956    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:11:43 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:43+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:43 vlan10fc8truenas k3s[7084]: E0905 15:11:43.802487    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:11:43 vlan10fc8truenas k3s[7084]: E0905 15:11:43.802557    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": context deadline exceeded" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:11:43 vlan10fc8truenas k3s[7084]: E0905 15:11:43.802945    7084 kuberuntime_manager.go:862] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-m69jt,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-58727_kube-system(f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": context deadline exceeded
Sep 05 15:11:43 vlan10fc8truenas k3s[7084]: E0905 15:11:43.803028    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": context deadline exceeded\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:11:45 vlan10fc8truenas k3s[7084]: E0905 15:11:45.305753    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:45 vlan10fc8truenas k3s[7084]: E0905 15:11:45.306106    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:48 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:48+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:53 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:53+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:11:55 vlan10fc8truenas k3s[7084]: E0905 15:11:55.323761    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:55 vlan10fc8truenas k3s[7084]: E0905 15:11:55.324112    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:11:56 vlan10fc8truenas k3s[7084]: E0905 15:11:56.701675    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:11:57 vlan10fc8truenas k3s[7084]: E0905 15:11:57.708358    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:11:58 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:11:58+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:03 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:03+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:05 vlan10fc8truenas k3s[7084]: E0905 15:12:05.341788    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:05 vlan10fc8truenas k3s[7084]: E0905 15:12:05.342080    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:08 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:08+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:09 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:12:09.697+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:12:10 vlan10fc8truenas k3s[7084]: E0905 15:12:10.702499    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:12:11 vlan10fc8truenas k3s[7084]: E0905 15:12:11.707364    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:12:13 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:13+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:15 vlan10fc8truenas k3s[7084]: E0905 15:12:15.359344    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:15 vlan10fc8truenas k3s[7084]: E0905 15:12:15.359699    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:18 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:18+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:23 vlan10fc8truenas k3s[7084]: E0905 15:12:23.707683    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:12:23 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:23+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:24 vlan10fc8truenas k3s[7084]: E0905 15:12:24.702742    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:12:25 vlan10fc8truenas k3s[7084]: E0905 15:12:25.376930    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:25 vlan10fc8truenas k3s[7084]: E0905 15:12:25.377204    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:28 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:28+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:33 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:33+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:35 vlan10fc8truenas k3s[7084]: E0905 15:12:35.394276    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:35 vlan10fc8truenas k3s[7084]: E0905 15:12:35.394563    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:36 vlan10fc8truenas k3s[7084]: E0905 15:12:36.702097    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:12:37 vlan10fc8truenas k3s[7084]: E0905 15:12:37.707639    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:12:38 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:38+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:42 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:12:42.068+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:12:43 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:43+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:45 vlan10fc8truenas k3s[7084]: E0905 15:12:45.412154    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:45 vlan10fc8truenas k3s[7084]: E0905 15:12:45.412562    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:47 vlan10fc8truenas k3s[7084]: E0905 15:12:47.702075    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:12:48 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:48+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:52 vlan10fc8truenas k3s[7084]: E0905 15:12:52.706931    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:12:54 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:54+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:12:55 vlan10fc8truenas k3s[7084]: E0905 15:12:55.429101    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:55 vlan10fc8truenas k3s[7084]: E0905 15:12:55.429477    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:12:59 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:12:59+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:02 vlan10fc8truenas k3s[7084]: E0905 15:13:02.702534    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:13:03 vlan10fc8truenas k3s[7084]: E0905 15:13:03.707786    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:13:04 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:04+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:05 vlan10fc8truenas k3s[7084]: E0905 15:13:05.447110    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:05 vlan10fc8truenas k3s[7084]: E0905 15:13:05.447470    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:09 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:09+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:14 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:14+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:14 vlan10fc8truenas k3s[7084]: E0905 15:13:14.707814    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:13:15 vlan10fc8truenas k3s[7084]: E0905 15:13:15.465312    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:15 vlan10fc8truenas k3s[7084]: E0905 15:13:15.465618    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:15 vlan10fc8truenas k3s[7084]: E0905 15:13:15.701376    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:13:19 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:19+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:23 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:13:23.330+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:13:24 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:24+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:25 vlan10fc8truenas k3s[7084]: E0905 15:13:25.482999    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:25 vlan10fc8truenas k3s[7084]: E0905 15:13:25.483280    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:28 vlan10fc8truenas k3s[7084]: E0905 15:13:28.702369    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:13:28 vlan10fc8truenas k3s[7084]: E0905 15:13:28.707242    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:13:29 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:29+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:34 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:34+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:35 vlan10fc8truenas k3s[7084]: E0905 15:13:35.500581    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:35 vlan10fc8truenas k3s[7084]: E0905 15:13:35.500954    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:39 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:39+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:41 vlan10fc8truenas k3s[7084]: E0905 15:13:41.707792    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:13:43 vlan10fc8truenas k3s[7084]: E0905 15:13:43.702834    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:13:44 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:44+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:45 vlan10fc8truenas k3s[7084]: E0905 15:13:45.518807    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:45 vlan10fc8truenas k3s[7084]: E0905 15:13:45.519234    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:49 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:49+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:54 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:54+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:13:55 vlan10fc8truenas k3s[7084]: E0905 15:13:55.536394    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:55 vlan10fc8truenas k3s[7084]: E0905 15:13:55.536741    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:13:58 vlan10fc8truenas k3s[7084]: E0905 15:13:58.702824    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:13:59 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:13:59+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:04 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:04+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:05 vlan10fc8truenas k3s[7084]: E0905 15:14:05.554052    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:05 vlan10fc8truenas k3s[7084]: E0905 15:14:05.554407    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:07 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:14:07.156+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:14:09 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:09+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:09 vlan10fc8truenas k3s[7084]: E0905 15:14:09.822338    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:14:09 vlan10fc8truenas k3s[7084]: E0905 15:14:09.822415    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:14:09 vlan10fc8truenas k3s[7084]: E0905 15:14:09.822584    7084 kuberuntime_manager.go:862] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:14:12 vlan10fc8truenas k3s[7084]: E0905 15:14:12.702431    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:14:14 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:14+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:15 vlan10fc8truenas k3s[7084]: E0905 15:14:15.572257    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:15 vlan10fc8truenas k3s[7084]: E0905 15:14:15.572542    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:19 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:19+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:24 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:24+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:24 vlan10fc8truenas k3s[7084]: E0905 15:14:24.888179    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": dial tcp 108.177.97.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:14:24 vlan10fc8truenas k3s[7084]: E0905 15:14:24.888256    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": dial tcp 108.177.97.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:14:24 vlan10fc8truenas k3s[7084]: E0905 15:14:24.888434    7084 kuberuntime_manager.go:862] container &Container{Name:csi-snapshotter,Image:k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 108.177.97.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)
Sep 05 15:14:25 vlan10fc8truenas k3s[7084]: E0905 15:14:25.589845    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:25 vlan10fc8truenas k3s[7084]: E0905 15:14:25.590212    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:27 vlan10fc8truenas k3s[7084]: E0905 15:14:27.702257    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:14:29 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:29+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:34 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:34+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:35 vlan10fc8truenas k3s[7084]: E0905 15:14:35.607853    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:35 vlan10fc8truenas k3s[7084]: E0905 15:14:35.608227    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:39 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:39+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:39 vlan10fc8truenas k3s[7084]: E0905 15:14:39.947363    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:14:39 vlan10fc8truenas k3s[7084]: E0905 15:14:39.947437    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:14:39 vlan10fc8truenas k3s[7084]: E0905 15:14:39.947596    7084 kuberuntime_manager.go:862] container &Container{Name:snapshot-controller,Image:k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0,Command:[],Args:[--v=5 --leader-election=true],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:14:39 vlan10fc8truenas k3s[7084]: E0905 15:14:39.949971    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-snapshotter\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": dial tcp 108.177.97.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"snapshot-controller\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:14:41 vlan10fc8truenas k3s[7084]: E0905 15:14:41.702409    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:14:41 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:14:41.798+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:14:44 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:44+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:45 vlan10fc8truenas k3s[7084]: E0905 15:14:45.625572    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:45 vlan10fc8truenas k3s[7084]: E0905 15:14:45.625850    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:49 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:49+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:51 vlan10fc8truenas k3s[7084]: E0905 15:14:51.707126    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:14:53 vlan10fc8truenas k3s[7084]: E0905 15:14:53.702516    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:14:54 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:54+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:14:55 vlan10fc8truenas k3s[7084]: E0905 15:14:55.642190    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:55 vlan10fc8truenas k3s[7084]: E0905 15:14:55.642566    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:14:59 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:14:59+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:04 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:04+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:05 vlan10fc8truenas k3s[7084]: E0905 15:15:05.659814    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:05 vlan10fc8truenas k3s[7084]: E0905 15:15:05.660093    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:05 vlan10fc8truenas k3s[7084]: E0905 15:15:05.707457    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:15:07 vlan10fc8truenas k3s[7084]: E0905 15:15:07.701611    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:15:09 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:09+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:13 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:15:13.201+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:15:14 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:14+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:15 vlan10fc8truenas k3s[7084]: E0905 15:15:15.676595    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:15 vlan10fc8truenas k3s[7084]: E0905 15:15:15.676958    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:19 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:19+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:22 vlan10fc8truenas k3s[7084]: E0905 15:15:22.702163    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:15:24 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:24+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:25 vlan10fc8truenas k3s[7084]: E0905 15:15:25.693265    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:25 vlan10fc8truenas k3s[7084]: E0905 15:15:25.693647    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:29 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:29+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:34 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:34+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:34 vlan10fc8truenas k3s[7084]: E0905 15:15:34.784768    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:15:34 vlan10fc8truenas k3s[7084]: E0905 15:15:34.784840    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:15:34 vlan10fc8truenas k3s[7084]: E0905 15:15:34.785054    7084 kuberuntime_manager.go:862] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:15:34 vlan10fc8truenas k3s[7084]: E0905 15:15:34.785215    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:15:35 vlan10fc8truenas k3s[7084]: E0905 15:15:35.711581    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:35 vlan10fc8truenas k3s[7084]: E0905 15:15:35.711866    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:37 vlan10fc8truenas k3s[7084]: E0905 15:15:37.701993    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:15:39 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:39+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:44 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:44+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:45 vlan10fc8truenas k3s[7084]: E0905 15:15:45.708141    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:15:45 vlan10fc8truenas k3s[7084]: E0905 15:15:45.728289    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:45 vlan10fc8truenas k3s[7084]: E0905 15:15:45.728580    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:48 vlan10fc8truenas k3s[7084]: E0905 15:15:48.702176    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:15:49 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:49+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:51 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:15:51.908+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:15:54 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:54+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:15:55 vlan10fc8truenas k3s[7084]: E0905 15:15:55.746166    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:55 vlan10fc8truenas k3s[7084]: E0905 15:15:55.746553    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:15:57 vlan10fc8truenas k3s[7084]: E0905 15:15:57.706834    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:15:59 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:15:59+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:03 vlan10fc8truenas k3s[7084]: E0905 15:16:03.702095    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:16:04 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:04+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:05 vlan10fc8truenas k3s[7084]: E0905 15:16:05.764690    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:05 vlan10fc8truenas k3s[7084]: E0905 15:16:05.765074    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:09 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:09+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:10 vlan10fc8truenas k3s[7084]: E0905 15:16:10.707103    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:16:14 vlan10fc8truenas k3s[7084]: E0905 15:16:14.702226    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:16:14 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:14+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:15 vlan10fc8truenas k3s[7084]: E0905 15:16:15.782619    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:15 vlan10fc8truenas k3s[7084]: E0905 15:16:15.782979    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:19 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:19+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:23 vlan10fc8truenas k3s[7084]: E0905 15:16:23.708289    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:16:24 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:24+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:25 vlan10fc8truenas k3s[7084]: E0905 15:16:25.800479    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:25 vlan10fc8truenas k3s[7084]: E0905 15:16:25.800758    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:28 vlan10fc8truenas k3s[7084]: E0905 15:16:28.701835    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:16:29 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:16:29.783+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:16:29 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:29+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:34 vlan10fc8truenas k3s[7084]: E0905 15:16:34.707041    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:16:35 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:35+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:35 vlan10fc8truenas k3s[7084]: E0905 15:16:35.818554    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:35 vlan10fc8truenas k3s[7084]: E0905 15:16:35.818890    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:39 vlan10fc8truenas k3s[7084]: E0905 15:16:39.701956    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:16:40 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:40+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:45 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:45+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:45 vlan10fc8truenas k3s[7084]: E0905 15:16:45.835886    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:45 vlan10fc8truenas k3s[7084]: E0905 15:16:45.836273    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:46 vlan10fc8truenas k3s[7084]: E0905 15:16:46.708026    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:16:50 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:50+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:55 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:16:55+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:16:55 vlan10fc8truenas k3s[7084]: E0905 15:16:55.854309    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:55 vlan10fc8truenas k3s[7084]: E0905 15:16:55.854667    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:16:58 vlan10fc8truenas k3s[7084]: E0905 15:16:58.707998    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:17:00 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:00+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:05 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:05+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:05 vlan10fc8truenas k3s[7084]: E0905 15:17:05.815938    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:17:05 vlan10fc8truenas k3s[7084]: E0905 15:17:05.816011    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:17:05 vlan10fc8truenas k3s[7084]: E0905 15:17:05.816239    7084 kuberuntime_manager.go:862] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-m69jt,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-58727_kube-system(f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:17:05 vlan10fc8truenas k3s[7084]: E0905 15:17:05.816339    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:17:05 vlan10fc8truenas k3s[7084]: E0905 15:17:05.872146    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:05 vlan10fc8truenas k3s[7084]: E0905 15:17:05.872528    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:06 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:17:06.694+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:17:10 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:10+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:11 vlan10fc8truenas k3s[7084]: E0905 15:17:11.707030    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:17:15 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:15+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:15 vlan10fc8truenas k3s[7084]: E0905 15:17:15.889389    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:15 vlan10fc8truenas k3s[7084]: E0905 15:17:15.889786    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:20 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:20+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:20 vlan10fc8truenas k3s[7084]: E0905 15:17:20.701555    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:17:25 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:25+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:25 vlan10fc8truenas k3s[7084]: E0905 15:17:25.707188    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:17:25 vlan10fc8truenas k3s[7084]: E0905 15:17:25.907349    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:25 vlan10fc8truenas k3s[7084]: E0905 15:17:25.907720    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:30 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:30+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:34 vlan10fc8truenas k3s[7084]: E0905 15:17:34.702086    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:17:35 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:35+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:35 vlan10fc8truenas k3s[7084]: E0905 15:17:35.924331    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:35 vlan10fc8truenas k3s[7084]: E0905 15:17:35.924683    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:38 vlan10fc8truenas k3s[7084]: E0905 15:17:38.707728    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:17:40 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:40+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:45 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:45+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:45 vlan10fc8truenas k3s[7084]: E0905 15:17:45.702008    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:17:45 vlan10fc8truenas k3s[7084]: E0905 15:17:45.941232    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:45 vlan10fc8truenas k3s[7084]: E0905 15:17:45.941512    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:46 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:17:46.575+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:17:50 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:50+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:52 vlan10fc8truenas k3s[7084]: E0905 15:17:52.707524    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:17:55 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:17:55+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:17:55 vlan10fc8truenas k3s[7084]: E0905 15:17:55.959603    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:55 vlan10fc8truenas k3s[7084]: E0905 15:17:55.960016    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:17:58 vlan10fc8truenas k3s[7084]: E0905 15:17:58.701859    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:18:00 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:00+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:05 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:05+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:05 vlan10fc8truenas k3s[7084]: E0905 15:18:05.976727    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:05 vlan10fc8truenas k3s[7084]: E0905 15:18:05.977014    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:07 vlan10fc8truenas k3s[7084]: E0905 15:18:07.707036    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:18:09 vlan10fc8truenas k3s[7084]: E0905 15:18:09.702322    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:18:10 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:10+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:15 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:15+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:15 vlan10fc8truenas k3s[7084]: E0905 15:18:15.994215    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:15 vlan10fc8truenas k3s[7084]: E0905 15:18:15.994570    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:18 vlan10fc8truenas k3s[7084]: E0905 15:18:18.707872    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:18:20 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:20+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:23 vlan10fc8truenas k3s[7084]: E0905 15:18:23.702783    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:18:25 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:25+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:26 vlan10fc8truenas k3s[7084]: E0905 15:18:26.011578    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:26 vlan10fc8truenas k3s[7084]: E0905 15:18:26.011964    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:30 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:30+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:30 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:18:30.572+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:18:33 vlan10fc8truenas k3s[7084]: E0905 15:18:33.706381    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:18:35 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:35+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:36 vlan10fc8truenas k3s[7084]: E0905 15:18:36.029763    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:36 vlan10fc8truenas k3s[7084]: E0905 15:18:36.030140    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:36 vlan10fc8truenas k3s[7084]: E0905 15:18:36.702229    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:18:40 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:40+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:45 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:45+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:46 vlan10fc8truenas k3s[7084]: E0905 15:18:46.047688    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:46 vlan10fc8truenas k3s[7084]: E0905 15:18:46.047962    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:47 vlan10fc8truenas k3s[7084]: E0905 15:18:47.702032    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:18:47 vlan10fc8truenas k3s[7084]: E0905 15:18:47.706781    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:18:50 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:50+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:55 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:18:55+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:18:56 vlan10fc8truenas k3s[7084]: E0905 15:18:56.065537    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:18:56 vlan10fc8truenas k3s[7084]: E0905 15:18:56.065887    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:00 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:00+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:01 vlan10fc8truenas k3s[7084]: E0905 15:19:01.702075    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:19:02 vlan10fc8truenas k3s[7084]: E0905 15:19:02.708491    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:19:05 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:05+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:06 vlan10fc8truenas k3s[7084]: E0905 15:19:06.083750    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:06 vlan10fc8truenas k3s[7084]: E0905 15:19:06.084118    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:06 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:19:06.118+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:19:10 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:10+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:15 vlan10fc8truenas k3s[7084]: E0905 15:19:15.702328    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:19:15 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:15+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:16 vlan10fc8truenas k3s[7084]: E0905 15:19:16.102163    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:16 vlan10fc8truenas k3s[7084]: E0905 15:19:16.102533    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:20 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:20+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:25 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:25+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:26 vlan10fc8truenas k3s[7084]: E0905 15:19:26.120324    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:26 vlan10fc8truenas k3s[7084]: E0905 15:19:26.120603    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:26 vlan10fc8truenas k3s[7084]: E0905 15:19:26.702557    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:19:30 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:30+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:32 vlan10fc8truenas k3s[7084]: E0905 15:19:32.831251    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": dial tcp 64.233.189.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:19:32 vlan10fc8truenas k3s[7084]: E0905 15:19:32.831320    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": dial tcp 64.233.189.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:19:32 vlan10fc8truenas k3s[7084]: E0905 15:19:32.831491    7084 kuberuntime_manager.go:862] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 64.233.189.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)
Sep 05 15:19:35 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:35+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:36 vlan10fc8truenas k3s[7084]: E0905 15:19:36.138718    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:36 vlan10fc8truenas k3s[7084]: E0905 15:19:36.139110    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:39 vlan10fc8truenas k3s[7084]: E0905 15:19:39.702701    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:19:40 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:40+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:45 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:45+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:46 vlan10fc8truenas k3s[7084]: E0905 15:19:46.156751    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:46 vlan10fc8truenas k3s[7084]: E0905 15:19:46.157105    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:46 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:19:46.855+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:19:47 vlan10fc8truenas k3s[7084]: E0905 15:19:47.850685    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:19:47 vlan10fc8truenas k3s[7084]: E0905 15:19:47.850759    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:19:47 vlan10fc8truenas k3s[7084]: E0905 15:19:47.850955    7084 kuberuntime_manager.go:862] container &Container{Name:csi-snapshotter,Image:k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:19:50 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:50+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:51 vlan10fc8truenas k3s[7084]: E0905 15:19:51.702800    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:19:55 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:19:55+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:19:56 vlan10fc8truenas k3s[7084]: E0905 15:19:56.174507    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:19:56 vlan10fc8truenas k3s[7084]: E0905 15:19:56.174855    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:00 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:00+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:02 vlan10fc8truenas k3s[7084]: E0905 15:20:02.895735    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:20:02 vlan10fc8truenas k3s[7084]: E0905 15:20:02.895816    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:20:02 vlan10fc8truenas k3s[7084]: E0905 15:20:02.895988    7084 kuberuntime_manager.go:862] container &Container{Name:snapshot-controller,Image:k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0,Command:[],Args:[--v=5 --leader-election=true],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:20:02 vlan10fc8truenas k3s[7084]: E0905 15:20:02.898142    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": dial tcp 64.233.189.82:443: i/o timeout (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-snapshotter\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"snapshot-controller\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:20:05 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:05+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:06 vlan10fc8truenas k3s[7084]: E0905 15:20:06.192577    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:06 vlan10fc8truenas k3s[7084]: E0905 15:20:06.192933    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:06 vlan10fc8truenas k3s[7084]: E0905 15:20:06.701460    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:20:10 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:10+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:15 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:15+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:16 vlan10fc8truenas k3s[7084]: E0905 15:20:16.211356    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:16 vlan10fc8truenas k3s[7084]: E0905 15:20:16.211717    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:16 vlan10fc8truenas k3s[7084]: E0905 15:20:16.707949    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:20:17 vlan10fc8truenas k3s[7084]: E0905 15:20:17.701782    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:20:20 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:20:20.982+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:20:21 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:21+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:26 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:26+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:26 vlan10fc8truenas k3s[7084]: E0905 15:20:26.228664    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:26 vlan10fc8truenas k3s[7084]: E0905 15:20:26.229061    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:31 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:31+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:31 vlan10fc8truenas k3s[7084]: E0905 15:20:31.702326    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:20:31 vlan10fc8truenas k3s[7084]: E0905 15:20:31.707700    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:20:36 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:36+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:36 vlan10fc8truenas k3s[7084]: E0905 15:20:36.247463    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:36 vlan10fc8truenas k3s[7084]: E0905 15:20:36.247840    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:41 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:41+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:46 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:46+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:46 vlan10fc8truenas k3s[7084]: E0905 15:20:46.265728    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:46 vlan10fc8truenas k3s[7084]: E0905 15:20:46.266114    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:46 vlan10fc8truenas k3s[7084]: E0905 15:20:46.703253    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:20:51 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:51+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:56 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:20:56+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:20:56 vlan10fc8truenas k3s[7084]: E0905 15:20:56.283894    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:56 vlan10fc8truenas k3s[7084]: E0905 15:20:56.284256    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:20:58 vlan10fc8truenas k3s[7084]: E0905 15:20:58.702175    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:20:59 vlan10fc8truenas k3s[7084]: E0905 15:20:59.794415    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:20:59 vlan10fc8truenas k3s[7084]: E0905 15:20:59.794485    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:20:59 vlan10fc8truenas k3s[7084]: E0905 15:20:59.794699    7084 kuberuntime_manager.go:862] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:20:59 vlan10fc8truenas k3s[7084]: E0905 15:20:59.794845    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:21:01 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:01+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:03 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:21:03.480+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:21:06 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:06+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:06 vlan10fc8truenas k3s[7084]: E0905 15:21:06.301364    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:06 vlan10fc8truenas k3s[7084]: E0905 15:21:06.301654    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:09 vlan10fc8truenas k3s[7084]: E0905 15:21:09.702567    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:21:11 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:11+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:12 vlan10fc8truenas k3s[7084]: E0905 15:21:12.707840    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:21:16 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:16+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:16 vlan10fc8truenas k3s[7084]: E0905 15:21:16.319491    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:16 vlan10fc8truenas k3s[7084]: E0905 15:21:16.319767    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:21 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:21+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:22 vlan10fc8truenas k3s[7084]: E0905 15:21:22.700494    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:21:23 vlan10fc8truenas k3s[7084]: E0905 15:21:23.706769    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:21:26 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:26+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:26 vlan10fc8truenas k3s[7084]: E0905 15:21:26.337799    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:26 vlan10fc8truenas k3s[7084]: E0905 15:21:26.338159    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:31 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:31+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:35 vlan10fc8truenas k3s[7084]: E0905 15:21:35.707175    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:21:36 vlan10fc8truenas k3s[7084]: E0905 15:21:36.354850    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:36 vlan10fc8truenas k3s[7084]: E0905 15:21:36.355202    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:36 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:36+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:37 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:21:37.187+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:21:37 vlan10fc8truenas k3s[7084]: E0905 15:21:37.702003    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:21:41 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:41+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:46 vlan10fc8truenas k3s[7084]: E0905 15:21:46.372144    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:46 vlan10fc8truenas k3s[7084]: E0905 15:21:46.372482    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:46 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:46+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:46 vlan10fc8truenas k3s[7084]: E0905 15:21:46.707456    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:21:50 vlan10fc8truenas k3s[7084]: E0905 15:21:50.703238    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:21:51 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:51+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:21:56 vlan10fc8truenas k3s[7084]: E0905 15:21:56.390191    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:56 vlan10fc8truenas k3s[7084]: E0905 15:21:56.390562    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:21:56 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:21:56+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:01 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:01+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:01 vlan10fc8truenas k3s[7084]: E0905 15:22:01.708511    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:22:02 vlan10fc8truenas k3s[7084]: E0905 15:22:02.701888    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:22:06 vlan10fc8truenas k3s[7084]: E0905 15:22:06.408122    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:06 vlan10fc8truenas k3s[7084]: E0905 15:22:06.408466    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:06 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:06+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:11 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:11+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:16 vlan10fc8truenas k3s[7084]: E0905 15:22:16.425743    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:16 vlan10fc8truenas k3s[7084]: E0905 15:22:16.426022    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:16 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:16+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:16 vlan10fc8truenas k3s[7084]: E0905 15:22:16.707018    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:22:21 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:21+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:21 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:22:21.778+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:22:26 vlan10fc8truenas k3s[7084]: E0905 15:22:26.443634    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:26 vlan10fc8truenas k3s[7084]: E0905 15:22:26.443918    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:26 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:26+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:27 vlan10fc8truenas k3s[7084]: E0905 15:22:27.707781    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:22:31 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:31+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:32 vlan10fc8truenas k3s[7084]: E0905 15:22:32.799337    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:22:32 vlan10fc8truenas k3s[7084]: E0905 15:22:32.799419    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:22:32 vlan10fc8truenas k3s[7084]: E0905 15:22:32.799641    7084 kuberuntime_manager.go:862] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-m69jt,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-58727_kube-system(f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:22:32 vlan10fc8truenas k3s[7084]: E0905 15:22:32.799734    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:22:36 vlan10fc8truenas k3s[7084]: E0905 15:22:36.461984    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:36 vlan10fc8truenas k3s[7084]: E0905 15:22:36.462261    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:36 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:36+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:39 vlan10fc8truenas k3s[7084]: E0905 15:22:39.707787    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:22:41 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:41+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:45 vlan10fc8truenas k3s[7084]: E0905 15:22:45.702160    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:22:46 vlan10fc8truenas k3s[7084]: E0905 15:22:46.479680    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:46 vlan10fc8truenas k3s[7084]: E0905 15:22:46.480033    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:46 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:46+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:50 vlan10fc8truenas k3s[7084]: E0905 15:22:50.707076    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:22:51 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:51+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:56 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:22:56.085+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:22:56 vlan10fc8truenas k3s[7084]: E0905 15:22:56.497554    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:56 vlan10fc8truenas k3s[7084]: E0905 15:22:56.497919    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:22:56 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:22:56+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:22:57 vlan10fc8truenas k3s[7084]: E0905 15:22:57.702009    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:23:01 vlan10fc8truenas k3s[7084]: E0905 15:23:01.707253    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:23:01 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:01+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:06 vlan10fc8truenas k3s[7084]: E0905 15:23:06.515481    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:06 vlan10fc8truenas k3s[7084]: E0905 15:23:06.515759    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:06 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:06+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:11 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:11+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:12 vlan10fc8truenas k3s[7084]: E0905 15:23:12.702274    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:23:15 vlan10fc8truenas k3s[7084]: E0905 15:23:15.707364    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:23:16 vlan10fc8truenas k3s[7084]: E0905 15:23:16.532812    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:16 vlan10fc8truenas k3s[7084]: E0905 15:23:16.533180    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:16 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:16+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:21 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:21+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:24 vlan10fc8truenas k3s[7084]: E0905 15:23:24.702099    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:23:26 vlan10fc8truenas k3s[7084]: E0905 15:23:26.550956    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:26 vlan10fc8truenas k3s[7084]: E0905 15:23:26.551324    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:26 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:26+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:27 vlan10fc8truenas k3s[7084]: E0905 15:23:27.707583    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:23:31 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:31+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:36 vlan10fc8truenas k3s[7084]: E0905 15:23:36.568409    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:36 vlan10fc8truenas k3s[7084]: E0905 15:23:36.568780    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:36 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:36+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:37 vlan10fc8truenas k3s[7084]: E0905 15:23:37.702241    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:23:38 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:23:38.429+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:23:39 vlan10fc8truenas k3s[7084]: E0905 15:23:39.708575    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:23:41 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:41+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:46 vlan10fc8truenas k3s[7084]: E0905 15:23:46.588236    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:46 vlan10fc8truenas k3s[7084]: E0905 15:23:46.588616    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:46 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:46+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:50 vlan10fc8truenas k3s[7084]: E0905 15:23:50.702342    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:23:51 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:51+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:23:53 vlan10fc8truenas k3s[7084]: E0905 15:23:53.707266    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:23:56 vlan10fc8truenas k3s[7084]: E0905 15:23:56.606449    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:56 vlan10fc8truenas k3s[7084]: E0905 15:23:56.606724    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:23:56 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:23:56+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:02 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:02+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:05 vlan10fc8truenas k3s[7084]: E0905 15:24:05.702545    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:24:06 vlan10fc8truenas k3s[7084]: E0905 15:24:06.623611    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:06 vlan10fc8truenas k3s[7084]: E0905 15:24:06.623893    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:07 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:07+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:07 vlan10fc8truenas k3s[7084]: E0905 15:24:07.707619    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:24:12 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:12+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:16 vlan10fc8truenas k3s[7084]: E0905 15:24:16.641727    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:16 vlan10fc8truenas k3s[7084]: E0905 15:24:16.642083    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:16 vlan10fc8truenas k3s[7084]: E0905 15:24:16.702375    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:24:17 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:17+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:21 vlan10fc8truenas k3s[7084]: E0905 15:24:21.707338    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:24:22 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:22+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:22 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:24:22.727+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:24:26 vlan10fc8truenas k3s[7084]: E0905 15:24:26.659936    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:26 vlan10fc8truenas k3s[7084]: E0905 15:24:26.660287    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:27 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:27+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:29 vlan10fc8truenas k3s[7084]: E0905 15:24:29.702226    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:24:32 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:32+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:32 vlan10fc8truenas k3s[7084]: E0905 15:24:32.708176    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:24:36 vlan10fc8truenas k3s[7084]: E0905 15:24:36.678077    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:36 vlan10fc8truenas k3s[7084]: E0905 15:24:36.678365    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:37 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:37+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:42 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:42+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:42 vlan10fc8truenas k3s[7084]: E0905 15:24:42.702638    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:24:46 vlan10fc8truenas k3s[7084]: E0905 15:24:46.696305    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:46 vlan10fc8truenas k3s[7084]: E0905 15:24:46.696681    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:47 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:47+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:52 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:52+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:56 vlan10fc8truenas k3s[7084]: E0905 15:24:56.714186    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:56 vlan10fc8truenas k3s[7084]: E0905 15:24:56.714570    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:24:57 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:24:57+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:24:57 vlan10fc8truenas k3s[7084]: E0905 15:24:57.702180    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:25:00 vlan10fc8truenas k3s[7084]: E0905 15:25:00.751042    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:25:00 vlan10fc8truenas k3s[7084]: E0905 15:25:00.751122    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
Sep 05 15:25:00 vlan10fc8truenas k3s[7084]: E0905 15:25:00.751299    7084 kuberuntime_manager.go:862] container &Container{Name:csi-resizer,Image:k8s.gcr.io/sig-storage/csi-resizer:v1.2.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:25:02 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:02+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:06 vlan10fc8truenas k3s[7084]: E0905 15:25:06.730534    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:06 vlan10fc8truenas k3s[7084]: E0905 15:25:06.730888    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:07 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:07+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:07 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:25:07.343+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:25:12 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:12+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:12 vlan10fc8truenas k3s[7084]: E0905 15:25:12.702532    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:25:15 vlan10fc8truenas k3s[7084]: E0905 15:25:15.869612    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:25:15 vlan10fc8truenas k3s[7084]: E0905 15:25:15.869688    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0"
Sep 05 15:25:15 vlan10fc8truenas k3s[7084]: E0905 15:25:15.869879    7084 kuberuntime_manager.go:862] container &Container{Name:csi-snapshotter,Image:k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --leader-election],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:25:16 vlan10fc8truenas k3s[7084]: E0905 15:25:16.747448    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:16 vlan10fc8truenas k3s[7084]: E0905 15:25:16.747812    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:17 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:17+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:22 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:22+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:26 vlan10fc8truenas k3s[7084]: E0905 15:25:26.702824    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:25:26 vlan10fc8truenas k3s[7084]: E0905 15:25:26.764959    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:26 vlan10fc8truenas k3s[7084]: E0905 15:25:26.765352    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:27 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:27+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:30 vlan10fc8truenas k3s[7084]: E0905 15:25:30.923559    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:25:30 vlan10fc8truenas k3s[7084]: E0905 15:25:30.923632    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0"
Sep 05 15:25:30 vlan10fc8truenas k3s[7084]: E0905 15:25:30.923791    7084 kuberuntime_manager.go:862] container &Container{Name:snapshot-controller,Image:k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0,Command:[],Args:[--v=5 --leader-election=true],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:25:30 vlan10fc8truenas k3s[7084]: E0905 15:25:30.925996    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-snapshotter\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"snapshot-controller\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:25:32 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:32+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:36 vlan10fc8truenas k3s[7084]: E0905 15:25:36.782720    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:36 vlan10fc8truenas k3s[7084]: E0905 15:25:36.783008    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:37 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:37+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:39 vlan10fc8truenas k3s[7084]: E0905 15:25:39.702764    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:25:42 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:42+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:43 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:25:43.387+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:25:45 vlan10fc8truenas k3s[7084]: E0905 15:25:45.706697    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:25:46 vlan10fc8truenas k3s[7084]: E0905 15:25:46.800571    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:46 vlan10fc8truenas k3s[7084]: E0905 15:25:46.800950    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:47 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:47+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:52 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:52+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:25:52 vlan10fc8truenas k3s[7084]: E0905 15:25:52.702726    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:25:56 vlan10fc8truenas k3s[7084]: E0905 15:25:56.707192    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:25:56 vlan10fc8truenas k3s[7084]: E0905 15:25:56.819088    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:56 vlan10fc8truenas k3s[7084]: E0905 15:25:56.819370    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:25:57 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:25:57+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:02 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:02+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:05 vlan10fc8truenas k3s[7084]: E0905 15:26:05.702413    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:26:06 vlan10fc8truenas k3s[7084]: E0905 15:26:06.837322    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:06 vlan10fc8truenas k3s[7084]: E0905 15:26:06.837721    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:07 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:07+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:12 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:12+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:15 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:26:15.409+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:26:16 vlan10fc8truenas k3s[7084]: E0905 15:26:16.854648    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:16 vlan10fc8truenas k3s[7084]: E0905 15:26:16.855025    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:17 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:17+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:19 vlan10fc8truenas k3s[7084]: E0905 15:26:19.702471    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:26:22 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:22+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:23 vlan10fc8truenas k3s[7084]: E0905 15:26:23.735760    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:26:23 vlan10fc8truenas k3s[7084]: E0905 15:26:23.735837    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" image="k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
Sep 05 15:26:23 vlan10fc8truenas k3s[7084]: E0905 15:26:23.736058    7084 kuberuntime_manager.go:862] container &Container{Name:csi-provisioner,Image:k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0,Command:[],Args:[--csi-address=$(ADDRESS) --v=5 --feature-gates=Topology=true --strict-topology --leader-election --extra-create-metadata=true --enable-capacity=true --default-fstype=ext4],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/var/lib/csi/sockets/pluginproxy/csi.sock,ValueFrom:nil,},EnvVar{Name:NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:socket-dir,ReadOnly:false,MountPath:/var/lib/csi/sockets/pluginproxy/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-s7js7,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-controller-0_kube-system(2728fa87-0c11-42bb-b5a8-9950ab76f184): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Sep 05 15:26:23 vlan10fc8truenas k3s[7084]: E0905 15:26:23.736212    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:26:26 vlan10fc8truenas k3s[7084]: E0905 15:26:26.871690    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:26 vlan10fc8truenas k3s[7084]: E0905 15:26:26.872047    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:27 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:27+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:30 vlan10fc8truenas k3s[7084]: E0905 15:26:30.702172    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:26:32 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:32+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:34 vlan10fc8truenas k3s[7084]: E0905 15:26:34.706596    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:26:36 vlan10fc8truenas k3s[7084]: E0905 15:26:36.889419    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:36 vlan10fc8truenas k3s[7084]: E0905 15:26:36.889827    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:37 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:37+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:42 vlan10fc8truenas k3s[7084]: E0905 15:26:42.702372    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:26:42 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:42+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:45 vlan10fc8truenas k3s[7084]: E0905 15:26:45.707148    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:26:46 vlan10fc8truenas k3s[7084]: E0905 15:26:46.906565    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:46 vlan10fc8truenas k3s[7084]: E0905 15:26:46.906952    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:47 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:47+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:52 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:52+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:26:56 vlan10fc8truenas k3s[7084]: E0905 15:26:56.701887    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:26:56 vlan10fc8truenas k3s[7084]: E0905 15:26:56.924689    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:56 vlan10fc8truenas k3s[7084]: E0905 15:26:56.925075    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:26:57 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:26:57.573+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:26:57 vlan10fc8truenas k3s[7084]: E0905 15:26:57.707828    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:26:57 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:26:57+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:02 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:02+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:06 vlan10fc8truenas k3s[7084]: E0905 15:27:06.942381    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:06 vlan10fc8truenas k3s[7084]: E0905 15:27:06.942667    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:07 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:07+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:08 vlan10fc8truenas k3s[7084]: E0905 15:27:08.707485    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:27:11 vlan10fc8truenas k3s[7084]: E0905 15:27:11.702777    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:27:12 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:12+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:16 vlan10fc8truenas k3s[7084]: E0905 15:27:16.959934    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:16 vlan10fc8truenas k3s[7084]: E0905 15:27:16.960221    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:17 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:17+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:20 vlan10fc8truenas k3s[7084]: E0905 15:27:20.707339    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:27:22 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:22+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:26 vlan10fc8truenas k3s[7084]: E0905 15:27:26.702694    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0\\\"\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:27:26 vlan10fc8truenas k3s[7084]: E0905 15:27:26.977550    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:26 vlan10fc8truenas k3s[7084]: E0905 15:27:26.977830    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:27 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:27+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:32 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:32+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:33 vlan10fc8truenas k3s[7084]: {"level":"warn","ts":"2023-09-05T15:27:33.089+0800","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004e3180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Sep 05 15:27:33 vlan10fc8truenas k3s[7084]: E0905 15:27:33.706788    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:27:36 vlan10fc8truenas k3s[7084]: E0905 15:27:36.995329    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:36 vlan10fc8truenas k3s[7084]: E0905 15:27:36.995621    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:37 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:37+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:42 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:42+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:44 vlan10fc8truenas k3s[7084]: E0905 15:27:44.710534    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:27:47 vlan10fc8truenas k3s[7084]: E0905 15:27:47.011785    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:47 vlan10fc8truenas k3s[7084]: E0905 15:27:47.012148    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:48 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:48+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:53 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:53+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:56 vlan10fc8truenas k3s[7084]: E0905 15:27:56.769953    7084 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": dial tcp 64.233.189.82:443: i/o timeout" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:27:56 vlan10fc8truenas k3s[7084]: E0905 15:27:56.770021    7084 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = Error response from daemon: Get \"https://k8s.gcr.io/v2/\": dial tcp 64.233.189.82:443: i/o timeout" image="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
Sep 05 15:27:56 vlan10fc8truenas k3s[7084]: E0905 15:27:56.770246    7084 kuberuntime_manager.go:862] container &Container{Name:csi-node-driver-registrar,Image:k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0,Command:[],Args:[--v=5 --csi-address=$(ADDRESS) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ADDRESS,Value:/plugin/csi.sock,ValueFrom:nil,},EnvVar{Name:DRIVER_REG_SOCK_PATH,Value:/var/lib/kubelet/plugins/zfs-localpv/csi.sock,ValueFrom:nil,},EnvVar{Name:KUBE_NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:NODE_DRIVER,Value:openebs-zfs,ValueFrom:nil,},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:plugin-dir,ReadOnly:false,MountPath:/plugin,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:registration-dir,ReadOnly:false,MountPath:/registration,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-api-access-m69jt,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:&Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[/bin/sh -c rm -rf /registration/zfs-localpv /registration/zfs-localpv-reg.sock],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod openebs-zfs-node-58727_kube-system(f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 64.233.189.82:443: i/o timeout
Sep 05 15:27:56 vlan10fc8truenas k3s[7084]: E0905 15:27:56.770328    7084 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"rpc error: code = Unknown desc = Error response from daemon: Get \\\"https://k8s.gcr.io/v2/\\\": dial tcp 64.233.189.82:443: i/o timeout\"" pod="kube-system/openebs-zfs-node-58727" podUID=f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3
Sep 05 15:27:57 vlan10fc8truenas k3s[7084]: E0905 15:27:57.027779    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:57 vlan10fc8truenas k3s[7084]: E0905 15:27:57.028132    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:27:58 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:27:58+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:27:59 vlan10fc8truenas k3s[7084]: E0905 15:27:59.706847    7084 pod_workers.go:965] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-resizer\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-resizer:v1.2.0\\\"\", failed to \"StartContainer\" for \"csi-snapshotter\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0\\\"\", failed to \"StartContainer\" for \"snapshot-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0\\\"\", failed to \"StartContainer\" for \"csi-provisioner\" with ImagePullBackOff: \"Back-off pulling image \\\"k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0\\\"\"]" pod="kube-system/openebs-zfs-controller-0" podUID=2728fa87-0c11-42bb-b5a8-9950ab76f184
Sep 05 15:28:03 vlan10fc8truenas k3s[7084]: time="2023-09-05T15:28:03+08:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Sep 05 15:28:07 vlan10fc8truenas k3s[7084]: E0905 15:28:07.045468    7084 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
Sep 05 15:28:07 vlan10fc8truenas k3s[7084]: E0905 15:28:07.045767    7084 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log\": failed to reopen container log \"1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-58727_f4358f39-aa58-4c9c-aa35-5e3d02bcf3f3/openebs-zfs-plugin/3.log" containerID="1db75a7a7b3b007867116aa4aa58e615e1d4f64527ae83ba8ec3a9afcb15cd62"
debug finished in 6 seconds for K3s Logs (last 1000 lines)


