Skip to main content

Troubleshooting

1. http: server gave HTTP response to HTTPS client​

CSGHub is installed by default using an insecure registry (i.e., <domain or IPv4>:5000 mentioned above). You need to ensure that Kubernetes can pull images from this registry. Therefore, you need to configure the following on each Kubernetes node:

  1. Before configuration, please confirm whether the configuration file /etc/containerd/config.toml exists. If it does not exist, you can use the following command to create it.
mkdir -p /etc/containerd/ && containerd config default >/etc/containerd/config.toml

  1. Configure config_path

    • Containerd 1.x

      version = 2

      [plugins."io.containerd.grpc.v1.cri".registry]
           config_path = "/etc/containerd/certs.d"

    • Containerd 2.x

       version = 3

       [plugins."io.containerd.cri.v1.images".registry]
            config_path = "/etc/containerd/certs.d"

  2. Configure hosts.toml

    # Create Registry certs.d directory
    mkdir /etc/containerd/certs.d/<domain or IPv4>:5000

    # Defined untrusted docker configuraiton
    cat > /etc/containerd/certs.d/<domain or IPv4>:5000/hosts.toml <<EOF
    server = "http://<domain or IPv4>:5000"

    [host."http://<domain or IPv4>:5000"]
       capabilities = ["pull", "resolve", "push"]
       skip_verify = true
       plain-http = true
    EOF

  3. Restart containerd service

    systemctl restart containerd

2. dial tcp: lookup casdoor.example.com on 10.43.0.10:53: no such host​

This problem occurs because the cluster cannot resolve the domain name. If it is a public domain name, please configure domain name resolution. If it is a custom domain name, please configure CoreDNS and Hosts resolution. CoreDNS resolution configuration is as follows:

# Add custom domain name resolution
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  example.server: |
    example.com {
      hosts {
        172.25.11.131 csghub.example.com csghub
        172.25.11.131 casdoor.example.com casdoor
        172.25.11.131 registry.example.com registry
        172.25.11.131 minio.example.com minio
        172.25.11.131 temporal.example.com temporal
        172.25.11.131 starship.example.com starship
        172.25.11.131 starship-api.example.com starship-api
      }
    }
EOF

# Update coredns pods
$ kubectl -n kube-system rollout restart deploy coredns

3. ssh: connect to host csghub.example.com port 22: Connection refused​

This problem is often caused by the failure of gitlab-shell job execution. If this problem occurs, please follow the following methods to troubleshoot:

  1. View
$ kubectl get cm csghub-ingress-nginx-tcp -n csghub -o yaml
apiVersion: v1
data:
  "22": default/csghub-gitlab-shell:22
......

Confirm whether the service name corresponding to port 22 is correct.

  1. If it is incorrect, modify it manually
$ kubectl -n csghub edit configmap/csghub-ingress-nginx-tcp
apiVersion: v1
data:
  "22": csghub/csghub-gitlab-shell:22

# Update ingress-nginx-controller
$ kubectl rollout restart deploy csghub-ingress-nginx-controller -n csghub

4. How to disable Multi-Sync and cleanup relative data​

CSGHub will start with multi-source synchronization enabled by default, but in some cases you may need to disable multi-source synchronization and only host local models. Therefore, you need to disable multi-source synchronization and clear the multi-source synchronization list data that has been synchronized to the local computer. The specific operation is as follows:

  • Docker

    • Set parametes:

      services:
        csghub:
          image: opencsg-registry.cn-beijing.cr.aliyuncs.com/opencsghq/omnibus-csghub:<version>
          environment:
            CSGHUB_OMNIBUS_CONFIG: |
              csghub:
                external_url: "http://csghub.example.com"
              ......
              server:
                multi_sync:
                  enabled: false   # Disable Multi-Sync
              ......

    • Apply changes

      docker compose up -d

    • Cleanup Multi-Sync data

      docker exec -it omnibus-csghub-1 csghub-psql "SELECT cleanup_multi_sync_data();"

  • Helm Chart

    • After v1.10.0

      helm upgrade --install csghub csghub/csghub \
        ......
        --set csghub.server.multiSync.enabled=false \
        ......

    • v1.10.0 and before

      helm upgrade --install csghub csghub/csghub \
        ......
        --set csghub.server.environments.STARHUB_SERVER_MULTI_SYNC_ENABLED=false \
        ......

    • Cleanup Multi-Sync data

      kubectl exec -it pod/csghub-postgresql-0 -n csghub -- su - postgres -lc "psql -d csghub_server -c 'SELECT cleanup_multi_sync_data();'"

5. failed to create pull mirror in mirror server: rpc error: code = PermissionDenied desc = permission denied​

The creation of the mirror repository failed. Please check if the Gitaly token is correct.

6. failed to set rlimit for open files. Try starting as root or requesting smaller maxconns value​

This issue arises because the machine's configuration or settings are insufficient to support too many Loki connections. Try reducing the connection limit setting. Here's how:

--set loki.chunksCache.connectionLimit=1000 --set loki.resultsCache.connectionLimit=1000

7. database "csghub_dataflow" does not exists​

This issue arises because the postgresql.databases list in the csghub application's initial installation does not contain the database csghub_dataflow. Creating it manually will resolve the problem.

kubectl exec -it csghub-postgresql-0 -n csghub -- su - postgres -lc 'psql -U csghub'

CREATE DATABASE csghub_dataflow ENCODING UTF8;

Wait for the Pod to rebuild automatically or manually rebuild quickly.

kubectl rollout restart sts csghub-dataflow -n csghub

8. After upgrading, refreshing the page prompts 'Unauthorized'.​

Please try logout and login again.