Skip to main content

Uploading Datasets

To upload datasets, you will need to create an account at CSGHub. Datasets are Git-based repositories, which give you versioning, branches, discoverability and sharing features. You can upload anything you want to the dataset repository.

Currently, we support four ways to upload files: via Web interface, Git, Command Line (CLI), and SDK.

💡 Which method should I choose?​

  • Web interface: Suitable for small files; check the current upload page and deployment configuration for the per-file limit.
  • Git: Suitable for version control and managing a large number of scattered code and configuration files.
  • Git/CLI: Use Git LFS for large Git files, or the repository-type-specific CLI instructions where provided. Limits depend on the site and storage configuration.

Upload Files to a Repository Using Git​

Prerequisites​

  1. Install Git and Git LFS, then run git lfs install.
  2. Configure the identity used for local commits:
git config --global user.name "Your name"
git config --global user.email "you@example.com"
  1. Obtain an Access Token from Account settings → Access Token on the target deployment. Use it as the password when HTTPS Git requests credentials. Organization repositories also require write permission.

Upload Steps​

Copy the HTTPS clone URL from the current repository. Replace the sample namespace and repository; use your deployment's actual address.

git lfs install
git clone https://your-csghub.example.com/datasets/your-namespace/example-dataset.git
cd example-dataset

Copy the intended files into the local repository. Check that no credentials or unintended files are included, then commit and push:

git add .
git commit -m "Upload repository files"
git push

Verify the files and commit in the target repository's Files tab.

[Note]​

Use Git LFS for large files. Before git add, check the repository's .gitattributes. Add a tracking rule if required:

git lfs track "path/to/large-file"
git add .gitattributes

File size and timeout limits depend on site, proxy and storage configuration. Preserve the file size and error, and use a supported workflow described on this page. Site limits are not a universal Git LFS protocol limit.

Managing and Uploading Files via Web Interface​

In the repository's Files tab, select Add file, then choose New folder, Create new file, or Upload file.

The per-file limit is defined by the current upload page. For larger files, use a transfer method documented as supported on this page.

Add file

Creating a New Directory​

Select New folder, enter a name and submit to create a subfolder under the current repository path.

New folder

Creating a New File​

Select Create new file, enter the file name and content, then select Create File to save and commit it.

Create new file

Uploading a File​

Select Upload file, choose the local file and submit. Confirm that it appears in the repository file list after the upload completes.

Upload file

Upload with the CLI​

Create the target repository first and confirm write access. Install a compatible csghub-sdk and inspect csghub-cli upload-large-folder --help. Configure this site's CSGHUB_TOKEN in the runtime. Replace the endpoint, repository ID and local directory; the platform API URL is not the inference gateway URL.

export CSGHUB_ENDPOINT="https://your-csghub.example.com"
csghub-cli upload-large-folder your-namespace/example-dataset ./my_dataset \
  --repo-type dataset --endpoint "$CSGHUB_ENDPOINT"

Specify model or dataset explicitly. Preserve upload progress files until completion. If your client lacks these options, check supported versions instead of omitting endpoint or type.

Upload with the SDK​

Repository uses Git and Git LFS. Reuse CSGHUB_ENDPOINT and CSGHUB_TOKEN, and set CSGHUB_USERNAME to your login account (which may differ from an organization namespace).

import os
from tempfile import TemporaryDirectory
from pycsghub.repository import Repository

with TemporaryDirectory(prefix="csghub-upload-") as work_dir:
    repo = Repository(
        repo_id="your-namespace/example-dataset",
        upload_path="./my_dataset",
        user_name=os.environ["CSGHUB_USERNAME"],
        token=os.environ["CSGHUB_TOKEN"],
        endpoint=os.environ["CSGHUB_ENDPOINT"],
        repo_type="dataset",
        work_dir=work_dir,
        auto_create=False,
    )
    repo.upload()

Verify​

Check the files, owner and commit in the target repository, then download a copy and compare content. A completed progress indicator alone is not sufficient verification.