Skip to main content

Uploading Datasets

To upload datasets, you will need to create an account at OpenCSG Community. Datasets are Git-based repositories, which give you versioning, branches, discoverability and sharing features. You can upload anything you want to the dataset repository.

Currently, we support four ways to upload files: via Git, Web side, command line, and SDK. The following details how to use these four methods.

Upload Files to a Repository Using Git​

  1. First clone the repository to your local machine. Copy the file to the corresponding repository.

  1. Assuming that your files are located in the /work/my_model_dir local directory, you can upload the local files to the platform with the following command:

    cd test_dataset_1
    cp -rf /work/my_dataset_dir/* .
    git add .
    git commit -m "commit message"
    git push

[Note]​

Files with the following suffixes are automatically uploaded with git-lfs:
.7z,.arrow,.bin,.bz2,.ckpt,.ftz,.gz,.h5,.joblib,.lz4,.mlmodel,.model,.msgpack,.npy,.npz,.onnx,.ot,.parquet,.pb,.pickle,.pkl,.pt,.pth,.rar,.safetensors,.tar,.tflite,.tgz,.wasm,.xz,.zip,.zst,.tfevents,.pcm,.sam,.raw,.aac,.flac,.mp3,.ogg,.wav,.bmp,.gif,.png,.tiff,.jpg,.jpeg,.webp

If there are other types of large files, run the following command to make them upload as lfs:

git lfs track <your_file_name>

Upload Files to a Repository Using Web Interface​

To add files to your repository with the web interface, start by selecting the Files tab, and then clicking Add file. You will be given the option to create a new file or upload a file.

Add new file

Creating a New File​

Click Create new file, add the contents and click Create File to save your file. Add new create

Uploading a File​

Click Upload file, you can choose a local file to upload.

Add new upload

Uploading Data via Command Line​

You can conveniently upload data using the command line tool csghub-cli. The installation method is as follows:

pip install csghub-sdk

Here is an example of uploading a local folder to the root path of a repo:

export CSG_TOKEN=your_access_token

# Upload the local folder '/Users/hhwang/temp/jsonl' to the root path of the repo 'wanghh2000/m01' with the default branch
csghub-cli upload wanghh2000/m01 /Users/hhwang/temp/jsonl

Uploading Data Using the SDK​

The CSGHub SDK provides a Python library that allows you to upload data through the SDK in your code.

Here is an example code to upload a repository:

from pycsghub.repository import Repository

token = "your access token"
r = Repository(
    repo_id="wanghh2003/ds15",
    upload_path="/Users/hhwang/temp/bbb/jsonl",
    user_name="wanghh2003",
    token=token,
    repo_type="dataset",
)
r.upload()

The SDK also supports uploading single or multiple files. For detailed examples, please refer to the SDK Documentation.