Object Storage

Object Storage provides an S3-compatible API for files, backups, assets, and other unstructured data. It is intended for workloads that need bucket-based object storage instead of a filesystem or relational database.

Capabilities​

The service is highly available: the underlying Ceph storage is set up redundantly and replicated at the storage layer.

Schema​

Config​

Secrets​

Details / Output​

Plan​

The provider exposes one plan, Generic (id: 0), which allows all quota parameters to be adjusted. Example plan: Generic (id: 0).

Example in a Landscape​

schemaVersion: v0.2

run:
  uploads:
    provider:
      name: s3
      version: v1
    plan:
      id: 0
      parameters:
        maxBuckets: 50
        maxObjects: 100000
        maxSizeKb: 10000000
        maxReadOpsPerS: 1000
        maxWriteOpsPerS: 1000
        maxReadBytesPerS: 100000000
        maxWriteBytesPerS: 100000000
    config:
      accessKey: "${{ workspace.env.S3_ACCESS_KEY }}"
      userDisplayName: "Landscape Upload User"
      initialBucketName: "${{ workspace.env.S3_BUCKET }}"
    secrets:
      secretKey: "${{ vault.s3SecretKey }}"

Within other runtimes, configure your S3 client with the returned url, the configured accessKey, and the stored secretKey. For Codesphere-managed S3, the endpoint is always http://rgw-load-balancer.rook-ceph.svc.cluster.local. The same S3-compatible endpoint is available from other Codesphere runtimes, including reactives, managed containers, and workloads inside a Virtual Cluster.

Backups​

The Object Storage provider supports automated backups and recovery to another S3-compatible backup store. The general concepts are covered in Managed Service Backups.

Under the hood, a backup syncs your buckets to a versioned destination bucket in the backup store, utilizing S3 object versioning to keep historical copies of every object. Restore copies object versions at the time of a given backup into your new service.

Backups are incremental: after the first one, only objects that were added, changed, or removed since the previous backup are synced, so regular backups stay fast and storage-efficient regardless of how much data is already in your buckets. A removed object is not deleted from the backup store's history; it is marked with a delete marker in the destination bucket so that its earlier versions remain available while future backups reflect that it was removed. Because each object is synced independently rather than as a single atomic operation, a backup does not guarantee cross-object consistency — if objects are being written while a backup is running, it may capture some of them before the change and others after.

Enabling Backups​

You can enable backups when creating a new Object Storage service or by updating an existing one, either in the UI or via the API. The following example enables backups on an existing service; the same backups block also works when creating a new service with POST /managed-services.

curl -X PATCH "https://api.codesphere.com/managed-services/YOUR_SERVICE_ID" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "backups": {
      "enabled": true,
      "intervalH": 12,
      "deleteRetentionDays": 30,
      "config": {
        "endpointUrl": "https://s3.eu-central-1.amazonaws.com",
        "path": "my-codesphere-backups/",
        "accessKeyId": "YOUR_S3_ACCESS_KEY"
      },
      "secrets": {
        "secretKey": "YOUR_S3_SECRET_KEY"
      }
    }
  }'

Required S3 Permissions​

The credentials (access key and secret key) used to access your S3-compatible backup store must have sufficient permissions to create the destination bucket, manage its versioning and lifecycle configuration, and read, write, and delete objects in it. For AWS S3, this means the IAM user or role associated with your access key needs at least the following permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketLevelOperations",
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:ListBucket",
        "s3:GetBucketVersioning",
        "s3:PutBucketVersioning",
        "s3:GetLifecycleConfiguration",
        "s3:PutLifecycleConfiguration"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
    },
    {
      "Sid": "ObjectLevelOperations",
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:ListMultipartUploadParts",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
    }
  ]
}

Restoring from a Backup​

Restoring always creates a new Object Storage service; the existing service remains untouched. Specify the backup to restore from with recoverFrom.id:

curl -X POST "https://api.codesphere.com/managed-services" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "teamId": 123,
    "name": "my-object-storage-recovered",
    "provider": {
      "name": "s3",
      "version": "v1"
    },
    "plan": {
      "id": 0
    },
    "config": {
      "accessKey": "YOUR_NEW_ACCESS_KEY",
      "initialBucketName": "my-bucket"
    },
    "secrets": {
      "secretKey": "YOUR_NEW_SECRET_KEY"
    },
    "recoverFrom": {
      "id": "BACKUP_UUID_HERE",
      "config": {
        "endpointUrl": "https://s3.eu-central-1.amazonaws.com",
        "path": "my-codesphere-backups/",
        "accessKeyId": "YOUR_S3_ACCESS_KEY"
      },
      "secrets": {
        "secretKey": "YOUR_S3_SECRET_KEY"
      }
    }
  }'

Backup Disclaimers and Limitations​

• Google Cloud Storage: Backups and restores work with a Google Cloud Storage (GCS) backup store, but automatic retention (deleteRetentionDays) isn't enforced yet against a GCS-backed store, and deleting an individual backup isn't yet supported either. This is a limitation of GCS itself rather than of a specific endpoint: GCS exposes a different, non-S3-compatible lifecycle/versioning API, so it applies regardless of which GCS endpoint the backup store's endpointUrl points at.
• No Compression: Backup data is transferred and stored as-is, without additional compression.

Connecting to an Object Store​

Prerequisites​

Once your Object Store is deployed, you can connect to it from your Codesphere workspaces. Each service lists non-sensitive connection details in their respective settings page, in the overview tab (or in the details property in the public API payload).

Connecting via Terminal (mc)​

The MinIO Client (mc) is a robust tool for interacting with S3-compatible APIs.

# Install mc
nix-env -iA nixpkgs.minio-client

# Configure alias
mc alias set my-storage http://rgw-load-balancer.rook-ceph.svc.cluster.local "$ACCESS_KEY" "$SECRET_KEY"

# List buckets
mc ls my-storage

# Copy file
mc cp myfile.txt my-storage/my-bucket/

Connecting via Node.js​

Using the AWS SDK for JavaScript v3 (@aws-sdk/client-s3):

const { S3 } = require("@aws-sdk/client-s3");

const s3 = new S3({
    endpoint: "http://rgw-load-balancer.rook-ceph.svc.cluster.local",
    region: "us-east-1",
    credentials: {
        accessKeyId: "YOUR_ACCESS_KEY",
        secretAccessKey: "YOUR_SECRET_KEY"
    },
    forcePathStyle: true,
    tls: false
});

const { Buckets } = await s3.listBuckets({});
console.log(Buckets);