Multiple Data Centers: Configuration

Before you start

A Codesphere installation can span several data centers. They share one PostgreSQL server, one platform domain and one OpenFGA instance; each data center runs its own Kubernetes cluster and its own Ceph cluster.

This page covers only what differs from a single-data-center install, see Installation Guide for the base flow. For the procedure, see Multiple data centers: installation.

Every data center has its own config.yaml and its own prod.vault.yaml. Nothing merges them: each oms install codesphere run installs exactly one data center.

                     codesphere.domain  =  cs.example.com
                               |
               +---------------+---------------+
               |                               |
        Data center 1                   Data center 2
        k0s + Ceph cluster A            k0s + Ceph cluster B
        jumpbox 1                       jumpbox 2
        config.yaml + its vault         config.yaml + its vault
               |                               |
               +---------------+---------------+
                               |
              one PostgreSQL server, one OpenFGA

Each data center is normally driven from its own jumpbox, so the file names below are the same on every machine. One jumpbox for the whole installation also works, and then every per-data-center path has to be made distinct, see Lay out the files.

What is shared

Components

• The PostgreSQL server holding the Codesphere database — its roles, its schema and every workspace, team and user row. One data center installs and hosts it; every other one connects to it as an external database.
• The OpenFGA instance holding the authorization tuples, together with the CloudNativePG database it ships with. One data center deploys it and publishes it through its gateway; every other one calls that URL.

Common configuration

Identical in every data center's config.yaml:

• codesphere.domain — one platform name for the whole installation.
• dataCenters and defaultDataCenterId — the topology every data center must know.
• codesphere.plans, deployConfig, features, internal, preview, gitProviders, oauth, managedServices — platform behaviour backed by the shared database.
• registry — every data center pulls the same images from the same registry.

And identical in every data center's prod.vault.yaml: the authentication and encryption keys, the registry credentials, and the roles of the shared database — every secret that is consumed on the far side of something shared. tokenPrivateKey mints the sessions that cross data centers, mounterHmacSecret and mongoDbPasswordEncryptionKey sign and encrypt rows in the shared database, openFgaPresharedKey authenticates against the one OpenFGA, and every postgresUser* / postgresPassword* pair is a role on the one server.

warning

Shared values are easy to change in one config and forget in the other. A diverged plans block is a workspace that can be created in one data center and not in the other, long after the install; a diverged tokenPrivateKey is a user signed in against one data center and rejected by the next.

What each data center owns

Components

Its own k0s cluster, its own Ceph cluster, and its own three externally reachable addresses — platform gateway, workspace gateway and workspace SSH proxy. None of these are shared, and nothing in one data center's config may point at another's hosts.

Common configuration

• dataCenter — this data center's identity.
• secrets.baseDir, its age identity and its vault file.
• codesphere.workspaceHostingBaseDomain, customDomains.cNameBaseDomain and publicIp — the domains and address workspaces of this data center are served from.
• kubernetes.*, ceph.*, cluster.gateway and cluster.publicGateway.
• operations.skip — which installer steps this data center never runs.

Plus the vault entries keyed to this data center's clusters, listed under its own configuration below.

Configure the first data center

The data center that hosts PostgreSQL and deploys OpenFGA. Its config is the normal single-data-center config, plus the topology list and the two hosting roles.

Shared configuration

Write the full topology into dataCenters, and pick where new teams land:

# Byte-identical in config.yaml and config-dc2.yaml
dataCenters:
  - id: 1
    name: karlsruhe
    city: Karlsruhe
    countryCode: DE
  - id: 2
    name: frankfurt
    city: Frankfurt
    countryCode: DE
defaultDataCenterId: 1

The installer rejects a list that contradicts itself: duplicate ids, a dataCenter missing from dataCenters, an entry whose name or city differs from dataCenter, or a defaultDataCenterId that is not in the list.

warning

Omitting dataCenters entirely is the one case that is silent: it defaults to [dataCenter] and defaultDataCenterId to dataCenter.id. The install succeeds, the cluster is healthy, and the platform simply behaves as a single-data-center installation — the picker offers only the data center you are logged in to. Set the list in every config, including this one.

This data center hosts the two shared components. PostgreSQL keeps the normal postgres.mode: install configuration, and OpenFGA is published through the gateway so the other data centers can reach it:

codesphere:
  domain: cs.example.com    # shared by the whole installation
  openFga:
    # deploy defaults to true, apiUrl to the in-cluster service.
    expose:
      enabled: true
      host: openfga.1.cs.example.com

• expose.host must resolve to this data center's platform gateway address, and is what the other data centers put in apiUrl. A host under this data center's own platform domain — openfga.1.cs.example.com — is already covered by the *.1.cs.example.com record from DNS records.
• The gateway certificate is issued by the ClusterIssuer named after codesphere.certIssuer.type. With ACME, the host must be publicly resolvable. With a self-signed issuer, the other data centers do not trust it — see their shared configuration.
• OpenFGA is only used when the openfga-authz preview flag is on (see Feature flags). Without it, leave the block out.

warning

The preshared key is what authenticates OpenFGA once it leaves the cluster. Without openFgaPresharedKey in the vault, OpenFGA is deployed unauthenticated, and the chart then refuses to render expose.enabled: true — the openfga application fails to sync rather than publishing an open authorization store. Add the key before exposing it:

SOPS_AGE_KEY_FILE=/etc/codesphere/secrets/age_key.txt \
  oms update install-config \
    --config /etc/codesphere/config.yaml \
    --vault /etc/codesphere/secrets/prod.vault.yaml

The rest of the shared configuration is whatever a single-data-center install would use. What matters is that the next data center repeats it exactly.

Its own configuration

Its identity, its secrets directory, and the domains and addresses its own workspaces are served from:

dataCenter:
  id: 1
  name: karlsruhe
  city: Karlsruhe
  countryCode: DE
secrets:
  baseDir: /etc/codesphere/secrets
codesphere:
  workspaceHostingBaseDomain: 1.example.com
  customDomains:
    cNameBaseDomain: 1.example.com
  publicIp: 203.0.113.20

dataCenter.id is stored on every workspace row in the shared database and appears in the per-data-center domains. dataCenter.name names the k0s cluster (codesphere-<dataCenter.name>), so it has to be unique too.

The workspace SSH proxy has no domain key of its own. What is configured per data center is the address its ssh-workspace-proxy service gets, under pcApps; the domain is only a DNS record pointing at that address:

pcApps:
  applications:
    ssh-workspace-proxy:
      enabled: true
      valuesObject:
        service:
          enabled: true
          type: LoadBalancer
          loadBalancerIP: 203.0.113.30

kubernetes.*, ceph.*, cluster.gateway and cluster.publicGateway describe this data center's own hosts and addresses, exactly as in a single-data-center install.

Configure every additional data center

Shared configuration

Copy dataCenters, defaultDataCenterId, codesphere.domain, plans, deployConfig, the flag buckets, gitProviders, oauth, managedServices and registry from the first data center's config unchanged.

Point this data center at the shared PostgreSQL server instead of installing one:

postgres:
  mode: external
  # An IP address, not a hostname — see the warning below.
  serverAddress: 10.10.0.10
  port: 5432
  # Copied verbatim from the hosting data center's config.yaml.
  caCertPem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

postgres.primary and postgres.replica are omitted — they describe a server this data center does not install. caCertPem lives in the config, not in the vault, and cannot be re-derived from the CA key, so copy the PEM block across.

warning

postgres.serverAddress must be the server's IP address, not its hostname. The server certificate that oms init install-config generates carries IP SANs only; the hostname appears in the Common Name, which modern TLS clients ignore. A hostname here produces a certificate verification failure in every service of that data center, with nothing pointing at the address as the cause.

The wizard's default for this prompt is postgres.example.com:5432 — replace it with the bare IP, and set the port in postgres.port.

If the server's IP ever changes, the certificate has to be reissued for the new IP and serverAddress updated in every data center that connects to it.

Point it at the OpenFGA instance the first data center deploys, instead of deploying its own:

codesphere:
  openFga:
    deploy: false
    apiUrl: https://openfga.1.cs.example.com
  # Only with a self-signed cluster issuer: trust the CA that signed
  # the OpenFGA gateway certificate.
  extraCaPem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

apiUrl must be the expose.host of the deploying data center. extraCaPem takes that data center's cluster.certificates.ca.certPem, and is only needed when codesphere.certIssuer.type is self-signed — an ACME certificate is trusted already.

Finally, the shared vault entries must be byte-identical to the first data center's. Deriving the vault rather than generating a fresh one is what guarantees that, see Derive the second data center's vault.

Its own configuration

Everything under its own configuration above applies here too, with this data center's own values: a unique dataCenter block, its own workspace domains and publicIp, its own kubernetes.*, ceph.* and gateway addresses, and its own pcApps SSH proxy address.

It also has its own vault, in its own secrets directory, and skips the installer step for the component it does not host:

secrets:
  baseDir: /etc/codesphere/secrets
operations:
  skip:
    - postgres

secrets.baseDir is a path on the machine that runs the installer, so on this data center's own jumpbox it is the same path data center 1 uses on its jumpbox — two directories on two machines, each holding one vault and one age identity. postgres in operations.skip keeps every run of this data center, including manual re-runs, from trying to manage the shared server.

danger

Data centers installed from one machine must not share a secrets.baseDir.

The installer reads the vault from that directory and writes results back into it — the kubernetes step writes kubeConfig, the ceph step writes every Ceph credential. If a shared jumpbox drives both data centers from one directory, installing the second one overwrites the first one's entries and decrypts its vault with the first one's age key. Nothing fails at install time, but the next oms install codesphere or oms add-cluster-admin run for data center 1 reads kubeConfig from the vault and applies data center 1's configuration to data center 2's cluster.

Give each data center its own directory there — /etc/codesphere/secrets and /etc/codesphere/secrets-dc2. The directory name also names the secrets directory staged on that data center's nodes, so distinct names keep the two apart on the nodes as well.

A separate age identity per vault is not strictly required, but it keeps the blast radius of a leaked key to one data center.

These vault entries belong to this data center's clusters and must be regenerated for it; every entry not listed here is shared:

Vault entry	Paired config field
selfSignedCaKeyPem	cluster.certificates.ca.certPem
cephSshPrivateKey	ceph.cephAdmSshKey.publicKey
acmeEabMacKey	codesphere.certIssuer.acme.eabKeyId
kubeConfig	— written by the kubernetes step, or by oms install k0s --vault
Everything starting with ceph, csi or rgw	— written by the ceph step
privNixSigningKey, pubNixSigningKey	— only present in recovered vaults

warning

Always change a config field and its vault secret as a pair. The generators are gated on the vault entry: if selfSignedCaKeyPem is absent, a new ingress CA is generated and cluster.certificates.ca.certPem overwritten. If the vault entry is absent but the config still carries the previous data center's certPem, you get a fresh key next to a stale certificate — ingress then serves certificates signed by a CA nobody trusts, visible only as a TLS error in a browser. The same applies to cephSshPrivateKey / ceph.cephAdmSshKey.publicKey and to acmeEabMacKey / codesphere.certIssuer.acme.eabKeyId.

DNS records

codesphere.domain resolves to one data center's platform gateway. Workspaces and their SSH endpoints resolve per data center, so a workspace's traffic reaches the cluster it runs on.

With base domain example.com, codesphere.domain: cs.example.com and six reserved addresses:

	Data center 1	Data center 2
Platform gateway (gateway-controller)	203.0.113.10	198.51.100.10
Workspace gateway (public-gateway-controller)	203.0.113.20	198.51.100.20
Workspace SSH proxy (ssh-workspace-proxy)	203.0.113.30	198.51.100.30

create:

Record	Type	Target
cs.example.com, *.cs.example.com	A	203.0.113.10
1.cs.example.com, *.1.cs.example.com	A	203.0.113.10
2.cs.example.com, *.2.cs.example.com	A	198.51.100.10
1.example.com, *.1.example.com	A	203.0.113.20
2.example.com, *.2.example.com	A	198.51.100.20
*.1.ssh.example.com	A	203.0.113.30
*.2.ssh.example.com	A	198.51.100.30

warning

The <dc-id>.cs.example.com rows are the ones that get missed. The platform builds each data center's service endpoint as <dataCenter.id>.<codesphere.domain> and the browser calls it directly, before rendering anything. Without an explicit record, 2.cs.example.com falls through to the *.cs.example.com wildcard, resolves to data center 1's gateway, which has no route for it — and the whole UI fails to load, not just the workspaces in data center 2.

Creating 2.cs.example.com also stops *.cs.example.com from covering names beneath it, so *.2.cs.example.com has to be created explicitly as well.

Before you install

Check, for every data center's config:

• dataCenter.id and dataCenter.name are unique across the installation.
• dataCenters is present and identical in every config, and defaultDataCenterId names an id that appears in it.
• The rest of the shared configuration — domain, plans, deploy config, flags, providers, managed services, registry — is identical in every config.
• secrets.baseDir holds this data center's own vault, and no other data center is installed from that same directory.
• Exactly one config has postgres.mode: install; every other has mode: external, serverAddress set to the server's IP, port, a copied caCertPem, no primary/replica, and postgres in operations.skip.
• Exactly one config has codesphere.openFga.expose.enabled: true; every other has deploy: false and an apiUrl pointing at it.
• The workspace domains, publicIp, kubernetes.*, ceph.* and both gateway addresses are this data center's own.
• The shared vault entries are byte-identical; the per-data-center ones are freshly generated, not inherited.

Then validate each data center's pair of files, on the jumpbox that holds them:

oms init install-config --validate -c /etc/codesphere/config.yaml \
  --vault /etc/codesphere/secrets/prod.vault.yaml

The validator checks one config in isolation. It will not catch a duplicate dataCenter.name, two data centers installed from one secrets directory, a missing dataCenters list or a diverged shared secret — those are the checks above.

Next

Multiple data centers: installation.