Skip to main content
Version: Weekly Build

Multiple Data Centers: Installation

Before you start​

One config and one vault per data center, prepared as described in Multiple data centers: configuration, and the infrastructure of every data center — private network, nodes, disks, reserved addresses, DNS — prepared exactly as for a single-data-center install.

Each data center is installed by its own oms install codesphere run against its own config file. There is no single command that installs the whole installation.

Install order​

Install the data center that hosts PostgreSQL to completion first, and never install two data centers in parallel. Its install creates the database, the roles and the schema every other data center reuses. A data center installed against a database that does not exist yet fails at the first service that connects; two installs at once race each other on the same roles and schema migrations.

"To completion" means the install returned successfully and the checks in Verify the installation pass for that data center.

Lay out the files​

Each data center is usually driven from its own jumpbox, which is also the topology the rest of this page assumes. Every jumpbox then holds one config, one secrets directory, one age identity and one vault — under the same paths, because they are different machines:

jumpbox 1 (data center 1)            jumpbox 2 (data center 2)
/etc/codesphere/config.yaml          /etc/codesphere/config.yaml
/etc/codesphere/secrets/             /etc/codesphere/secrets/
  age_key.txt                          age_key.txt      (its own identity)
  prod.vault.yaml                      prod.vault.yaml  (derived from jumpbox 1's)
install -d -m 0700 /etc/codesphere/secrets   # on each jumpbox

Nothing about the installation requires a jumpbox per data center — what it requires is that no two data centers are installed from one secrets directory. If a single jumpbox drives the whole installation, give each data center its own config file and its own directory there, and use those paths wherever this page says /etc/codesphere/config.yaml or /etc/codesphere/secrets:

/etc/codesphere/config.yaml          /etc/codesphere/config-dc2.yaml
/etc/codesphere/secrets/             /etc/codesphere/secrets-dc2/

Sharing one directory silently destroys the first data center's vault, see its own configuration.

Derive the second data center's vault​

Data center 2's vault must carry data center 1's shared entries byte-identically and its own per-data-center entries. Derive it rather than generating one from scratch: copy, drop what belongs to data center 1's clusters, and let oms regenerate exactly those.

Start on jumpbox 2, with its own age identity. Its public recipient is what jumpbox 1 encrypts the result to, so the derived vault is never readable in transit and the plaintext never leaves jumpbox 1:

# On jumpbox 2.
install -d -m 0700 /etc/codesphere/secrets
age-keygen -o /etc/codesphere/secrets/age_key.txt
chmod 0600 /etc/codesphere/secrets/age_key.txt
age-keygen -y /etc/codesphere/secrets/age_key.txt      # age1... — take this along
# On jumpbox 1.
umask 077

# 1. Data center 1's vault as a starting point.
SOPS_AGE_KEY_FILE=/etc/codesphere/secrets/age_key.txt \
  sops --decrypt /etc/codesphere/secrets/prod.vault.yaml > /tmp/dc2.vault.yaml

# 2. Drop everything that belongs to data center 1's clusters.
yq -i 'del(.secrets[] | select(.name | test("^(ceph|csi|rgw)")))' /tmp/dc2.vault.yaml
yq -i 'del(.secrets[] | select(.name == "selfSignedCaKeyPem"
                            or .name == "kubeConfig"
                            or .name == "acmeEabMacKey"
                            or .name == "privNixSigningKey"
                            or .name == "pubNixSigningKey"))' /tmp/dc2.vault.yaml

# 3. Encrypt to jumpbox 2's recipient and hand it over.
sops --encrypt --age age1... /tmp/dc2.vault.yaml > /tmp/dc2.vault.enc.yaml
shred -u /tmp/dc2.vault.yaml
scp /tmp/dc2.vault.enc.yaml jumpbox2:/etc/codesphere/secrets/prod.vault.yaml
shred -u /tmp/dc2.vault.enc.yaml

On a single shared jumpbox this is one machine and one step less: skip the transfer and encrypt straight into /etc/codesphere/secrets-dc2/prod.vault.yaml, using that directory's own age identity.

cephSshPrivateKey is dropped along with the other ceph* names, but unlike the Ceph cluster credentials it has to be back before the install — the installer needs it to reach data center 2's Ceph hosts. selfSignedCaKeyPem likewise. Regenerating both, together with the config fields paired with them, is what the next step does.

Copy data center 1's config.yaml to jumpbox 2 and edit it into data center 2's config as described in Multiple data centers: configuration, including clearing the three fields whose keys you just dropped:

cluster:
  certificates:
    ca:
      certPem: ''         # regenerated with selfSignedCaKeyPem
ceph:
  cephAdmSshKey:
    publicKey: ''         # regenerated with cephSshPrivateKey
codesphere:
  certIssuer:
    acme:
      eabKeyId: ''        # only if you use ACME with external account binding

Then, on jumpbox 2, let oms fill in what is missing. It generates every absent secret and writes the config field paired with it, so the two halves can never end up mismatched:

SOPS_AGE_KEY_FILE=/etc/codesphere/secrets/age_key.txt \
  oms update install-config \
    --config /etc/codesphere/config.yaml \
    --vault /etc/codesphere/secrets/prod.vault.yaml

It reports what it is about to add and asks before writing; --yes approves up front.

warning

Set SOPS_AGE_KEY_FILE for the vault you are working on, as above. It takes precedence over the identity next to the vault, so on a shared jumpbox a value left over from data center 1 re-encrypts data center 2's vault to data center 1's recipient — and a jumpbox that only has its own key can then no longer read it.

An external account binding cannot be regenerated locally: obtain a new one from your ACME CA and set acmeEabMacKey and codesphere.certIssuer.acme.eabKeyId together.

Finally validate each data center's pair of files, as described in Before you install.

Run the installs​

Data center 1 first, from jumpbox 1, and only then data center 2 from jumpbox 2. The command is the same on both — each jumpbox holds only its own data center's files:

# On jumpbox 1: the data center that hosts PostgreSQL. Must finish and verify first.
# Then the same command on jumpbox 2.
oms install codesphere \
  -c /etc/codesphere/config.yaml \
  -k /etc/codesphere/secrets/age_key.txt \
  --vault /etc/codesphere/secrets/prod.vault.yaml \
  -p codesphere-<version>-installer-lite.tar.gz \
  -s load-container-images

Data center 2 does not need postgres on the command line — operations.skip in its config carries it, which is what keeps a colleague's manual re-run from touching the shared server. Every other step runs in every data center.

warning

From a shared jumpbox, every run points -c, -k and --vault at a different set of files: three flags, three chances to mix them up. Reusing data center 1's --vault for data center 2 corrupts data center 1's vault; reusing its -c installs data center 2's nodes with data center 1's identity. oms install codesphere logs a warning when the config and the vault disagree — do not ignore it:

Warning: config secrets.baseDir (/etc/codesphere/secrets) does not match the directory of --vault (/etc/codesphere/secrets-dc2)

Back up every config, every encrypted vault and every age identity to an approved secret store. An age identity is required to recover or update its data center.

Verify the installation​

Run the normal single-data-center checks (see Run the installer) in each data center first, then the four checks below.

Each cluster is its own​

On a control plane node of each data center:

/etc/codesphere/deps/kubernetes/files/k0s kubectl get nodes -o wide

The node list must match that data center's kubernetes.controlPlanes and kubernetes.workers. If both commands return the same list, the two configs point at one cluster — check kubernetes.apiServerHost and the kubeConfig entry in each vault.

The Ceph clusters must be separate too. Their FSIDs must differ:

grep fsid /etc/ceph/ceph.conf          # on each data center's Ceph master

Two identical FSIDs mean the data centers shared a secrets.baseDir and one install overwrote the other's Ceph credentials.

Both data centers know the full topology​

On a control plane node of each data center:

/etc/codesphere/deps/kubernetes/files/k0s kubectl -n codesphere \
  get configmap -l codesphere.com/purpose=config \
  -o jsonpath='{.items[0].data.dataCenters}'

Each cluster must report the same availableDcs list — every data center of the installation — with currentDc set to its own dataCenter.id. A list containing only the local data center means dataCenters was omitted from that config.

Both data centers write to the same database​

Create one workspace in each data center, then query the shared server. Every workspace row carries the id of the data center it runs in, so one row per data center is only possible if both clusters write to the same database:

psql "host=10.10.0.10 port=5432 dbname=codesphere user=postgres sslmode=verify-full" \
  -c 'select data_center_id, count(*)
        from "workspaceService".workspaces
       group by data_center_id
       order by data_center_id;'
 data_center_id | count
----------------+-------
              1 |     1
              2 |     1

If data center 2's workspaces are missing, its services are talking to a different database — check postgres.serverAddress in its config. (codesphere is the default database name; use postgres.database if the config overrides it.)

A session crosses data centers​

The end-to-end check, and the one that catches a divergent tokenPrivateKey or a missing <dc-id>.<codesphere.domain> DNS record:

  1. Sign in at https://cs.example.com.
  2. Switch to the other data center in the picker. If the picker offers only one, go back to the topology check above.
  3. Create and open a workspace there, without signing in again.
  4. Open a terminal in it and reach it over SSH at its *.<dc-id>.ssh.<base-domain> name.

A redirect back to the sign-in page at step 3 means the token keys differ between data centers. A connection error before any UI renders means <dc-id>.cs.example.com is missing or points at the wrong gateway.

Add a data center to a running installation​

The existing data center keeps running throughout; it needs a configuration update and a re-run of its platform install at the end.

  1. Plan it. Pick an unused dataCenter.id and a dataCenter.name that differs from every existing one. Reserve its three external addresses.

  2. Prepare its infrastructure as for a single-data-center install — including its own jumpbox, and connectivity from its nodes to the existing PostgreSQL server on TCP 5432.

  3. Derive its config and vault from the existing data center's, as above. This is what carries the shared roles and token keys across.

  4. Update DNS. Add the new data center's records — and the existing data center's <dc-id>.<codesphere.domain> records, which a single-data-center install never needed. Verify them from outside the private network.

  5. Install the new data center from its own jumpbox. The existing one already hosts PostgreSQL and is already complete, so the ordering rule is satisfied.

  6. Update the existing data center's config with the dataCenters list containing both data centers and defaultDataCenterId, then re-apply its platform:

    oms install codesphere \
      -c /etc/codesphere/config.yaml \
      -k /etc/codesphere/secrets/age_key.txt \
      --vault /etc/codesphere/secrets/prod.vault.yaml \
      -p codesphere-<version>-installer-lite.tar.gz \
      -s copy-dependencies,extract-dependencies,load-container-images,docker,postgres,ceph,kubernetes

    If the new data center is to run the shared OpenFGA, also set codesphere.openFga.apiUrl in the existing data center's config.

  7. Verify with the checks above. Both clusters must now report both data centers.

warning

Step 6 is easy to defer and easy to forget. Its absence is invisible in every health check: both clusters are green, both serve their own workspaces, and the only symptom is that the existing data center's users never see the new one in the picker.

Test on GCP​

For evaluation, oms beta bootstrap-gcp --multi-dc provisions a complete two-data-center installation in one GCP project, applying everything on this page automatically:

oms beta bootstrap-gcp \
  --project-name multidc-test \
  --billing-account "$BILLING_ACCOUNT" \
  --base-domain oms-testing.example.com \
  --multi-dc \
  --datacenter-name multidc \
  --install-version <version>

The data centers share the project's VPC, PostgreSQL VM and — unlike a production setup — a single jumpbox; each gets its own three Ceph nodes, three k0s nodes and three static addresses. Data center 2's resources therefore carry a -dc2 suffix, including /etc/codesphere/config-dc2.yaml and /etc/codesphere/secrets-dc2/ on that jumpbox, which is the shared-jumpbox layout described above.

Two data centers mean 14 VMs (~100 vCPUs) and 6 regional static addresses, so the region's quotas usually need raising first. --multi-dc cannot be combined with --datacenter-id — the ids are derived (1 and 2) because they drive the per-data-center domains. The bootstrap serves workspaces at <dc-id>.ws.<base-domain> and SSH at *.<dc-id>.ssh.cs.<base-domain>, rather than the scheme used in the examples on this page; only the DNS records and workspaceHostingBaseDomain have to agree with each other.

oms beta bootstrap-gcp is not for production use. See the bootstrap-gcp documentation for the full flag list.