One-click deployment (Bundle)
The shortest path to a first install: install the operator, apply one Bundle, and confirm against the acceptance checklist. If something goes wrong, switch to manual per-service deployment.
Base layer installation
This is the only step in the entire guide that you must do by hand. The reason is simple: a Bundle is a custom resource, and neither the operator that installs it nor the CRDs that define it can be installed by the Bundle itself — the chicken has to come before the egg, so this layer can only be applied on its own.
The base layer should be as small, static, and reproducible as possible. The smaller it is, the lower the chance that it fails to install in your environment.
1. Namespace and ServiceAccount
The operator runs in fewbox-system and reaches the API Server as the fewbox ServiceAccount
(in-cluster, through a projected token; no endpoint or token to configure):
kubectl create namespace fewbox-system
kubectl create serviceaccount fewbox -n fewbox-system
2. RBAC
kubectl apply -f deploy/operator-role.yaml # ClusterRole
kubectl apply -f deploy/operator-role-binding.yaml # ClusterRoleBinding -> fewbox-system/fewbox
That ClusterRole covers workloads, CRDs, Istio and RBAC itself — its
customresourcedefinitions write permission is what makes the next step's CRD self-install possible.
It does not cover nodes or metrics: the console's node page and utilisation charts read nodes and
metrics.k8s.io (NodeMetricses goes through metrics-server). Add a ClusterRole for those if you use them.
3. The PackingTable operator
The operator is three manifests, reproduced in full below — the first is the permissions, the second binds
them to the fewbox ServiceAccount, the third is the Deployment. Replace <your-registry> and <tag> with
your own image first, then:
kubectl apply -f operator-role.yaml
kubectl apply -f operator-role-binding.yaml
kubectl apply -f operator-deployment.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: packingtable-operator
rules:
# Operator framework needs (leader election & events)
- apiGroups:
- ""
resources:
- events
verbs:
- get
- list
- create
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
# FewBox CRDs
- apiGroups:
- fewbox.com
resources:
- tenants
- tenants/status
- registries
- infrastructures
- infrastructures/status
- middlewares
- middlewares/status
- bundles
- bundles/status
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
# Operator manages K8S core resources
- apiGroups:
- ""
resources:
- namespaces
- services
- configmaps
- secrets
- pods
- pods/log
- serviceaccounts
- persistentvolumes
- persistentvolumeclaims
- replicationcontrollers
- resourcequotas
- endpoints
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- apps
resources:
- deployments
- replicasets
- statefulsets
- daemonsets
- controllerrevisions
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- batch
resources:
- jobs
- cronjobs
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- rbac.authorization.k8s.io
resources:
- clusterroles
- clusterrolebindings
- roles
- rolebindings
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- networking.k8s.io
resources:
- ingresses
- networkpolicies
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- networking.istio.io
resources:
- virtualservices
- destinationrules
- serviceentries
- gateways
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
# RabbitMQ / MySQL operators
- apiGroups:
- rabbitmq.com
resources:
- rabbitmqclusters
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- mysql.oracle.com
resources:
- innodbclusters
- mysqlbackups
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- zalando.org
resources:
- kopfpeerings
- clusterkopfpeerings
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: packingtable-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: packingtable-operator
subjects:
- kind: ServiceAccount
name: fewbox
namespace: fewbox-system
apiVersion: apps/v1
kind: Deployment
metadata:
name: packingtable
namespace: fewbox-system
labels:
app: packingtable
spec:
replicas: 1
revisionHistoryLimit: 3
selector:
matchLabels:
app: packingtable
template:
metadata:
labels:
app: packingtable
spec:
serviceAccountName: fewbox
containers:
- name: packingtable
image: <your-registry>/packingtable:<tag>
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# KubeOps connects with in-cluster config (same ServiceAccount); no extra endpoint/token needed.
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
terminationGracePeriodSeconds: 10
Three things worth noticing:
- The first one is the complete permission list: workloads, CRDs, Istio, RBAC and
apiextensions.k8s.io. The operator is inherently a privileged component and belongs only in a controlled cluster. It does not includenodesormetrics.k8s.io— the console's node page and utilisation charts need those, so add a ClusterRole for them if you use those pages. - The Deployment runs one replica, and its only environment variable is
POD_NAMESPACE(from the downward API — its own namespace); Kubernetes access uses the in-cluster config, so there is no endpoint or token to fill in. - Its objects are named
packingtable-operator/packingtable; they are unrelated to thefewboxServiceAccount inside a tenant.
You do not apply the CRDs by hand. At startup the operator transpiles its entity types into the
fewbox.com/v1 CRDs and writes them into the cluster (the code is the single source of truth, so the schema
and the program never drift apart). Verify once it is up:
kubectl get crd | grep fewbox.com
# tenants / infrastructures / middlewares / bundles / families / registries
kubectl get pods -n fewbox-system -l app=packingtable
Until the operator is up, Bundle objects just sit there and nothing gets created — the symptom is
"apply succeeded but there is nothing in the cluster". This is the most common first blocker. A
Failed to ensure FewBox CRDs line in the startup log means the CRD write permission is missing; fix the
RBAC first. Do not overwrite the CRDs with an older generated copy — that rolls the schema back and
fields written afterwards are silently dropped.
4. Tenant prerequisites (before you apply the Bundle)
A Bundle is a namespaced resource, so four things must already exist in the target tenant:
| Object | Why |
|---|---|
Namespace <tenant> |
It is metadata.namespace; apply fails outright if it does not exist |
ServiceAccount <tenant> |
Ordinary packs run as it (serviceAccountName defaults to the tenant name) |
ServiceAccount fewbox + ClusterRoleBinding |
Used by packs with isRunAsClusterAdmin: true (such as fewbox-tethys-shipping); without the binding the Pod starts fine but every Kubernetes API call returns 403 |
Istio Gateway internet |
isPublic: true services bind their VirtualService to it; without it nothing fails, the hosts just return 404 |
The Tenant CR is not how you create a tenant. It is a record only (the console's tenant list, base
domains, registry source) and has no controller — applying a Tenant by hand does nothing. The tenant's
namespace, ServiceAccount and Gateway are created by tethys-shipping through
POST /api/v1/tenants (the console's "add tenant" action).
So a fresh cluster has two ways to start; pick one:
Create it by hand first (no tethys needed):
kubectl create namespace <tenant> kubectl label namespace <tenant> istio-injection=enabled kubectl create serviceaccount <tenant> -n <tenant> kubectl create serviceaccount fewbox -n <tenant> kubectl -n <tenant> create rolebinding fewbox-admin \ --clusterrole=packingtable-operator --serviceaccount=<tenant>:fewboxThen create the entry
Gateway(the shape is in Install Istio; theselectormust match your cluster's ingressgateway labels):# tethys-shipping reads and writes this Gateway as v1beta1; on Istio 1.22+ the v1 form is the same object apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: name: internet namespace: <tenant> spec: selector: istio: ingressgateway servers: - hosts: ["<your-domain>", "*.<your-domain>"] port: number: 80 name: http protocol: HTTPOnce tethys is up, complete the tenant from the console (domains, quotas, registry):
Registeronly skips namespaces, ServiceAccounts and Gateways that already exist, it never recreates them, so creating them by hand and creating the tenant later compose.Bring tethys up first, use "add tenant" to create all four objects in one go, then import the Bundle in the UI. This suits environments that already have a working tethys instance bootstrapped from outside the cluster.
tethys-gateway, tethys-shipping and tethys-web are themselves three of the packs in the Bundle, so
path A gives you the console; you do not need a separate manifest for it.
5. Scheduled jobs (not in the Bundle)
The thia-jobs-tick CronJob is not part of a Bundle export (the exported spec has no
scheduledTaskPacks). It is a separate manifest, k8s/thia-jobs-tick.yaml in the agent repository,
containing one ConfigMap and one CronJob:
kubectl apply -f thia-jobs-tick.yaml -n <tenant>
Without it the system still installs and every page still opens, but heartbeat checks, memory maintenance and credit settlement never run — and nothing reports an error. See Scheduled jobs (required).
Bootstrap credentials and API Server exposure
tethys is itself a workload in the cluster, and on first install there is no management entry point available
for it yet — this chicken-and-egg situation is unavoidable. There are two bootstrap methods; pick either one:
| Bootstrap method | How to install | When to use |
|---|---|---|
| In-cluster apply | kubectl apply the RBAC, the operator and the tenant prerequisites, then the Bundle (Path A, B) |
Routine installs with cluster access |
| Out-of-cluster bootstrap | Run tethys on a machine outside the cluster, connect to the API Server, and install from the console | You already have a working tethys instance, or you want to avoid touching YAML entirely; see "Out-of-cluster bootstrap: a local tethys" below |
Both methods have the same reachability requirement for the API Server, but external bootstrap adds two places you must tighten.
① API Server exposure scope
External bootstrap requires the API Server to be reachable from the deployment machine. Do not open 6443 to
0.0.0.0/0 for this:
- Allow only the deployment machine's egress IP (or the jump host, or the VPN subnet) in the security group / firewall;
- Close public access once bootstrap is done; day-to-day upgrades go through the in-cluster console;
tethys-shippingin the cluster must reach the API Server through an in-cluster address, i.e.DevOps__Endpoints__K8SEndpoint__Host=kubernetes.default.svc,Port=443. Otherwise the runtime will keep depending on that public hole, and you will not be able to close it.
② Lifetime of bootstrap credentials
kubectl create token issues a ServiceAccount JWT, not a Secret: until it expires it is a long-lived bearer
credential and cannot be revoked on its own — revocation only works by deleting / modifying that ServiceAccount,
or by rotating the API Server signing key (the latter kicks out every SA token at once).
The SA the operator uses is also a privileged component (it creates Deployments, Services, Ingresses, ClusterRoles, and even CRDs), so this credential is equivalent to control of the cluster. Therefore:
- Mint on demand and use short-term, for example
--duration 1h; - Do not issue tokens measured in years, and do not write tokens into config files, databases, or deliverables;
- For any credential you have already persisted, clean it up after bootstrap and generate a new one.
The in-cluster apply path also needs a kubeconfig that can reach the API Server. The difference is only the credential form, not whether risk exists — the client certificate embedded in a kubeconfig should likewise be kept short-term and taken back after use.
Out-of-cluster bootstrap: a local tethys
If you would rather not write YAML against the cluster, invert the order: run tethys-gateway,
tethys-shipping and tethys-web on a machine that can reach the API Server (your laptop or a jump host),
connect with a credential minted by kubectl create token, click through the base layer and the operator in the
UI, and import the Bundle last.
The order cannot be reversed: until PackingTable is installed, importing a Bundle fails — nothing is there to reconcile it. So:
Start the three services locally (the same images as the three packs in the Bundle), changing only the connection settings: point
tethys-shipping'sDevOps__Endpoints__K8SEndpoint__Host/Portat the API Server and put the token inAppSettingsToken; pointtethys-gateway'sSHIPPING_ENDPOINTat the local shipping. While the identity service is not in the cluster yet,PERMISSIONS_OPEN=trueskips both the JWT public-key startup check and ADMIN enforcement (the log prints a[SECURITY]warning).Namespace and ServiceAccount: create
fewbox-systemin the console, then thefewboxServiceAccount in it.ClusterRole / ClusterRoleBinding: exactly the manifests in section 3 — do not trim a single permission; a trimmed role still lets the operator start and only returns 403 at reconcile time.
The PackingTable Deployment:
serviceAccountName: fewbox,POD_NAMESPACEfrom the downward API.Tenant → Bundle → scheduled job: "add tenant" creates the namespace, the ServiceAccount and the
internetGateway; import the Bundle; then createthia-jobs-tickseparately.Retire the bootstrap: the cluster now runs the three
tethys-*packs itself — switch the console to that copy. The local shipping SQLite is only the bootstrap record (release history, tenant rows), not a source of truth. Before you walk away, do two things: export what the UI created back into manifests (objects clicked into existence have no declaration anywhere — without an export nobody knows which image or env that copy runs) —kubectl get deploy,clusterrole,clusterrolebinding -n fewbox-system -o yaml > operator-as-installed.yamlkeep it with the delivery records for the next upgrade or incident; then stop the three local services and revoke the bootstrap token — do not leave a management plane running with
PERMISSIONS_OPEN=truepointed at a customer's cluster.
What this path costs, and where it stops:
| Item | Note |
|---|---|
| The local tethys needs its own user table | SUPABASE (Tethys predates the unified identity, so its user table is separate). Path A needs it too — those three packs carry the same requirement |
| The bootstrap credential equals control of the cluster | Mint short-term, never persist, revoke after use; see ② above |
| Objects created in the UI drift from the manifests | There is no diff to read on clicked objects. Export once when you retire the bootstrap (step 6), and realign with section 3 when you upgrade the operator or change its permissions |
| The long-term entry point is the in-cluster tethys | The local one only has to install the first version |
Path A: one-click Bundle deployment
Use this path for both first install and day-to-day upgrades. The Bundle you receive is exported by us from a reference environment and delivered as an artifact; you do not need to assemble it yourself.
For field semantics and reconcile / deletion behavior, see Bundle format.
Where the Bundle comes from
It is exported by us from a reference environment and delivered as an artifact. You do not need to export it yourself.
To see what the file looks like and which fields to change, go straight to Sample manifests: a complete export from a reference environment (21 applications plus 17 Secrets), sanitized and ready to fill in.
Before you import
Open the YAML, deal with the following items first, then apply:
① Image registry prefix
Every pack has a registry field, defaulting to our registry address:
spec:
packs:
- registry: <YOUR_REGISTRY>/<PROJECT>
app: thia-web
version: v1.2.3
Replacing it globally is the safest; editing entry by entry is easy to miss.
② Namespace
metadata.namespace is the tenant name. A private deployment usually uses just one:
metadata:
name: agentic-hat
namespace: <YOUR_NAMESPACE>
③ Image pull credentials
imagePullSecret holds only the Secret name, not its contents. Create it in the target namespace first:
kubectl create secret docker-registry <secret-name> \
--docker-server=<YOUR_REGISTRY> \
--docker-username=<YOUR_USERNAME> \
--docker-password=<YOUR_PASSWORD> \
-n <YOUR_NAMESPACE>
④ Business Secrets that are referenced
runtimeVariables[].source: Secret and settings[].source: Secret in the Bundle only keep the
secretName / secretKey references, and contain no secret material. These Secrets must exist in the target
namespace before you apply, and their names must match the Bundle exactly.
# List every Secret referenced by the Bundle
grep -E '^\s*(- )?secretName:' bundle.yaml | awk '{print $NF}' | sort -u
Using Themis as an example (for what each key is used for, see Path B / Themis: identity):
# ES256 key pair: auth signs (private + public key), gateway verifies (public key)
kubectl create secret generic <AUTH_SECRET_NAME_FROM_BUNDLE> -n <YOUR_NAMESPACE> \
--from-file=JWT_PRIVATE_KEY=jwt-private.pem \
--from-file=JWT_PUBLIC_KEY=jwt-public.pem
kubectl create secret generic <GATEWAY_PUBLIC_KEY_SECRET_NAME_FROM_BUNDLE> -n <YOUR_NAMESPACE> \
--from-file=JWT_PUBLIC_KEY=jwt-public.pem
# Session signing key: required, no default; an empty value is rejected by the startup self-check
kubectl create secret generic <SESSION_SECRET_NAME_FROM_BUNDLE> -n <YOUR_NAMESPACE> \
--from-literal=IDP_SESSION_SECRET="$(openssl rand -base64 48)"
If any one is missing, the corresponding Pod stops at CreateContainerConfigError: secret "..." not found;
after you create the missing Secret there is no need to apply again — kubelet retries the mount automatically.
⚠️ The Bundle file itself is a sensitive file. The values in
spec.packs[].runtimeVariablesare plaintext, and config file contents (spec.customPacks[].settings[].settingsTemplate) are base64, which is no protection at all. Do not commit a Bundle filled with real values to git, and do not send it by email.
Apply
kubectl apply -f bundle.yaml
Watch the reconcile
kubectl get bundles -n <YOUR_NAMESPACE>
kubectl get deploy,svc -n <YOUR_NAMESPACE>
The operator creates Deployments first, then Services. Only after everything is READY can you move on to the
first-start acceptance check section.
If the reconcile is stuck, check in this order:
- Is the operator Pod there, and does its log show errors?
- Does the image pull succeed (check
ImagePullBackOffwithkubectl describe pod)? - Is the resource quota sufficient (a failed
ReplicaSetcreation leaves events)?
First-start acceptance check
It counts as installed only when all six items pass. The order runs from infrastructure to business — if an earlier item fails, you cannot check the later one.
| # | Check | How to verify |
|---|---|---|
| 1 | All Pods READY |
kubectl get pods -n <ns> |
| 2 | Health checks pass | Each service's /livez, /readyz (themis-auth, rhea-email, rhea-push expose them under /api/v1; thia-web has no health route, see Service and ports) |
| 3 | You can register and log in | Open accounts.titanides.com and complete register → receive mail → verify → log in |
| 4 | You can send one conversation | Log in to thia-web, create a conversation, send a message, and see a reply |
| 5 | The credits ledger is correct | The balance decreases with the conversation, and you can see the matching ledger entry in the database |
| 6 | Scheduled jobs are running | kubectl get jobs -n <ns> shows Jobs generated by thia-jobs-tick (names like thia-jobs-tick-xxxxx), and the latest one is within 5 minutes |
Acceptance item 3 is the most likely to go wrong — it exercises both the mail service and the identity service.
If the verification email does not arrive, check rhea-email's SMTP configuration first, then check whether
themis-auth can reach its Service.