To demonstrate this, I’m going to intentionally break a Kubernetes service in three different ways, and examine the raw packet captures from each.
For the set up, I created a simple deployment which runs a single instance of socat, and I put a NodePort Kubernetes service in front of it. This is all running on a single node instance of k3d, which means the service is ultimately exposed at <k3d node>:<node port>:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
app: socat
name: socat
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: socat
template:
metadata:
labels:
app: socat
spec:
containers:
- args:
- TCP-LISTEN:12345,fork,reuseaddr
- EXEC:/bin/cat
command:
- socat
image: alpine/socat:latest
imagePullPolicy: Always
name: socat
ports:
- containerPort: 12345
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app: socat
name: socat
namespace: default
spec:
ports:
- nodePort: 31234
port: 12345
protocol: TCP
type: NodePort
The socat process in the pod listens on port 12345, and we want packets to the k3d node at port 31234 to be routed there.
As a quick demonstration of what this socat command does, it just echoes back whatever is sent to it on a TCP connection. I can telnet into the node port and get whatever I send echo’d back. Note the 127.0.0.1 is because I have the k3d docker container port forwarded to localhost on my MacBook host network:
tyler@MacBookPro sysobs % telnet 127.0.0.1 31234
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
hello
world
world
Under the hood, Kubernetes services are implemented by EndpointSlice resources, which are translated by kube-proxy into iptables/netfilter rules that the Linux Kernel can use directly. Essentially, these rules match packets for specific IP address / port tuples and then use network address translation to rewrite the destination address on the packet.
The EndpointSlice typically gets updated dynamically by a controller based on the selector field on the service, so you usually don’t have to touch them. For the purposes of this post, I removed the selector field from the service and created the EndpointSlice manually.
I’ll set the correct pod IP and correct port in the EndpointSlice spec. From kubectl get pod we can see the pod’s IP address:
socat-5c97445bc5-vw5dz 1/1 Running 0 23h 10.42.0.142 k3d-k3s-default-server-0 <none> <none>
Which just needs to be set on the EndpointSlice, along with the correct port:
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
- 10.42.0.142
conditions:
ready: true
kind: EndpointSlice
metadata:
labels:
app: socat
kubernetes.io/service-name: socat
name: socat
namespace: default
ports:
- name: ""
port: 12345
protocol: TCP
This creates the following chain of IPTables rules:
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/socat" -m tcp --dport 31234 -j KUBE-EXT-6GHM6NOKDRAB634Q
...
-A KUBE-SEP-STAYFALK6DPGANUO -p tcp -m comment --comment "default/socat" -m tcp -j DNAT --to-destination 10.42.0.142:12345
I removed some of the intermediate rules to simplify, but basically – this matches packets coming in to port 31234 (the nodePort on our service) and changes the destination address on them to 10.42.0.142:12345.
And if I attempt a TCP connection against the NodePort service, this is packet capture:
+0.401s [pktcap] capture-started default/socat bpf="(ip and ((host 10.43.161.224 and port 12345) or ((net 10.42.0.0/24) and port 12345) or ((host 192.168.97.3) and port 31234))) or (arp and (host 10.42.0.142))" link=Linux SLL2
+3.519s [pktcap] tcp-packet [eth0] 192.168.97.2:54398 -> 192.168.97.3:31234 SYN
+3.519s [pktcap] tcp-packet [cni0] 10.42.0.1:10639 -> 10.42.0.142:12345 SYN
+3.519s [pktcap] tcp-packet [veth0829e839] 10.42.0.1:10639 -> 10.42.0.142:12345 SYN
+3.519s [pktcap] tcp-packet [veth0829e839] 10.42.0.142:12345 -> 10.42.0.1:10639 SYN,ACK
+3.519s [pktcap] tcp-packet [cni0] 10.42.0.142:12345 -> 192.168.97.2:54398 SYN,ACK
+3.519s [pktcap] tcp-packet [eth0] 192.168.97.3:31234 -> 192.168.97.2:54398 SYN,ACK
+3.519s [pktcap] tcp-packet [eth0] 192.168.97.2:54398 -> 192.168.97.3:31234 ACK
+3.519s [pktcap] tcp-packet [cni0] 10.42.0.1:10639 -> 10.42.0.142:12345 ACK
+3.519s [pktcap] tcp-packet [veth0829e839] 10.42.0.1:10639 -> 10.42.0.142:12345 ACK
This is just a standard TCP handshake, but we see three different network interfaces being traversed for each packet. The first is eth0, which is the “external” interface of the k3d node. The packet is addressed to 192.168.97.3:31234, which is the k3d node IP and node port.
Next we see the same SYN packet, this time on the cni0 interface, the virtual bridge connecting every pod on the node, and the destination address has been changed to 10.42.0.142:12345, which is the pod’s IP and listening port. This is the direct effect of the DNAT iptables rule we looked at above, and the packet was routed to cni0 because it now belonged to the 10.42.0.0/24 network.
Notice also that the source address changed, which is another effect of the iptables rules that kube-proxy installs, which is then reversed on the return path.
Finally, each pod has its own veth pair which bridges the pod’s network namespace to the host’s network namespace. The host end of the pair is connected to the cni0 virtual bridge, so the packet is re-transmitted to this interface.
When the server sends packets back to the client, the three interfaces are traversed in reverse order (veth -> cni0 -> eth0), as expected.
A simple way to break the service is to point at a port nothing in the pod is listening on. Since we can modify the EndpointSlice however we want, we can just change the port on it:
ports:
- name: ""
port: 12346 # changed from 12345
protocol: TCP
This changes the iptables rule to look like:
-A KUBE-SEP-XVRNL33HWWUQHJC7 -p tcp -m comment --comment "default/socat" -m tcp -j DNAT --to-destination 10.42.0.142:12346
Nothing is listening on this port, so let’s see the packet capture:
+0.626s [pktcap] capture-started default/socat bpf="(ip and ((host 10.43.161.224 and port 12345) or ((net 10.42.0.0/24) and port 12346) or ((host 192.168.97.3) and port 31234))) or (arp and (host 10.42.0.142))" link=Linux SLL2
+50.032s [pktcap] tcp-packet [eth0] 192.168.97.2:58364 -> 192.168.97.3:31234 SYN
+50.032s [pktcap] tcp-packet [cni0] 10.42.0.1:6999 -> 10.42.0.142:12346 SYN
+50.032s [pktcap] tcp-packet [veth0829e839] 10.42.0.1:6999 -> 10.42.0.142:12346 SYN
+50.032s [pktcap] tcp-packet [veth0829e839] 10.42.0.142:12346 -> 10.42.0.1:6999 ACK,RST
+50.032s [pktcap] tcp-packet [cni0] 10.42.0.142:12346 -> 192.168.97.2:58364 ACK,RST
+50.032s [pktcap] tcp-packet [eth0] 192.168.97.3:31234 -> 192.168.97.2:58364 ACK,RST
Since the IP address on the EndpointSlice is still correct, the packet still gets routed to the correct pod veth pair, but since nothing inside the pod’s network namespace is listening on port 12346, a TCP RST packet is sent back through the interfaces to the client.
This typically manifests on the client as a “connection refused” error.
Again, we can just update the IP address on the EndpointSlice directly:
endpoints:
- addresses:
- 10.42.0.143
conditions:
ready: true
Which changes the iptables rule to look like:
-A KUBE-SEP-L7Q23S57UQHHMQS4 -p tcp -m comment --comment "default/socat" -m tcp -j DNAT --to-destination 10.42.0.143:12345
Which results in the following packet capture:
+0.564s [pktcap] capture-started default/socat bpf="(ip and ((host 10.43.161.224 and port 12345) or ((net 10.42.0.0/24) and port 12345) or ((host 192.168.97.3) and port 31234))) or (arp and (host 10.42.0.143))" link=Linux SLL2
+7.389s [pktcap] tcp-packet [eth0] 192.168.97.2:47514 -> 192.168.97.3:31234 SYN
+7.389s [pktcap] arp-request [cni0] arp who-has 10.42.0.143 tell 10.42.0.1
+7.389s [pktcap] arp-request [veth0829e839] arp who-has 10.42.0.143 tell 10.42.0.1
+7.389s [pktcap] arp-request [veth261c8b4a] arp who-has 10.42.0.143 tell 10.42.0.1
+7.389s [pktcap] arp-request [vethb5026255] arp who-has 10.42.0.143 tell 10.42.0.1
+7.389s [pktcap] arp-request [vethf88de69a] arp who-has 10.42.0.143 tell 10.42.0.1
+7.389s [pktcap] arp-request [vethfd2cfe3e] arp who-has 10.42.0.143 tell 10.42.0.1
+7.389s [pktcap] arp-request [veth03bb59f4] arp who-has 10.42.0.143 tell 10.42.0.1
+7.389s [pktcap] arp-request [veth71d61d24] arp who-has 10.42.0.143 tell 10.42.0.1
+7.389s [pktcap] arp-request [veth683aa91d] arp who-has 10.42.0.143 tell 10.42.0.1
After the packet’s destination address changes, the Kernel attempts to discover the corresponding MAC address for this IP, and broadcasts an ARP who-has request on the pod bridge network. The veth pair for each pod on the node receives a copy of the ARP request, and since the IP address isn’t attached to any of the pods, nobody is able to reply, and the TCP connection is unable to complete.
From the client you’re not going to see an “ARP failed” error message, you’ll probably just see it hanging while it retries the TCP handshake until it gives up with a timeout.
An EndpointSlice can contain multiple pod IP addresses, so let’s put the correct address alongside the wrong one:
endpoints:
- addresses:
- 10.42.0.143
conditions:
ready: true
- addresses:
- 10.42.0.142
conditions:
ready: true
This creates iptables rules with a probabilistic load balancing effect:
-A KUBE-SVC-6GHM6NOKDRAB634Q -m comment --comment "default/socat -> 10.42.0.142:12345" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-STAYFALK6DPGANUO
-A KUBE-SVC-6GHM6NOKDRAB634Q -m comment --comment "default/socat -> 10.42.0.143:12345" -j KUBE-SEP-L7Q23S57UQHHMQS4
And now the packet capture shows only intermittent failures:
+20.003s [tcp-probe] io-error 127.0.0.1:57014 -> 127.0.0.1:31234 read: read tcp 127.0.0.1:57014->127.0.0.1:31234: i/o timeout
+20.064s [pktcap] arp-request [cni0] arp who-has 10.42.0.143 tell 10.42.0.1
+20.064s [pktcap] arp-request [veth0829e839] arp who-has 10.42.0.143 tell 10.42.0.1
+20.064s [pktcap] arp-request [veth261c8b4a] arp who-has 10.42.0.143 tell 10.42.0.1
+20.064s [pktcap] arp-request [vethb5026255] arp who-has 10.42.0.143 tell 10.42.0.1
+20.064s [pktcap] arp-request [vethf88de69a] arp who-has 10.42.0.143 tell 10.42.0.1
+20.064s [pktcap] arp-request [vethfd2cfe3e] arp who-has 10.42.0.143 tell 10.42.0.1
+20.064s [pktcap] arp-request [veth03bb59f4] arp who-has 10.42.0.143 tell 10.42.0.1
+20.064s [pktcap] arp-request [veth71d61d24] arp who-has 10.42.0.143 tell 10.42.0.1
+20.064s [pktcap] arp-request [veth683aa91d] arp who-has 10.42.0.143 tell 10.42.0.1
+21.007s [tcp-probe] echo-ok 127.0.0.1:57015 -> 127.0.0.1:31234 connect=0ms echo=4ms (5B)
+21.005s [pktcap] tcp-packet [eth0] 192.168.97.2:46896 -> 192.168.97.3:31234 SYN
+21.005s [pktcap] tcp-packet [cni0] 10.42.0.1:29742 -> 10.42.0.142:12345 SYN
+21.005s [pktcap] tcp-packet [veth0829e839] 10.42.0.1:29742 -> 10.42.0.142:12345 SYN
+21.005s [pktcap] tcp-packet [veth0829e839] 10.42.0.142:12345 -> 10.42.0.1:29742 SYN,ACK
+21.005s [pktcap] tcp-packet [cni0] 10.42.0.142:12345 -> 192.168.97.2:46896 SYN,ACK
+21.005s [pktcap] tcp-packet [eth0] 192.168.97.3:31234 -> 192.168.97.2:46896 SYN,ACK
+21.005s [pktcap] tcp-packet [eth0] 192.168.97.2:46896 -> 192.168.97.3:31234 ACK
+21.005s [pktcap] tcp-packet [cni0] 10.42.0.1:29742 -> 10.42.0.142:12345 ACK
+21.005s [pktcap] tcp-packet [veth0829e839] 10.42.0.1:29742 -> 10.42.0.142:12345 ACK
We get a mix of behavior from the Happy Path and Break #2.
As I showed in the previous post about race conditions, this can happen when a pod is terminated before kube-proxy has a chance to remove the iptables rule DNAT-ing traffic to it. Unlike the first two examples – in practice nobody is going to intentionally modify EndpointSlices with the wrong IP addresses / ports – this can very well happen in real world production scenarios. You’d observe general flakiness / intermittent connection failures around rolling deploys.
The experiments in this post clearly reveal that Kubernetes services are just an abstraction on top of the Linux kernel’s iptables/netfilter subsystem. All of the errors didn’t really have to do with the Kubernetes components themselves, but were just standard Linux networking errors you’d get any time you tried to access the wrong port or IP address.
We had a traefik proxy running inside of a Kubernetes cluster, acting as an API gateway in front of various backend services and we noticed that there would be occasional network errors during rolling restarts of the corresponding K8s Deployment. There were small windows of downtime during deploys. Granted this was small enough not to be noticed most of the time, but it flatly contradicted my previous expectation, and I wanted to understand why this was happening.
Some googling led me to this Gruntwork blog post and then this excerpt from Kubernetes In Action which gave a good explanation of the problem.
To summarize – the basic problem is that there are two processes that happen in parallel when a pod which is fronted by a Kubernetes service shuts down. The first is that SIGTERM is sent to the actual process running inside the pod container, which is handled according to whatever the application is coded to do. The second is that the pod’s IP is removed from the list of endpoints belonging to the service, which is then picked up by kube-proxy, which then removes the corresponding iptables rule for the pod’s IP address. There is no built in mechanism for keeping these two in sync, so it can often happen that the process is terminated (and thus is not listening on its IP address) before the iptables rule for routing traffic to its IP address is removed.
This makes sense in theory – Kubernetes deployments and services are managed by separate controllers that operate independently, so why should I have assumed they would magically sync up? Still, I wanted to really prove to myself that this was actually happening. There are a lot of moving parts here – how could I possibly keep track of them at once?
The answer is that I needed a custom-built tool. And the realization I came to is that building this kind of tool is now possible in a way that just wasn’t previously. The specific unlock was being able to use LLM-assisted coding to dramatically lower the tedium involved.
The first problem was to reliably reproduce the network errors. Because the window of downtime was so small, the error didn’t always happen “organically”. I needed to create some artificial traffic while performing the rolling restart in order to trigger the error.
I’m not ashamed to admit that in the past I would have written a bash loop that cURLs the traefik URL, opened up a separate terminal to trigger the rolling restart, and then tried to move my eyes back and forth fast enough to catch the error. Instead, this time I had Claude Code write a Go program that performs continuous HTTP requests against Traefik’s URL, and in a separate goroutine triggers a rolling restart of the deployment. Then I had it log timestamped events for each operation.
This isn’t a hard problem, but it would have been tedious to code it all out by hand, and I probably would have decided it wasn’t worth the effort for this “small” network blip.
I’ve recreated the environment for the purposes of this blog post. Here’s an example of running the tool against a stock k3d cluster, hitting the built-in traefik proxy. I set up docker port-forwarding so it was exposed on localhost:80. I also modified it so there would be 2 replicas, and created an Ingress for a basic webpage at the path /.
What’s interesting is that the availability gap on a single node k3d cluster is very small, much smaller than the gap I noticed on the “real” cluster, so I had to be pretty aggressive with the rate of HTTP requests. Here’s an example run where it fires a request every 100ms:
+0.006s [deployment-rollout] restarting deployment=traefik namespace=kube-system
+0.009s [http] response duration=7.967706ms method=GET status=200 url=http://localhost:80
+0.037s [deployment-rollout] restarted deployment=traefik namespace=kube-system
+0.113s [http] response duration=3.19735ms method=GET status=200 url=http://localhost:80
...
+2.312s [http] response duration=2.380564ms method=GET status=200 url=http://localhost:80
+2.377s [pod-termination] terminating deletion_timestamp=2026-04-17T05:50:32-04:00 namespace=kube-system pod=traefik-6fdcf5b898-rwbfd pod_ip=10.42.0.83
+2.412s [http] error error=Get "http://localhost:80": EOF method=GET url=http://localhost:80
+2.512s [http] response duration=2.286649ms method=GET status=200 url=http://localhost:80
+2.614s [http] response duration=4.442221ms method=GET status=200 url=http://localhost:80
You can see that it immediately triggers a restart of the k8s deployment. In the background this spins up a new pod, and after 2.377s it terminates one of the old ones, and then the next request 35ms later fails with EOF. Presumably this was the gap between when the traefik process had closed its listening socket but traffic was still being forwarded to it. Then in another 100ms the requests start succeeding again.
So with this I was able to reliably reproduce the error.
Next I prompted Claude Code to “add endpointslice events” to the output of the tool. Now I could see, next to the logs for the HTTP requests and the rolling restart events, the precise timestamped moment when the pod’s IP address was removed from the endpoint slice for the service.
+12.010s [http] response duration=3.896189ms method=GET status=200 url=http://localhost:80
+12.309s [http] response duration=3.026349ms method=GET status=200 url=http://localhost:80
+12.609s [http] response duration=2.908761ms method=GET status=200 url=http://localhost:80
+12.856s [endpointslice] slice-modified changes=[~10.42.0.99(ready)] slice=traefik-gmfgw
+12.884s [pod-termination] terminating deletion_timestamp=2026-04-17T06:16:40-04:00 namespace=kube-system pod=traefik-75f59cb466-qjccp pod_ip=10.42.0.97
+12.909s [http] error error=Get "http://localhost:80": EOF method=GET url=http://localhost:80
+12.914s [endpointslice] slice-modified changes=[~10.42.0.97(terminating)] slice=traefik-gmfgw
+13.058s [endpointslice] slice-modified changes=[-10.42.0.97(terminating)] slice=traefik-gmfgw
+13.863s [pod-termination] deleted namespace=kube-system pod=traefik-75f59cb466-qjccp pod_ip=10.42.0.97
This was interesting, but the endpoint slices are just abstract API objects, in themselves they don’t have any impact on the actual network packets flying around. What we really care about are the actual iptables rules, so that’s what I decided to add next.
Now this is really starting to get in the realm of things that would have not been worth the time without AI assistance. Without going into too much detail about iptables/nftables (that’s worth its own post), the solution I landed on was to:
nft monitor to capture rule changes when kube-proxy makes them. Modern Linux distributions convert iptables -> nftables under the hood, so everything shows up under nft monitor.kube-proxy calls iptables-restore, which deletes and recreates everything on every change.stdoutThe algorithm is easy to describe, so I was able to explain to Claude Code how to do it, but coding it all out would have taken quite a bit of time. I was also able to clone the source code for both kube-proxy and iptables and point Claude Code at it for reference material when designing the implementation.
With this I was able to see the actual iptables changes on the same timeline as the pod termination events and the tcp network errors.
+12.610s [http] response duration=2.260901ms method=GET status=200 url=http://localhost:80
+12.803s [endpointslice] slice-modified changes=[~10.42.0.105(ready)] slice=traefik-gmfgw
+12.825s [pod-termination] terminating deletion_timestamp=2026-04-17T06:30:28-04:00 namespace=kube-system pod=traefik-7b685bbcdd-cktdt pod_ip=10.42.0.102
+12.846s [endpointslice] slice-modified changes=[~10.42.0.102(terminating)] slice=traefik-gmfgw
+12.910s [http] error error=Get "http://localhost:80": EOF method=GET url=http://localhost:80
+13.010s [endpointslice] slice-modified changes=[-10.42.0.102(terminating)] slice=traefik-gmfgw
+13.211s [http] response duration=3.169981ms method=GET status=200 url=http://localhost:80
+13.566s [nftables] endpoint-added pod=10.42.0.105
+13.566s [nftables] endpoint-removed pod=10.42.0.102
This shows that the pod began terminating about 700ms before the corresponding iptables rule was removed, leaving a gap of downtime. Seeing the theory confirmed in real time like this was very cool. And I think this is only scratching the surface. We could keep digging by adding:
strace captures for when the Traefik pod actually closed its listening socket or called exit()Finally, seeing that the gap was consistently a few hundred milliseconds, the solution was to add a preStop lifecycle hook to the Traefik pod, to give the endpoints / iptables rules time to update before it stopped serving traffic:
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- sleep 10
Then the error is consistently avoided:
+20.944s [pod-termination] terminating deletion_timestamp=2026-04-17T07:03:43-04:00 namespace=kube-system pod=traefik-54fcb567cc-4j6hk pod_ip=10.42.0.112
+20.972s [endpointslice] slice-modified changes=[~10.42.0.112(terminating)] slice=traefik-gmfgw
+21.041s [http] response duration=3.703407ms method=GET status=200 url=http://localhost:80
+21.342s [http] response duration=3.984828ms method=GET status=200 url=http://localhost:80
...
+21.947s [nftables] endpoint-removed pod=10.42.0.112
...
+31.120s [endpointslice] slice-modified changes=[-10.42.0.112(terminating)] slice=traefik-gmfgw
+31.209s [pod-termination] deleted namespace=kube-system pod=traefik-54fcb567cc-4j6hk pod_ip=10.42.0.112
+31.240s [http] response duration=2.995166ms method=GET status=200 url=http://localhost:80
Now you can see that there is a big gap between the terminating and deleted events, more than big enough for the iptables changes to be applied, and so the HTTP errors are avoided.
There’s a lot of discussion about LLMs atrophying your programming skills, giving you a shallower understanding of the system you’re working on, and just generally reducing your competence as an engineer. In general, there may be some truth to that, but this case was the opposite experience. I was able to gain a much deeper and much more visceral understanding of a complex networking/distributed systems problem than would have been possible previously, at least on a reasonable timeframe.
]]>Organizations in segments like government, healthcare, and finance all have extensive legal regulations which make on-prem a de-facto requirement. They could potentially benefit from your software but need you to package it in a way that works with this on-prem requirement.
In order to achieve this, it’s important to avoid certain assumptions that wouldn’t hold in an on-premises environment. Any of these assumptions could end up being a future dealbreaker for a lucrative contract.
Here’s a few examples that I used to take for granted when I had unlimited access to whatever cloud services I wanted and had unlimited time to fine-tune the singleton SaaS environment to my liking. The purpose is not to be comprehensive, but just to get you thinking.
Your SaaS offering might assume it has free rein to access the internet. Maybe it tries to access a third party service hosted on the internet – cloud hosted email providers, payment processors, telemetry collectors, etc. Maybe your install/deploy process tries to download dependencies from the internet.
Any of these can result in network connectivity errors, where your application hangs for 30 seconds on startup and then crashes with a cryptic error message.
You should identify every outbound network dependency that your application has. It’s probably a lot more than you think. Then as much as possible think about an internal solution that could replace these, or make the dependency optional and think about a way for your application to handle it gracefully.
Your SaaS offering might assume it can pull secrets from AWS Secrets Manager, a managed Hashicorp Vault instance, or Google Cloud Secrets. On prem these secrets need to be pulled from some secure storage inside the customer’s environment.
Audit your code for assumptions about where it pulls secrets from. Add support for general purpose injection methods like environment variables, or bundle your own secret management solution. Make it easy for the customer to provide the secret inputs your application needs without exposing them to unnecessary security risks.
Six months after you deploy to the customer’s environment, they start getting confusing errors about certificate expiration. Your SaaS application used Let’s Encrypt, and certificate rotation was handled automatically, so no one thought about this problem in the customer’s environment.
Additionally, your SaaS offering might serve TLS with a certificate obtained from a public CA, but your on-prem customer might have their own internal CA they want you to use, or maybe you need to come with a solution to package some kind of mini-PKI with your application.
Your SaaS offering might run on cloud hosted Kubernetes. On-prem, the customer might not have a Kubernetes cluster, or might have their own in house cluster. Or maybe you need to ship a self-hosted Kubernetes distribution like K3s, which may have different constraints than your cloud-hosted Kubernetes.
Ideally you should have integration tests which spin up your application from scratch in the type of environment you expect your customer to have.
Your SaaS offering might have integrations with cloud hosted identity providers like Google, but on-prem customers might want to authenticate users with LDAP or Kerberos (Active Directory). Try to keep your authentication methods de-coupled from the rest of your business logic.
This is potentially a more complex undertaking than the others, but could really move the needle with enterprise buyers.
Your SaaS offering is available at my-cool-saas.com. On-prem it becomes my-cool-saas.corp.internal. Same for any auxiliary services like databases. Same for internal service to service communication if your app has multiple services.
Audit your code and make sure it doesn’t assume a particular service is available at a hard-coded hostname. All hostnames should be somehow injectable from the environment – configuration files, environment variables, etc.
These are just a few of the assumptions that can creep in when you’re only developing for a SaaS environment that you have full control over. The common thread running through them is that they couple the business logic of your software (what your buyer is paying for) with the environment that it happens to run in today.
And even if you’re not planning to deploy on-prem any time soon, de-coupling these two will lead to more robust code that will help even a purely SaaS deployment. It’s a great exercise to really sit down and think critically about the actual requirements of your application, and how you could make it more portable.
]]>