Some notes on creating website proxies in Nginx with only Kubernetes resources.

To use a recent example I worked on, say we've been using an Amazon S3 bucket with a custom domain name - files.example.com. This works nicely as you can CNAME your custom domain to the bucket's FQDN. However we need to migrate this to a different object-storage provider, and they pass the bucket name in the URL path. i.e.:

  • CNAME files.example.com -> files.example.com.s3.amazonaws.com
  • New bucket URI on Oracle Cloud: objectstorage.eu-frankfurt-1.oraclecloud.com/n/example/b/files.example.com

This poses a new problem - we need to rewrite URLs to be prefixed by the bucket path. This can all be achived with a minimal Nginx reverse proxy:

server {
  rewrite ^/?(.*) /n/example/b/files.example.com/$1 break;
  proxy_pass https://objectstorage.eu-frankfurt-1.oraclecloud.com;
  proxy_redirect off;
}

That works, but it would be nice to add this into our cluster ingress controller. After some experimentation, I found that Ingress-Nginx can only proxy to internal services. I also found that you can declare a service of type ExternalName, that is effectively a DNS CNAME. This allows Ingresses to route externally out of the cluster.

apiVersion: v1
kind: Service
metadata:
  name: objectstore
spec:
  type: ExternalName
  externalName: objectstorage.eu-frankfurt-1.oraclecloud.com
  ports:
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP

Then, we can add an ingress to expose this service:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: files-example-com
  annotations:
    kubernetes.io/ingress.class: "nginx"

    # URL rewriting - see paths for $1
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: "/n/example/b/files.example.com/o/$1"
spec:
  rules:
    - host: "files.example.com"
      http:
        paths:
          - path: "/(.*)"
            backend:
              serviceName: objectstore
              servicePort: 443

SSL termination and other Nginx configurations can be applied in the usual way. Oracle's object store has a couple of undocumented quirks - it only operates on HTTPS and requires a Host header. We'd also like to have an alias directory to redirect requests to an older resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    [...]

    # Use https for upstream proxy_pass
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

    # Send a Host header to the upstream
    nginx.ingress.kubernetes.io/upstream-vhost: "objectstorage.eu-frankfurt-1.oraclecloud.com"

    # Redirect certain paths with a temporary redirect (302)
    nginx.ingress.kubernetes.io/configuration-snippet: |
     rewrite ^/myapp/(.*) /apps/myapp/1.0.0/$1 redirect;