Debbuging RDS EKS Network

Introduction

In Kubernetes environments, debugging RDS connectivity issues can be challenging due to the complexity of network configurations, security policies, and container isolation. A common scenario I’ve encountered involves a pod that fails to connect to an RDS database, leaving me uncertain whether the issue lies with the pod, the network setup, or the RDS instance itself.

To streamline the debugging process, I use a temporary bastion pod deployed inside the Kubernetes cluster. This pod acts as a simple, controlled environment where I can directly test RDS connectivity. By doing so, I can quickly verify if the network configuration, security groups, and database credentials are correctly set up. Once a successful connection is established from the bastion pod, I know that the underlying infrastructure is sound, allowing me to focus on other potential causes if the problem persists within the application pod itself.

This approach helps to systematically isolate and troubleshoot connectivity issues, saving time and effort in identifying the root cause.

Steps and Explanation:

1. Pull Ubuntu/Postgres Docker Image

docker pull ubuntu/postgres

• This pulls the ubuntu/postgres Docker image from DockerHub. This image combines the Ubuntu OS with PostgreSQL tools, you could install this in a second instance but you might have your node in a private subnet and won’t be able access the internet, which you will use for testing RDS connections.

2. Create ECR Repository

aws ecr create-repository --repository-name ubuntu --region your_region

• This command creates an Amazon Elastic Container Registry (ECR) repository called ubuntu in the your_region region. You will push your Docker image here for use in your Kubernetes cluster.

3. Describe ECR Repository

aws ecr describe-repositories --repository-names ubuntu --region your_region

• If you accidentally exit or lose track of your repository, you can retrieve its details using this command.

4. Tag the Docker Image

docker tag ubuntu/postgres:latest {account_id}.dkr.ecr.{region}.amazonaws.com/ubuntu

• This tags your local Docker image for pushing to your newly created ECR repository. Replace {account_id} and {region} with your AWS account ID and region.

5. Login to ECR

aws ecr get-login-password --region your_region | docker login --username AWS --password-stdin {account_id}.dkr.ecr.{region}.amazonaws.com

• This authenticates your Docker client with ECR so that you can push your image. The login credentials are fetched automatically using the AWS CLI.

6. Kubernetes Pod Configuration (YAML)

apiVersion: v1
kind: Pod
metadata:
  name: bastion-pod
  labels:
    app: ubuntu
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
            - key: purpose
              operator: In
              values:
              - public
  containers:
  - name: ubuntu-container
    image: {account_id}.dkr.ecr.{region}.amazonaws.com/ubuntu
    command: ["/bin/bash", "-c", "sleep infinity"] # Keeps the container running
    tty: true # Enable terminal support
    stdin: true # Enable interaction
  restartPolicy: Always

• Purpose: This YAML configuration creates a pod named bastion-pod with an ubuntu-container inside it. The container uses the Ubuntu image you pushed to ECR and runs a bash command that keeps it running indefinitely. This allows you to access the pod later for testing RDS connectivity.
• Network Setup: Make sure that the network setup, including Route Tables and Security Groups, is configured properly to allow the pod to access the RDS instance.

7. Apply Kubernetes Pod Configuration

kubectl apply -f bastion_pod.yaml

• This applies the configuration for your bastion pod, which you’ll use to connect to the RDS instance. The YAML configuration is detailed below.

8. Connect to the RDS Instance

psql -h your_rds_endpoint.amazonaws.com -p 5432 -U your_username -d postgres

• Use the PostgreSQL CLI psql to connect to your RDS instance. Replace the placeholder values with your RDS endpoint, username, and database name.

9. Validate Network Setup

If you start to encounter network issue start ensuring that:

• Route Tables allow traffic to/from the RDS instance.
• Security Groups attached to both the bastion pod and the RDS instance allow necessary inbound/outbound traffic on port 5432 (PostgreSQL default port).

10. Clean Up

kubectl delete pod bastion-pod

• After testing the connection, delete the bastion pod to clean up resources.