CDC with Postgres Debezium to Kafka Strimzi

Postgres Database — Kafka Connect — Kafka

A little intro to Strimzi:

A little intro to Debezium:

Starting up Minikube

minikube config set memory 4096
minikube config set cpus 4
minikube start

Create a Postgres database

Kubectl create namespace postgres
Username: data_engineerPassword: password
kubectl get pods -n postgres

Strimzi for Kafka

Kubectl create namespace kafka
curl -L https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.16.1/strimzi-cluster-operator-0.16.1.yaml | sed ‘s/namespace: .*/namespace: kafka/’ | kubectl apply -f — -n kafka
kubectl -n kafka apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.16.1/examples/kafka/kafka-persistent-single.yaml && kubectl wait kafka/my-cluster — for=condition=Ready — timeout=300s -n kafka

Create the KafkaConnect

Create Postgres Connector

kubectl -n kafka port-forward svc/postgres-connect-cluster-connect-api 8083:8083

How To Test:

kubectl -n postgres port-forward svc/postgres 5432:5432
kubectl -n kafka exec my-cluster-kafka-0 -c kafka -i -t --   bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic server1.inventory.customers

References:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store