Data Pipelining and Transformation System

Invisible Technologies

This is my specialization, Design and Develop data streaming systems using Apache Airflow that pulls data from multiple sources.

AirflowFastApiPostgresDockerKubernetesAWS EKSGCP GKELeadership

At Invisible, I’ve worked on a workflow management engine that was used to train AI models by the specialists. The platform is similar to n8n but need specific integration support for cohere/OpenAI models which was prevalent at that time

Integations

  1. LLM Models (Cohere/OpenAI)
  2. Whatsapp
  3. Zapier

Infrastructure

  1. CI/CD pipelining
  2. Doppler setup
  3. Docker & Kubernetes
  4. Datadog
  5. Celery

Tech stack

  1. Python
  2. Typescript