⚡
Atul Kumar Pandey

Available for hire • Boston, USA

Atul
Kumar
Pandey

AI Data Engineer

Building production data platforms processing 500K+ records daily for a 150M+ user platform across AWS, GCP, and Azure.

Specialized in real-time streaming, lakehouse architecture, and agentic AI & GenAI/RAG infrastructure — building the data systems that make AI reliable in production.

Get in touch View Projects
6+
Years Experience
40
Member Team Led
4.0
GPA (MS Analytics)
Atul Kumar Pandey - AI Data Engineer

Measurable Impact

Quantifiable results from data and AI engineering initiatives across different organizations

75%

Processing Time Cut

Reduced manual data processing effort on 500K+ daily records

500K+

Records / Day

Production ETL pipelines serving a 150M+ user platform

99%

SLA Achievement

Consistent data pipeline reliability and reporting adherence

15%

Analytics-Driven Insights

Increase from unified data model architecture

40

Member Team Led

Cross-functional Agile data engineering team

5min

CDC Latency

End-to-end change data capture from S3 to warehouse

96%

Customer Retention

Achieved through data-driven e-commerce operations

20%

Faster Resolution

Via Salesforce automation and Python-driven workflows

About Me

I'm an AI Data Engineer with 6+ years of experience building production pipelines, analytics infrastructure, and the data systems behind AI applications. My journey began in e-commerce operations, where I discovered the transformative power of data-driven decision making.

Completed an MS in Business Analytics (AI and Data Analytics) with a perfect 4.0 GPA at UMass Boston, I specialize in building enterprise-scale data platforms that process hundreds of thousands of records daily for platforms serving 150M+ users.

My expertise spans the full data stack: from real-time streaming with Kafka and Spark to lakehouse architectures on Databricks, and cutting-edge agentic AI and GenAI infrastructure including LangGraph agents, RAG pipelines, and LLMOps with full observability and evaluation.

Current Focus

  • Real-time streaming & event-driven architectures
  • Lakehouse & medallion architecture patterns
  • GenAI infrastructure: RAG pipelines & vector databases
  • MLOps pipeline automation & monitoring

Core Principles

Observability
Reliability
Simplicity
Scalability

Technical Expertise

Comprehensive skill set spanning the modern data engineering stack

Cloud Platforms

Amazon Web Services

S3GlueLambdaRedshiftEMRKinesisAthena

Google Cloud Platform

BigQueryDataflowPub/SubComposer

Microsoft Azure

DatabricksADLS Gen2SynapseData Factory

Big Data & Streaming

Processing Engines

Apache SparkPySparkSpark Structured StreamingApache Flink

Streaming Platforms

Apache KafkaAWS KinesisGCP Pub/SubApache NiFi

GenAI Infrastructure

NEW

RAG & LLMOps

RAG PipelinesLangChainLLMOps

Vector Databases

PineconeChromaDBEmbeddings

AI Integrations

Semantic SearchHybrid RetrievalBM25 + Vector

Data Warehousing & Orchestration

Data Warehouses

SnowflakeBigQueryRedshiftDelta Lake

Orchestration & Transform

Apache AirflowdbtMage AI

Data Governance

Great ExpectationsCDC / SCDIAM

Programming & DevOps

Languages

PythonSQLPySparkRBash

DevOps & IaC

DockerTerraformGitHub ActionsCI/CD

Visualization

TableauPower BILookerStreamlit

Featured Projects

Real-world data and AI engineering solutions with measurable business impact

FEATURED AGENTIC AI LLMOps

Agentic Data Quality Monitor

A self-healing pipeline agent built with LangGraph — detects data quality failures, diagnoses root cause by querying Snowflake, and proposes fixes with human-in-the-loop approval, full LangSmith tracing, and LLM-as-judge evaluation

Agent Flow

dbt Test Fails
Agent Investigates
Diagnose + Propose
Human Approval
Remediate

Key Features

  • LangGraph stateful agent with ReAct reasoning loop and custom tool use over dbt, Snowflake, and Airflow
  • Human-in-the-loop checkpoint via LangGraph interrupt — engineer approves the proposed SQL fix before any remediation runs
  • All tools exposed through a custom MCP server for cross-agent reuse
  • LangSmith tracing on every run with LLM-as-judge evaluation across synthetic failure scenarios

Impact

Category Accuracy 100%
Explanation Quality 4.4 / 5
Pattern Self-Healing Pipeline
LangGraph LangChain LangSmith MCP Groq LLM Snowflake dbt Airflow Docker GitHub Actions
View on GitHub
FEATURED REAL-TIME GenAI + RAG

Stock Market Intelligence Pipeline V2

Production-grade lakehouse with dual batch + streaming pipelines, Apache Iceberg on MinIO, Snowflake RBAC governance, hybrid RAG + NL2SQL AI layer, and full LLM observability

Data Flow

Finnhub API
Kafka Topics
Spark + Iceberg
dbt → Snowflake
FastAPI Layer
RAG + NL2SQL

Key Features

  • Apache Iceberg on MinIO replacing Parquet — ACID transactions, time travel, and schema evolution on the data lake layer
  • Hybrid RAG fusing dbt BUY/SELL signals with 300+ Pinecone news embeddings via Groq LLM — flags conflicts between price momentum and news sentiment
  • FastAPI layer (7 endpoints) with NL2SQL, rate limiting, and Snowflake RBAC with column-level masking for production governance
  • AI observability: every LLM call logged into Bronze/Silver/Gold Snowflake tables; 27 dbt tests halt the pipeline before bad data reaches consumers

Impact

Data Lake Apache Iceberg
Architecture Lakehouse + Lambda
AI Layer RAG + NL2SQL + Observability
Kafka Apache Spark Apache Iceberg dbt Snowflake FastAPI Groq LLM Pinecone Airflow Docker GitHub Actions
View on GitHub
GenAI RAG PIPELINE

Enterprise Document Intelligence RAG Pipeline

Airflow-orchestrated RAG system with hybrid semantic search and CDC-driven embedding auto-refresh

Pipeline Flow

PDFs / Reports
Recursive Chunking
LangChain Embeddings
ChromaDB
Hybrid Search API

Key Features

  • Hybrid retrieval: BM25 + vector similarity with reranking for high precision
  • CDC-driven auto-refresh: embeddings update when source documents change
  • Semantic search API for enterprise document Q&A
  • Dockerized deployment with Airflow sensor orchestration

Technical Highlights

Retrieval BM25 + Vector Hybrid
Vector Store ChromaDB
Freshness CDC auto-refresh
LangChain ChromaDB Airflow BM25 Python Docker CDC
View on GitHub
LAKEHOUSE

E-Commerce Analytics Lakehouse

Medallion architecture on Azure Databricks with ADF event-triggered ingestion

Bronze → Silver → Gold · 15+ PySpark quality rules · ACID

Delta LakeTime-travel + schema evolution
Databricks Delta Lake ADLS Gen2 Data Factory PySpark
SERVERLESS

Spotify Music Analytics ETL

Serverless AWS pipeline with Snowpipe for zero-delay loading

SchemaDimensional (Artist / Album / Song)
TriggerS3KeySensor → Snowpipe
Airflow Lambda Glue S3 Snowflake
GCP

Uber Data Analytics Pipeline

Star schema DWH on BigQuery with Looker dashboards

Query Performance60% faster
Trip Records100K+
Mage AI BigQuery Cloud Storage Looker
CDC

Near Real-Time CDC Pipeline

5-min end-to-end CDC from AWS S3 to Snowflake with SCD Type 1/2

  • • NiFi ListFile/FetchFile + Snowpipe — 5-min S3→warehouse
  • • SCD Type 1/2 via Snowflake Streams & Tasks
  • • Dockerized with Faker for automated CDC testing
NiFi Snowpipe Streams & Tasks Docker

Professional Journey

Progressive career growth from operations to data engineering leadership

Data Engineer (Assistant Manager)

Think and Learn Pvt Ltd (Byju's) Jul 2022 – Jul 2024

Data engineering & analytics leadership for India's largest ed-tech platform (150M+ users)

Key Achievements

  • Built production ETL pipelines processing 500K+ daily records across student activity, attendance, assessments, and class interactions
  • 75% reduction in manual data processing effort through Python/SQL automation
  • Led cross-functional team of up to 40 members in an Agile environment — roadmaps, mentoring, stakeholder coordination
  • Unified data model architecture integrating siloed platforms — 15% increase in analytics-driven insights

Technical Impact

Data Quality Framework

Reusable schema validation, null checks, SLA monitoring — standardized across analytics verticals

Airflow Orchestration

Built and optimized DAGs across ingestion, transformation, and reporting with automated alerting and retry logic

PythonSQLAWSAirflowTableauData QualityAgile/Scrum

Data Analyst (Product Expert)

Think and Learn Pvt Ltd (Byju's) Oct 2020 – Jun 2022

Analytics leadership for customer operations team of 20+ members

Key Achievements

  • Automated Salesforce case routing using Python, reducing issue resolution time by 20%
  • Built executive KPI dashboards in Power BI and SQL for strategic decision-making
  • Maintained 99% SLA adherence for reporting and data workflows

Technical Work

Automated Reporting Pipelines

Eliminated manual overhead by extracting from Salesforce and internal databases

Operational Monitoring

Real-time tracking of team productivity and business-critical metrics

SalesforcePythonSQLPower BIKPI Dashboards

Operations Manager (E-Commerce Ops)

Binarify Jun 2018 – Sep 2020

E-commerce operations across Amazon, eBay, and Shopify platforms

Key Achievements

  • 96% customer retention through business analytics and data-driven process optimization
  • Built inventory tracking dashboards and streamlined order-to-delivery workflows
  • Developed logistics reporting & vendor tracking systems in SQL — foundation for data automation

Process Automation

Fulfillment Efficiency

Improved operational visibility and fulfillment workflows across multi-platform e-commerce

SQLE-Commerce AnalyticsInventory ManagementAmazon/eBay/Shopify

Education & Certifications

Continuous learning through formal education and industry certifications

Formal Education

MS Business Analytics (AI and Data Analytics)

University of Massachusetts Boston

GPA: 4.0May 2026

Advanced Machine Learning, Predictive Analytics, Big Data Processing, Statistical Modeling

BE Mechanical Engineering

Visvesvaraya Technological University, Bangalore

GPA: 3.8Jun 2018

Senior project on manufacturing process optimization

Professional Certifications

Data Engineering Professional Certificate

DeepLearning.AI • 2024

Vector Databases & Embeddings, RAG, LangChain for LLM Applications, LLMOps

DeepLearning.AI • 2024

Data Warehousing with Snowflake

Datavidhya • 2024

Apache Spark with Databricks

Datavidhya • 2024

Apache Airflow & Apache Kafka

Datavidhya • 2024

Let's Build Something Amazing

Ready to discuss your data engineering needs? Let's connect and explore how we can transform your data infrastructure.

Boston, Massachusetts

Available for remote/hybrid roles

atulpandey02@gmail.com

Quick response within 24h

LinkedIn Profile

Professional network
View My Work