Atul Kumar Pandey — Analytics & Data Engineer • Applied AI Systems

Measurable Impact

Results from professional roles and project repositories. Project figures are measured on synthetic data.

10M+

Records Modeled

Star-schema warehouse on Amazon Redshift for student and course reporting

4→1

Shared Data Model

One model across four platforms, so teams stopped reporting different numbers for the same metric

99%

SLA Adherence

Held across reporting workflows for a 20+ member operations team

20%

Faster Resolution

Salesforce case routing automated in Python

96%

Customer Retention

Multi-channel e-commerce operations across Amazon, eBay and Shopify

33→53%

Precision

Fraud false-positive fix; recall 86% → 81%, measured on synthetic data

5/5

Scenarios Classified

Agentic data quality monitor, judge-scored on synthetic failure scenarios

27

dbt Data Tests

Staging and mart models in the stock market lakehouse

About Me

I'm an Analytics & Data Engineer with about six years across e-commerce operations, product analytics and data engineering. My journey began in e-commerce operations, reconciling orders and inventory across marketplaces that rarely agreed. That's where I learned a number is only useful if people trust it.

At BYJU'S I built modeled warehouse layers and shared metric definitions for reporting across four platforms. I have since completed an MS in Business Analytics (AI and Data Analytics) at UMass Boston with a 4.0 GPA.

Today I focus on governed agentic workflows. In these systems a language model proposes, deterministic checks and policies decide what runs, and people approve anything that leaves the system. I build them with Kafka, Spark, Snowflake, dbt, PostgreSQL and LangGraph.

Current Focus

Governed agentic workflows with human approval
Analytics engineering: dbt layers & shared metric definitions
Grounded analytics, RAG & LLM evaluation
Reliable enterprise reporting & data quality

Core Principles

Observability

Reliability

Simplicity

Governance

Technical Expertise

Tools I have used in professional roles and in the projects below

Cloud Platforms

Amazon Web Services

S3GlueLambdaRedshiftAthena

Google Cloud Platform

BigQueryCloud StorageComposer

Microsoft Azure

DatabricksADLS Gen2Data Factory

Big Data & Streaming

Processing Engines

Apache SparkPySparkSpark Structured StreamingApache Flink

Streaming Platforms

Apache KafkaApache NiFiSnowpipe

Applied AI & Agentic Systems

NEW

Agentic Workflows

LangGraphMulti-Agent SystemsHuman-in-the-LoopStructured Outputs

RAG & Retrieval

RAG PipelinesPineconeWeaviateNL2SQL

Evaluation & Guardrails

LLM-as-JudgeLangSmith TracingSQL GuardrailsAudit Logging

Data Warehousing & Orchestration

Data Warehouses

SnowflakePostgreSQLRedshiftBigQueryApache IcebergDelta Lake

Orchestration & Transform

Apache AirflowdbtMage AI

Data Governance

dbt TestsGreat ExpectationsCDC / SCD2Row-Level SecurityRBAC

Programming & DevOps

Languages

PythonSQLPySpark

Backend & DevOps

FastAPIDockerGitHub ActionsPrometheus

Visualization

Power BITableauLookerStreamlit

Featured Projects

Independent projects. Each description reflects what the repository implements, including its limits.

FEATURED GOVERNED AI LIVE PILOT

Natural Language Workflow

A controlled natural-language analytics workflow, deployed as an invitation-only pilot. An LLM proposes; deterministic feasibility and policy checks decide what runs; answers are grounded in evidence; and a human approves before anything is delivered to Slack.

Request Flow

Plain-English Request

LLM Typed Proposal

Feasibility & Policy Checks

Durable Run (PostgreSQL)

Grounded Result

Human Approval

Slack Delivery

Key Features

LLM returns a typed proposal via structured output; deterministic checks decide to run, ask for clarification, require approval or reject
Grounded analytics: every KPI and finding links to the execution step that produced it
Tenant isolation enforced in PostgreSQL with row-level security and signed, expiring request context
A different admin must approve before Slack delivery (at-least-once, no silent resends); container images deploy only after build provenance is verified

Scope

Status Invitation-only pilot

Data Synthetic Sales & Support

Delivery Human-approved Slack

Python FastAPI PostgreSQL Redis / Dramatiq Anthropic Claude Next.js Docker GitHub Actions Prometheus

Repository Live pilot Demo video

FEATURED MULTI-AGENT HUMAN-IN-THE-LOOP

Agentic Fraud Decisioning Platform

Streaming fraud scoring on synthetic transactions with a LangGraph multi-agent review layer that cannot skip policy: decisions are tiered by risk, routed to human review when needed, and every reasoning step is logged for audit.

Decision Flow

Key Features

Spark Structured Streaming computes velocity, geo and device features into Redis (online) and Snowflake (offline)
LangGraph supervisor routes feature, risk, policy and decision agents; policy-before-decision is enforced as a routing rule in code, not a prompt
Decisions tiered into auto-approve, notify-only or suggest-for-review by confidence and amount; reviewers confirm, override or escalate
False-positive investigation traced excessive geo flags to rule logic; SELECT-only SQL guard and audit traces in Snowflake

Impact

Precision 33% → 53%

Recall 86% → 81%

Data Synthetic transactions

Kafka Spark Structured Streaming Redis Snowflake LangGraph Weaviate FastAPI Docker

View on GitHub

FEATURED BATCH + STREAM NL2SQL + RAG

Stock Market Intelligence Lakehouse

Lakehouse with batch and Kafka stream paths into Apache Iceberg on MinIO, a dbt-modeled Snowflake layer, and an NL2SQL + RAG layer with LLM observability

Data Flow

Key Features

Apache Iceberg on MinIO for ACID transactions and time travel on the lake layer
Airflow runs batch and stream DAGs with idempotent Snowflake MERGE loads; dbt staging and mart models carry 27 data tests
Hybrid RAG combines dbt BUY/SELL signals with news embeddings in Pinecone via a Groq-hosted LLM, flagging conflicts between price momentum and news sentiment
FastAPI service with NL2SQL over Snowflake; every LLM call logged into bronze/silver/gold tables to track answer quality over time

Highlights

Data Lake Apache Iceberg

Data Tests 27 dbt tests

AI Layer NL2SQL + RAG + Observability

Kafka Apache Spark Apache Iceberg MinIO Airflow dbt Snowflake FastAPI Pinecone Groq LLM

View on GitHub

FEATURED AGENTIC AI LLMOps

Agentic Data Quality Monitor

A LangGraph triage agent for failed dbt tests: it investigates upstream Snowflake data and Airflow logs, diagnoses the root cause and proposes a fix, with human approval before anything executes and LLM-as-judge evaluation

Agent Flow

Key Features

LangGraph agent with a ReAct loop over dbt results, failing-row samples, upstream schema and Airflow logs
Human-in-the-loop checkpoint via LangGraph interrupt: an engineer approves the proposed SQL fix before it runs
Guardrails validate proposed SQL; every run is written to a Snowflake audit table
LangSmith tracing, with a separate judge model scoring five synthetic failure scenarios

Evaluation

Classification 5 / 5 scenarios

Explanation Quality 4.4 / 5

Pattern Human-approved remediation

LangGraph LangSmith Groq LLM Snowflake dbt Airflow GitHub Actions

View on GitHub

LAKEHOUSE

E-Commerce Analytics Lakehouse

Medallion architecture on Azure Databricks with ADF event-triggered ingestion

Bronze → Silver → Gold · 15+ PySpark quality rules · ACID

Delta LakeTime-travel + schema evolution

Databricks Delta Lake ADLS Gen2 Data Factory PySpark

GenAI · RAG

Enterprise Document RAG Pipeline

Airflow-orchestrated RAG over enterprise documents with change-detected re-embedding

FreshnessMD5 change detection

RetrievalPinecone + Groq LLM

Airflow Pinecone Groq LLM Streamlit Python

GCP

Uber Data Analytics Pipeline

ELT into a BigQuery star schema with Looker reporting

ModelFact + dimension tables

QualityData quality checks

Mage AI BigQuery Cloud Storage Looker

CDC

Snowflake CDC Pipeline with SCD Type 2

Change data capture from AWS S3 into Snowflake with SCD Type 2 history

• Apache NiFi lands files in S3; Snowpipe loads them into Snowflake
• SCD Type 2 history via Snowflake Streams & Tasks
• Synthetic change data generated with Faker

NiFi Snowpipe Streams & Tasks Docker

Professional Journey

From e-commerce operations to product analytics to data engineering

Data Engineer

Think and Learn Pvt Ltd (Byju's) Jul 2022 – Jul 2024

Warehouse modeling, pipelines and data quality for ed-tech reporting

Key Achievements

Designed and tuned a star-schema warehouse on Amazon Redshift covering 10M+ student records, using distribution and sort keys to keep reporting fast
Replaced manual CSV exports with Airflow-orchestrated AWS Glue jobs and automated MongoDB and DynamoDB ingestion in Python
Coordinated data-delivery requirements across a cross-functional group and mentored junior engineers
Built one shared data model across four platforms for learning, course-completion and revenue reporting, so teams stopped reporting different numbers for the same metric

Technical Impact

Data Quality Framework

Shared schema, null and SLA checks that other analytics teams adopted in their own pipelines

Airflow Orchestration

Airflow-orchestrated Glue jobs replacing manual exports across ingestion and reporting

PythonSQLAmazon RedshiftAWS GlueAirflowMongoDBData Quality

Data Analyst (Product Expert)

Think and Learn Pvt Ltd (Byju's) Oct 2020 – Jun 2022

Data operations and reporting for a 20+ member team

Key Achievements

Automated Salesforce case routing using Python, reducing issue resolution time by 20%
Built executive KPI dashboards in Power BI and SQL for strategic decision-making
Maintained 99% SLA adherence for reporting and data workflows

Technical Work

SQL Reconciliation

Reconciled reported figures against source systems and traced discrepancies to the step that caused them

Fit / Gap Analysis

Compared what stakeholders asked for with what the systems could support, and proposed the workable option

SalesforcePythonSQLPower BIKPI Dashboards

Operations Manager (E-Commerce Ops)

Binarify Jun 2018 – Sep 2020

E-commerce operations across Amazon, eBay, and Shopify platforms

Key Achievements

96% customer retention through business analytics and data-driven process optimization
Built inventory tracking dashboards and streamlined order-to-delivery workflows
Developed logistics reporting & vendor tracking systems in SQL — foundation for data automation

Process Automation

Fulfillment Efficiency

Improved operational visibility and fulfillment workflows across multi-platform e-commerce

SQLE-Commerce AnalyticsInventory ManagementAmazon/eBay/Shopify

Education & Certifications

Formal education, a professional certificate and short courses

Formal Education

MS Business Analytics (AI and Data Analytics)

University of Massachusetts Boston

GPA: 4.0May 2026

Advanced Machine Learning, Predictive Analytics, Big Data Processing, Statistical Modeling

BE Mechanical Engineering

Visvesvaraya Technological University, Bangalore

GPA: 3.8Jun 2018

Certification & Courses

Data Engineering Professional Certificate

DeepLearning.AI • Professional Certificate

Vector Databases & Embeddings, RAG, LangChain for LLM Applications, LLMOps

DeepLearning.AI • Short courses

Data Warehousing with Snowflake

Datavidhya • Course

Apache Spark with Databricks

Datavidhya • Course

Apache Airflow & Apache Kafka

Datavidhya • Courses

Let's Work Together

I'm open to full-time roles in analytics engineering, data engineering and applied AI, especially on teams where reporting has to be trusted and AI has to be governed.

Boston, Massachusetts

Available for remote/hybrid roles

atulpandey02@gmail.com

Best way to reach me

LinkedIn Profile

Professional network

View My Work