Building a Cluster-Aware Semantic Cache for 20 Newsgroups | AI/ML Engineer Task

Описание: An intelligent Information Retrieval system featuring Fuzzy logic and optimized Semantic Caching. Implements a 'bucketed' search architecture to reduce latency by searching specific clusters. Built with Python, FastAPI, ChromaDB, and skfuzzy.

Overview: A complete walkthrough of a production-ready Semantic Search Engine built on the 20 Newsgroups dataset (~20k articles). This project was developed as an AI/ML Engineer technical assessment, focusing on high-performance vector retrieval, dynamic clustering, and a custom-built semantic cache.

🔗 GitHub Repository: https://github.com/Shanmuk4622/Tradem...

🚀 Key Features Demonstrated in this Video:

End-to-End Data Pipeline: Loading, cleaning, taking out headers/footers, and embedding the 20 Newsgroups corpus using all-MiniLM-L6-v2.
Two-Phase Fuzzy C-Means Clustering: Intelligently bucketing documents into 13 high-dimensional semantic clusters to map the true underlying topics rather than assuming existing labels.
Cluster-Aware Semantic Cache: A highly optimized caching engine (built from scratch in pure Python) that achieves O(n/k) lookup times by exclusively scanning within the user's target semantic cluster.
Minimalist Web UI: A custom, Vercel-inspired light-mode frontend with glassmorphism, dynamic animations, and beautifully rendered contextual search results.
Docker & GPU Fast-Path Execution: Demonstrating the dual-path setup where the pipeline runs natively on NVIDIA GPUs (~2 mins) and is served instantly via Docker (~5 secs).
🛠 Technology Stack:

Backend Flow: FastAPI, Python 3.10
Machine Learning Flow: PyTorch (CUDA), Sentence-Transformers, SciPy, scikit-learn
Vector Database: ChromaDB
Frontend / UI: HTML5, TailwindCSS, Vanilla JavaScript