India Fashion Data Directory

About This Project

The brief. A fashion market intelligence client needed something the Indian apparel market does not publish anywhere: a single queryable catalogue of it. Every major retailer, Indian and international, across sixteen style categories, with prices, reductions, sizes, colours, materials, imagery and classification in one consistent shape, refreshable on demand rather than bought once as a static export. Why one scraper was never going to work. No two storefronts publish product data the same way. Marketplaces expose private search APIs that return deeply nested payloads. Direct to consumer brands run hosted storefronts that publish a complete machine readable catalogue. Heritage retailers run enterprise commerce clouds with their own product APIs and their own pagination rules. A long tail sits on open source carts, CMS templates, and headless applications where the product grid only exists after the page has executed. Some sources expect a browser accurate client and simply do not answer a plain HTTP library. Writing one scraper for that surface produces something that is broken more often than it is working. The architecture. So the pipeline is registry driven rather than script driven. Every retailer is declared once in a JSON registry with its domain, resolved engine, country, market segment and seed paths. At runtime a router reads that registry and dispatches to one of 58 platform adapters, and every adapter, whatever it talked to, emits the same 27 field record. Adding a retailer is a registry entry, not a new program. Adding a platform is one adapter that the whole corpus immediately benefits from. The output contract never moves, so the client loads a marketplace listing and a boutique storefront with exactly the same code. Extraction tiers, priced by cost. Full browser rendering costs minutes per catalogue where a structured endpoint costs seconds, so rendering is the last resort rather than the default. Each source is profiled once and pinned to the cheapest tier that returns complete data, and the profile is written back into the registry so later runs start in the right place. 194 sources sit on tier one, a structured catalogue endpoint that returns price, every size and colourway, imagery and merchandising attributes in one paginated request. 35 sit on tier two, an XML sitemap traversal with structured product markup parsed off each document. Tier three, session and identity rotation, is promoted at runtime for sources that expect a browser accurate client: the transport replays a genuine handshake and rotates IP across a managed proxy pool along with headers, user agent and session identity. Only four sources are pinned to tier four, full browser rendering with scroll to load traversal and network response interception. Crawling responsibly at this volume. A corpus of this size is only reachable if the crawl behaves. Request pacing is tuned per host, concurrency is capped and adapts under load, transient failures retry with exponential backoff, and traversal respects robots and sitemap declarations with canonical URL resolution so the same product cannot enter the corpus twice. Every crawl checkpoints, so a run that spans hours re-enters mid catalogue rather than starting again, and each run writes a machine readable summary of exactly what was collected and what was held back. Reproducible by construction. Every harvest is a container, not a laptop. Ten image variants cover the different transport profiles, from a slim HTTP image through to a full browser image with a pinned build, and twenty four container services map onto the sixteen categories and their sub batches. Output paths are identical on every machine, the browser build ships inside the image rather than being inherited from the host, and rotation policy is supplied entirely through environment. Any category, or any single retailer, can be re-run by anyone, anywhere, and land in the same place. Normalisation is where the value is. Harvesting is the easy half. The normaliser reduces every platform's shape to one record: value formats collapse to a single numeric representation, reductions are recomputed from list and selling value rather than trusted from a badge, size options are flattened from variant trees into one ordered list, colour is separated from the sibling colourways many sources bundle into the same field, material is parsed out of free text, and imagery is deduplicated and consistently ordered. Category, audience and style classification are derived rather than harvested, because almost no source publishes them as data, and every derived value is flagged so the client can always separate harvested fact from inference. The result. 1,458 CSV datasets and 21,751,464 product rows covering all sixteen categories, from 2.6 million rows of casual wear through to a bridal and wedding guest section built from smaller designer catalogues. Two festive categories carry over half the corpus on their own, because festive and party wear catalogues run to tens of thousands of products per brand. Quality, measured rather than claimed. The client audit flagged field gaps in one category. On investigation the same gaps existed across the corpus and the same fix applied, so remediation ran corpus wide rather than on the flagged category alone. Every figure was measured on disk before and after the work, never estimated: roughly 3.0 million empty cells populated, 401 datasets improved, 264 storefront catalogues newly harvested contributing 779,868 products. A live page audit across 24 sources then recorded a verdict per field with the exact URL checked, and proved that two fields in the delivery schema are published by effectively no retailer in the market. Rather than leave a permanent unclosable gap, we reported that back with evidence and recommended reclassifying them. The safety rails that made it defensible. Merging enrichment into a corpus of more than twenty million rows is where this kind of work usually goes wrong. Every merge is blank fill only. Row counts are asserted non decreasing before each write, appends are de duplicated on identity then URL then name, and a full value set snapshot is verified after every pass. No delivered row was ever lost and no populated cell was ever replaced. When a single multi brand source contaminated one designer dataset, growing it from 80 rows to 15,522, an automated brand consistency audit caught it. The dataset was restored to its 80 genuine rows with the field enrichment already applied to them left intact, a brand match guard was added so no append can enter a dataset whose brand does not match, and all 132 datasets that had gained rows were re-audited to confirm it was the only case. What the client holds now. Not a one off export, but a refreshable market dataset with a stable contract, a documented source registry, and a run history that says exactly where every row came from.

Key Features

• 241 source JSON registry with per source engine, country and market segment
• 58 extraction engines covering hosted storefronts, enterprise commerce clouds, marketplace APIs, open source carts and headless applications
• Four tier extraction strategy from structured endpoints through to full browser rendering
• IP rotation across a managed proxy pool with header and user agent rotation
• Session persistence, cookie continuity and transport level fingerprinting
• Per host request pacing, concurrency limits and exponential backoff
• Robots and sitemap aware traversal with canonical URL resolution
• Pagination and infinite scroll traversal with network response interception
• One 27 field output contract shared by every engine
• Automatic tagging into the sixteen market categories
• Container profiles for per category harvests with a pinned browser build
• Incremental append and mid catalogue resume for multi hour runs
• Blank fill merge engine that never replaces or removes existing data
• Brand consistency guard preventing cross brand contamination on append
• Per run summaries recording every source excluded from delivery
• Corpus wide field completeness auditing with before and after evidence files

Project Gallery