Rank Tracker — SaaS website rank tracker for Yandex and Google

Pet project: a full SaaS service for monitoring website search rankings — from idea and specs to CI/CD and production deployment.

Stack in one line: TypeScript · NestJS 11 · Prisma 7 · PostgreSQL 16 · Redis 8 · BullMQ · Next.js 16 (App Router, React 19) · Tailwind CSS 4 · shadcn/Base UI · TanStack Query 5 · Zod 4 · Turborepo · Docker · GitHub Actions → GHCR · Dokploy

---

Idea

An SEO specialist or website owner needs to know where their site ranks in search results for important queries — and how those rankings change over time. Rank Tracker solves this:

• the user creates a project (a website domain) and adds keywords — specifying the search engine (Yandex / Google), region, and device type (desktop / mobile / tablet);
• the system on a schedule (or on manual trigger) queries the search results via a SERP API (xmlstock.com), finds the domain's position, and saves it to history;
• the dashboard shows a pivot table of position history (dates as columns, most recent on the left), where a top-3 placement is highlighted gold, improvement green, and decline red, plus summary cards: total keywords, average position, number in top-3.

The project is designed as a SaaS: FREE / PAID plans, daily check quotas (FREE — 20/day), abuse protection, admin tools for managing plans.

Features

• Authentication: registration with email confirmation, JWT login (15-min access + 30-day refresh with rotation) in httpOnly cookies; Cloudflare Turnstile and a per-IP registration limit.
• Projects: CRUD, domain unique per user, configurable auto-check interval.
• Keywords: bulk add by pasting text or importing CSV/XLSX (up to 2 MB, auto-detection of broken win1251 encoding); cascading selection: search engine → region (filtered by engine) → device; deduplication at the DB level.
• Position checks: daily cron + manual trigger (all keywords or a selection); Yandex — top 100, Google — top 50; per-user daily quotas.
• Dashboard: 30-day position history pivot table with color-coded trends, stat cards, Yandex/Google switch (state kept in the URL).

Architecture

A pnpm + Turborepo monorepo with three workspaces: apps/api (NestJS), apps/web (Next.js), and packages/shared — a package of Zod schemas that keeps API contracts consistent between the frontend and backend.

⤢

Position check pipeline

The core of the project is an asynchronous rank-collection pipeline built on BullMQ:

⤢

Step by step:

1. Trigger — daily cron or manual run from the UI.
2. Selection — the scheduler enqueues only keywords whose last check is older than the project's checkInterval.
3. Quota — an atomic Redis Lua script reserves slots against the user's daily limit. Two modes: reserveExact (all-or-nothing for manual checks, otherwise a 429 with used/limit details) and reserveUpTo (best-effort for the nightly cron — takes whatever is left).
4. Queue — jobs get 3 attempts with exponential backoff; only errors flagged as retryable are retried.
5. Fetch — a shared token bucket limits calls to the paid API to 20 req/s; Yandex returns top-100 in a single request thanks to deep grouping, while Google caps at 10 results per page — the service walks up to 5 pages with early exit and handles xmlstock's quirk: code 110 ("request queued") → retry the same request up to 4 times with a 7s pause.
6. Parsing and saving — defensive XML parsing without any, domain normalization (protocol/www/port/path), writing a PositionCheck record (null = domain not in the top results).

Data model

⤢

Technology stack

Layer	Technologies
Backend	NestJS 11, Prisma 7 (@prisma/adapter-pg, multi-file schema), PostgreSQL 16, Redis 8 + ioredis, BullMQ, passport-jwt, argon2, nodemailer, fast-xml-parser, Joi (env) + Zod/nestjs-zod (DTO)
Frontend	Next.js 16 (App Router, RSC, standalone), React 19, TypeScript, Tailwind CSS 4, shadcn on Base UI, TanStack Query 5, Cloudflare Turnstile
Shared	@rank-tracker/shared — Zod 4 schemas and API contracts (single source of truth shared by both apps)
Import	csv-parse, SheetJS (xlsx), iconv-lite
Infra	Turborepo, Docker multi-stage (turbo prune --docker, node:22-alpine, non-root), docker-compose (Postgres + Redis with ACL), GitHub Actions → GHCR, deployed on Dokploy
Tests	Jest 30, unit tests for the SERP parser against real XML fixtures, e2e via supertest

Engineering decisions

Atomic quotas on Redis Lua. A manual check and the nightly cron can race for the same user's daily limit. Instead of a racy GET+SET, a Lua script run via EVAL checks and reserves atomically; the key is tied to the date in the service's timezone (25h TTL). Overspending the limit is structurally impossible.

Single Zod contracts instead of tRPC. One schema in packages/shared produces: request validation in NestJS (createZodDto), TypeScript types, and client-side form validation. The frontend and backend physically cannot drift apart on data shape, while the API stays plain REST, usable by any client.

BFF proxy with transparent refresh. Tokens live only in httpOnly cookies — JS never sees them. Next.js route handlers proxy requests to NestJS; on a 401 the proxy itself calls /auth/refresh, mixes the refreshed set-cookie into the retried request, and forwards them to the browser. The user never notices the access token expiring.

Dual-engine SERP client that handles quirks. Yandex and Google behave differently through xmlstock: Yandex returns top-100 in a single request (deep grouping), while Google caps at 10 results per page, so pagination with early exit and retries for the vendor's "code 110" was implemented. Errors are typed (SerpFetchError with an isRetryable flag) — the queue's retry policy is driven by the nature of the error, not blanket retries.

Rate limiting at three levels, each doing its own job. Per-IP registration limit (Redis, 3/day — counts failed attempts too, to choke captcha brute-forcing), the BullMQ worker limiter (100 jobs/s), and an in-process token bucket for the paid SERP API (20 req/s, matching Yandex.XML's limits).

Import that survives real-world files. CSVs from Russian Excel often arrive as win1251: the parser detects broken decoding via U+FFFD characters and re-reads the file through iconv-lite; both ;/, delimiters and XLSX are supported. Parsed results go through the same Zod schema as manual input.

Security by default. argon2id for passwords, refresh token rotation, every data request scoped by userId at the service layer (WHERE {id, userId} — IDOR is architecturally impossible), Turnstile on registration, Redis's default user disabled in favor of an ACL user with restricted permissions.

A verifiable pipeline. Parser unit tests run against real recorded XML responses from both search engines; every pipeline step has a smoke script (test-xmlstock, test-queue, test-processor); an admin CLI (set-user-plan) is bundled into the production image and runs via tsx directly in the container.

Development process

• Spec-first: every pipeline increment (xmlstock client → Redis compose → queue → processor → HTTP trigger) started with a design doc with explicit Scope / Goals / Non-Goals (docs/superpowers/specs/).
• Schema evolution — 11 Prisma migrations: from init to dropping server-side sessions in favor of stateless JWT, cascading deletes, anti-abuse fields, and moving the search engine from the project level to the region level (supporting Google within the same projects).
• CI/CD: a lint + test gate → matrix build of API and Web Docker images → push to GHCR with layer caching and latest/sha/tag tags → deploy to Dokploy. The production API container applies migrations on startup itself (a deliberate single-replica assumption, documented in a comment).
• Feature branches via git worktree, merged through PRs.

Roadmap

• SearchProvider abstraction — plugging in DataForSeo / SerpApi as alternative SERP sources.
• Aggregation worker: visibility score, average-position trend, project-level gains/losses.
• Billing for the PAID plan.
• Position history charts alongside the pivot table.
• Per-word checking
• Data export to XLSX and CSV
• AI-generated summary reports over the table data for a selected period