10.9-bob: Monitoring va logging — Prometheus, Grafana, Sentry

  OBSERVABILITY (kuzatuvchanlik) — tizim ICHIDAGI holatni
  TASHQI signallardan tushunish (qutini ochmasdan bilish):

  3 USTUN (three pillars of observability):

  1. METRICS (metrikalar) — RAQAMLI vaqt qatori (time-series):
     "so'ngi 1 daqiqada 1200 so'rov, p99 latency 240ms, RAM 60%"
      ARZON, ixcham, trend ko'rsatadi (NIMA bo'lyapti?)

  2. LOGS (loglar) — diskret HODISALAR yozuvi (matn/JSON):
     "2026-06-22 10:03 ERROR user 42 login failed: bad password"
      BATAFSIL, kontekst (NEGA bo'ldi?)

  3. TRACES (izlar) — bitta so'rovning XIZMATLAR bo'ylab yo'li:
     "so'rov  API Gateway (5ms)  Auth (20ms)  DB (180ms)  sekin!"
      BOG'LIQLIK, qaysi bo'lak sekin (QAYERDA bo'ldi?)

   Monitoring = "men nimani kuzatishni BILAMAN" (oldindan)
     Observability = "BILMAGAN savolga ham javob topa olaman"

  MONITORING — tizim holatini DOIMIY o'lchash + og'ish bo'lsa OGOHLANTIRISH:

  NEGA KERAK (production haqiqati):
   Muammoni FOYDALANUVCHIDAN OLDIN bilish (alert, sezgi emas)
   Trend ko'rish (RAM asta-sekin o'syaptimi? — memory leak)
   Tezroq tuzatish (qachon, qayerda buzildi — log/trace)
   Sig'im rejalashtirish (qachon server qo'shish kerak — capacity)
   SLO/SLA bajarish (99.9% uptime — biznes va'dasi)

  MTTR (Mean Time To Recovery — tiklash o'rtacha vaqti):
  - Monitoringsiz: foydalanuvchi shikoyat qiladi  1 soat qidirasan
  - Monitoring bilan: alert keladi  log/trace  5 daqiqada tuzatasan

   Maqsad — "ma'lumot yig'ish" emas, MTTR'ni kamaytirish + ishonchlilik

  METRIC — vaqt bo'yicha o'lchanadigan RAQAM (time-series: vaqt + qiymat):

  1. COUNTER (hisoblagich) — faqat O'SADI (nol'ga reset bo'ladi restart'da):
     http_requests_total, errors_total, jobs_completed_total
      "jami nechta?" (so'rov, xato, tugagan vazifa)
      qiymatning O'ZI emas, o'sish TEZLIGI muhim  rate()

  2. GAUGE (o'lchagich) — UP/DOWN (ixtiyoriy o'zgaradi):
     memory_usage_bytes, active_connections, queue_size, temperature
      "hozir nechta?" (joriy qiymat — RAM, ulanish soni)

  3. HISTOGRAM (taqsimot) — qiymatlarni BUCKET'larga sanaydi:
     http_request_duration: <0.1s, <0.5s, <1s, <5s ... + sum + count
      "taqsimot qanday?"  p50/p95/p99 (server tomonda hisoblanadi)

  4. SUMMARY (xulosa) — quantile'ni MIJOZ tomonda hisoblaydi (sliding window):
     histogram'ga o'xshash, lekin quantile client'da (agregatsiya qiyin)
      ko'pincha HISTOGRAM afzal (Prometheus tomonda quantile)

   p99 latency = "so'rovlarning 99% shu vaqtdan TEZ" (eng muhim ko'rsatkich)

  PROMETHEUS — ochiq kodli metrika tizimi (CNCF, 2012, sanoat standarti):

  PULL MODEL (tortib olish — push EMAS):
  Prometheus HAR X soniyada ilovaga GET /metrics yuboradi (scrape):

  ┌──────────────┐   GET /metrics    ┌──────────────────────┐
  │  Ilova       │ ◄──── 15s ─────── │  PROMETHEUS SERVER   │
  │  /metrics    │ ──── matn ──────► │  - scrape (yig'ish)  │
  │ (prom-client)│   javob qaytaradi │  - TSDB (saqlash)    │
  └──────────────┘                   │  - PromQL (so'rov)   │
                                     │  - alert qoidalar    │
  ┌──────────────┐                   └──────┬───────────────┘
  │ EXPORTER     │ ◄──── scrape ─────────────┘
  │ (DB/Node/OS) │   (ilova o'zi metrika bermasa — exporter)
  └──────────────┘

  KOMPONENTLAR:
  - SCRAPE — Prometheus targetlardan metrika TORTIB oladi (interval)
  - EXPORTER — metrika bermaydigan tizim uchun "tarjimon" (node_exporter, postgres_exporter)
  - TSDB — Time-Series DataBase (metrikani vaqt bo'yicha saqlaydi)
  - PromQL — Prometheus Query Language (metrikadan so'rov: rate, sum, avg)

   PULL afzalligi: Prometheus target RO'YXATini biladi  "target down" ni
     o'zi sezadi (push'da yiqilgan ilova xabar bermaydi — bilmay qolasan)

  GRAFANA — metrikani CHIROYLI ko'rsatuvchi vizualizatsiya platformasi:
  (Prometheus raqam saqlaydi; Grafana uni grafik/dashboard qiladi)

  ASOSIY TUSHUNCHALAR:
  - DATASOURCE (manba) — qayerdan ma'lumot (Prometheus, Loki, PostgreSQL, ...)
                          170+ manba qo'llab-quvvatlaydi (2026)
  - DASHBOARD — bir necha PANEL to'plami (bitta ekranda umumiy ko'rinish)
  - PANEL — bitta vizualizatsiya (grafik, gauge, jadval, stat, heatmap)
  - QUERY — panel ichidagi PromQL so'rov (qaysi metrikani ko'rsatish)
  - ALERTING — shart buzilsa OGOHLANTIRISH (panelga bog'lanadi)

  ┌─────────────────── DASHBOARD: "API holati" ──────────────────┐
  │ ┌─ Panel: RPS ──┐ ┌─ Panel: p99 latency ┐ ┌─ Panel: Errors ┐│
  │ │   1200/s ▲    │ │  240ms  ╱╲___╱╲      │ │  0.2% ▁▁▃▁     ││
  │ └───────────────┘ └─────────────────────┘ └────────────────┘│
  │ ┌─ Panel: CPU/RAM ───────────────┐ ┌─ Panel: DB so'rovlar ─┐│
  │ │  CPU 45%  RAM 60%  ╱╲╱╲╱       │ │  ...                  ││
  │ └────────────────────────────────┘ └───────────────────────┘│
  └──────────────────────────────────────────────────────────────┘

   Grafana ma'lumot SAQLAMAYDI — faqat datasource'dan O'QIYDI va ko'rsatadi

  SENTRY — XATOLARNI real vaqtda tutuvchi tizim (error/exception tracking):
  (Prometheus metrika; Sentry — XATO tafsiloti: stack trace, kontekst)

  NIMA TUTADI:
  - Exception (tutilmagan xato), unhandled rejection, throw
  - Har xato: STACK TRACE (qaysi qator), so'rov ma'lumoti, user, env

  ASOSIY TUSHUNCHALAR:
  - ISSUE — bir XIL xatolar guruhi (10000 marta bo'lsa — 1 issue, son bilan)
  - STACK TRACE — xato qaysi fayl/qatorda yuz berdi (manba bilan)
  - RELEASE — qaysi versiyada paydo bo'ldi (v1.2.3 — deploy bog'lash)
  - CONTEXT — qo'shimcha: user (kim), tags, breadcrumbs (oldingi qadamlar)
  - ENVIRONMENT — production / staging / development

  ┌────────────────────────────────────────────────────┐
  │  TypeError: Cannot read 'id' of undefined         │
  │   UserService.findOne (user.service.ts:42)         │
  │   Seen 1,204 times · 89 users · release v1.2.3     │
  │   First seen: 2h ago · Last: 10s ago               │
  └────────────────────────────────────────────────────┘

   Log'da xato KO'MILIB ketadi (millionlab qator); Sentry — GURUHLAYDI,
     ustuvorlashtiradi, alert beradi (qaysi xato ko'p, kimga ta'sir qildi)

  STRUCTURED LOGGING — log'ni MATN emas, JSON sifatida yozish (5.12 davomi):

   Oddiy matn (parse qilish qiyin, qidirish og'ir):
  "User 42 logged in from 1.2.3.4 at 10:03"

   Structured (JSON — mashina o'qiydi, filter/qidirish oson):
  {"level":"info","time":"2026-06-22T10:03:00Z","msg":"login",
   "userId":42,"ip":"1.2.3.4","requestId":"abc-123","traceId":"xyz"}

  NEGA JSON:
   Filter: level=error AND userId=42 (aniq qidirish)
   Agregatsiya: Loki/ELK avtomatik tahlil qiladi (matn'da qiyin)
   Bog'lash: requestId/traceId  log'ni trace bilan ulash 2.10-bob

  PINO vs WINSTON 5.12-bob:
  - Pino — TEZ (minimal overhead, JSON default)  production tavsiya
  - Winston — moslashuvchan (ko'p transport, format)

  LOG DARAJALARI: trace < debug < info < warn < error < fatal
   production: info+ (debug o'chiq — shovqin/narx kamaytirish)

  LOG AGREGATSIYA — ko'p server/podning loglarini BIR JOYGA yig'ish + qidirish:
  (10 ta pod — 10 ta log fayli emas; markazlashgan, qidiriladigan)

  GRAFANA LOKI (yengil, Prometheus uslubi):
  - Log MAZMUNINI indekslamaydi  faqat LABEL'larni (app, level, pod)
  - Arzon (object storage'da saqlaydi), Grafana'da ko'riladi
  - Promtail/agent loglarni yig'ib Loki'ga yuboradi
  - "Prometheus loglar uchun"  bir xil ekotizim (label, LogQL)

  ELK / Elastic Stack (kuchli, og'irroq):
  - Elasticsearch (saqlash + HAR token indekslanadi  tez full-text qidiruv)
  - Logstash/Beats (yig'ish) + Kibana (ko'rish)
  - Kuchli qidiruv, lekin resurs ko'p (JVM, shard, disk)

  ┌─────────┐         ┌─────────┐
  │ Loki    │ arzon,  │ ELK     │ kuchli qidiruv,
  │ label   │ yengil  │ full    │ og'ir/qimmat
  │ indeks  │         │ indeks  │
  └─────────┘         └─────────┘
   2026 amaliyot: Loki (ilova loglari — ko'p, arzon) + ELK (audit/security)

  ALERTING — shart buzilganda AVTOMATIK xabar (sen qaramasang ham):

  PROMETHEUS ALERT QOIDASI (PromQL shart + davomiylik):
  - expr: rate(errors_total[5m]) > 0.05    xato 5%'dan oshsa
  - for: 5m                                 5 daqiqa DAVOM etsa (lop emas)
   firing (yonadi)  ALERTMANAGER'ga yuboriladi

  ALERTMANAGER — alertlarni boshqaradi (Prometheus'ning alert qismi):
  - GROUPING — o'xshash alertlarni guruhlaydi (10 pod yiqilsa — 1 xabar)
  - ROUTING — kimga (Slack? Telegram? PagerDuty? email?)
  - SILENCING — vaqtincha o'chirish (deploy/texnik ish paytida)
  - INHIBITION — bog'liq alertni bostirish (server o'lsa — uning 100 alertini emas)

  ALERT SEVIYALARI:
  - WARNING — diqqat (p99 oshyapti — keyin qara)
  - CRITICAL — darrov (server down — uyqudan tur, PagerDuty qo'ng'iroq)

   ALERT NOISE (shovqin) eng katta dushman: ko'p/yolg'on alert  "alert fatigue"
      odam e'tibor bermay qo'yadi  haqiqiy alertni o'tkazib yuboradi

  NIMANI O'LCHASH KERAK? — tasodifiy emas, METODLAR bor:

  GOLDEN SIGNALS (Google SRE — 4 ta asosiy):
  - LATENCY    — so'rov qancha vaqt oldi (p50/p95/p99)
  - TRAFFIC    — qancha so'rov (RPS — request per second)
  - ERRORS     — necha % xato (error rate)
  - SATURATION — resurs qanchalik to'la (CPU/RAM/disk — "to'yinganlik")

  RED METODI (servis/so'rov uchun — Golden'ning soddasi):
  - Rate (so'rov tezligi) · Errors (xato) · Duration (davomiylik)

  USE METODI (resurs uchun — server/disk/CPU):
  - Utilization (band %) · Saturation (navbat) · Errors (xato)

  DISTRIBUTED TRACING — so'rovning XIZMATLAR bo'ylab yo'lini kuzatish:
  - trace_id butun so'rovga; har bo'lak — SPAN (qaysi xizmat, qancha vaqt)
  - OPENTELEMETRY (OTel) — vendor'siz standart (trace/metric/log yig'ish)
  - JAEGER / Tempo — trace'larni saqlash va ko'rish

   Boshla: GOLDEN SIGNALS (latency, traffic, errors, saturation) — yetadi

  HEALTH CHECK — "tirikmi?" so'rovi (oddiy endpoint, alohida):
  - GET /health   200 OK {"status":"ok"} (ilova javob beradimi)
  - GET /ready    bog'liqliklar tayyormi (DB, Redis ulanganmi)
   Kubernetes 10.8-bob liveness/readiness probe shuni o'qiydi (yiqilsa qayta yoqadi)
   Uptime monitor (UptimeRobot) tashqaridan har 1 daqiqa tekshiradi

  SLI / SLO / SLA (ishonchlilik shartnomasi — 9.6 ishonchlilik):

  SLI (Indicator)  — O'LCHANADIGAN ko'rsatkich:
                     "checkout p99 latency" yoki "muvaffaqiyatli so'rov %"
  SLO (Objective)  — MAQSAD (ichki): "99.9% so'rov < 300ms, 30 kun ichida"
  SLA (Agreement)  — SHARTNOMA (mijoz bilan): "99.9% uptime, buzilsa — jarima"

  ERROR BUDGET (xato byudjeti): 99.9% SLO  oyiga ~43 daqiqa "ruxsat etilgan" downtime
   byudjet bor: yangi feature deploy; byudjet tugadi: faqat barqarorlik

   SLI o'lchanadi (metric) · SLO — ichki maqsad · SLA — tashqi va'da (jarima)

  OBSERVABILITY STACK (hammasi birga):

  ┌──────────┐   metrika    ┌────────────┐  so'rov  ┌─────────┐
  │  ILOVA   │ ──/metrics──►│ PROMETHEUS │◄─PromQL──│ GRAFANA │
  │ (NestJS) │              │  (TSDB)    │          │(dashboard│
  │          │   loglar     ├────────────┤          │ + alert) │
  │ prom-    │ ──Promtail──►│   LOKI     │◄─LogQL───│         │
  │ client   │              └────────────┘          └────┬────┘
  │ + Sentry │   xatolar                                  │ alert
  │ + Pino   │ ──────────►  ┌────────────┐          ┌────▼────┐
  └──────────┘   exception  │   SENTRY   │──alert──►│ Telegram│
                            │(error track)│         │/Slack   │
                            └────────────┘          └─────────┘

  HAR BIRI O'Z ISHIDA:
  - Prometheus  METRIKA (raqam, trend, alert)
  - Grafana     KO'RSATISH (dashboard) + ALERT (Alertmanager)
  - Loki        LOG (qidirish, kontekst)
  - Sentry      XATO (stack trace, guruh, release)
  - Pino        structured log ishlab chiqaradi (Loki o'qiydi)

   Bittasi yetmaydi — birga to'liq ko'rinish (metric+log+trace+error)

   Golden signals'dan boshla (latency, traffic, errors, saturation — 2.10)
   /metrics + prom-client (har xizmatda — 2.4); /health endpoint 2.11-bob
   Structured JSON log (Pino) + requestId/traceId (bog'lash — 2.7)
   Kam, MAZMUNLI alert (noise emas — actionable: chora bor — 2.9)
   Alert davomiylik (for: 5m — lahzalik tebranishga emas — 2.9)
   p99/p95 kuzat (o'rtacha emas — "dum"ni yashiradi — 2.3)
   Sentry'da PII'ni TOZALA (parol/token yubormagin — 6-xato)
   Metric cardinality nazorat (label cheklangan — userId LABEL emas — 6-xato)
   Dashboard'ni Git'da saqla (JSON — IaC, qayta tiklanadi — 10.10)
   SLO belgila + error budget (ishonchlilik o'lchovi — 2.11, 9.6)
   Production'da debug log o'chiq (info+ — narx/shovqin — 2.7)

PROM-CLIENT 2.4-bob: new client.Counter({name,help,labelNames}) | .inc() | .observe()
                   client.register.metrics()  /metrics endpoint matni
METRIC TURI 2.3-bob: Counter (.inc) | Gauge (.set/.inc/.dec) | Histogram (.observe)

PROMQL 2.4-bob: rate(http_requests_total[5m])             so'rov tezligi (RPS)
              sum by (status) (rate(...[5m]))            status bo'yicha guruh
              histogram_quantile(0.99, rate(..._bucket[5m]))   p99 latency
              up == 0                                     target yiqilgan
              rate(errors_total[5m])/rate(requests_total[5m])   error rate

PROMETHEUS CONFIG 2.4-bob: scrape_configs: job_name + targets + scrape_interval
ALERT QOIDA 2.9-bob:       alert + expr (PromQL) + for (davomiylik) + labels

SENTRY 2.6-bob: Sentry.init({dsn, tracesSampleRate, environment, release})
              Sentry.captureException(err) | Sentry.setUser({id})

PINO 2.7-bob: pino({level:'info'}) | logger.info({userId}, 'msg') | .error(err)
HEALTH 2.11-bob: GET /health  200 {status:'ok'} | /ready  DB tayyormi

// metrics.ts — prom-client bilan asosiy metrikalar (npm i prom-client)
import client from "prom-client";

// Default metrikalar (process CPU, RAM, event loop — avtomatik — 5.1)
const register = new client.Registry();
client.collectDefaultMetrics({ register });          // node_* metrikalar

// COUNTER — jami HTTP so'rovlar (faqat o'sadi — 2.3)
export const httpRequestsTotal = new client.Counter({
  name: "http_requests_total",                       // metrika nomi (konvensiya: _total)
  help: "Jami HTTP so'rovlar soni",                  // tavsif (majburiy)
  labelNames: ["method", "route", "status"],         // o'lchamlar (filtrlash uchun)
  registers: [register],
});

// HISTOGRAM — so'rov davomiyligi (p50/p95/p99 hisoblash uchun — 2.3)
export const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP so'rov davomiyligi (soniya)",
  labelNames: ["method", "route", "status"],
  buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 3, 5],     // bucket chegaralari (soniya)
  registers: [register],
});

export { register };
//  buckets — taqsimot chegaralari; histogram_quantile shulardan p99 hisoblaydi (2.3)

// instrumentation.ts — har so'rovni o'lchaydigan middleware (5.6 middleware)
import { Request, Response, NextFunction } from "express";
import { httpRequestsTotal, httpDuration, register } from "./metrics";

export function metricsMiddleware(req: Request, res: Response, next: NextFunction) {
  const end = httpDuration.startTimer();             // taymerni boshlash (histogram)
  res.on("finish", () => {                           // javob tugagach o'lchaymiz
    const route = req.route?.path ?? "unknown";      // route shabloni (/users/:id — 6-xato!)
    const labels = { method: req.method, route, status: String(res.statusCode) };
    httpRequestsTotal.inc(labels);                   // counter +1 (2.3)
    end(labels);                                     // histogram'ga davomiylik yoziladi
  });
  next();
}

// /metrics endpoint — Prometheus SHU YERNI scrape qiladi (2.4)
export async function metricsHandler(_req: Request, res: Response) {
  res.set("Content-Type", register.contentType);     // text/plain; prometheus format
  res.end(await register.metrics());                 // barcha metrikalar matni
}
//  route'ni :id bilan (real ID emas) — aks holda cardinality portlaydi (6-xato)

# prometheus.yml — Prometheus serverning asosiy sozlamasi
global:
  scrape_interval: 15s              # har 15 soniyada scrape (default — 2.4)
  evaluation_interval: 15s          # alert qoidalarini har 15s tekshir

# Alert qoidalari va Alertmanager (2.9)
rule_files:
  - "alert_rules.yml"               # alert qoidalari (Misol 6)
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  # 1-job: Prometheus o'zini ham kuzatadi
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  # 2-job: bizning NestJS ilova (/metrics — Misol 2)
  - job_name: "nestjs-api"
    metrics_path: "/metrics"        # qayerdan scrape (default /metrics)
    static_configs:
      - targets: ["api:3000"]       # Docker network nomi (10.4)

  # 3-job: server resurslari (node_exporter — CPU/RAM/disk — 2.4)
  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]
#  Kubernetes'da 10.8-bob static emas, service discovery ishlatiladi (avtomatik)

# === GOLDEN SIGNALS — PromQL so'rovlar 2.10-bob ===

# 1. TRAFFIC — so'rov tezligi (RPS — request per second):
rate(http_requests_total[5m])
# rate() — counter'ning 5 daqiqadagi o'rtacha o'sish tezligi (counter'ga DOIM rate)

# 2. ERRORS — xato foizi (5xx javoblar nisbati):
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))
# status=~"5.." — regex (500-599); xato / jami = error rate 2.10-bob

# 3. LATENCY — p99 (so'rovlarning 99% shu vaqtdan tez — 2.3):
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# histogram'ning _bucket'idan quantile hisoblaydi (le = "less than or equal")

# 4. SATURATION — RAM ishlatilishi (foiz):
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# 5. TARGET DOWN — qaysi ilova yiqilgan (pull model afzalligi — 2.4):
up == 0
# up — Prometheus avtomatik metrikasi (1 = scrape OK, 0 = target o'lik)

{
  "title": "API Golden Signals",
  "panels": [
    {
      "title": "So'rov tezligi (RPS)",
      "type": "timeseries",
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "targets": [
        { "expr": "sum(rate(http_requests_total[5m]))", "legendFormat": "RPS" }
      ]
    },
    {
      "title": "p99 latency",
      "type": "timeseries",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
          "legendFormat": "p99"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "s", "thresholds": {
        "steps": [{ "color": "green", "value": 0 }, { "color": "red", "value": 0.5 }]
      } } }
    },
    {
      "title": "Error rate %",
      "type": "stat",
      "targets": [{ "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100" }]
    }
  ]
}

# alert_rules.yml — Prometheus alert qoidalari (prometheus.yml ulaydi — Misol 3)
groups:
  - name: api_alerts
    rules:
      # 1. Xato foizi 5%'dan oshsa (critical)
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m                            # 5 daqiqa DAVOM etsa (lahzalik emas — 2.9)
        labels:
          severity: critical              # routing uchun (Alertmanager — 2.9)
        annotations:
          summary: "Xato foizi yuqori ({{ $value | humanizePercentage }})"
          description: "API 5xx xatolari 5 daqiqada 5%'dan oshdi"

      # 2. p99 latency sekin (warning)
      - alert: SlowResponses
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.5
        for: 10m
        labels: { severity: warning }
        annotations: { summary: "p99 latency 500ms'dan oshdi" }

      # 3. Ilova yiqilgan (critical — pull model — 2.4)
      - alert: TargetDown
        expr: up == 0
        for: 1m
        labels: { severity: critical }
        annotations: { summary: "{{ $labels.job }} target javob bermayapti" }
#  for — eng muhim: b'siz har lahzalik tebranish alert beradi (noise — 2.9)

// instrument.ts — LOYIHA ILDIZIDA, eng birinchi import (rasmiy talab — 2.6)
import * as Sentry from "@sentry/nestjs";

Sentry.init({
  dsn: process.env.SENTRY_DSN,               // loyiha kaliti (.env — 10.11, secret)
  environment: process.env.NODE_ENV,         // production/staging (filtr — 2.6)
  release: process.env.npm_package_version,  // versiya — qaysi deploy'da xato (2.6)
  tracesSampleRate: 0.1,                      // tranzaksiyalarning 10% (production — narx)
  // PII himoyasi — parol/token'ni TOZALA (6-xato, GDPR — 14.x)
  beforeSend(event) {
    if (event.request?.data) delete (event.request.data as any).password;
    return event;
  },
});

// main.ts — instrument BOSHQA importlardan OLDIN (2.6 — kritik!)
import "./instrument";                       //  eng birinchi qator
import { NestFactory } from "@nestjs/core";
import { AppModule } from "./app.module";

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}
bootstrap();

// app.module.ts — global filter (tutilmagan xatolarni Sentry'ga — 8.6 filter)
import { Module } from "@nestjs/common";
import { APP_FILTER } from "@nestjs/core";
import { SentryModule } from "@sentry/nestjs/setup";
import { SentryGlobalFilter } from "@sentry/nestjs/setup";

@Module({
  imports: [SentryModule.forRoot()],
  providers: [{ provide: APP_FILTER, useClass: SentryGlobalFilter }],
})
export class AppModule {}
// Qo'lda tutish: Sentry.captureException(err) + Sentry.setUser({ id: userId })

// logger.ts — Pino structured JSON logger (5.12 davomi, tez)
import pino from "pino";

export const logger = pino({
  level: process.env.LOG_LEVEL ?? "info",    // production: info (debug o'chiq — 2.7)
  // Production: toza JSON (Loki o'qiydi — 2.8); dev: chiroyli rangli
  transport: process.env.NODE_ENV !== "production"
    ? { target: "pino-pretty" }              // dev: o'qiladigan format
    : undefined,                             // prod: JSON (mashina o'qiydi)
  formatters: { level: (label) => ({ level: label }) },  // "level":"info" (raqam emas)
  redact: ["req.headers.authorization", "password", "*.token"],  // PII tozalash (2.13)
});

// Ishlatish — har log'ga KONTEKST (requestId/traceId — bog'lash — 2.7)
logger.info({ userId: 42, requestId: "abc-123" }, "Foydalanuvchi kirdi");
logger.error({ err, orderId: 7 }, "Buyurtma yaratishda xato");
//  Birinchi argument — obyekt (struktura); ikkinchi — xabar (msg)
// Natija: {"level":"info","time":...,"userId":42,"requestId":"abc-123","msg":"..."}

// health.controller.ts — @nestjs/terminus bilan (DB/Redis tekshiradi)
import { Controller, Get } from "@nestjs/common";
import { HealthCheck, HealthCheckService, TypeOrmHealthIndicator } from "@nestjs/terminus";

@Controller("health")
export class HealthController {
  constructor(
    private health: HealthCheckService,
    private db: TypeOrmHealthIndicator,
  ) {}

  // GET /health — liveness: ilova tirikmi? (Kubernetes probe — 10.8)
  @Get()
  @HealthCheck()
  check() {
    return this.health.check([
      () => this.db.pingCheck("database"),   // DB ulanish ishlayaptimi (2.11)
    ]);
    //  200 {"status":"ok","info":{"database":{"status":"up"}}}
    //  biror bog'liqlik o'lsa: 503 (Kubernetes podni qayta yoqadi — 10.8)
  }
}
//  /health (tirikmi) + /ready (bog'liqliklar tayyormi) — K8s ikkalasini o'qiydi (2.11)

# docker-compose.monitoring.yml — to'liq monitoring stack (10.4 davomi)
services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml   # config (Misol 3)
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml # alertlar (Misol 6)
      - prom_data:/prometheus                              # TSDB saqlash (doimiy)

  grafana:
    image: grafana/grafana:latest
    ports: ["3001:3000"]                                   # 3000 ilovada band
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}     # admin parol (.env — 10.11)
    volumes:
      - grafana_data:/var/lib/grafana                      # dashboard saqlash
    depends_on: [prometheus]                               # Prometheus avval (10.4)

  node-exporter:
    image: prom/node-exporter:latest                       # server resurs metrikasi (2.4)
    ports: ["9100:9100"]

  alertmanager:
    image: prom/alertmanager:latest                        # alert boshqaruvi (2.9)
    ports: ["9093:9093"]
    volumes: ["./alertmanager.yml:/etc/alertmanager/alertmanager.yml"]

volumes: { prom_data: {}, grafana_data: {} }               # doimiy saqlash (10.3)
#  docker compose up  Prometheus:9090, Grafana:3001 (Prometheus'ni datasource qo'sh)

 Tasodifiy metrika ("har narsani o'lchayman" — shovqin, narx)
 Golden signals (latency, traffic, errors, saturation — 2.10)

 O'rtacha (average) latency (sekin "dum"ni yashiradi — 2.3)
 p95/p99 (foydalanuvchilarning eng sekin tajribasini ko'rsatadi)

 console.log("User " + id + " login") (matn — qidirish/filter qiyin)
 Structured JSON (Pino) + requestId/traceId (filter/bog'lash — 2.7)

 Ko'p/yolg'on alert (alert fatigue  e'tibor bermay qo'yiladi — 2.9)
 Kam, mazmunli, actionable + for: 5m (chora bor — davomiy — 2.9)

 labelNames: ["userId", "requestId"] (cheksiz qiymat — cardinality portlaydi)
 Cheklangan label (method, route shabloni, status — 6-xato)

 Xato bilan parol/token/karta Sentry'ga ketadi (PII leak — 6-xato)
 beforeSend / redact bilan tozalash (GDPR — 2.13)