JobSignals

JobSignals is a reusable, governed data product that ingests job postings from multiple sources — LinkedIn, Indeed, YC Jobs, and company career pages — and transforms raw listings into a structured, enriched, and queryable dataset. It surfaces hiring trends, skill demand signals, salary benchmarks, and company hiring velocity for a broad range of downstream consumers.

Core Outputs

The Core Problem

Job market data is fragmented, noisy, and inaccessible as a reusable asset. Today, any team that wants to answer a question like "Which ML skills are companies hiring for most in Q1 2026?" has to manually scrape or purchase data, spend weeks cleaning it, build bespoke pipelines, and produce an answer that goes stale within days.

This work is being done redundantly — by bootcamps building curriculum, by VCs doing diligence, by job seekers benchmarking salaries, and by HR teams setting headcount budgets. Each consumer solves the same underlying data problem independently, with no shared standard or infrastructure.

Consequences

• Wasted engineering effort: Every consumer rebuilds the same extraction and normalization logic
• Low data quality: One-off scrapes lack deduplication, standardized taxonomy, or historical depth
• Stale intelligence: Without continuous pipelines, job market signals decay within days
• No shared language: Skill names, job titles, and company identifiers are inconsistent across sources

Total Addressable Market

Why Now

• LLM-native tools have dramatically lowered the cost of parsing unstructured job descriptions into structured data
• The rise of Data Mesh architecture has increased organizational appetite for owned, governed data products
• Real-time labor market intelligence has become a strategic asset as hiring cycles compress and skills shift rapidly

Persona Map

Persona Deep Dives

This section defines JobSignals according to Data Mesh and data-as-a-product principles.

Product Identity

Data Product Contracts

Ownership Model

Source Registry

Source Tiering

Entity Relationship Overview

job_postings (raw → normalized)
    │
    ├── job_skills (extracted from description)
    ├── job_salaries (extracted / inferred)
    └── companies (resolved entity)
            │
            └── company_signals (aggregated per company)
                        │
                        └── skill_trends (time-series aggregations)

Core Table: job_postings

Core Table: job_skills

Derived Table: skill_trends

┌──────────────────────────────────────────────────────────────┐
│                      SOURCE LAYER                             │
│  LinkedIn  │  Indeed  │  YC Jobs  │  Career Pages  │  BLS    │
└──────────────────────┬───────────────────────────────────────┘
                        │ HTTP / API / RSS
                        ▼
┌──────────────────────────────────────────────────────────────┐
│                   INGESTION LAYER (Airflow)                    │
│  ├── Source adapters (Python, async, per-source)              │
│  ├── Rate limiting + retry with exponential backoff           │
│  └── Raw landing zone → GCS / S3 (Bronze / partitioned)      │
└──────────────────────┬───────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│              TRANSFORMATION LAYER (dbt on BigQuery)           │
│  ├── Deduplication (fingerprint + LSH near-duplicate)         │
│  ├── Normalization (title taxonomy, company resolution)       │
│  ├── Skill extraction (LLM pipeline + rule-based fallback)    │
│  ├── Salary parsing + currency normalization                  │
│  └── Aggregations → skill_trends, company_signals            │
└──────────────────────┬───────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────────────────────┐
│                    SERVING LAYER                              │
│  ├── Data Warehouse (BigQuery) — Analytics consumers          │
│  ├── Operational DB (PostgreSQL) — API backend                │
│  ├── Search Index (Elasticsearch) — Faceted search            │
│  └── Cache (Redis) — Hot path API responses                   │
└──────────────────────┬───────────────────────────────────────┘
                        │
             ┌──────────┴──────────┐
             ▼                     ▼
       REST / GraphQL          Dashboard
            API               (Metabase /
         (FastAPI)              Superset)

Pipeline Stage Summary

Deduplication Strategy

Job postings are duplicated across sources frequently. JobSignals uses multi-signal fingerprinting:

1. Exact match: (company_id, title_normalized, location_city, posted_at_date) — strongest signal
2. Near-duplicate: Locality-sensitive hashing (LSH) on description_cleaned — catches reposts with minor edits
3. Cross-source linking: If source URLs differ but fingerprint matches, merge into canonical record with an array of originating sources

Design Principles

• RESTful with resource-oriented URLs; GraphQL endpoint for flexible consumer queries (Phase 3)
• All responses return JSON with consistent envelope: { data, meta, errors }
• Pagination: cursor-based for large result sets, offset-based for fixed queries
• Authentication: API key (external) + service account JWT (internal)
• All timestamps in ISO 8601 UTC

Endpoint: GET /v1/jobs

// Search and filter job postings
// Query params: q, title_family, company_id, location, modality,
//   seniority, salary_min, posted_after, skills[], page_size, cursor

// Response 200
{
  "data": [{
    "job_id": "jb_01HXYZ...",
    "title_normalized": "Senior Data Engineer",
    "company": {
      "company_name": "Stripe",
      "industry": "Fintech",
      "company_stage": "public"
    },
    "location": { "city": "San Francisco", "modality": "hybrid" },
    "seniority_level": "senior",
    "salary": { "min": 180000, "max": 240000, "currency": "USD" },
    "skills": ["Apache Spark", "dbt", "Python", "Kafka"],
    "posted_at": "2026-03-10T14:22:00Z"
  }],
  "meta": { "total_count": 4820, "next_cursor": "eyJqb2Ii..." }
}

Endpoint: GET /v1/skills/trends

// Ranked skill demand over time
// Query params: title_family, country, window (7d|30d|90d|365d), limit, order_by

{
  "data": [{
    "skill_name": "dbt",
    "skill_category": "tool",
    "posting_count": 12480,
    "posting_share": 0.34,
    "mom_change": 0.08,
    "yoy_change": 0.31,
    "rank": 1,
    "rank_previous_period": 3
  }],
  "meta": {
    "window": "30d",
    "title_family": "Data Engineering",
    "total_postings_in_scope": 36708
  }
}

Endpoint: GET /v1/companies/{id}/signals

{
  "data": {
    "company_name": "Stripe",
    "window": "90d",
    "total_postings": 143,
    "active_postings": 58,
    "hiring_velocity_score": 82.4,
    "top_skills": ["Python", "Go", "Kubernetes", "PostgreSQL", "dbt"],
    "salary_benchmarks": { "median_min": 170000, "median_max": 230000 },
    "trend": [
      { "period": "2025-12-01", "postings": 38 },
      { "period": "2026-01-01", "postings": 52 },
      { "period": "2026-02-01", "postings": 53 }
    ]
  }
}

Endpoint: GET /v1/salaries/benchmark

{
  "data": {
    "title_family": "Data Engineering",
    "seniority": "senior",
    "country": "US",
    "percentile_10": 140000,
    "percentile_25": 165000,
    "percentile_50": 195000,
    "percentile_75": 230000,
    "percentile_90": 270000,
    "sample_size": 2840
  }
}

Error Codes

Feature Registry

Access Tiers

Performance

Availability & Reliability

Security

• All data in transit encrypted (TLS 1.3); at rest (AES-256)
• No PII stored — recruiter/poster identity stripped at ingestion
• RBAC for internal warehouse access: read_raw, read_curated, admin
• API keys rotatable; 90-day audit log of all key usage retained
• API layer is read-only; no write paths exposed publicly

Observability

Data Quality Dimensions

Schema Change Management

Supply-Side Metrics (Data Quality)

Demand-Side Metrics (Adoption)

Anti-Metrics

• Number of job postings ingested: Volume ≠ quality. More noise reduces product value.
• Dashboard pageviews: Vanity metric — does not reflect whether data is informing decisions.
• Number of data sources: More sources create maintenance burden without proportional signal gain.

MVP Inclusions

MVP Timeline

MVP Success Criteria

• ≥ 3 distinct internal teams making regular API queries by day 30 post-launch
• Pipeline success rate ≥ 98% in first 30 days
• Zero P0 data quality incidents (schema breaks, widespread nulls in required fields)
• Consumer NPS ≥ 30 in post-launch survey (n ≥ 10)

Visual Timeline

Document Owner: [Product Owner Name]  ·  Last Reviewed: 2026-03-13  ·  Next Review: 2026-04-13
Questions: #jobsignals-team (Slack)  ·  jobsignals-pm@[company].com