HOW IT WORKS · the product in one minute
Someone fakes ShopNova's site. BrandWatch finds it, proves it, takes it down.
Six steps, shown on this workspace's real data: ShopNova, its look-alikes, a cloaked kit and the attacker behind it. Press play, or scroll.
First, it learns what the real ShopNova looks like.
BrandWatch opens shopnova.com once and keeps a fingerprint: the favicon's hash, a picture of the homepage, its colours, its login form and the words a phisher would copy.
- AI reads the site and writes the Google searches an attacker would target.
Every suspect is later compared against this fingerprint, not against a guess.
mmh3 -2029202433Then it hunts for look-alikes — before anyone reports them.
Typo domains on 1,000+ endings, new HTTPS certificates as they are issued, yesterday's registrations and Google results — nothing has to be on a blocklist first.
- Social: Facebook pages, Instagram profiles and ads under the brand's name, read by an AI.
- AI assistants: ChatGPT and Gemini are asked what a customer asks; a phone or e-mail that is not on the brand's site becomes a finding.
Found before anyone reports it — the brand's own sites and big platforms are dropped first.
- shopnova-login.comAI answer (ChatGPT / Gemini / Google AI Mode) critical
- shopnova-refund.netFacebook pages / Instagram profiles critical
- shoqnova.comGoogle search critical
- shopnowa.comGoogle search critical
- shopn0va.comGoogle search critical
- shospnova.comLook-alike names watching
- sholpnova.comLook-alike names watching
- shapnova.comLook-alike names watching
It visits every site the way a victim would.
Many phishing kits hide from security scanners: a harmless page for datacenter IPs, the fake login only for real people. So BrandWatch also visits as a real person.
- A real Chrome: its User-Agent, Accept-Language, client hints, language and time zone all agree — no "HeadlessChrome" giveaway.
- A real address: through a residential or a mobile 4G / 5G proxy (or both) in the brand's country, as a desktop and / or a phone.
- A locked-down browser that can't reach internal systems; everything it sees is kept as evidence nobody can edit.
The score uses what a victim sees, not what the kit chose to show a scanner.


It scores the site — and shows its work.
50 signals in 7 capped groups, each worth fixed points: does the address use the brand, does the page look the same, does it ask for a password, how new is it, is it hiding.
- The same evidence always gives the same score; AI is a second opinion and never changes the number.
No black box: every point traces back to a piece of evidence.
Registered 4 days ago. Puts the ShopNova name in its address, copies ShopNova's favicon, homepage and login form and asks for a password. It hides from security scanners: the real page only appears to a normal visitor (home IP / browser).
A person decides. Then it maps the attacker.
Nothing is confirmed automatically: an analyst reads the evidence and presses Confirm. From that one site, BrandWatch follows what the attacker reused — server, certificate, favicon, tracking code, owner — to the rest of the operation.
- Neighbours that only share a cheap server are dropped as noise, with the reason written down.
One fake site leads to the whole operation, and every indicator exports to a SIEM as STIX 2.1.
Finally, it drafts the takedown.
A request backed by the evidence, addressed to the people who can pull the site: the registrar, the hosting provider and the CDN — contacts found automatically.
- 26 report portals (browsers, anti-virus, blocklists) so the site is blocked for everyone while the takedown is pending.
- A fake Facebook / Instagram ad or page gets Meta's report flow and a ready-to-paste report.
BrandWatch never sends anything by itself: a person reviews and sends, and every step lands in the audit log.
That's BrandWatch, in the six steps above. Every step is explainable, every piece of evidence is tamper-proof, and a person decides everything that matters.
About this project · numbers read from the code, not typed by hand
Stack: FastAPI · SQLAlchemy 2 · Alembic · Jinja2 · Playwright (Chromium) · httpx · dnspython · cryptography. Scoring version v7-f1ba3ebf. CI: lint · test-sqlite · test-postgres.
Go deeper
For the questions after the demo: every step with its exact APIs and limits, the presenter's script, the engineering, and the glossary every “?” on the site links to.
The pipeline, step by step· 13 stages in order — “see it” opens a real example, “More info” shows the exact APIs, limits and code
Fingerprint the brand's real site: favicon hash, screenshot hash, colours, the login and sign-up pages, keywords — and an AI-written list of the Google searches a phisher would target.
Learn the real site
- Calls: Headless Chromium, OpenAI
- Cost: free · local · paid per token · $0.15 / M in, $0.60 / M out
- Keeps: brand_fingerprints row, versioned (+1 per rebuild): hashes, screenshots, keywords, palette, AI profile
- Limit: ≤ 3 auth pages followed · ≤ 20 keywords · ≤ 20 app links
All the details ▾
External calls
the brand's primary domain + up to 3 linked login / sign-up pagespage.goto (DOMContentLoaded), viewport screenshot, favicon in a second routed tabapi.openai.comPOST /v1/chat/completions · gpt-4o-mini (OPENAI_MODEL) · temperature 0 · strict JSON schema brand_profileFallback chain
Computed locally
- favicon MurmurHash3 over the base64 bytes (the Shodan / Censys convention) + 64-bit pHash (numpy DCT)
- screenshot pHash + dHash; 6-colour palette by PIL median-cut on a 64×40 thumbnail
- login-form DOM signature: the token list of forms, inputs and buttons
- keywords = brand token + <title> / <h1> / <h2> words + the validated AI keywords
- official app-store listings and login / sign-up links found on the homepage
Stored
- brand_fingerprints row, versioned (+1 per rebuild): hashes, screenshots, keywords, palette, AI profile
- favicon and screenshots as content-addressed artifacts (SHA-256 file names) in the evidence store
Limits
- ≤ 3 auth pages followed · ≤ 20 keywords · ≤ 20 app links
- AI input truncated to 4,000 chars; output validated: ≤ 20 keywords, ≤ 15 search queries
- browser: navigation 15 s, settle 3 s, whole operation ≤ 45 s, viewport 1280×800
- reused on the next scan unless 'include fingerprinting' is ticked
Demo vs live
Code
Five streams look for look-alikes: typo and homoglyph permutations on 1,000+ TLDs (DNS-checked), new TLS certificates, yesterday's newly registered domains, Google results, and the urlscan archive — plus fake Facebook / Instagram accounts and ads, and what ChatGPT, Gemini and Google AI Mode tell users about the brand's contacts.
Find look-alikes
- Calls: DNS, crt.sh, CertSpotter, WhoisDS newly registered domains, SerpApi, ScraperAPI, urlscan.io search, SearchApi.io · Meta Ad Library
- Cost: free · free · 5xx retried 4× · free · rate-limited · free · paid · $0.01 / query · paid · about 25 credits (~$0.012) / query · free tier · 100 free · then ~$0.004 / request — only for brands whose products other shops resell
- Keeps: scan_stages 'discovery' row (hits, API calls) and one live event per stream
- Limit: streams per track: Light dnstwist + CT · Standard + urlscan · Deep all five
All the details ▾
External calls
13 root servers → TLD → authoritative, or 1.1.1.1 / 9.9.9.9 / 8.8.8.8dnspython NS / A query per generated namecrt.shGET /?q=%<brand>%&output=jsonapi.certspotter.comGET /v1/issuances?domain=<official>&include_subdomains=true&match_wildcards=truewww.whoisds.comGET the zip of yesterday's list (falls back 2–3 days)serpapi.comGET /search.json engine=google, q, gl, hl, num=20api.scraperapi.comGET /structured/google/search api_key, query, country_code, tld, hlurlscan.ioGET /api/v1/search/?q=page.url:"<key>" OR page.domain:*<key>*&size=50www.searchapi.ioGET /api/v1/search engine=meta_ad_library, q=<ad keyword>, search_type=keyword_exact_phrase, country, platforms=facebook,instagramFallback chain
Computed locally
- dnstwist as a library: permutations interleaved by technique (dictionary, TLD swap, typos … homoglyphs last)
- combosquats: 44 generic words + the brand's own keywords (3–12 letters), at most 60
- certificate / new-domain relevance: brand key in the host, or Levenshtein distance ≤ 2
- brand key shorter than 4 letters (on.com → “on”): the brand name is used instead (“oncloud”) when it starts with the label; otherwise the key must stand as its own word, typo permutations are skipped and the urlscan substring search is off
- Google results triaged before any visit: official · known platform · app listing · free hosting · brand + login words
- merge: shops advertised on Meta first, dnstwist last, so the cap keeps the rarer hits
- Meta ads: Facebook redirect links unwrapped, utm / fbclid stripped, one candidate per shop host, every ad kept for reporting
- Meta ads relevance: Meta matches words, not meaning (“on cloud” also finds cloud-storage ads), so an ad counts only when it names the brand or a product line, or says the brand's words within 60 characters of what the brand sells (shoes, sizes, נעליים …); unclear ads are read by the AI in one batch per query, or skipped without an AI
- shop checks only for pages about the brand: a shop that sells something else is skipped, so it can never become a “suspicious shop” for this brand
- fake social accounts: Facebook pages by name (SearchApi meta_ad_library_page_search, every country) + site:instagram.com / site:facebook.com Google results → only accounts about the brand, never the brand's own (brand profile + links on its website) → page info (created, renamed, admin countries, about), Instagram profile (bio, links, posts), profile picture pHash vs the brand's favicon → 10 signals, 2 penalties (verified, says unofficial) and the AI's reading (≤ 20 points; “unrelated” is never a finding) → a separate social score; bio links off the brand's domains go to the website checks
- suspicious shops: page facts (language, policies, contact, company number, prices incl. '2 for …', invented 'regular price' discounts), the brand's product lines on the page, an ad page named after the brand, the company in its country's registry (Israel data.gov.il, UK Companies House, EU VIES, GLEIF), the shop on Google, reviews, an advisory AI opinion → a separate shop score (shop-v2: six capped groups — the seller, the offer, the brand, contact, reputation, freshness — so signals with one cause count once; a registered company with the shop's own name and an old domain take points off; a company number that belongs to another business is a warning, not a reassurance)
- a conclusion, not just a score: BrandWatch reads “likely counterfeit seller” only from evidence about the goods (a price at half the official one or less, reviews that say fakes) and about the seller; otherwise “suspicious shop”; “confirmed counterfeit” only from an analyst, with the proof written down
- price check: only prices of the brand's own items are compared (the product named next to the price, or a multi-buy deal on a page about the brand) — never the shop's other goods; a sale price keeps the product of the regular price before it (“ShopNova Runner ₪799 ₪349”)
- AI Google queries follow the business: sale / outlet / discount for products, login / wallet / deposit only for crypto, banking and betting brands
Stored
- scan_stages 'discovery' row (hits, API calls) and one live event per stream
- trace: every generated name with its DNS answer, every Google result with its verdict
- the new-domain download, cached once a day
Limits
- streams per track: Light dnstwist + CT · Standard + urlscan · Deep all five
- TLD sets: 12 / 35 / 1,284 · names DNS-checked: 1,000 / 3,000 / 20,000 · ≤ 250 homoglyphs per domain
- DNS: 40 concurrent, batches of 200; > 80 % no answer in the first batch stops the stream with an alert
- ≤ 10 search queries × each country · SerpApi timeout 45 s
- Meta ads: per brand — ads per search (30 / 60 / 100 / 200, ~30 a page, one request each), searches per scan (default 6, up to 20) and shops checked in full (default 15, up to 60), set in the brand profile with the cost shown
- social accounts: ≤ 2 page searches + 2 Google searches per scan; full look (page info + Instagram profile + AI) for ≤ 8 accounts; profile pictures fetched only from Meta's image hosts
- hits capped at min(track 150 / 500 / 1,500, workspace max sites per scan)
Demo vs live
Code
One candidate per URL with every stream that found it; official domains, whitelisted partners and big platforms (YouTube, app stores) are dropped before anything is fetched.
Clean the list
- Calls: nothing outside — it works on what is already stored
- Keeps: candidates, unique on (brand, canonical URL); a known one only gets last_seen bumped
- Limit: 34 known platforms · 33 shared-hosting suffixes · 29 app stores and APK mirrors
All the details ▾
External calls · none
How it runs
Computed locally
- normalise: https by default, IDNA → punycode, registrable domain via a bundled public-suffix snapshot
- whitelist (fail-open): official + partner domains and their registrable roots; on shared hosting only the exact site
- known platforms (YouTube, social networks, wikis …) skipped; one visit per host per scan except shared hosting and app stores
- app listings compared by canonical listing URL (Google Play keeps ?id=); the brand's own listings are skipped
- rows created before a domain was whitelisted are dismissed and their open findings closed (audited)
Stored
- candidates, unique on (brand, canonical URL); a known one only gets last_seen bumped
- candidate_sources: one row per stream that found it (SAVEPOINT guard against concurrent workers)
Limits
- 34 known platforms · 33 shared-hosting suffixes · 29 app stores and APK mirrors
- visit cap = min(requested or track maximum, workspace max sites per scan); the rest waits for the next scan
- dismissed candidates are not revisited
Demo vs live
Code
Each site is visited in a sandboxed browser behind an SSRF guard: DNS, WHOIS / RDAP, TLS, screenshot, HTML, favicon and reputation feeds — all stored as evidence.
Visit in a sandbox
- Calls: DNS, Headless Chromium, RDAP via rdap.org, VirusTotal v3, URLhaus, Google Safe Browsing v4, ip-api.com
- Cost: free · free · local · free · free public key · 4 req / min · free · free · non-commercial · free · ~45 req / min
- Keeps: immutable evidence rows (database triggers reject UPDATE / DELETE) with a parent chain dns → http → html / screenshot / favicon
- Limit: response ≤ 5 MB · ≤ 5 redirects · fetch timeout 15 s · connect 5 s (frozen in code)
All the details ▾
External calls
own root walk, or 1.1.1.1 / 9.9.9.9 / 8.8.8.8A query; without an A record, an NS check 'is it registered?'the candidate URLfresh browser per fetch, as a person's Chrome (User-Agent, Accept-Language, sec-ch-ua client hints, language, time zone): goto, settle 3 s, screenshot, faviconrdap.org → the registry's RDAP serverGET /domain/<registrable> (application/rdap+json)www.virustotal.comGET /api/v3/domains/<host> · x-apikey headerurlhaus-api.abuse.chPOST /v1/host/ · Auth-Key headersafebrowsing.googleapis.comPOST /v4/threatMatches:find · X-Goog-Api-Key header · http:// and https:// root URLip-api.comGET /json/<ip>?fields=status,country,city,lat,lon,org,asFallback chain
Computed locally
- SSRF guard: http/https only, ports 80 / 443 / 8080 / 8443, no credentials; every resolved IP checked, fail-closed
- every Chromium sub-request re-validated; media, fonts, websockets, service workers and downloads blocked
- the HTTP fallback pins the validated IP (Host header + SNI) and re-validates each redirect hop
- HTML features: password / card / seed-phrase forms, off-site form action, e-mail prefill, lure words, tracking IDs, parking
- screenshot pHash / dHash + palette; favicon MurmurHash3 + pHash
Stored
- immutable evidence rows (database triggers reject UPDATE / DELETE) with a parent chain dns → http → html / screenshot / favicon
- child tables: DNS, HTTP, HTML, screenshot, WHOIS, TLS, one reputation row per provider
- HTML and PNG artifacts named by their SHA-256; one observation row per visit
Limits
- response ≤ 5 MB · ≤ 5 redirects · fetch timeout 15 s · connect 5 s (frozen in code)
- 24 blocked networks (RFC 1918, CGNAT, loopback, 169.254/16 metadata, ULA, NAT64 …) + the AWS IPv6 metadata address
- browser operation ≤ 45 s · favicon ≤ 512 KB in 4 s
- every provider call: 3 attempts, backoff from 0.5 s, circuit opens after 3 failures for 60 s, 1 h cache
- no A record → no browser visit: registered goes to the watchlist as parked, otherwise dismissed
Demo vs live
Code
Kits that hide from scanners are fetched again as a bot, as a victim link, from a Google referer — and as a person: a real Chrome (User-Agent, Accept-Language, client hints, language and time zone) through a residential or a mobile 4G / 5G proxy, or both, as a desktop and / or a phone. The score uses what a victim sees.
Catch cloaking
- Calls: Headless Chromium, Residential proxy, Mobile proxy, Webshare API
- Cost: free · local · paid · metered 0.002 GB per fetch · paid · metered like the residential one · paid plan
- Keeps: one HTTP evidence row per probe (diverged, challenge, credential form)
- Limit: probes on in every track; home-IP probe gated on Light / Standard, always on Deep
All the details ▾
External calls
the candidate URL3 re-fetches: a crawler user-agent · a victim link (?email=victim@example.com) · Referer https://www.google.com/ (+ a phone when chosen); all but the crawler as a real Chrome with matching headersp.webshare.io:80 by defaultChromium through the proxy; Webshare username <user>-rotate-country-<cc>the provider's host:port, entered in SettingsChromium through a phone-network address — instead of the residential proxy, or as a second visit with bothproxy.webshare.ioGET /api/v2/proxy/config/ · Authorization: TokenFallback chain
Computed locally
- runs when the host looks like the brand, the page has a credential / card / seed form, or the first score is ≥ Medium
- signals: cloaking_ua_based · bot_blocking (CAPTCHA) · parameter_gating · referer_gating · device_gating (phone only) · cloaking_ip_based (home or mobile IP)
- a person's Chrome: Accept-Language and navigator.languages, sec-ch-ua / userAgentData say 'Google Chrome' (not HeadlessChrome), locale and time zone of the proxy's country
- suspected = a probe revealed a form, any probe diverged, or a thin brand-less page on a look-alike host
- the revealing probe's capture is adopted, so the score uses what a victim sees; the decoy is kept as evidence
- the proxy probe runs only if still suspected: residential, mobile or both (Settings → Proxy), as a desktop and / or a phone; navigator.webdriver is hidden
Stored
- one HTTP evidence row per probe (diverged, challenge, credential form)
- the revealed page as new HTML / screenshot / favicon evidence; a 'residential' observation
- an api_usage row per home-IP fetch
Limits
- probes on in every track; home-IP probe gated on Light / Standard, always on Deep
- home-IP probe needs a proxy and a daily GB budget (default 0 → skipped)
- suspected but skipped → 'residential' marked unavailable (confidence −8)
- proxy country: the brand's override, else its search country, else its language, else US
Demo vs live
Code
50 signals in 7 groups compare the site with the brand: identity tricks, visual copy, what the page asks for, shared infrastructure, reputation, freshness, evasion.
Compare with the brand
- Calls: nothing outside — it works on what is already stored
- Keeps: candidates.last_analysis: signals, unavailable evidence, score — for every scored site, used to train the classifier
- Limit: 50 signals in 7 groups (config/brand_config.py)
All the details ▾
External calls · none
How it runs
Computed locally
- Identity: brand in the name, combosquat, Levenshtein ≤ 2, IDN homoglyph, TLD swap, title claims the brand, brand in subdomain / path
- Visual: favicon exact or pHash ≤ 6 bits; screenshot ≥ 0.90 similar to the home or login page; palette ≥ 0.67; login DOM ≥ 0.80
- Behaviour: password / card / seed fields, off-site form action, e-mail prefill, 17 lure phrases, /login-style path
- Infrastructure: shares an IP, certificate or tracking ID with a confirmed or critical finding, or a mapped attacker node
- Reputation fires only on a positive verdict (unknown is never clean) · Freshness: < 7 / < 30 days, woke up · Evasion: cloaking
Stored
- candidates.last_analysis: signals, unavailable evidence, score — for every scored site, used to train the classifier
- one trace line per site with every fired signal and its plain-language reason
Limits
- 50 signals in 7 groups (config/brand_config.py)
- favicon near-match ≤ 6 bits · screenshot ≥ 0.90 · palette ≥ 0.67 · login DOM ≥ 0.80
- colours count as close at L1 distance ≤ 54 · DOM signature ≤ 400 tokens
Demo vs live
Code
Deterministic and versioned: points per signal, a cap per group, combination rules for known kit shapes, false-positive penalties — plus a separate confidence axis. AI and a classifier give advisory opinions only.
Score & explain
- Calls: OpenAI
- Cost: paid per token
- Keeps: risk_scores: append-only history (only is_current flips) with signals, group breakdown and the scoring version
- Limit: bands: Medium ≥ 25 · High ≥ 50 · Critical ≥ 75 (per brand, must stay in order)
All the details ▾
External calls
api.openai.comPOST /v1/chat/completions · strict JSON schema phishing_triage · temperature 0 · page text framed as untrustedFallback chain
Computed locally
- raw = Σ min(group points, group cap) + cluster bonus − false-positive penalties, clamped 0..150 → score = raw / 150 × 100
- group caps: identity 20 · visual 30 · behaviour 30 · infrastructure 20 · reputation 15 · freshness 10 · evasion 25
- 5 decisive clusters (+15, capped at 15); 8 combination rules set a minimum severity
- confidence: breadth 0–70 + hard confirmers (≤ 30) + completeness (≤ 15) + agreement − 8 per unavailable source
- classifier: L2 logistic regression in numpy on 47 binary features, 5-fold cross-validation; shown, never scored
Stored
- risk_scores: append-only history (only is_current flips) with signals, group breakdown and the scoring version
- trained models as rule_versions rows (weights, metrics, top features)
- the AI and classifier opinions inside the breakdown, labelled advisory
Limits
- bands: Medium ≥ 25 · High ≥ 50 · Critical ≥ 75 (per brand, must stay in order)
- false-positive penalties: redirects to the official site −60 · domain older than 2 years −15
- AI opinion only for Medium+ when enabled (off on Light) · page text ≤ 4,000 chars
- the classifier needs ≥ 8 labelled sites of both classes
Demo vs live
Code
Medium and above become a finding for an analyst; the rest stays on the watchlist and is re-checked on a schedule (a parked domain that grows a login form jumps to the front).
Raise a finding
- Calls: Slack incoming webhook, Telegram Bot API, SMTP e-mail
- Cost: free · free · free
- Keeps: findings + a risk_scores row + evidence links with a role (primary, supporting, context, contradicting)
- Limit: re-check every 24 / 12 / 6 h for 7 / 14 / 30 days (Light / Standard / Deep)
All the details ▾
External calls
SLACK_WEBHOOK_URLPOST {text} per new or updated findingapi.telegram.orgPOST /bot<token>/sendMessageSMTP_HOST:587the finding alert and the daily digestFallback chain
Computed locally
- score ≥ Medium → create or update the site's finding (one finding per site, even after a confirm)
- below the threshold → watching, re-checked on the track's cadence
- triggers vs the last visit: a login form appeared +10 priority · woke up +6 · favicon changed +4 · page changed +2
- scheduler tick: expire stale domains, re-queue due ones by priority, start the automatic scans that are due (only brands switched on; chosen frequency, local hour and tasks)
Stored
- findings + a risk_scores row + evidence links with a role (primary, supporting, context, contradicting)
- candidate state (flagged, watching, long watch, dismissed) and an audit event for every transition
Limits
- re-check every 24 / 12 / 6 h for 7 / 14 / 30 days (Light / Standard / Deep)
- past the window → long watch at 3× the cadence; past 3× the window → dismissed
- scheduler tick every 60 s · ≤ 200 re-checks per tick · automatic scans are off per brand until switched on (brand page → Automatic scans); one scan at a time per brand; the watchlist re-check is one of the tasks
- alerts at ≥ High by default
Demo vs live
Code
The live console narrates every step while it happens and ends with a plain-language report; an engineering trace records every provider call, AI prompt and the scoring maths.
Report
- Calls: nothing outside — it works on what is already stored
- Keeps: scan_events (unique job + sequence), scan_stages (hits, API calls, cost, time), api_usage per provider call
- Limit: SSE stream resumes from Last-Event-ID; polls every 0.5 s; keep-alive when idle; an 'end' event
All the details ▾
External calls · none
How it runs
Computed locally
- plain-language report: one line per stage (fingerprint, each stream, visit cap, visits, cloaking, scoring, provider issues)
- event emitter: monotonic sequence per job, secrets redacted, committed per event so the console updates live
- provider failures classified (quota, rate limit, bad key, outage, circuit open, not configured) → one alert with a fix
- engineering trace: every provider call in and out, the full AI prompt and answer, every DNS answer, the scoring maths
Stored
- scan_events (unique job + sequence), scan_stages (hits, API calls, cost, time), api_usage per provider call
- trace file per scan, served at GET /api/jobs/{id}/trace?host=
- the job's cost and a final summary event
Limits
- SSE stream resumes from Last-Event-ID; polls every 0.5 s; keep-alive when idle; an 'end' event
- polling fallback: GET /api/jobs/{id}/events?from_seq=N (≤ 5,000 per page)
- trace values cut at 1,200 chars (AI 12,000) · secrets redacted
Demo vs live
Code
The finding page explains itself in two sentences and shows every signal, its points and the evidence. A human decides; the reason goes to the audit log and the training set.
Confirm or dismiss
- Calls: nothing outside — it works on what is already stored
- Keeps: finding state + verdict; analyst_labels (label, reason, analyst)
- Limit: reason ≤ 2,000 chars
All the details ▾
External calls · none
How it runs
Computed locally
- admin / analyst only, and only from 'awaiting verdict' (409 otherwise)
- confirm → phishing; dismiss → legitimate, suspicious or unknown (never phishing)
- confirming queues the infrastructure map automatically (one already running is reused)
- confirmed sites feed later scans: shares a server / certificate / tracking ID with phishing
- verdicts train the classifier (phishing 1, legitimate 0)
Stored
- finding state + verdict; analyst_labels (label, reason, analyst)
- audit_events (who, old → new, reason) — the database rejects edits and deletes
Limits
- reason ≤ 2,000 chars
- 10 immutable tables: audit_events, observations, evidence + 7 evidence child tables
Demo vs live
Code
VirusTotal-style lookup on the finding page itself: detections, DNS, certificate, RDAP and abuse contact, WHOIS history (every past owner, and details hidden later), passive DNS, every other site on the server, urlscan similar pages.
Site & network investigation
- Calls: VirusTotal v3, RDAP via rdap.org, crt.sh → CertSpotter, urlscan.io, WhoisXML WHOIS History, ip-api.com, AbuseIPDB v2, Live DNS + TLS handshake
- Cost: free public · 4 req / min; premium-only relations are marked as such · free · free · free tier · paid · WhoisXML credits · free · free daily quota · free
- Keeps: intel_lookups keyed by (workspace, subject, source, relation): payload, status, expiry
- Limit: cache 24 h (errors 10 min) · Refresh forces new lookups
All the details ▾
External calls
www.virustotal.com/api/v3GET /domains/{d}, /ip_addresses/{ip}, /{kind}/{v}/{relationship}?limit=40&cursor=rdap.orgGET /domain/{registrable}crt.sh · api.certspotter.comCT search for the registrable domainurlscan.io/api/v1POST /scan/ (unlisted) · GET /result/{uuid}/ · GET /search/?q=hash:|page.ip:|page.domain:whois-history.whoisxmlapi.com/api/v1GET ?domainName=<registrable>&mode=purchase&outputFormat=JSONip-api.comGET /json/{ip}api.abuseipdb.comGET /api/v2/check?ipAddress={ip}&maxAgeInDays=90 · Key headerpublic resolvers · the site's validated IP:443A / AAAA / CNAME / NS / MX / TXT / SOA; one handshake with SNIFallback chain
Computed locally
- a sliding-window pacer keeps VirusTotal at ≤ 4 calls per 60 s; extra pages pace themselves
- live DNS and the live certificate only when VirusTotal has none
- IPs = VirusTotal A records + live A records + the finding's evidence, de-duplicated, first 4
- urlscan: submit and poll up to 16 × every 5 s, else the latest public scan of the domain
- WHOIS history → timeline: registration, re-registration after expiry, transfers, owner changes, privacy on / off
- owner details public before privacy was switched on, cross-referenced with every other domain in the workspace
Stored
- intel_lookups keyed by (workspace, subject, source, relation): payload, status, expiry
- events and a trace for the intel job; plan-gated relations stored as 'not available on this plan'
Limits
- cache 24 h (errors 10 min) · Refresh forces new lookups
- ≤ 4 IPs, ≤ 48 calls, ≤ 300 s per job; the daily spend cap still applies
- VirusTotal resolutions: 2 pages for a domain, 3 for an IP (40 per page)
- IP intel only for public IPs (the SSRF block list applies)
Demo vs live
Code
From one confirmed site, a budgeted pivot over shared servers, certificates, favicons and tracking IDs; co-hosted noise is triaged out. Indicators export as STIX 2.1.
Attack infrastructure
- Calls: VirusTotal resolutions, Shodan, crt.sh, OpenAI + Chromium
- Cost: free public · 4 req / min · paid membership for search · free · paid per token
- Keeps: campaigns, campaign_nodes (included, triage reason, AI verdict), campaign_edges (typed, linked to evidence)
- Limit: budget: depth 2, 100 nodes, 300 edges (frozen in code) · 60 API calls · 120 s · the workspace's investigation cost cap
All the details ▾
External calls
www.virustotal.comGET /ip_addresses/{ip}/resolutions or /domains/{d}/resolutions?limit=40api.shodan.ioGET /shodan/host/{ip} · /shodan/host/search?query=http.favicon.hash:<mmh3>crt.shGET /?q=<certificate SHA-256>&output=jsonapi.openai.com · the neighbour sitefetch the neighbour, then the phishing_triage schemaFallback chain
Computed locally
- breadth-first search from the finding's evidence: IPs, certificate SHA-256, favicon hash, tracking IDs, registrant e-mail
- tracking-ID and registrant pivots query this brand's own evidence (no API call)
- triage: own sites out; existing findings in; brand pattern in; shared certificate / favicon / tracking ID in; bare reverse-IP neighbour out
- AI triage of ≤ 8 noise neighbours: in only if phishing for this brand at ≥ 0.75; other brands → actor intel
- indicators exported as STIX 2.1 (identity, campaign, indicators, 'indicates' relationships), CSV or JSON
Stored
- campaigns, campaign_nodes (included, triage reason, AI verdict), campaign_edges (typed, linked to evidence)
- iocs, unique per (workspace, type, value): domain, hostname, URL, IP, certificate, favicon hash, tracking ID, e-mail
- pivoted domains become new watchlist candidates (priority 20, source 'pivot')
Limits
- budget: depth 2, 100 nodes, 300 edges (frozen in code) · 60 API calls · 120 s · the workspace's investigation cost cap
- a server with > 25 names is shared hosting → no further pivot from it
- ≤ 8 AI-triaged neighbours per map · IOC export ≤ 20,000 rows
Demo vs live
Code
An evidence-backed request with an audited lifecycle; contacts for the registrar, the hosting provider and the CDN; 26 blocklists, anti-virus and abuse-report portals. BrandWatch never sends anything itself.
Takedown
- Calls: nothing outside — it works on what is already stored
- Keeps: takedown_drafts: state, contacts, reference, body, sent / response / resolved times, channels and portals reported
- Limit: only for confirmed findings · editable in draft, review or reopened
All the details ▾
External calls · none
How it runs
Computed locally
- registrar: WHOIS abuse e-mail → RDAP abuse contact → VirusTotal WHOIS text → known desk
- hosting, per IP (≤ 4): the network owner's abuse e-mail, or the known desk for its organisation
- CDN when the IP owner or nameservers are Cloudflare, Akamai or Fastly
- templated request: signals, domain age, screenshot similarity, case id BW-<8 hex>; addressed per channel
- mailto links capped at 1,900 characters (Outlook's limit)
Stored
- takedown_drafts: state, contacts, reference, body, sent / response / resolved times, channels and portals reported
- an audit event for every draft, edit, transition, channel sent and portal reported
- evidence package: HTML (screenshots inline, SHA-256 per row) + JSON
Limits
- only for confirmed findings · editable in draft, review or reopened
- draft → review → approved → sent → acknowledged → in progress → taken down (+ rejected, reopened)
- 'sent' needs an abuse contact and is recorded by the analyst after mailing it
- 26 report portals in 5 groups (browsers, anti-virus, corporate web filters, phishing feeds, national reporting), each linked as close to the form as the vendor allows
- suspicious shops: a letter worded to the conclusion — a request for review for a suspicious shop, “likely selling counterfeit” for a likely one, “selling counterfeit … confirmed by a test purchase” only when confirmed — plus Report to Meta — every ad (Ad Library link → Report ad) and every advertiser page that sends buyers to the shop, and Meta's trademark / counterfeit forms for the brand owner; each marked reported with an audit event
Demo vs live
Code
Guided tour · 8 stops· a ten-minute walk through the app, with the questions people ask
- Dashboard — the mission line and the five-step strip
“BrandWatch finds sites impersonating a brand — on the web and in Facebook / Instagram ads — proves it with tamper-proof evidence, maps the attacker's servers and drafts the takedown. A human decides every step.”
Likely question: what's real here? → In demo mode every external service is a recorded scenario; the same code calls VirusTotal, crt.sh, Google and Chromium in live mode.
- The brand page — tick “include fingerprinting”, press START SCAN
“First it learns what the real site looks like, then it hunts for copies on five streams.”
Likely question: how do you find domains that don't exist in any list yet? → Permutations checked in DNS, new certificates in CT logs, yesterday's registrations.
Likely question: what about ChatGPT and Gemini? → With AI answers on, the approved questions (support phone, contact, login, fraud) are asked to ChatGPT, Gemini and Google AI Mode; every contact in the answers is compared with the ones on the brand's own website, a mismatch gets a "show me the official page" follow-up, and BrandWatch opens the cited pages itself before it raises a finding.
Likely question: what about social media? → Facebook pages by name (Meta's page search, every country) and Instagram / Facebook profiles Google knows; each one's page history, logo, links and posts are checked and an AI reads the content. Fakes become findings with a Report-to-Meta box, a link in their bio goes to the website checks, and the brand's own accounts are never flagged. Ads about the brand are searched too, per country.
- The live scan console — then “▶ replay highlights”
“Every step is narrated as it happens; the report at the bottom says what the scan did in plain words.”
Likely question: what if VirusTotal is down? → Retries with backoff, a circuit breaker, then the next provider in the chain; the alert panel says what happened and how to fix it.
- A cloaked kit — read the two-sentence summary, then the three screenshots
“This kit shows a blank page to scanners and the login page to real visitors. We fetch it like a victim would.”
Likely question: why is VirusTotal at 0? → It's days old; blocklists lag. Unknown is never treated as clean.
- Explain Finding — signals, group caps, the score line, “open the arithmetic”
“Deterministic, explainable scoring: every point traces back to evidence. Risk and confidence are separate axes. The AI is advisory.”
Likely question: false positives? → Penalties (old domain, redirects to the brand), a whitelist, and a golden-dataset regression gate (precision / recall ≥ 0.90).
- Press Confirm phishing, then Investigate site & network
“Confirming maps the attacker's infrastructure automatically. The investigation tabs on the finding page replace twenty minutes of VirusTotal, urlscan and RDAP tabs.”
- The attack infrastructure map — drag the graph, read two triage reasons
“One site led to the whole operation. Noise is triaged out with a reason; the indicators go to a SIEM as STIX.”
-
“Three channels — registrar, host, CDN — plus the blocklists. Every step is in an append-only audit log the database won't let anyone edit.”
Under the hood· the engineering an interviewer asks about
SSRF guard & sandbox
Every fetch: validate the URL → resolve A/AAAA → fail closed on 24 blocked networks (private, loopback, cloud metadata) → connect pinned to the validated IP with Host + SNI → re-validate every redirect hop (≤ 5) → size ≤ 5.0 MB, timeout 15.0 s.
The same policy gates every Chromium sub-request. Limits are frozen in code — no brand, workspace or env var can change them.
Immutable evidence
10 append-only tables: database triggers reject UPDATE and DELETE — even for an admin, on SQLite and PostgreSQL. Artifacts are content-addressed: the SHA-256 is the file name.
Each finding links its evidence with a role (primary, supporting, contradicting, context): a chain of custody for a takedown or a court.
Provider resilience
Every external call goes through one wrapper: TTL cache → budget check → circuit breaker → token bucket → retry with backoff (honouring Retry-After) → the next provider in the fallback chain.
A bad key disables the provider for the job instead of failing 500 calls; "unknown" is never treated as clean.
Cost ledger & budgets
Every paid call is recorded with its cost; daily call and spend caps are checked over the trailing 24 h before the call, so a scan stops spending instead of overspending.
Golden dataset gate
65 recorded scenarios (16 phishing incl. cloaked kits, 17 legitimate, 15 brand-related legitimate, 10 suspicious-but-not-phishing) run through the real pipeline in CI.
The build fails below precision 0.9 / recall 0.9 / FP rate 0.05; last measured 1.00 / 1.00 / 0.00. The same corpus drives the $0 demo.
Durable live events
Each scan writes an ordered event stream (sequence-numbered, secrets redacted) before publishing it over SSE; a refresh resumes from the last event. Cancellation is cooperative at safe checkpoints.
Secrets, tenants & roles
API keys are envelope-encrypted (random 32-byte key per record, AES-256-GCM, bound to workspace:provider). Passwords: Argon2id. Personal API tokens are stored as SHA-256 only; a read-scoped token caps even an admin at viewer.
Every query is workspace-scoped; RBAC (admin / analyst / viewer) is identical in the UI and the API.
Tests & CI
659 test functions in 73 files: schema and trigger enforcement, SSRF cases, provider resilience, every UI page, the golden gate. CI runs lint, test-sqlite, test-postgres, including migrations up and down on PostgreSQL 16.
Trust & safety
- A human decides every verdict and every takedown; nothing is ever sent automatically.
- Residential IPs are used to look like a victim, never to interact; probes never submit forms.
- The AI only gives advisory opinions under a strict JSON schema, with the page text framed as untrusted.
- Only brands you are authorised to protect; every action is in the append-only audit log.
Glossary· every “?” on the site links here
- Baseline
- The fingerprint of the brand's real site — favicon hash, screenshot hash, colours, login pages, keywords. Every suspect is compared against it.
- Candidate / watched domain
- Any domain discovered for a brand. It lives on the Watchlist until it scores high enough to become a finding, expires, or is dismissed.
- Finding
- A candidate that scored Medium or higher. An analyst confirms it as phishing or dismisses it; nothing happens automatically.
- Signal
- One piece of evidence (e.g. 'identical favicon'). 50 signals in 7 groups; each adds points.
- Group cap
- Each group can add at most its cap, so ten weak visual hints never outweigh one strong behaviour signal.
- Risk score
- How dangerous the site is, 0–100. Medium ≥ 25, High ≥ 50, Critical ≥ 75. Combination rules can raise the severity.
- Confidence
- How complete and consistent the evidence is (breadth, completeness, provider agreement) — separate from risk.
- Cloaking
- A kit that shows a harmless decoy to scanners and the phishing page to victims. BrandWatch fetches as a bot, as a victim link, from a search referer and from a home IP.
- Detection track
- Light, Standard or Deep: which discovery streams run, how many sites are visited, how often the watchlist is re-checked.
- Attack infrastructure
- The map built from one confirmed site: every server, certificate and sister domain it shares, triaged so reverse-IP noise is excluded. The pivot is budgeted (at most 2 hops, 100 nodes, a fixed number of API calls) and those limits are frozen in code.
- Advisory opinions
- An AI model and a trained classifier each give a second opinion on Medium+ findings. The page text is passed as untrusted input and the answer must match a strict JSON schema. Shown for context only: the deterministic score never depends on them.
- IOC
- Indicator of compromise: a domain, IP, certificate hash or tracking ID a firewall / SIEM can block. Exported as CSV, JSON or STIX 2.1.
- Takedown
- The request to remove a site. Draft → review → approved → sent → taken down; sent to the registrar, the hosting provider and the CDN, and reported to blocklists.
- Evidence
- Every screenshot, HTML page, DNS / WHOIS answer and reputation lookup, stored append-only with its SHA-256 — the database rejects edits and deletes.
- Discovery stream
- One way of finding look-alikes: look-alike names (dnstwist), new certificates (CT), newly registered domains (NRD), Google search, the urlscan.io archive, social media, ads. A track decides which streams run.
- dnstwist
- Generates the names a phisher would register — typos, swapped letters, other TLDs, look-alike Unicode — and checks in DNS which of them exist.
- Certificate transparency (CT)
- Every HTTPS certificate is published in public logs within minutes; a new certificate carrying the brand's name is often the first sign of a kit.
- Newly registered domains (NRD)
- Yesterday's list of every newly registered domain (about 70,000 a day), searched for the brand's name.
- urlscan.io
- A public archive of scanned web pages: past scans that mention the brand, and a screenshot of a site that blocks BrandWatch.
- Home / mobile IP probe
- A visit through a home (residential) or 4G/5G address, for kits that show the real page only to people, never to a datacentre. Needs a proxy and a GB budget.
- STIX 2.1
- The standard file format security tools exchange threat indicators in; the infrastructure map's indicators can be exported as STIX for a SIEM or a firewall.
- Golden dataset
- Recorded, hand-labelled scenarios (phishing, cloaked kits, scam shops, legitimate sites) the real pipeline is run on in every build; precision and recall are measured on them.
- TLD set
- Which top-level domains dnstwist tries: common (.com .net …), popular (about 100) or all (about 1,280).
- Vertical
- The kind of business ("footwear and apparel", "crypto exchange") — read by the AI from the official site, used to pick the right Google queries.
- Who sells it
- Only the brand (a service, an app, a bank) or other shops too: then scam shops advertising its products are hunted as well.
- Demo mode
- Recorded scenarios replace every external service: the full pipeline runs deterministically, offline, at $0.