Web Security, WAF & Bot Management Engineering
Starting from $5,500
(9 ratings) 1240 views
I separate incoming traffic into humans, declared bots and hidden automation, and tie every block decision to a threshold and the reason behind it. Edge WAF rules are managed from code, requests clearing the edge meet a second filter at the origin, and every block is counted by category. Bot claims are verified against vendor-published address ranges.
SERVICE 4: Web Security, WAF & Bot Management Engineering
Making a security decision without knowing how much of your traffic is actually human is like firing in the dark. Part of today's incoming requests is automation impersonating a browser, part is search and AI crawlers that declare their identity honestly, and part is scanners looking directly for a way in. All three look alike in the same log line. I build a defence layer that separates these three, grounds every decision in a measurement, and records the reason behind every block.
The work here is not "turn on the WAF and enable bot protection". It is making measurable what the edge caught, what an escaping request ran into at the origin, who was blocked at which threshold and for how long, and whether any of it is actually working. None of the numbers below are illustrative scenarios; they were measured from code running in production on this very site.
Layered Defence – High-Level Flow
| Edge (CDN/WAF) Custom rules, rate limits, known-bad traffic |
Origin Filter Path pattern and encoding variant matching |
Classification Human / declared bot / hidden automation |
Verification Vendor IP ranges, ASN, header signature |
Decision and Record Score, escalating block, metric |
Why This Service?
In most setups security is treated as an on/off switch. The outcome splits in two: either the rules are too loose and an attacker slips through, or they are too tight and a real visitor is left at the door. Both happen for the same reason — there is no measurement behind the decision.
| Conventional Approach | My Approach |
|---|---|
| A managed rule set is switched on and never revisited | Every rule is tied to a threshold and to the reason for that threshold |
| Bots are assumed bad and thrown into one bucket | Search crawlers, AI crawlers and hidden automation are governed separately |
| The user-agent claim is taken at face value | The claim is checked against the vendor's own published IP ranges |
| Block duration is single and fixed | Duration escalates with repetition; a first mistake and persistence are not punished alike |
| The edge is the only line of defence | A request that clears the edge meets a second filter at the origin |
| Protection is assumed to be working | Every block is counted with its category; whether it works is measured |
My Architectural Approach
1) Signature Matching and Evasion Detection
The first layer of attack detection is pattern matching, but catching a pattern written literally in a list is not enough. An attacker URL-encodes the path to slip past the list. That is why path normalisation applies two decoding passes; if the output of the second pass differs from the first, that alone counts as an attack signal, because a legitimate browser does not send a double-encoded path.
Every matched signature is more than a "matched / did not match" pair; it carries severity 1-5 and confidence 0-100. The block decision comes from the score these two produce, so a weak signal never leaves anyone at the door on its own.
2) Verification, Not Declaration
A request's user-agent field is a claim, not evidence; anyone can declare themselves a search engine. So the IP ranges vendors publish themselves are fetched and compared against the claim. If the requesting address is not in the range, the verified-bot label is dropped.
There is an important distinction here: for a token with no known mapping the result is left empty. Could not verify and forged are not placed in the same box. Without that distinction, legitimate bots without verification infrastructure get quietly mislabelled.
3) Escalating Blocks and Edge Quota
Block duration is not a single constant. The first block lasts 24 hours, a returning address gets 7 days, and a third offence 30 days. A one-off scanner and an attacker who keeps coming back do not receive the same penalty.
Only blocks of 7 days and above are pushed to the edge list. The reason is practical: edge lists are quota-bound, and filling them with one-off scanners leaves no room for genuine repeat offenders.
4) Rate Limits per Endpoint
A single global request limit is both crude and wrong; a read page and a contact form do not share a usage profile. The limit is split into 9 separate buckets and each threshold is chosen from that endpoint's legitimate profile. A rejected request returns 429 with a Retry-After header, telling the client when it may try again.
What I Deliver
| Area | What I Provide |
|---|---|
| WAF rule management | Edge rules managed from code, version tracking, reversibility |
| Bot classification | Separating search, AI, social, SEO and monitoring bots, each with its own policy |
| Attack detection | Signature library, encoding evasion detection, score-based block decisions |
| Authentication protection | Rate limits on the login endpoint, username enumeration detection, protection without account lockout |
| Measurement and dashboard | Blocks counted by category, edge–origin reconciliation, traffic distribution |
| Privacy compliance | Hashing the visitor identifier, pinning the retention period in code |
The Numbers Running on My Own Site
The following are not example scenarios but values measured from code running in production on this site. The most honest way to describe a service is to run it on yourself and show the number.
| Component | Measure | What It Means |
|---|---|---|
| Attack signature library | 129 signatures / 30 categories | 95 path patterns, 18 scanner user-agents, 16 query payloads; each carries severity and confidence |
| Encoding evasion detection | 2 decoding passes | If the second pass differs from the first, that alone is an attack signal |
| Bot user-agent classification | 143 tokens / 8 kinds | 62 tokens are in the AI family; training crawls, AI search indexing and user-triggered fetches are marked separately |
| Vendor verification | 7 vendors / 14 IP ranges | 32 token–vendor mappings, plus 55 hosting ASNs for datacentre origin |
| Automatic block thresholds | 3 attempts in 15 minutes | Threat score 11; block duration escalates 24 hours, 7 days, 30 days |
| Second layer (origin) | 47 path patterns + 7 variants | A request clearing the edge gets a 404 without generating log noise; every block is counted as a metric |
| Rate limit policies | 9 separate buckets | From 200 per minute on reads down to 5 per hour on the contact form |
| Login abuse detection | 2 distinct patterns | Distributed attempts against one account and username enumeration from one address are handled separately |
Technologies I Use
Cloudflare WAF and Rulesets API, the ASP.NET Core middleware pipeline, sliding-window rate limiting, Prometheus metrics, Grafana dashboards, HMAC-SHA256 visitor hashing, attack and block telemetry on MSSQL, additional constraints at the Kubernetes ingress layer.
What I Deliberately Do Not Do
In a security design, what is not done matters as much as what is. The following are not gaps but deliberate decisions:
| Decision | Reasoning |
|---|---|
| No account lockout | Lockout hands an attacker a free account-disabling tool; protection is applied on the address side |
| ISP and transit ASNs stay off the list | False-positive risk cuts off real visitors; only hosting ASNs are kept |
| An unverifiable bot is not called forged | Uncertainty is its own state; it is left empty rather than producing a wrong label |
| Short blocks are not pushed to the edge | The edge list is quota-bound; a one-off scanner should not consume that quota |
Project Deliverables
| Deliverable | Description |
|---|---|
| Threat surface breakdown | Existing traffic separated into human, declared bot and hidden automation |
| Edge rule set | Version-controlled WAF and rate-limit rules managed from code |
| Origin defence layer | In-application signature matching, classification and escalating block mechanism |
| Measurement dashboard | Block categories, edge–origin reconciliation, false-positive tracking |
| Threshold documentation | Every threshold and why it is that number, so the team inheriting it does not change values blindly |
Who Is It For?
Teams that do not know how much of their traffic is real; that have bot protection enabled yet still receive complaints from genuine visitors; that see constant probing on the login endpoint; that want to govern how AI crawlers use their content; or that, after a security incident, are looking for an answer to "why was this not caught".
Why Me?
I built this layer on my own site first and I run it in production. Every number above was measured from that code — the signature count, the threshold values, the bucket count. Every mechanism I describe has a working counterpart; it comes from a file, not from a slide.
A second point: building this also showed me my own mistakes. A badly written edge rule broke part of the site for visitors outside Türkiye; I spotted it on the dashboard and fixed it. Being able to measure what your protection breaks is as much a part of the job as building it.
Conclusion
A good security layer works quietly, but it does not work blindly. Every block should rest on a reason, every threshold on a number, and every number on a decision that can be changed. The aim is to stop the attacker without leaving the real visitor at the door — and to be able to prove that is what actually happens.