Fail2Ban Filter Regex: Writing Custom Patterns That Work

You run a Flask API that logs authentication failures in a custom format: 2024-01-15 14:23:11 AUTH_FAIL user=admin ip=203.0.113.42. Attackers probe it relentlessly, but none of Fail2Ban's default filters match this line. You need a fail2ban custom filter, but your first attempt sits silently in /etc/fail2ban/filter.d/, never triggering a ban. The regex looked right, yet it captures nothing. This happens because Fail2Ban's matching logic is strict: a single misplaced quantifier or missing <HOST> placeholder turns a filter into dead weight.

This guide walks through the entire workflow from raw log line to tested, deployed filter. You will write failregex patterns that handle variable whitespace and multiline logs, use fail2ban-regex to catch mistakes before they reach production, and avoid the silent failures that waste hours of troubleshooting. By the end, you will have a working custom jail protecting your application.

How Fail2Ban Filters Work

Fail2Ban filters live in /etc/fail2ban/filter.d/ as .conf files. Each filter contains a [Definition] section with one or more failregex = lines. When a jail references the filter, Fail2Ban opens the configured logpath, tails new lines, and applies every failregex pattern to each line. A match increments a counter for the extracted IP address. When the counter exceeds maxretry within findtime, Fail2Ban bans the IP using the jail's action (typically an iptables rule).

The critical requirement: your regex must produce a captured host. Fail2Ban provides the <HOST> placeholder, which expands to a pattern matching IPv4 or IPv6 addresses. Here is a minimal working filter for SSH:

[Definition]
failregex = ^%(__prefix_line)sFailed password for .* from <HOST>

The <HOST> placeholder tells Fail2Ban where to extract the IP. The %(__prefix_line)s macro matches common syslog timestamp prefixes; you can omit it for custom logs with different formats. Filters run per-jail: you associate a filter with a jail in jail.local or jail.d/, specifying filter = filtername and logpath = /path/to/log.

Fail2Ban auto-detects timestamp formats in most cases. If your logs use an unusual format, set datepattern = in the filter definition to a strptime-style pattern or one of Fail2Ban's named patterns. For standard ISO timestamps or syslog formats, the default detection works.

Anatomy of a Working failregex

The <HOST> Placeholder and IP Capture

The <HOST> placeholder expands to a comprehensive pattern that matches both IPv4 addresses (dotted quad) and IPv6 addresses (colon-hex with optional compression). Fail2Ban extracts the matched text as the offending host. Without <HOST>, your regex may match the line but produce no ban because Fail2Ban does not know which part is the IP.

Consider this log line from a web application:

2024-01-15 10:42:33 LOGIN_FAIL user=bob ip=192.0.2.15

A working failregex:

failregex = LOGIN_FAIL user=\S+ ip=<HOST>

Fail2Ban matches LOGIN_FAIL user=bob ip=192.0.2.15, extracts 192.0.2.15, and increments the counter for that address. The \S+ matches the username (one or more non-whitespace characters), then the pattern looks for the literal text ip= followed by <HOST>.

Anchoring and Whitespace Handling

Use ^ to anchor the regex at the start of the line when your log format is consistent. Without an anchor, Fail2Ban matches the pattern anywhere in the line, which works but may be slower or match unintended content in verbose logs. For the example above, you might write:

failregex = ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} LOGIN_FAIL user=\S+ ip=<HOST>

This anchors at the timestamp. Alternatively, skip the timestamp if it varies or if you rely on Fail2Ban's datepattern detection:

failregex = LOGIN_FAIL user=\S+ ip=<HOST>

For whitespace, prefer \s+ or \s* over literal spaces when logs might have tabs or multiple spaces. Use \S+ to match a field (username, session ID) and .* sparingly; greedy quantifiers can cause problems.

Common Pitfalls: Greedy Quantifiers and Ambiguous Patterns

A frequent mistake: using .* before <HOST> in a way that swallows the IP. Consider this broken regex:

failregex = LOGIN_FAIL.*<HOST>

If the log line is LOGIN_FAIL user=bob ip=192.0.2.15 session=abc, the .* matches user=bob ip=192.0.2.15 session=ab greedily, leaving only c for <HOST>, which fails. The corrected version anchors the variable parts:

failregex = LOGIN_FAIL user=\S+ ip=<HOST>

Another pitfall: matching the line but forgetting <HOST>. This regex matches but never bans:

failregex = LOGIN_FAIL user=\S+ ip=\d+\.\d+\.\d+\.\d+

It hard-codes an IPv4 pattern instead of using <HOST>, so Fail2Ban sees a match but extracts no IP. Always use <HOST> where the address appears.

Fail2Ban uses Python's re module, so you have access to standard Python regex syntax. Patterns are case-sensitive by default; prefix with (?i) for case-insensitive matching if your logs vary in case. For detailed syntax, see the Python re module documentation.

Writing Your First Custom Filter

Identify the Log Line and Attack Signature

Start with a real failed authentication event from your application log. Suppose your web app writes to /var/log/webapp.log:

2024-01-15 14:23:11 Auth failed for user alice from 198.51.100.42

The attack signature is Auth failed for user. The variable parts are the username (alice) and the IP (198.51.100.42). Your goal: write a regex that matches this pattern and captures the IP.

Draft the Regex and Add <HOST>

Build the regex step by step. Match the fixed text Auth failed for user, then the username with \S+, then from, then insert <HOST>:

Auth failed for user \S+ from <HOST>

You can anchor it at the timestamp if you want strict matching:

^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} Auth failed for user \S+ from <HOST>

Or rely on Fail2Ban's datepattern auto-detection and skip the timestamp in the regex. For simplicity, the unanchored version works fine here.

Save the Filter File

Create /etc/fail2ban/filter.d/webapp.conf (name it to match your jail or service):

[Definition]
failregex = Auth failed for user \S+ from <HOST>

That is the minimal filter. You can add multiple failregex = lines; Fail2Ban applies them with OR logic, so a match on any line counts. For example, if your app also logs Invalid token from <IP>, add a second line:

[Definition]
failregex = Auth failed for user \S+ from <HOST>
            Invalid token from <HOST>

Optionally, add ignoreregex = to exclude known good patterns (internal IPs, health checks). Save the file and proceed to testing. Never deploy a filter without validating it with fail2ban-regex first.

Testing Filters with fail2ban-regex

The fail2ban-regex tool validates your filter against real or sample log data before you enable the jail. The basic syntax: fail2ban-regex /path/to/logfile /path/to/filter.conf. It reads the log, applies every failregex in the filter, and reports how many lines matched and which IPs it would ban. This catches silent failures (patterns that look correct but match nothing) before they waste production time.

Basic Syntax and Output Interpretation

Create a test log file with a few sample lines. For the webapp filter above, save this to /tmp/test.log:

2024-01-15 14:23:11 Auth failed for user alice from 198.51.100.42
2024-01-15 14:23:15 Successful login for user bob from 203.0.113.10
2024-01-15 14:23:19 Auth failed for user charlie from 198.51.100.42

Run the test:

fail2ban-regex /tmp/test.log /etc/fail2ban/filter.d/webapp.conf

The output shows:

Running tests
=============

Use   failregex filter file : webapp, basedir: /etc/fail2ban
Use         log file : /tmp/test.log
Use         encoding : UTF-8

Results
=======

Failregex: 2 total
|-  #) [# of hits] regular expression
|   1) [2] Auth failed for user \S+ from <HOST>
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [3] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?:\s*Zone offset)?

Lines: 3 lines, 0 ignored, 2 matched, 1 missed
[processed in 0.00 sec]

|- Missed line(s):
|  2024-01-15 14:23:15 Successful login for user bob from 203.0.113.10
`-

Two lines matched (the Auth failed entries), one missed (the successful login). The summary shows 2 matched, and the filter extracted 198.51.100.42 twice. If you see 0 matched, your regex is broken.

Debugging Non-Matching Patterns

Suppose your log uses tabs instead of spaces between fields, but your regex expects spaces. You test and see 0 matched. Add the verbose flag to see what happened:

fail2ban-regex -v /tmp/test.log /etc/fail2ban/filter.d/webapp.conf

The -v flag (or --print-all-matched) shows every line and whether it matched. If nothing matches, check for whitespace mismatches: your regex has literal spaces ( ) but the log has tabs (\t). Fix it with \s+:

failregex = Auth\s+failed\s+for\s+user\s+\S+\s+from\s+<HOST>

Re-run fail2ban-regex. Now you see matches. Another common issue: the timestamp format does not auto-detect. If Fail2Ban cannot parse the date, it ignores the line. The output will show 0 lines processed or Date template hits: 0. Set datepattern = in the filter (covered next section) or verify your log actually has timestamps.

Testing Against Live Logs

Once the filter works on your test file, validate against recent production data. Grab the last 100 lines of the real log:

tail -100 /var/log/webapp.log > /tmp/live-test.log
fail2ban-regex /tmp/live-test.log /etc/fail2ban/filter.d/webapp.conf

Check the output for matches and extracted IPs. If you see legitimate traffic getting matched (internal monitoring, health checks), add an ignoreregex to the filter:

[Definition]
failregex = Auth failed for user \S+ from <HOST>
ignoreregex = from 10\.

This ignores any line with from 10., excluding your internal 10.0.0.0/8 network. Test again to confirm. Always run fail2ban-regex before enabling the jail; a broken filter in production either bans no one or bans everyone.

Handling Multiline Logs and Complex Patterns

Some applications log failures across multiple lines: a summary line followed by detail lines, or a stack trace that spans ten lines. Fail2Ban can match these with the maxlines directive and multiline regex patterns. This is essential for Java applications, JSON error objects, or any log format that does not fit one event per line.

Multiline Mode and the datepattern Directive

Set maxlines = in the [Definition] section to tell Fail2Ban how many consecutive lines to consider as one event. For example, if a login failure logs three lines (timestamp, error message, client IP), set maxlines = 3:

[Definition]
maxlines = 3
failregex = (?m)^Login attempt failed$.*^Client: <HOST>

The (?m) flag enables multiline mode, so ^ and $ match at line breaks within the buffered text. The .* matches everything between the two lines (including newlines, if you use (?s) as well). Fail2Ban buffers up to maxlines lines, concatenates them with newlines, and applies the regex. If it matches, it extracts the IP.

For non-standard timestamps, set datepattern =. If Fail2Ban cannot parse your date format, it skips the line. Most common formats (ISO 8601, syslog, Apache combined) auto-detect, but custom formats need explicit patterns. For example, if your log uses 15/Jan/2024:14:23:11, you might write:

datepattern = {^LN-BEG}Day/MON/Year:24hour:Minute:Second

The {^LN-BEG} prefix tells Fail2Ban the timestamp is at the start of the line. Consult the Fail2Ban filter documentation for the full datepattern syntax and named patterns. For most logs, omit datepattern and let auto-detection work.

Matching Stack Traces and Wrapped Messages

A Java application logs authentication failures as:

2024-01-15 14:30:00 ERROR AuthService
Authentication failed for user admin
Client IP: 203.0.113.99

The IP is on the third line. Set maxlines = 3 and write a regex that spans all three:

[Definition]
maxlines = 3
failregex = (?m)^ERROR AuthService$\nAuthentication failed for user \S+$\nClient IP: <HOST>

The \n matches the literal newline character in the buffered string. Test with fail2ban-regex using a sample file that has the exact spacing and line breaks. If your log uses \r\n (Windows-style), adjust accordingly or use \s+ to match any whitespace including newlines.

For JSON logs that span multiple lines, you can match the opening brace on one line and the IP field on another. Set maxlines high enough to cover the entire object (5 or 10 lines), then use a regex that matches the relevant keys. This works but can be slow; prefer single-line JSON if you control the log format.

Creating and Enabling the Jail

With a tested filter, you create a jail configuration that ties the filter to a log file and defines ban parameters. Jails live in /etc/fail2ban/jail.local or in separate files under /etc/fail2ban/jail.d/. The jail.d approach keeps custom jails isolated, making upgrades cleaner.

Jail Configuration: logpath, filter, bantime

Create /etc/fail2ban/jail.d/webapp.conf with this content:

[webapp]
enabled  = true
filter   = webapp
logpath  = /var/log/webapp.log
maxretry = 5
findtime = 10m
bantime  = 1h

Each key:

• enabled = true: activates the jail on Fail2Ban restart.
• filter = webapp: references /etc/fail2ban/filter.d/webapp.conf.
• logpath = /var/log/webapp.log: the file Fail2Ban tails for new lines.
• maxretry = 5: ban after 5 failures.
• findtime = 10m: count failures within the last 10 minutes.
• bantime = 1h: ban for 1 hour (use -1 for permanent bans).

Restart Fail2Ban to load the jail:

systemctl restart fail2ban.service

Check that the jail is active:

fail2ban-client status webapp

You should see output like:

Status for the jail: webapp
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/webapp.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

The jail is watching /var/log/webapp.log. If File list is empty or the command fails, check for typos in the jail name or logpath.

Testing the Jail Without Banning Production IPs

Before going live, test the jail with a short ban time so you do not lock yourself out. Edit the jail to set bantime = 10 (10 seconds):

[webapp]
enabled  = true
filter   = webapp
logpath  = /var/log/webapp.log
maxretry = 3
findtime = 5m
bantime  = 10

Restart Fail2Ban, then trigger failures yourself. If your app is a web service, make bad login requests:

for i in {1..3}; do curl -X POST http://localhost:8080/login -d "user=test&pass=wrong"; done

Watch /var/log/fail2ban.log for the ban:

tail -f /var/log/fail2ban.log

You should see lines like:

2024-01-15 14:35:22,123 fail2ban.actions [12345]: NOTICE  [webapp] Ban 127.0.0.1

Check the banned IP list:

fail2ban-client get webapp banned

Or inspect the iptables chain:

iptables -L f2b-webapp -v -n

You will see a DROP rule for your IP. After 10 seconds, Fail2Ban unbans automatically. To unban manually:

fail2ban-client unban 127.0.0.1

Once you confirm the jail works, set bantime back to a production value (1h or 24h), increase maxretry to a reasonable threshold (5 or 10), and restart Fail2Ban. Your custom filter is now live.

Common Regex Mistakes and How to Fix Them

The most frequent mistake: forgetting <HOST> or placing it where the IP does not appear. This regex matches the line but extracts nothing:

failregex = Auth failed for user \S+ from \d+\.\d+\.\d+\.\d+

Symptom: fail2ban-regex shows 2 matched, but fail2ban.log reports Unable to find a corresponding IP address for ... and no bans occur. The fix: replace the IP pattern with <HOST>:

failregex = Auth failed for user \S+ from <HOST>

Another common error: greedy .+ that consumes the IP. Given Auth failed from 203.0.113.42 user admin, this broken regex fails:

failregex = Auth failed from .+ user <HOST>

The .+ matches 203.0.113.42 user admi, leaving only n for <HOST>. Fix it by anchoring variable parts: Auth failed from <HOST> user \S+.

Case mismatch is silent. If your log says Failed login but the regex says failed login, you get zero matches. Python regex is case-sensitive by default. Either match the exact case or prefix with (?i) for case-insensitive matching: (?i)failed login.

Forgetting to escape special characters breaks patterns. Literal brackets in logs like [AUTH] must be escaped: \[AUTH\], not [AUTH] (which is a character class). The symptom: fail2ban-regex shows error: nothing to repeat at position ... or 0 matched. Check the regex syntax with the Regex Quick Reference cheat sheet.

Finally, including the timestamp in the failregex when Fail2Ban already strips it causes silent failures. If you write ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} Auth failed but Fail2Ban auto-detects and removes the timestamp before applying the regex, the pattern never matches. Either omit the timestamp from the regex or verify that datepattern is not stripping it.

Performance and Maintenance

Fail2Ban scans every new log line against every failregex in the filter. A slow regex can lag the scanner, especially on high-traffic logs. Avoid heavy backtracking: nested quantifiers like (.*)+ or alternations with overlapping patterns cause exponential matching attempts. This bad pattern is slow:

failregex = (Auth|Login|Signin).*(failed|error|denied).* from <HOST>

The multiple .* and alternations force the engine to explore many paths. Rewrite it with specific anchors and one \S+ for the variable field:

failregex = (?:Auth|Login|Signin) (?:failed|error|denied) for user \S+ from <HOST>

This anchors the keywords and matches the username once, reducing backtracking. Test performance with fail2ban-regex --print-all-missed on a large log file; if it takes more than a second or two, simplify the regex.

Log rotation is automatic: Fail2Ban detects when a log file is rotated (via inode change) and reopens the new file. If you use glob patterns in logpath (e.g., /var/log/app/*.log), Fail2Ban watches all matching files. Verify the glob expands correctly with fail2ban-client get webapp logpath.

To update a filter in production, edit the .conf file in /etc/fail2ban/filter.d/, then reload the specific jail:

fail2ban-client reload webapp

Or reload all jails with:

systemctl reload fail2ban.service

Check /var/log/fail2ban.log immediately after reload for errors like error: invalid regular expression. If you see errors, the old filter remains active until you fix the syntax. Version-control your filters in /etc/fail2ban/filter.d/ with Git to track changes and roll back mistakes. Add a comment at the top of each filter with the date and reason for the last change.

Pre-Deployment Checklist

Before you enable a custom Fail2Ban jail in production, verify these points to avoid locking out legitimate users or leaving attackers unbanned:

• Run fail2ban-regex against recent production logs (last 500-1000 lines) and confirm matches. Zero matches means a broken regex; thousands of matches may indicate overly broad patterns catching legitimate traffic.
• Check that <HOST> appears exactly once in each failregex and captures the attacker's IP, not a local address or NAT gateway.
• Set a short bantime (10 seconds) and low maxretry (2 or 3) for initial testing. Trigger failures yourself and confirm the ban appears in iptables -L f2b-<jailname> or fail2ban-client status <jailname>.
• Add ignoreregex patterns for internal monitoring, health checks, or known-good IPs (office networks, monitoring services). Test that these lines are ignored by fail2ban-regex.
• Review /var/log/fail2ban.log after enabling the jail. Look for Unable to find a corresponding IP address (missing <HOST>) or error: invalid regular expression (syntax error).
• Document the filter: add a comment block at the top of the .conf file explaining what it matches, which log format it expects, and an example log line. Future you (or a colleague) will thank you.
• If you manage multiple servers, version-control /etc/fail2ban/filter.d/ and /etc/fail2ban/jail.d/ in a private Git repository. Deploy filters with Ansible, Puppet or a simple rsync script to keep them consistent.

For deeper coverage of regex patterns in system administration contexts, see the Regex Mastery for System Administrators eBook. It covers advanced anchoring, lookarounds, and performance tuning for log parsing at scale.

Frequently Asked Questions

Why does my Fail2Ban filter match in fail2ban-regex but not ban anyone?

The most common cause: your failregex is missing <HOST> or places it in the wrong position, so Fail2Ban cannot extract an IP address. Check /var/log/fail2ban.log for Unable to find a corresponding IP address. Another possibility: the jail is not enabled (enabled = false in the jail config) or the logpath is wrong. Run fail2ban-client status jailname to verify the jail is active and watching the correct file.

How do I capture an IP address that appears in JSON logs?

JSON logs often quote the IP as a value: "client_ip": "203.0.113.42". Your failregex should match the key and use <HOST> for the value: "client_ip":\s*"<HOST>". If the JSON spans multiple lines, set maxlines in the filter and use (?m) multiline mode. Test with fail2ban-regex on a sample log to confirm the pattern extracts the IP correctly.

Can I test a Fail2Ban filter without restarting the service?

Yes. Use fail2ban-regex /path/to/logfile /etc/fail2ban/filter.d/filtername.conf to test the filter offline. This validates the regex and shows which lines match without affecting running jails. Once the filter works, you can reload a specific jail with fail2ban-client reload jailname instead of restarting the entire service. The jail picks up the updated filter without dropping existing bans or interrupting other jails.

What does 'Unable to find a corresponding IP address' mean in Fail2Ban logs?

This error appears when your failregex matches a log line but does not capture an IP address. The usual cause: you forgot <HOST> or used a hard-coded IP pattern like \d+\.\d+\.\d+\.\d+ instead of the placeholder. Fail2Ban needs the <HOST> tag to know which part of the match is the offending address. Fix the regex by replacing the IP pattern with <HOST> and re-test with fail2ban-regex.

How do I write a failregex for logs with no fixed timestamp position?

If the timestamp moves or is absent, omit it from the failregex and rely on Fail2Ban's datepattern auto-detection. Fail2Ban extracts the timestamp separately before applying the regex. Your failregex should match only the failure message and IP: Auth failed for user \S+ from <HOST>. If the log truly has no timestamp, Fail2Ban cannot determine event order or apply findtime correctly; you may need to add timestamps at the application level or use datepattern = NONE (which disables time-based filtering).