Log Parsing with Regex

Parse web server logs with Python's re module in your browser: capture fields, name groups, count status codes and convert timestamps.

Python
import re

log_entry = '192.168.1.100 - - [20/Feb/2024:10:23:15 +0000] "GET /index.html HTTP/1.1" 200 2326'
pattern = r'(\d+\.\d+\.\d+\.\d+) .+ \[(.+)\] "(\w+) (.+) HTTP/\d\.\d" (\d+) (\d+)'
match = re.search(pattern, log_entry)
if match:
    ip, timestamp, method, path, status, bytes_sent = match.groups()
    print(f"IP: {ip}")
    print(f"Timestamp: {timestamp}")
    print(f"Method: {method}")
    print(f"Path: {path}")
    print(f"Status: {status}")
    print(f"Bytes Sent: {bytes_sent}")
IP: 192.168.1.100
Timestamp: 20/Feb/2024:10:23:15 +0000
Method: GET
Path: /index.html
Status: 200
Bytes Sent: 2326

Python's built-in re module finds text that matches a regular expression, a pattern that describes what the text looks like. Log files suit it well: each line follows a fixed layout, and a pattern with capture groups pulls out fields such as the client address, time, request and status code. On this page it runs in your browser, so you can try it without installing anything. Run the example first, then paste any snippet below into a new cell to try it.

What the example does

log_entry is one line of a web server access log. Each pair of parentheses in the pattern captures a field. (\d+\.\d+\.\d+\.\d+) is the IP address, with \. matching a literal dot. The .+ between two spaces skips the two - fields. \[(.+)\] captures the timestamp; the brackets are escaped because [ normally starts a character set. "(\w+) (.+) HTTP/\d\.\d" takes the method and path, and the last two groups take the status code and bytes sent. re.search() returns a match object, or None if the line doesn't fit. match.groups() returns the six captured strings in order, so even the status is the string "200". To try a pattern on your own lines, use the regex tester.

Name the groups

(?P<name>...) names a group, match["name"] reads it, and groupdict() returns every field as a dictionary. With re.VERBOSE, whitespace in the pattern is ignored and # starts a comment, so a long pattern can span several lines:

import re

LOG_PATTERN = re.compile(r"""
    (?P<ip>\S+)\s\S+\s\S+\s              # client IP, then two unused fields
    \[(?P<time>[^\]]+)\]\s               # timestamp inside [ ]
    "(?P<method>\w+)\s(?P<path>\S+)\s[^"]*"\s
    (?P<status>\d{3})\s(?P<size>\d+|-)   # status code, then bytes sent or -
""", re.VERBOSE)

line = '192.168.1.100 - - [20/Feb/2024:10:23:15 +0000] "GET /index.html HTTP/1.1" 200 2326'
match = LOG_PATTERN.match(line)
print(match["path"], match["status"])
print(match.groupdict())

[^\]]+ means one or more characters other than ], and \d+|- accepts a number or a dash.

Parse a whole log and count status codes

re.compile() turns the pattern into a reusable object. Its match() method only matches at the start of a line and returns None for lines that don't fit, which the list comprehension drops:

import re
from collections import Counter

log = """\
192.168.1.100 - - [20/Feb/2024:10:23:15 +0000] "GET /index.html HTTP/1.1" 200 2326
10.0.0.7 - - [20/Feb/2024:10:23:18 +0000] "GET /missing.png HTTP/1.1" 404 512
192.168.1.100 - - [20/Feb/2024:10:24:02 +0000] "POST /login HTTP/1.1" 302 -
2001:db8::1 - - [20/Feb/2024:10:24:09 +0000] "GET /index.html HTTP/1.1" 200 2326
this line is not a log entry
10.0.0.9 - - [20/Feb/2024:10:25:40 +0000] "GET /api/data HTTP/1.1" 500 87
10.0.0.7 - - [20/Feb/2024:10:25:41 +0000] "GET /missing.png HTTP/1.1" 404 512
"""

pattern = re.compile(r'(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "(?P<request>[^"]*)" (?P<status>\d{3}) (?P<size>\d+|-)')
entries = [m.groupdict() for m in map(pattern.match, log.splitlines()) if m]

print(len(entries), "of", len(log.splitlines()), "lines parsed")
print(Counter(e["status"] for e in entries).most_common())
print(Counter(e["request"] for e in entries if int(e["status"]) >= 400))

Convert fields to numbers and dates

Captured groups are always strings. int() converts the status, and datetime.strptime() reads the timestamp: %b is the abbreviated month name and %z the offset from UTC.

import re
from datetime import datetime

line = '192.168.1.100 - - [20/Feb/2024:10:24:02 +0000] "POST /login HTTP/1.1" 302 -'
m = re.match(r'(\S+) \S+ \S+ \[([^\]]+)\] "[^"]*" (\d{3}) (\d+|-)', line)
ip, stamp, status, size = m.groups()

when = datetime.strptime(stamp, "%d/%b/%Y:%H:%M:%S %z")
status = int(status)
size = 0 if size == "-" else int(size)
print(repr(when), status, size)
print(when.strftime("%Y-%m-%d %H:%M"), when.tzinfo)

Pull key=value pairs from application logs

Application logs often carry details as key=value pairs. With two groups in the pattern, re.findall() returns (key, value) tuples, which dict() turns into a dictionary:

import re

lines = [
    "2024-02-20 10:23:15,120 INFO user=alice action=login ms=42",
    "2024-02-20 10:23:16,884 ERROR user=bob action=upload ms=1530 error=timeout",
    "2024-02-20 10:23:17,002 WARNING user=alice action=upload ms=870",
]

for line in lines:
    level = re.search(r"\b(INFO|WARNING|ERROR)\b", line).group(1)
    fields = dict(re.findall(r"(\w+)=(\S+)", line))
    print(level, fields)

slow = [line for line in lines if int(re.search(r"ms=(\d+)", line).group(1)) > 500]
print(len(slow), "requests took over 500 ms")

Good to know

  • .+ is greedy: it matches as much as it can while the rest of the pattern still fits. On [2024-02-20 10:23:15] [ERROR] Disk full, \[(.+)\] captures 2024-02-20 10:23:15] [ERROR. Use [^\]]+ or the lazy .+? to stop at the first ].
  • A line that doesn't fit the pattern is skipped without an error. The example's pattern misses two valid lines in the log above: the one with - as its size, and the IPv6 one, which \d+\.\d+\.\d+\.\d+ can't match. Count skipped lines so gaps show up.