How it works
Start by defining which actions require evidence: permission changes, account recovery, refunds, or administrator access. Unlike debug logs, required audit events should not be sampled or silently dropped. Record successful changes and, where required, denied or failed attempts.
Use a structured event with an event ID, tenant ID, authenticated actor, action, resource ID, outcome, server timestamp, and request ID. Include relevant changed fields or resource versions, but exclude passwords, tokens, and unnecessary personal data. For delegated actions, distinguish the initiating user from the service executing the request.
Capture committed changes atomically with their audit record. For example, update a user's role and insert an audit event or outbox entry in the same database transaction; a worker then delivers it to the audit store. A denied attempt needs a separate durable capture path because a rolled-back business transaction cannot preserve its audit entry.
Deliver events with retries and deduplicate by event ID at the destination. Separate occurrence time from ingestion time, and use resource versions or sequence numbers when change order matters; timestamps alone do not establish order across machines. Monitor delivery lag, rejected records, and capture coverage rather than assuming a healthy queue proves completeness.
Keep the authoritative history append-only, with corrections recorded as new events. Use restricted writers, separate administrative privileges, and retention-locked storage where required; build searchable indexes as derived views. Hash chains with independently protected checkpoints can reveal changes to recorded history, but they cannot prove that every action was captured.
A worked example
Auditing merchant refunds
Suppose a payment service processes 100 refunds per second. Each successful refund transaction inserts the refund, updates the refundable amount, and writes an outbox event containing the event ID, merchant ID, actor ID, payment ID, amount, outcome, and payment version. A worker copies events to the audit store with at-least-once delivery; a unique event ID prevents duplicate records. If the worker stops for 10 minutes, 60,000 events accumulate, but committed refunds retain their evidence in the database. Search may lag, so the audit UI reports ingestion freshness. A denied refund attempt uses a separate durable capture path. The financial ledger remains responsible for balances; the audit trail explains who initiated each action and its result.
Trade-offs to say out loud
- Completeness versus availability
- Failing a sensitive operation when durable audit capture is unavailable preserves evidence but makes audit infrastructure part of that operation's availability boundary.
- Atomic capture versus integration effort
- Transactional audit inserts or an outbox prevent database-to-log gaps, but every relevant mutation path must participate.
- Durability versus search freshness
- Asynchronous indexing keeps search failures off the write path, but investigators may need to wait for recent events to become searchable.
- Detail versus privacy
- Rich change records help investigations but increase exposure, so field selection, redaction, encryption, and retention need explicit policies.
- Tamper resistance versus operational flexibility
- Retention locks and separated privileges make unauthorized changes harder while also making accidental sensitive-data capture harder to remediate.
- Retention versus cost
- Long retention increases storage and indexing costs, so older events often move to cheaper storage with slower retrieval.
When to reach for it
- The prompt requires financial history, dispute investigation, or an explanation of how balances changed.
- Administrators can change permissions, credentials, configuration, or ownership.
- Users or investigators need to answer who changed a resource and reconstruct the sequence of changes.
- The requirements include compliance evidence, retention periods, or protection against privileged tampering.
How to bring it up in the interview
- Name the audited actions and required evidence before choosing storage; interviewers may ask whether reads, denials, and administrator impersonation are covered.
- Draw the business-write-to-audit boundary and explain a crash between steps; defend a transactional audit insert or outbox instead of an uncoordinated dual write.
- State exactly when success is acknowledged and what happens when audit capture or downstream delivery fails; distinguish durable capture from searchable availability.
- Separate completeness, integrity, and access control: interviewers may probe missing events, privileged deletion, tenant isolation, and whether audit-log access is itself audited.
- Give concrete query patterns and retention requirements, then explain indexes, archive retrieval, deduplication, and ordering without promising a global timestamp order.
Common mistakes
- Treating sampled application logs or distributed traces as a complete audit trail.
- Updating business state and then making a best-effort network call to the audit service.
- Claiming append-only tables or hash chains alone prevent privileged tampering or prove completeness.
- Logging credentials, full request bodies, or unrestricted before-and-after snapshots.
- Confusing an audit log with a financial ledger or event sourcing: it records evidence, but is not automatically sufficient to calculate balances or rebuild application state.
Where it comes up
Interview questions where audit logs is part of a strong answer, and why.
- Design a payment systemAuditable balances, refunds, and payouts require durable evidence linking actors and requests to financial state changes.
- Design a wire transfer systemTransfer investigations and reconciliation require a reliable history of instructions, external responses, and settlement outcomes alongside the ledger.
- Design a login and authentication systemAccount recovery, credential changes, and session revocation need security audit records covering both successful actions and rejected attempts.
Questions people ask
What is the difference between an audit log and an application log?
Application logs primarily help engineers debug and operate software; they may be sampled, rotated quickly, or changed freely. Audit logs preserve defined evidence about actions and outcomes, with explicit completeness, access, integrity, and retention requirements.
How do you prevent audit logs from being tampered with?
Restrict writes to an append-only ingestion path, separate administrative privileges, and use retention-locked storage when required. Hash chains or signed batches with independently protected checkpoints provide tamper evidence, not absolute prevention. They also cannot detect actions that were never captured.
Should audit logging be synchronous or asynchronous?
Durable capture should be part of the success boundary for operations that must never succeed without evidence. Delivery to a central store and search indexing can happen asynchronously through an outbox. Explicitly decide whether operations fail closed when durable capture is unavailable.