One Missed Test Case Cost Me 8 Hours — How I Built a Zero-Regression Memory Test Suite with Pytest + Docker

I got woken up by an alert call at 2 a.m. – the production chatbot had suddenly developed “amnesia.” It was completely unable to recall order information we had just discussed that morning. I scrambled to investigate and discovered the root cause: we had migrated the memory storage from Redis to PostgreSQL + pgvector. The migration script looked fine, but one historical memory entry’s vector index hadn’t been properly built, causing the similarity recall to return empty results. Before going live, I had manually spot-checked 20 conversations as a regression check – and you guessed it, the one broken entry was the 21st I missed. From that moment on, I resolved that regression testing for LLM memory storage must be automated, and every change must replay historical memories. This post shares how I built that automated regression suite with Pytest + Docker, along with two near-fatal pitfalls along the way. Problem Breakdown: Why Memory Storage Breaks Silently “LLM memory storage” is the mechanism that stores chat histories, user preferences, summary snippets, etc., so the bot can retrieve them later via vector similarity or keyword search and inject them into prompts as context. Common implementations include Redis (with RediSearch/Vector modules), Milvus, PostgreSQL + pgvector, or even local FAISS files. The biggest engineering pain point isn’t “can we store it?” but rather “does the system still remember after we upgrade?” Memory storage has many churn points: switching the underlying database, changing the embedding model, tweaking similarity thresholds, altering expiration policies, or even bumping a minor pgvector version. Any of these can cause missing memories, score deviations, or empty result lists – but traditional manual tests only cover a few happy paths and simply can’t cover weeks of accumulated historical data. The conventional approaches aren’t great either: full end-to-end load testing with manual comparison is slow and irreproducible; mocking the storage layer to test business logic bypasses the exact vector retrieval logic that’s most fragile. What you really need is an environment that can quickly spin up a real storage instance, replay real historical data, and assert that every memory is correctly recalled. That’s exactly where Pytest + Docker shine. Design Decisions: Why Not an Off-the-Shelf Integration Test Platform? I considered several options: A permanent test database: requires dedicated infrastructure, easily polluted by test data, and breaks when multiple people run tests in parallel. Pure unit tests with mocks: never exercises real vector retrieval behavior, so the false-negative rate is dangerously high. Pytest + Docker combo: each test session pulls a clean container and destroys it afterwards – perfect isolation. Plus, you can parameterize tests against multiple storage backends easily. The core idea is to use testcontainers-python to dynamically manage storage containers. During regression tests, you load a snapshot of historical data (sampled from production or generated golden files), then call the memory storage service’s query APIs and assert that the recalled IDs, scores, and ordering match expectations. If someone changes storage logic or upgrades a model, running this regression suite will tell you within 10 minutes whether old memories are broken. Core Implementation: From Container Management to Regression Assertions 1. Managing Real Storage Environments with testcontainers This code solves “each test needs a fresh, disposable database.” I chose PostgreSQL + pgvector because it’s the most common and hides the most surprises. # conftest.py import pytest from testcontainers.postgres import PostgresContainer from testcontainers.redis import RedisContainer import psycopg2 import redis # Custom pgvector container: the official postgres image doesn't include pgvector class PostgresWithVector(PostgresContainer): def __init__(self, image="pgvector/pgvector:pg16"): super().__init__(image=image) self.with_env("POSTGRES_USER", "test") self.with_env("POSTGRES_PASSWORD", "test") self.with_env("POSTGRES_DB", "memory_test") @pytest.fixture(scope="session") def pg_vector_store(): """Start a pgvector container and return usable connection params""" postgres = PostgresWithVector() postgres.start() # Get the dynamically mapped host port host = postgres.get_container_host_ip() port = postgres.get_exposed_port(5432) dsn = f"postgresql://test:test@{host}:{port}/memory_test" # Initialize schema and vector extension conn = psycopg2.connect(dsn) with conn.cursor() as cur: cur.execute("CREATE EXTENSION IF NOT EXISTS vector;") cur.execute(""" CREATE TABLE IF NOT EXISTS memory_blocks ( id TEXT PRIMARY KEY, user_id TEXT, content TEXT, embedding vector(1536), created_at TIMESTAMP DEFAULT NOW() ); CREATE INDEX ON memory_blocks USING ivfflat (embedding vector_cosine_ops); """) conn.close() yield {"dsn": dsn, "host": host, "port": port} postgres.stop() @pytest.fixture(scope="session") def redis_memory_store(): """Similarly, use Redis as an alternative storage for comparison tests""" redis_container = RedisContainer("redis/redis-stack-server:latest") redis_container.start() host = redis_container.get_container_host_ip Enter fullscreen mode Exit fullscreen mode