Before You Burn GPU Hours, Smoke-Test One Task on a Cheap CPU Runtime

This article was originally published in Japanese on Zenn. Introduction Hi, I'm Kohei Matsunobe, a second-year student in the online (correspondence) program at ZEN University. I'm working toward becoming an AI engineer, and on my own I run experiment pipelines that evaluate LLM outputs and code on notebooks and cloud runtimes. GPU runtimes are great, but they're expensive per hour, and there's a cap on how many compute units you can use. I've made the same mistakes more than once: "I grabbed a GPU, launched the real job, and it crashed in the first few minutes," or "it crashed, I didn't notice, and the runtime just sat there running." Looking back, almost all of those failures could have been caught without a GPU. In this post, I'll walk through the idea of a "smoke test" — a minimal check you run on a cheap CPU runtime before moving up to the expensive one — plus a small script you can use right away. What tends to go wrong In pipelines that evaluate LLMs or generated code, these problems come up a lot: Missing test dependencies: Your setup grades generated code with pytest, but the production runtime doesn't have pytest, or the libraries the tested code needs. Cache or data not found: The path where you thought you'd put model weights or dataset caches is empty on the fresh runtime. So downloads run every time, or you get errors because offline mode is still on. Mismatched environment variables or paths: It worked locally, but on a different runtime the working directory or cache location has changed. An exception on the very first task: Simple mistakes, like a missing output directory or a misspelled argument name. None of these have anything to do with GPU performance. But when they happen on a GPU runtime, you lose all the time spent on setup and model loading too. Retry the job enough times and you might suddenly find you've used up your compute units. The idea behind the smoke test It boils down to three points: Keep dependencies identical to production: Build the environment from the same definition as production — the same requirements file, lock file, or container image. If you maintain a separate dependency list for CPU, you hide the very differences you want to catch. (The only exception: packages that need a GPU-specific build get handled separately.) Run just one task on the cheap runtime: Don't run the whole eval set — just one item. Inference may be slow on CPU, but all you need to know is "does it make it to the end?" Promote to the expensive runtime only if it passes: Only when the smoke test succeeds do you run the real job on the GPU runtime, with the same code and the same config. If you think of it as a CI pipeline, it's a staged gate: put a light job in front of the heavy one, and don't move on unless the light one passes. The minimum checklist Here's what I check every time: Can the key packages be imported? (Including pytest for evaluation) Do the cache and data directories exist, and are they non-empty? Are the required environment variables set? Can I write to the output directory? Does at least one pytest test actually pass? Can the same entry point as production run to completion as a dry run with just one task? You don't need to do all of these perfectly. Just adding "anything that has bitten me even once" to the list goes a long way. A small script Here's a script that bundles the checks above. Adjust REQUIRED_MODULES, REQUIRED_PATHS, and run_task to fit your own pipeline. #!/usr/bin/env python3 # smoke_test.py """Smoke test to run on a cheap runtime before moving to an expensive one.""" from __future__ import annotations import argparse import importlib import json import os import subprocess import sys from pathlib import Path # Modules to verify with the same dependency stack as production REQUIRED_MODULES = ["numpy", "pytest", "transformers"] # Cache/data paths that should exist and have contents REQUIRED_PATHS = [ Path(os.environ.get("HF_HOME", "~/.cache/huggingface")).expanduser(), Path("data"), ] REQUIRED_ENV = ["HF_HOME"] def check_imports(modules: list[str]) -> list[str]: errors = [] for name in modules: try: importlib.import_module(name) except Exception as e: # also catch init errors, not just ImportError errors.append(f"import failed: {name} ({type(e).__name__}: {e})") return errors def check_paths(paths: list[Path]) -> list[str]: errors = [] for p in paths: if not p.exists(): errors.append(f"path missing: {p}") elif p.is_dir() and not any(p.iterdir()): errors.append(f"directory is empty: {p}") return errors def check_env(names: list[str]) -> list[str]: return [f"env var not set: {n}" for n in names if not os.environ.get(n)] def check_writable(out_dir: Path) -> list[str]: try: out_dir.mkdir(parents=True, exist_ok=True) probe = out_dir / ".smoke_write_test" probe.write_text("ok", encoding="utf-8") probe.unlink() except OSError as e: return [f"not writable: {out_dir} ({e})"] return [] def run_one_pytest(test_id: str) -> list[str]: # e.g. tests/test_eval.py::test_single_case result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", test_id]) # Exit code 5 means "no tests were collected", so treat it as a failure too if result.returncode != 0: return [f"pytest failed: {test_id} (exit={result.returncode})"] return [] def load_tasks(path: Path) -> list[dict]: with path.open(encoding="utf-8") as f: return [json.loads(line) for line in f if line.strip()] def run_task(task: dict) -> dict: # Replace this with the same processing as production (inference, grading) return {"id": task.get("id"), "status": "ok"} def main() -> int: parser = argparse.ArgumentParser(description="cheap runtime smoke test") parser.add_argument("--tasks", type=Path, help="JSONL file, one task per line") parser.add_argument("--out", type=Path, default=Path("outputs")) parser.add_argument("--pytest", metavar="TEST_ID", help="run a single test ID") parser.add_argument("--dry-run", action="store_true", help="run only one task") args = parser.parse_args() errors: list[str] = [] errors += check_imports(REQUIRED_MODULES) errors += check_paths(REQUIRED_PATHS) errors += check_env(REQUIRED_ENV) errors += check_writable(args.out) if args.pytest: errors += run_one_pytest(args.pytest) if errors: print("[NG] Pre-flight checks found problems:") for e in errors: print(" -", e) return 1 if args.tasks: tasks = load_tasks(args.tasks) if args.dry_run: tasks = tasks[:1] for task in tasks: result = run_task(task) print(json.dumps(result, ensure_ascii=False)) print("[OK] Smoke test passed. Safe to promote to the expensive runtime.") return 0 if __name__ == "__main__": sys.exit(main()) Enter fullscreen mode Exit fullscreen mode On the cheap CPU runtime, run it like this: python smoke_test.py --tasks data/tasks.jsonl --dry-run \ --pytest tests/test_eval.py::test_single_case Enter fullscreen mode Exit fullscreen mode If you only start the real run when the exit code is 0, the same pattern works in notebooks and in CI: python smoke_test.py --tasks data/tasks.jsonl --dry-run && python run_full_eval.py Enter fullscreen mode Exit fullscreen mode Wrap-up Most expensive GPU job failures come from things that have nothing to do with the GPU: dependencies, caches, and path settings. Keep the same dependency stack as production, push just one task through on a cheap CPU runtime, and promote to the expensive runtime only when it passes. For the checklist, adding things you've personally been bitten by, little by little, is plenty. A few minutes of smoke testing protects both your compute units and your motivation. I hope this helps anyone else running experiments on their own!