Growing Pains with Five Repositories: Gitlinks and Dual CI

Usually, infrastructure stories start with the words: "Our project got too big, and we decided to split it into parts." But with us, everything was different. From the very first minute of the project, we consciously chose an architecture consisting of a scatter of independent repositories. The Pipeline, engine, design-docs, and user-docs projects were created almost simultaneously, and the .gitmodules configuration file sat in the very first commit of the parent repository. For the sake of code cleanliness, CI isolation, independent history, and separation of access rights, we built the system on Git Submodules from day one. The idea looked flawless on paper. But here is what we had to pay for it in practice. The Gitlink Trap: The Ghost of a Local Commit The first price we paid was the non-obvious mechanics of state synchronization. When you use submodules, the workflow demands strict discipline. If you edit code inside engine/ (while physically located in the folder of the parent repository Pipeline), you need to make two pushes. First, a git push from the folder of the submodule itself, and then, after moving up to the root of Pipeline—you must commit the updated pointer (gitlink) to the new engine commit and push Pipeline itself. It is incredibly easy to forget the first push. On your local disk, the commit physically exists. Everything builds, the tests are green, and Pipeline happily goes to the cloud. But this only surfaces on a fresh checkout in CI. The git submodule update --init command fails with the text: fatal: remote error: upload-pack: not our ref Enter fullscreen mode Exit fullscreen mode ext GitHub honestly replies: "This commit does not exist in the remote repository." One day, three submodules fell into this trap simultaneously (design-docs, engine, user-docs). The cause was identical—small commits were made right inside the submodule folders, the gitlink in Pipeline was synchronized, but the local commits never left the developer's laptop. There was also a colorful nuance: the fourth submodule, ude_promotion (which hosts the site), broke so badly that a normal push couldn't fix it—its local branch fell so far behind upstream that an interactive rebase was required before pushing. Dual CI: One Code in Two Personalities The second reckoning came from CI. The engine repository (our core) had two personalities. On the one hand, it was a standalone repository with its own build process, which checked out only this repository. On the other—it was a submodule inside Pipeline, which was tested in the context of the entire parent project. These two personalities inevitably clashed, and this happened in two different stages. The First Strike: Missing Dependencies The simplest case. The test_integration_scripts.py test tried to reach the verify_pages/check_links modules, which existed only in the parent Pipeline repository. During a standalone checkout of engine, these files simply weren't on the disk. We fixed it roughly but effectively—we added a single line to the ci.yml configuration, strictly ignoring this test from the standalone run (pytest --ignore=...). But this was a one-off patch, not a systemic solution. The Fatal Directory Ascent Soon the bug returned in a new guise. One by one, tests started failing that used a classic trick to find parent artifacts: Path(__file__).resolve().parents[2] Enter fullscreen mode Exit fullscreen mode ext In a nested run, this code ascended exactly to the root of the parent repository. But in the standalone build of engine, it jumped above the root of the repository, straight into the system filesystem of the CI runner, and crashed with a hard AssertionError. It was after the second occurrence of this bug that we realized point patches no longer worked. We needed a general convention. We introduced an elegant marker hack: tests began checking for the presence of our system marker directory (let's call it .workspace_config) at the ascended level. If the directory is there—it means we are in a nested context, and the tests run. If it's not there (standalone)—we politely use pytest.mark.skipif, and the tests are quietly skipped without failing with a red status. Conclusion A Multi-repo architecture does not give you magical "loose coupling" for free on day one. It demands flawless push synchronization discipline and forces you to design code so that it is ready to work in several different execution contexts simultaneously. These lessons cost us broken pipelines, but without them, the system would not have survived further growth. But the problems with Git and dual CI turned out to be just a prelude. The real mysticism began when the network intervened, and our pipeline started crashing because of a "ghost" in the wires. How we investigated an elusive network bug—read about it in the next episode. Originally published on our blog: https://blog.flude.guide/blog/multi-repo-gitlinks-dual-ci Also read us: Telegram channel RSS feed