Building a Gomoku AI from Scratch — C++, pybind11, and a Lot of Pain

I wanted to build a Gomoku AI. Not a rule-based one that just looks up patterns — a real one, trained end-to-end with deep reinforcement learning, the kind that gets better by playing against itself. So I did what any reasonable person would do: I picked the most annoying possible tech stack and started digging.

This post is about the first phase: getting the C++ engine talking to Python without everything exploding. Spoiler — it did explode. Several times.

The Plan

The idea is a classic AlphaZero-style setup:

C++ Board Engine  ──(pybind11)──▶  Python MCTS  ──▶  PyTorch Neural Net
                                        │                     │
                                        └──── self-play ◀─────┘
                                              keeps going

Why C++ for the engine? Because MCTS needs to simulate thousands of board positions per second. Pure Python just can't keep up. C++ handles the heavy lifting; Python handles the brains.

Tech stack at a glance:

Straightforward enough on paper. In practice? Let me walk you through it.

The Board: What the AI Actually "Sees"

Before anything else, I needed to decide how to represent the board state as a tensor — because this is literally what gets fed into the neural network.

I went with 3-Channel Spatial Binary Planes, borrowed from the AlphaGo paper:

Input shape: (3, 15, 15)

Channel 0 → Current player's stones   (1 = stone, 0 = empty)
Channel 1 → Opponent's stones         (1 = stone, 0 = empty)
Channel 2 → Turn flag                 (all 0s = Black's turn, all 1s = White's turn)

The key design choice here: the representation is always from "my" perspective. Regardless of whether the current player is Black or White, Channel 0 is always me and Channel 1 is always them. This means the network doesn't need to learn two different "roles" — it just learns to play well from any position. Clean.

Data conversion runs in O(n) time where n = L^2 = 225. Fast enough that I'm not worried about it being a bottleneck.

The Disasters (Troubleshooting Log)

Okay, this is the part you're actually here for.

🔴 Problem 1 — CLion and the terminal were living in different realities

I pip-installed PyTorch into my .venv, ran import torch in the terminal — worked fine. Opened the same file in CLion — red squiggles everywhere, "No module named torch."

The cause: CLion's Python interpreter setting was pointing to the system Python, not the .venv. They were two completely separate environments and I was somehow confused about which one was running what.

Fix: Settings → Python Interpreter → point it at .venv\Scripts\python.exe. Obvious in hindsight.

🔴 Problem 2 — CMake couldn't find pybind11

Could not find a package configuration file provided by "pybind11"

CMake had no idea where pybind11 was installed, even though it was sitting right there in the .venv. The issue is that CMake doesn't automatically look inside virtual environments.

Fix: Ask Python where it is, dynamically:

execute_process(
  COMMAND ${Python3_EXECUTABLE} -m pybind11 --cmakedir
  OUTPUT_VARIABLE pybind11_DIR
  OUTPUT_STRIP_TRAILING_WHITESPACE
)
find_package(pybind11 REQUIRED)

Now CMake asks the actual Python interpreter "hey, where's pybind11?" and uses that path. Works regardless of where the venv lives.

🔴 Problem 3 — The compiled module went to the wrong place

Compiled gomoku_ai.pyd successfully. Ran python model.py. Got an ImportError.

Turns out CMake was dumping the output into cmake-build-debug/, but model.py was in the project root and had no idea that folder existed.

Fix: Tell CMake where to put it:

set_target_properties(gomoku_ai PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY ${CMAKE_SOURCE_DIR}
)

Now the .pyd lands right next to model.py. Problem solved, should've done this from the start.

🔴 Problem 4 — The big one: DLL load failed

This one took the longest. Everything looked fine — .pyd was in the right place, Python could see it — but running import gomoku_ai gave me:

ImportError: DLL load failed while importing gomoku_ai: 找不到指定的模組

找不到指定的模組. Module not found. But the module is right there.

The real issue: the .pyd itself was fine, but it dynamically links to MinGW's runtime DLLs at load time, and Python had no idea where to find them. The three culprits:

libgcc_s_seh-1.dll
libstdc++-6.dll
libwinpthread-1.dll

These are MinGW's C++ runtime libraries. On Linux this is never an issue because GCC's runtime is system-wide. On Windows, especially with a non-standard MinGW install, they can be anywhere — or effectively nowhere, from Python's perspective.

Finding them: I scanned the whole drive and found them in CLion's bundled MinGW:

C:\Program Files\JetBrains\CLion 2025.2.5\bin\mingw\bin\

Quick fix — copy them to the project root:

$mingw = "C:\Program Files\JetBrains\CLion 2025.2.5\bin\mingw\bin"
$dest  = "C:\Users\shawn\CLionProjects\Gomoku"

Copy-Item "$mingw\libgcc_s_seh-1.dll"  $dest
Copy-Item "$mingw\libstdc++-6.dll"     $dest
Copy-Item "$mingw\libwinpthread-1.dll" $dest

After that, import gomoku_ai worked.

Cleaner long-term fix — static linking in CMake:

target_link_options(gomoku_ai PRIVATE
    -static-libgcc
    -static-libstdc++
    -static
    -lpthread
)

This bakes the runtime directly into the .pyd, so you don't need to ship DLLs alongside it. The tradeoff is a slightly larger file, but for a dev project I'd take that any day.

Validating the Interface

Once everything was finally working, I ran a quick sanity check:

import gomoku_ai

board = gomoku_ai.Board(15)
obs = board.get_observation()

print(obs.shape)   # (3, 15, 15) ✓
print(obs.dtype)   # float32     ✓

兩行程式碼，證明三天的環境地獄是值得的。

What's Next

The C++/Python bridge is solid. Next up:

1. MCTS implementation — Pure Python first to get the logic right, optimize later
2. PolicyValueNet — A lightweight ResNet that takes (3, 15, 15) and outputs a move distribution + position evaluation
3. Self-play training loop — The part where the AI actually starts learning

I'll write about the MCTS + neural net design once that's done. The interesting part is how they interact — the network guides the search, the search improves the network, repeat until it beats you.

如果你也在做類似的專案，歡迎交流。這條路上的坑比想像中多，但也比想像中值得。