Common bugs of Lichess bots

Threefold repetition bug

---

The most common and critical bug among "weak" bots.

What's happening?

Let's see how the mighty @maia9 bot converts this completely winning position...

Ok, perhaps it's because @maia9 has no search. Let's see how the significantly stronger @maia9_10n, searching 10 nodes (instead of 1) handles this...

Alright. Perhaps these Maia bots can't understand repetition, because they lack searching capability. Let's see how @ChessChildren, a stronger bot, using the old fashioned brute force approach, searching 1Mn/s solves a mate in one...

How to fix it

1. Maintain a stack of position hash keys, from the initial position to the current node.
2. Iterate backwards to see if the current position is a repetition. You can accelerate the loop by starting 4 plies ago, iterating by 2 plies, and stopping once you exceed the 50 move rules ply count (at which point you would traverse an irreversible move).

Common mistake: Waiting to find 2 repetitions to score the node as a draw instead of doing it immediately at the first one. Not only does it require a lot more depth to detect the second repetition, but it can even fail completely, because of the (recursive) tree grafting effect of transposition tables.

Determinism

---

Yes, and we're not talking about philosophy here. Determinism in bots is a flaw, that can - hence will - be exploited.

First attempt

Let's abuse @sargon-4ply for a few games. Let's try to beat it in bullet. After analysis of the game, Stockfish shows me that Nxc6 wins a piece, which I missed:

Second attempt

Ok, if the bot is determistic, it will play the same move in the same position, so we can repeat and improve. Now, whenever I play white, I can get into this easily winning endgame:

In fact, I can even pre-move the whole sequence from the starting position to this winning endgame. Big help in bullet!

Obviously, I can do the same thing with the black pieces.

Infinite Elo farming unlocked

Ok, it just took me 2 attempts, because I'm not a beginner. But even a 400 elo noob can do it. It will just take them more iterations.

Disclaimer: Please don't abuse the Lichess rating system by farming bogus bots in this way. I did this playing Casual, not Rated, just to illustrate.

How to fix it

The simplest fix doesn't even require modifying the engine code. Merely configuring the Lichess interface script to use an opening book from which to select moves randomly.

An even stronger fix (in addition to opening book), is to modify the engine to add some randomness. Multi threaded engines get a bit of this for free. But single threaded ones don't, and need explicit logic for it, typically add random noise to the evaluation.Of course, you must seed your PRNG with a random seed (typically system clock), or else you still draw from a deterministic PRNG sequence, fully determined by the seed.

Unable to perform trivial mates

---

Can we solve K+Q vs. K endgame?

White to play, and win easily. Just promote to a Queen and win the game. Simple plan, right? Well @larrybot has a more elaborate plan, ending in a 50 move draw some 82 moves later...

How about K+QQ vs. K endgame?

Surely a pair of Queens should suffice. Let's see how @UltraBrick wins this...

It didn't even shuffle for a 50 move draw. The game was interrupted by an unexpected threefold repetition that @UltraBrick doesn't seem to understand either.

How to fix it

A few things are needed:

1. Fix the aforementioned repetition bug.
2. Implement the 50 move draw rule, if not already done. You never know. Just look at how the engine refused to promote in the KQvK game, until the 50 move rule started to hit the search horizon...
3. Incentives in evaluation. Simplest is to use a Piece Square Table (PST), which gives bonus for centralised king in endgame, and malus for king in the corner or edge.
4. Searching deeper, of course.
5. Disabling null move pruning may also help, but this should be done carefully (it's a trade-off).