Challenge mode is a special way of starting games that allows you to target a specific player instead of opening a lobby that everyone can join.
You can challenge another player by right-clicking on them, hovering over "Apps", and clicking on "Challenge". This will open a menu that lets you pick a game to challenge them to.
If you are on mobile, you can also challenge someone with the
/challenge slash command.