The player holding the prisoner can't refuse the ransom, as long as it is the player's turn, and he can afford the ransom (he must have 3 or more points with which to pay). For instance, blue has captured a red follower. On red's turn, he may elect to pay the ransom for the prisoner at the beginning of their turn. Blue can't refuse. Blue gives red back his follower. Blue moves forward 3 points on the scoring track, red moves back 3 points on the scoring track. If red had a score of 2 points on the scoring track, he would not be able to buy back a prisoner on his turn, as this would put him at -1 point, which isn't possible. Once red has his follower back, he continues his turn and can redeploy that follower to a newly laid tile, etc.
In a similar way, players who have prisoners of each of their respective colours immediately exchange those prisoners (on a 1 for 1 exchange), but it is the owner of the prisoner that decides which piece of his is returned to him. The person who captured the prisoner in the first place (if he has more than 1 of the same colour) does not get to decide which prisoner is returned, the owner does. For instance, the red capturer might hold a large follower and a normal follower of blue. Blue then captures a normal red follower. The captured pieces are exchanged immediately, but blue can choose either to have his large follower or his normal follower returned to him; but not both followers. The large follower does not make any difference on the exchange rate - it is still 1 for 1.
