Tell Duke you won't go to the Tower of Gabnid five times. Each refusal must be in a different room and the Master's room will only count if you speak with him.