Empower Agents with Guidance