Well, I was close.
When you're flagged for PVP, any action you take will reset the flag timer.
Things that reset it out of combat:
Flash of Light
Eating/drinking (Succulent Orca Stew)
Landing at a flightpath...?
Things that reset it in combat:
Art of war procs from melee hits (but not melee hits)
Vengance procs from melee hits
Or any ability really
Dismissing buffs doesn't appear to cause a reflag, but changing auras does. If you sit there and do nothing (but travel), you should drop the flag in 5 minutes as per normal (I did anyways).
Probably why people have issues getting it to go away. Until the devs can find a fix for this, just sit tight, and it will fade on its own.