Any disruption(control surface here) over the surface of a smooth fluid foil causes the airflow to be disrupted and creates drag.
Additionally, the movement of a control surface from its flush(neutral) position on a Low observable body will create a change in its reflectivity and increase its RCS.
What TVC allows is the ability to keep the airframe completely smooth(no RCS changes or surface breaks) by using thrust direction to point the aircraft. This ability is very useful for a stealth airframe to make flight path changes without exposing itself through the movement of its control surfaces.
Well, its not for the F-16 anyway. So the swaying may be or not be swinging or swaying at all.