You cannot use transform.Rotate because that will always rotate relative to whichever direction is currently facing. What you want to use instead is Transform.eulerAngles which will explicitly set a rotation despite whatever the current rotation is:
transform.eulerAngles = new Vector3(0f, yAngleRotation, 0f);
Each swipe direction would have a different yAngleRotation value (I'm not sure exactly which as it depends on what axis and direction your game is on / displayed at). These values would probably each be 0f, 90f, 180f, 270f.
One key thing to note is to make sure you don't just set one axis rotation by itself or you will get bad results... as in don't do this:
// this is very very bad, don't do this!!!
transform.eulerAngles.y = yAngleRotation;
Thanks to Leuthil from unity3d forum Who solve my problem. Thanks allot :) all of the others who sincerly tried for me.