This lag-vs-responsiveness issue is the situation with virtually all motion controllers, whether something like the Hydra, the Wii Remote, the Kinect, or the PlayStation Move.
The problem is this:
When an input stream is coming in, you're making a decision on a frame-by-frame basis about whether or not to trust the input data; whether the trends you're seeing now will have continued in the data you receive a dozen milliseconds from now. For example, if there's a sudden shift to the right this frame, you don't know whether it's a real bit of input data (and so you should act on it) or whether it's merely jitter (and so you should ignore it). Whichever you pick, if you later discover that you were wrong, you've either allowed the input jitter to make it into your game (in the first case), or introduced lag into your game (in the latter case).
There is no good solution to this. A "correct" solution to determining whether input is real or jitter requires knowing what the input stream is going to do in the future, as well as what it did in the past. We can't do that in games, for obvious reasons. So no, there is no way to filter rotational data to remove jitter without losing precision, in the context of a game which is working with input data in real-time.
I've seen a major manufacturer recommend that developers deal with this problem by having players hold down a button while rotating the control, so that the game can turn off its anti-jitter code at that point, so it's non-laggy. (I don't recommend this, but it's one approach).
I've seen a few motion input middleware libraries which deal with this problem by introducing an artificial delay in the input -- there's a quarter-second buffer that input data goes into, and your game only hears about the input a quarter second later, so that the library can smooth out the jitter for you, by knowing what happens both before and after the "present" from the game's point of view. That works great, apart from introducing a quarter second of lag to everything. But it's one way to solve the problem, and it can do an awesome job of accurately representing a motion with jitter removed, at the expense of constant lag.
But without going to that extreme, there are still some things we can do to give vastly improved behaviour, even though we know that there will always be "worst case scenarios" which behave in a non-ideal fashion.
The core insight is that we only really care about jitter when the controller is mostly stationary, and we only really care about lag when the controller is being moved. So our strategy should be to try to deal with things so that we have lag when the controller is stationary, and have jitter when the controller is in motion.
Here are two possible ways to do that:
One common approach is a "locked/unlocked" system, in which you keep track of the device's orientation, and if it doesn't change for a short while (half a second or so), you 'lock' that orientation, taking no action based upon the device's reported orientation until it differs enough to 'unlock' again. This can completely squelch orientation-based jitter, without introducing lag when the orientation is actively changing. There might be a hint of lag before the code decides it needs to switch into "unlocked" mode, but it'll be a lot better than having lag everywhere.
Another approach is to average together input data from frames. The important point here is to only average together the input data from frames where the input data was vaguely similar -- This meant that small jitters will get blurred together and softened, but larger changes don't get blurred, because their data isn't similar enough to the data from previous frames.
There are other ways to get a similar effect as well. The core insight is that you can't have both non-jitter and non-lag in your real-time game at the same time, because to do that would require knowledge of the future. So you need to pick when to bias your control's behaviour toward accepting jitter and when to bias it toward accepting lag, in order to make the overall experience as non-bad as possible.