I'll do as TobiasW has suggested and turn my comments into an answer format.
First thing to keep in mind when using physics with bullets or short-lived objects is that there can be MANY of them. If your game is anything like Battlefield, you may want to rethink the usage of physics on every single bullet.
For CPUs, doing math is always much cheaper (in terms of performance) than doing physics calculation. So, try not to use physics if you can help it.
Also, try to use stored variables. For example, a gun will shoot a bullet towards a trajectory and calculating that trajectory every time you create a new bullet is costly. Instead, calculate it for the gun and keep that as a stored variable (if you're raycasting, store it in a Raycast variable).
If the number of bullets on screen at any given time is not that high, then it's not a bad idea to use physics, but still, try to avoid using physics calculations as much as you can. For example, if you're never going to have 100 and rarely going to have more than ten rockets on the screen at any given time, then it's not a bad idea to use physics with those. Especially if you're thinking of having "magnetic" traps (like the magnets from Worms games) which attract or deflect the rocket.
If you wish to use physics on an object but not have it push other stuff back, it's a good idea to encase the
rigidbody inside a
collider, so if you have a 1cm square (or cube if you're in 3D) rigidbody on a bullet, then make its collider into a 1.1cm square. This will mean that before the physics calculations of the hit can take place, the collider will do the calculations it's supposed to and then get rid of the bullet. Of course, colliders can still do a push-back like effect, but it should be infinitesimal compared to a rigidbody. Besides, you still don't want the physics calculations kicking off for a minigun that's dishing out 6000 bullets per minute.