it is your driving.
theres a few things you can do to try and correct understeer mid drift such as handbraking or switching off the accelerator to the brake and back as fast as possible, BUT the point is that you didnt take the right line, didnt have enough speed or didnt get your car sideways enough when initiating your drift with the feint. this happens to everyone when they first start out, its just experience and practice so just keep doing what your doing and trying different speeds and levels of throttle.
also you can try starting on a snow track until you get used to the power band of the car.
dont, i repeat dont keep screwing with your setup until you can drift the stock silvia or if you have a set up you like just stick to it even if its not perfect. this will only get in the way of your real understanding of the angles you need. once you understand whats going wrong or once you can drift quite well in a stock car, then start worrying about mods to make your car faster or to link drifts which you cant in a normal car etc etc
good luck and keep doing what your doing.