I don't know about removing it in a Neon, but I did race one at my 2nd driverrs school with the belt thrown on lap 2. Noticeable increase in power and I was happy with both feel and response. Of course my normal car is manual steering so I'm used to the extra effort. Typical quoted losses for power steering pumps range from 3-5 and higher so the gain isn't trivial.
But, just pulling the pump isn't legal, you would have to swap the rack also. And as Chris pointed out the ratio is slower so I'm not sure that's the right way to go. Crank and accesory pulleys are free so you could just gear the thing down to be next to inoperable and fluids are free as well. So a lightweight oil in a barely turning pump is an option.