In terms of Newtonian physics, that mass needs to be accelerated in longitudinally (engine power, positive and braking, negative), and both left and right around corners, so it impacts performance everywhere the car is on the track - except where the driver's being a wuss and coasting.
The math to determine the influence on lap time isn't theoretically complex - it just requires a gazillion calculations. The best way to do it might be to boot up the race car dynamics simulation software and change just that variable. I don't have that software but I'll be that someone around here does.
In reality, it's got to be considered as a percentage change in all respects, depending on what the total mass of the car is. It might be a meaningful difference if 100# is a 5% change (2000# car) rather than a 4% change (2500# car). Your designations of "handling" vs. "horsepower" are proxies for this, I think...
The REAL question however - and the assumption on which the 100# noise tolerance is predicated - is whether or not the variance in lap times resulting from that weight is
smaller than the difference between what a car is actually capable of, and what the typical SCCA club racing driver can get out of the hardware. Unless that influence is the determining factor in whether one make/model car beats another make/model car - with ALL of the other variables being held constant - it's just too small to try to attend to.
Similarly, if that weight contributes less to a lap time than the real lap-to-lap variance of the driver, it's meaningless. This is about repeatability.
Either (or even both) can be issues in any given case. A driver can be inconsistently fast or consistently slow (relative to what the car is capable of) and in either situation, it's not weight that's the issue.
K