If it's a short-term surge, that sounds iffy but not too bad. If the idle moves and stays while you're on the brakes, then I'd suspect a bad booster.
another way to check is to stick a vacuum pump & gauge on the booster. Try to pull a vacuum on it and see if it bleeds down with time.