Follow the steel brake line from the brake master cylinder on the firewall.If it joins a round steel drum like thing in the drivers side of the engine bay about 9 inches diamemter with another brake cylinder attached to it then you have a brake booster.It may or may not have the letters PBR written on it somewhere depending on what model booster it is.
It is possible that ( if you are running a brake booster that is ) you may

have a perforated vacuum diaphragm in the booster which is allowing the the inlet manifold to suck air via the vaccum supply line from inlet manifold to booster at stages of brake operation.You can test this by disconnecting the vacuum supply line from the inlet manifold and block the end of the hose off with something.Then try brake application.
If the engine idle is unaffected then it is possible the fault may be in the booster.
I have an EK sedan here with a VH 40 brake booster that does the same thing during application of the brakes although it's not a currently driving car and I haven't done any investigation myself as to the actual cause of the problem, I have theorised that it may have a perforated diaphragm.
It's hard to diagnose problems without looking at the car and even harder to write a way of explaining how to check things.
I'd certainly be taking it to Dr Terry's shop, he'll find the cause of it for sure.
I can't think what to write here so this will do.