The United States' “stop and frisk" rule originated from the Fourth Amendment's language regarding unreasonable searches and seizures. The policy actually allows police to stop and “pat down" anyone they suspect may be committing, have committed, or are about to commit a crime.
Can you refuse to be patted down?
You can't stop the officer from patting you down if he decides he is going to do it, even if he is not legally allowed to. But you can tell him that you do not consent. If you repeatedly and calmly let an officer know you do not consent it will do several good things for you.
When can you be patted down?
A pat-down search constitutes a search under the Fourth Amendment. Pat-down searches, incident to an investigatory stops, are usually made without a warrant and justified if the officer has reasonable suspicion that the person being searched is armed and dangerous.