I've found over the years that there is a bit of a black art in using frisket. But, it really comes down to a few things.
1) Support Selection and Surface prep
2) Layers and Curing time
3) Optional Intercoats
****** Support Selection and Surface prep ******
Porus Supports:
Frisket was really a big thing in illustration done on hot press illustration card, and to this day that is about the best support to use it on. It has just enough rag (tooth) to hold paint, and make it resistant to lifting, and any paint lifted tends to be very subtle. However, the moment you have paint on top of paint, the surface changes, and you need to start treating it like a non-porus support. Other papers will perform similar to Illustration Card (IC) to a greater or lesser degree.
What you need if using paper is thick, strong, well pressed paper (hot press), such as vellum brisol, ink papers, or some watercolor papers. The danger with watercolor papers however is that they tend to be loosely bound, so tear easy. If you go watercolor, use a high cotton content paper that will resist the surface lifting and learn to use any knife or cutting tool very carefully.
Optionally, a light coat of a transparent binder can really help strengthen a paper, WHEN PROPERLY CURED
Wood such as birch is also a good choice, properly prepared with at least a thin layer of gesso. Too thick a gesso layer and it starts to act like a non-porus surface when it comes time to cure.
Advantages of porus supports are in curing time and cost.
Non-Porus Supports:
Non porus supports, such as synthetic papers, photo paper, metal, all have unique advantages, but they do not hold paint as well without giving them some kind of tooth. A light sanding with 200 or 300 grit paper will help keep your paint adhering to the support and not your frisket. Also, absolutely use degreaser or alcohol to rub down any metal or plastic surface you will be working on, and use gloves while handling it to avoid getting finger oils on it.
Adding tooth to a non porus surface and very, very carefully removing any grease, dust or oil is vital to avoid lifting paint when using frisket
****** Layers and Curing time ******
The biggest mistake that people make when using frisket is assuming paint sticks to other paint. That is to say, while the first layer of paint or basecoat may adhere well to the support, they find that subsequent layers of paint peel easily off. This is because paint itself is a largely non-porus media on which to put other paint. Even worse, paint that is dry to the touch (especially waterborne paints) will still contain moisture or solvent, and may not be fully cured for days with each successive layer. The curing time goes up exponentially for each successive layer, as the lower layers now have to fight with the upper layers to outgas and crosslink. This makes top layers weak longer, and that means your nice skull on a black background, suddenly lifts when you go to use frisket if you happened to paint the skull on top of a previous layer of black. The way of avoiding this is to avoid layering as much as possible, and to give adequate time between laying down layers for a full cure of the previous layer. It takes time, and we are all of us, guilty of rushing things. Dry is not cured. If you lay down multiple layers in a session it could take 24 to 72 hours before it became frisket safe on a porus support, even longer on non-porus.
So, plan out your spraying to use as few layers as possible, and try to avoid spraying paint over paint on any area you plan to mask with frisket. Before you ever put frisket over top of paint, your first question should be - had it had 72 hours to safely cure?
Best entirely to never even consider frisket overtop of paint you've laid down, and instead only use it on the bare support.
****** Optional Intercoats ******
Not something I am fond of, except to stop paint bleed. Intercoat clears are wonderful to stop pigment from a lower layer bleeding into and chaging color of something in an upper layer, but are very poor at preventing paint lift. Better perhaps than nothing insofar as that you'll likely lift the clear, and it might save the underlaying paint, but they won't save you from multiple lift events, and after using frisket, their bleed protection goes out the window. Worst of all, nearly everyone applies them before the previous layer has properly cured. A clear overtop of an uncured lower layer is just a time bomb waiting to mess you up. It can lead to orangepeel effects on later layers, and prevent lower layers from outgassing and fully curing.
If you absolutely, positively, stake your life on it, know, and I mean KNOW that the previous layer are cured, then apply an intercoat and wait for htat to properly cure, it actually can be a life saver for frisket use. It's just that 9 time in 10, I see curing of the clear itself tripping up people learning airbrush and giving them a false confidence that leads to their beautiful and hard work being lifted with frisket.
So, unless you are really experienced with your paint system, I do not recommend the technique. While it can perform miracles when used by those that understand curing and cross linking of paint, for nearly everyone else it is a recipe for disaster.
Last words of advice for frisket...
Frisket use is a more advanced technique and should be used only for very specific purposes when absolutely nothing else will get the desired effect. It is inherently risky, requires incredible skill in cutting without cutting into the support or paint below and just begs to lift paint or leave residue on a surface. It really is intended only to be used directly on a support or a well sealed surface properly cured, and not really over top of paint. So many easier and faster ways to mask things are out there, from hand shields to paper templates, to freehand techniques.
If you are going to use it, that is awesome, but it's good to learn out strategies to use it safely.
If you can lay your hands on it. Peter Owen & Jane Rollanson's "The Complete Manual of Airbrushing" from 1988 has an invaluable section on how to strategically plan out making and frisket strategies. I make all of my airbrushing students read that section before ever touching a roll of frisket.
Frisket is all about strategy and avoiding paint, paired with surface prep and knowledge of curing.
Good Luck!