I am using a LLM in a one shot context here, so it is not applicable to the situation you described. But to do what you are talking about, you would need to train a spacy model on your dataset for the classification to work (It is pretty easy to do if you have enough data). You probably won't need the LLM component of the pipeline. Or if you have money to spare, you can still fine-tune GPT-3, but I wouldn't recommend for your specific use case.