I know with work we tend to get fiberglass on us and its unpleasant. If the fibers from the cactus are as small a super cold shower may help. Closes the pores of your skin and helps bring the object to the surface. Good luck. I had it happen on the top of my head once. It was a royal biotch for weeks. Everytime i put a hat on i found more.
You could also try to wax it?