Passive stylus systems offer a simple and cost-effective solution for digital input, compatible with a wide range of surfaces and devices. This study reviews the domain of passive stylus tracking on passive surfaces, a topic previously underexplored in existing literature. We answer four key research questions: what type of systems exist in this domain, what methods do they use for tracking styli, how accurate are they, and what are their limitations? A systematic literature review resulted in 24 papers describing passive stylus systems. Their methods primarily fall into four categories: monocular cameras with image processing, multiple camera systems with image processing, machine learning systems using high-speed cameras or motion capture hardware, and radio frequency signal-based systems with signal processing. We found the system with the highest accuracy used a single monocular camera. In many systems, markers such as retroreflective spheres, tape, or fiducial markers were used to enhance the feature matching. We have also found stagnation and in some cases, regression in the precision and reliability of these systems over time. The limitations in these systems include the lack of varied stylus form factor support, the restriction to specific camera positions and angles, and the requirement of expensive hardware. Given these findings, we discuss the important characteristics and features of passive stylus systems and propose ways forward in this field.