Autonomous recording units are increasingly being used to monitor wildlife on large geographic and temporal scales, paired with machine learning (ML) to automate detection of wildlife. However, false positive detections from ML classifiers can result in erroneous ecological models that can lead to misguided management and conservation actions. We used a two-stage general approach to understand and reduce false positive detections, a technique in which outputs of the primary classification model are passed to a secondary classification model to yield the probability that a detection from the primary model is a true positive detection. This approach is demonstrated on two open-source models, BirdNET and the Drumming Model, that detect Ruffed Grouse (Bonasa umbellus). We analyzed over 9500 hours of acoustic data collected in 2022-2023 from the Green Mountain National Forest in Vermont, USA, and found the two models detected different types of acoustic signals associated with differing life history traits. The Drumming Model yielded 4106 detections (71.5% true positives); BirdNET yielded 524 detections (17.0% true positives). Secondary logistic regression models separated true positives and false positives with high accuracy (BirdNET = 84.5%; Drumming Model = 89.8%). Our findings go beyond improving Ruffed Grouse monitoring and conservation efforts to, more broadly, illustrate how two-stage ML approaches can improve the use of model-derived detections in wildlife research.