We developed and tested neural network-based models to recognize and count emigrating juvenile river herring in continuous video. Continuous video was collected from an underwater camera installed at Great Herring Pond in Bourne, Massachusetts (USA) between June and November 2017. Our algorithm extracts video frames to assess presence/absence of fish and count numbers of fish emigrating. We used extracted video frames to assess model performance. Provided datasets include information about extracted frames that were used for model assessment. This data release includes four datasets that were used to test model performance and select the best fitting model. (1) The “Model Evaluation Dataset” includes count and presence/absence classifications for 19,498 video frames that were evaluated by each tested model variation and by human counters. (2) The “Count Evaluation Dataset” includes expert counts for 189 video frames along with outputs for the top 4 models and associated differences in counts. Both the model evaluation dataset and count evaluation dataset also include environmental conditions associated with captured video (time of day, moon phase, cloud cover), that were downloaded from Visual Crossing Weather API (Visual Crossing Corporation, 2023). (3) The “Volunteer Validation Dataset” was used to test volunteer count and classification accuracy compared to expert accuracy and includes 5009 volunteer and expert classifications. Volunteer counts were assessed by participants on an online public participation science website (Zooniverse). (4) The “Expert Validation Dataset” was used to differences in classifications of the same image by two expert observers and includes numerical and categorical classifications for 65 images and 500 fish occurrences.