During controversies over the uprightness of Grand Secretary Zhang Juzheng following Zhang's death in 1582, Pan was accused of siding with Zhang, and was dismissed from the civil service entirely in 1584.
In 1587, the dikes along the Yellow River burst in several places, and Grand Secretary Shen Shixing sought to engineer Pan's return to service; he had proved himself an able commissioner of riparian works in the past, and though it was politically unpopular, Shen worked to bring him back to address this new iteration of flooding problems. A temporary commissioner was assigned for the time being, but within a few months, Shen was able to rally enough support to bring Pan back into the service.
Pan served admirably in heading the project, which included repairing dikes along the river, among other work, and remained in good standing at court so long as Shen did. When Shen was forced to resign in the early 1590s, however, Pan was as well.
- Ray Huang, 1587: A Year of No Significance, Yale University Press (1981), 106-108.