Designing Adaptive Reservoir Operations Using Multi-Objective Reinforcement Learning
PI: Antonia Hadjimichael (EMS, Geosciences & EESI)
Level of effort appropriate for the proposed project: Two semesters of pre-comps RA support
Plan for funding the remainder of the researcher’s salary: Startup funds can cover tuition
A list of specific areas of computational and/or data science expertise: reinforcement learning, evolutionary optimization, time series analysis
Any other requirements or expectations of potential ICDS Junior Researchers: attend weekly group meetings
A list of specific objectives for work supported by this call: develop reservoir operations model with additional optimization constraint; publish reproducible repository of model and analysis
At least one medium to long-term goal: medium term goal to submit a scientific publication on the findings; long term goal to use results to acquire sustained funding for the effort from NSF (Collaborations in Artificial Intelligence and Geosciences program or other)
Connection of the project to ICDS’s mission: This multi-disciplinary approach combines multi-objective evolutionary optimization and reinforcement learning techniques with a hydrologic and reservoir operations model. The project supports ICDS’s mission by applying cutting-edge reinforcement learning approaches to address critical environmental management challenges. The research demonstrates the value of integrating advanced methods to train dynamically adaptive and state-aware control strategies, applicable to a variety of applications outside reservoir operations. The computationally intensive nature of reinforcement learning with multi-objective evolutionary algorithms makes this project particularly well-suited for ICDS resources.
Team member’s recent and/or planned engagement with ICDS: PI Hadjimichael is a member of ICDS’s Cyberinfrastructure Faculty Advisory Committee (CiFAC). The project will also utilize ROAR Collab computational time in the PIs allocation.
A brief description of the proposed project.
Reservoirs are critical infrastructure for water resource management, requiring operators to balance competing objectives such as hydropower generation, water supply, and maintaining environmental flows. The Conowingo Reservoir on the Susquehanna River is a prime example of these challenges, where operators must provide water for municipal supplies and a nuclear power plant while maintaining downstream flows and maximizing hydropower revenue. Current operating policies fail to adequately account for the system’s internal variability and emerging threats from climate change. Particularly concerning is freshwater salinization—when riverine flows are low, brackish water from Chesapeake Bay can intrude inland, compromising water quality at the Havre de Grace intake. This project will develop dynamically adaptive and state-aware reservoir operation policies for the Conowingo Dam that explicitly address saltwater intrusion while balancing other management objectives under deeply uncertain future climate conditions. By applying advanced computational approaches, we aim to identify flexible control policies that map system states (reservoir levels, forecasted inflows) to optimal release decisions. These policies will be designed to dilute salt concentrations during low-flow conditions while meeting other operational objectives, and will be tested across a range of plausible future hydroclimatic scenarios to ensure robustness to climate change.
The project has multi-objective reinforcement learning at its core. We will first establish new probabilistic flow requirements by analyzing salinity data at the Havre de Grace water intake and accounting for tidal, seasonal, and annual variability. We will then train multiobjective, state-aware, and dynamically adaptive reservoir release policies through Evolutionary Multi-objective Direct Policy Search, a reinforcement learning method. Unlike traditional approaches, our method trains policies represented by nonlinear approximating networks (e.g., artificial neural networks or radial basis functions) that continuously map system states (reservoir levels, forecasted inflows, current salinity conditions) to optimal release decisions. This approach enables policies to learn from experience across thousands of simulations, developing the ability to dynamically respond to changing conditions and even generalize to previously unseen states. The methods employed are computationally intensive as they rigorously explore objective spaces across multiple stochastic hydrological simulations. Therefore, we will leverage the ROAR Collab cluster to enable efficient policy identification and testing.
This proposal requests funding for two semesters of RA support for a pre-comps graduate student. The graduate student will collaborate with the PI to implement the reinforcement learning framework with salinity constraints, conduct ensemble simulations, and analyze multi-objective tradeoffs. The student will also lead the development of a reproducible and transferable code repository, as well as the submission of scientific article. The student will also engage with ICDS through regular participation in seminars and presentation at the Fall ICDS Symposium.