Programmatic Curtailment: The Problem With Manually Curtailing Mining Facilities

Shutting down power throughout a mining facility may seem routine, but the consequences of inaccurately or manually flipping breakers can be far-reaching and potentially cause permanent damage to ASICs. This article will explore other reasons why programmatically utilizing strike price-based curtailments can be beneficial. We’ll also focus on a case study of an immersion facility that decided to initiate a power shutdown to participate in a demand response program.

Even when carried out with good intentions, manual power shutdowns (via breakers) can unleash a series of catastrophic events that disrupt operations, jeopardize equipment, and lead to substantial financial losses. The case study discussed here is a tale of caution, highlighting the importance of understanding the potential consequences of not implementing a comprehensive program like Foreman to mitigate risks through automated curtailment.

Programmatic Consistency in a Chaotic Market

Consistency in chaos is crucial to programmatic curtailment processes. Real-time index pricing volatility, seasonal price increases, quarterly changes in time-of-day pricing, and minute-by-minute variable factors underscore the need for a programmatic strategy in the wild west of real-time energy markets like ERCOT. By leveraging automation and programmatically curtailing, mining facilities can respond swiftly to pricing fluctuations, optimize energy usage, and maintain operational efficiency in an ever-changing market environment.

Real-Time Index Pricing Volatility

Index volatility poses significant challenges for manual curtailment processes. Many resources are required to manage curtailment based on real-time pricing effectively. Working curtailment this way includes dedicating multiple individuals to constantly monitor ERCOT real-time pricing for day and night shifts, covering all daytime hours. Such a manual approach would demand significant time, effort, and financial investment, as personnel are due full-time or be paid at least around the clock to fulfill this monitoring role. By contrast, automating curtailment processes eliminates the need for constant human oversight, reduces staffing costs, and allows for efficient and cost-effective curtailment management based on real-time index pricing volatility.

The graph below illustrates the importance of programmatically automated curtailment during unprofitable periods. Each dot represents the average hourly price in ERCOT, with yellow indicating unprofitable times and blue representing profitable ones. Foreman’s software-based curtailment enables efficient energy usage and maximizes profitability without needing constant manual intervention.

How Summer Heat Affects Energy Prices

The ERCOT energy price market is significantly impacted by summer heat, with the hottest days often leading to increased electricity prices. Despite the substantial contribution of solar energy production, more generation capacity is still needed to offset the high demand during scorching summer days. As a result, energy prices soar due to the strain on the grid. These spikes in pricing highlight the need for expanded energy production capacity to meet the surging demand during peak periods. Additionally, implementing effective curtailment strategies becomes crucial, particularly during low-demand months, to ensure a balanced energy supply and manage pricing fluctuations from a macro and statewide perspective.

Implementing Foreman enables automated curtailment, reducing stress on the grid and streamlining mining operations. By optimizing energy consumption based on grid conditions, Foreman contributes to grid stability while promoting energy efficiency and cost savings on a company-to-company level.

The Purpose of Programmatic Curtailment

This case study highlights the necessity of integrating Foreman into mining operations to manage curtailment. Foreman integration is critical in mining facility operations by providing automated software and firmware capabilities that contribute to seamless and uninterrupted workflows. One of Foreman’s key features is automated power ramping, gradually reducing power consumption through enabled software and firmware. This controlled approach ensures a smooth transition, avoiding disruptions and potential issues associated with sudden power fluctuations.

This article explores the scenario of an immersion facility that opted to participate in a demand response program by “breaker flipping” or shutting down power facility-wide. It delves into their challenges, the extensive damage caused, and the efforts required to restore normal operations. By analyzing this case study, we demonstrate the need to implement robust management systems, such as Foreman, to ensure the smooth execution of automated curtailment and minimize the associated risks.

Key Takeaways from Case Study:

• Increased Risk of Equipment Damage.
• Missed Revenue for Load Optimization.
• Exacerbated Intervention and Response.
• Manual, not automated curtailment.

Immersion Case Study

Recently, a mining facility faced a series of damaging events when the CSP (Centralized Service Provider) abruptly interrupted their service without a safe power-down of the machinery. At the time, the facility had yet to incorporate Foreman. Instead of a controlled shutdown, this unexpected power cut led to severe complications. The repercussions of this situation underscore the importance of robust tools like Foreman. These systems manage service interruptions in a way that prevents abrupt shutdowns.

Curtailment and Immersion Shutdown:

Due to the circumstances above, the CSP decided to curtail the mining facility's recloser. As a result, the mining equipment, including the immersion cooling systems, was abruptly shut off. Immersion cooling is a method that uses a dielectric liquid to cool mining rigs, enhancing their efficiency. With the power cut off, the cooling systems ceased functioning, which had a cascading effect on the entire facility.

Air Bubbles and Miner Failures:

When the site was reenergized after the curtailment, the mining facility faced a critical issue—air bubbles had become trapped in the cooling tubes of the immersion system. These air bubbles disrupted the flow of the dielectric liquid, causing the mining rigs to overheat. The issue went unnoticed without Foreman's proper monitoring and control, exacerbating the situation.

In some cases, temp gauges can short out from the massive surge of electricity, prolonging the issue and possibly damaging machines.

Secondary Equipment Failures

The consequences extended beyond the primary mining rigs. The sudden power surge during the restoration process also impacted expensive secondary equipment. Computers, servers, and other essential components experienced failures and required hard reboots. These critical system failures prolonged the downtime and introduced the risk of data loss and additional damage to the facility's infrastructure.

Facility-Wide Disruption:

The facility was left in disarray for an extended period with no on-site personnel available to promptly diagnose and address the issues. Miner failures, secondary equipment malfunctions, and power surges caused significant disruptions, rendering the facility non-operational for several hours. The financial losses incurred during this time were substantial, and the impact on production and operational efficiency was severe.

The case study above serves as a stark reminder of the potential consequences that can arise when mining facilities operate without integrating a comprehensive management system like Foreman. The events following the curtailment, from miner failures to secondary equipment malfunctions, underscore the critical need for continuous monitoring, early detection of issues, and remote management capabilities. By integrating Foreman into their operations, mining facilities can proactively safeguard against such incidents, ensure operational continuity, and minimize financial losses caused by unforeseen events.