Enhanced system metadata availability

Oct 13, 2025 by recoveryshake.com 146 views

Overview

In this article, we will discuss the importance of fine-tuning the schema/DDL lease refresh interval in TiDB, particularly when running on attached disks in the cloud. We will explore the current limitations and propose a solution to improve the lease refresh interval, ensuring that TiDB can refresh schema on time, even in cases of long latency or hot key collocation.

Current Limitations

When TiDB runs on attached disks in the cloud, some disks may exhibit long latency for a minute duration. This can cause issues with schema lease expiration, as the system metadata cannot be served in time. To mitigate this problem, the lease refresh interval has been increased to 5 minutes. However, this change has a side effect: the lease refresh runs only every 2-3 minutes, as it is configured to run half of the lease interval.

The Need for Fine-Grained Control

While the current lease refresh interval of 5 minutes may seem sufficient, it can still lead to issues in certain scenarios. For instance, when there is a hot key collocated on the same node as system metadata, or when there are cache misses due to stale reads, the lease refresh interval can become a bottleneck. In such cases, it would be beneficial to have a more fine-grained control over the lease refresh interval, allowing TiDB to refresh schema on time.

Proposed Solution

To address the limitations mentioned above, we propose to introduce a more fine-grained schema/DDL lease refresh interval. Specifically, we suggest keeping the lease refresh interval to 30 seconds (or 1/10 of the lease). This approach can help in several ways:

Improved lease refresh: By reducing the lease refresh interval to 30 seconds, TiDB can refresh schema on time, even in cases of long latency or hot key collocation.
Enhanced system metadata availability: With a more frequent lease refresh, system metadata can be served more reliably, reducing the likelihood of schema lease expiration.
Better cache performance: By reducing the lease refresh interval, cache misses due to stale reads can be minimized, leading to improved cache performance.

Benefits of Fine-Grained Control

Introducing a more fine-grained schema/DDL lease refresh interval can bring several benefits to TiDB:

Improved system reliability: By ensuring that schema lease is refreshed on time, TiDB can provide a more reliable experience for users.
Enhanced performance: With a more frequent lease refresh, system metadata can be served more quickly, leading to improved performance.
Better scalability: Fine-grained control over the lease refresh interval can help TiDB scale more effectively, even in large-scale deployments.

Implementation Details

To implement the proposed solution, the following changes can be made:

Update the lease refresh interval: The lease refresh interval can be updated to 30 seconds (or 1/10 of the lease) in the TiDB configuration file.
Modify the lease refresh logic: The lease refresh logic can be modified to run more frequently, taking into account the new lease refresh interval.
Test and validate: The proposed solution can be tested and validated to ensure that it meets the desired performance and reliability requirements.

Conclusion

In conclusion, introducing a more fine-grained schema/DDL lease refresh interval can bring several benefits to TiDB, including improved system reliability, enhanced performance, and better scalability. By reducing the lease refresh interval to 30 seconds, TiDB can refresh schema on time, even in cases of long latency or hot key collocation. We propose to update the lease refresh interval and modify the lease refresh logic to implement this solution. With thorough testing and validation, we can ensure that the proposed solution meets the desired performance and reliability requirements.

Future Work

Future work can focus on further optimizing the lease refresh interval and logic to improve system performance and reliability. Some potential areas of exploration include:

Dynamic lease refresh interval: Investigating the possibility of dynamically adjusting the lease refresh interval based on system load and performance metrics.
Lease refresh prioritization: Exploring ways to prioritize lease refresh operations to ensure that critical schema updates are processed more quickly.
Cache optimization: Investigating cache optimization techniques to minimize cache misses due to stale reads.

Frequently Asked Questions

In this article, we will address some of the most frequently asked questions related to introducing a more fine-grained schema/DDL lease refresh interval in TiDB.

Q: What is the current lease refresh interval in TiDB?

A: The current lease refresh interval in TiDB is 5 minutes. This interval is used to refresh the schema lease, which is essential for ensuring that system metadata is up-to-date and available.

Q: Why is the current lease refresh interval not sufficient?

A: The current lease refresh interval of 5 minutes may not be sufficient in certain scenarios, such as when there is a hot key collocated on the same node as system metadata, or when there are cache misses due to stale reads. In such cases, the lease refresh interval can become a bottleneck, leading to issues with system performance and reliability.

Q: What is the proposed solution to improve the lease refresh interval?

A: The proposed solution is to introduce a more fine-grained schema/DDL lease refresh interval, with a refresh interval of 30 seconds (or 1/10 of the lease). This approach can help improve system performance and reliability by ensuring that schema lease is refreshed on time, even in cases of long latency or hot key collocation.

Q: How will the proposed solution improve system performance and reliability?

A: The proposed solution can improve system performance and reliability in several ways:

Improved lease refresh: By reducing the lease refresh interval to 30 seconds, TiDB can refresh schema on time, even in cases of long latency or hot key collocation.
Enhanced system metadata availability: With a more frequent lease refresh, system metadata can be served more reliably, reducing the likelihood of schema lease expiration.
Better cache performance: By reducing the lease refresh interval, cache misses due to stale reads can be minimized, leading to improved cache performance.

Q: What are the benefits of introducing a more fine-grained schema/DDL lease refresh interval?

A: The benefits of introducing a more fine-grained schema/DDL lease refresh interval include:

Improved system reliability: By ensuring that schema lease is refreshed on time, TiDB can provide a more reliable experience for users.
Enhanced performance: With a more frequent lease refresh, system metadata can be served more quickly, leading to improved performance.
Better scalability: Fine-grained control over the lease refresh interval can help TiDB scale more effectively, even in large-scale deployments.

Q: How can I implement the proposed solution in my TiDB deployment?

A: To implement the proposed solution, you can update the lease refresh interval to 30 seconds (or 1/10 of the lease) in the TiDB configuration file. Additionally, you may need to modify the lease refresh logic to run more frequently, taking into account the new lease refresh interval.

Q: What are the potential risks and challenges associated with introducing a more fine-grained schema/DDL lease refresh interval?

A: Some potential risks and challenges associated with introducing a more fine-grained schema/DDL lease refresh interval include:

Increased system load: More frequent lease refresh operations can increase system load, potentially leading to performance issues.
Complexity: Introducing a more fine-grained schema/DDL lease refresh interval can add complexity to the system, potentially leading to issues with system reliability and performance.

Q: How can I mitigate the potential risks and challenges associated with introducing a more fine-grained schema/DDL lease refresh interval?

A: To mitigate the potential risks and challenges associated with introducing a more fine-grained schema/DDL lease refresh interval, you can:

Monitor system performance: Regularly monitor system performance to ensure that the more frequent lease refresh operations are not causing issues.
Tune system configuration: Adjust system configuration to optimize performance and reliability.
Test and validate: Thoroughly test and validate the proposed solution to ensure that it meets the desired performance and reliability requirements.

Conclusion

In conclusion, introducing a more fine-grained schema/DDL lease refresh interval can bring several benefits to TiDB, including improved system reliability, enhanced performance, and better scalability. By reducing the lease refresh interval to 30 seconds, TiDB can refresh schema on time, even in cases of long latency or hot key collocation. We hope that this Q&A article has provided valuable insights and information to help you implement the proposed solution in your TiDB deployment.