Skip to content

injectHeartbeat writing failed on teardown #1663

@pasha132

Description

@pasha132

Hi all!

Environment:
Azure Mysql Flexible Sever, 8.0.44-azure
gh-ost 1.1.8

We have an instance with a number of partitioned tables and a large schema metadata. With this setup, the dry run check fails with the following errors (from the first to the last):

Closed streamer connection. err=<nil>
Dropping table `<redacted>_ghc`
Table dropped
Error 1146 (42S02): Table '<redacted>_ghc' doesn't exist
...
Error 1146 (42S02): Table '<redacted>_ghc' doesn't exist
...
injectHeartbeat writing failed 61 times, last error: Error 1146 (42S02): Table '<redacted>_ghc' doesn't exist

It appears that on slow disks/thousands of tables, finalCleanup is executed too early and drops the table before teardown (where injectHeartbeat actually stops). And deleting a '_ghc' table takes longer than default-retries (60) * heartbeat-interval-millis (100 ms).

Currently, this can be fixed by increasing default-retries or heartbeat-interval-millis (or both), but this seems like a workaround.

Is it possible to fix the order of execution?

Thanks in advance!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions