It seems like a relatively straightforward marketing article. I was pleasantly surprised to learn about Erathos though, nice product!
I am personally not a big fan of CDC in prod. Streaming data movement is generally prone to confusion, and it feeds into bad data patterns like hard deletes without any audit logs, no timestamps on updates or deletes, etc. which are usually the reason why batch loads cannot be utilized. They require a decent operational understanding of the underlying database, and have some gotchas like the Erathos folks mentioned in the article. We offer CDC both in our cloud platform, as well our open-source tools, but if I could, I would always pick an incremental batch load with a cursor value over a CDC connection.
I understand it is sometimes required due to organizational complexity or legacy database reasons, mine is just a personal preference.
> I would always pick an incremental batch load with a cursor value over a CDC connection.
What's the benefit vs. something like Postgres's logical replication for CDC? IMO, the hard part of CDC is maintaining consistency in the face of potential network issues or downstream slowdowns. One is forced to choose between scylla: generate excess trx logs if replication slows, and charybdis: lose consistency. I don't see how an open transaction helps here?
2) downstream consumers consuming a business-events table. Kind of like inverted event-sourcing pattern.
CDC is sound for sure but only in the lowest technically sense. Downstream consumers consuming business events gets you all the benefits of event sourcing(Namabilty, Replayability etc) while keeping relational guarantees and transactional safety.
Wrote this after seeing the same failure mode too many times: a MySQL sync job that looks fine in the dashboard but has been silently dropping deletes and intermediate updates for months, because it's comparing snapshots instead of reading the binlog. Tried to lay out exactly what has to be true on the MySQL side (row image, binlog_row_value_options, server-id, retention) for CDC to actually be complete, not just "eventually consistent." Happy to go deeper on any of it in the comments.
It is interesting to me! Do you know if the above applies to Datastream for CloudSQL as well, is it the same thing, or are you talking about a more DIY approach to CDC here?
I assume not everyone using MySQL and BigQuery are using CloudSQL and so don’t have Datastream available to them.
Datastream solves the issues mentioned in the article, because it relies on CDC, which is the same solution the article is proposing.
Note that the article seems to have a major error in its first sentence: “MySQL CDC syncs miss deletes and intermediate updates.” CDC is a solution to that issue, it doesn’t suffer from it. The second paragraph correctly describes the approach that has limitations: “a scheduled job selects rows…”
Disclaimer: I am cofounder of Bruin (https://github.com/bruin-data/bruin), we are a competitor to Erathos.
It seems like a relatively straightforward marketing article. I was pleasantly surprised to learn about Erathos though, nice product!
I am personally not a big fan of CDC in prod. Streaming data movement is generally prone to confusion, and it feeds into bad data patterns like hard deletes without any audit logs, no timestamps on updates or deletes, etc. which are usually the reason why batch loads cannot be utilized. They require a decent operational understanding of the underlying database, and have some gotchas like the Erathos folks mentioned in the article. We offer CDC both in our cloud platform, as well our open-source tools, but if I could, I would always pick an incremental batch load with a cursor value over a CDC connection.
I understand it is sometimes required due to organizational complexity or legacy database reasons, mine is just a personal preference.
If anyone is looking for an open-source CDC tool that runs as a standalone Go CLI, check out ingestr: https://github.com/bruin-data/ingestr
> I would always pick an incremental batch load with a cursor value over a CDC connection.
What's the benefit vs. something like Postgres's logical replication for CDC? IMO, the hard part of CDC is maintaining consistency in the face of potential network issues or downstream slowdowns. One is forced to choose between scylla: generate excess trx logs if replication slows, and charybdis: lose consistency. I don't see how an open transaction helps here?
I have no horse in this race, but CDC sounds more robust? It will capture all changes, regardless of how the application maintainers use the database.
I prefer following:
1) CDC for archiving/recovery or as an audit log
2) downstream consumers consuming a business-events table. Kind of like inverted event-sourcing pattern.
CDC is sound for sure but only in the lowest technically sense. Downstream consumers consuming business events gets you all the benefits of event sourcing(Namabilty, Replayability etc) while keeping relational guarantees and transactional safety.
Wrote this after seeing the same failure mode too many times: a MySQL sync job that looks fine in the dashboard but has been silently dropping deletes and intermediate updates for months, because it's comparing snapshots instead of reading the binlog. Tried to lay out exactly what has to be true on the MySQL side (row image, binlog_row_value_options, server-id, retention) for CDC to actually be complete, not just "eventually consistent." Happy to go deeper on any of it in the comments.
It is interesting to me! Do you know if the above applies to Datastream for CloudSQL as well, is it the same thing, or are you talking about a more DIY approach to CDC here?
I assume not everyone using MySQL and BigQuery are using CloudSQL and so don’t have Datastream available to them.
FYI, Datastream works with pretty much any MySQL instance, including self-hosted and even AWS RDS and MariaDB. See: https://docs.cloud.google.com/datastream/docs/sources-mysql
Datastream solves the issues mentioned in the article, because it relies on CDC, which is the same solution the article is proposing.
Note that the article seems to have a major error in its first sentence: “MySQL CDC syncs miss deletes and intermediate updates.” CDC is a solution to that issue, it doesn’t suffer from it. The second paragraph correctly describes the approach that has limitations: “a scheduled job selects rows…”