High-Performance Order Storage: Backward Compatibility and Synchronization

This is the second deep dive into the High-Performance Order Storage solution principles. Please find the links to all the chapters in the initial progress report.

We understand that breaking stores with any change is unacceptable. This is also our main guiding principle when designing the High-Performance Order Storage (HPOS) and so we attempted to minimize the negative impact of this change as much as possible. Since not all plugins migrated to and use the CRUD layer introduced in WooCommerce 3.0 in 2017, it was apparent that we need to provide a solution that wouldn’t break for customers using those plugins, but would also motivate plugin developers to invest in updating their plugins.

The main mitigations we put in place are:

  1. The HPOS feature will be available as an opt-in experience so that no store should break on the WooCommerce plugin update.
  2. Order data is synchronized (duplicated) between the  wp_posts/wp_postmeta tables and the HPOS tables by default when the HPOS feature is enabled.
  3. Order id in the new table is always equal to the post id in the wp_posts table.

This means that once the HPOS feature is rolled out, merchants would be able to switch between the posts/postmeta tables and HPOS tables as authoritative tables.

User can switch between data stores freely when orders are in sync

Please note that switching between tables is only possible when the posts/postmeta and the new tables are in sync. This means you’ll need to wait for the initial sync to finish before you’d be able to switch:

Switching data storage is disabled when orders are out of sync

If plugins are interacting with order data via the WooCommerce-provided WC_Order class, then the transition should be seamless, and the data source in use should be transparent.

If any plugins interact with posts and postmeta directly, they should still work without issue as the synchronization will update the data. However, these plugins will need to be updated as we plan to stop the synchronization to posts/postmeta tables next year in WooCommerce 8.0.

Synchronization is an option that can also be turned off, so once the store admin is confident that all plugins are working successfully, they can disable the sync process and realize the full performance gains of HPOS tables.

Synchronization

Once the HPOS feature and synchronization are enabled, WC populates the HPOS tables with data from posts & postmeta tables. This is the first part of the project we implemented and tested with the community in the first call for testing in May. This initial population can be either run via Action Scheduler or via WP-CLI (using a new command wp wc cot sync).

Authoritative source set to HPOS & synchronization active

When the HPOS data store is active and the sync is on, the data is kept in sync via direct updates during the CRUD operations. This means the HPOS data store updates not only the new order tables but also the posts/postmeta tables. The updates need to happen at the same time because some plugins might still expect the data to be instantly readable from the posts/postmeta tables.

If some operation writes directly to posts & postmeta, the HPOS tables would get out of sync. Thus, we compare the orders during the read operation and if the post got updated later, we update the HPOS tables to match. The logic to determine the out of sync scenarios is the following:

  • If update time in CPT < HPOS, WC assumes a failed write to CPT, and it updates the CPT data with the HPOS data
  • If update time in CPT == HPOS, WC assumes a direct write to CPT, and it updates the HPOS data based on the CPT data

Authoritative source set to HPOS & synchronization disabled

With the HPOS active and data sync turned off, the HPOS data store adds a placeholder order record (which is a custom post type shop_order_placehold) to the post table. Nothing gets written to the postmeta table.

We’ve made this decision to ensure the invariant post.ID == order.id will always be true, which makes synchronization much easier. It also means any historically stored order ids will remain as correct references. This should help avoid difficult problems where e.g. a scheduled action plans to process an order in the future, only to suddenly realize during the execution of the scheduled action that the object stored at the given id is not an order.

Furthermore, this also allows us to make the transition easier for any objects that refer to orders, such as order notes or terms related to orders (used by some extensions). This means we can focus on implementation in areas where we believe we can make more performance gains.

We don’t currently support disabling the creation of placeholder posts, as it requires further implementation efforts. We believe adding this one INSERT to the `wp_posts` table shouldn’t be a performance problem. We can revisit this decision in the future, should the need arise.

Authoritative source set to old CPT & synchronization active

When the posts/postmeta tables are active, the old CPT data store writes to the posts and postmeta tables as usual, plus it enqueues an action via Action Scheduler to write the data to HPOS tables. 

As the authoritative source is set to the old CPT tables, the immediate presence of data in the HPOS tables is not required. A slight delay in sync shouldn’t create issues for anything that reads data via WC API, since switching over to HPOS is only possible once the tables are in sync.

Authoritative source set to old CPT & synchronization off

This configuration should work exactly the same as it’s worked since WooCommerce 3.0 until now, without any changes or overhead.

A word on performance

While we recognize this synchronization adds a small overhead over both the old CPT and the new HPOS tables compared to the situation when they’d be used by themselves, it’s a necessary transition phase while some plugins still expect records in the posts and postmeta tables. We expect this to be temporary mitigation to prevent stores from breaking. This also enables stores to roll back easily, as the orders would still be stored as they were previously. 

At the same time, this gives advanced users the option to realize the full potential of gains from HPOS tables, should they feel confident it’s safe to enable the new solution with their configuration. As soon as an individual store can verify that their plugins/code are not relying on the legacy CPT data, they will be able to disable the syncing process to fully rely on the new tables.

Verification

On the fly

To make the troubleshooting of potential problems encountered during sync easier, we’ve also updated the code that reads orders to read both data sources and compare the data, logging all discrepancies. This is very handy for making sure orders are in sync (in case no discrepancies have been reported) and allows us to debug potential problems more easily when bugs get reported back to us.

En masse

In addition to on-the-fly verification, we created a CLI tool that allows site owners to verify the consistency of the sync. This tool compares all the data between the posts/postmeta and the HPOS tables and reports all the differences to the standard output. To run this tool, execute wp wc cot verify_cot_data in the wp-cli environment.

Transactions

We’ve added support to run each synchronization batch to the HPOS tables inside a transaction to ensure data consistency. We’d be interested to hear from you if you encounter any issues running both with and without the transactions enabled.

This can be enabled in the HPOS feature settings, along with the required transaction isolation level:

Feedback

Please comment on this post, we’re excited to hear your thoughts! Let’s give WooCommerce the order storage it’s needed for a long time: scalable, performant, and flexible!


8 responses to “High-Performance Order Storage: Backward Compatibility and Synchronization”

  1. gileshold Avatar

    Hi there, is there a publicly available list of plugins that are already HPOS compatible? I ask only as a dev agency we rely on official Woo plugins to add functionality that we then develop to our client’s needs with additional features and integrations. An example of this is we use custom order status for a client which we then have an API that checks the statuses to feed to various warehouses for orders. We can adapt our solution to work with this but would need to know if the initial plugin is HPOS supported.

    Are you also planning to introduce a compatibility tag feature and some way of seeing this data, such as within system status reports?

    1. is there a publicly available list of plugins that are already HPOS compatible?

      Hi, while it’s not available yet, it should be available soon. We already have implemented compatibility tag feature, where plugins can choose to declare their compatibility with the HPOS feature. See this section of the upgrade guide for more details: https://github.com/woocommerce/woocommerce/wiki/High-Performance-Order-Storage-Upgrade-Recipe-Book#declaring-extension-incompatibility.

      Eventually, we will use these declarations to guide merchants on whether they can safely enable the feature in their shops or not. This GitHub issue is tracking this: https://github.com/woocommerce/woocommerce/issues/34861

      1. Is there a standard method for declaring theme compatibility (or incompatibility) with HPOS?

  2. If the option to disable posts/postmeta table synchronisation is available to users in WC 7.1, this effectively forces plugin developers to make their plugins HPOS compatible from WC 7.1, which is not too much time. I’m really excited about this feature, but I hope that disabling posts table synchronisation can be moved further in time to give us developers more time for this transition.

    1. Hey, if you need more time to make your plugins compatible, I’d recommend making use of the feature compatibility API to declare the plugin incompatible for now : https://github.com/woocommerce/woocommerce/wiki/High-Performance-Order-Storage-Upgrade-Recipe-Book#declaring-extension-incompatibility.

      We plan to use this in a UX which will guide merchants on whether they can enable HPOS or not. Not that the deprecation plan for posts table is August 2023, although you may want to make your plugin compatible way earlier.

      1. Thanks for your reply. Well, unfortunately declaring my WooCommerce-compatible plugin as incompatible with such a major feature of WooCommerce is not a solution for me. So, with the option to disable posts table synchronisation landing in WC 7.1, I’ll have no other option but make my plugin HPOS compatible in a really short time frame (not like it was initially stated that we would have time to transition until WC 8.0).

        1. Peter Fabian Avatar
          Peter Fabian

          Hi Dennis!

          We wanted to give people the option to disable the sync to allow them to better realize the performance benefit of HPOS. In the beginning, there will be many incompatible plugins. We very much appreciate you wanting to be compatible with HPOS for WC 7.1 in November, the transition period will take some time.

          Also, please don’t hesitate to join our October upgrade party where we are happy to guide you and help you out with making your plugins compatible. Our experience with the plugin catalog we maintain in Woo has been positive—plugins generally don’t require a lot of changes, unless they are using their own order data stores.

  3. Not totally related to this particular topic, but I’m curious about the new admin screens for viewing order lists. I have a site that displays a custom post_meta field as a sortable column in the orders list view, which relies on get_post_meta; will there be documentation on how to do this with the new admin screens?

Leave a Reply

Your email address will not be published. Required fields are marked *