Pacemaker
The jump from Pacemaker 1.2 to 1.3 was a big step, especially since we took the opportunity to streamline our product a bit. For those who haven't noticed it yet: From now on M2IF is called Pacemaker Community. Pacemaker Community is still freely available as a community project and is supported on the same level as M2IF. You can find more information in the post. Have fun reading.
Community Edition 3.8.x
Pacemaker Community, formerly M2IF remains, as the name suggests, open source and a community project. In the course of the conversion, numerous bugfixes, optimizations and further developments were made in order to increase stability, add missing functions and simplify the integration into own projects.
Configuration
One of the most important enhancements of the Community Edition is certainly the decomposition of the monolithic configuration into small snippets, which can be quickly and easily overwritten in case of necessary adjustments such as changing the date format. In addition, further optimizations have been made to allow developers to make necessary adjustments as declarative as possible. For example, the configuration of the operations and the actual import workflow has been further decoupled. This means that if a new operation, e.g. the conversion of data, is necessary within the scope of a project, it is sufficient to simply integrate it at the desired point in the workflow. If an operation is modified, however, it can be explicitly overwritten with version 3.8.x without having to store the complete configuration in a separate file and to explicitly specify this file during the import.
Validation
Validation is a completely new feature. Based on the idea of the framework, it was completely implemented as a PlugIn and can be easily declaratively deactivated or activated. Furthermore, a CLI option can be used to define the number of errors to be issued. Additionally all errors are stored in a JSON file within the working directory to allow automated processing. Apart from errors in the validation of the files being imported, the most important database variables are also checked. Should optimization options be found at this point, these are displayed on the console as a note
Sortation
With version 3.8.x it is possible to control almost all sortings via import. This includes from now on
- The sorting of the attribute sets and groups
- The sorting of the attributes within an attribute group, in the filter navigation and the attribute options
- Sorting the subcategories within a category
- Sorting of products within a bundle, grouped, up sell, cross sell and accessory product
- Sorting the option values for the configuration of a bundle product
- Sorting the images of a product
- Sorting the configurable attributes for the creation of a configurable product
When importing the sort order, there is always the possibility to overwrite it in the Magento Admin like other values. However, from the moment you change the columns in the Magento Admin, the columns must either be empty or completely removed from the CSV file.
Dynamic processing of columns (also for non-EAV entities)
The dynamic processing of columns, especially for non-SAP entities, offers the possibility for customers to make certain adjustments such as sorting products within categories in the Magento Admin without overwriting these adjustments during import.
Clean-Up for product-web site relations
Starting with version 3.8.x, the Clean-Up functionality has been extended to include cleaning up the product-website relationship. Similar to the other Clean-Up options, the functionality can be activated by a corresponding flag in the configuration file.
Support for database prefixes
Newly added is the option to use database prefixes. Here the prefix can either be conveniently specified as an option directly during import via the CLI or it can be permanently stored in the configuration.
Create categories and attributes on-the-fly
Starting with version 3.7.x it was already possible to create missing attribute options on-the-fly. With version 3.8.x missing attribute options and categories are now created on-the-fly by default. In general, this functionality will be used mainly in the project phase, as data inconsistencies can occur and these are tolerated in order to have continuous data in the shop and to ensure further development. As soon as a shop goes live, however, the functionality is often deactivated, since from this point on it must be ensured that the data is consistent and valid. The dedicated import of the data is a good way to do this. In addition, only the values for the standard store can be created in the course of the on-the-fly option; translation is not possible.
Dedicated Product URL Import
Product URLs are usually imported via the product import. Pacemaker Community also provides a CLI command that can be used to import Product URLs. The required CSV file has been reduced to the minimum to make the import as efficient as possible
Professional Edition 1.3.0
The Professional Edition was completely revised during the migration from M2IF to Pacemaker. Besides additional functionalities, the focus was especially on performance. For example, depending on the available resources and the requirements, we succeeded in doubling the performance by combining features such as cache warming and change set detection. Furthermore, the Professional Edition now also contains a GUI to process files directly and conveniently via the Magento Admin area.
Upload GUI
The Magento Admin can also be used to upload files from version 1.3 and higher via a GUI. Alongside the selection of available entities, the operation, mode and other important parameters can be configured. Especially during the project phase, the GUI simplifies debugging and enables the project manager or the PO to import data for test purposes in an uncomplicated way.
See the documentation for more details and the upload GUI can be used.
Import Videos
Pacemaker Professional offers the option of importing videos and their sorting in combination with the images. To do this, the data can either be delivered directly in the file containing the products to be imported or in a dedicated file. If the data is delivered in a dedicated file, the import can be started with a separate command.
Individual Product Options
Similar to the import of videos, the Professional Edition also includes the option to import Custom Product Options. In addition, the order of the options and the option values for the Custom Product Options can be specified during import. Custom Product Options can be imported either directly with the products or via a dedicated file, in which case a command is available.
Caching
Version 1.3 has completely revised the existing caching functionality and can now be used from Professional Edition on. Caching is especially aimed at distributed systems where access to an external database is expensive, either in terms of cost or time. If caching is activated, data that has been loaded once is mostly kept in the cache and does not have to be loaded again. The caching has been designed in such a way that it can also be used in the context of own extensions of the import framework.
Caching is deactivated by default as the available memory can be quickly used up when caching is activated. Before activating caching, it is recommended that you ensure that the size of the available memory does not exceed the size of the data stored there during the import.
Cache warming
Building on the caching, the cache warming functionality provides a powerful feature that allows data to be loaded into the cache once upon import initialization in a optimized way and from then on to be used for imports. Cache warming, which can be activated via the configuration, accelerates the product import in particular, since database accesses are significantly reduced.
Want to know more about caching and cache warming? Have a look at our documentation.
Change Set Detection
Change Set Detection is an additional option to reduce access to the database. If the Change-Set Detection is activated, the system checks during the import of products whether the data in the database has changed and only then updates the database. This reduces write accesses to the database to a minimum. Especially in combination with caching and cache warming the performance of the product import can be optimized to a maximum.
Read how the Change Set Detection can be configured.
Timestamp Detection
An extremely effective optimization of the import is the ability to use a timestamp to check whether the data in the database has changed. To do this, the timestamp of the entity is compared with the timestamp in the CSV file and the data is only updated in the database if it differs. The timestamp detection can only be used if it is ensured that the CSV file contains an updated timestamp if the data in the third party system has changed and that the data is not maintained in the Magento Admin because the timestamp also changes and this can lead to conflict situations.
Read more about this feature in our documentation.
Enterprise Edition 1.3.0
The Enterprise Edition is and remains the solution for large and complex projects. Besides the clear positioning as THE Solution for the management of complex processes, the addition of the high-performance import framework and the know-how that can be used out-of-the-box in the form of ready-to-use pipelines is aimed towards the most cost-effective and quickest possible implementation of complex connections of third-party systems.
Generic Magento Executor
With the generic Magento Executor, all commands provided by the Magento CLI can be used in a pipeline. This makes it quick and easy to configure completely new pipelines or extend existing ones with additional functionality.
Upload GUI
In contrast to the Professional Edition, the GUI does not directly display the entities available for upload but rather the corresponding pipelines. For the import, the desired pipeline step corresponding to an entity can be selected and configured. Pacemaker then uses the pipeline to ensure that the import does not collide with other processes and that the consistency of the data is guaranteed at all times.
What's next?
The next Pacemaker version is already in the starting blocks and is already being developed diligently. When planning the next versions we try to incorporate the requirements of customers and solution partners, so wishes and suggestions are more than welcome.
Demo system
We are currently working on a demo system which will allow us to put the modules through their paces and check whether they can meet the requirements of a project. As soon as the demo system is available, all Solution Partners will be informed accordingly.
Upcoming Pacemaker v1.4, October 2020
We have already started work on Pacemaker 1.4 and currently plan to have it ready by early October 2020. Our roadmap includes the following features, but may change depending on requirements
- Generic order workflow functionality to synchronize the status of an order with ERP systems like SAP or Dynamics
- Extension of the SAP connector with a so-called pre-checkout functionality that compares prices with SAP before a product is added to the shopping cart
- Add/Update functionality for Custom Product Options based on SKUs for the Custom Option itself or its option values
- More transparency in processing the pipelines, e.g. by making the output of errors and additional information more transparent in the steps
- Enhancement of the clean-up functionality by the possibility to use a string constant like __EMPTY__VALUE__ to explicitly empty single fields of an entity
- Output of the live status of import processes as part of a step in the Magento Admin and via the CLI
- BMECat 1.2 Adapter for processing BMCat files e.g. from ERP systems
- Additional CLI commands to simplify debugging and prepare the directory structure required for import