The more complex the menu structure is, the more important it is to the customer is. And at the same time, the more complex the menu structure is, the less menu links are migrated to Drupal 9. We immediately realized this when we understood the bug in my previous post.

Motivation

Although we’ve built some patches for this customer, we aim to help not only Acquia customers, but as many Drupal 7 users as possible. This is why the latest Migrate Magician submodule will be created soon.

In this post, I share my basic notes I took while I was transforming the sanitized customer database to a minimal database fixture that I have been using during the development.

Fundaments

Maybe you already know this about me: I really like tests! Not just because they give me some extra confidence about the code I write, but because tests do the tedious and time-consuming, repetitive steps for me, and I can focus more on the actual development.

But this case was a bit special: we had a great, difficult menu structure in a client’s database which I wanted to use as test data during the development, when running the “real” test.

Maybe you don’t know about it, but Drupal 9 has a built-in CLI database application which can export MySQL/MariaDB databases into a PHP file. The only problem with it is that it creates one single PHP file. I wrote a smarter replacement on its fundamentals which creates per-table database fixture files, and it also can be configured to split big tables into chunks. This is Smart DB Tools, that is what I was using here.

Preparation

  1. I’ve exported our customer’s sanitized source database to a database fixture using Smart DB Tools.

    

    1
2
3

    		

     php ./modules/contrib/smart_db_tool/scripts/smart-db-tools.php dump\
   --database fixture_connection\
   --split-destination ./temp/d7-menu-link-raw.php

    The exported database fixture file, and the subdirectory which holds the per-table fixtures The exported database fixture file, and the subdirectory which holds the per-table fixtures

  2. I created a tests/fixtures/d7-menu-link-db.php file and a tests/fixtures/d7-menu-link-db subdirectory, moved the exported menu_links.php and menu_custom.php table fixture files into the subdirectory – because I assumed that executing the d7_menu_link migration will probably need these tables 😉.

  3. I wrote a very basic kernel test based on MigrateDrupalTestBase, specified the tests/fixtures/d7-menu-link-db.php file created previously as a fixture file, and executed the test, which only migrated d7_menu and d7_menu_links. Each time the test failed, I checked the error message, and either:
    • added the missing migration to the executeMigrations() method argument array
    • installed the missing core module, or the missing entity schema or module config
    • or if the error was about a missing database table a migration source plugin tried to access, then I created the corresponding table fixture file in the tests/fixtures/d7-menu-link-db subdirectory, copied the table schema from the original fixture into it, and added an include statement in the main tests/fixtures/d7-menu-link-db.php file.

  4. There are some tables which should be present,but can be empty. These were field_config, field_config_instance, role, role_permissions, users and user_roles. We obviously need a user for authoring the migrated nodes, but user 1 will be available in our kernel test.

  5. I also created empty fixture files for the node and the node_revision tables temporarily, and made a cleaned-up system table fixture file where I only kept the rows of the field, field_storage, menu, node, system and user modules (all of them enabled).

And this was the kernel test I used:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64



<?php

namespace Drupal\Tests\migmag_menu_link_migrate\Kernel;

use Drupal\menu_link_content\Entity\MenuLinkContent;
use Drupal\Tests\migmag\Traits\MigMagKernelTestDxTrait;
use Drupal\Tests\migrate_drupal\Kernel\MigrateDrupalTestBase;

/**
 * Tests the enhanced menu link migration.
 *
 * @group migmag_menu_link_migrate
 */
class MenuLinkMigrateTest extends MigrateDrupalTestBase {

  use MigMagKernelTestDxTrait;

  /**
   * {@inheritdoc}
   */
  protected static $modules = [
    'comment',
    'link',
    'menu_link_content',
    'node',
  ];

  /**
   * {@inheritdoc}
   */
  protected function setUp() {
    parent::setUp();

    $this->installEntitySchema('menu_link_content');

    $fixture_path = implode(DIRECTORY_SEPARATOR, [
      drupal_get_path('module', 'migmag_menu_link_migrate'),
      'tests',
      'fixtures',
      'd7-menu-link-db.php',
    ]);

    $this->loadFixture($fixture_path);
  }

  /**
   * Test the enhanced menu link migration.
   */
  public function testMenuLinkMigration() {
    $this->startCollectingMessages();
    $this->executeMigrations([
      'd7_node_type',
      'd7_user_role',
      'd7_user',
      'd7_node_complete',
      'd7_menu',
      'd7_menu_links',
    ]);
    $this->assertNoMigrationMessages();

    $this->assertCount(999, MenuLinkContent::loadMultiple());
  }

}

Write a database fixture with a test!

I removed the d7_node_type, d7_user_role, d7_user and d7_node_complete migrations, set a breakpoint after the migrations were executed, and checked how many records I have in the migrate map table of d7_menu_links: I had 289 messages! I went ahead and updated the count assertion: I want to have all of them migrated.

At this point, I had only one menu link which had been migrated. And almost every other menu link which failed to be migrated had a message in the migrate message table! Most of them (276 out of 283) contained a message like this:

d7_menu_links:link/uri: The path "internal:/node/801" failed validation.

This customer has more than 2500 nodes. Obviously, I don’t want to sanitize the whole client DB when I publish this work including a test with the database fixture. First, because it is an overhead, and on the other side I don’t want to migrate 2500 nodes (17000 revisions in case of using the complete node migration) just for being able to migrate 289 menu links. I only need the minimal data being available about these nodes. And the very minimal data is their ID. I can get them very easily, by parsing these migration messages!

  1. Let’s get the messages! This method was my tool:

    

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

    		

    /**
 * Returns the migration messages saved for the specified migration.
 *
 * @param string $plugin_id
 *   The (full) plugin ID of the corresponding migration plugin instance.
 * 
 * @return array[]
 *   The list of the migrate message record properties, containing only the 
 *   message (keyed by its column name 'message').
 */
protected function getMigrationMessages($plugin_id) {
  $migration = $this->getMigration($plugin_id);
  $idmap = $migration->getIdMap();
  assert($idmap instanceof Sql);
   
  return \Drupal::database()
    ->select($idmap->messageTableName(), 'm')
    ->fields('m', ['message'])
    ->execute()
    ->fetchAll(\PDO::FETCH_ASSOC);
}

  2. And now, I can get the IDs of those nodes which have a menu link:

    

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

    		

    /**
 * Obtains the required node IDs from the migration messages.
 * 
 * @param array[] $messages
 *   An array of the migrate map table records. At the very minimum, the 
 *   message key must be present (and its value should be the message).
 * 
 * @return int[]
 *   The "missing" node IDs.
 */
protected function getMissingNodeIds(array $messages) {
  $missing_node_ids = array_reduce(
    $messages,
    function (array $carry, array $message_data) {
      if (preg_match('/\sThe path "\w+:\/node\/(\d+).*" failed validation/', $message_data['message'], $matches)) {
        $carry[] = (int) $matches[1];
      }
      return $carry;
    },
    []
  );
  $missing_node_ids = array_unique($missing_node_ids);
  natsort($missing_node_ids);
  return array_values($missing_node_ids);
}

    This method returned 210 node IDs. Yes, this number is less than 276, but this difference means that we have some menu links which are pointing to the same node.

  3. The next task was adding records of these node IDs into the node and the node_revision DB table fixture. So I wrote a new helper method which consumes the list of these node IDs returned by ::getMissingNodeIds(), builds the appropriate data for the fixture, and exports it into a file.

    

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

    		

    /**
 * Saves a developer friendly (but incomplete) node table fixture file.
 * 
 * @param int[] $missing_node_ids
 *   The missing node IDs.
 */
protected function saveToNodeFixture(array $missing_node_ids): void {
  $raw_data = array_reduce(
    $missing_node_ids,
    function (array $carry, int $node_id) {
      $type = [
        'test_type1',
        'test_type2',
        'test_type3',
      ][random_int(0, 2)];
   
      $carry[$node_id] = [
        'nid' => $node_id,
        'vid' => $node_id,
        'type' => $type,
        'language' => 'und',
        'title' => "Node #{$node_id} title",
        'uid' => 1,
        'status' => 1,
        'created' => 1600000000 + $node_id,
        'changed' => 1600000000 + $node_id,
        'comment' => 0,
        'promote' => 0,
        'sticky' => 0,
        'tnid' => 0,
        'translate' => 0,
      ];
      return $carry;
    },
    []
  );
   
  $data_to_save = array_reduce(
    $raw_data,
    function (string $carry, array $data) {
      $carry .= "\n->values(";
      $carry .= trim(Variable::export($data));
      $carry .= ')';
      return $carry;
    },
    "<?php\n// phpcs:ignoreFile\n"
  );
  $data_to_save .= "\n->execute();";
   
  file_put_contents(implode(DIRECTORY_SEPARATOR, [
    drupal_get_path('module', 'migmag_menu_link_migrate'),
    'tests/fixtures',
    'node-cleaned.php'
  ]), $data_to_save);
}

  4. And the last step was that I modified the method above to write a fixture for my super-clean node_revisions table (node_revisions has slightly different table structure), and re-ran the test.

Profit

Basically, that was all. Although I did not dare to delete the original fixture file for a couple of hours, after a while I realized that if I still needed it, I could regenerate it anytime.

And I think I’ll finish this soonish! 🥳