<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.2.2">Jekyll</generator><link href="https://huzooka.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://huzooka.github.io/" rel="alternate" type="text/html" /><updated>2023-06-19T05:27:28+00:00</updated><id>https://huzooka.github.io/feed.xml</id><title type="html">HuZooka</title><subtitle>The Github Page where I temporarily publish my professional posts</subtitle><author><name>Zoltán Horváth</name></author><entry><title type="html">Views with Flaggings? You Can Make Them Fast!</title><link href="https://huzooka.github.io/development/2023/05/18/flagging-views.html" rel="alternate" type="text/html" title="Views with Flaggings? You Can Make Them Fast!" /><published>2023-05-18T14:24:57+00:00</published><updated>2023-05-18T14:24:57+00:00</updated><id>https://huzooka.github.io/development/2023/05/18/flagging-views</id><content type="html" xml:base="https://huzooka.github.io/development/2023/05/18/flagging-views.html"><![CDATA[<p class="lead">Nowadays, I am working on the migration of a Drupal 7 site with millions of nodes and 10 millions of comments. We also have some flaggings to import. But after we managed to migrate all the <em>Flagging</em> entities we needed, we found that our views on the main page got extremely slow 😳. As it turned out, we had an extremely slow Views query which used <em>flagging</em> relationship.</p>

<h2 id="the-problem">The problem</h2>

<p>The extremely slow query is caused by the different data type used in the <em>flagging</em> table. Nodes, comments (basically most the content entities I’m aware of) are using unsigned integers for their ID column, while <em>flagging</em> uses varchar. And in my opinion, <strong>Flag module is right!</strong> There is no constraint which forces entities to have integer IDs: it is just the default type in case of <em>content</em> entities.</p>

<p>Anyway, the query we need to execute joins <code class="language-plaintext highlighter-rouge">flagging</code> and <code class="language-plaintext highlighter-rouge">node_field_data</code> tables by comparing a <code class="language-plaintext highlighter-rouge">VARCHAR</code> column to an <code class="language-plaintext highlighter-rouge">INT</code> column. This has two disadvantages:</p>
<ol>
  <li>Comparison falls back to floating-point comparison which isn’t precize at all and this is also much slower than comparing a <code class="language-plaintext highlighter-rouge">VARCHAR</code> to <code class="language-plaintext highlighter-rouge">VARCHAR</code> or <code class="language-plaintext highlighter-rouge">INT</code> to <code class="language-plaintext highlighter-rouge">INT</code>. Size and other configurations also must match – in an ideal situation, an <code class="language-plaintext highlighter-rouge">INT(10) UNSIGNED</code> column should be compared to another <code class="language-plaintext highlighter-rouge">INT(10) UNSIGNED</code>).</li>
  <li><a href="https://dev.mysql.com/doc/refman/8.0/en/type-conversion.html">MySQL/MariaDB cannot use index</a>, so the query will be much slower:
    <blockquote>
      <p>For comparisons of a string column with a number, MySQL cannot use an index on the column to look up the value quickly.</p>
    </blockquote>

    <p>(By the way, there is no such index on the <code class="language-plaintext highlighter-rouge">flagging</code> table anyway.)</p>
  </li>
</ol>

<p>The simple solution would be to change the data type of the <code class="language-plaintext highlighter-rouge">entity_id</code> in the <code class="language-plaintext highlighter-rouge">flagging</code> table <strong>and</strong> add an index, but I think that would be a very bad idea, because if we did, it would break Flag module’s current capabilities and would possibly cause issues even if newer Flag releases try to change or update the column.</p>

<h2 id="the-solution-we-applied-for-views">The solution we applied (for Views)</h2>

<p>We decided to add a new integer column to the <code class="language-plaintext highlighter-rouge">flagging</code> table, then in Flag module’s <a href="https://git.drupalcode.org/project/flag/-/blob/9e12a609b2/src/Plugin/views/relationship/FlagViewsRelationship.php">Views relationship plugin</a>, we could use this column if the base table’s column’s type is also integer. So the tasks were:</p>

<ul>
  <li>Add a new column to the base table of flagging entities (so to the <code class="language-plaintext highlighter-rouge">flagging</code> table: <code class="language-plaintext highlighter-rouge">entity_id_int INT(10) UNSIGNED</code> defaulting to <code class="language-plaintext highlighter-rouge">NULL</code>).</li>
  <li>Ask the Flag module to fill this column with the value of flagging’s <code class="language-plaintext highlighter-rouge">entity_id</code> property if the property value is numeric.</li>
  <li>Change the related query added by the <code class="language-plaintext highlighter-rouge">flag_relationship</code> plugin, so if flagging records are joined and the other table’s appropriate column has integer type, we change <code class="language-plaintext highlighter-rouge">flagging.entity_id</code> column to <code class="language-plaintext highlighter-rouge">flagging.entity_id_int</code></li>
  <li>Add the appropriate index with an enhanced flagging storage schema handler.</li>
  <li>In an update hook, install the new column and the new storage schema handler.</li>
  <li>In a post update function, fill the new column with the values fetched from the preexisting record’s <code class="language-plaintext highlighter-rouge">entity_id</code> column.</li>
</ul>

<h3 id="new-base-field">New base field</h3>

<p>The easiest (and the Drupal-) way to add a new column to the base table of an entity is adding a new base field. Base fields are available for every entity bundle, which makes them the best fit for our situation too.</p>

<p>To add a new base field to an entity type provided by another module, we can implement <code class="language-plaintext highlighter-rouge">hook_entity_base_field_info()</code> (see the <a href="https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Entity%21ContentEntityBase.php/function/ContentEntityBase%3A%3AbaseFieldDefinitions">documentation at <code class="language-plaintext highlighter-rouge">ContentEntityBase::baseFieldDefinitions</code></a>):</p>

<div data-file="better_flag.module" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Implements hook_entity_base_field_info().
 */</span>
<span class="k">function</span> <span class="n">better_flag_entity_base_field_info</span><span class="p">(</span><span class="kt">EntityTypeInterface</span> <span class="nv">$entity_type</span><span class="p">):</span> <span class="kt">array</span> <span class="p">{</span>
  <span class="k">if</span> <span class="p">(</span><span class="nv">$entity_type</span><span class="o">-&gt;</span><span class="nf">id</span><span class="p">()</span> <span class="o">!==</span> <span class="s1">'flagging'</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="p">[];</span>
  <span class="p">}</span>

  <span class="k">return</span> <span class="p">[</span>
    <span class="s1">'entity_id_int'</span> <span class="o">=&gt;</span> <span class="nc">BaseFieldDefinition</span><span class="o">::</span><span class="nf">create</span><span class="p">(</span><span class="s1">'integer'</span><span class="p">)</span>
      <span class="o">-&gt;</span><span class="nf">setLabel</span><span class="p">(</span><span class="nf">t</span><span class="p">(</span><span class="s1">'Entity ID as integer'</span><span class="p">))</span>
      <span class="o">-&gt;</span><span class="nf">setDescription</span><span class="p">(</span><span class="nf">t</span><span class="p">(</span><span class="s1">'The ID of the flagged entity if it is integer.'</span><span class="p">))</span>
      <span class="o">-&gt;</span><span class="nf">setSetting</span><span class="p">(</span><span class="s1">'unsigned'</span><span class="p">,</span> <span class="kc">TRUE</span><span class="p">)</span>
      <span class="c1">// Provider must be set to 'flag'.</span>
      <span class="c1">// - If it's set to 'better_flag' then on uninstall, ModuleInstaller tries</span>
      <span class="c1">//   to remove it even if we delete this field in our uninstall hook. Why?</span>
      <span class="c1">//   Because this hook gets invoked!</span>
      <span class="c1">// - But if we don't do anything on uninstall, the entity schema won't be</span>
      <span class="c1">//   updated and the 'flagging' entity's schema will be outdated - meaning</span>
      <span class="c1">//   that the extra column in DB won't be removed.</span>
      <span class="o">-&gt;</span><span class="nf">setProvider</span><span class="p">(</span><span class="s1">'flag'</span><span class="p">)</span>
      <span class="o">-&gt;</span><span class="nf">setInitialValue</span><span class="p">(</span><span class="kc">NULL</span><span class="p">),</span>
  <span class="p">];</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Of course, if the entity was already available before our module was installed, we have to provide an update hook to have its column installed to the database table. This is one of the tasks that <code class="language-plaintext highlighter-rouge">EntityDefinitionUpdateManager</code> was designed for!</p>

<div data-file="better_flag.install" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Installs 'entity_id_int' column to Flagging entities' DB table.
 */</span>
<span class="k">function</span> <span class="n">better_flag_update_9001</span><span class="p">():</span> <span class="kt">void</span> <span class="p">{</span>
  <span class="err">\</span><span class="nc">Drupal</span><span class="o">::</span><span class="nf">entityDefinitionUpdateManager</span><span class="p">()</span><span class="o">-&gt;</span><span class="nf">installFieldStorageDefinition</span><span class="p">(</span>
    <span class="s1">'entity_id_int'</span><span class="p">,</span>
    <span class="s1">'flagging'</span><span class="p">,</span>
    <span class="s1">'flag'</span><span class="p">,</span>
    <span class="nc">BaseFieldDefinition</span><span class="o">::</span><span class="nf">create</span><span class="p">(</span><span class="s1">'integer'</span><span class="p">)</span>
      <span class="o">-&gt;</span><span class="nf">setLabel</span><span class="p">(</span><span class="nf">t</span><span class="p">(</span><span class="s1">'Entity ID as integer'</span><span class="p">))</span>
      <span class="o">-&gt;</span><span class="nf">setDescription</span><span class="p">(</span><span class="nf">t</span><span class="p">(</span><span class="s1">'The ID of the flagged entity if it is integer.'</span><span class="p">))</span>
      <span class="o">-&gt;</span><span class="nf">setSetting</span><span class="p">(</span><span class="s1">'unsigned'</span><span class="p">,</span> <span class="kc">TRUE</span><span class="p">)</span>
      <span class="o">-&gt;</span><span class="nf">setInitialValue</span><span class="p">(</span><span class="kc">NULL</span><span class="p">)</span>
  <span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>After our new base field was installed in the update hook above, we weren’t done yet:</p>
<ol>
  <li>We still had to update every pre-existing record, so they will have the right value in the new base field.</li>
  <li>We also had to ensure that new flaggings would have the appropriate value there.</li>
</ol>

<p>Let’s fill the new column with the right value for the preexisting flagging entities! Ideally, this should be done in a post update function. We only updated those records where the value of the original <code class="language-plaintext highlighter-rouge">entity_id</code> column only contained integer characters:</p>

<div data-file="better_flag.post_update.php" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Fill 'entity_id_int' column values.
 */</span>
<span class="k">function</span> <span class="n">better_flag_post_update_fill_entity_id_integer_column</span><span class="p">():</span> <span class="kt">void</span> <span class="p">{</span>
  <span class="k">try</span> <span class="p">{</span>
    <span class="nv">$flagging_definition</span> <span class="o">=</span> <span class="err">\</span><span class="nc">Drupal</span><span class="o">::</span><span class="nf">entityTypeManager</span><span class="p">()</span><span class="o">-&gt;</span><span class="nf">getDefinition</span><span class="p">(</span><span class="s1">'flagging'</span><span class="p">);</span>
    <span class="nv">$connection</span> <span class="o">=</span> <span class="err">\</span><span class="nc">Drupal</span><span class="o">::</span><span class="nf">database</span><span class="p">();</span>
    <span class="nv">$regexp_operator</span> <span class="o">=</span> <span class="nv">$connection</span><span class="o">-&gt;</span><span class="nf">databaseType</span><span class="p">()</span> <span class="o">===</span> <span class="s1">'pgsql'</span> <span class="o">?</span> <span class="s1">'~'</span> <span class="o">:</span> <span class="s1">'REGEXP'</span><span class="p">;</span>
    
    <span class="nv">$connection</span><span class="o">-&gt;</span><span class="nf">update</span><span class="p">(</span><span class="nv">$flagging_definition</span><span class="o">-&gt;</span><span class="nf">getBaseTable</span><span class="p">())</span>
      <span class="o">-&gt;</span><span class="nb">expression</span><span class="p">(</span><span class="s1">'entity_id_int'</span><span class="p">,</span> <span class="s1">'entity_id'</span><span class="p">)</span>
      <span class="o">-&gt;</span><span class="nf">condition</span><span class="p">(</span><span class="s1">'entity_id'</span><span class="p">,</span> <span class="s1">'^[0-9]+$'</span><span class="p">,</span> <span class="nv">$regexp_operator</span><span class="p">)</span>
      <span class="o">-&gt;</span><span class="nf">execute</span><span class="p">();</span>
  <span class="p">}</span>
  <span class="k">catch</span> <span class="p">(</span><span class="err">\</span><span class="nc">Exception</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">throw</span> <span class="k">new</span> <span class="nc">UpdateException</span><span class="p">(</span><span class="nv">$e</span><span class="o">-&gt;</span><span class="nf">getMessage</span><span class="p">());</span>
  <span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>After this, in a <code class="language-plaintext highlighter-rouge">hook_ENTITY_TYPE_presave()</code> implementation, we ensured that new flaggings have the right value in the new <code class="language-plaintext highlighter-rouge">entity_id_int</code> field.</p>

<div data-file="better_flag.module" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Implements hook_ENTITY_TYPE_presave() for flagging.
 */</span>
<span class="k">function</span> <span class="n">better_flag_flagging_presave</span><span class="p">(</span><span class="kt">EntityInterface</span> <span class="nv">$flagging</span><span class="p">)</span> <span class="p">{</span>
  <span class="nb">assert</span><span class="p">(</span><span class="nv">$flagging</span> <span class="k">instanceof</span> <span class="nc">FlaggingInterface</span><span class="p">);</span>
  <span class="nb">assert</span><span class="p">(</span><span class="nv">$flagging</span> <span class="k">instanceof</span> <span class="nc">ContentEntityBase</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nb">preg_match</span><span class="p">(</span><span class="s1">'/^\d{1,10}$/'</span><span class="p">,</span> <span class="nv">$flagged_entity_id</span> <span class="o">=</span> <span class="nv">$flagging</span><span class="o">-&gt;</span><span class="nf">getFlaggableId</span><span class="p">()))</span> <span class="p">{</span>
    <span class="k">return</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="c1">// We know that the entity ID is numeric (and it isn't longer than 10 digits),</span>
  <span class="c1">// so we can store the ID value in our extra integer field too.</span>
  <span class="nv">$flagging</span><span class="o">-&gt;</span><span class="nf">set</span><span class="p">(</span><span class="s1">'entity_id_int'</span><span class="p">,</span> <span class="nv">$flagged_entity_id</span><span class="p">);</span>
  <span class="nv">$flagging</span><span class="o">-&gt;</span><span class="nf">updateOriginalValues</span><span class="p">();</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="views-relationship">Views relationship</h3>

<p>The next task was to make the <code class="language-plaintext highlighter-rouge">flag_relationship</code> Views plugin to use the new integer column for joining its table whenever the base table’s corresponding field also has an integer type.</p>

<p>Luckily, Views utilizes Plugin API for most of the components it is using. It is very easy to alter for example the <code class="language-plaintext highlighter-rouge">flag_relationship</code> plugin. We don’t have too many options, but all what we need to do is to replace the plugin’s class in the plugin definition with our enhanced, “smarter” class:</p>

<div data-file="src/Plugin/views/relationship/BetterFlagViewsRelationship.php" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Replacement class for FlagViewsRelationship.
 */</span>
<span class="kd">class</span> <span class="nc">BetterFlagViewsRelationship</span> <span class="kd">extends</span> <span class="nc">FlagViewsRelationship</span> <span class="p">{</span>

  <span class="cd">/**
   * {@inheritdoc}
   */</span>
  <span class="k">public</span> <span class="k">function</span> <span class="n">query</span><span class="p">()</span> <span class="p">{</span>
    <span class="k">if</span> <span class="p">(</span>
      <span class="c1">// We have a real DB column to join on.</span>
      <span class="k">isset</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="n">realField</span><span class="p">)</span> <span class="o">&amp;&amp;</span>
      <span class="c1">// The current display has an entity type.</span>
      <span class="o">!</span><span class="nb">empty</span><span class="p">(</span><span class="nv">$entity_type</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">getEntityType</span><span class="p">())</span> <span class="o">&amp;&amp;</span>
      <span class="c1">// The column is one of the base fields of the entity type.</span>
      <span class="p">(</span><span class="nv">$base_field_definition</span> <span class="o">=</span> <span class="err">\</span><span class="nc">Drupal</span><span class="o">::</span><span class="nf">service</span><span class="p">(</span><span class="s1">'entity_field.manager'</span><span class="p">)</span><span class="o">-&gt;</span><span class="nf">getBaseFieldDefinitions</span><span class="p">(</span><span class="nv">$entity_type</span><span class="p">)[</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="n">realField</span><span class="p">]</span> <span class="o">??</span> <span class="kc">NULL</span><span class="p">)</span> <span class="o">&amp;&amp;</span>
      <span class="c1">// The type of the column is integer.</span>
      <span class="p">(</span><span class="nv">$base_field_definition</span> <span class="k">instanceof</span> <span class="nc">FieldDefinitionInterface</span> <span class="o">&amp;&amp;</span> <span class="nv">$base_field_definition</span><span class="o">-&gt;</span><span class="nb">getType</span><span class="p">()</span> <span class="o">===</span> <span class="s1">'integer'</span><span class="p">)</span>
    <span class="p">)</span> <span class="p">{</span>
      <span class="nv">$this</span><span class="o">-&gt;</span><span class="n">definition</span><span class="p">[</span><span class="s1">'base field'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'entity_id_int'</span><span class="p">;</span>
    <span class="p">}</span>

    <span class="k">parent</span><span class="o">::</span><span class="nf">query</span><span class="p">();</span>
  <span class="p">}</span>

<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>After this we could go ahead and change the plugin definition by using a <code class="language-plaintext highlighter-rouge">hook_views_plugins_relationship_alter()</code> hook:</p>

<div data-file="better_flag.module" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Implements hook_views_plugins_relationship_alter().
 */</span>
<span class="k">function</span> <span class="n">better_flag_views_plugins_relationship_alter</span><span class="p">(</span><span class="kt">array</span> <span class="o">&amp;</span><span class="nv">$plugins</span><span class="p">):</span> <span class="kt">void</span> <span class="p">{</span>
  <span class="c1">// Replaces flag_relationship views plugin with our own, improved version</span>
  <span class="c1">// which then speeds up queries using integer type of entity ID column in case</span>
  <span class="c1">// of joins.</span>
  <span class="k">if</span> <span class="p">(</span>
    <span class="k">isset</span><span class="p">(</span><span class="nv">$plugins</span><span class="p">[</span><span class="s1">'flag_relationship'</span><span class="p">])</span> <span class="o">&amp;&amp;</span>
    <span class="nv">$plugins</span><span class="p">[</span><span class="s1">'flag_relationship'</span><span class="p">][</span><span class="s1">'class'</span><span class="p">]</span> <span class="o">===</span> <span class="nc">FlagViewsRelationship</span><span class="o">::</span><span class="n">class</span>
  <span class="p">)</span> <span class="p">{</span>
    <span class="nv">$plugins</span><span class="p">[</span><span class="s1">'flag_relationship'</span><span class="p">][</span><span class="s1">'class'</span><span class="p">]</span> <span class="o">=</span> <span class="nc">BetterFlagViewsRelationship</span><span class="o">::</span><span class="n">class</span><span class="p">;</span>
  <span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="extend-the-entity-table-with-new-indexes">Extend the entity table with new indexes</h3>

<p>Finally, we extended the <a href="https://git.drupalcode.org/project/flag/-/blob/9e12a609b2/src/Entity/Storage/FlaggingStorageSchema.php">storage schema</a> of flagging entities. Why? Because without having the matching index in place, the query still took about 1200—1500ms to finish with about 1 million flaggings.</p>

<p><em>This is a bit of a fragile solution</em>: we cannot decorate these handlers, we can only change them. If another module has to do the same to flaggings, our replacement storage schema handler might be replaced, and we can’t take advantage of our dedicated index. All we can do is add a PHPUnit test that checks the schema handler of flaggings with all the modules in our project installed.</p>

<p>This is our replacement storage schema handler which adds the new index with <code class="language-plaintext highlighter-rouge">flag_id</code> and <code class="language-plaintext highlighter-rouge">entity_id_int</code>:</p>

<div data-file="src/Entity/Storage/BetterFlaggingStorageSchema.php" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Replacement class for Flagging entity storage schema handler.
 */</span>
<span class="kd">class</span> <span class="nc">BetterFlaggingStorageSchema</span> <span class="kd">extends</span> <span class="nc">FlaggingStorageSchema</span> <span class="p">{</span>

  <span class="cd">/**
   * {@inheritdoc}
   */</span>
  <span class="k">protected</span> <span class="k">function</span> <span class="n">getEntitySchema</span><span class="p">(</span><span class="kt">ContentEntityTypeInterface</span> <span class="nv">$entity_type</span><span class="p">,</span> <span class="nv">$reset</span> <span class="o">=</span> <span class="kc">FALSE</span><span class="p">)</span> <span class="p">{</span>
    <span class="nv">$schema</span> <span class="o">=</span> <span class="k">parent</span><span class="o">::</span><span class="nf">getEntitySchema</span><span class="p">(</span><span class="nv">$entity_type</span><span class="p">,</span> <span class="nv">$reset</span><span class="p">);</span>
    <span class="nv">$schema</span><span class="p">[</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="n">entityType</span><span class="o">-&gt;</span><span class="nf">getBaseTable</span><span class="p">()][</span><span class="s1">'indexes'</span><span class="p">]</span> <span class="o">+=</span> <span class="p">[</span>
      <span class="s1">'better__flag_id__entity_id_int'</span> <span class="o">=&gt;</span> <span class="p">[</span>
        <span class="s1">'flag_id'</span><span class="p">,</span>
        <span class="s1">'entity_id_int'</span><span class="p">,</span>
      <span class="p">],</span>
    <span class="p">];</span>

    <span class="k">return</span> <span class="nv">$schema</span><span class="p">;</span>
  <span class="p">}</span>

<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Of course, we also had to replace the original handler with our own. This can be done with a <code class="language-plaintext highlighter-rouge">hook_entity_type_alter()</code> implementation:</p>

<div data-file="better_flag.module" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Implements hook_entity_type_alter().
 */</span>
<span class="k">function</span> <span class="n">better_flag_entity_type_alter</span><span class="p">(</span><span class="kt">array</span> <span class="nv">$entity_types</span><span class="p">):</span> <span class="kt">void</span> <span class="p">{</span>
  <span class="c1">// Replace storage schema class of flagging entities.</span>
  <span class="nb">assert</span><span class="p">(</span><span class="nv">$entity_types</span><span class="p">[</span><span class="s1">'flagging'</span><span class="p">]</span> <span class="k">instanceof</span> <span class="nc">EntityTypeInterface</span><span class="p">);</span>
  <span class="nv">$entity_types</span><span class="p">[</span><span class="s1">'flagging'</span><span class="p">]</span><span class="o">-&gt;</span><span class="nf">setHandlerClass</span><span class="p">(</span><span class="s1">'storage_schema'</span><span class="p">,</span> <span class="nc">BetterFlaggingStorageSchema</span><span class="o">::</span><span class="n">class</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>As the last step, we needed yet another update hook where we could install the new storage schema. In our case, this means that – under the hood – Drupal will drop every preexisting index on the corresponding table(s) and then recreate the new ones.</p>

<div data-file="better_flag.install" class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Install the new storage schema handler of flagging entities.
 */</span>
<span class="k">function</span> <span class="n">better_flag_update_9002</span><span class="p">():</span> <span class="kt">void</span> <span class="p">{</span>
  <span class="nv">$entity_definition_update_manager</span> <span class="o">=</span> <span class="err">\</span><span class="nc">Drupal</span><span class="o">::</span><span class="nf">entityDefinitionUpdateManager</span><span class="p">();</span>
  <span class="nv">$entity_type</span> <span class="o">=</span> <span class="nv">$entity_definition_update_manager</span><span class="o">-&gt;</span><span class="nf">getEntityType</span><span class="p">(</span><span class="s1">'flagging'</span><span class="p">);</span>
  <span class="nv">$entity_type</span><span class="o">-&gt;</span><span class="nf">setHandlerClass</span><span class="p">(</span><span class="s1">'storage_schema'</span><span class="p">,</span> <span class="nc">BetterFlaggingStorageSchema</span><span class="o">::</span><span class="n">class</span><span class="p">);</span>
  <span class="nv">$entity_definition_update_manager</span><span class="o">-&gt;</span><span class="nf">updateEntityType</span><span class="p">(</span><span class="nv">$entity_type</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p><strong>That was all!</strong></p>

<h2 id="diagnostic">Diagnostic</h2>

<p>If you’ve reached this line, I’m pretty sure you’re wondering what happened to the Views database query. “Unfortunately”, I was not patient enough to wait for the unoptimized database query to finish 😶.</p>

<h3 id="original">Original</h3>

<p>This was the original query (no result after <strong>~30 minutes</strong>):</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre><span class="k">SELECT</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">langcode</span> <span class="k">AS</span> <span class="n">node_field_data_langcode</span><span class="p">,</span> 
    <span class="n">flagging_node_field_data</span><span class="p">.</span><span class="n">uid</span> <span class="k">AS</span> <span class="n">flagging_node_field_data_uid</span><span class="p">,</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">changed</span> <span class="k">AS</span> <span class="n">node_field_data_changed</span><span class="p">,</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">nid</span> <span class="k">AS</span> <span class="n">nid</span><span class="p">,</span> 
    <span class="n">flagging_node_field_data</span><span class="p">.</span><span class="n">id</span> <span class="k">AS</span> <span class="n">flagging_node_field_data_id</span> 
<span class="k">FROM</span> <span class="n">node_field_data</span> <span class="n">node_field_data</span> 
<span class="k">LEFT</span> <span class="k">JOIN</span> <span class="n">flagging</span> <span class="n">flagging_node_field_data</span> <span class="k">ON</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">nid</span> <span class="o">=</span> <span class="n">flagging_node_field_data</span><span class="p">.</span><span class="n">entity_id</span> <span class="k">AND</span> 
    <span class="n">flagging_node_field_data</span><span class="p">.</span><span class="n">flag_id</span> <span class="o">=</span> <span class="s1">'custom_flag_id'</span> 
<span class="k">LEFT</span> <span class="k">JOIN</span> <span class="n">comment_entity_statistics</span> <span class="n">comment_entity_statistics</span> <span class="k">ON</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">nid</span> <span class="o">=</span> <span class="n">comment_entity_statistics</span><span class="p">.</span><span class="n">entity_id</span> <span class="k">AND</span> 
    <span class="n">comment_entity_statistics</span><span class="p">.</span><span class="n">entity_type</span> <span class="o">=</span> <span class="s1">'node'</span> 
<span class="k">WHERE</span> 
    <span class="p">(</span><span class="n">node_field_data</span><span class="p">.</span><span class="n">status</span> <span class="o">=</span> <span class="s1">'1'</span><span class="p">)</span> <span class="k">AND</span> 
    <span class="p">(</span><span class="n">node_field_data</span><span class="p">.</span><span class="k">type</span> <span class="k">IN</span> <span class="p">(</span><span class="s1">'topic'</span><span class="p">))</span> <span class="k">AND</span> 
    <span class="p">(</span><span class="n">comment_entity_statistics</span><span class="p">.</span><span class="n">comment_count</span> <span class="o">&gt;</span> <span class="s1">'0'</span><span class="p">)</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">node_field_data_changed</span> <span class="k">DESC</span> 
<span class="k">LIMIT</span> <span class="mi">10</span> <span class="k">OFFSET</span> <span class="mi">0</span><span class="p">;</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Original <code class="language-plaintext highlighter-rouge">EXPLAIN</code> result:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre><span class="o">+</span><span class="c1">----+-------------+---------------------------+------------+-------+--------------------------------------------------------------------------------------------+--------------------------------------+---------+--------------------------------------------+---------+----------+-----------------------------------------------------------+</span>
<span class="o">|</span> <span class="n">id</span> <span class="o">|</span> <span class="n">select_type</span> <span class="o">|</span> <span class="k">table</span>                     <span class="o">|</span> <span class="n">partitions</span> <span class="o">|</span> <span class="k">type</span>  <span class="o">|</span> <span class="n">possible_keys</span>                                                                              <span class="o">|</span> <span class="k">key</span>                                  <span class="o">|</span> <span class="n">key_len</span> <span class="o">|</span> <span class="k">ref</span>                                        <span class="o">|</span> <span class="k">rows</span>    <span class="o">|</span> <span class="n">filtered</span> <span class="o">|</span> <span class="n">Extra</span>                                                     <span class="o">|</span>
<span class="o">+</span><span class="c1">----+-------------+---------------------------+------------+-------+--------------------------------------------------------------------------------------------+--------------------------------------+---------+--------------------------------------------+---------+----------+-----------------------------------------------------------+</span>
<span class="o">|</span>  <span class="mi">1</span> <span class="o">|</span> <span class="k">SIMPLE</span>      <span class="o">|</span> <span class="n">comment_entity_statistics</span> <span class="o">|</span> <span class="k">NULL</span>       <span class="o">|</span> <span class="k">range</span> <span class="o">|</span> <span class="k">PRIMARY</span><span class="p">,</span><span class="n">comment_count</span><span class="p">,</span><span class="n">testing__entity_id__entity_type__comment_count</span>                       <span class="o">|</span> <span class="n">comment_count</span>                        <span class="o">|</span> <span class="mi">4</span>       <span class="o">|</span> <span class="k">NULL</span>                                       <span class="o">|</span> <span class="mi">1874806</span> <span class="o">|</span>    <span class="mi">10</span><span class="p">.</span><span class="mi">00</span> <span class="o">|</span> <span class="k">Using</span> <span class="k">where</span><span class="p">;</span> <span class="k">Using</span> <span class="k">index</span><span class="p">;</span> <span class="k">Using</span> <span class="k">temporary</span><span class="p">;</span> <span class="k">Using</span> <span class="n">filesort</span> <span class="o">|</span>
<span class="o">|</span>  <span class="mi">1</span> <span class="o">|</span> <span class="k">SIMPLE</span>      <span class="o">|</span> <span class="n">node_field_data</span>           <span class="o">|</span> <span class="k">NULL</span>       <span class="o">|</span> <span class="k">ref</span>   <span class="o">|</span> <span class="k">PRIMARY</span><span class="p">,</span><span class="n">node__id__default_langcode__langcode</span><span class="p">,</span><span class="n">node_field__type__target_id</span><span class="p">,</span><span class="n">node__status_type</span> <span class="o">|</span> <span class="n">node__id__default_langcode__langcode</span> <span class="o">|</span> <span class="mi">4</span>       <span class="o">|</span> <span class="n">drupal</span><span class="p">.</span><span class="n">comment_entity_statistics</span><span class="p">.</span><span class="n">entity_id</span> <span class="o">|</span>       <span class="mi">1</span> <span class="o">|</span>     <span class="mi">5</span><span class="p">.</span><span class="mi">00</span> <span class="o">|</span> <span class="k">Using</span> <span class="k">where</span>                                               <span class="o">|</span>
<span class="o">|</span>  <span class="mi">1</span> <span class="o">|</span> <span class="k">SIMPLE</span>      <span class="o">|</span> <span class="n">flagging_node_field_data</span>  <span class="o">|</span> <span class="k">NULL</span>       <span class="o">|</span> <span class="k">ref</span>   <span class="o">|</span> <span class="n">flagging_field__flag_id__target_id</span><span class="p">,</span><span class="n">entity_id__uid</span>                                          <span class="o">|</span> <span class="n">flagging_field__flag_id__target_id</span>   <span class="o">|</span> <span class="mi">34</span>      <span class="o">|</span> <span class="n">const</span>                                      <span class="o">|</span>    <span class="mi">3091</span> <span class="o">|</span>   <span class="mi">100</span><span class="p">.</span><span class="mi">00</span> <span class="o">|</span> <span class="k">Using</span> <span class="k">where</span>                                               <span class="o">|</span>
<span class="o">+</span><span class="c1">----+-------------+---------------------------+------------+-------+--------------------------------------------------------------------------------------------+--------------------------------------+---------+--------------------------------------------+---------+----------+-----------------------------------------------------------+</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="optimized-query">Optimized query</h3>

<p>This is the optimized query – it finishes in <strong>1ms-3ms</strong>!:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre><span class="k">SELECT</span>
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">langcode</span> <span class="k">AS</span> <span class="n">node_field_data_langcode</span><span class="p">,</span> 
    <span class="n">flagging_node_field_data</span><span class="p">.</span><span class="n">uid</span> <span class="k">AS</span> <span class="n">flagging_node_field_data_uid</span><span class="p">,</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">changed</span> <span class="k">AS</span> <span class="n">node_field_data_changed</span><span class="p">,</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">nid</span> <span class="k">AS</span> <span class="n">nid</span><span class="p">,</span> 
    <span class="n">flagging_node_field_data</span><span class="p">.</span><span class="n">id</span> <span class="k">AS</span> <span class="n">flagging_node_field_data_id</span> 
<span class="k">FROM</span> <span class="n">node_field_data</span> <span class="n">node_field_data</span> 
<span class="k">LEFT</span> <span class="k">JOIN</span> <span class="n">flagging</span> <span class="n">flagging_node_field_data</span> <span class="k">ON</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">nid</span> <span class="o">=</span> <span class="n">flagging_node_field_data</span><span class="p">.</span><span class="n">entity_id_int</span> <span class="k">AND</span> 
    <span class="n">flagging_node_field_data</span><span class="p">.</span><span class="n">flag_id</span> <span class="o">=</span> <span class="s1">'custom_flag_id'</span>
<span class="k">LEFT</span> <span class="k">JOIN</span> <span class="n">comment_entity_statistics</span> <span class="n">comment_entity_statistics</span> <span class="k">ON</span> 
    <span class="n">node_field_data</span><span class="p">.</span><span class="n">nid</span> <span class="o">=</span> <span class="n">comment_entity_statistics</span><span class="p">.</span><span class="n">entity_id</span> <span class="k">AND</span> 
    <span class="n">comment_entity_statistics</span><span class="p">.</span><span class="n">entity_type</span> <span class="o">=</span> <span class="s1">'node'</span> 
<span class="k">WHERE</span> 
    <span class="p">(</span><span class="n">node_field_data</span><span class="p">.</span><span class="n">status</span> <span class="o">=</span> <span class="s1">'1'</span><span class="p">)</span> <span class="k">AND</span> 
    <span class="p">(</span><span class="n">node_field_data</span><span class="p">.</span><span class="k">type</span> <span class="k">IN</span> <span class="p">(</span><span class="s1">'topic'</span><span class="p">))</span> <span class="k">AND</span> 
    <span class="p">(</span><span class="n">comment_entity_statistics</span><span class="p">.</span><span class="n">comment_count</span> <span class="o">&gt;</span> <span class="s1">'0'</span><span class="p">)</span> 
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">node_field_data_changed</span> <span class="k">DESC</span> 
<span class="k">LIMIT</span> <span class="mi">10</span> <span class="k">OFFSET</span> <span class="mi">0</span><span class="p">;</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Optimized <code class="language-plaintext highlighter-rouge">EXPLAIN</code> result:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre><span class="o">+</span><span class="c1">----+-------------+---------------------------+------------+-------+--------------------------------------------------------------------------------------------+------------------------------------------------+---------+----------------------------------+------+----------+--------------------------+</span>
<span class="o">|</span> <span class="n">id</span> <span class="o">|</span> <span class="n">select_type</span> <span class="o">|</span> <span class="k">table</span>                     <span class="o">|</span> <span class="n">partitions</span> <span class="o">|</span> <span class="k">type</span>  <span class="o">|</span> <span class="n">possible_keys</span>                                                                              <span class="o">|</span> <span class="k">key</span>                                            <span class="o">|</span> <span class="n">key_len</span> <span class="o">|</span> <span class="k">ref</span>                              <span class="o">|</span> <span class="k">rows</span> <span class="o">|</span> <span class="n">filtered</span> <span class="o">|</span> <span class="n">Extra</span>                    <span class="o">|</span>
<span class="o">+</span><span class="c1">----+-------------+---------------------------+------------+-------+--------------------------------------------------------------------------------------------+------------------------------------------------+---------+----------------------------------+------+----------+--------------------------+</span>
<span class="o">|</span>  <span class="mi">1</span> <span class="o">|</span> <span class="k">SIMPLE</span>      <span class="o">|</span> <span class="n">node_field_data</span>           <span class="o">|</span> <span class="k">NULL</span>       <span class="o">|</span> <span class="k">index</span> <span class="o">|</span> <span class="k">PRIMARY</span><span class="p">,</span><span class="n">node__id__default_langcode__langcode</span><span class="p">,</span><span class="n">node_field__type__target_id</span><span class="p">,</span><span class="n">node__status_type</span> <span class="o">|</span> <span class="n">node_field__changed</span>                            <span class="o">|</span> <span class="mi">4</span>       <span class="o">|</span> <span class="k">NULL</span>                             <span class="o">|</span>   <span class="mi">36</span> <span class="o">|</span>     <span class="mi">5</span><span class="p">.</span><span class="mi">00</span> <span class="o">|</span> <span class="k">Using</span> <span class="k">where</span>              <span class="o">|</span>
<span class="o">|</span>  <span class="mi">1</span> <span class="o">|</span> <span class="k">SIMPLE</span>      <span class="o">|</span> <span class="n">comment_entity_statistics</span> <span class="o">|</span> <span class="k">NULL</span>       <span class="o">|</span> <span class="k">ref</span>   <span class="o">|</span> <span class="k">PRIMARY</span><span class="p">,</span><span class="n">comment_count</span><span class="p">,</span><span class="n">testing__entity_id__entity_type__comment_count</span>                       <span class="o">|</span> <span class="n">testing__entity_id__entity_type__comment_count</span> <span class="o">|</span> <span class="mi">38</span>      <span class="o">|</span> <span class="n">drupal</span><span class="p">.</span><span class="n">node_field_data</span><span class="p">.</span><span class="n">nid</span><span class="p">,</span><span class="n">const</span> <span class="o">|</span>    <span class="mi">1</span> <span class="o">|</span>    <span class="mi">50</span><span class="p">.</span><span class="mi">00</span> <span class="o">|</span> <span class="k">Using</span> <span class="k">where</span><span class="p">;</span> <span class="k">Using</span> <span class="k">index</span> <span class="o">|</span>
<span class="o">|</span>  <span class="mi">1</span> <span class="o">|</span> <span class="k">SIMPLE</span>      <span class="o">|</span> <span class="n">flagging_node_field_data</span>  <span class="o">|</span> <span class="k">NULL</span>       <span class="o">|</span> <span class="k">ref</span>   <span class="o">|</span> <span class="n">flagging_field__flag_id__target_id</span><span class="p">,</span><span class="n">better__flag_id__entity_id_int</span>                          <span class="o">|</span> <span class="n">better__flag_id__entity_id_int</span>                 <span class="o">|</span> <span class="mi">39</span>      <span class="o">|</span> <span class="n">drupal</span><span class="p">.</span><span class="n">node_field_data</span><span class="p">.</span><span class="n">nid</span><span class="p">,</span><span class="n">const</span> <span class="o">|</span>    <span class="mi">1</span> <span class="o">|</span>   <span class="mi">100</span><span class="p">.</span><span class="mi">00</span> <span class="o">|</span> <span class="k">NULL</span>                     <span class="o">|</span>
<span class="o">+</span><span class="c1">----+-------------+---------------------------+------------+-------+--------------------------------------------------------------------------------------------+------------------------------------------------+---------+----------------------------------+------+----------+--------------------------+</span>
</pre></td></tr></tbody></table></code></pre></div></div>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Drupal" /><category term="Flag" /><category term="Views" /><category term="Performance" /><category term="Database API" /><summary type="html"><![CDATA[How we accelerated our extremely slow Views queries which were using flagging relationship.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/files/image/snail.jpg" /><media:content medium="image" url="https://huzooka.github.io/files/image/snail.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Scrollable Tables in Jekyll Posts</title><link href="https://huzooka.github.io/development/2022/07/26/jekyll-table-wrapper.html" rel="alternate" type="text/html" title="Scrollable Tables in Jekyll Posts" /><published>2022-07-26T20:20:53+00:00</published><updated>2022-07-26T20:20:53+00:00</updated><id>https://huzooka.github.io/development/2022/07/26/jekyll-table-wrapper</id><content type="html" xml:base="https://huzooka.github.io/development/2022/07/26/jekyll-table-wrapper.html"><![CDATA[<p class="lead">I have been very satisfied with Jekyll. It is very comfortable for me to write my blog posts in Markdown, and the maintenance of a static site generator based project needs a lot less time than anything else. But when I was writing <a href="/development/2018/12/06/using-nightwatch-for-screenshots.html">my post about Claro’s screenshot automation</a> I found a very interesting challenge: the table I put into the article caused vertical overflow on narrow screen width devices.</p>

<p>The solution for this kind of issue is pretty easy: we only have to wrap the table into a (block level) HTML element; let the wrapper element inherit the available width but make it overflow vertically. But how can I achieve this with Jekyll?</p>

<p>I was sure I am not the only person having this issue. But <a href="https://github.com/gettalong/kramdown/issues/69">people at this issue</a> were suggesting to add the needed HTML markup to the Markdown document – what I don’t really like to do: On one hand, I want to keep my Markdown files as “clean” as possible. On the other hand, it is cleaner and much more maintainable to process the HTML output at a very late phase somehow, and then automatically add the wrapper HTML tag where it is needed.</p>

<p>Luckily, I found some good news: Jekyll provides a <a href="https://jekyllrb.com/docs/plugins/hooks/">helpful infrastructure called “Hooks”</a> that we can use for this! We only have to implement a <a href="https://jekyllrb.com/docs/plugins/">Jekyll plugin</a>, register its (static) method as a hook, which then gets invoked on the desired event and can modify the generated output.</p>

<p>The simplest and easiest approach imho is to create a file in the <code class="language-plaintext highlighter-rouge">&lt;project root&gt;/_plugins</code> directory named <code class="language-plaintext highlighter-rouge">table-wrapper.rb</code>. This file contains the class of the table wrapper Jekyll plugin, and also registers the appropriate method:</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="rouge-code"><pre><span class="nb">require</span> <span class="s1">'nokogiri'</span><span class="p">;</span>

<span class="k">module</span> <span class="nn">Jekyll</span>
  <span class="k">class</span> <span class="nc">TableWrapper</span>
    <span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">process</span><span class="p">(</span><span class="n">document</span><span class="p">)</span>
    <span class="k">end</span>
  <span class="k">end</span>
<span class="k">end</span>

<span class="no">Jekyll</span><span class="o">::</span><span class="no">Hooks</span><span class="p">.</span><span class="nf">register</span> <span class="p">[</span><span class="ss">:pages</span><span class="p">,</span> <span class="ss">:documents</span><span class="p">],</span> <span class="ss">:post_render</span> <span class="k">do</span> <span class="o">|</span><span class="n">document</span><span class="o">|</span>
  <span class="no">Jekyll</span><span class="o">::</span><span class="no">TableWrapper</span><span class="p">.</span><span class="nf">process</span><span class="p">(</span><span class="n">document</span><span class="p">)</span> <span class="k">if</span> <span class="n">document</span><span class="p">.</span><span class="nf">write?</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>To parse and modify the actual file’s output, I chose <a href="https://nokogiri.org/">Nokogiri</a>, mostly because it seems to be pretty common in the Ruby community. My strategy is pretty simple: if the current output contains a <code class="language-plaintext highlighter-rouge">&lt;table&gt;</code>, I wrap it into a <code class="language-plaintext highlighter-rouge">&lt;div class="vertical-scroll-wrapper"&gt;</code> HTML element – unless the table’s parent is an overflow container.</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
</pre></td><td class="rouge-code"><pre><span class="nb">require</span> <span class="s1">'nokogiri'</span><span class="p">;</span>

<span class="k">module</span> <span class="nn">Jekyll</span>
  <span class="k">class</span> <span class="nc">TableWrapper</span>
    <span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">process</span><span class="p">(</span><span class="n">document</span><span class="p">)</span>
      <span class="n">wrapper_tag</span> <span class="o">=</span> <span class="s1">'div'</span><span class="p">;</span>
      <span class="n">wrapper_css_class</span> <span class="o">=</span> <span class="s1">'vertical-scroll-wrapper'</span><span class="p">;</span>
      <span class="n">wrapper</span> <span class="o">=</span> <span class="s2">"&lt;</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2"> class=</span><span class="se">\"</span><span class="si">#{</span><span class="n">wrapper_css_class</span><span class="si">}</span><span class="se">\"</span><span class="s2">&gt;&lt;/</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2">&gt;"</span>

      <span class="n">xpath_selector</span> <span class="o">=</span> <span class="s1">'//body//table'</span>
      <span class="n">xpath_selector</span> <span class="o">&lt;&lt;</span> <span class="s2">"[not(parent::</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2">[contains(concat(' ', normalize-space(@class), ' '), ' </span><span class="si">#{</span><span class="n">wrapper_css_class</span><span class="si">}</span><span class="s2"> ')])]"</span>

      <span class="n">parsed_document</span> <span class="o">=</span> <span class="no">Nokogiri</span><span class="o">::</span><span class="no">HTML</span><span class="p">(</span><span class="n">document</span><span class="p">.</span><span class="nf">output</span><span class="p">)</span>
      <span class="n">parsed_document</span><span class="p">.</span><span class="nf">search</span><span class="p">(</span><span class="n">xpath_selector</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">table_node</span><span class="o">|</span>
        <span class="n">table_node</span><span class="p">.</span><span class="nf">wrap</span><span class="p">(</span><span class="n">wrapper</span><span class="p">)</span>
      <span class="k">end</span>

      <span class="n">document</span><span class="p">.</span><span class="nf">output</span> <span class="o">=</span> <span class="n">parsed_document</span><span class="p">.</span><span class="nf">to_html</span>
    <span class="k">end</span>
  <span class="k">end</span>
<span class="k">end</span>

<span class="no">Jekyll</span><span class="o">::</span><span class="no">Hooks</span><span class="p">.</span><span class="nf">register</span> <span class="p">[</span><span class="ss">:pages</span><span class="p">,</span> <span class="ss">:documents</span><span class="p">],</span> <span class="ss">:post_render</span> <span class="k">do</span> <span class="o">|</span><span class="n">output</span><span class="o">|</span>
  <span class="no">Jekyll</span><span class="o">::</span><span class="no">TableWrapper</span><span class="p">.</span><span class="nf">process</span><span class="p">(</span><span class="n">output</span><span class="p">)</span> <span class="k">if</span> <span class="n">document</span><span class="p">.</span><span class="nf">write?</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The version above worked well in case of HTML files… But it turned out that in Jekyll, <code class="language-plaintext highlighter-rouge">:documents</code> also means <strong>CSS</strong> files! So every time I’ve called <code class="language-plaintext highlighter-rouge">to_html</code> on the HTML-parsed version of my CSS files, Nokogiri prepended <code class="language-plaintext highlighter-rouge">&lt;!DOCTYPE html&gt;</code> to them.</p>

<p>At this point I’ve stopped for a minute. I was pretty sure that the <a href="https://github.com/keithmifsud/jekyll-target-blank">Jekyll Target Blank plugin</a> is also using Jekyll’s plugin system. I checked how it solves this situation – well, it also <a href="https://github.com/keithmifsud/jekyll-target-blank/blob/6d74c8baed/lib/jekyll-target-blank.rb#L46">checks the extension</a> of the generated file!</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
</pre></td><td class="rouge-code"><pre><span class="nb">require</span> <span class="s1">'nokogiri'</span><span class="p">;</span>

<span class="k">module</span> <span class="nn">Jekyll</span>
  <span class="k">class</span> <span class="nc">TableWrapper</span>
    <span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">process</span><span class="p">(</span><span class="n">document</span><span class="p">)</span>
      <span class="n">wrapper_tag</span> <span class="o">=</span> <span class="s1">'div'</span><span class="p">;</span>
      <span class="n">wrapper_css_class</span> <span class="o">=</span> <span class="s1">'vertical-scroll-wrapper'</span><span class="p">;</span>
      <span class="n">wrapper</span> <span class="o">=</span> <span class="s2">"&lt;</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2"> class=</span><span class="se">\"</span><span class="si">#{</span><span class="n">wrapper_css_class</span><span class="si">}</span><span class="se">\"</span><span class="s2">&gt;&lt;/</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2">&gt;"</span>

      <span class="n">xpath_selector</span> <span class="o">=</span> <span class="s1">'//body//table'</span>
      <span class="n">xpath_selector</span> <span class="o">&lt;&lt;</span> <span class="s2">"[not(parent::</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2">[contains(concat(' ', normalize-space(@class), ' '), ' </span><span class="si">#{</span><span class="n">wrapper_css_class</span><span class="si">}</span><span class="s2"> ')])]"</span>

      <span class="n">parsed_document</span> <span class="o">=</span> <span class="no">Nokogiri</span><span class="o">::</span><span class="no">HTML</span><span class="p">(</span><span class="n">document</span><span class="p">.</span><span class="nf">output</span><span class="p">)</span>
      <span class="n">parsed_document</span><span class="p">.</span><span class="nf">search</span><span class="p">(</span><span class="n">xpath_selector</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">table_node</span><span class="o">|</span>
        <span class="n">table_node</span><span class="p">.</span><span class="nf">wrap</span><span class="p">(</span><span class="n">wrapper</span><span class="p">)</span>
      <span class="k">end</span>

      <span class="n">document</span><span class="p">.</span><span class="nf">output</span> <span class="o">=</span> <span class="n">parsed_document</span><span class="p">.</span><span class="nf">to_html</span>
    <span class="k">end</span>
  <span class="k">end</span>
<span class="k">end</span>

<span class="no">Jekyll</span><span class="o">::</span><span class="no">Hooks</span><span class="p">.</span><span class="nf">register</span> <span class="p">[</span><span class="ss">:pages</span><span class="p">,</span> <span class="ss">:documents</span><span class="p">],</span> <span class="ss">:post_render</span> <span class="k">do</span> <span class="o">|</span><span class="n">output</span><span class="o">|</span>
  <span class="no">Jekyll</span><span class="o">::</span><span class="no">TableWrapper</span><span class="p">.</span><span class="nf">process</span><span class="p">(</span><span class="n">output</span><span class="p">)</span> <span class="k">if</span> <span class="n">document</span><span class="p">.</span><span class="nf">write?</span> <span class="n">and</span> <span class="n">document</span><span class="p">.</span><span class="nf">output_ext</span> <span class="o">==</span> <span class="s1">'.html'</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The plugin above is “almost” perfect. The missing pieces were:</p>

<ol>
  <li>Early return for documents which do not contain <code class="language-plaintext highlighter-rouge">&lt;table&gt;</code> elements at all.</li>
  <li>Exclude those tables which are added by the default Kramdown code highlighter when it is configured to show code line numbers.</li>
</ol>

<p>This is the “perfect” plugin file:</p>

<div data-file="./_plugins/table-wrapper.rb" class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
</pre></td><td class="rouge-code"><pre><span class="nb">require</span> <span class="s1">'nokogiri'</span><span class="p">;</span>

<span class="k">module</span> <span class="nn">Jekyll</span>
  <span class="k">class</span> <span class="nc">TableWrapper</span>
    <span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">process</span><span class="p">(</span><span class="n">document</span><span class="p">)</span>
      <span class="c1"># Early return if no tables are present.</span>
      <span class="k">if</span> <span class="n">document</span><span class="p">.</span><span class="nf">output</span><span class="p">.</span><span class="nf">index</span><span class="p">(</span><span class="sr">/&lt;table\b/</span><span class="p">).</span><span class="nf">nil?</span>
        <span class="k">return</span>
      <span class="k">end</span>

      <span class="n">wrapper_tag</span> <span class="o">=</span> <span class="s1">'div'</span><span class="p">;</span>
      <span class="n">wrapper_css_class</span> <span class="o">=</span> <span class="s1">'vertical-scroll-wrapper'</span><span class="p">;</span>
      <span class="n">wrapper</span> <span class="o">=</span> <span class="s2">"&lt;</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2"> class=</span><span class="se">\"</span><span class="si">#{</span><span class="n">wrapper_css_class</span><span class="si">}</span><span class="se">\"</span><span class="s2">&gt;&lt;/</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2">&gt;"</span>

      <span class="n">xpath_selector</span> <span class="o">=</span> <span class="s1">'//body//table'</span>
      <span class="n">xpath_selector</span> <span class="o">&lt;&lt;</span> <span class="s2">"[not(parent::</span><span class="si">#{</span><span class="n">wrapper_tag</span><span class="si">}</span><span class="s2">[contains(concat(' ', normalize-space(@class), ' '), ' </span><span class="si">#{</span><span class="n">wrapper_css_class</span><span class="si">}</span><span class="s2"> ')])]"</span>
      <span class="c1"># Exclude tables added to syntax highlighted code blocks.</span>
      <span class="n">xpath_selector</span> <span class="o">&lt;&lt;</span> <span class="s1">'[not(ancestor::pre)][not(ancestor::code)]'</span>

      <span class="n">parsed_document</span> <span class="o">=</span> <span class="no">Nokogiri</span><span class="o">::</span><span class="no">HTML</span><span class="p">(</span><span class="n">document</span><span class="p">.</span><span class="nf">output</span><span class="p">)</span>
      <span class="n">parsed_document</span><span class="p">.</span><span class="nf">search</span><span class="p">(</span><span class="n">xpath_selector</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">table_node</span><span class="o">|</span>
        <span class="n">table_node</span><span class="p">.</span><span class="nf">wrap</span><span class="p">(</span><span class="n">wrapper</span><span class="p">)</span>
      <span class="k">end</span>

      <span class="n">document</span><span class="p">.</span><span class="nf">output</span> <span class="o">=</span> <span class="n">parsed_document</span><span class="p">.</span><span class="nf">to_html</span>
    <span class="k">end</span>
  <span class="k">end</span>
<span class="k">end</span>

<span class="no">Jekyll</span><span class="o">::</span><span class="no">Hooks</span><span class="p">.</span><span class="nf">register</span> <span class="p">[</span><span class="ss">:pages</span><span class="p">,</span> <span class="ss">:documents</span><span class="p">],</span> <span class="ss">:post_render</span> <span class="k">do</span> <span class="o">|</span><span class="n">document</span><span class="o">|</span>
  <span class="no">Jekyll</span><span class="o">::</span><span class="no">TableWrapper</span><span class="p">.</span><span class="nf">process</span><span class="p">(</span><span class="n">document</span><span class="p">)</span> <span class="k">if</span> <span class="n">document</span><span class="p">.</span><span class="nf">write?</span> <span class="n">and</span> <span class="n">document</span><span class="p">.</span><span class="nf">output_ext</span> <span class="o">==</span> <span class="s1">'.html'</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>The CSS which makes my tables vertically scrollable is pretty obvious:</p>

<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre><span class="nc">.vertical-scroll-wrapper</span> <span class="p">{</span>
  <span class="nl">overflow</span><span class="p">:</span> <span class="nb">hidden</span><span class="p">;</span>
  <span class="nl">overflow-x</span><span class="p">:</span> <span class="nb">auto</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>And finally, the end result:</p>

<p class="figure"><img src="/files/image/cover-table-wrapper.jpg" alt="A photo about a mobile screen showing a table with a horizontal scroll bar on the bottom." />
The end result of the plugin. Note the light vertical scroll bar on the bottom of the table.</p>

<p>Happy coding!</p>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Jekyll" /><category term="Jekyll Plugin System" /><category term="Jekyll Hooks" /><category term="Ruby" /><summary type="html"><![CDATA[How to wrap tables used in Jekyll pages into a div automatically and prevent horizontal overflow -- with a custom Jekyll plugin!]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/files/image/cover-table-wrapper.jpg" /><media:content medium="image" url="https://huzooka.github.io/files/image/cover-table-wrapper.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Conditional VirtualHost in Apache</title><link href="https://huzooka.github.io/development/2022/07/05/apache-conditional-vhost.html" rel="alternate" type="text/html" title="Conditional VirtualHost in Apache" /><published>2022-07-05T20:30:29+00:00</published><updated>2022-07-05T20:30:29+00:00</updated><id>https://huzooka.github.io/development/2022/07/05/apache-conditional-vhost</id><content type="html" xml:base="https://huzooka.github.io/development/2022/07/05/apache-conditional-vhost.html"><![CDATA[<p>I usually spin up a new project directory and PHPStorm project each time I have to improve or fix something in a Drupal contrib extension. Then after I finished the task (and especially if I’m running out of storage), I just simply delete these project directories, without cleaning up the corresponding virtual hosts in my local Apache configuration.</p>

<p>This obviously causes issues on <code class="language-plaintext highlighter-rouge">httpd</code> service [re]starts, but I have learned to tolerate that I have a couple of messages logged to my console. Like this one:</p>

<pre><code class="language-cli">AH00112: Warning: DocumentRoot [/Users/zoli/projects/foo/workroom/public_html] does not exist
</code></pre>

<p>Yes, this also means that my server log is <em>never ever</em> empty. I only delete the virtual hosts of missing projects if I need to debug Apache 🙃. But I try to be more aware of my laziness. From Apache 2.4.34, there is a directive that might help in such cases – <a href="https://httpd.apache.org/docs/2.4/mod/core.html#iffile">this is <code class="language-plaintext highlighter-rouge">&lt;IfFile&gt;</code></a>. If you are using it (I try to do so) then you won’t pollute your log unnecessarily.</p>

<p>Here is an example:</p>

<div class="language-apache highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
</pre></td><td class="rouge-code"><pre><span class="p">&lt;</span>IfFile<span class="sr"> /Users/zoli/projects/foo/dist/foo/public_html/index.php</span><span class="p">&gt;
</span>    <span class="p">&lt;</span><span class="nl">VirtualHost</span><span class="sr"> *:80</span><span class="p">&gt;
</span>        <span class="nc">DocumentRoot</span> "/Users/zoli/projects/foo/dist/foo/public_html"
        <span class="nc">ServerName</span> test.foo.localhost
    <span class="p">&lt;/</span><span class="nl">VirtualHost</span><span class="p">&gt;
</span>    <span class="p">&lt;</span><span class="nl">VirtualHost</span><span class="sr"> *:443</span><span class="p">&gt;
</span>        <span class="nc">DocumentRoot</span> "/Users/zoli/projects/foo/dist/foo/public_html"
        <span class="nc">ServerName</span> test.foo.localhost
        <span class="nc">SSLEngine</span> <span class="ss">on</span>
        <span class="nc">SSLCertificateFile</span> "/Users/zoli/projects/foo/ssl/test.foo.localhost.crt"
        <span class="nc">SSLCertificateKeyFile</span> "/Users/zoli/projects/foo/ssl/test.foo.localhost.key"
    <span class="p">&lt;/</span><span class="nl">VirtualHost</span><span class="p">&gt;
&lt;/</span>IfFile<span class="p">&gt;
</span></pre></td></tr></tbody></table></code></pre></div></div>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Apache" /><summary type="html"><![CDATA[How we can declare a virtual host in Apache 2's configuration only if artifact exists.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/simple.png" /><media:content medium="image" url="https://huzooka.github.io/simple.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Database Query Trick for SQL-based Migrate Source Plugins</title><link href="https://huzooka.github.io/development/2022/06/12/migrate-source-query-trick.html" rel="alternate" type="text/html" title="Database Query Trick for SQL-based Migrate Source Plugins" /><published>2022-06-12T04:37:37+00:00</published><updated>2022-06-12T04:37:37+00:00</updated><id>https://huzooka.github.io/development/2022/06/12/migrate-source-query-trick</id><content type="html" xml:base="https://huzooka.github.io/development/2022/06/12/migrate-source-query-trick.html"><![CDATA[<p class="lead">In my previous post about <a href="/development/2022/05/29/empty-properties-change-tracking.html">strange empty destination property handling</a> I mentioned <a href="https://www.drupal.org/u/yashrode">Yash Rode</a>’s brilliant <a href="https://drupal.org/i/3272705">YouTube field → media migration</a> feature. But I missed taking a note about a crucial trick we used there: how we managed to use a column alias as migration source item identifier.</p>

<h2 id="the-problem">The problem</h2>

<p>Let’s suppose there are multiple YouTube fields on the source Drupal 7 site! Each of them have different field names, so the field property column names are also different:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">Field name</th>
      <th style="text-align: left">Input property column</th>
      <th style="text-align: left">Video ID column</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_foo</code></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_foo_input</code></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_foo_video_id</code></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_bar</code></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_bar_input</code></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_bar_video_id</code></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_baz</code></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_baz_input</code></td>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">field_yt_baz_video_id</code></td>
    </tr>
  </tbody>
</table>

<p>In Drupal’s Migrate API, SQL-based source plugins <a href="https://git.drupalcode.org/project/drupal/-/blob/3aaef92fe5/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php#L379-L382">should return a database select query</a> (a <code class="language-plaintext highlighter-rouge">Drupal\Core\Database\Query\SelectInterface</code>). But how can we get all the data in a single database query? Well, we can’t. But we can use a separate query to get the list of the field names with the specified type:</p>

<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
</pre></td><td class="rouge-code"><pre><span class="k">protected</span> <span class="k">function</span> <span class="n">getYouTubeFieldNames</span><span class="p">():</span> <span class="kt">array</span> <span class="p">{</span>
  <span class="nv">$youtube_fields</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">select</span><span class="p">(</span><span class="s1">'field_config'</span><span class="p">)</span>
    <span class="o">-&gt;</span><span class="nf">fields</span><span class="p">(</span><span class="s1">'field_config'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'field_name'</span><span class="p">])</span>
    <span class="o">-&gt;</span><span class="nf">condition</span><span class="p">(</span><span class="s1">'type'</span><span class="p">,</span> <span class="s1">'youtube'</span><span class="p">)</span>
    <span class="o">-&gt;</span><span class="nf">condition</span><span class="p">(</span><span class="s1">'module'</span><span class="p">,</span> <span class="s1">'youtube'</span><span class="p">)</span>
    <span class="o">-&gt;</span><span class="nf">execute</span><span class="p">()</span>
    <span class="o">-&gt;</span><span class="nf">fetchAllKeyed</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>

  <span class="k">return</span> <span class="nb">array_values</span><span class="p">(</span><span class="nv">$youtube_fields</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>…Then use the list to build a <code class="language-plaintext highlighter-rouge">UNION</code> query (the <code class="language-plaintext highlighter-rouge">static::addUnionQuery()</code> is explained in this<sup id="fnref:union" role="doc-noteref"><a href="#fn:union" class="footnote" rel="footnote">1</a></sup> footnote):</p>

<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * {@inheritdoc}
 */</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">query</span><span class="p">()</span> <span class="p">{</span>
  <span class="nv">$union_query</span> <span class="o">=</span> <span class="kc">NULL</span><span class="p">;</span>
  <span class="k">foreach</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">getYouTubeFieldNames</span><span class="p">()</span> <span class="k">as</span> <span class="nv">$field_name</span><span class="p">)</span> <span class="p">{</span>
    <span class="nv">$subquery</span> <span class="o">=</span> <span class="nv">$this</span>
      <span class="o">-&gt;</span><span class="nf">select</span><span class="p">(</span><span class="s2">"field_data_</span><span class="nv">$field_name</span><span class="s2">"</span><span class="p">,</span> <span class="nv">$field_name</span><span class="p">)</span>
      <span class="o">-&gt;</span><span class="nf">fields</span><span class="p">(</span><span class="s2">"field_data_</span><span class="si">{</span><span class="nv">$field_name</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="p">[</span><span class="s2">"</span><span class="si">{</span><span class="nv">$field_name</span><span class="si">}</span><span class="s2">_input"</span><span class="p">]);</span>
    <span class="k">static</span><span class="o">::</span><span class="nf">addUnionQuery</span><span class="p">(</span><span class="nv">$union_query</span><span class="p">,</span> <span class="nv">$subquery</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="k">return</span> <span class="nv">$union_query</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Query string:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="rouge-code"><pre><span class="k">SELECT</span>
  <span class="nv">"field_data_field_yt_foo"</span><span class="p">.</span><span class="nv">"field_yt_foo_input"</span> <span class="k">AS</span> <span class="nv">"field_yt_foo_input"</span>
<span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_foo</span><span class="p">}</span> <span class="nv">"field_yt_foo"</span>
<span class="k">UNION</span>
<span class="k">SELECT</span>
  <span class="nv">"field_data_field_yt_bar"</span><span class="p">.</span><span class="nv">"field_yt_bar_input"</span> <span class="k">AS</span> <span class="nv">"field_yt_bar_input"</span>
<span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_bar</span><span class="p">}</span> <span class="nv">"field_yt_bar"</span>
<span class="k">UNION</span>
<span class="k">SELECT</span>
  <span class="nv">"field_data_field_yt_baz"</span><span class="p">.</span><span class="nv">"field_yt_baz_input"</span> <span class="k">AS</span> <span class="nv">"field_yt_baz_input"</span>
<span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_baz</span><span class="p">}</span> <span class="nv">"field_yt_baz"</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Unfortunately, we cannot really use this query as the iterator source of our migrate source: Migrate source plugins must specify which columns are identifying a single migration row. (These are called “<em>source IDs</em>”.) The identifier<sup id="fnref:idplural" role="doc-noteref"><a href="#fn:idplural" class="footnote" rel="footnote">2</a></sup> should be unique for every migrate source item. If they aren’t unique, every next item with the same ID will be ignored during the migration<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">3</a></sup>.</p>

<p>So instead of using the name of the <code class="language-plaintext highlighter-rouge">input</code> field property, let’s use an alias, and define the alias as source item identifier!</p>

<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * {@inheritdoc}
 */</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">query</span><span class="p">()</span> <span class="p">{</span>
  <span class="nv">$union_query</span> <span class="o">=</span> <span class="kc">NULL</span><span class="p">;</span>
  <span class="k">foreach</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">getYouTubeFieldNames</span><span class="p">()</span> <span class="k">as</span> <span class="nv">$field_name</span><span class="p">)</span> <span class="p">{</span>
    <span class="nv">$subquery</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">select</span><span class="p">(</span><span class="s2">"field_data_</span><span class="nv">$field_name</span><span class="s2">"</span><span class="p">,</span> <span class="nv">$field_name</span><span class="p">);</span>
    <span class="nv">$subquery</span><span class="o">-&gt;</span><span class="nf">addField</span><span class="p">(</span><span class="nv">$field_name</span><span class="p">,</span> <span class="s2">"</span><span class="si">{</span><span class="nv">$field_name</span><span class="si">}</span><span class="s2">_input"</span><span class="p">,</span> <span class="s1">'input'</span><span class="p">);</span>
    <span class="k">static</span><span class="o">::</span><span class="nf">addUnionQuery</span><span class="p">(</span><span class="nv">$union_query</span><span class="p">,</span> <span class="nv">$subquery</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="k">return</span> <span class="nv">$union_query</span><span class="p">;</span>
<span class="p">}</span>

<span class="cd">/**
 * {@inheritdoc}
 */</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getIds</span><span class="p">()</span> <span class="p">{</span>
  <span class="k">return</span> <span class="p">[</span><span class="s1">'input'</span> <span class="o">=&gt;</span> <span class="p">[</span><span class="s1">'type'</span> <span class="o">=&gt;</span> <span class="s1">'string'</span><span class="p">]];</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Query string:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="rouge-code"><pre><span class="k">SELECT</span>
  <span class="nv">"field_data_field_yt_foo"</span><span class="p">.</span><span class="nv">"field_yt_foo_input"</span> <span class="k">AS</span> <span class="nv">"input"</span>
<span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_foo</span><span class="p">}</span> <span class="nv">"field_yt_foo"</span>
<span class="k">UNION</span>
<span class="k">SELECT</span>
  <span class="nv">"field_data_field_yt_bar"</span><span class="p">.</span><span class="nv">"field_yt_bar_input"</span> <span class="k">AS</span> <span class="nv">"input"</span>
<span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_bar</span><span class="p">}</span> <span class="nv">"field_yt_bar"</span>
<span class="k">UNION</span>
<span class="k">SELECT</span>
  <span class="nv">"field_data_field_yt_baz"</span><span class="p">.</span><span class="nv">"field_yt_baz_input"</span> <span class="k">AS</span> <span class="nv">"input"</span>
<span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_baz</span><span class="p">}</span> <span class="nv">"field_yt_baz"</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>But this is still wrong! Although the query works as expected, but as soon as we try to use joinable source and ID map databases, we will have an ugly message in our migrate kernel test:</p>
<pre><code class="language-cli">Migration failed with source plugin exception: SQLSTATE[42703]: Undefined column: 7  ERROR:  column "input" does not exist
LINE 4: ...ublic.60254974m_map_d7_youtube_field "map" ON input = ma...
                                                             ^: 
SELECT "field_yt_foo"."field_yt_foo_input" AS "input", "map"."sourceid1" AS "migrate_map_sourceid1", "map"."source_row_status" AS "migrate_map_source_row_status"
FROM
"602549740field_data_field_yt_foo" "field_yt_foo"
LEFT OUTER JOIN .public.60254974m_map_d7_youtube_field "map" ON input = map.sourceid1
WHERE ("map"."sourceid1" IS NULL) OR ("map"."source_row_status" = :db_condition_placeholder_0) UNION SELECT "field_yt_bar"."field_yt_bar_input" AS "input"
FROM
"602549740field_data_field_yt_bar" "field_yt_bar" UNION SELECT "field_yt_baz"."field_yt_baz_input" AS "input"
FROM
"602549740field_data_field_yt_baz" "field_yt_baz"; Array
(
  [:db_condition_placeholder_0] =&lt;; 1
)
in /Users/zoli/projects/media_migration/zdev/public_html/core/lib/Drupal/Core/Database/ExceptionHandler.php line 79
</code></pre>

<p>In essence this happens because the source plugin tries to exclude the source records from the database query which are already migrated. Although the outer joined query SqlBase has added is unaware of that we have a UNION query, the exception totally makes sense: We really don’t have an “input” column – since it’s just an alias of the <code class="language-plaintext highlighter-rouge">field_yt_foo_input</code>, <code class="language-plaintext highlighter-rouge">field_yt_bar_input</code> and <code class="language-plaintext highlighter-rouge">field_yt_baz_input</code> columns.</p>

<h2 id="the-solution">The solution</h2>

<p>Just check out <a href="https://git.drupalcode.org/project/drupal/-/blob/3aaef92fe5/core/lib/Drupal/Core/Database/Connection.php#L1197-1215">the docblock of <code class="language-plaintext highlighter-rouge">Drupal\Core\Database\Connection::select()</code></a>, focusing on the type of the first method parameter<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup>! 🧐</p>

<blockquote>
  <p>@param <code class="language-plaintext highlighter-rouge">string</code>|<code class="language-plaintext highlighter-rouge">\Drupal\Core\Database\Query\SelectInterface</code> <code class="language-plaintext highlighter-rouge">$table</code></p>

  <p>The base table name or subquery for this query, used in the FROM clause. If a string, the table specified will also be used as the “base” table for query_alter hook implementations.</p>
</blockquote>

<p>Because of the name of the variable, we all<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">5</a></sup> think that <code class="language-plaintext highlighter-rouge">$table</code> must be a string – the name of the table – but it <strong>can be</strong> either <strong>a select statement</strong>! And this is the essence of our solution as well: <em>We wrap the “big” union query into a new query</em>, which then allows us to use any kind of alias as source item identifier – be it a column alias or alias of a complex SQL expression!</p>

<p>It’s that simple:</p>

<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * {@inheritdoc}
 */</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">query</span><span class="p">()</span> <span class="p">{</span>
  <span class="nv">$union_query</span> <span class="o">=</span> <span class="kc">NULL</span><span class="p">;</span>
  <span class="k">foreach</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">getYoutubeFieldNames</span><span class="p">()</span> <span class="k">as</span> <span class="nv">$field_name</span><span class="p">)</span> <span class="p">{</span>
    <span class="nv">$subquery</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">select</span><span class="p">(</span><span class="s2">"field_data_</span><span class="nv">$field_name</span><span class="s2">"</span><span class="p">,</span> <span class="nv">$field_name</span><span class="p">);</span>
    <span class="nv">$subquery</span><span class="o">-&gt;</span><span class="nf">addField</span><span class="p">(</span><span class="nv">$field_name</span><span class="p">,</span> <span class="s2">"</span><span class="si">{</span><span class="nv">$field_name</span><span class="si">}</span><span class="s2">_input"</span><span class="p">,</span> <span class="s1">'input'</span><span class="p">);</span>
    <span class="k">static</span><span class="o">::</span><span class="nf">addUnionQuery</span><span class="p">(</span><span class="nv">$union_query</span><span class="p">,</span> <span class="nv">$subquery</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="nv">$wrapper_query</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">select</span><span class="p">(</span><span class="nv">$union_query</span><span class="p">,</span> <span class="s1">'all_yt'</span><span class="p">)</span><span class="o">-&gt;</span><span class="nf">fields</span><span class="p">(</span><span class="s1">'all_yt'</span><span class="p">);</span>
  <span class="nv">$wrapper_query</span><span class="o">-&gt;</span><span class="nf">orderBy</span><span class="p">(</span><span class="s1">'all_yt.input'</span><span class="p">);</span>
  <span class="k">return</span> <span class="nv">$wrapper_query</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>This is the query string of the Drupal select above:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
</pre></td><td class="rouge-code"><pre><span class="k">SELECT</span> 
  <span class="nv">"all_yt"</span><span class="p">.</span><span class="o">*</span>
<span class="k">FROM</span> <span class="p">(</span>
  <span class="k">SELECT</span>
    <span class="nv">"field_data_field_yt_foo"</span><span class="p">.</span><span class="nv">"field_yt_foo_input"</span> <span class="k">AS</span> <span class="nv">"input"</span>
  <span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_foo</span><span class="p">}</span> <span class="nv">"field_yt_foo"</span>
  <span class="k">UNION</span>
  <span class="k">SELECT</span>
    <span class="nv">"field_data_field_yt_bar"</span><span class="p">.</span><span class="nv">"field_yt_bar_input"</span> <span class="k">AS</span> <span class="nv">"input"</span>
  <span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_bar</span><span class="p">}</span> <span class="nv">"field_yt_bar"</span>
  <span class="k">UNION</span>
  <span class="k">SELECT</span>
    <span class="nv">"field_data_field_yt_baz"</span><span class="p">.</span><span class="nv">"field_yt_baz_input"</span> <span class="k">AS</span> <span class="nv">"input"</span>
  <span class="k">FROM</span> <span class="p">{</span><span class="n">field_data_field_yt_baz</span><span class="p">}</span> <span class="nv">"field_yt_baz"</span>
<span class="p">)</span> <span class="nv">"all_yt"</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="nv">"all_yt"</span><span class="p">.</span><span class="nv">"input"</span> <span class="k">ASC</span> <span class="n">NULLS</span> <span class="k">FIRST</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>Have nice <em>database queries</em>! 🙂</p>

<hr />

<p><em>Footnotes</em>:</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:union" role="doc-endnote">
      <p>This <code class="language-plaintext highlighter-rouge">::addUnionQuery()</code> is a very simple helper function:</p>

      <div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre><span class="k">protected</span> <span class="k">static</span> <span class="k">function</span> <span class="n">addUnionQuery</span><span class="p">(</span><span class="o">&amp;</span><span class="nv">$union_destination</span><span class="p">,</span> <span class="nc">SelectInterface</span> <span class="nv">$query</span><span class="p">)</span><span class="o">:</span> <span class="n">void</span> <span class="p">{</span>
  <span class="k">if</span> <span class="p">(</span><span class="nv">$union_destination</span> <span class="k">instanceof</span> <span class="nc">SelectInterface</span><span class="p">)</span> <span class="p">{</span>
    <span class="nv">$union_destination</span><span class="o">-&gt;</span><span class="nf">union</span><span class="p">(</span><span class="nv">$query</span><span class="p">);</span>
    <span class="k">return</span><span class="p">;</span>
  <span class="p">}</span>
  <span class="nv">$union_destination</span> <span class="o">=</span> <span class="k">clone</span> <span class="nv">$query</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div>      </div>

      <p>If the destination query is empty, then it just sets <code class="language-plaintext highlighter-rouge">$destination</code> to the clone of the source query. But if it is already a <code class="language-plaintext highlighter-rouge">SelectInterface</code>, then it ‘unions’ the two queries. <a href="#fnref:union" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:idplural" role="doc-endnote">
      <p>We can define multiple identifiers too. <a href="#fnref:idplural" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2" role="doc-endnote">
      <p>This is a very handy feature! It is actively used in Drupal core migrate source plugins too: This is why <code class="language-plaintext highlighter-rouge">d7_field</code> and <code class="language-plaintext highlighter-rouge">d7_field_instance</code> can use the same database query to get their source data. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:4" role="doc-endnote">
      <p>This option is available in <code class="language-plaintext highlighter-rouge">::join()</code>, <code class="language-plaintext highlighter-rouge">::isNull()</code> etc too. As well as in <code class="language-plaintext highlighter-rouge">::union()</code>! 😉 <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3" role="doc-endnote">
      <p>Or at least I’ve only known about this feature for two years (since 2020, when I started working on migrations). <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Drupal" /><category term="Database API" /><category term="Migration" /><summary type="html"><![CDATA[How we can use DB column or DB expression aliases as migration source item identifiers.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/simple.png" /><media:content medium="image" url="https://huzooka.github.io/simple.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Front End Regression Testing with Nightwatch.js and Nightwatch VRT</title><link href="https://huzooka.github.io/development/2022/06/05/frontend-regression.html" rel="alternate" type="text/html" title="Front End Regression Testing with Nightwatch.js and Nightwatch VRT" /><published>2022-06-05T05:02:19+00:00</published><updated>2022-06-05T05:02:19+00:00</updated><id>https://huzooka.github.io/development/2022/06/05/frontend-regression</id><content type="html" xml:base="https://huzooka.github.io/development/2022/06/05/frontend-regression.html"><![CDATA[<p class="lead">If you have to maintain a huge or quite complex CSS codebase, it is a big challenge to make sure you don’t break certain parts of the site when you try to fix an issue or just simply add a new component.</p>

<p>Do you want to keep track of how your code works on the current browser versions? Or you just want to do a refactor without causing any accidental regressions? Either way, in 2022, we have well-functioning, widely used visual regression testing tools at our disposal. In this post, I want to show you how we can use <a href="https://nightwatchjs.org/">Nightwatch.js</a> and <a href="https://github.com/Crunch-io/nightwatch-vrt">Nightwatch VRT</a>.</p>

<h2 id="my-motivation">My motivation</h2>

<p>Since I am interested in improving and optimizing the performance of websites in particular, I try to reduce the size of the CSS asset(s) as much as possible. In many cases, serious results can be achieved only by rearranging the order of the selectors. I thought since my CSS codebase is small enough, I can easily refactor it without causing any issues. And after a couple weeks, I noticed that paragraphs which contain a <code class="language-plaintext highlighter-rouge">&lt;sup&gt;</code> tag have very weird vertical rhythm:</p>

<p class="figure"><img src="/files/image/Visual-regression-motivation.png" alt="Mixed line height in a HTML paragraph containing a &lt;sup&gt; tag. The light purple “shadow” shows where words should be." />
Mixed line height in a HTML paragraph containing a <code class="language-plaintext highlighter-rouge">&lt;sup&gt;</code> tag. The light purple “shadow” shows where words should be.</p>

<p>Well, this was the time I decided I will set up visual regression testing.</p>

<h2 id="basics-background">Basics, background</h2>

<p>You may remember that <a href="/development/2018/12/06/using-nightwatch-for-screenshots.html">I used Nightwatch.js tests about three and a half years ago</a> in a quite similar situation. Well, Nightwatch VRT uses very similar <a href="https://nodejs.org/">Node.js</a> modules and tools under the hood to perform visual regression tests<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>

<p>Essentially, Nightwatch VRT provides a Nightwatch.js command and a Nightwatch.js assertion. The way it works is <a href="https://github.com/Crunch-io/nightwatch-vrt/tree/v0.2.9#usage">very developer friendly</a>: In your Nightwatch.js test, you can use the <a href="https://github.com/Crunch-io/nightwatch-vrt/blob/v0.2.9/assertions/screenshotIdenticalToBaseline.js"><code class="language-plaintext highlighter-rouge">screenshotIdenticalToBaseline()</code> assertion</a> to compare the actual visual representation of an HTML element to a base image. If the base image does not exist – because you are executing the test for the first time, or you just added a new <code class="language-plaintext highlighter-rouge">screenshotIdenticalToBaseline()</code> assertion – then this assertion will generate the base image at the first execution.</p>

<h2 id="setup-nightwatchjs-and-nightwatch-vrt">Setup Nightwatch.js and Nightwatch VRT</h2>

<ol>
  <li>
    <p>Initialize the node.js part of your project (only necessary if it does not exist yet):</p>

    <p><code class="language-plaintext highlighter-rouge">yarn init .</code></p>
  </li>
  <li>
    <p>Add Nightwatch.js and Nightwatch VRT as development dependencies:</p>

    <p><code class="language-plaintext highlighter-rouge">yarn add --dev nightwatch-vrt nightwatch@1</code></p>

    <p>At the time I am writing this post, we have to use Nightwatch.js with <code class="language-plaintext highlighter-rouge">@1</code> version constraint, because Nightwatch VRT needs this version.</p>
  </li>
  <li>
    <p>Add a <code class="language-plaintext highlighter-rouge">nightwatch.json</code> file to the root of the project then add the assertions and commands folder of Nightwatch VRT to this configuration. I used to copy this file from <code class="language-plaintext highlighter-rouge">node_modules/nightwatch/examples</code> then slightly customize it to match my needs. This is what I use:</p>

    <div data-file="./nightwatch.json" class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
</pre></td><td class="rouge-code"><pre><span class="p">{</span><span class="w">
  </span><span class="nl">"src_folders"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"./tests/Nightwatch/tests"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"output_folder"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"./tests/Nightwatch/reports"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"custom_commands_path"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
    </span><span class="s2">"./tests/Nightwatch/Commands"</span><span class="p">,</span><span class="w">
    </span><span class="s2">"./node_modules/nightwatch-vrt/commands"</span><span class="w">
  </span><span class="p">],</span><span class="w">
  </span><span class="nl">"custom_assertions_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"./node_modules/nightwatch-vrt/assertions"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"globals_path"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"./tests/Nightwatch/globals.js"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"test_workers"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
  </span><span class="nl">"webdriver"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"server_path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/usr/local/bin/chromedriver"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"start_process"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
  </span><span class="p">},</span><span class="w">
  </span><span class="nl">"test_settings"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"screenshots"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"enabled"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
      </span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"./tests/Nightwatch/reports/screenshots"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"on_failure"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
      </span><span class="nl">"on_error"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
    </span><span class="p">},</span><span class="w">
    </span><span class="nl">"default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"disable_colors"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
      </span><span class="nl">"screenshots"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nl">"enabled"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
        </span><span class="nl">"path"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="w">
      </span><span class="p">},</span><span class="w">
      </span><span class="nl">"request_timeout_options"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nl">"timeout"</span><span class="p">:</span><span class="w"> </span><span class="mi">1000</span><span class="p">,</span><span class="w">
        </span><span class="nl">"retry_attempts"</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w">
      </span><span class="p">},</span><span class="w">
      </span><span class="nl">"desiredCapabilities"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{}</span><span class="w">
    </span><span class="p">},</span><span class="w">
    </span><span class="nl">"mobile"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{}</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
  <li>
    <p>(Optional) Nightwatch VRT searches for its configuration either at Nightwatch <code class="language-plaintext highlighter-rouge">globals.test_settings.default.visual_regression_settings</code> or at Nightwatch <code class="language-plaintext highlighter-rouge">globals.visual_regression_settings</code>. We have to create the file we specified at the <code class="language-plaintext highlighter-rouge">globals_path</code> configuration (line 9). We can define other pre- and post-test hooks here, or store the “default” landing page url like I do. But if we don’t specify any <code class="language-plaintext highlighter-rouge">visual_regression_settings</code> config anywhere, then Nightwatch VRT will use its default configuration. If you think you don’t need this “globals” file, you can delete it, but don’t forget to remove the <code class="language-plaintext highlighter-rouge">globals_path</code> from your <code class="language-plaintext highlighter-rouge">nightwatch.js</code> JSON.</p>

    <div data-file="./tests/Nightwatch/globals.js" class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="rouge-code"><pre><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span>
  <span class="na">landingPageUrl</span><span class="p">:</span> <span class="dl">'</span><span class="s1">http://127.0.0.1:4000</span><span class="dl">'</span><span class="p">,</span>
  <span class="na">beforeEach</span><span class="p">:</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">browser</span><span class="p">,</span> <span class="nx">done</span><span class="p">)</span> <span class="p">{</span>
    <span class="nx">browser</span><span class="p">.</span><span class="nx">resizeWindow</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">600</span><span class="p">,</span> <span class="nx">done</span><span class="p">);</span>
  <span class="p">},</span>
  <span class="na">visual_regression_settings</span><span class="p">:</span> <span class="p">{</span>
    <span class="dl">"</span><span class="s2">baseline_screenshots_path</span><span class="dl">"</span><span class="p">:</span> <span class="dl">'</span><span class="s1">tests/Nightwatch/vrt</span><span class="dl">'</span><span class="p">,</span>
    <span class="dl">"</span><span class="s2">threshold</span><span class="dl">"</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
    <span class="dl">"</span><span class="s2">prompt</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
    <span class="dl">"</span><span class="s2">always_save_diff_screenshot</span><span class="dl">"</span><span class="p">:</span> <span class="kc">false</span>
  <span class="p">}</span>
<span class="p">};</span>
</pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
  <li>
    <p>(Optional) Since I can’t be sure that the path to the chromedriver binary is always <code class="language-plaintext highlighter-rouge">/usr/local/bin/chromedriver</code> (see line 12 above), I also have a <code class="language-plaintext highlighter-rouge">nightwatch.conf.js</code> file in my root which changes this path to the output of the <code class="language-plaintext highlighter-rouge">which chromedriver</code> command:</p>

    <div data-file="./nightwatch.conf.js" class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre><span class="kd">const</span> <span class="nx">which</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">which</span><span class="dl">'</span><span class="p">);</span>
   
<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">(</span><span class="nx">settings</span> <span class="o">=&gt;</span> <span class="p">{</span>
  <span class="kd">const</span> <span class="nx">chromedriverPath</span> <span class="o">=</span> <span class="nx">which</span><span class="p">.</span><span class="nx">sync</span><span class="p">(</span><span class="dl">'</span><span class="s1">chromedriver</span><span class="dl">'</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="nx">chromedriverPath</span><span class="p">)</span> <span class="p">{</span>
    <span class="nx">settings</span><span class="p">.</span><span class="nx">webdriver</span><span class="p">.</span><span class="nx">server_path</span> <span class="o">=</span> <span class="nx">chromedriverPath</span><span class="p">;</span>
  <span class="p">}</span>
  <span class="k">return</span> <span class="nx">settings</span><span class="p">;</span>
<span class="p">})(</span><span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./nightwatch.json</span><span class="dl">'</span><span class="p">));</span>
</pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
</ol>

<p>In the config file above, inside <code class="language-plaintext highlighter-rouge">test_settings</code>, we have a <code class="language-plaintext highlighter-rouge">default</code> (line 22) and a <code class="language-plaintext highlighter-rouge">mobile</code> (line 34) key. These are the environments our tests can be executed on. The default environment is a special one: on one hand, if the environment isn’t specified, Nightwatch.js will use this environment. On the other hand, every other environment we specify (now it is <code class="language-plaintext highlighter-rouge">mobile</code>) will inherit the configuration of the default one.</p>

<h2 id="set-up-test-browsers">Set up test browsers</h2>

<p>I think using <code class="language-plaintext highlighter-rouge">chromedriver</code> as the (default) browser for executing Nightwatch.js tests on is the most obvious choice. This is what I know the best, for example I can configure it to act as a mobile browser (with touch-capable screen). So I chose Chrome and <code class="language-plaintext highlighter-rouge">chromedriver</code>. I added these configurations to my <code class="language-plaintext highlighter-rouge">nightwatch.json</code>:</p>

<div data-file="./nightwatch.json" class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
</pre></td><td class="rouge-code"><pre><span class="p">{</span><span class="w">
  </span><span class="nl">"src_folders"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"./tests/Nightwatch/tests"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"output_folder"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"./tests/Nightwatch/reports"</span><span class="p">,</span><span class="w">
  </span><span class="err">…</span><span class="w">
  </span><span class="nl">"test_settings"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="err">…</span><span class="w">
    </span><span class="nl">"default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="err">…</span><span class="w">
      </span><span class="nl">"desiredCapabilities"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nl">"browserName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"chrome"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"loggingPrefs"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nl">"driver"</span><span class="p">:</span><span class="w"> </span><span class="s2">"INFO"</span><span class="p">,</span><span class="w"> </span><span class="nl">"server"</span><span class="p">:</span><span class="w"> </span><span class="s2">"OFF"</span><span class="p">,</span><span class="w"> </span><span class="nl">"browser"</span><span class="p">:</span><span class="w"> </span><span class="s2">"INFO"</span><span class="p">},</span><span class="w">
        </span><span class="nl">"chromeOptions"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
          </span><span class="nl">"args"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
            </span><span class="s2">"--headless"</span><span class="p">,</span><span class="w">
            </span><span class="s2">"--force-device-scale-factor=1"</span><span class="p">,</span><span class="w">
            </span><span class="s2">"--hide-scrollbars"</span><span class="p">,</span><span class="w">
            </span><span class="s2">"--font-render-hinting=full"</span><span class="p">,</span><span class="w">
            </span><span class="s2">"--enable-font-antialiasing"</span><span class="p">,</span><span class="w">
            </span><span class="s2">"--force-color-profile=srgb"</span><span class="w">
          </span><span class="p">]</span><span class="w">
        </span><span class="p">}</span><span class="w">
      </span><span class="p">}</span><span class="w">
    </span><span class="p">},</span><span class="w">
    </span><span class="nl">"mobile"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"desiredCapabilities"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nl">"chromeOptions"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
          </span><span class="nl">"mobileEmulation"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
            </span><span class="nl">"deviceMetrics"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"width"</span><span class="p">:</span><span class="w"> </span><span class="mi">360</span><span class="p">,</span><span class="w"> </span><span class="nl">"height"</span><span class="p">:</span><span class="w"> </span><span class="mi">640</span><span class="p">,</span><span class="w"> </span><span class="nl">"pixelRatio"</span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="w"> </span><span class="p">},</span><span class="w">
            </span><span class="nl">"userAgent"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19"</span><span class="w">
          </span><span class="p">}</span><span class="w">
        </span><span class="p">}</span><span class="w">
      </span><span class="p">}</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div></div>

<h2 id="writing-the-first-visual-regression-test">Writing the first visual regression test</h2>

<p>In the configuration JSON, the <code class="language-plaintext highlighter-rouge">"src_folders" : "./tests/Nightwatch/tests"</code> line declares that our Nightwatch.js tests should be in that directory. Of course, we can execute tests which aren’t stored in this folder, but if we just run <code class="language-plaintext highlighter-rouge">./node_modules/.bin/nightwatch</code>, only the tests in <code class="language-plaintext highlighter-rouge">./tests/Nightwatch/tests</code> are discovered.</p>

<p>Make sure your chromedriver is compatible with the Chrome version you use, then create a simple test in the <code class="language-plaintext highlighter-rouge">./tests/Nightwatch/tests</code> folder:</p>

<div data-file="./tests/Nightwatch/tests/huzooka.js" class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span>
  <span class="dl">'</span><span class="s1">HuZooka</span><span class="dl">'</span><span class="p">:</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">client</span><span class="p">)</span> <span class="p">{</span>
    <span class="nx">client</span>
      <span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="dl">'</span><span class="s1">https://huzooka.github.io/</span><span class="dl">'</span><span class="p">)</span>
      <span class="p">.</span><span class="nx">waitForElementVisible</span><span class="p">(</span><span class="dl">'</span><span class="s1">css</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">main .intro</span><span class="dl">'</span><span class="p">)</span>
      <span class="p">.</span><span class="nx">assert</span><span class="p">.</span><span class="nx">screenshotIdenticalToBaseline</span><span class="p">(</span><span class="dl">'</span><span class="s1">main .intro</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">HuZooka intro</span><span class="dl">'</span><span class="p">)</span>
      <span class="p">.</span><span class="nx">end</span><span class="p">();</span>
  <span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h3 id="how-it-works">How it works</h3>

<p>Nightwatch.js tests can be performed by executing this command on CLI:</p>
<pre><code class="language-cli">./node_modules/.bin/nightwatch
</code></pre>

<p>For the first time, Nightwatch VRT will save a base image in the configured <code class="language-plaintext highlighter-rouge">./tests/Nightwatch/vrt</code> folder, in a subdirectory that’s name is the same as the test name (by default), and the base image file’s name will be <code class="language-plaintext highlighter-rouge">Huzooka intro.png</code>.</p>

<p class="figure"><img src="/files/image/Visual-regression-base.png" alt="A test base image" />
The base image saved into <code class="language-plaintext highlighter-rouge">./tests/Nightwatch/vrt/huzooka/Huzooka intro.png</code></p>

<p>If the base image exists, each subsequent test run will compare the base image with the actually taken image, and save a diff image file if the comparison fails: if I change my shortened name “Zoltán Horváth” to my full name, “Zoltán Attila Horváth”, then the test will fail:</p>

<pre><code class="language-cli">FAILED: 1 assertions failed and  15 passed (5.626s)
_________________________________________________

TEST FAILURE:  1 assertions failed, 15 passed (6.693s)

 ✖ huzooka
 – HuZooka (5.626s)
   Visual regression test results for element &lt;.page-content .intro&gt; in 5000ms - expected "true" but got: "/Users/zoli/projects/github-blog/vrt/diff/huzooka/HuZooka intro.diff.png" (5021ms)
       at Object.HuZooka (/Users/zoli/projects/github-blog/tests/Nightwatch/tests/huzooka.js:15:15)
       at processTicksAndRejections (internal/process/task_queues.js:97:5)

error Command failed with exit code 5.
</code></pre>

<p>With the default Nightwatch VRT configuration, a failed visual regression test also saves a “latest” and a “diff” image. These will be very helpful if the test fails on a remote environment (like a CI server), because then we can see the differences visually too.</p>

<p class="figure"><img src="/files/image/Visual-regression-diff.png" alt="A Nightwatch VRT diff image" />
The diff image saved into <code class="language-plaintext highlighter-rouge">./vrt/diff/huzooka/Huzooka intro.diff.png</code></p>

<h3 id="my-testing-approach">My testing approach</h3>

<p>My visual regression test cases are very basic:</p>

<ul>
  <li>I am testing the header and the beginning of a “lorem ipsum” post.</li>
  <li>Then the footer and the end of the same “lorem ipsum” post.</li>
  <li>I have an “element” test which covers standard post elements, such as:
    <ul>
      <li>an HTML <code class="language-plaintext highlighter-rouge">&lt;p&gt;</code> tag containing sort sentences with some emphasized, bold, striked words</li>
      <li>HTML blockquotes</li>
      <li>Ordered and unordered lists</li>
      <li>Inline <code class="language-plaintext highlighter-rouge">&lt;code&gt;</code> tags and code blocks etc.</li>
    </ul>
  </li>
  <li>And I also have a test which checks the spacing between elements frequently used in my posts:
    <ul>
      <li>Spacing between <code class="language-plaintext highlighter-rouge">&lt;p&gt;</code> and <code class="language-plaintext highlighter-rouge">&lt;p&gt;</code> tags,</li>
      <li>Spacing between <code class="language-plaintext highlighter-rouge">&lt;p&gt;</code> tags and code blocks etc.</li>
    </ul>
  </li>
</ul>

<p>I have some posts and pages for these tests I don’t delete before I create the build artifact that is published - so if you’re curious and lucky<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> at the same time, you can check them if you navigate to the public repository of this site and look for HTML files that’s name starts with <code class="language-plaintext highlighter-rouge">VRT-</code>.</p>

<h2 id="profit">Profit</h2>

<p>As you can see, we can get an evaluable amount of test coverage by a small amount of code very quickly. Although the solution I described here only uses <code class="language-plaintext highlighter-rouge">chromedriver</code>, it could be a quite solid base if you want to extend it with cross-browser testing. And if you think so, you also can improve the component tests by using <a href="https://storybook.js.org/">Storybook</a>.</p>

<hr />

<p><em>Footnotes</em>:</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>It uses <a href="https://github.com/oliver-moran/jimp">Jimp</a> for capturing the elements based on the provided CSS selector. However, unlike me, they also worked on neatly tidying up the code and documenting it. Many thanks! <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2" role="doc-endnote">
      <p>I used to change my mind sometimes, so it can happen that at the time you read this post the visual regression test pages aren’t available anymore. I’m sorry about that! <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Visual Regression" /><category term="Nightwatch VRT" /><category term="Nightwatch.js" /><category term="Testing" /><summary type="html"><![CDATA[Visual regression tests for your CSS codebase.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/files/image/Visual-regression-motivation.png" /><media:content medium="image" url="https://huzooka.github.io/files/image/Visual-regression-motivation.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Strange Empty Destination Property Handling in Drupal’s Migration API</title><link href="https://huzooka.github.io/development/2022/05/29/empty-properties-change-tracking.html" rel="alternate" type="text/html" title="Strange Empty Destination Property Handling in Drupal’s Migration API" /><published>2022-05-29T12:34:32+00:00</published><updated>2022-05-29T12:34:32+00:00</updated><id>https://huzooka.github.io/development/2022/05/29/empty-properties-change-tracking</id><content type="html" xml:base="https://huzooka.github.io/development/2022/05/29/empty-properties-change-tracking.html"><![CDATA[<p class="lead">One of my colleagues, <a href="https://www.drupal.org/u/yashrode">Yash Rode</a> developed a very nice improvement for <a href="https://drupal.org/project/media_migration">Media Migration</a> about one and half months ago: he added a <a href="https://drupal.org/i/3272705">YouTube field → media migration</a> to the module. After I committed it, the question immediately arose in my mind: what will happen during incremental migrations? I definitely cannot migrate file IDs to media entity IDs anymore!</p>

<p>So I started getting rid of the <code class="language-plaintext highlighter-rouge">mid</code> destination properties in the media entity migrations <code class="language-plaintext highlighter-rouge">d7_file_entity</code> and <code class="language-plaintext highlighter-rouge">d7_file_plain</code> and also update all migrate process plugins which assumed that the source file ID will be the ID of the migrated media entity, like the plugins which are processing text fields<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>

<p>The task wasn’t that difficult if we didn’t count with Migrate Upgrade… Well, that was what I thought before I ran into a core bug (I think it is a bug).</p>

<h2 id="the-situation">The situation</h2>

<p>The first thing I did was removing the <code class="language-plaintext highlighter-rouge">mid: fid</code> process pipelines from <code class="language-plaintext highlighter-rouge">d7_file_entity</code> and <code class="language-plaintext highlighter-rouge">d7_file_plain</code>. Then I took a look at the process plugins which are parsing text fields<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> and I updated them accordingly. Latter was required because I couldn’t assume anymore that the destination media entity’s ID is the same as the source file ID – by the way, I already knew that it was a wrong assumption… See <a href="https://www.drupal.org/project/media_migration/issues/3214791">this (#3214791)</a> and <a href="https://www.drupal.org/project/media_migration/issues/3267064">this (#3267064)</a> issue.</p>

<p>Pretty slowly I refactored all the components I knew to handle the new situation properly, and then I ran into a whole unexpected <code class="language-plaintext highlighter-rouge">EntityStorageException</code> while running the refresh test:</p>

<pre><code class="language-cli">SQLSTATE[23502]: Not null violation: 7 ERROR:  null value in column "uuid" violates not-null constraint
DETAIL:  Failing row contains (4, 4, image, null, en).: UPDATE "test68160068media" SET "vid"=:db_update_placeholder_0, "bundle"=:db_update_placeholder_1, "uuid"=:db_update_placeholder_2, "langcode"=:db_update_placeholder_3
WHERE "mid" = :db_condition_placeholder_0; Array
(
  "db_update_placeholder_0" =&gt; 4,
  "db_update_placeholder_1" =&gt; "image",
  "db_update_placeholder_2" =&gt; NULL,
  "db_update_placeholder_3" =&gt; "en"
)
(/Users/zoli/projects/media_migration/zdev/public_html/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php:811)
</code></pre>

<h2 id="debugging">Debugging</h2>

<p>Obviously I started debugging this situation by setting up a breakpoint in <code class="language-plaintext highlighter-rouge">SqlContentEntityStorage</code> at line 811 then checking the <em>backtrace</em>. There was no surprise: the <code class="language-plaintext highlighter-rouge">uuid</code> property of the updated entity in <code class="language-plaintext highlighter-rouge">EntityContentBase::save()</code><sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup> really was empty!</p>

<p class="figure"><img src="/files/image/Empty-dest-prop-uuid.png" alt="" /></p>

<p>I checked the next trace (<code class="language-plaintext highlighter-rouge">EntityContentBase::import()</code>), and found a very weird thing in the <code class="language-plaintext highlighter-rouge">$row</code> variable’s empty destination properties:</p>

<p class="figure"><img src="/files/image/Empty-dest-prop-empty-props.png" alt="" /></p>

<p>Why is <code class="language-plaintext highlighter-rouge">uuid</code> listed twice? I flagged it as an empty destination in the <code class="language-plaintext highlighter-rouge">MediaMigrateUuid</code> process plugin if there is no <em>prophecy</em> for the given file ID (managed files are the data sources of the migrated media entities). So I set up another breakpoint there, and then followed the processing of the actual migration row.</p>

<h2 id="change-between-92x-and-93x">Change between 9.2.x and 9.3.x</h2>

<p>It turned out that the second time <code class="language-plaintext highlighter-rouge">uuid</code> is marked as empty happens in <code class="language-plaintext highlighter-rouge">MigrateExecutable</code>, more precisely <a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/modules/migrate/src/MigrateExecutable.php#L411-L419">here, at the end of its <code class="language-plaintext highlighter-rouge">processRow</code></a> method:</p>

<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre><span class="c1">// Ensure all values, including nulls, are migrated.</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$plugins</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">if</span> <span class="p">(</span><span class="k">isset</span><span class="p">(</span><span class="nv">$value</span><span class="p">))</span> <span class="p">{</span>
    <span class="nv">$row</span><span class="o">-&gt;</span><span class="nf">setDestinationProperty</span><span class="p">(</span><span class="nv">$destination</span><span class="p">,</span> <span class="nv">$value</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="k">else</span> <span class="p">{</span>
    <span class="nv">$row</span><span class="o">-&gt;</span><span class="nf">setEmptyDestinationProperty</span><span class="p">(</span><span class="nv">$destination</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>I was absolutely sure that this was not the way how destination property values were handled until recently, so I started checking the contents of the <code class="language-plaintext highlighter-rouge">MigrateExecutable</code> class switching between core development branches.</p>

<p>It turned out I remembered well! There was a bigger change between <code class="language-plaintext highlighter-rouge">9.2.x</code> and <code class="language-plaintext highlighter-rouge">9.3.x</code>. <a href="https://git.drupalcode.org/project/drupal/-/commit/8616edd24f8ded10f25a259e8d01244d9fb44f29">A commit (with a very less expressive commit message) has introduced</a> even the <code class="language-plaintext highlighter-rouge">MigrateExecutable::processRow</code> method and with it, the current way of handling empty destination properties.</p>

<p>I went back to my <code class="language-plaintext highlighter-rouge">MediaMigrateUuid</code> process plugin, and made the empty property flagging conditional: I do it only if the actual core version is lower than <code class="language-plaintext highlighter-rouge">9.3.0</code>:</p>

<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre><span class="c1">// No UUID was found – lets set the destination property to empty before</span>
<span class="c1">// throwing a skip process exception (this is only required for 9.2.x and</span>
<span class="c1">// below).</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">version_compare</span><span class="p">(</span><span class="err">\</span><span class="nc">Drupal</span><span class="o">::</span><span class="no">VERSION</span><span class="p">,</span> <span class="s1">'9.3.0'</span><span class="p">,</span> <span class="s1">'lt'</span><span class="p">))</span> <span class="p">{</span>
  <span class="nv">$row</span><span class="o">-&gt;</span><span class="nf">setEmptyDestinationProperty</span><span class="p">(</span><span class="nv">$destination_property</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>I hoped that this would fix the issue, but unfortunately this fix wasn’t enough: my change tracking test was wailing with the same error message that I saw at first.</p>

<h2 id="still-wrong-but-why">Still wrong! But why?…</h2>

<p>I knew I needed to know how my fully processed migration row looks and then what is the entity that is attempted to be saved by the migration destination plugin. In every similar situation I used to set up a breakpoint in <code class="language-plaintext highlighter-rouge">MigrateExecutabe</code>, <a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/modules/migrate/src/MigrateExecutable.php#L230">at the line where the destination plugin’s <code class="language-plaintext highlighter-rouge">import</code> method is invoked</a>.</p>

<p>I tracked down what happened: Before the import, my row contained the calculated destination properties and values I expected. But in <a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php#L159"><code class="language-plaintext highlighter-rouge">EntityContentBase::import</code></a>, the entity instance returned by <a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/modules/migrate/src/Plugin/migrate/destination/Entity.php#L155"><code class="language-plaintext highlighter-rouge">::getEntity()</code> method</a> (this is inherited from the parent <code class="language-plaintext highlighter-rouge">Entity</code> class) contained weird things: At the entity’s <code class="language-plaintext highlighter-rouge">values</code> key, I could see the current uuid, but at the <code class="language-plaintext highlighter-rouge">fields</code> key (this is where the processed destination property values are pushed into), the <code class="language-plaintext highlighter-rouge">uuid</code> property was present, and it was an empty array!</p>

<p class="figure"><img src="/files/image/Empty-dest-prop-update-uuid.png" alt="" /></p>

<p>Let’s see what happens in <code class="language-plaintext highlighter-rouge">Entity::getEntity()</code>! <a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/modules/migrate/src/Plugin/migrate/destination/Entity.php#L155-L161">This method basically starts with a condition</a>: if old destination IDs are available, then it tries to load the preexisting entity, then invokes <code class="language-plaintext highlighter-rouge">::updateEntity()</code>; and <a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php#L287-L289">at the end of the <code class="language-plaintext highlighter-rouge">EntityContentBase::updateEntity()</code></a> method we can see why the value of the <code class="language-plaintext highlighter-rouge">uuid</code> property will be <code class="language-plaintext highlighter-rouge">NULL</code>:</p>

<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="rouge-code"><pre><span class="k">protected</span> <span class="k">function</span> <span class="n">updateEntity</span><span class="p">(</span><span class="kt">EntityInterface</span> <span class="nv">$entity</span><span class="p">,</span> <span class="kt">Row</span> <span class="nv">$row</span><span class="p">)</span> <span class="p">{</span>
  <span class="nv">$empty_destinations</span> <span class="o">=</span> <span class="nv">$row</span><span class="o">-&gt;</span><span class="nf">getEmptyDestinationProperties</span><span class="p">();</span>
  
  <span class="p">(</span><span class="mf">...</span><span class="p">)</span>
  
  <span class="k">foreach</span> <span class="p">(</span><span class="nv">$empty_destinations</span> <span class="k">as</span> <span class="nv">$field_name</span><span class="p">)</span> <span class="p">{</span>
    <span class="nv">$entity</span><span class="o">-&gt;</span><span class="nv">$field_name</span> <span class="o">=</span> <span class="kc">NULL</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">setRollbackAction</span><span class="p">(</span><span class="nv">$row</span><span class="o">-&gt;</span><span class="nf">getIdMap</span><span class="p">(),</span> <span class="nv">$rollback_action</span><span class="p">);</span>

  <span class="c1">// We might have a different (translated) entity, so return it.</span>
  <span class="k">return</span> <span class="nv">$entity</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>I’ll be honest: I have no idea why this is needed. I would definitely unset the destination property values of the current <code class="language-plaintext highlighter-rouge">$row</code> object instead of setting the fields of the entity to <code class="language-plaintext highlighter-rouge">NULL</code>. <code class="language-plaintext highlighter-rouge">NULL</code> is a value, it does not mean emptiness by default. The <em>“empty”</em> attribution is determined by the field’s <em>FieldType</em> class (<a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/lib/Drupal/Core/TypedData/ComplexDataInterface.php#L97-L103">see <code class="language-plaintext highlighter-rouge">ComplexDataInterface::isEmtpy()</code></a>) or item list class (<a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/lib/Drupal/Core/TypedData/ListInterface.php#L28-L34">see <code class="language-plaintext highlighter-rouge">ListInterface::isEmpty()</code></a>).</p>

<p>Maybe I totally misunderstood what empty destination properties are used for…</p>

<h2 id="workaround">Workaround</h2>

<p>I worked this around with a complex process pipeline. Luckily, I had to write <a href="https://www.drupal.org/docs/contributed-modules/migrate-magician/migrate-magician-process-plugins-110/migmaggetentityproperty-120"><code class="language-plaintext highlighter-rouge">MigMagGetEntityProperty</code></a> for working around <a href="/development/2021/09/21/menu-link-migration-mess.html">menu link migration weirdnesses of Drupal core</a>, and this process plugin was a big help this time too.</p>

<p>So the actual solution:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">track_changes_uuid</code>: I do a lookup for the ID of an already migrated entity, and if it exists, then I get the entity’s UUID. This destination property only has value if the same entity has been migrated previously.</li>
  <li><code class="language-plaintext highlighter-rouge">oracle_uuid</code>: This is the process pipeline that computed the <code class="language-plaintext highlighter-rouge">uuid</code> property previously, by only taking the <em>UUID prophecy</em> table into account – but it was renamed to <code class="language-plaintext highlighter-rouge">oracle_uuid</code>.</li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">uuid</code>: Here is the magic! One of my favorite process plugins: <code class="language-plaintext highlighter-rouge">null_coalesce</code>! If <code class="language-plaintext highlighter-rouge">track_changes_uuid</code> is not null, then the computed <code class="language-plaintext highlighter-rouge">uuid</code> will be the value of <code class="language-plaintext highlighter-rouge">track_changes_uuid</code>. If it is <code class="language-plaintext highlighter-rouge">NULL</code>, but <code class="language-plaintext highlighter-rouge">oracle_uuid</code> has a non-null value, then that will be set.</p>

    <p>If both are empty, then this destination property will be set to <code class="language-plaintext highlighter-rouge">NULL</code> (and will be marked as being empty like before), but this happens only if we migrate the entity for the first time. And fortunately, the database exception is only triggered during entity updates.</p>
  </li>
</ul>

<p>Here is the code:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
</pre></td><td class="rouge-code"><pre><span class="na">id</span><span class="pi">:</span> <span class="s">d7_file_entity</span>
<span class="c1"># Label, source config, deriver class configuration...</span>
<span class="na">process</span><span class="pi">:</span>
  <span class="na">track_changes_uuid</span><span class="pi">:</span>
    <span class="pi">-</span>
      <span class="na">plugin</span><span class="pi">:</span> <span class="s">migration_lookup</span>
      <span class="na">source</span><span class="pi">:</span> <span class="s">fid</span>
      <span class="na">migration</span><span class="pi">:</span> <span class="s">d7_file_entity</span>
      <span class="na">no_stub</span><span class="pi">:</span> <span class="no">true</span>
    <span class="pi">-</span>
      <span class="na">plugin</span><span class="pi">:</span> <span class="s">skip_on_empty</span>
      <span class="na">method</span><span class="pi">:</span> <span class="s">process</span>
    <span class="pi">-</span>
      <span class="na">plugin</span><span class="pi">:</span> <span class="s">migmag_get_entity_property</span>
      <span class="na">entity_type_id</span><span class="pi">:</span> <span class="s1">'</span><span class="s">media'</span>
      <span class="na">property</span><span class="pi">:</span> <span class="s1">'</span><span class="s">uuid'</span>
  <span class="na">oracle_uuid</span><span class="pi">:</span>
    <span class="na">plugin</span><span class="pi">:</span> <span class="s">media_migrate_uuid</span>
    <span class="na">source</span><span class="pi">:</span> <span class="s">fid</span>
  <span class="na">uuid</span><span class="pi">:</span>
    <span class="na">plugin</span><span class="pi">:</span> <span class="s1">'</span><span class="s">null_coalesce'</span>
    <span class="na">source</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s1">'</span><span class="s">@track_changes_uuid'</span>
      <span class="pi">-</span> <span class="s1">'</span><span class="s">@oracle_uuid'</span>
<span class="c1"># The rest of the migration YAML remaind unchanged.</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<hr />

<p><em>Footnotes</em>:</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>These plugins are parsing test fields and transforming the old media embed codes to their Drupal 9+ equivalents and also replace inline <code class="language-plaintext highlighter-rouge">&lt;img&gt;</code> tags with embed code. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2" role="doc-endnote">
      <p>These are the process plugins which are changing Drupal 7 Media embed JSONs to Drupal 9 embed tags and which are transforming image and other file links to linkit tags, and which are converting <em>directly</em> used images (<code class="language-plaintext highlighter-rouge">&lt;img src="..."&gt;</code>) to Drupal 9 equivalent embed HTML tags. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3" role="doc-endnote">
      <p><code class="language-plaintext highlighter-rouge">EntityContentBase</code> is the destination plugin class of Drupal content entities. It is derived by <code class="language-plaintext highlighter-rouge">MigrateEntity</code> which decides <a href="https://git.drupalcode.org/project/drupal/-/blob/86c7ed07/core/modules/migrate/src/Plugin/Derivative/MigrateEntity.php#L59-L61">what plugin class should be used per entity type</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Drupal" /><category term="Migration" /><summary type="html"><![CDATA[Do you have any idea why it works this way?]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/files/image/Empty-dest-prop-uuid.png" /><media:content medium="image" url="https://huzooka.github.io/files/image/Empty-dest-prop-uuid.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">How to Serve a Public GitHub Pages Instance from a Private Repository for Free</title><link href="https://huzooka.github.io/development/2022/05/14/move-to-actions-from-pages.html" rel="alternate" type="text/html" title="How to Serve a Public GitHub Pages Instance from a Private Repository for Free" /><published>2022-05-14T12:02:19+00:00</published><updated>2022-05-14T12:02:19+00:00</updated><id>https://huzooka.github.io/development/2022/05/14/move-to-actions-from-pages</id><content type="html" xml:base="https://huzooka.github.io/development/2022/05/14/move-to-actions-from-pages.html"><![CDATA[<p class="lead">After publishing this site using <a href="https://pages.github.com/">GitHub Pages</a> (and <a href="https://jekyllrb.com/">Jekyll</a>), I immediately started looking for ways to open all the external URLs in a separate browser tab. I came across the <a href="https://github.com/keithmifsud/jekyll-target-blank">Jekyll Target Blank plugin</a> very quickly, but then I realized I can’t use it with the standard <a href="https://docs.github.com/en/pages/setting-up-a-github-pages-site-with-jekyll/about-github-pages-and-jekyll#plugins">GitHub Pages workflow</a> – because <a href="https://pages.github.com/versions/">it only supports a certain set of plugins</a>.</p>

<p>I did not settle for this situation: there must be a solution to this problem! And it turned out that <a href="https://blog.samplasion.js.org/build-jekyll-with-custom-plugins-on-github-pages/">there is one</a>: I only have to create my own <a href="https://github.com/features/actions">GitHub Actions</a> <em>workflow</em> that builds the artifact which is then deployed to the GitHub Pages web server.<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup></p>

<h2 id="controlling-the-jekyll-build-process">Controlling the Jekyll build process</h2>

<p>I basically followed <a href="https://blog.samplasion.js.org/build-jekyll-with-custom-plugins-on-github-pages/">Samplasion’s post</a>. I don’t want to copy it here, please read it carefully. There is only one thing to take care of: <strong><em>Read and write permissions</em> must be granted to workflows in the repository for all scopes</strong>.</p>

<p><em id="workflow">​</em>​You have to navigate to the repository <em>Settings</em>, select <em>Actions</em> on the right-hand side, and you can change this configuration on the bottom of the page, at the <em>Workflow permissions</em> section.</p>

<p class="figure"><img src="/files/image/Jekyll-workflow-permissions.png" alt="Workflow permissions settings on GitHub" />
Workflow permissions settings on GitHub at https://github.com/<wbr />[name]/[repository]/<wbr />settings/actions</p>

<p>The <code class="language-plaintext highlighter-rouge">joshlarsen/jekyll4-deploy-gh-pages</code> plugin I’m using to build and publish the artifacts is designed to push the build artifact into a <code class="language-plaintext highlighter-rouge">gh-pages</code> branch, and this cannot be modified<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>. So before you proceed, you have to create the <code class="language-plaintext highlighter-rouge">gh-pages</code> branch.</p>

<p>This was the workflow YAML I used:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
</pre></td><td class="rouge-code"><pre><span class="na">name</span><span class="pi">:</span> <span class="s">Github Pages Custom Build</span>
<span class="na">on</span><span class="pi">:</span>
  <span class="na">push</span><span class="pi">:</span>
    <span class="na">branches</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">main</span>
<span class="na">jobs</span><span class="pi">:</span>
  <span class="na">github-pages</span><span class="pi">:</span>
    <span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
    <span class="na">steps</span><span class="pi">:</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">GitHub Actions Checkout</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">Cache</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/cache@v2</span>
        <span class="na">with</span><span class="pi">:</span>
          <span class="na">path</span><span class="pi">:</span> <span class="s">vendor/bundle</span>
          <span class="na">key</span><span class="pi">:</span> <span class="s">${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}</span>
          <span class="na">restore-keys</span><span class="pi">:</span> <span class="pi">|</span>
            <span class="s">${{ runner.os }}-gems-</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">Jekyll Build &amp; Deploy</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">joshlarsen/jekyll4-deploy-gh-pages@v1.8</span>
        <span class="na">env</span><span class="pi">:</span>
          <span class="na">GITHUB_TOKEN</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_TOKEN }}</span>
          <span class="na">GITHUB_REPOSITORY</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_REPOSITORY }}</span>
          <span class="na">GITHUB_ACTOR</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_ACTOR }}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<p>After I saw that this workflow finished successfully, I changed my GitHub Pages configuration: I set the source branch to <code class="language-plaintext highlighter-rouge">gh-pages</code>, pushed a test commit into the main branch – and it was published in less than 2 minutes!</p>

<h2 id="being-able-to-push-from-a-private-repository-intoa-public-one">Being able to push from a private repository into a public one</h2>

<p>Maybe because of the success, I wanted to solve another new task. I’m quite shy 😉, I don’t want to publish all the changes I make or my public commit history. Can’t we publish the build from a private repository somehow (for <em>free</em>)?</p>

<p>Yes, we can!</p>

<ol>
  <li>I created an orphan branch in my public GitHub Pages repository temporarily, and named it <code class="language-plaintext highlighter-rouge">test</code>. My goal was to push into this branch.</li>
  <li>I created a new private repository (let’s say it is <code class="language-plaintext highlighter-rouge">huzooka/github-site</code>. This is the repository which will contain all the source files.</li>
  <li>In the private repository, I created a branch named <code class="language-plaintext highlighter-rouge">main</code>. This contained a single <code class="language-plaintext highlighter-rouge">test.txt</code> with some random text.</li>
  <li><a href="#workflow">I granted read and write permissions to workflows</a> in the private repo.</li>
  <li>To be able to push to the public repository, I had to create a <abbr title="Personal Access Token">PAT</abbr> in the public GitHub Pages repository, and then add it as a <em>secret</em> to the private repository. This is essential: without a token you won’t be able to push the build to the public repo. Please follow <a href="https://kuros.in/ci/cd/use-private-repo-to-publish-website-with-github-pages/">Kumar Rohit’s post (I did that too)</a>.</li>
  <li>
    <p>I started to write a very simple workflow in the private repo, and each commit I have been sitting on needles and pins waiting for the text file to appear in the public repository. This worked:</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
</pre></td><td class="rouge-code"><pre><span class="na">name</span><span class="pi">:</span> <span class="s">Build and release</span>
<span class="na">on</span><span class="pi">:</span>
  <span class="na">push</span><span class="pi">:</span>
    <span class="na">branches</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">main</span>
<span class="na">env</span><span class="pi">:</span>
  <span class="na">BUILD_DIRECTORY</span><span class="pi">:</span> <span class="s1">'</span><span class="s">destination'</span>
  <span class="na">TARGET_REPOSITORY</span><span class="pi">:</span> <span class="s1">'</span><span class="s">huzooka/huzooka.github.io'</span>
  <span class="na">TARGET_BRANCH</span><span class="pi">:</span> <span class="s">test</span>
<span class="na">jobs</span><span class="pi">:</span>
  <span class="na">build</span><span class="pi">:</span>
    <span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
    <span class="na">steps</span><span class="pi">:</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">GitHub Actions Checkout</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
        <span class="na">with</span><span class="pi">:</span>
          <span class="na">path</span><span class="pi">:</span> <span class="s">source</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">Do some dummy action</span> <span class="c1"># @todo: Replace with a Jekyll build.</span>
        <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
          <span class="s">mkdir -p "${GITHUB_WORKSPACE}/${BUILD_DIR}"</span>
          <span class="s">cp ${GITHUB_WORKSPACE}/source/*.* ${GITHUB_WORKSPACE}/${BUILD_DIR}/</span>
          <span class="s">md5sum "$GITHUB_WORKSPACE/${BUILD_DIR}/test.txt" &gt; "$GITHUB_WORKSPACE/${BUILD_DIR}/test.md5sum"</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">Release to Public Repository</span>
        <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
          <span class="s">cd "$GITHUB_WORKSPACE/_site"</span>
          <span class="s">export BUILD_BRANCH=`git rev-parse --abbrev-ref HEAD`</span>
          <span class="s">git push --force "https://${GITHUB_ACTOR}:${{ secrets.GH_PAGES_REPO_TOKEN }}@github.com/${TARGET_REPOSITORY}.git" ${BUILD_BRANCH}:${TARGET_BRANCH}</span>
</pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
</ol>

<h2 id="combining-the-above">Combining the above</h2>

<p>Finally, I combined the steps above together. I’m pretty sure there are more configurable Jekyll build plugins, but I kept using <code class="language-plaintext highlighter-rouge">joshlarsen/jekyll4-deploy-gh-pages</code>. It has another limitation though: because of some reasons<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>, I wasn’t able to make it push into another repository.</p>

<p>I can live with that: I let it push into the private repository’s <code class="language-plaintext highlighter-rouge">gh-pages</code> branch 🤷‍♂️. But after that step finishes, my next workflow task pushes the same artifact into my <code class="language-plaintext highlighter-rouge">public</code> GitHub Pages repo too.</p>

<p>This is my private GitHub Actions workflow:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
</pre></td><td class="rouge-code"><pre><span class="na">name</span><span class="pi">:</span> <span class="s">Build and release</span>
<span class="na">on</span><span class="pi">:</span>
  <span class="na">push</span><span class="pi">:</span>
    <span class="na">branches</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">main</span>
<span class="na">env</span><span class="pi">:</span>
  <span class="na">TARGET_REPOSITORY</span><span class="pi">:</span> <span class="s1">'</span><span class="s">huzooka/huzooka.github.io'</span>
  <span class="na">TARGET_BRANCH</span><span class="pi">:</span> <span class="s">gh-pages</span>
<span class="na">jobs</span><span class="pi">:</span>
  <span class="na">build</span><span class="pi">:</span>
    <span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
    <span class="na">steps</span><span class="pi">:</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">GitHub Actions Checkout</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">Cache</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/cache@v2</span>
        <span class="na">with</span><span class="pi">:</span>
          <span class="na">path</span><span class="pi">:</span> <span class="s">vendor/bundle</span>
          <span class="na">key</span><span class="pi">:</span> <span class="s">${{ runner.os }}-gems-${{hashFiles('**/Gemfile.lock') }}</span>
          <span class="na">restore-keys</span><span class="pi">:</span> <span class="pi">|</span>
            <span class="s">${{ runner.os }}-gems-</span>
      <span class="pi">-</span> 
        <span class="na">name</span><span class="pi">:</span> <span class="s">Jekyll Build &amp; Dummy Deploy</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">joshlarsen/jekyll4-deploy-gh-pages@v1.8</span>
        <span class="na">env</span><span class="pi">:</span>
          <span class="na">GITHUB_TOKEN</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_TOKEN }}</span>
          <span class="na">GITHUB_REPOSITORY</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_REPOSITORY }}</span>
          <span class="na">GITHUB_ACTOR</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_ACTOR }}</span>
      <span class="pi">-</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">Release to Public Repository</span>
        <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
          <span class="s">cd "$GITHUB_WORKSPACE/_site"</span>
          <span class="s">export BUILD_BRANCH=`git rev-parse --abbrev-ref HEAD`</span>
          <span class="s">git push --force "https://${GITHUB_ACTOR}:${{ secrets.GH_PAGES_REPO_TOKEN }}@github.com/${TARGET_REPOSITORY}.git" ${BUILD_BRANCH}:${TARGET_BRANCH}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<hr />

<p><em>Sources</em>:</p>

<ul>
  <li><a href="https://blog.samplasion.js.org/build-jekyll-with-custom-plugins-on-github-pages/">Build Jekyll with custom plugins on GitHub Pages</a> by <a href="https://github.com/Samplasion">Samplasion</a></li>
  <li><a href="https://jekyllrb.com/docs/continuous-integration/github-actions/">GitHub Actions</a> documentation and examples on <a href="https://jekyllrb.com/">https://jekyllrb.com/</a></li>
  <li><a href="https://kuros.in/ci/cd/use-private-repo-to-publish-website-with-github-pages/">Use private repo to publish websites with github pages</a> by Kumar Rohit</li>
</ul>

<hr />

<p><em>Footnotes</em>:</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>This is performed by the <a href="https://github.com/huzooka/huzooka.github.io/actions/workflows/pages/pages-build-deployment">pages build and deployment</a> <em>action</em>. This action is the essence of GitHub Pages, but it is <em>not completely</em> under our control, we cannot modify it or find it in our codebase. As soon as we enable GitHub Pages on a repository, it suddenly appears, and if we turn it off, it disappears 🙂. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2" role="doc-endnote">
      <p>Yes, I know I can fork any repository anytime. But why would I want to maintain yet another package? <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3" role="doc-endnote">
      <p>It seems that we can configure it to push to another repo, but it doesn’t work – and I really don’t have an idea why. The action finished successfully, but nothing was pushed to the public repository <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="GitHub Pages" /><category term="GitHub Actions" /><category term="Jekyll" /><summary type="html"><![CDATA[How I moved from a standard GitHub Pages setup to custom Jekyll builds deployed from a private repository.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/simple.png" /><media:content medium="image" url="https://huzooka.github.io/simple.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Composer Package Trick for Installing Incompatible Composer Extensions</title><link href="https://huzooka.github.io/development/2022/05/08/composer-package-trick.html" rel="alternate" type="text/html" title="Composer Package Trick for Installing Incompatible Composer Extensions" /><published>2022-05-08T17:30:00+00:00</published><updated>2022-05-08T17:30:00+00:00</updated><id>https://huzooka.github.io/development/2022/05/08/composer-package-trick</id><content type="html" xml:base="https://huzooka.github.io/development/2022/05/08/composer-package-trick.html"><![CDATA[<p class="lead">This is the up-to-date, user-friendly version of the <a href="/development/2020/04/08/composer-package-drupal9.html">post I wrote 2 years ago</a> about the same topic.</p>

<p><a href="#steps-to-follow">⇩ Jump to the steps</a>.</p>

<p>In this post you will be able to see how to add an incompatible Drupal module to a Drupal 10 project with Composer. At the time I’m writing (well, making up-to-date) this post, I’m not able to do this with <a href="https://drupal.org/project/eme">Entity Migrate Export</a> module: both the latest tag release (<code class="language-plaintext highlighter-rouge">1.0.0-alpha10</code>) and the latest branch release (<code class="language-plaintext highlighter-rouge">1.0.x-dev</code>) require Drupal core <code class="language-plaintext highlighter-rouge">^8.9 || ^9</code>:</p>

<pre><code class="language-cli">$ composer require drupal/eme
Using version ^1.0@alpha for drupal/eme
./composer.json has been updated
Running composer update drupal/eme
Gathering patches for root package.
No patches supplied.
Loading composer repositories with package information
Updating dependencies
Info from https://repo.packagist.org: #StandWithUkraine
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - drupal/eme[1.0.0-alpha1, ..., 1.0.0-alpha5] require drupal/core ^8.7.7 || ^9 -&gt; found drupal/core[8.7.7, ..., 8.9.x-dev, 9.0.0-alpha1, ..., 9.5.x-dev] but the package is fixed to 10.0.x-dev (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.
    - drupal/eme[1.0.0-alpha6, ..., 1.0.0-alpha10] require drupal/core ^8.9 || ^9 -&gt; found drupal/core[8.9.0-beta1, ..., 8.9.x-dev, 9.0.0-alpha1, ..., 9.5.x-dev] but the package is fixed to 10.0.x-dev (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.
    - Root composer.json requires drupal/eme ^1.0@alpha -&gt; satisfiable by drupal/eme[1.0.0-alpha1, ..., 1.0.0-alpha10].

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.

Installation failed, reverting ./composer.json and ./composer.lock to their original content.
</code></pre>

<h2 id="steps-to-follow">Steps to follow</h2>

<h3 id="find-the-corresponding-package-info-json-file">Find the corresponding package info JSON file</h3>

<ol>
  <li>We will be using the package info that’s already available for your composer. So, locate where your composer cache is, by executing <code class="language-plaintext highlighter-rouge">composer config cache-dir</code>:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre><span class="nv">$ </span>composer config cache-dir
/Users/zoli/Library/Caches/composer
</pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
  <li>
    <p>The package info will be inside the cache directory’s <code class="language-plaintext highlighter-rouge">repo</code> subdirectory, in the corresponding repository subdirectory. On my development environment, in case of Drupal extensions, this is <code class="language-plaintext highlighter-rouge">/Users/zoli/Library/Caches/composer/repo/https---packages.drupal.org-8/</code>.</p>
  </li>
  <li>For Entity Migrate Export, we will have two different JSON files: one for the <em>usual</em> tag-based releases (this is <code class="language-plaintext highlighter-rouge">provider-drupal~eme.json</code>), and an another one for the <em>branch</em> based releases (<code class="language-plaintext highlighter-rouge">provider-drupal~eme~dev.json</code>). Imho the best practice is to <em>replace</em> the latest release we found in the <em>tag</em> info JSON: so if the maintainer releases a newer (and already compatible) release, then we will also be informed through Drupal’s Module Update UI.</li>
</ol>

<h3 id="add-the-right-json-object-to-your-root-composerjson-as-custom-package">Add the right JSON object to your root <code class="language-plaintext highlighter-rouge">composer.json</code> as custom package</h3>

<p>One rule here: your custom package entry must be added before the Drupal packagist repository.</p>

<ol>
  <li>Open the package file, locate the latest release. This is the info you will be using:
    <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
</pre></td><td class="rouge-code"><pre><span class="p">{</span><span class="w">
  </span><span class="nl">"packages"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"drupal/eme"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
      </span><span class="p">{</span><span class="w">
        </span><span class="nl">"keywords"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"Drupal"</span><span class="p">],</span><span class="w">
        </span><span class="nl">"homepage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://www.drupal.org/project/eme"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0-alpha10"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"version_normalized"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0.0-alpha10"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"license"</span><span class="p">:</span><span class="w"> </span><span class="s2">"GPL-2.0-or-later"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"authors"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="err">…</span><span class="p">],</span><span class="w">
        </span><span class="nl">"support"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="err">…</span><span class="p">},</span><span class="w">
        </span><span class="nl">"source"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="err">…</span><span class="p">},</span><span class="w">
        </span><span class="nl">"dist"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="err">…</span><span class="p">},</span><span class="w">
        </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"drupal-module"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"uid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"eme-3219974"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"drupal/eme"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"extra"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="err">…</span><span class="p">},</span><span class="w">
        </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Exports content entities into a migration module"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"require"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
          </span><span class="nl">"drupal/core"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^8.9 || ^9"</span><span class="w">
        </span><span class="p">},</span><span class="w">
        </span><span class="nl">"require-dev"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
          </span><span class="nl">"drupal/migrate_tools"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^5"</span><span class="p">,</span><span class="w">
          </span><span class="nl">"drupal/migrate_plus"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^5"</span><span class="p">,</span><span class="w">
          </span><span class="nl">"drush/drush"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^9 || ^10"</span><span class="w">
        </span><span class="p">}</span><span class="w">
      </span><span class="p">},</span><span class="w">
      </span><span class="p">{</span><span class="err">…</span><span class="p">},</span><span class="w">
      </span><span class="p">{</span><span class="err">…</span><span class="p">}</span><span class="w">
    </span><span class="p">]</span><span class="w">
  </span><span class="p">},</span><span class="w">
  </span><span class="nl">"minified"</span><span class="p">:</span><span class="w"> </span><span class="s2">"composer/2.0"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"last-modified"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Sat, 30 Apr 2022 16:30:38 GMT"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
  <li>The section you will paste into the root <code class="language-plaintext highlighter-rouge">composer.json</code> is a single release info, an object which starts with the <code class="language-plaintext highlighter-rouge">keywords</code> key. Add this section to your root <code class="language-plaintext highlighter-rouge">composer.json</code>, above Drupal’s repository:
    <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
</pre></td><td class="rouge-code"><pre><span class="p">{</span><span class="w">
  </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"z.a.horvath/incubator"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"drupal-project"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"repositories"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"custom repo key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"package"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"package"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nl">"keywords"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"Drupal"</span><span class="p">],</span><span class="w">
        </span><span class="nl">"homepage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://www.drupal.org/project/eme"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0-alpha10"</span><span class="w">
        </span><span class="err">…</span><span class="w">
      </span><span class="p">}</span><span class="w">
    </span><span class="p">},</span><span class="w">
    </span><span class="nl">"drupal"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"composer"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://packages.drupal.org/8"</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">},</span><span class="w">
  </span><span class="err">…</span><span class="w">
</span><span class="p">}</span><span class="w">       
</span></pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
</ol>

<h3 id="fix-the-requirements-version-constraints-which-cause-the-conflict">Fix the requirement’s version constraints which cause the conflict</h3>

<p>Locate the <code class="language-plaintext highlighter-rouge">require</code> key in the package info, and modify the version constraints which cause conflicts:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre><span class="nl">"require"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
  </span><span class="nl">"drupal/core"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^8.9 || ^9"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div></div>
<p>Let’s change the version constraint of <code class="language-plaintext highlighter-rouge">drupal/core</code> requirement, and allow Drupal 10 as well:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre><span class="nl">"require"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
  </span><span class="nl">"drupal/core"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^8.9 || ^9 || ^10"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div></div>

<h2 id="summary">Summary</h2>

<p>With the solution explained above we are able to download and use incompatible extensions in our Composer project. Of course these extensions might be <em>really</em> incompatible, but as soon as we can download them we can use <a href="https://github.com/cweagans/composer-patches">cweagans/composer-patches</a> to apply patches fixing the issues, or we can start writing the fix and share it with the community.</p>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Composer" /><category term="Drupal" /><summary type="html"><![CDATA[How to suppress the repository info of a Composer package to make it installable in your Drupal 10 project.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/simple.png" /><media:content medium="image" url="https://huzooka.github.io/simple.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Prepare a Customer’s Database to be Used in a Test – with a Drupal Kernel Test!</title><link href="https://huzooka.github.io/development/2021/12/04/ttd-test-data.html" rel="alternate" type="text/html" title="Prepare a Customer’s Database to be Used in a Test – with a Drupal Kernel Test!" /><published>2021-12-04T14:34:26+00:00</published><updated>2021-12-04T14:34:26+00:00</updated><id>https://huzooka.github.io/development/2021/12/04/ttd-test-data</id><content type="html" xml:base="https://huzooka.github.io/development/2021/12/04/ttd-test-data.html"><![CDATA[<p class="lead">The more complex the menu structure is, the more important it is to the customer is. And at the same time, the more complex the menu structure is, the less menu links are migrated to Drupal 9. We immediately realized this when we understood <a href="/development/2021/09/21/menu-link-migration-mess.html">the bug in my previous post</a>.</p>

<h2 id="motivation">Motivation</h2>

<p>Although we’ve built some patches for this customer, we aim to help not only Acquia customers, but as many Drupal 7 users as possible. This is why the latest <a href="https://drupal.org/project/migmag/">Migrate Magician</a> submodule will be created soon.</p>

<p>In this post, I share my basic notes I took while I was transforming the sanitized customer database to a minimal database fixture that I have been using during the development.</p>

<h2 id="fundaments">Fundaments</h2>

<p>Maybe you already know this about me: I really like tests! Not just because they give me some extra confidence about the code I write, but because tests do the tedious and time-consuming, repetitive steps for me, and I can focus more on the actual development.</p>

<p>But this case was a bit special: we had a great, difficult menu structure in a client’s database which I wanted to use as test data during the development, when running the “real” test.</p>

<p>Maybe you don’t know about it, but Drupal 9 has a built-in <a href="https://api.drupal.org/api/drupal/core%21scripts%21db-tools.php">CLI database application</a> which can export MySQL/MariaDB databases into a PHP file. The only problem with it is that it creates one single PHP file. I wrote a smarter replacement on its fundamentals which creates per-table database fixture files, and it also can be configured to split big tables into chunks. This is <a href="https://www.drupal.org/project/smart_db_tools">Smart DB Tools</a>, that is what I was using here.</p>

<h2 id="preparation">Preparation</h2>

<ol>
  <li>
    <p>I’ve exported our customer’s sanitized source database to a database fixture using Smart DB Tools.</p>

    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre> php ./modules/contrib/smart_db_tool/scripts/smart-db-tools.php dump<span class="se">\</span>
   <span class="nt">--database</span> fixture_connection<span class="se">\</span>
   <span class="nt">--split-destination</span> ./temp/d7-menu-link-raw.php
</pre></td></tr></tbody></table></code></pre></div>    </div>

    <p class="figure"><img src="/files/image/tdd-menu-link-raw-export.png" alt="The exported database fixture file, and the subdirectory which holds the per-table fixtures" />
 The exported database fixture file, and the subdirectory which holds the per-table fixtures</p>
  </li>
  <li>
    <p>I created a <code class="language-plaintext highlighter-rouge">tests/fixtures/d7-menu-link-db.php</code> file and a <code class="language-plaintext highlighter-rouge">tests/fixtures/d7-menu-link-db</code> subdirectory, moved the exported <code class="language-plaintext highlighter-rouge">menu_links.php</code> and <code class="language-plaintext highlighter-rouge">menu_custom.php</code> table fixture files into the subdirectory – because I assumed that executing the <code class="language-plaintext highlighter-rouge">d7_menu_link</code> migration will probably need these tables 😉.</p>
  </li>
  <li>I wrote a very basic kernel test based on MigrateDrupalTestBase, specified the <code class="language-plaintext highlighter-rouge">tests/fixtures/d7-menu-link-db.php</code> file created previously as a fixture file, and executed the test, which only migrated <code class="language-plaintext highlighter-rouge">d7_menu</code> and <code class="language-plaintext highlighter-rouge">d7_menu_links</code>. Each time the test failed, I checked the error message, and either:
    <ul>
      <li>added the missing migration to the <code class="language-plaintext highlighter-rouge">executeMigrations()</code> method argument array</li>
      <li>installed the missing core module, or the missing entity schema or module config</li>
      <li>or if the error was about a missing database table a migration source plugin tried to access, then I created the corresponding table fixture file in the <code class="language-plaintext highlighter-rouge">tests/fixtures/d7-menu-link-db</code> subdirectory, copied the table schema from the original fixture into it, and added an <code class="language-plaintext highlighter-rouge">include</code> statement in the main <code class="language-plaintext highlighter-rouge">tests/fixtures/d7-menu-link-db.php</code> file.</li>
    </ul>
  </li>
  <li>
    <p>There are some tables which <em>should</em> be present,but can be empty. These were <code class="language-plaintext highlighter-rouge">field_config</code>, <code class="language-plaintext highlighter-rouge">field_config_instance</code>, <code class="language-plaintext highlighter-rouge">role</code>, <code class="language-plaintext highlighter-rouge">role_permissions</code>, <code class="language-plaintext highlighter-rouge">users</code> and <code class="language-plaintext highlighter-rouge">user_roles</code>. We obviously need a user for authoring the migrated nodes, but user 1 will be available in our kernel test.</p>
  </li>
  <li>I also created empty fixture files for the <code class="language-plaintext highlighter-rouge">node</code> and the <code class="language-plaintext highlighter-rouge">node_revision</code> tables temporarily, and made a cleaned-up <code class="language-plaintext highlighter-rouge">system</code> table fixture file where I only kept the rows of the <code class="language-plaintext highlighter-rouge">field</code>, <code class="language-plaintext highlighter-rouge">field_storage</code>, <code class="language-plaintext highlighter-rouge">menu</code>, <code class="language-plaintext highlighter-rouge">node</code>, <code class="language-plaintext highlighter-rouge">system</code> and <code class="language-plaintext highlighter-rouge">user</code> modules (all of them enabled).</li>
</ol>

<p>And this was the kernel test I used:</p>

<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
</pre></td><td class="rouge-code"><pre><span class="cp">&lt;?php</span>

<span class="kn">namespace</span> <span class="nn">Drupal\Tests\migmag_menu_link_migrate\Kernel</span><span class="p">;</span>

<span class="kn">use</span> <span class="nf">Drupal\menu_link_content</span><span class="err">\</span><span class="nc">Entity\MenuLinkContent</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Drupal\Tests\migmag\Traits\MigMagKernelTestDxTrait</span><span class="p">;</span>
<span class="kn">use</span> <span class="nf">Drupal\Tests\migrate_drupal</span><span class="err">\</span><span class="nc">Kernel\MigrateDrupalTestBase</span><span class="p">;</span>

<span class="cd">/**
 * Tests the enhanced menu link migration.
 *
 * @group migmag_menu_link_migrate
 */</span>
<span class="kd">class</span> <span class="nc">MenuLinkMigrateTest</span> <span class="kd">extends</span> <span class="nc">MigrateDrupalTestBase</span> <span class="p">{</span>

  <span class="kn">use</span> <span class="nc">MigMagKernelTestDxTrait</span><span class="p">;</span>

  <span class="cd">/**
   * {@inheritdoc}
   */</span>
  <span class="k">protected</span> <span class="k">static</span> <span class="nv">$modules</span> <span class="o">=</span> <span class="p">[</span>
    <span class="s1">'comment'</span><span class="p">,</span>
    <span class="s1">'link'</span><span class="p">,</span>
    <span class="s1">'menu_link_content'</span><span class="p">,</span>
    <span class="s1">'node'</span><span class="p">,</span>
  <span class="p">];</span>

  <span class="cd">/**
   * {@inheritdoc}
   */</span>
  <span class="k">protected</span> <span class="k">function</span> <span class="n">setUp</span><span class="p">()</span> <span class="p">{</span>
    <span class="k">parent</span><span class="o">::</span><span class="nf">setUp</span><span class="p">();</span>

    <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">installEntitySchema</span><span class="p">(</span><span class="s1">'menu_link_content'</span><span class="p">);</span>

    <span class="nv">$fixture_path</span> <span class="o">=</span> <span class="nb">implode</span><span class="p">(</span><span class="no">DIRECTORY_SEPARATOR</span><span class="p">,</span> <span class="p">[</span>
      <span class="nf">drupal_get_path</span><span class="p">(</span><span class="s1">'module'</span><span class="p">,</span> <span class="s1">'migmag_menu_link_migrate'</span><span class="p">),</span>
      <span class="s1">'tests'</span><span class="p">,</span>
      <span class="s1">'fixtures'</span><span class="p">,</span>
      <span class="s1">'d7-menu-link-db.php'</span><span class="p">,</span>
    <span class="p">]);</span>

    <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">loadFixture</span><span class="p">(</span><span class="nv">$fixture_path</span><span class="p">);</span>
  <span class="p">}</span>

  <span class="cd">/**
   * Test the enhanced menu link migration.
   */</span>
  <span class="k">public</span> <span class="k">function</span> <span class="n">testMenuLinkMigration</span><span class="p">()</span> <span class="p">{</span>
    <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">startCollectingMessages</span><span class="p">();</span>
    <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">executeMigrations</span><span class="p">([</span>
      <span class="s1">'d7_node_type'</span><span class="p">,</span>
      <span class="s1">'d7_user_role'</span><span class="p">,</span>
      <span class="s1">'d7_user'</span><span class="p">,</span>
      <span class="s1">'d7_node_complete'</span><span class="p">,</span>
      <span class="s1">'d7_menu'</span><span class="p">,</span>
      <span class="s1">'d7_menu_links'</span><span class="p">,</span>
    <span class="p">]);</span>
    <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">assertNoMigrationMessages</span><span class="p">();</span>

    <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">assertCount</span><span class="p">(</span><span class="mi">999</span><span class="p">,</span> <span class="nc">MenuLinkContent</span><span class="o">::</span><span class="nf">loadMultiple</span><span class="p">());</span>
  <span class="p">}</span>

<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>

<h2 id="write-a-database-fixture-with-a-test">Write a database fixture with a test!</h2>

<p>I removed the <code class="language-plaintext highlighter-rouge">d7_node_type</code>, <code class="language-plaintext highlighter-rouge">d7_user_role</code>, <code class="language-plaintext highlighter-rouge">d7_user</code> and <code class="language-plaintext highlighter-rouge">d7_node_complete</code> migrations, set a breakpoint after the migrations were executed, and checked how many records I have in the migrate map table of <code class="language-plaintext highlighter-rouge">d7_menu_links</code>: I had 289 messages! I went ahead and updated the count assertion: I want to have all of them migrated.</p>

<p>At this point, I had only one menu link which had been migrated. And almost every other menu link which failed to be migrated had a message in the migrate message table! Most of them (276 out of 283) contained a message like this:</p>

<pre><code class="language-cli">d7_menu_links:link/uri: The path "internal:/node/801" failed validation.
</code></pre>

<p>This customer has more than 2500 nodes. Obviously, I don’t want to sanitize the whole client DB when I publish this work including a test with the database fixture. First, because it is an overhead, and on the other side I don’t want to migrate 2500 nodes (17000 revisions in case of using the complete node migration) just for being able to migrate 289 menu links. I only need the minimal data being available about these nodes. And the very minimal data is <em>their ID</em>. I can get them very easily, by parsing these migration messages!</p>

<ol>
  <li>
    <p>Let’s get the messages! This method was my tool:</p>

    <div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Returns the migration messages saved for the specified migration.
 *
 * @param string $plugin_id
 *   The (full) plugin ID of the corresponding migration plugin instance.
 * 
 * @return array[]
 *   The list of the migrate message record properties, containing only the 
 *   message (keyed by its column name 'message').
 */</span>
<span class="k">protected</span> <span class="k">function</span> <span class="n">getMigrationMessages</span><span class="p">(</span><span class="nv">$plugin_id</span><span class="p">)</span> <span class="p">{</span>
  <span class="nv">$migration</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="nf">getMigration</span><span class="p">(</span><span class="nv">$plugin_id</span><span class="p">);</span>
  <span class="nv">$idmap</span> <span class="o">=</span> <span class="nv">$migration</span><span class="o">-&gt;</span><span class="nf">getIdMap</span><span class="p">();</span>
  <span class="nb">assert</span><span class="p">(</span><span class="nv">$idmap</span> <span class="k">instanceof</span> <span class="nc">Sql</span><span class="p">);</span>
   
  <span class="k">return</span> <span class="err">\</span><span class="nc">Drupal</span><span class="o">::</span><span class="nf">database</span><span class="p">()</span>
    <span class="o">-&gt;</span><span class="nf">select</span><span class="p">(</span><span class="nv">$idmap</span><span class="o">-&gt;</span><span class="nf">messageTableName</span><span class="p">(),</span> <span class="s1">'m'</span><span class="p">)</span>
    <span class="o">-&gt;</span><span class="nf">fields</span><span class="p">(</span><span class="s1">'m'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'message'</span><span class="p">])</span>
    <span class="o">-&gt;</span><span class="nf">execute</span><span class="p">()</span>
    <span class="o">-&gt;</span><span class="nf">fetchAll</span><span class="p">(</span><span class="err">\</span><span class="no">PDO</span><span class="o">::</span><span class="no">FETCH_ASSOC</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
  <li>
    <p>And now, I can get the IDs of those nodes which have a menu link:</p>

    <div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Obtains the required node IDs from the migration messages.
 * 
 * @param array[] $messages
 *   An array of the migrate map table records. At the very minimum, the 
 *   message key must be present (and its value should be the message).
 * 
 * @return int[]
 *   The "missing" node IDs.
 */</span>
<span class="k">protected</span> <span class="k">function</span> <span class="n">getMissingNodeIds</span><span class="p">(</span><span class="kt">array</span> <span class="nv">$messages</span><span class="p">)</span> <span class="p">{</span>
  <span class="nv">$missing_node_ids</span> <span class="o">=</span> <span class="nb">array_reduce</span><span class="p">(</span>
    <span class="nv">$messages</span><span class="p">,</span>
    <span class="k">function</span> <span class="p">(</span><span class="kt">array</span> <span class="nv">$carry</span><span class="p">,</span> <span class="kt">array</span> <span class="nv">$message_data</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">if</span> <span class="p">(</span><span class="nb">preg_match</span><span class="p">(</span><span class="s1">'/\sThe path "\w+:\/node\/(\d+).*" failed validation/'</span><span class="p">,</span> <span class="nv">$message_data</span><span class="p">[</span><span class="s1">'message'</span><span class="p">],</span> <span class="nv">$matches</span><span class="p">))</span> <span class="p">{</span>
        <span class="nv">$carry</span><span class="p">[]</span> <span class="o">=</span> <span class="p">(</span><span class="n">int</span><span class="p">)</span> <span class="nv">$matches</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
      <span class="p">}</span>
      <span class="k">return</span> <span class="nv">$carry</span><span class="p">;</span>
    <span class="p">},</span>
    <span class="p">[]</span>
  <span class="p">);</span>
  <span class="nv">$missing_node_ids</span> <span class="o">=</span> <span class="nb">array_unique</span><span class="p">(</span><span class="nv">$missing_node_ids</span><span class="p">);</span>
  <span class="nb">natsort</span><span class="p">(</span><span class="nv">$missing_node_ids</span><span class="p">);</span>
  <span class="k">return</span> <span class="nb">array_values</span><span class="p">(</span><span class="nv">$missing_node_ids</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div>    </div>

    <p>This method returned 210 node IDs. Yes, this number is less than 276, but this difference means that we have some menu links which are pointing to the same node.</p>
  </li>
  <li>
    <p>The next task was adding records of these node IDs into the <code class="language-plaintext highlighter-rouge">node</code> and the <code class="language-plaintext highlighter-rouge">node_revision</code> DB table fixture. So I wrote a new helper method which consumes the list of these node IDs returned by <code class="language-plaintext highlighter-rouge">::getMissingNodeIds()</code>, builds the appropriate data for the fixture, and exports it into a file.</p>

    <div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
</pre></td><td class="rouge-code"><pre><span class="cd">/**
 * Saves a developer friendly (but incomplete) node table fixture file.
 * 
 * @param int[] $missing_node_ids
 *   The missing node IDs.
 */</span>
<span class="k">protected</span> <span class="k">function</span> <span class="n">saveToNodeFixture</span><span class="p">(</span><span class="kt">array</span> <span class="nv">$missing_node_ids</span><span class="p">):</span> <span class="kt">void</span> <span class="p">{</span>
  <span class="nv">$raw_data</span> <span class="o">=</span> <span class="nb">array_reduce</span><span class="p">(</span>
    <span class="nv">$missing_node_ids</span><span class="p">,</span>
    <span class="k">function</span> <span class="p">(</span><span class="kt">array</span> <span class="nv">$carry</span><span class="p">,</span> <span class="kt">int</span> <span class="nv">$node_id</span><span class="p">)</span> <span class="p">{</span>
      <span class="nv">$type</span> <span class="o">=</span> <span class="p">[</span>
        <span class="s1">'test_type1'</span><span class="p">,</span>
        <span class="s1">'test_type2'</span><span class="p">,</span>
        <span class="s1">'test_type3'</span><span class="p">,</span>
      <span class="p">][</span><span class="nb">random_int</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">)];</span>
   
      <span class="nv">$carry</span><span class="p">[</span><span class="nv">$node_id</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span>
        <span class="s1">'nid'</span> <span class="o">=&gt;</span> <span class="nv">$node_id</span><span class="p">,</span>
        <span class="s1">'vid'</span> <span class="o">=&gt;</span> <span class="nv">$node_id</span><span class="p">,</span>
        <span class="s1">'type'</span> <span class="o">=&gt;</span> <span class="nv">$type</span><span class="p">,</span>
        <span class="s1">'language'</span> <span class="o">=&gt;</span> <span class="s1">'und'</span><span class="p">,</span>
        <span class="s1">'title'</span> <span class="o">=&gt;</span> <span class="s2">"Node #</span><span class="si">{</span><span class="nv">$node_id</span><span class="si">}</span><span class="s2"> title"</span><span class="p">,</span>
        <span class="s1">'uid'</span> <span class="o">=&gt;</span> <span class="mi">1</span><span class="p">,</span>
        <span class="s1">'status'</span> <span class="o">=&gt;</span> <span class="mi">1</span><span class="p">,</span>
        <span class="s1">'created'</span> <span class="o">=&gt;</span> <span class="mi">1600000000</span> <span class="o">+</span> <span class="nv">$node_id</span><span class="p">,</span>
        <span class="s1">'changed'</span> <span class="o">=&gt;</span> <span class="mi">1600000000</span> <span class="o">+</span> <span class="nv">$node_id</span><span class="p">,</span>
        <span class="s1">'comment'</span> <span class="o">=&gt;</span> <span class="mi">0</span><span class="p">,</span>
        <span class="s1">'promote'</span> <span class="o">=&gt;</span> <span class="mi">0</span><span class="p">,</span>
        <span class="s1">'sticky'</span> <span class="o">=&gt;</span> <span class="mi">0</span><span class="p">,</span>
        <span class="s1">'tnid'</span> <span class="o">=&gt;</span> <span class="mi">0</span><span class="p">,</span>
        <span class="s1">'translate'</span> <span class="o">=&gt;</span> <span class="mi">0</span><span class="p">,</span>
      <span class="p">];</span>
      <span class="k">return</span> <span class="nv">$carry</span><span class="p">;</span>
    <span class="p">},</span>
    <span class="p">[]</span>
  <span class="p">);</span>
   
  <span class="nv">$data_to_save</span> <span class="o">=</span> <span class="nb">array_reduce</span><span class="p">(</span>
    <span class="nv">$raw_data</span><span class="p">,</span>
    <span class="k">function</span> <span class="p">(</span><span class="kt">string</span> <span class="nv">$carry</span><span class="p">,</span> <span class="kt">array</span> <span class="nv">$data</span><span class="p">)</span> <span class="p">{</span>
      <span class="nv">$carry</span> <span class="mf">.</span><span class="o">=</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">-&gt;values("</span><span class="p">;</span>
      <span class="nv">$carry</span> <span class="mf">.</span><span class="o">=</span> <span class="nb">trim</span><span class="p">(</span><span class="nc">Variable</span><span class="o">::</span><span class="nf">export</span><span class="p">(</span><span class="nv">$data</span><span class="p">));</span>
      <span class="nv">$carry</span> <span class="mf">.</span><span class="o">=</span> <span class="s1">')'</span><span class="p">;</span>
      <span class="k">return</span> <span class="nv">$carry</span><span class="p">;</span>
    <span class="p">},</span>
    <span class="s2">"&lt;?php</span><span class="se">\n</span><span class="s2">// phpcs:ignoreFile</span><span class="se">\n</span><span class="s2">"</span>
  <span class="p">);</span>
  <span class="nv">$data_to_save</span> <span class="mf">.</span><span class="o">=</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">-&gt;execute();"</span><span class="p">;</span>
   
  <span class="nb">file_put_contents</span><span class="p">(</span><span class="nb">implode</span><span class="p">(</span><span class="no">DIRECTORY_SEPARATOR</span><span class="p">,</span> <span class="p">[</span>
    <span class="nf">drupal_get_path</span><span class="p">(</span><span class="s1">'module'</span><span class="p">,</span> <span class="s1">'migmag_menu_link_migrate'</span><span class="p">),</span>
    <span class="s1">'tests/fixtures'</span><span class="p">,</span>
    <span class="s1">'node-cleaned.php'</span>
  <span class="p">]),</span> <span class="nv">$data_to_save</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div>    </div>
  </li>
  <li>
    <p>And the last step was that I modified the method above to write a fixture for my super-clean <code class="language-plaintext highlighter-rouge">node_revisions</code> table (<code class="language-plaintext highlighter-rouge">node_revisions</code> has slightly different table structure), and re-ran the test.</p>
  </li>
</ol>

<h2 id="profit">Profit</h2>

<p>Basically, that was all. Although I did not dare to delete the original fixture file for a couple of hours, after a while I realized that if I still needed it, I could regenerate it anytime.</p>

<p>And I think I’ll finish this soonish! 🥳</p>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Drupal" /><category term="Migration" /><category term="Testing" /><category term="PHPUnit" /><summary type="html"><![CDATA[How TDD helped creating a minimal test database used for the enhanced menu link migration submodule of Migrate Magician.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/simple.png" /><media:content medium="image" url="https://huzooka.github.io/simple.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Migrating moderated Drupal 7 content to Drupal 9? There is a module for that!</title><link href="https://huzooka.github.io/development/2021/11/14/workbench-moderation-migrate.html" rel="alternate" type="text/html" title="Migrating moderated Drupal 7 content to Drupal 9? There is a module for that!" /><published>2021-11-14T12:06:43+00:00</published><updated>2021-11-14T12:06:43+00:00</updated><id>https://huzooka.github.io/development/2021/11/14/workbench-moderation-migrate</id><content type="html" xml:base="https://huzooka.github.io/development/2021/11/14/workbench-moderation-migrate.html"><![CDATA[<p class="lead">This August we got a <em>bug report</em> from a customer: They had an issue with the published state of their migrated nodes. Their complaint was that some of their nodes were migrated with the latest revision being active for anyone, while the correct visible revision should have been a previous node version.</p>

<p>We checked their installed Drupal 7 modules, and our suspicion was confirmed: the source site was using <a href="https://www.drupal.org/project/workbench_moderation">Workbench Moderation</a> configured to provide moderation workflows for some of their content types. And at the time we didn’t have any solutions for correctly migrating moderated content to Drupal 9.</p>

<p>I guess you won’t be too surprised if I say that Acquia decided to fund the development of moderated content types and their moderation states. This ended in <a href="https://www.drupal.org/project/workbench_moderation_migrate"><strong>Workbench Moderation Migrate</strong></a>, and I am telling you about how the moderation worked in Drupal 7 and how it works in Drupal 9, and what was the greatest challenge I had to solve in the migration paths.</p>

<h2 id="moderated-content">Moderated content</h2>

<p>The main feature of content moderation consists of the following things:</p>

<ul>
  <li>It provides <em>moderation states</em> a content can be in (<em>Draft</em>, <em>In Review</em>, <em>Published</em>, <em>Archived</em> etc.), and</li>
  <li><em>Transitions</em> between the states; and rules around when a transition can be used on a moderated content, and what state will be applied on the content if the transition is performed.</li>
</ul>

<p>This ecosystem allows you to have a published content version that is live and also have a separate working copy that is undergoes reviewing before it gets published, replacing the previous live content version.</p>

<h3 id="moderated-content-in-drupal7">Moderated content in Drupal 7</h3>

<p>Drupal 7 doesn’t have any support for “forward” (<a href="https://drupal.org/i/2890364">or “pending”</a>): it always checks the revision with the highest revision ID. And users always see (or don’t see) this most recent node version.</p>

<p><a href="https://www.drupal.org/project/workbench_moderation">Drupal 7 Workbench Moderation</a> works around this behavior. Whenever a user creates or edits a non-public draft of an already published node, the module saves a new, published revision which is the clone of the most recent published version. and since the last saved published revision has a higher revision ID than the “pending” version, users will not notice any change. (But in fact they will see a new revision.)</p>

<p>Workbench Moderation tracks the state transitions in its history database table, which contains an incremental history ID, an entity and an entity revision ID, the previous and the new state of the version, and the time when the transition happened.</p>

<p>An example how this works in Drupal 7:</p>

<table caption="Published node revision is #9, which is a clone of #7. But editors are editing #5 (this is what “current” means).">
  <thead>
    <tr>
      <th style="text-align: right">#</th>
      <th style="text-align: left">Action performed</th>
      <th style="text-align: right">Rev. ID</th>
      <th style="text-align: left">From</th>
      <th style="text-align: left">To</th>
      <th style="text-align: center">Public</th>
      <th style="text-align: center"><em>Current</em></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: right">1</td>
      <td style="text-align: left">New node, sent to review</td>
      <td style="text-align: right">1</td>
      <td style="text-align: left"> </td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: center">◯</td>
      <td style="text-align: center">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">2</td>
      <td style="text-align: left">Reviewer publishes</td>
      <td style="text-align: right">2</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">published</td>
      <td style="text-align: center">◯</td>
      <td style="text-align: center">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">3</td>
      <td style="text-align: left">User creates a new draft</td>
      <td style="text-align: right">3</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">draft</td>
      <td style="text-align: center">◯</td>
      <td style="text-align: center">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">4</td>
      <td style="text-align: left"><em>WBM clones rev. #2</em></td>
      <td style="text-align: right">4</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
      <td style="text-align: center">◯</td>
      <td style="text-align: center">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">5</td>
      <td style="text-align: left">Draft sent to review</td>
      <td style="text-align: right">5</td>
      <td style="text-align: left">draft</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: center">◯</td>
      <td style="text-align: center">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">6</td>
      <td style="text-align: left"><em>WBM clones rev #4</em></td>
      <td style="text-align: right">6</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
      <td style="text-align: center">◯</td>
      <td style="text-align: center">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">7</td>
      <td style="text-align: left">Pending draft published</td>
      <td style="text-align: right">7</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">published</td>
      <td style="text-align: center">◯</td>
      <td style="text-align: center">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">8</td>
      <td style="text-align: left">New draft</td>
      <td style="text-align: right">8</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">draft</td>
      <td style="text-align: center">◯</td>
      <td style="text-align: center">◉</td>
    </tr>
    <tr>
      <td style="text-align: right">9</td>
      <td style="text-align: left"><em>WBM clones rev #7</em></td>
      <td style="text-align: right">9</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
      <td style="text-align: center">◉</td>
      <td style="text-align: center">◯</td>
    </tr>
  </tbody>
</table>

<p><small>Published node revision is #9, which is a clone of #7. But editors are editing #5 (this is what “current” means).</small></p>

<h3 id="content-moderation-module-in-drupal9">Content Moderation module in Drupal 9</h3>

<p>Workflows and Content Moderation added to Drupal 8.2.x have clearer foundations: moderated workflows can track and maintain not only the moderation states of content revisions, but they can manage a flag on the revisions. This flag contains the info whether the version is a default revision or not. And Drupal’s Entity API is also aware of this. If an entity specific (and not entity revision specific!) read operation happens, Entity API checks the most recent revision flagged as being default. And this version’s state will determine whether the user can access the content, and if it is a public state, this will be the version they will see.</p>

<p>Here is the equivalent history data of the steps performed from the previous point on Drupal 9:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: right">#</th>
      <th style="text-align: left">Action performed</th>
      <th style="text-align: right">Rev. ID</th>
      <th style="text-align: left">From</th>
      <th style="text-align: left">To</th>
      <th style="text-align: right">Public</th>
      <th style="text-align: right">Default</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: right">1</td>
      <td style="text-align: left">New node, sent to review</td>
      <td style="text-align: right">1</td>
      <td style="text-align: left"> </td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: right">◯</td>
      <td style="text-align: right">◉</td>
    </tr>
    <tr>
      <td style="text-align: right">2</td>
      <td style="text-align: left">Reviewer publishes</td>
      <td style="text-align: right">2</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">published</td>
      <td style="text-align: right">◉</td>
      <td style="text-align: right">◉</td>
    </tr>
    <tr>
      <td style="text-align: right">3</td>
      <td style="text-align: left">User creates a new draft</td>
      <td style="text-align: right">3</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">draft</td>
      <td style="text-align: right">◯</td>
      <td style="text-align: right">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">4</td>
      <td style="text-align: left">Draft sent to review</td>
      <td style="text-align: right">4</td>
      <td style="text-align: left">draft</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: right">◯</td>
      <td style="text-align: right">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">5</td>
      <td style="text-align: left">Pending draft published</td>
      <td style="text-align: right">5</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">published</td>
      <td style="text-align: right">◉</td>
      <td style="text-align: right">◉</td>
    </tr>
    <tr>
      <td style="text-align: right">5</td>
      <td style="text-align: left">New pending draft</td>
      <td style="text-align: right">6</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: right">◯</td>
      <td style="text-align: right">◯</td>
    </tr>
  </tbody>
</table>

<p>Please note how this new “default” flag works: as long as there is no published version, all versions are “default”. It looks weird at first, but it totally makes sense: if a visitor tries to access the content early on, when only the draft revision #1 exists, Drupal should return a 403 response.</p>

<h2 id="how-can-we-move-this-from-drupal7-to-drupal9">How can we move this from Drupal 7 to Drupal 9?</h2>

<p>You may already have a clue what the solution is: we have to identify which node versions are the clones of a previous published revision, and skip their migration. <a href="https://git.drupalcode.org/project/workbench_moderation_migrate/-/blob/63f033bf/src/ModerationStateMigrate.php#L99-L113">And yes, this is its essence</a>, beside other nits – like being able to identify the first published revision. Unfortunately, it wasn’t this easy to solve.</p>

<p>But I had the first rule I set up: <strong>Clones of a published revision shouldn’t be migrated</strong>.</p>

<h2 id="the-biggest-challenges">The biggest challenges</h2>

<p>Drupal 7 Node + Workbench Moderation allows users to change the state of a revision and even to delete revisions on the revision UI<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>. It isn’t a big problem if a <em>published version clone</em> was deleted, but in every other case, we will have “holes” in the revision history, and this needs to be bridged somehow.</p>

<h3 id="states-changed-on-revision-ui">States changed on revision UI</h3>

<p>Why is it a problem that the state was changed on the revision UI? Because then Drupal does not create a new revision for the pending draft, so we will have more than one history record for the same revision. This is how it looks (watch out what happens between the sixth and eighth lines):</p>

<table>
  <thead>
    <tr>
      <th style="text-align: right">#</th>
      <th style="text-align: left">Action performed</th>
      <th style="text-align: right">Rev. ID</th>
      <th style="text-align: left">From</th>
      <th style="text-align: left">To</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: right">1</td>
      <td style="text-align: left">New node, sent to review</td>
      <td style="text-align: right">1</td>
      <td style="text-align: left"> </td>
      <td style="text-align: left">in_review</td>
    </tr>
    <tr>
      <td style="text-align: right">2</td>
      <td style="text-align: left">Reviewer approves and publishes</td>
      <td style="text-align: right">2</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">3</td>
      <td style="text-align: left">Published version was edited</td>
      <td style="text-align: right">3</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">4</td>
      <td style="text-align: left">New draft</td>
      <td style="text-align: right">4</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">draft</td>
    </tr>
    <tr>
      <td style="text-align: right">5</td>
      <td style="text-align: left"><em>WBM clones rev #3</em></td>
      <td style="text-align: right">5</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">6</td>
      <td style="text-align: left">New draft sent to review</td>
      <td style="text-align: right"><strong>6</strong></td>
      <td style="text-align: left">draft</td>
      <td style="text-align: left">in_review</td>
    </tr>
    <tr>
      <td style="text-align: right">7</td>
      <td style="text-align: left"><em>WBM clones rev #5</em></td>
      <td style="text-align: right"><strong>7</strong></td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">8</td>
      <td style="text-align: left">Reviewer sets back to draft on rev. UI</td>
      <td style="text-align: right"><strong>6</strong></td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">draft</td>
    </tr>
    <tr>
      <td style="text-align: right">9</td>
      <td style="text-align: left"><em>WBM clones rev #7</em></td>
      <td style="text-align: right"><strong>8</strong></td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">10</td>
      <td style="text-align: left">Draft sent back ro review</td>
      <td style="text-align: right">9</td>
      <td style="text-align: left">draft</td>
      <td style="text-align: left">in_review</td>
    </tr>
    <tr>
      <td style="text-align: right">11</td>
      <td style="text-align: left"><em>WBM clones rev #9</em></td>
      <td style="text-align: right">10</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">12</td>
      <td style="text-align: left">New draft reviewed and published</td>
      <td style="text-align: right">11</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">13</td>
      <td style="text-align: left">First published rev. is restored</td>
      <td style="text-align: right">12</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
    </tr>
  </tbody>
</table>

<p>This led me to set up the second rule that I applied when I was checking the <em>“from”</em> or the <em>“to” states</em> of a node revision: <strong><em>from state</em> should be fetched from the first history entry, <em>to state</em> should be fetched from the last history entry</strong> of a node revision.</p>

<h3 id="removed-revisions">Removed revisions</h3>

<p>I hope that it is obvious why it is a problem if revisions are deleted. Assume that only revision #4, #6 and #12 are available in the source. These are the only revisions which will be migrated by <code class="language-plaintext highlighter-rouge">d7_node_complete</code>. Just take a look at this history where I removed those history entries whose revision was deleted:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: right">#</th>
      <th style="text-align: left">Action performed</th>
      <th style="text-align: right">Rev. ID</th>
      <th style="text-align: left">From</th>
      <th style="text-align: left">To</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: right">4</td>
      <td style="text-align: left">New draft</td>
      <td style="text-align: right">4</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">draft</td>
    </tr>
    <tr>
      <td style="text-align: right">6</td>
      <td style="text-align: left">New draft sent to review</td>
      <td style="text-align: right">6</td>
      <td style="text-align: left">draft</td>
      <td style="text-align: left">in_review</td>
    </tr>
    <tr>
      <td style="text-align: right">8</td>
      <td style="text-align: left">Reviewer sets back to draft on rev. UI</td>
      <td style="text-align: right">6</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">draft</td>
    </tr>
    <tr>
      <td style="text-align: right">13</td>
      <td style="text-align: left">First published rev. is restored</td>
      <td style="text-align: right">12</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
    </tr>
  </tbody>
</table>

<p>How can we migrate these to the destination site by retaining the highest level of data integrity?</p>

<p>We can see the connection between the two non-published revision<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>. But how can we connect the last one’s <em>to state</em> with the last, published revision’s state? We cannot assume that there is a direct transition from <em>draft</em> to <em>published</em>!</p>

<p>In such cases, the solution is a bit sad: <strong>If there is no connection</strong> between the <em>first</em> drafts and the <em>first</em> published version, then we ask <code class="language-plaintext highlighter-rouge">d7_node_complete</code> to <strong>skip migrating the <em>stale</em> drafts</strong>. This was the third rule.</p>

<h3 id="unpublished-nodes-which-were-published-before">Unpublished nodes which were published before</h3>

<p>I want to show another tricky example<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: right">#</th>
      <th style="text-align: left">Action performed</th>
      <th style="text-align: right">Rev. ID</th>
      <th style="text-align: left">From</th>
      <th style="text-align: left">To</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: right">1</td>
      <td style="text-align: left">New draft sent to review</td>
      <td style="text-align: right">1</td>
      <td style="text-align: left"> </td>
      <td style="text-align: left">in_review</td>
    </tr>
    <tr>
      <td style="text-align: right">2</td>
      <td style="text-align: left">Reviewer publishes</td>
      <td style="text-align: right">2</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">3</td>
      <td style="text-align: left">New draft</td>
      <td style="text-align: right">3</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">draft</td>
    </tr>
    <tr>
      <td style="text-align: right">4</td>
      <td style="text-align: left"><em>WBM clones rev #2</em></td>
      <td style="text-align: right">4</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">published</td>
    </tr>
    <tr>
      <td style="text-align: right">5</td>
      <td style="text-align: left">Unpublished on revision UI</td>
      <td style="text-align: right">4</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">draft</td>
    </tr>
  </tbody>
</table>

<p>We can migrate rev #1 and #2, but what should we do with #3 and #4? I obviously cannot drop #4 – although it was a clone initially, it became a real transition because this was the published revision which <em>archived</em> the whole content. And on the other hand, if I didn’t migrate it, then the node would be published in Drupal 9, because revision #2 was public, and #3 is only a new, pending revision.</p>

<p>Well, I checked what happens on Drupal 9 if I repeat the same steps there, and there was a big relief: every single moderated content can be archived, even if it has a pending version.</p>

<p>So this transition log migrated into Drupal 9 looks like this :</p>

<table>
  <thead>
    <tr>
      <th style="text-align: right">#</th>
      <th style="text-align: left">Action performed</th>
      <th style="text-align: right">Rev. ID</th>
      <th style="text-align: left">From</th>
      <th style="text-align: left">To</th>
      <th style="text-align: right">Public</th>
      <th style="text-align: right">Default</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: right">1</td>
      <td style="text-align: left">New draft sent to review</td>
      <td style="text-align: right">1</td>
      <td style="text-align: left"> </td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: right">◯</td>
      <td style="text-align: right">◉</td>
    </tr>
    <tr>
      <td style="text-align: right">2</td>
      <td style="text-align: left">Reviewer publishes</td>
      <td style="text-align: right">2</td>
      <td style="text-align: left">in_review</td>
      <td style="text-align: left">published</td>
      <td style="text-align: right">◉</td>
      <td style="text-align: right">◉</td>
    </tr>
    <tr>
      <td style="text-align: right">3</td>
      <td style="text-align: left">New draft</td>
      <td style="text-align: right">3</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">draft</td>
      <td style="text-align: right">◯</td>
      <td style="text-align: right">◯</td>
    </tr>
    <tr>
      <td style="text-align: right">4</td>
      <td style="text-align: left">Content unpublished</td>
      <td style="text-align: right">4</td>
      <td style="text-align: left">published</td>
      <td style="text-align: left">archive</td>
      <td style="text-align: right">◯</td>
      <td style="text-align: right">◉</td>
    </tr>
  </tbody>
</table>

<hr />

<p><em>Drupal core issues discovered</em>:</p>

<ul>
  <li><a href="https://drupal.org/i/3200949#comment-14014993">#3200949-9</a>: Non-default entity revisions are migrated as default revision because EntityContentComplete does not allow creating forward (and non-default) revisions</li>
  <li><a href="https://drupal.org/i/3052115#comment-14296084">#3052115-32</a>: Mark an entity as ‘syncing’ during a migration ‘update’ and possibly test syncing semantics (no changed item bump, no content moderation revisions)</li>
  <li><a href="https://drupal.org/i/2329253#comment-14239586">#2329253-78</a>: Allow the ChangedItem to skip updating when synchronizing (f.e. when migrating)</li>
</ul>

<hr />

<p><em>Footnotes</em>:</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>In these cases, the moderation history table will still contain the history of the node, so we won’t have the revision data available. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2" role="doc-endnote">
      <p>Yes, there are <em>three</em> non-public table rows in the table, but two of them were recorded for the <em>same node revision ID</em>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3" role="doc-endnote">
      <p>This represents the typical history entries of the nodes we got the incident notice from the customer about. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Zoltán Horváth</name></author><category term="Development" /><category term="Drupal" /><category term="Migration" /><category term="Workbench Moderation" /><category term="Workflows" /><summary type="html"><![CDATA[Why the Workbench Moderation Migrate module works the way it does.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://huzooka.github.io/simple.png" /><media:content medium="image" url="https://huzooka.github.io/simple.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>