How to migrate content to Drupal 8 from a CSV file

When ordering a Drupal website from Omitsis, it’s very common that it’s an existing website and therefore has content that they want to migrate.

In Drupal there’s the migrate module that’s incorporated into the core and allows us to migrate content. It’s a very powerful, versatile module that facilitates performing rollbacks out of the box. Rollbacks allow us to “unimport”, very useful when we’re still developing and there are errors.

The migrate module allows importing from different sources: a Drupal 7, a database with whatever structure, from a csv, JSON, XML, SOAP, etc.

It’s very common for them to give us an excel with all the fields to migrate for each content type. In this post we’re going to explain how to perform this type of migration: from a csv, since all spreadsheets (excel for example) allow saving as csv and this is a better supported standard format.

Everything we’re going to explain is valid for Drupal 8 and for Drupal 9, since all modules are available for both versions except config_devel which seems to be coming soon and isn’t an essential module.

The first thing is to install the necessary modules. migrate is already in Drupal core but we’ll need some more: migrate_plus, migrate_tools, migrate_source_csv and config_devel.

We install these modules using composer:

composer require drupal/migrate_plus drupal/migrate_tools drupal/migrate_source_csv drupal/config_devel

And we activate them, with drush it would be like this:

drush -y en migrate migrate_plus migrate_tools migrate_source_csv config_devel

Now we have to create the yml files that will define the migrations. This we have to do within a custom module, which we may already have or we can create one specifically for migrate (cleaner option).

You can generate a module easily with the console, in our case we’ll call it custom_migrate.

drupal gm

In this module’s directory we create a subfolder called config and inside this another one called install. There we’ll put the yml files that will define the migrations.

The nomenclature of these files must be:

migrate_plus.migration.[MIGRATION_ID].yml

For example, if we want to import products the file name could be this:

migrate_plus.migration.products.yml

In this example the file would be in the following path:

web/modules/custom/custom_migrate/config/install/migrate_plus.migration.products.yml

The migrate_plus module also allows us to create groups, to then do imports all at once (and rollbacks). You can see in this post how to do it. In this example we won’t use them, but it’s quite practical.

Now we have to tell Drupal that it must take into account the import file we defined. That is put in the .info.yml of our module as follows:

config_devel:
  install:
    - migrate_plus.migration.products

The complete file would be like this:

name: 'Custom migrate'
type: module
description: 'Content import module'
core: 8.x
package: 'Custom'
dependencies:
  - migrate
  - migrate_plus
  - migrate_tools
  - config_devel
config_devel:
  install:
    - migrate_plus.migration.products
#   - migrate_plus.migration.other_import

We can do this thanks to the config_devel module, since without this module this configuration would only be set when installing the module, and it’s not practical to have to be installing and uninstalling.

Now we’re missing the most important thing: the content of the import yml. This would be enough for many posts, so let’s put a simple example:

First it’s necessary to define the id and a label will be good to give it a nice name.

id: products
label: Import products

Then comes the source section. Here we have to tell it which plugin we’ll use and for the csv case a path where the csv will be. In this case we’ve put it inside the module to be able to integrate it in our git.

We’ve also indicated the unique ids, which in our csv was called “code”.

And finally we indicate the csv fields.

source:
  plugin: 'csv'
  # Full path to the file.
  path: 'modules/custom/custom_migration/data/products.csv'
  # Column delimiter. Comma (,) by default.
  delimiter: ','
  # Field enclosure. Double quotation marks (") by default.
  enclosure: '"'
  # The row to be used as the CSV header (indexed from 0),
  # or null if there is no header row.
  header_offset: 0
  # The column(s) to use as a key. Each column specified will
  # create an index in the migration table and too many columns
  # may throw an index size error.
  ids:
    - code
  # Here we identify the columns of interest in the source file.
  # Each numeric key is the 0-based index of the column.
  # For each column, the key below is the field name assigned to
  # the data on import, to be used in field mappings below.
  # The label value is a user-friendly string for display by the
  # migration UI.
  fields:
    0:
      name: cat
      label: 'Category'
    1:
      name: code
      label: 'Product code'
    2:
      name: title
      label: 'Title'
    3:
      name: image
      label: 'Image'

Now the process part, where we indicate the fields where to put the data, the plugin we’re going to use and the source.

If it’s something very simple it could be as easy as this:

field_field_name_in_our_drupal: source_column_name

But often it’s not that simple. For example they give us the title all in uppercase, so we combine several plugins. You’ll also see the part of importing to a media field. We’ll explain this in another post soon.

A real case could be this:

process:
  type:
    plugin: default_value
    default_value: producto
  field_cat:
    plugin: entity_lookup
    source: division
    value_key: name
    bundle_key: vid
    bundle: cat_producto
    entity_type: taxonomy_term
    ignore_case: true
  field_producto_codigo: code
  title:
    -
      source: title
      plugin: callback
      callable: mb_strtolower
    -
      plugin: callback
      callable: ucfirst
  field_imagen/target_id:
    plugin: entity_lookup
    value_key: name
    source: image
    bundle_key: bundle
    bundle: image
    entity_type: media
    ignore_case: 1
    access_check: 0

There are many plugins available, you can see them here. Additionally we can create plugins very easily.

Now we only need to indicate where all this content will go with destination. It will depend on the entity we’re importing, in this case they are nodes.

destination:
  plugin: entity:node

And here it is complete:

id: products
label: Import products

source:
  plugin: 'csv'
  # Full path to the file.
  path: 'modules/custom/custom_migration/data/products.csv'
  # Column delimiter. Comma (,) by default.
  delimiter: ','
  # Field enclosure. Double quotation marks (") by default.
  enclosure: '"'
  # The row to be used as the CSV header (indexed from 0),
  # or null if there is no header row.
  header_offset: 0
  # The column(s) to use as a key. Each column specified will
  # create an index in the migration table and too many columns
  # may throw an index size error.
  ids:
    - code
  # Here we identify the columns of interest in the source file.
  # Each numeric key is the 0-based index of the column.
  # For each column, the key below is the field name assigned to
  # the data on import, to be used in field mappings below.
  # The label value is a user-friendly string for display by the
  # migration UI.
  fields:
    0:
      name: cat
      label: 'Category'
    1:
      name: code
      label: 'Product code'
    2:
      name: title
      label: 'Title'
    3:
      name: image
      label: 'Image'

process:
  field_cat:
    plugin: entity_lookup
    source: division
    value_key: name
    bundle_key: vid
    bundle: cat_producto
    entity_type: taxonomy_term
    ignore_case: true
  field_producto_codigo: code
  title:
    -
      source: title
      plugin: callback
      callable: mb_strtolower
    -
      plugin: callback
      callable: ucfirst
  type:
    plugin: default_value
    default_value: producto
  field_imagen/target_id:
    plugin: entity_lookup
    value_key: name
    source: image
    bundle_key: bundle
    bundle: image
    entity_type: media
    ignore_case: 1
    access_check: 0

destination:
  plugin: entity:node

migration_dependencies: {}

Now finally we only have to execute the commands from drush to perform the import. It can also be done from UI in admin/structure/migrate.

The first thing is to tell drupal to load all import files with the config_devel module:

drush cdi [module_name]

Which in our case would be:

drush cdi custom_migrate

Then we can see the migrations we have and their status with:

drush ms

To perform the import:

drush mim [import_id]

Which in our case would be:

drush mim products

If something goes wrong, which normally happens, we can do a rollback with this command:

drush mr products

It’s also possible that the migration gets stuck, in which case first before doing the rollback we have to put it in idle:

drush mrs products

And this is more or less everything, as an introduction to a migration with Drupal from a csv using the migrate module.