Description

Easy import approaches Ensembl data import from a file-based perspective, beginning with sequences and gff and optionally loading additional data directly from program output files.

Configuration

All parameters for both EasyMirror and EasyImport are set and passed to scripts through .ini files. The configuration file format mirrors the use of .ini files in ensembl-webcode, which has the benefit that a user who sets up an Ensembl instance using EasyImport will gain familiarity with the syntax conventions needed to further customise the instance. Use of configuration files rather than command line parameters also improves reproducibility since all parameters must be saved for the pipeline to be run.

Server setup

Setting up an Ensembl server to host locally imported data is essentially the same as setting up a mirror site (with just a couple of changes to config files) so the server setup and hosting steps simply reuse easy mirror.

Core import

The challenge for a file-based approach to data import is dealing with the fragmentation of the .gff format and the diverse ways that names, ids and descriptions can be captured both in .gff attributes and across a range of other file types. Most of the effort in designing easy import has been in creating a mechanism to allow gene models and annotations to be extracted from diverse, potentially "broken" files while maintaining a full and clear record of the import procedure to ensure full reproducibility.

One drawback to this approach is that separate configuration files are required for conceptually separate parts of the pipeline. This means that database connection data are repeated across multiple .ini files so care must be taken to use the correct template when making changes to the default settings.

Database creation

Consistency with the latest Ensembl database schema is ensured by importing the database schema for each new core database from the ensembl table.sql file and using an existing database with the same schema version as a template with the ensembl-production populate_production_db_tables.pl script.

Sequence data import

Scaffold/contig data will almost always be available as .fasta format (or possibly .fasta plus .agp). This is also the starting point for the Ensembl import pipeline so sequence import uses scripts directly from ensembl-pipeline.

Summary statistics

Basic summary statistics can be calculated at the beginning of the core import. A summary of .gff attribute counts by feature type is particularly useful in determining which fields may contain IDs, descriptions, etc. and whether any discrepancies in the counts of each attribute suggest that the .gff needs to be repaired by the gff parser.

Parsing and repairing GFF

Our flexible gff parser lies at the heart of easy import. It has been designed to embrace the diversity of real world .gff files by allowing full customisation of expected relationships and properties with functions to repair, warn or ignore errors during validation. gff parser is a perl module that provides a mechanism to assign expectations and validation rules to specific .gff feature types while having very little hard-coded awareness of any official gff specification allowing the flexibility to handle many more edge cases than other parsers. For easy import, a subset of the full functionality can be controlled through a meta-syntax in the core import .ini files and while the gff parser can accommodate most forms of .gff, including .gtf, easy import requires a .gff3 compatible format in column nine. This approach initially adds some complexity to the parameter specification, but many patterns can be reused across most .gff3 files and the benefit is that all modifications to the .gff3 can be preserved in the .ini file.

Gene model import

While the gff parser takes care of standardising .gff, additional functionality in easy import allows for retrieval of gene IDs, synonyms and descriptions from .gff attributes as well as from .fasta headers and simple text files. For a community resource like Lepbase, this provides the flexibility to easily incorporate information from diverse sources as supplied by individual labs rather than demanding a standardised format, which could act as a deterrent to full data sharing. For others implementing this pipeline, it offers the flexibility to integrate with existing protocols without the need to reformat data prior to import.

Import verification

In our experience, most problems with data import from .gff files can be detected through comparison of provider protein sequences with translations exported from an Ensembl database. easy import checks that the same IDs are present in each of these sets, and that the sequence lengths are identical. The most common causes of differences are alternate interpretations of phase ([0,1,2] vs. [0,2,1]), and manual editing of the provided sequences file to terminate translations at the first stop codon.

Additional annotations

Some xrefs can be imported via Dbxref attributes in a .gff file, however, we have deliberately limited the extent to which additional annotations can be imported from .gff due to the complexity of validating additional feature types and the complexity of mapping from potentially variable attribute names to specific fields in the core database tables. Several xref types can be richly represented in the Ensembl database if all required attributes are provided and this is easiest to ensure by working directly with program outputs. This also fits most closely with the Lepbase model of ensuring consistency across genomes from diverse sources through annotating features with consistent databases/parameters. easy import currently supports direct import of blastp, interproscan and repeatmasker output files.

File export

Sequence file export (scaffold, protein and cds) makes it simple to access the bulk data files for analysis or to provide bulk downloads. In addition to the basic formats, additional files are also exported for use with our comparative analysis pipeline. Detailed summary statistics can also be exported in .json format, which are used on ensembl.lepbase.org to populate the assembly statistics tables and in our assembly stats and codon usage visualisations.

Search indexing

Ensembl supports a very basic (direct MySQL) search out of the box, this is best replaced with a search plugin so we have made index_database.pl available to generate an autocomplete/search index compatible with the GenomeHubs gh-ensembl-plugin plugin.

The final round of summary statistic generation also produces template files for use on the species pages of an Ensembl instance, including our highly detailed assembly stats plots as used on ensembl.lepbase.org.

Compara import

A powerful feature of Ensembl is the integration of single and cross-species analyses in a single genome browser. At lepbase.org, we have recently finished implementing our own compara analysis for import using easy import and will add a full description of a containerised version of the analysis pipeline soon.