{"_id":"5735abf3eceb872200abbb85","category":{"_id":"5735b55beceb872200abbc6c","project":"5735936aafab441700723a50","version":"5735936aafab441700723a53","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-05-13T11:07:07.935Z","from_sync":false,"order":1,"slug":"configuration-options","title":"Stage 1 - Server Setup"},"project":"5735936aafab441700723a50","user":"573592b84b0ab120000b7d44","parentDoc":null,"__v":14,"version":{"_id":"5735936aafab441700723a53","__v":12,"project":"5735936aafab441700723a50","createdAt":"2016-05-13T08:42:18.615Z","releaseDate":"2016-05-13T08:42:18.615Z","categories":["5735936aafab441700723a54","5735a32931a73b1700887c94","5735b55beceb872200abbc6c","5735b56eb667601700d3bd6f","5735b9ba4b0ab120000b7dd4","5735b9c94b0ab120000b7dd5","5735cb131f16241700c8a0f7","5735e5c4e4824c3400aa1f21","5735e5d9e4824c3400aa1f23","5735e5f2ec67f6290013ac72","573ecfe0804f901700a9dfc7","573f276c7eeb8b190094ca7d"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":false,"codename":"","version_clean":"1.0.0","version":"1.0"},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2016-05-13T10:26:59.223Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":2,"body":"[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"Multiple .ini files\",\n  \"body\": \"All configuration options are stored in ``.ini`` files.  This ensures reproducibility as all options must be saved before a script is executed and avoids the need for numerous command-line flags.  Where practical, different scripts use common ``.ini`` files, reading only those parameters that are relevant.  However the options for easy import can be conceptually divided into four distinct groups, and it is convenient to keep the options for each of these groups in a separate ``.ini`` file:\\n- Server/Ensembl instance configuration\\n- Database hosting\\n- Core import (genome assembly/annotation specific data)\\n- Compara import (multiple assembly comparative data)\\n\\nThis means that database connection data are repeated across multiple ``.ini`` files so care must be taken to use the correct template when making changes to the default settings.\"\n}\n[/block]\nTo host an Ensembl mirror with remotely hosted data, at least one local database must be created with write access, to host additional data locally and to allow data import, additional users/databases must be created. These instructions assume that both the webserver and database are on ``localhost``. Use of separate hosts is supported (in which case this script may be run on a different host to the rest of Stage 1) but will require changes to  ``/etc/mysql/my.cnf`` to allow external connections.\n\n```\ncd ~/ei/em\n./setup-databases.sh ../conf/setup-db.ini\n```\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Configuration options\"\n}\n[/block]\n``setup-db.ini`` provides the following options:\n\n- [[DATABASE]](doc:database-db)\n```\n[DATABASE]\n    DB_USER = anonymous\n    DB_PASS =\n\n    DB_SESSION_USER = ensrw\n    DB_SESSION_PASS = ensrw\n\n    DB_IMPORT_USER = importer\n    DB_IMPORT_PASSWORD = importpassword\n\n    DB_ROOT_USER = root\n    DB_ROOT_PASSWORD = secretpassword\n    DB_PORT = 3306\n    DB_HOST = localhost\n```\nRoot user connection details and user names (and passwords) for database users to be created.  ``DB_USER`` has ``SELECT`` permissions only and will be used as the 'ro' user for the Ensembl instance. ``DB_SESSION_USER`` has permissions on the ``ensembl_accounts`` database and will be used as the 'rw' user for the Ensembl instance.  ``DB_IMPORT_USER`` has more extensive permissions on all databases and will be used during Core and Compara Import.\n \n- [[WEBSITE]](doc:website-db)\n```\n[WEBSITE]\n    ENSEMBL_WEBSITE_HOST = localhost\n```\n  The name of the ``ENSEMBL_WEBSITE_HOST`` host (on which [Step 1.1](doc:step-11-install-dependencies), etc. are run) is used when setting up the database users. If this is anything other than ``localhost`` then changes will be required to ``/etc/mysql/my.cnf`` to support external connections.\n\n- [[DATA_SOURCE]](doc:data-source-db)\n```\nDATA_SOURCE]\n    ENSEMBL_DB_URL = ftp://ftp.ensembl.org/pub/current_mysql/\n    ENSEMBL_DB_REPLACE =\n    ENSEMBL_DBS = [ ensembl_accounts ]\n\n    EG_DB_URL = ftp://ftp.ensemblgenomes.org/pub/current/pan_ensembl/mysql/\n    EG_DB_REPLACE = 1\n    EG_DBS = [ ncbi_taxonomy ensembl_website_84 ]\n\n    SPECIES_DB_URL = ftp://ftp.ensemblgenomes.org/pub/current/metazoa/mysql/\n    SPECIES_DB_REPLACE =\n    SPECIES_DB_AUTO_EXPAND =\n    SPECIES_DBS = [ bombyx_mori_core_31_84_1 ]\n\n    MISC_DB_URL =\n    MISC_DB_REPLACE =\n    MISC_DBS =\n```\n  Locations and names of database dumps to fetch and load locally.\n  - ``ENSEMBL_DB_URL`` - the URL containing the Ensembl database dumps\n  - ``ENSEMBL_DB_REPLACE`` - a flag to specify whether to overwrite databases that already exist on the ``DB_HOST``\n  - ``ENSEMBL_DBS`` - a space separated list of database dump names in square braces. ``ensembl_accounts`` is required, all others are optional\n  - The equivalent variables may be set for ``EG_DB_URL`` to fetch and download EnsemblGenomes database dumps and for ``MISC_DB_URL`` to support situations where the required databases are spread across multiple hosts.\n  - An additional variable may be set for species databases, ``SPECIES_DB_AUTO_EXPAND`` - a space separated list of database types to use as replacement strings for ``core`` to facilitate downloading multiple database types for each species in ``SPECIES_DBS``","excerpt":"","slug":"step-12-setup-database-connections","type":"basic","title":"Step 1.2: Setup database connections"}

Step 1.2: Setup database connections


[block:callout] { "type": "warning", "title": "Multiple .ini files", "body": "All configuration options are stored in ``.ini`` files. This ensures reproducibility as all options must be saved before a script is executed and avoids the need for numerous command-line flags. Where practical, different scripts use common ``.ini`` files, reading only those parameters that are relevant. However the options for easy import can be conceptually divided into four distinct groups, and it is convenient to keep the options for each of these groups in a separate ``.ini`` file:\n- Server/Ensembl instance configuration\n- Database hosting\n- Core import (genome assembly/annotation specific data)\n- Compara import (multiple assembly comparative data)\n\nThis means that database connection data are repeated across multiple ``.ini`` files so care must be taken to use the correct template when making changes to the default settings." } [/block] To host an Ensembl mirror with remotely hosted data, at least one local database must be created with write access, to host additional data locally and to allow data import, additional users/databases must be created. These instructions assume that both the webserver and database are on ``localhost``. Use of separate hosts is supported (in which case this script may be run on a different host to the rest of Stage 1) but will require changes to ``/etc/mysql/my.cnf`` to allow external connections. ``` cd ~/ei/em ./setup-databases.sh ../conf/setup-db.ini ``` [block:api-header] { "type": "basic", "title": "Configuration options" } [/block] ``setup-db.ini`` provides the following options: - [[DATABASE]](doc:database-db) ``` [DATABASE] DB_USER = anonymous DB_PASS = DB_SESSION_USER = ensrw DB_SESSION_PASS = ensrw DB_IMPORT_USER = importer DB_IMPORT_PASSWORD = importpassword DB_ROOT_USER = root DB_ROOT_PASSWORD = secretpassword DB_PORT = 3306 DB_HOST = localhost ``` Root user connection details and user names (and passwords) for database users to be created. ``DB_USER`` has ``SELECT`` permissions only and will be used as the 'ro' user for the Ensembl instance. ``DB_SESSION_USER`` has permissions on the ``ensembl_accounts`` database and will be used as the 'rw' user for the Ensembl instance. ``DB_IMPORT_USER`` has more extensive permissions on all databases and will be used during Core and Compara Import. - [[WEBSITE]](doc:website-db) ``` [WEBSITE] ENSEMBL_WEBSITE_HOST = localhost ``` The name of the ``ENSEMBL_WEBSITE_HOST`` host (on which [Step 1.1](doc:step-11-install-dependencies), etc. are run) is used when setting up the database users. If this is anything other than ``localhost`` then changes will be required to ``/etc/mysql/my.cnf`` to support external connections. - [[DATA_SOURCE]](doc:data-source-db) ``` DATA_SOURCE] ENSEMBL_DB_URL = ftp://ftp.ensembl.org/pub/current_mysql/ ENSEMBL_DB_REPLACE = ENSEMBL_DBS = [ ensembl_accounts ] EG_DB_URL = ftp://ftp.ensemblgenomes.org/pub/current/pan_ensembl/mysql/ EG_DB_REPLACE = 1 EG_DBS = [ ncbi_taxonomy ensembl_website_84 ] SPECIES_DB_URL = ftp://ftp.ensemblgenomes.org/pub/current/metazoa/mysql/ SPECIES_DB_REPLACE = SPECIES_DB_AUTO_EXPAND = SPECIES_DBS = [ bombyx_mori_core_31_84_1 ] MISC_DB_URL = MISC_DB_REPLACE = MISC_DBS = ``` Locations and names of database dumps to fetch and load locally. - ``ENSEMBL_DB_URL`` - the URL containing the Ensembl database dumps - ``ENSEMBL_DB_REPLACE`` - a flag to specify whether to overwrite databases that already exist on the ``DB_HOST`` - ``ENSEMBL_DBS`` - a space separated list of database dump names in square braces. ``ensembl_accounts`` is required, all others are optional - The equivalent variables may be set for ``EG_DB_URL`` to fetch and download EnsemblGenomes database dumps and for ``MISC_DB_URL`` to support situations where the required databases are spread across multiple hosts. - An additional variable may be set for species databases, ``SPECIES_DB_AUTO_EXPAND`` - a space separated list of database types to use as replacement strings for ``core`` to facilitate downloading multiple database types for each species in ``SPECIES_DBS``