{"_id":"5735ac021f16241700c8a098","user":"573592b84b0ab120000b7d44","parentDoc":null,"project":"5735936aafab441700723a50","version":{"_id":"5735936aafab441700723a53","__v":12,"project":"5735936aafab441700723a50","createdAt":"2016-05-13T08:42:18.615Z","releaseDate":"2016-05-13T08:42:18.615Z","categories":["5735936aafab441700723a54","5735a32931a73b1700887c94","5735b55beceb872200abbc6c","5735b56eb667601700d3bd6f","5735b9ba4b0ab120000b7dd4","5735b9c94b0ab120000b7dd5","5735cb131f16241700c8a0f7","5735e5c4e4824c3400aa1f21","5735e5d9e4824c3400aa1f23","5735e5f2ec67f6290013ac72","573ecfe0804f901700a9dfc7","573f276c7eeb8b190094ca7d"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":false,"codename":"","version_clean":"1.0.0","version":"1.0"},"__v":16,"category":{"_id":"5735b55beceb872200abbc6c","project":"5735936aafab441700723a50","version":"5735936aafab441700723a53","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-05-13T11:07:07.935Z","from_sync":false,"order":1,"slug":"configuration-options","title":"Stage 1 - Server Setup"},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2016-05-13T10:27:14.412Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":3,"body":"[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"Reusing this script\",\n  \"body\": \"The ``update-ensembl-code.sh`` script should be run again (as Step 3.1) following data import in Stage 2/Stage 3.  To set up a mirror site, continue directly to [Step 3.1: Update Ensembl webcode](doc:step-31-update-ensembl-webcode) for full details of additional configuration options.\"\n}\n[/block]\nRunning the ``update-ensembl-code.sh`` script will ``git clone``/``git pull`` a set of (mostly Ensembl) repositories required for Core/Compara Import and for hosting an Ensembl instance.\n```\ncd ~/ei/em\n./update-ensembl-code.sh ../conf/setup.ini\n```\n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"Testing your setup\",\n  \"body\": \"With the basic setup complete, you can now jump directly to [Step 3.2: Reload Ensembl website](doc:step-32-reload-ensembl-website) to launch a local Ensembl instance.\"\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Configuration options\"\n}\n[/block]\nTwo stanzas in ``setup.ini`` are relevant at this stage:\n\n- [[DATABASE]](doc:database)\n```\n[DATABASE]\n    DB_HOST = localhost\n    DB_PORT = 3306\n    DB_USER = anonymous\n    DB_PASS =\n\n    DB_SESSION_HOST = localhost\n    DB_SESSION_PORT = 3306\n    DB_SESSION_USER = ensrw\n    DB_SESSION_PASS = ensrw\n\n    DB_FALLBACK_HOST = mysql-eg-publicsql.ebi.ac.uk\n    DB_FALLBACK_PORT = 4157\n    DB_FALLBACK_USER = anonymous\n    DB_FALLBACK_PASS =\n\n    DB_FALLBACK2_HOST = ensembldb.ensembl.org\n    DB_FALLBACK2_PORT = 3306\n    DB_FALLBACK2_USER = anonymous\n    DB_FALLBACK2_PASS =\n```\n  Four subsections with ``DB_[*_]HOST``, ``DB_[*_]PORT``, ``DB_[*_]USER`` and ``DB_[*_]PASS`` variables specify connection settings for:\n  - ``DB_HOST`` etc. - the primary database host with species/multi-species databases.\n  - ``DB_SESSION_HOST`` etc. - user-specific information, typically the only database to require read-write access and therefore a password protected connection.\n  - ``DB_FALLBACK_HOST`` etc. - to reduce the amount of locally hosted data, it is often desirable to use alternate sources for some databases, the ``DB_FALLBACK_HOST`` host will be queried to find any required databases that are not available on ``DB_HOST``\n  - ``DB_FALLBACK2_HOST`` etc. - especially with EnsemblGenomes sites, remote databases may be found on more than one host, the ``DB_FALLBACK2_HOST`` host will be queried to find any required databases that are not available on ``DB_HOST`` or ``DB_FALLBACK_HOST``\n\n- [[REPOSITORIES]](doc:repositories)\n```\n[REPOSITORIES]\n    ENSEMBL_URL = https://github.com/Ensembl\n    ENSEMBL_BRANCH = release/84\n\n    BIOPERL_URL = https://github.com/bioperl\n    BIOPERL_BRANCH = master\n\n    EG_METAZOA_PLUGIN_URL = https://github.com/EnsemblGenomes/eg-web-metazoa\n    EG_METAZOA_PLUGIN_BRANCH = release/eg/31\n    EG_METAZOA_PLUGIN_PACKAGE = EG::Metazoa\n\n    API_PLUGIN_URL = https://github.com/EnsemblGenomes/ensemblgenomes-api\n    API_PLUGIN_BRANCH = release/eg/31\n    API_PLUGIN_PACKAGE = EG::API\n\n    EG_COMMON_PLUGIN_URL = https://github.com/EnsemblGenomes/eg-web-common\n    EG_COMMON_PLUGIN_BRANCH = release/eg/31\n    EG_COMMON_PLUGIN_PACKAGE = EG::Common\n\n    PUBLIC_PLUGINS = [ ]\n```\n  Connection/branch information for the Github repositories to be cloned\n  - ``ENSEMBL_URL``/``ENSEMBL_BRANCH`` and ``BIOPERL_URL``/``BIOPERL_BRANCH`` are always required.  Several Ensel]mbl repositories are ``git clone``d from ``ENSEMBL_URL`` for both the Ensembl website and to support the import pipeline.\n  - Additional plugins can be loaded from any public ``git`` repository by specifying ``<NAME>_PLUGIN_URL``, ``<NAME>_PLUGIN_BRANCH`` and ``<NAME>_PLUGIN_PACKAGE`` as above.  Plugins specified in this way will be added to ``ensembl-webcode/conf/Plugins.pm`` in the order they are listed in the ``.ini`` file so those at the top of the list will overwrite functions in plugins further down the list.\n  - ``PUBLIC_PLUGINS`` can be loaded from the Ensembl ``public-plugins`` repository by specifying the directory and package name of each repository as in ``PUBLIC_PLUGINS = [ ensembl|EnsEMBL::Ensembl genoverse|EnsEMBL::Genoverse ]``\n\nThe remaining section in ``setup.ini``, [[DATA_SOURCE]](doc:data_source), can also be edited at this stage - see [Step 4.1: Update Ensembl webcode](doc:step-41-update-ensembl-webcode) for details.","excerpt":"","slug":"step-13-git-clone-ensembl-repositories","type":"basic","title":"Step 1.3: git clone ensembl repositories"}

Step 1.3: git clone ensembl repositories


[block:callout] { "type": "info", "title": "Reusing this script", "body": "The ``update-ensembl-code.sh`` script should be run again (as Step 3.1) following data import in Stage 2/Stage 3. To set up a mirror site, continue directly to [Step 3.1: Update Ensembl webcode](doc:step-31-update-ensembl-webcode) for full details of additional configuration options." } [/block] Running the ``update-ensembl-code.sh`` script will ``git clone``/``git pull`` a set of (mostly Ensembl) repositories required for Core/Compara Import and for hosting an Ensembl instance. ``` cd ~/ei/em ./update-ensembl-code.sh ../conf/setup.ini ``` [block:callout] { "type": "info", "title": "Testing your setup", "body": "With the basic setup complete, you can now jump directly to [Step 3.2: Reload Ensembl website](doc:step-32-reload-ensembl-website) to launch a local Ensembl instance." } [/block] [block:api-header] { "type": "basic", "title": "Configuration options" } [/block] Two stanzas in ``setup.ini`` are relevant at this stage: - [[DATABASE]](doc:database) ``` [DATABASE] DB_HOST = localhost DB_PORT = 3306 DB_USER = anonymous DB_PASS = DB_SESSION_HOST = localhost DB_SESSION_PORT = 3306 DB_SESSION_USER = ensrw DB_SESSION_PASS = ensrw DB_FALLBACK_HOST = mysql-eg-publicsql.ebi.ac.uk DB_FALLBACK_PORT = 4157 DB_FALLBACK_USER = anonymous DB_FALLBACK_PASS = DB_FALLBACK2_HOST = ensembldb.ensembl.org DB_FALLBACK2_PORT = 3306 DB_FALLBACK2_USER = anonymous DB_FALLBACK2_PASS = ``` Four subsections with ``DB_[*_]HOST``, ``DB_[*_]PORT``, ``DB_[*_]USER`` and ``DB_[*_]PASS`` variables specify connection settings for: - ``DB_HOST`` etc. - the primary database host with species/multi-species databases. - ``DB_SESSION_HOST`` etc. - user-specific information, typically the only database to require read-write access and therefore a password protected connection. - ``DB_FALLBACK_HOST`` etc. - to reduce the amount of locally hosted data, it is often desirable to use alternate sources for some databases, the ``DB_FALLBACK_HOST`` host will be queried to find any required databases that are not available on ``DB_HOST`` - ``DB_FALLBACK2_HOST`` etc. - especially with EnsemblGenomes sites, remote databases may be found on more than one host, the ``DB_FALLBACK2_HOST`` host will be queried to find any required databases that are not available on ``DB_HOST`` or ``DB_FALLBACK_HOST`` - [[REPOSITORIES]](doc:repositories) ``` [REPOSITORIES] ENSEMBL_URL = https://github.com/Ensembl ENSEMBL_BRANCH = release/84 BIOPERL_URL = https://github.com/bioperl BIOPERL_BRANCH = master EG_METAZOA_PLUGIN_URL = https://github.com/EnsemblGenomes/eg-web-metazoa EG_METAZOA_PLUGIN_BRANCH = release/eg/31 EG_METAZOA_PLUGIN_PACKAGE = EG::Metazoa API_PLUGIN_URL = https://github.com/EnsemblGenomes/ensemblgenomes-api API_PLUGIN_BRANCH = release/eg/31 API_PLUGIN_PACKAGE = EG::API EG_COMMON_PLUGIN_URL = https://github.com/EnsemblGenomes/eg-web-common EG_COMMON_PLUGIN_BRANCH = release/eg/31 EG_COMMON_PLUGIN_PACKAGE = EG::Common PUBLIC_PLUGINS = [ ] ``` Connection/branch information for the Github repositories to be cloned - ``ENSEMBL_URL``/``ENSEMBL_BRANCH`` and ``BIOPERL_URL``/``BIOPERL_BRANCH`` are always required. Several Ensel]mbl repositories are ``git clone``d from ``ENSEMBL_URL`` for both the Ensembl website and to support the import pipeline. - Additional plugins can be loaded from any public ``git`` repository by specifying ``<NAME>_PLUGIN_URL``, ``<NAME>_PLUGIN_BRANCH`` and ``<NAME>_PLUGIN_PACKAGE`` as above. Plugins specified in this way will be added to ``ensembl-webcode/conf/Plugins.pm`` in the order they are listed in the ``.ini`` file so those at the top of the list will overwrite functions in plugins further down the list. - ``PUBLIC_PLUGINS`` can be loaded from the Ensembl ``public-plugins`` repository by specifying the directory and package name of each repository as in ``PUBLIC_PLUGINS = [ ensembl|EnsEMBL::Ensembl genoverse|EnsEMBL::Genoverse ]`` The remaining section in ``setup.ini``, [[DATA_SOURCE]](doc:data_source), can also be edited at this stage - see [Step 4.1: Update Ensembl webcode](doc:step-41-update-ensembl-webcode) for details.