Services #

💡 You can replace the default values of the commands for PROJECT, STAGE and APPNAME in the form on the right side! A standard pattern might be myproject, production and web (see concepts).

A Service is an additional component such as a PostgreSQL. Services provide additional functionality and can be linked to be used in Apps.

Services Overview #

Services are configured with the service field in the stage definition.

...
services:
  # The name for services must only contain lowercase letters a-z and numbers 0-9 and dashes -. The name must be between 3 and 18 characters long and start with a lowercase letter.
  my_db1:
    type: <TYPE>
    plan:  <PLAN>
    version: <VERSION>
    backup_retention_days: <BACKUP_RETENTION_DAYS>
    options: ... # options depend on the service
...

We currently support the Service types described in the table below. Refer to the details section for each Service type for specific information on how to create and use this Service.

Deleting a service by removing it from the stage definition is a destructive action. When you apply the changes you will lose access to the database and all data/objects in it. Think carefully before deleting a Service!

If you accidentally deleted the service, please reach out to us as we MAY be able to recover data – but this is just a last resort.

Configure Services #

You can configure your Service with Links and Options.

Manage Links for a Service #

A Service must be linked to an App to make use of it. Links can be created in two ways, depending on the service type:

• Environment Variable: the App is provided with an Environment Variable describing how to access the service
• Path: the service (volume) gets mounted into the App container at a user-specified path

To find out, which to choose for a specific service, see the services overview table. The specifics of all supported Service types are discussed in the Service Specifics section below.

Link with Environment Variable #

apps:
  my_app:
    ...
    links:
      my_db1:
        env_key: MY_DB1_CONNECTION_URL
    ...
services:
  my_db1:
    type: <TYPE>
    plan:  <PLAN>
    version: <VERSION>
    options: ...

For services with an environment variable link, you can connect the service to an app via a link object with env_key. The env_key is the name of the environment variable which contains the connection information.

Link with Path #

apps:
  my_app:
    ...
    links:
      my_storage:
        path: /data
        read_only: false
    ...
services:
  my_storage:
    type: volume

For services with a path, you can connect the service to an app via a link object with path and optionally with read_only (which defaults to false). path is the path in the container where the service is mounted.

Manage Backup retention for a Service #

Backups for Services are managed automatically. The retention time is configurable, with a maximum time of 3650 days (10 years) and a minimum of 0 days, which will turn off automatic backups. Depending on the type of service, the maximum retention days might be less than the configurable maximum value, in that case the maximum of both values is chosen. The default retention time is 14 days.

...
services:
  database:
    ...
    backup_retention_days: 30

This configures the database service to keep backups for 30 days.

Service Specifics #

PostgreSQL #

The postgresql Service provides a PostgreSQL database with a dedicated user. The Service is linked with environment variables.

Your database user will own the database and all objects within the database.

Plan #

With the shared plan (currently only supported for type postgresql and version 11), this database is provisioned in a database cluster shared between Apps. All databases are created with a unique database user to ensure data separation. Using the shared plan is a cost-efficient way to add a database to an App.

If you require a dedicated database instance for compliance, performance or reliability, the type must currently be set to postgresql and to version 13 and the plan value can be set to any valid RDS PostgreSQL node type. db.t3.micro is a sensible default for small workloads. To enable the Multi-AZ feature, which keeps a hot standby instance in a second availability zones for higher reliability, add .ha at the end of the node type: e.g. db.t3.micro.ha for the Multi-AZ version of db.t3.micro. Using plans with .ha doubles the cost of the database instance.

Version #

For dedicated instances (when plan does not equal shared) can choose between the versions. A major and minor version must be provided, e.g. 15.2. See all available minor versions for version 13, 14 & 15.

You can (and should) upgrade to the latest version on a regular basis. When updating by changing the version please check the migration path table. An update will cause a downtime of the service for a few minutes.

Link Service to App #

To use the database, your app needs to know the credentials and the hostname of the database. SetOps uses Environment Variables to provide your App with this information.

The format is as follows:

DATABASE_URL=postgresql://username:password@shared.something.eu-central-1.rds.amazonaws.com:5432/databasename

Configure extensions for your database #

The PostgreSQL service supports user-configurable database extensions. See AWS' RDS documentation for a complete list. Extensions are configured as Service options:

...
services:
  database:
    type: postgresql
    ...
    options:
      extensions:
        - uuid-ossp
        - postgis

You can update the list of enabled extensions later as well. If your database is still dependent on a particular extension, it may not be possible for it to be deactivated by removing it from the list. The stage:apply would simply fail. In the future, we will implement a feature to let you know about this in advance.

Configure storage (non-shared plans only) #

When using a dedicated database instance (plan is not shared), the following options are available:

...
services:
  database:
    type: postgresql
    ...
    options:
      min_storage: 20
      max_storage: 1000
      storage_iops: 3000
      storage_throughput: 125

min_storage sets the amount of storage in gigabytes. The minimum and default is currently 20 gigabytes. You can set this option to a higher value whenever you need more space.

You can also activate storage autoscaling by setting max_storage to a higher value than min_storage. With storage autoscaling enabled, when the database instance is running out of free space it automatically scales up your storage.

Storage modification for an autoscaling-enabled database instance is triggered when these factors apply:

• Free available space is less than 10 percent of the allocated storage.
• The low-storage condition lasts at least five minutes.
• At least six hours have passed since the last storage modification.

The additional storage is in increments of whichever of the following is greater:

• 5 GiB
• 10 percent of currently allocated storage
• Storage growth prediction for 7 hours change in the past hour.

To receive alerts when PostgreSQL runs out of storage, make sure to set up notifications properly.

For storage_iops and storage_throughput see the AWS GP3 volume performance documentation.

S3 Object Store #

The s3 Service provides an S3-API compatible object store bucket. The plan or version value is not required for this service. SetOps will create a dedicated user authorized for all API actions for the bucket. The Service is linked with environment variables.

...
services:
  storage:
    type: s3
    options:
      blockPublicAccess: true

Block Public Access #

By default all objects within the bucket will have their Access Control List (ACL) set to private, i.e., the objects will not be able to be read or modified with unauthorized, public requests. You can set the option blockPublicAccess (defaults to true) to false to allow setting public object access via ACL. Please make sure, that you really want to do this since it increases the risk of exposing private data. Bucket level ACL modification is not supported.

Link Service to App #

To use the s3 service, your app needs to know the credentials and the hostname of the bucket. SetOps uses Environment Variables to provide your App with this information.

The format is as follows:

S3_DATA_URL=s3://ACCESS_KEY_URL_ENCODED:SECRET_KEY_URL_ENCODED@bucket-name?region=eu-central-1&endpoint=https%3A%2F%2Fs3.eu-central-1.amazonaws.com

The AWS Access Key ID (the username), and the AWS Secret Access Key (the password) are URI-encoded because they may contain special characters. Make sure to decode them in your app.

In Ruby this can be done with CGI.unescape:

s3_url = URI.parse(ENV['S3_DATA_URL'])
secret_access_key = CGI.unescape(s3_url.password)

Redis #

The redis Service provides a Redis instance. The plan value can be set to any valid ElastiCache for Redis Node Type, but cache.t3.micro is a sensible default. Currently only version 6 is supported. The Service is linked with environment variables.

...
services:
  cache:
    type: redis
    plan: <PLAN>
    version: 3

SetOps will create a dedicated Redis instance for the service and authorize your user for access to that instance. Note that your App will need to use the AUTH command when it connects to the instance.

In-Transit Encryption (TLS) will be activated for the Redis instance. The Environment Variable injected into your App by the Service’s Link will have the rediss:// protocol to reflect this (unencrypted Redis connections use redis://).

Link Service to App #

To use the redis service, your app needs to know the credentials and the hostname of the database. SetOps uses Environment Variables to provide your App with this information.

The format is as follows:

REDIS_URL=rediss://:AUTHTOKEN@master.parkscheibe-staging-somenumber.something.euc1.cache.amazonaws.com:6379

master.parkscheibe-staging-somenumber.something.euc1.cache.amazonaws.com is the hostname you need to configure your Redis API client to use. The AUTH token is encoded in the password value of the Environment Variable.

Version #

For this service you can choose between the available versions 6 and 7. A major and minor version must be provided, e.g. 6.2.

You can (and should) upgrade to the latest version on a regular basis by increasing the version in the definition. An update will cause a downtime of the service for a few minutes.

Volume #

The volume Service provides a persistent storage volume. The volume is created when you create the Service, and it can be used by any number of apps simultaneously. The Service is linked with a path.

...
services:
  cache:
    type: volume

Link Service to App #

To use the redis service, your App needs to be linked with the Service so that it becomes available at a path you specify. The Service is linked with a path.

File Permissions #

All files and directories on the volume will per default be owned by root (UID 0), not by the user who created them. However, any user can still create, modify, and delete files and directories on the volume. This might seem confusing when you’re working with the volume as it is not how files and directories behave on your local filesystem. However, it allows for maximum interoperability since you don’t need to configure the users and groups used by your App images.

Note that the term “user” in this section refers to an UNIX user account, not a SetOps account.

This is how it looks from a terminal. /vol is the mount path of a volume service.

user@host:/vol$ touch test
user@host:/vol$ ls -al
total 12
drwxr-xr-x 2 root root 6144 Apr 27 06:08 .
drwxr-xr-x 1 root root 4096 Apr 27 06:07 ..
-rw-r--r-- 1 root root    0 Apr 27 06:08 test

You can still use chown, chmod etc. to change file and directory permissions, which will be persisted across container restarts. However, they are not verified and any user may read and write to the volume.

Effectively, this means that all Apps which link with a Volume Service gain full access to the linked volume. You can use the --read-only flag to prohibit a certain app from writing to the volume.

Elasticsearch #

The elasticsearch Service provides an Elasticsearch cluster with version 7.

...
services:
  search:
    type: elasticsearch
    plan: <PLAN>
    version: 7

Plan #

The value for plan is composed of three to four parts:

• First, the instance type (as in hardware generation, e.g. t3).
• Second, the instance size (as in CPU and RAM resources, e.g. medium).
• Third, elasticsearch.
• Last, a modifier for an optional high-availability setup (e.g. sharded or ha).

This a valid plan value: t3.medium.elasticsearch. For highly-available deployment, you could use t3.medium.elasticsearch.ha.

For most basic use cases, t3.small.elasticsearch will do the job just fine. Using high-availability deployments is really justified only if your App is also developed and deployed with high availability in mind. In all other cases, you will just end up paying for more resources than your App can practically make use of.

Sharded setups deploy Elasticsearch Service instances in two Availability Zones (rather than one). Highly-available (ha) configurations utilize three Elasticsearch Master Nodes in different Availability Zones.

Storage #

The Elasticsearch service is provisioned with 10 GB of storage initially. You can set the storage option to increase the size:

...
services:
  search:
    type: elasticsearch
    plan: <PLAN>
    version: 7
    options:
      storage: 20
      storage_iops: 3000
      storage_throughput: 125

Note that the valid range for the storage option is between 1 GB and 1024 GB.

For storage_iops and storage_throughput see the AWS GP3 volume performance documentation.

Link Service to App #

To use the service, your App needs to be linked with the Service so that it knows the credentials for the Elasticsearch service. SetOps uses environment variables to provide your App with this information.

The format is as follows:

https://user:password@somehost.eu-central-1.es.amazonaws.com:443

Your App needs to authenticate itself using HTTP Basic Auth with the credentials from the Environment Variable. The traffic to the Elasticsearch Service needs to be TLS-encrypted (https).

DocumentDB #

The documentdb Service with version 4 provides an AWS DocumentDB (with MongoDB Compatibility) cluster. The SetOps DocumentDB service targets MongoDB API version 4.0.

AWS DocumentDB provides a subset of the full MongoDB API. This means that there are certain differences on how MongoDB and DocumentDB works, and which APIs it supports:

• There are some functional differences between DocumentDB and MongoDB.
• DocumentDB supports a subset of MongoDB APIs.
• DocumentDB enforces different naming constraints than MongoDB.

This means you need to verify that your application works with DocumentDB. When it does, DocumentDB provides certain benefits:

• It’s a highly-available, fault-tolerant, and scalable service.
• It has a lower cost, compared to other hosted MongoDB offerings.

We plan to introduce an additional SetOps Service for full MongoDB compatibility in a future SetOps version.

To create a DocumentDB cluster, set up the Service:

...
services:
  search:
    type: documentdb
    plan: <PLAN>
    version: 4

Plan #

See the table below for valid plan values.

For most basic use cases, db.t3.medium will do the job just fine. You can control the level of reliability (i.e., high availability) with the scale option

Scale #

Initially, the MongoDB cluster is created with one replica, which is also the primary instance. You can set the scale option to any number between 0 and 14. Any number greater than 1 will provision a multi-node cluster with one primary instance and n-1 replicas. The follower instances can be used to increase read throughput. The endpoint your App receives is the Cluster endpoint, which allows the MongoDB client to discover all instances in the cluster. You can specify the read behavior by configuring the MongoDB client (Read Preference).

...
services:
  search:
    type: documentdb
    plan: <PLAN>
    version: 4
    options:
      scale: 2

Link Service to App #

To use the service, your App needs to be linked with the Service so that it knows the credentials for the DocumentDB service. SetOps uses environment variables to provide your App with this information.

The format is as follows:

mongodb://username:password@somehost.eu-central-1.docdb.amazonaws.com:27017/?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false

Your App needs to authenticate itself with the credentials from the Environment Variable.

In-Transit Encryption #

Traffic to the MongoDB Service needs to be encrypted. The CA certificates for the encrypted connection can be downloaded from https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem. The path to the downloaded CA bundle needs to be specified in the ssl_ca_cert client option.

For example, add a line in the Dockerfile used to build your App image:

ADD https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem /app/rds-combined-ca-bundle.pem

Next, you include the certificate bundle the MongoDB Ruby client initialization:

mongo = Mongo::Client.new(ENV['MONGODB_URL'], ssl_ca_cert: '/app/rds-combined-ca-bundle.pem')

For Java apps, the rds-combined-ca-bundle.pem does not work since it contains two certs with the same alias. Instead, use the RDS root certificate. Add this to your Dockerfile:

# the combined bundle does not work with Java since it contains two certs with the same alias
ADD https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem /tmp/rds-ca-2019-root.pem

RUN $JAVA_HOME/bin/keytool \
    -import \
    -trustcacerts \
    -keystore $JAVA_HOME/lib/security/cacerts \
    -storepass changeit \
    -noprompt \
    -file /tmp/rds-ca-2019-root.pem

Going further #

Monitor your App with Notifications.