Genvid Forum

Genvid Technologies

GenvidSDK 1.22 - New ConsulTemplate backwards-compatibility fix

API Changes for ConsulTemplateTool

We added the new class ConsulTemplate, which is a more
flexible API implementation of ConsulTemplateTool. However, this
change is not backwards-compatible.

The goal is to make using the most common cases simpler:

  • If your template has stored configurations in Consul or Vault, use the
    use_consul=True or use_value=True parameter, as needed.
  • The env parameter lets you inject environment variables in the
    environment consul-template will run in.
  • All other keyword-parameters are the options of the consul-template
    executable.
    # This example assumes we are working in a class that is a subclass
    # of ``ConsulTemplateTool``.
    # Before
    process = self.consul_template(template,
                                   stdout=PIPE,
                                   stderr=PIPE,
                                   log_level="debug",
                                   timeout=1,
                                   env=env)
    # After
    output = self.consul_template_once(use_consul=True,
                                       use_vault=True,
                                       env=env,
                                       template=template,
                                       dry=True)

In this example, we use the template and dry options of the
underlying consul-template command directly. The
template value is the path to the templated configuration. The
dry value specifies that the resulting configuration is to be placed
inside output.configuration instead of writing it to disk.

Issue with Sample Files from Releases Prior to 1.20.0

This change to the API causes an issue with the sample scripts that predate
the 1.20.0 release. If you used those scripts as a base for your project, you
need to adjust them to work with the updated API.
Specifically, you will need to replace this block of code:

    try:
        process = self.consul_template(
            file_path,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            log_level="debug",
            cwd=self.base_dir,
            timeout=1,
            env=env)
    except subprocess.SubprocessError as ex:
        error_message = ex.stderr.decode()
        self.logger.error("Consul template has return an error:\n" +
                            error_message)
        raise
    output = process.stdout.decode()
    return hcl.loads(output[1:])

with the skd.load_config_template function we added for ease of use:

    return self.sdk.load_config_template(file_path, env=env)

If You Were Using Advanced Features of the Subprocess

If you were using the subprocess.CompletedProcess instance returns by
the previous API, you can use the previous implementation from the class
directly in your code. Include this code in your Python file:

    # Copyright Genvid Technologies 2016-2020
    from genvid.toolbox import VaultTool
    class ConsulTemplateTool(VaultTool):
        """A wrapper over consul-template that generates files from variables set
        in Consul, Vault, or from the environment.
        .. warning::
            This class is only used for starting the supervisor jobs. We do not
            recommend using it for any other purpose.
        """
        consul_template_filter = r"(?m)^.((\[DEBUG\] \(runner\) (final|checking|running|missing|diffing|was|watching|rendering|stopping))|(\[INFO\])).(\r\n|\n)"
        def _init_(self, **kwargs):
            super(ConsulTemplateTool, self)._init_(**kwargs)
        @property
        def CONSUL_TEMPLATE(self):
            "The consul-template binary location"
            return self.which("consul-template")
        def consul_template(self,
                            template,
                            dest=None,
                            *,
                            consul_addr=None,
                            log_level="warn",
                            need_vault: bool = False,
                            timeout=30,
                            **kwargs):
            """Build a new file from a template using consul-template.
            Args:
            template: The path of the template file.
            dest: The destination. If none, do a dry run instead.
            consul_addr: The Consul IP to use. If None, it will use
            get_consul_ip() on port 8500.
            log_level: The log level of consul-template. Valid values
            are 'debug', 'info', 'warn' and 'err'.
            timeout: A timeout value (in seconds) for consul-template
            to finish. Consul-template is blocking if a key is
            required but not present. This will kill the child process
            and raise a :class:subprocess.TimeoutExpired if the child
            process doesn't terminate in time. Set to None to wait
            indefinitely.
            need_vault: If consul template requires vault to run.
            .. versionadded:: 1.14.0
            kwargs: Any other keyword arguments pass to :func:~BaseTool.run.
            Note::
                Since absolute paths aren't well supported with
                consul-template on Windows, you must pass paths relative
                to the current working directory (:attr:ROOTDIR by
                default).
                A dry run always outputs a first line containing '> '.
            """
            if consul_addr is None:
                consul_addr = self.get_consul_ip() + ":8500"
            args = [
                self.CONSUL_TEMPLATE,
                "-consul-addr",
                consul_addr,
                "-log-level",
                log_level,
                "-once",
            ]
            if need_vault:
                args.extend(["-vault-renew-token=false"])
                self.refresh_token()
                if self.VAULT_TOKEN:
                    # TODO: Use a role defined for our services
                    self.auth_vault()
                    token = self.create_vault_token(ttl="1m")["client_token"]
                    args.extend(["-vault-token", token])
            if dest is None:
                args.extend(["-dry", "-template", template])
            else:
                args.extend(["-template", template + ":" + dest])
            kwargs.setdefault("cwd", self.ROOTDIR)
            return self.run(*args, timeout=timeout, **kwargs)