collective.recipe.template 2.2
Buildout recipe to generate a text file from a template
License: BSD
Keywords: template recipe

Introduction
************

This recipe can be used to generate textfiles from a (text)
template.

.. contents::

A short example::

  [buildout]
  parts = message

  [message]
  recipe = collective.recipe.template
  input = templates/message.in
  output = ${buildout:parts-directory}/etc/message

  mymessage = Hello, World!


In the template you can use the exact same variables as you can use
in the buildout configuration. For example an input file can look like this::

  My top level directory is ${buildout:directory}
  Executables are stored in ${buildout:bin-directory}


As an extension to the buildout syntax you can reference variables from
the current buildout part directly. For example::

  My message is: ${mymessage}


Features
========

* Starting with version 1.3, you can also specify a path to the output
  file and the path will be created if it does not exist.
* Starting with version 1.5, you can use inline templates.
* Starting with version 1.7, you can use `genshi text templates`_.
* Starting with version 1.9, you can use a URL to specify template input.
* Starting with version 1.12, you can specify ``timeout`` as an option to
  configure ``urllib2`` requests.
* Starting with version 2.1, you can set ``input-encoding`` and
  ``output-encoding`` to specify the character encoding.

Genshi text templates
---------------------

A short example::

  [buildout]
  parts = message

  [message]
  recipe = collective.recipe.template[genshi]:genshi
  input = templates/message.in
  output = ${buildout:parts-directory}/etc/message
  some-option = value

  mymessage = Hello, World!

In the template you can use the exact same variables as you can use
in the buildout configuration, but instead of colons as the separator you
either have to use attribute access, or for options with a dash dictionary
syntax. The global buildout config is accessible through ``parts``, the
current part through ``options``.

For example an input file can look like this::

  My top level directory is ${parts.buildout.directory}
  Executables are stored in ${parts.buildout['bin-directory']}
  Accessing the current part: ${options['some-option']}


Why another template recipe?
============================

Both `iw.recipe.template`_ and `inquant.recipe.textfile`_ claim to do the
same thing. I have found them to be undocumented and too buggy for real
world use, and neither are in a public repository where I could fix them. In
addition this implementation leverages the buildout variable substitution
code, making it a lot simpler.


.. _genshi text templates: http://genshi.edgewall.org/wiki/Documentation/text-templates.html
.. _iw.recipe.template: http://pypi.python.org/pypi/iw.recipe.template
.. _inquant.recipe.textfile: http://pypi.python.org/pypi/inquant.recipe.textfile


Detailed Description
********************

Simple creation of a file out of a template
===========================================

Lets create a minimal `buildout.cfg` file::

  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... parts = template
  ... offline = true
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... input = template.in
  ... output = template
  ... ''')

We create a template file::

  >>> write('template.in',
  ... '''#
  ... My templåte knows about buildout path:
  ...   ${buildout:directory}
  ... ''')

Now we can run buildout::

  >>> print(system(join('bin', 'buildout')))
  Installing template.

The template was indeed created::

  >>> cat('template')
  #
  My templåte knows about buildout path:
  .../sample-buildout

The variable ``buildout:directory`` was also substituted by a path.

Overriding output file
======================

By default re-execute buildout, makes that output file is overwrited, by new
output file. But, if you want generate this file ONLY when it doesn't exist,
you can use overwrite option:

Once again check output file content::

  >>> cat('template')
  #
  My templåte knows about buildout path:
  .../sample-buildout

Let's change this file::
  >>> print(system("sed 's/sample-buildout/spam-ham-eggs/g' template > out && mv out template"))
  <BLANKLINE>

Let's check content now::

  >>> cat('template')
  #
  My templåte knows about buildout path:
  .../spam-ham-eggs

Now try re-execute buildout, and then check our file again::

  >>> print(system(join('bin', 'buildout')))
  Updating template.

  >>> cat('template')
  #
  My templåte knows about buildout path:
  .../sample-buildout

Like you see, re-execute buildout, caused overwrite ourmodified file. Let's try
to prevent this behavior. So we must modify buildout.cfg, re-execute buildout,
and then modify again output file::

  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... parts = template
  ... offline = true
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... input = template.in
  ... output = template
  ... overwrite = False
  ... ''')

  >>> print(system(join('bin', 'buildout')))
  Uninstalling template.
  Installing template.

  >>> cat('template')
  #
  My templåte knows about buildout path:
  .../sample-buildout

  >>> print(system("sed 's/sample-buildout/spam-ham-eggs/g' template > out && mv out template"))
  <BLANKLINE>

  >>> cat('template')
  #
  My templåte knows about buildout path:
  .../spam-ham-eggs

Let's check output file again - it shouldn't be modyfied this time::

  >>> print(system(join('bin', 'buildout')))
  Updating template.

  >>> cat('template')
  #
  My templåte knows about buildout path:
  .../spam-ham-eggs

Using inline input
==================

For very short script it can make sense to put the source directly into
`buildout.cfg`::

  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... parts = template
  ... offline = true
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... input = inline:
  ...    #!/bin/bash
  ...    echo foo
  ... output = ${buildout:parts-directory}/template
  ... ''')

Now we can run buildout::

  >>> print(system(join('bin', 'buildout')))
  Uninstalling template.
  Installing template.

The template should have been created::

  >>> cat('parts', 'template')
  #!/bin/bash
  echo foo

Normally the file mode gets copied from the template, but it can also be
specified manually, which especially makes sense in this case:

  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... parts = template
  ... offline = true
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... inline =
  ...    #!/bin/bash
  ...    echo foo
  ... output = ${buildout:parts-directory}/template
  ... mode = 755
  ... ''')

Run buildout again ::

  >>> print(system(join('bin', 'buildout')))
  Uninstalling template.
  Installing template.

The template should have the specified file mode::

  >>> from os import stat
  >>> from stat import S_IMODE
  >>> print('%o' % S_IMODE(stat('parts/template').st_mode))
  755

Using URL input
===============

.. Warning:: There is a security risk inherent with using URL input.
    Please be careful.

Similarly, you may want to read input from a URL, e.g.::

  >>> import os
  >>> tmpfn = os.path.abspath(join('template.in'))
  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... parts = template
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... url = file://%s
  ... output = template
  ... ''' % tmpfn)

To demonstrate this, first we create a template file::

  >>> write(tmpfn,
  ... '''#
  ... My templåte knows about buildout path:
  ...   ${buildout:directory}
  ... ''')

Now we can run buildout::

  >>> lines = system(join('bin', 'buildout')).splitlines()
  >>> lines = [x for x in lines if not x.startswith('Not found:')]
  >>> print('\n'.join(lines))
  Uninstalling template.
  Installing template.

The template should have been created::

  >>> cat('template')
  #
  My templåte knows about buildout path:
    /sample-buildout

Creating a template in a variable path
======================================

Lets create a minimal `buildout.cfg` file. This time the output should
happen in a variable path::

  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... parts = template
  ... offline = true
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... input = template.in
  ... output = ${buildout:parts-directory}/template
  ... ''')

Now we can run buildout::

  >>> print(system(join('bin', 'buildout')))
  Uninstalling template.
  Installing template.

The template was indeed created::

  >>> cat('parts', 'template')
  #
  My templåte knows about buildout path:
  .../sample-buildout


Creating missing paths
======================

If an output file should be created in a path that does not yet exist,
then the missing items will be created for us::

  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... parts = template
  ... offline = true
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... input = template.in
  ... output = ${buildout:parts-directory}/etc/template
  ... ''')

  >>> print(system(join('bin', 'buildout')))
  Uninstalling template.
  Installing template.

Also creation of several subdirectories is supported::


  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... parts = template
  ... offline = true
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... input = template.in
  ... output = ${buildout:parts-directory}/foo/bar/template
  ... ''')

  >>> print(system(join('bin', 'buildout')))
  Uninstalling template.
  Installing template.

  >>> cat('parts', 'foo', 'bar', 'template')
  #
  My templåte knows about buildout path:
  .../sample-buildout

When changes happen to the output path, then the old path is removed
on uninstall. Therefore the ``etc/`` directory created above has
vanished now::

  >>> ls('parts')
  d  foo


Substituting variables with options of other parts
==================================================

When substituting variables in a template, dependencies on other buildout
parts can occur. Buildout will resolve them by determining the values of those
other parts' options first. To see this, we create a buildout involving a
template that uses a variable computed by a part that would not otherwise be
built:

  >>> write('dummy.py',
  ... '''
  ... class Recipe(object):
  ...
  ...     def __init__(self, buildout, name, options):
  ...         options['foo'] = 'bar'
  ...
  ...     def install(self):
  ...         return ()
  ...
  ...     def update(self):
  ...         pass
  ... ''')

  >>> write('setup.py',
  ... '''
  ... from setuptools import setup
  ...
  ... setup(name='dummyrecipe',
  ...       entry_points = {'zc.buildout': ['default = dummy:Recipe']})
  ... ''')

  >>> write('buildout.cfg',
  ... '''
  ... [buildout]
  ... develop = .
  ... parts = template
  ... offline = true
  ...
  ... [template]
  ... recipe = collective.recipe.template
  ... input = template.in
  ... output = template
  ...
  ... [other]
  ... recipe = dummyrecipe
  ... ''')

  >>> write('template.in',
  ... '''#
  ... My templåte knows about another buildout part:
  ... ${other:foo}
  ... ''')

  >>> print(system(join('bin', 'buildout')))
  Develop: '/sample-buildout/.'
  Uninstalling template.
  Installing other.
  Installing template.

  >>> cat('template')
  #
  My templåte knows about another buildout part:
  bar

Unchanged output files are not rewritten on update
==================================================

When output content is unchanged, the output file is not rewritten on update.
The advantage is that the modification timestamp of the file is not changed.
(E.g. systemd notices if the timestamp of any unit files change, and issues
helpful "nags" reminding the user to rerun "systemctl daemon-reload".)

Note the mtime of the output file:
  >>> from os.path import getmtime
  >>> from time import sleep
  >>> orig_mtime = getmtime('template')

Wait until new files get a different mtime
  >>> def mtime_tick():
  ...     write('test.stamp', '')
  ...     return getmtime('test.stamp') > orig_mtime
  >>> wait_until('mtime_tick', mtime_tick)

Rerun the buildout:
  >>> print(system(join('bin', 'buildout')))
  Develop: '/sample-buildout/.'
  Uninstalling other.
  Installing other.
  Updating template.

The file's mtime is not changed:
  >>> getmtime('template') == orig_mtime
  True

Change the template:
  >>> write('template.in',
  ... '''#
  ... My template still knows about another buildout part:
  ... Foo is ${other:foo}
  ... ''')

Rerun the buildout:
  >>> print(system(join('bin', 'buildout')))
  Develop: '/sample-buildout/.'
  Uninstalling other.
  Installing other.
  Updating template.

The file's mtime is changed:
  >>> getmtime('template') > orig_mtime
  True

The output has changed:
  >>> cat('template')
  #
  My template still knows about another buildout part:
  Foo is bar


Changelog
*********

2.2 (2021-12-01)
================

* Port code and tests to Python 3 instead of using no longer supported 2to3.

* Add support for Python 3.7, 3.8, 3.9, and 3.10.

* Drop support for Python 3.4.


2.1 (2018-07-14)
================

* Support new ``input-encoding`` and ``output-encoding`` options.
  [fschulze]

* On update, do not rewrite the output file (thus preserving its
  modification timestamp) unless its content has changed.
  [dairiki]


2.0 (2017-01-17)
================

* Claim support for Python 3.5 and drop support for Python 2.6.
  [sallner]


1.13 (2015-10-20)
=================

* Back compatibility with zc.buildout 1.7.1 [#11]
  [mstaniszczak]


1.12 - 2015-07-23
=================

* Add timeout configuration option.
  [davidjb]

* Fix encoding problem in python 3.
  [cedricmessiant]

* Added overwrite option - possibility to disable overwrite output file after
  re-execute buildout.
  [mstaniszczak]


1.11 - 2014-02-07
=================

* Python 3 support for Genshi and doctests.
  [mitchellrj]

* Delete script before writing to it, this way we avoid chmod permission errors
  when the current user is not the script owner.
  [alecghica]


1.10 - 2012-02-26
=================

* Add Python 3 support using 2to3 flag in setup.
  [mitchellrj]


1.9 - 2011-06-19
================

* Add support for URL input. Use ``url =`` (instead of ``input =``) to specify URL.
  [aclark]


1.8 - 2010-06-08
================

* WARNING! Backward incompatible change for Genshi templates.
  It wasn't possible to access parts with a dash in the name, so now you have
  to use ${parts.partname} or ${parts['part-name']}. In addition it is now
  possible to access the current part with ``options``.
  [fschulze]

* Import genshi modules very late to prevent issues with zc.buildout.
  [fschulze]


1.7 - 2010-05-21
================

* Added support for genshi text templates. Use them with this as the
  recipe:
  `recipe = collective.recipe.template[genshi]:genshi`
  Use a dot between the section name and the option name instead of a colon.
  [fschulze]


1.6 - 2010-02-24
================

* Output file mode is now assumed to be octal, like chmod.
  [elro]

* Inline template can now be specified with the inline option.
  [elro]


1.5 - 2010-02-23
================

* Add support for explicitly setting the output file mode.
  [witsch]

* Add support for inline templates.
  [witsch]


1.4 - 2009-07-29
================

* Fixed the way variables in templates are substituted to allow buildout to
  determine dependencies on other parts and prepare those correctly. [tlotze]


1.3 - 2009-04-28
================

* Add support for output path creation. You can do::

    output = /path/to/target

  and intermediate path items will be created if they do not exist.
  [ulif]

* Add tests.
  [ulif]


1.2 - 2008-12-09
================

(By accident the 1.1 release was marked as 1.2. So in fact they are
the same.)

1.1 - 2008-12-09
================

* Correct handling of multiple variables in a line. Bugreport and patch from
  Roman Susi.
  [wichert]


1.0 - 2008-10-16
================

* Copy the mode of the input file to the output file. This makes it possible
  to create executable scripts.
  [wichert]

* Add missing link in README.
  [wichert]


1.0rc2 - 2008-07-04
===================

* Add a MANIFEST.in with instructions to include docs/, otherwise the package
  will not install.
  [wichert]


1.0rc1 - 2008-07-04
===================

* Initial release.
  [wichert]