collective.generic.webbuilder 1.1
Yet another WSGI Paste factory for paste by Makina Corpus
License: BSD

==========================
Introduction
==========================

.. contents::


CGWB is a web interface to ``paster``, its goal is to generate a webinterface to selection options aggregated from a set of templates.

Imagine that you have 2 templates, the one that can deploy an application, and the other which generates the application in itself.

Declaring the two templates as a ``cgwb set`` will make a webinterface for those 2 templates. Answering correctly to the questions will produce a tarball that you ll be able download and unpack to have your base installation setup.

To make the templates available, you must define the set using ZCML.


As this server was developped as a quick and efficient interface to paster, *it is not safe to open it to wide internet.*
For security reason, just launch/use when you need it.

Next versions will include some sessions/roles and improved security, it may be possible at this stage to leave it open.


See in action `here <http://cgwb-makinacorpus.rhcloud.com/>`_



Credits
=========================================


Companies
----------------
|makinacom|_

* `Planet Makina Corpus <http://www.makina-corpus.org>`_
* `Contact us <mailto:python@makina-corpus.org>`_

.. |makinacom| image:: http://depot.makina-corpus.org/public/logo.gif
.. _makinacom:  http://www.makina-corpus.com

Authors
---------------

    - kiorky <kiorky@cryptelium.net>
    - Jean-Philippe Camguilhem <jpc@makina-corpus.com>



Installation
==============

Installing cgwb in a minitage
-----------------------------------
You are not obliged to run with `minitage`_ even if it is the recommended mode for running at least the plones template.

Assuming that your minitage lives in ~/minitage, issue the following::

    export MT=~/minitage

Install or udpate minitage in your dedicated virtualenv if any
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Just do that (you must refer to `minitage installation`_ for prerequisites)

Install virtualenv
::

    virtualenv --no-site-packages --distribute $MT

Update minitage packages
::

    source $MT/bin/activate
    easy_install -U minitage.core
    easy_install -U minitage.paste
    minimerge -s


Install cgwb
++++++++++++++++++++++
Download & install via the minibuild
::

    source $MT/bin/activate
    git clone http://github.com/collective/collective.generic.webbuilder-minilay.git $MT/minilays/cgwb
    minimerge -v cgwb


Cgwb lives in ``$MT/bfg/cgwb``.

Generating & deploying your project using minitage
-----------------------------------------------------------
Launching the cgwb server
++++++++++++++++++++++++++++++++
Launch via ``bin/cgwb``.
This binary includes some options to let you override the default port (--port) and listenning address (--host)
To see all the available options, just use::

    bin/cgwb --help

If you use minitage, mandatory to use the minitage.instances.env profile::

    $MT/bin/easy_install -U minitage.paste
    $MT/bin/paster create -t minitage.instances.env cgwb

MINITAGE .ENV
++++++++++++++++++++
Each time you use cgwb, you use the .ENV::

    source $MT/bfg/cgwb/sys/share/minitage/minitage.env

Use it
++++++++++++++
Launch it::

    cd $INS
    ./bin/cgwb --port=6253


- At the moment, cgwb do not have some session mecanism, so the only way to replay a generation is to use the selenium firefox plugin.
- If you want to store your choices to redo an updated tarball later, just install the SeleniumIDE firefox plugin and use it to record your session.
- Maybe, activate selenium and

    - Go to the `cgwb`_
    - Choose `Generic Portal Plone3`.

Filling the settings, some notes
+++++++++++++++++++++++++++++++++++++++++++
- project name is mandatory and must be in the form in `project` or `subproject`.
- You can choose in the `Plone Products to auto checkout in development mode` the products from the community from which we should check out & use in development mode

THE IMPORTANT PART AROUND INITIATING A PROJECT
+++++++++++++++++++++++++++++++++++++++++++++++++
- It would be good unless you have some minitage experience to version the code prior to build, because of minitage update mecanism.
- Before version/import the code in your SCM you must elude the following points:

    * By default, the generated tarball contains the buildout layout and all the eggs in src, and the buildout use them as develop eggs and NOT WITH MR.DEVELOPER.
      Thus for running the buildout in standalone mode
    * You may decide not to include them as-is but to separate the code and version the code elsewhere.
    * I would advice you to checkout the packages with mr.developer.

An example of using svn which generic/pyramid
+++++++++++++++++++++++++++++++++++++++++++++
What i would do from a generated tarball for using subversion as my SCM could be to produce this layout::

    import
    |-- import/eggs
    |   |-- import/eggs/myproject.core
    |   |   `-- import/eggs/myproject.core/trunk
    `-- import/buildout


- Exporting base variables::

    export PROJECT="myproject" # your project name as filled in the web interfacE
    export TARBALL="$(ls -1t ~/cgwb/${PROJECT}-*.tar.gz|head -n1)" # produced tarball
    export IMPORT_URL="https://subversion.xxx.net/scrumpy/${PROJECT}/" # base svn place to import

- Create a temporary workspace::

    mkdir -p  $PROJECT/tarball
    cd $PROJECT
    tar xzvf  $TARBALL -C tarball/

- Create the base layout to be imported::

    mkdir -p import/buildout import/eggs

- Move the generated plone extensions eggs to a separate place to be imported::

    for i in tarball/src/${PROJECT}*;do if [[ -d $i ]] && [[ $(basename $i) != "themes" ]];then j=$(basename $i);dest=import/eggs/$j/trunk; mkdir -pv  $(dirname $dest); mv -v $i $dest; fi; done

- Move the buildout structure in the import layout::

    cp -rf tarball/* import/buildout

- Update buildout to use mr.developer instead of basic develop::

    * move off the develop declaration::

        sed -re "s:(src/)?$PROJECT\.((skin)|(tma)|(core)|(testing))::g" -i import//buildout/etc/project/$PROJECT.cfg

    * add to mr.developer sources::

        sed -re "/\[sources\]/{
        a $PROJECT.core = svn $IMPORT_URL/eggs/$PROJECT.core/trunk
        }" -i import/buildout/etc/project/sources.cfg

    * add to auto checkout packages::

        sed -re "/auto-checkout \+=/{
        a \    $PROJECT.core
        }"  -i import/buildout/etc/project/sources.cfg
        sed -re "/eggs \+=.*buildout:eggs/{
        a \    $PROJECT.core
        }"  -i import/buildout/etc/project/$PROJECT.cfg
        sed -re "/zcml \+=/{
        a \    $PROJECT.core
        }"  -i import/buildout/etc/project/$PROJECT.cfg

* be sure to use the right svn url to checkout::

    sed -re "s|src_uri.*|src_uri=$IMPORT_URL/buildout/|g" -i import/buildout/minilays/$PROJECT/*

* Be sure to use svn

    sed -re "s|src_type.*|src_type=svn|g" -i import/buildout/minilays/$PROJECT/*

* Import::

   svn import import/ $IMPORT_URL -m "initial import" 

An example of using svn which generic/plone
+++++++++++++++++++++++++++++++++++++++++++++
What i would do from a generated tarball for using subversion as my SCM could be to produce this layout::

    import
    |-- import/eggs
    |   |-- import/eggs/myproject.policy
    |   |   `-- import/eggs/myproject.policy/trunk
    |   |-- import/eggs/myproject.skin
    |   |   `-- import/eggs/myproject.skin/trunk
    |   |-- import/eggs/myproject.testing
    |   |   `-- import/eggs/myproject.testing/trunk
    |   `-- import/eggs/myproject.tma
    |       `-- import/eggs/myproject.tma/trunk
    `-- import/minitage
        |-- import/minitage/buildouts
        |   `-- import/minitage/buildouts/zope
        |       `-- import/minitage/buildouts/zope/myproject


- Exporting base variables::

    export PROJECT="myproject" # your project name as filled in the web interfacE
    export TARBALL="$(ls -1t ~/cgwb/${PROJECT}-*.tar.gz|head -n1)" # produced tarball
    export IMPORT_URL="https://subversion.xxx.net/scrumpy/${PROJECT}/ # base svn place to import

- Create a temporary workspace::

    mkdir -p  $PROJECT/tarball
    cd $PROJECT
    tar xzvf  $TARBALL -C tarball/

- Create the base layout to be imported::

    mkdir -p import/buildout import/eggs

- Move the generated plone extensions eggs to a separate place to be imported::

    for i in tarball/src/${PROJECT}*;do if [[ -d $i ]] && [[ $(basename $i) != "themes" ]];then j=$(basename $i);dest=import/eggs/$j/trunk; mkdir -pv  $(dirname $dest); mv -v $i $dest; fi; done

- Move the buildout structure in the import layout::

    cp -rf tarball/* import/buildout

- Update buildout to use mr.developer instead of basic develop::

    * move off the develop declaration::

        sed -re "s:(src/)?$PROJECT\.((skin)|(tma)|(policy)|(testing))::g" -i import//buildout/etc/project/$PROJECT.cfg

    * add to mr.developer sources::

        sed -re "/\[sources\]/{
        a $PROJECT.policy = svn $IMPORT_URL/eggs/$PROJECT.policy/trunk
        a $PROJECT.tma = svn $IMPORT_URL/eggs/$PROJECT.tma/trunk
        a $PROJECT.skin = svn $IMPORT_URL/eggs/$PROJECT.skin/trunk
        a $PROJECT.testing = svn $IMPORT_URL/eggs/$PROJECT.testing/trunk
        }" -i import/buildout/etc/project/sources.cfg

    * add to auto checkout packages::

        sed -re "/auto-checkout \+=/{
        a \    $PROJECT.policy
        a \    $PROJECT.tma
        a \    $PROJECT.skin
        a \    $PROJECT.testing
        }"  -i import/buildout/etc/project/sources.cfg
        sed -re "/eggs \+=.*buildout:eggs/{
        a \    $PROJECT.policy
        a \    $PROJECT.tma
        a \    $PROJECT.skin
        a \    $PROJECT.testing
        }"  -i import/buildout/etc/project/$PROJECT.cfg
        sed -re "/zcml \+=/{
        a \    $PROJECT.policy
        a \    $PROJECT.tma
        a \    $PROJECT.skin
        }"  -i import/buildout/etc/project/$PROJECT.cfg

* be sure to use the right svn url to checkout::

    sed -re "s|src_uri.*|src_uri=$IMPORT_URL/buildout/|g" -i import/buildout/minilays/$PROJECT/*

* Be sure to use svn

    sed -re "s|src_type.*|src_type=svn|g" -i import/buildout/minilays/$PROJECT/*

* Import::

   svn import import/ $IMPORT_URL -m "initial import"

An example of using git which generic
++++++++++++++++++++++++++++++++++++++++
What i would do from a generated tarball for using subversion as my SCM could be to produce this layout::

    import
        |-- myproject.policy
        |-- myproject.skin
        |-- myproject.testing
        `-- myproject.tma
        `-- myproject.buildout
        `-- myproject.minilay


- Exporting base variables::

    export PROJECT="myproject"                                     # your project name as filled in the web interfacE
    export TARBALL="$(ls -1t ~/cgwb/${PROJECT}-*.tar.gz|head -n1)" # produced tarball
    export IMPORT_URL="ssh://git.makina-corpus.net/var/git"              # base svn place to import

- Create a temporary workspace & the base layout to be imported::

    mkdir -p  $PROJECT/
    cd $PROJECT
    mkdir tarball import
    tar xzvf  $TARBALL -C tarball/

- Move the generated plone extensions eggs to a separate place to be imported::

    for i in tarball/src/*;do if [[ -d $i ]] && [[ $i != "tarball/src/themes" ]];then j=$(basename $i);dest=import/$j;mkdir -pv  $(dirname $dest); mv -v $i $dest; fi; done

- Move the buildout structure in the import layout::

    cp -rf tarball/minilays/$PROJECT   import/$PROJECT.minilay
    rm -rf tarball/minilays
    cp -rf tarball/ import/$PROJECT.buildout

- Update buildout to use mr.developer instead of basic develop::

    * move off the develop declaration::

        sed -re "s:(src/)?$PROJECT\.((skin)|(tma)|(policy)|(testing))::g" -i import/$PROJECT.buildout/etc/project/$PROJECT.cfg

    * add to mr.developer sources::

        sed -re "/\[sources\]/{
        a $PROJECT.policy =  git $IMPORT_URL/$PROJECT.policy
        a $PROJECT.tma =     git $IMPORT_URL/$PROJECT.tma
        a $PROJECT.skin =    git $IMPORT_URL/$PROJECT.skin
        a $PROJECT.testing = git $IMPORT_URL/$PROJECT.testing
        }" -i import/$PROJECT.buildout/etc/project/sources.cfg

    * add to auto checkout packages::

        sed -re "/auto-checkout \+=/{
        a \    $PROJECT.policy
        a \    $PROJECT.tma
        a \    $PROJECT.skin
        a \    $PROJECT.testing
        }"  -i import/$PROJECT.buildout/etc/project/sources.cfg
        sed -re "/eggs \+=.*buildout:eggs/{
        a \    $PROJECT.policy
        a \    $PROJECT.tma
        a \    $PROJECT.skin
        a \    $PROJECT.testing
        }"  -i import/$PROJECT.buildout/etc/project/$PROJECT.cfg
        sed -re "/zcml \+=/{
        a \    $PROJECT.policy
        a \    $PROJECT.tma
        a \    $PROJECT.skin
        }"  -i import/$PROJECT.buildout/etc/project/$PROJECT.cfg

* be sure to use the right git url to checkout::

    sed -re "s|src_uri.*|src_uri=$IMPORT_URL/$PROJECT.buildout|g" -i import/*.minilay/*

* Be sure to use git

    sed -re "s|src_type.*|src_type=git|g" -i import/*.minilay/*

* Import::

   pushd import;for i in *;do echo "Importing $i";pushd $i;git init;git add *;git commit -am "initial revision";git remote add origin "$IMPORT_URL/$i";git push --all origin;popd;done;popd

Deploy the project
++++++++++++++++++++++
* install the minilay::

    export MT=~/minitage
    svn co $IMPORT_URL/buildout/minilays/$PROJECT/ $MT/minilays/$PROJECT
    # or
    git clone  $IMPORT_URL/$PROJECT.minilay $MT/minilays/$PROJECT

* Install it::

    minimerge -v $PROJECT

.. _`minitage installation`: http://minitage.org/installation.html
.. _`cgwb`: http://localhost:6253
.. _`minitage`: http://www.minitage.org




Tests & docs
==============
Defining sets via ZCML
---------------------------------------------------

A set is a collection of templates, it is also known as a 'PasterConfiguration'.
::

    -------------------------------------------
    | configuration                           |
    |                                         |
    |       -----------------------------------
    |       |  templates                      |
    |       -----------------------------------
    |       |       |  group                  |
    |       |       [--------------------------
    |       |       |      |  options         |
    |       |       |      --------------------
    |       |       |      |                  |
    -------------------------------------------


We will redefine the 'well known' plone template as an example.

First of all, we need to define a template
::

    >>> from zope.configuration import xmlconfig
    >>> from zope.configuration.config import ConfigurationMachine
    >>> from collective.generic.webbuilder.zcml import PasterConfiguration, Template, Group, ExcludeOption, Option
    >>> from collective.generic.webbuilder.models import root
    >>> from minitage.paste.projects import plone3
    >>> import collective.generic.webbuilder
    >>> context = ConfigurationMachine()
    >>> xmlconfig.registerCommonDirectives(context)
    >>> xmlconfig.include(context, 'meta.zcml', collective.generic.webbuilder)
    >>> context = xmlconfig.string("""
    ... <configure xmlns="http://webbuilder.org/webbuilder">
    ...  <genericpaster name="Test Generic Portal Plone">
    ...    <!--<plugin name="dummy_plugin" order="1"/>-->
    ...    <plugin name="egg_plugin" order="2"/>
    ...    <template name="collective.generic.policy" output="src" order="200">
    ...       <excludeoptions prefix="project_.*" />
    ...       <excludeoption  name="python" />
    ...    </template>
    ...    <template name="minitage.plone3" order="1">
    ...       <group name="Minitage" order="05">
    ...         <option name="install_method" alias="ai"/>
    ...         <options prefix=".*with.*" default="true" type="boolean"/>
    ...         <excludeoptions prefix="project_.*" />
    ...         <excludeoption  name="python" />
    ...       </group>
    ...    </template>
    ...  </genericpaster>
    ... </configure>
    ... """, context = context)


It will register/update the ``collective.generic.webbuilder.root.configurations`` module variable

The *genericpaster* directive
++++++++++++++++++++++++++++++
- Must be used at top level.
- Name of a configuration of templates.

::

    <genericpaster name="Name of the configuration"/>

It contains a list of underlying configurations
::

    >>> 'Test Generic Portal Plone' in root.configurations
    True

The configurations objects contain a list of templates and plugins
::

    >>> templates = root.configurations['Test Generic Portal Plone'].templates
    >>> sorted(templates.keys())
    ['collective.generic.policy', 'minitage.plone3']


The *template* directive
+++++++++++++++++++++++++
- Must be used at genericpaster level.
- It describe a relative "paster template". The name which you could get with ``paster create -t --list-templates``.
- It has also an order which is used to order templates in the webinterface for lower to upper.

::

     <template name="Template Name" order="int">

::

    >>> t = templates['minitage.plone3']
    >>> t.order
    1
    >>> t.name
    'minitage.plone3'


A template can also say that it must be generated under a 'subdirectory' with the ``output`` attribute.
::

    >>> templates['collective.generic.policy'].output
    'src'


The *group* directive
++++++++++++++++++++++
- A template has a list of groups of options.
- Groups are represented by a block of questions surrounded by the group name in the webinterface.

::

    <group name="GroupName" order="int"/>

Those groups group 'paster questions'.

::

    >>> groups = t.groups
    >>> groups.keys()
    ['default', 'Minitage']
    >>> g = t.groups['Minitage']
    >>> t.groups['Minitage'].order
    5
    >>> t.groups['Minitage'].name
    'Minitage'

The *options* directive
++++++++++++++++++++++++
- Must be used at group level.
- Groups group options, Which can be grabbed by a regular expression with this directive.

::

    <options prefix="Regular expression"  type="boolean|" default="value"/>

- *type* can be omitted and defaults to None (text).
- *default* can be omitted and no default value will be assigned (or the paster default value).

::

    >>> opts = g.options['.*with.*']
    >>> opts.type, opts.default
    ('boolean', 'true')

As you can see, there is a default group where go non-matched options which are not excluded via the ``excludeoptions`` directive.


The *option* directive
++++++++++++++++++++++++
- Must be used at group level.
- Groups group also 'single options', Which can be grabbed by their name.
- Single options and can have an alias. It is useful  if we have the same 'option name' in 2 templates of the configuration and we don't want that they share the same value (default behaviour). To be clear, we have the option 'project', in template 'a' and 'b', by default, if we choose 'foo' for 'project', the value will be 'foo' in template 'a' and 'b', and with an alias, we can choose the value for 'a' *_and_* for 'b'.

::

    <options name="name" alias="alias name"  type="boolean|" default="value"/>

- *alias* can be omitted.
- *type* can be omitted and defaults to None (text).
- *default* can be omitted and no default value will be assigned (or the paster default value).

::

    >>> opt = g.single_options['install_method']
    >>> opt.type, opt.alias, opt.default
    (None, 'ai', None)


The *excludeoptions* & *excludeoption* directives
+++++++++++++++++++++++++++++++++++++++++++++++++++
- Must be used at group or template level (in any group of the template).

::

    <excludeoptions prefix="regular expression"/>

- exclude options from the interface
- *prefix*: regular expression for the options to exclude.


::

    <excludeoption name="option name"/>

- exclude an option from the interface
- *name*: name for the options to exclude.

::

    >>> [[getattr(templates[template].groups['default'], attr).keys() for attr in 'exclude_options', 'excludes_options'] for template in 'minitage.plone3', 'collective.generic.policy']
    [[['python'], ['project_.*']], [['python'], ['project_.*']]]


The *plugin* directive
++++++++++++++++++++++++
- Must be used at template level.
- Declare which plugin must run after the templates collection generation.
- This is useful for example, to rearrange things which are generated.

To run a plugin which is declared  under "plugin name".
::

    <plugin name="plugin name" order="int"/>

- *name*: name of the adapter
- *order*: control order to run if there are more than one plugin

A plugin, is a simple adapter which takes a IPasterAssembly and provides IPostGenerationPlugin
::

    <adapter
      name="plugin name"
      provides=".interfaces.IPostGenerationPlugin"
      factory=".plugins.MyPluginFactory"
      for=".interfaces.IPasterAssembly"
    />

::

    >>> plugins = root.configurations['Test Generic Portal Plone'].plugins
    >>> plugins
    [('egg_plugin', 2)]



The paster dance
--------------------------

Heart of cgwb is pythonpaste, take some of paster templates, gather them in an ihm for user inputs and answears for generating a final composition of those templates, with or without been modificated by surrounded plugins.
::

    User choose a configuration
        --------->
            read variables from templates which are in the configuration and give the appropriate choice to the user
        ------------->
                User inputs and submit it
        -------------------->
                    We generate a tarball of the assembled templates according to the answers

* An option is asked only once, only you make aliases for each of the options which have the same name among templates.
* As a question is asked only once, if its type is not default, you must define it in the configuration of the template which has the less order number, because there will be there the question will be asked.


Loading a zcml representation of a configuration
---------------------------------------------------

Testing the zcml to python represetation
+++++++++++++++++++++++++++++++++++++++++++++

Load our test package where we have three templates
::

    >>> import collective.generic.webbuilder.tests
    >>> testegg = os.path.join( collective.generic.webbuilder.tests.__path__[0], 'egg', 'src')
    >>> pkg_resources.working_set.add_entry(testegg)
    >>> env = pkg_resources.Environment()
    >>> egg = env['cgwb.tp'][0]


We have 3 templates in there waiting to be assembled
::

    >>> pprint(egg.get_entry_map())
    {'paste.paster_create_template': {'cgwb.testpackage1': EntryPoint.parse('cgwb.testpackage1 = tp.package:Package'),
                                      'cgwb.testpackage2': EntryPoint.parse('cgwb.testpackage2 = tp1.package:Package'),
                                      'cgwb.testpackage3': EntryPoint.parse('cgwb.testpackage3 = tp2.package:Package')}}


The configuration
+++++++++++++++++++
It is more described in the zcml part of the documentation, but it's a zcml representation of which variables from the pastertemplates we want to extract and how we want to present them to users.


A sample zcml needed to assemble the packages we declared before is as follow:
::

    >>> paster_zcml = """
    ... <configure xmlns="http://namespaces.repoze.org/bfg" xmlns:meta="http://namespaces.zope.org/meta">
    ...   <include package="collective.generic.webbuilder" file="meta.zcml"/>
    ...   <configure xmlns="http://webbuilder.org/webbuilder">
    ...     <genericpaster name="test Assembler">
    ...         <template name="cgwb.testpackage1" output="1" order="1000">
    ...           <group name="Minitage" order="05">
    ...             <option name="tp1option" type="boolean"/>
    ...             <option name="tp1option3" default="y"/>
    ...           </group>
    ...         </template>
    ...         <template name="cgwb.testpackage2" output="2" order="200">
    ...           <group name="Authors" order="20">
    ...             <options prefix="^author.*"/>
    ...           </group>
    ...           <excludeoptions prefix=".*"/>
    ...         </template>
    ...         <template name="cgwb.testpackage3" output="3" order="500">
    ...           <group name="Plone Settings" order="8">
    ...             <option name="tp2opton2" />
    ...             <option name="author_email" />
    ...           </group>
    ...           <group name="Authors" order="20">
    ...             <options prefix="^author.*"/>
    ...           </group>
    ...           <group name="Package tuning" order="1">
    ...             <option name="project_name" type="hidden" default="tma" alias="tmapn"/>
    ...           </group>
    ...         </template>
    ...     </genericpaster>
    ...   </configure>
    ... </configure>
    ... """
    >>> noecho = xmlconfig.string(paster_zcml)
    >>> root.configurations['test Assembler']
    <collective.generic.webbuilder.zcml.PasterConfiguration object at ...>


PasterAssembly object
++++++++++++++++++++++++++++
have some "log variables" and the configuration name to search for in the "bfgroot".configurations dictionnary.


    - ``template_data``: list of mappings in the form::

        [
         {
          'self': template 'zcml' object,
          'name': paster template name,
          'added_options': option added by this template,
          'not_explicit_options': option added by this template which were not explicitly matched,
          'display' : display a template or not
          'groups':
             {
                groupname: 
                    {
                        'name': groupname,
                        'group': zcml group object:
                        'options': [(paster variablen, type, optionn name, alias|None, zcml optionn|None )]
                    }

             },
          'aliases': [ (varName, aliasName),]

         }
        ]


    - ``added_options``: All options added for all templates


Get an assembly for the wanted configuration
::

    >>> ta = gpaster.PasterAssembly('test Assembler')
    >>> pprint(ta.__dict__.items())
    [('templates_data', []),
     ('configuration',
      <collective.generic.webbuilder.zcml.PasterConfiguration object at ...>),
     ('added_options', []),
     ('configuration_name', 'test Assembler')]

The PasterAssemblyReader object
+++++++++++++++++++++++++++++++++
This adapter takes as input a IPasterAssembly object and implements the IPasterAssemblyReader interface.

We have configurations stored into zcml representation, now we need to gather and map the configuration informations with the content of each "paster template" into a python friendly structure.
This Reader component is responsible for storing in the assembly object:

    - The extracted template name
    - Each group of options
    - For each of those groups:

        - The excluded options
        - For the options which are not excluded, if applicable:

          - making its alias
          - Assign the default value

What will finnally load the assembly data structures is a reader that now how to parse a configuration
::

    >>> reader = gpaster.PasterAssemblyReader(ta)
    >>> reader.readed
    False
    >>> reader.read()
    >>> len(ta.added_options) > 0
    True
    >>> reader.readed
    True


We will check now that the structure loaded is as we wanted
::

    >>> t1 = pkg_resources.load_entry_point('cgwb.tp', 'paste.paster_create_template', 'cgwb.testpackage1')
    >>> t2 = pkg_resources.load_entry_point('cgwb.tp', 'paste.paster_create_template', 'cgwb.testpackage2')
    >>> t3 = pkg_resources.load_entry_point('cgwb.tp', 'paste.paster_create_template', 'cgwb.testpackage3')


Templates
++++++++++

Order of templates is respected
::

    >>> [t['name'] for t in ta.templates_data]
    ['cgwb.testpackage2', 'cgwb.testpackage3', 'cgwb.testpackage1']
    >>> rt2, rt3, rt1 =  ta.templates_data

Groups
++++++
Options in template3, on the paster side, that we ll find on the next example
::

    >>> pprint([v.name for v in t3.vars])
    ['namespace',
     'nested_namespace',
     'version',
     'author',
     'author_email',
     'tp3option',
     'tp3option3',
     'keywords',
     'license_name',
     'project_name']

For each templates, options are grouped, and groups respect order defined in zcml::

    >>> pprint([(rt3['groups'][n]['name'] , rt3['groups'][n]['options']) for n in range(len(rt3['groups']))])
    [('Package tuning',
      [(<var project_name default='tma' should_echo=True>,
        'hidden',
        'tmapn',
        <collective.generic.webbuilder.zcml.Option object at ...>)]),
     ('Plone Settings',
      [(<var author_email default='bar@localhost' should_echo=True>,
        'default',
        None,
        <collective.generic.webbuilder.zcml.Option object at ...>)]),
     ('Authors',
      [(<var author default='foo' should_echo=True>,
        'default',
        None,
        <collective.generic.webbuilder.zcml.Options object at ...>)]),
     ('default',
      [(<var namespace default='%(namespace)s' should_echo=True>,
        'default',
        None,
        None),
       (<var nested_namespace default='%(package)s' should_echo=True>,
        'default',
        None,
        None),
       (<var version default='1.0' should_echo=True>, 'default', None, None),
       (<var tp3option default='http://python.org' should_echo=True>,
        'default',
        None,
        None),
       (<var tp3option3 default='Project %s' should_echo=True>,
        'default',
        None,
        None),
       (<var keywords default='' should_echo=True>, 'default', None, None),
       (<var license_name default='GPL' should_echo=True>,
        'default',
        None,
        None)])]


As you can see project_name has been aliased and will be explained after.


Consumed options
+++++++++++++++++
Goal is to insist loudly on consumed option.
When an option is consumed by another template, it is not available in others to be asked only once.
That's why , template1 has no variables which were first asked in template3.
::

    >>> rt3options = []; noecho = [rt3options.extend(g['options']) for g in rt3['groups']]; rt3options = [opt[0].name for opt in rt3options]
    >>> rtp1options = []; noecho = [rtp1options.extend(g['options']) for g in rt1['groups']]; rtp1options = [opt[0].name for opt in rtp1options]

project_name, author, etc. are not part of template1 options even if they are in the paster template.
They have been consumed by template3
::

    >>> pprint([v.name for v in t1.vars])
    ['namespace',
     'nested_namespace',
     'version',
     'author',
     'author_email',
     'tp1option',
     'tp1option2',
     'tp1option3',
     'keywords',
     'license_name',
     'project_name']
    >>> rt3options
    ['project_name', 'author_email', 'author', 'namespace', 'nested_namespace', 'version', 'tp3option', 'tp3option3', 'keywords', 'license_name']
    >>> rtp1options
    ['tp1option', 'tp1option3', 'tp1option2', 'project_name']

Excluded options
+++++++++++++++++

Template2 ignore all opions per default, even adding an option can't precedence over ignoring options.
Take care of your regexes !
::

    >>> [g['options'] for g in rt2['groups']]
    [[], []]


Typed  options
++++++++++++++++

We can assign type to values to use different widgets to display them in the UI for example.
Supported types are:

    - boolean (checkbox)
    - hidden (hidden)
    - default (textarea)

template3 define project_name as ``hidden``
::

    >>> rt3['groups'][0]['options'][0][1]
    'hidden'

template1 define tp1option as ``boolean``.
::

    >>> rt1['groups'][0]['options'][0][1]
    'boolean'

If an option default startswith 'y', 'true', or 'on', we switch the option type to boolean
::

    >>> rt1['groups'][0]['options'][1][1]
    'boolean'

Options in the default group have ``default`` as type, as for options without explicit type
::

    >>> rt3["groups"][3]['options'][0][1]
    'default'
    >>> rt3["groups"][2]['options'][0][1]
    'default'

Option aliases
+++++++++++++++
We have defined a default value and a default type for template3.project_name which is also an alias.
Alias allow options with the same name but not the same value to exists within the same Assembly.
Default behaviour tells that one value is asked only once and used for all options that have the same name unless they are aliased explicitly each one of them.::

    >>> rt3['groups'][0]['options']
    [(<var project_name default='tma' should_echo=True>, 'hidden', 'tmapn', <collective.generic.webbuilder.zcml.Option object at ...>)]


Added options
+++++++++++++

We can retrieve options added

    * For one template ::

        >>> rt1['added_options']
        ['tp1option', 'tp1option2', 'tp1option3', 'project_name']
        >>> rt2['added_options']
        []
        >>> rt3['added_options']
        ['namespace', 'nested_namespace', 'version', 'author', 'author_email', 'tp3option', 'tp3option3', 'keywords', 'license_name', 'project_name']


    * For all templates ::

        >>> ta.added_options
        ['namespace', 'nested_namespace', 'version', 'author', 'author_email', 'tp3option', 'tp3option3', 'keywords', 'license_name', 'tmapn', 'tp1option', 'tp1option2', 'tp1option3', 'project_name']



Creating plugins to rearrange things after a successfull templates generation
-------------------------------------------------------------------------------



A plugin is a simple adapter
++++++++++++++++++++++++++++

Creating plugins to run after a generation is really simple.
It is just a matter of implementing an adapter which takes an ``\IPasterConfiguration`` and provided ``IPostGenerationPlugin``.
::

    <adapter
      name="plugin name"
      provides=".interfaces.IPostGenerationPlugin"
      factory=".plugins.MyPluginFactory"
      for=".interfaces.IPasterAssembly"
    />

The eggs plugin
+++++++++++++++++

For example, here is a simple plugin which take all eggs in a 'src' directory and register them in 'zcml' and 'develop' in the relative ''buildout.cfg''
It will add just the 'policy' egg to the intance's zcml option.


Boiler plate to simulate a generation

    >>> import tempfile, shutil, os
    >>> c = os.getcwd()
    >>> d = tempfile.mkdtemp()
    >>> os.chdir(d)
    >>> open('buildout.cfg', 'w').write('[buildout]\ndevelop+=\n   foo\n[instance]\nzcml=  too\n')
    >>> os.makedirs('src/bar/src')
    >>> os.makedirs('src/policy/src')
    >>> open('src/bar/setup.py', 'w').write('')
    >>> open('src/policy/setup.py', 'w').write('')

Running the plugin

    >>> from collective.generic.webbuilder.models import root
    >>> conf = root.configurations['Generic Portal Plone3']
    >>> from collective.generic.webbuilder import interfaces, paster
    >>> pa = paster.PasterAssembly('Generic Portal Plone3')
    >>> plugin = zope.component.queryAdapter(pa, interfaces.IPostGenerationPlugin, name='egg_plugin')
    >>> plugin.process(d, 'foo', {})
    >>> print open('buildout.cfg').read()
    [buildout]
    develop+=src/policy
       src/bar
       foo
    eggs += policy
            bar
    [instance]
    zcml=  too  
                policy
    <BLANKLINE>

Cleanup

    >>> os.chdir(c);shutil.rmtree(d)


Registering plugins
+++++++++++++++++++++
Please refer to the *plugin* zcml directive to know how to add plugins for a 'Configuration'.

Here is an example about the "eggs_plugins"
::

    <configure xmlns="http://namespaces.repoze.org/bfg"
        xmlns:meta="http://namespaces.zope.org/meta"
        xmlns:cgwb=xmlns="http://webbuilder.org/webbuilder">
        <adapter
            name="egg_plugin"
            provides=".interfaces.IPostGenerationPlugin"
            factory=".plugins.EggPlugin"
            for=".interfaces.IPasterAssembly"
        />
        <cgwb:genericpaster name="Generic Portal Plone">
            <cgwb:plugin name="egg_plugin" order="2"/>
            <cgwb:template name="minitage.plone3" order="1">
            </template>
        </genericpaster>
    </configure>



The paster dance
--------------------------

Heart of cgwb is pythonpaste, take some of paster templates, gather them in an ihm for user inputs and answears for generating a final composition of those templates, with or without been modificated by surrounded plugins.
::

    User choose a configuration
        --------->
            read variables from templates which are in the configuration and give the appropriate choice to the user
        ------------->
                User inputs and submit it
        -------------------->
                    We generate a tarball of the assembled templates according to the answers

* An option is asked only once, only you make aliases for each of the options which have the same name among templates.
* As a question is asked only once, if its type is not default, you must define it in the configuration of the template which has the less order number, because there will be there the question will be asked.


Loading a zcml representation of a configuration
---------------------------------------------------

Testing the zcml to python represetation
+++++++++++++++++++++++++++++++++++++++++++++

Load our test package where we have three templates
::

    >>> import collective.generic.webbuilder.tests
    >>> testegg = os.path.join( collective.generic.webbuilder.tests.__path__[0], 'egg', 'src')
    >>> pkg_resources.working_set.add_entry(testegg)
    >>> env = pkg_resources.Environment()
    >>> egg = env['cgwb.tp'][0]

The configuration
+++++++++++++++++++
It is more described in the zcml part of the documentation, but it's a zcml representation of which variables from the pastertemplates we want to extract and how we want to present them to users.


A sample zcml needed to assemble the packages we declared before is as follow:
::

    >>> paster_zcml = """
    ... <configure xmlns="http://namespaces.repoze.org/bfg" xmlns:meta="http://namespaces.zope.org/meta">
    ...   <include package="collective.generic.webbuilder" file="meta.zcml"/>
    ...   <configure xmlns="http://webbuilder.org/webbuilder">
    ...     <genericpaster name="test Assembler">
    ...         <template name="cgwb.testpackage1" output="1" order="1000">
    ...           <group name="Minitage" order="05">
    ...             <option name="tp1option" type="boolean"/>
    ...             <option name="tp1option3" default="y"/>
    ...           </group>
    ...         </template>
    ...         <template name="cgwb.testpackage2" output="2" order="200">
    ...           <group name="Authors" order="20">
    ...             <options prefix="^author.*"/>
    ...           </group>
    ...           <excludeoptions prefix=".*"/>
    ...         </template>
    ...         <template name="cgwb.testpackage3" output="3" order="500">
    ...           <group name="Plone Settings" order="8">
    ...             <option name="tp2opton2" />
    ...             <option name="author_email" />
    ...           </group>
    ...           <group name="Authors" order="20">
    ...             <options prefix="^author.*"/>
    ...           </group>
    ...           <group name="Package tuning" order="1">
    ...             <option name="project_name" type="hidden" default="tma" alias="tmapn"/>
    ...           </group>
    ...         </template>
    ...     </genericpaster>
    ...   </configure>
    ... </configure>
    ... """
    >>> noecho = xmlconfig.string(paster_zcml)
    >>> root.configurations['test Assembler']
    <collective.generic.webbuilder.zcml.PasterConfiguration object at ...>

We will check now that the structure loaded is as we wanted
::

    >>> t1 = pkg_resources.load_entry_point('cgwb.tp', 'paste.paster_create_template', 'cgwb.testpackage1')
    >>> t2 = pkg_resources.load_entry_point('cgwb.tp', 'paste.paster_create_template', 'cgwb.testpackage2')
    >>> t3 = pkg_resources.load_entry_point('cgwb.tp', 'paste.paster_create_template', 'cgwb.testpackage3')
    >>> server, url = launch_server()
    >>> browser = Browser(url)

We can see that in the main page we have the default configurations and the custom loaded one
::

    >>> 'test Assembler' in browser.contents
    True
    >>> 'Generic Portal Plone4' in browser.contents
    True
    >>> 'Generic Portal Plone3' in browser.contents
    True

Following the test assembler link
::

    >>> browser.getLink('test Assembler').click()
    >>> htmlS(browser.contents).xpath('//input[@name="project"]')[0]
    <InputElement ... name='project' type='text'>

I can submit a form and a valid it
::

    >>> browser.getControl(name='project').value = 'myproject'
    >>> browser.getControl(name='author').value = 'tim burton'
    >>> browser.getControl(name='author_email').value = 'tim burton@foo.com'
    >>> browser.getControl(name='tp1option').value = False
    >>> browser.getControl(name='tp1option2').value = 'Project Monster'
    >>> browser.getControl(name='project_name').value = 'My Big Project'
    >>> browser.getControl(name='submit_cgwbDownload').click()
    >>> '.tar' in browser.contents
    True

The sucessful produced result is a tarball
::

    >>> pprint(browser.headers.headers)
    ['Server:...
     'Date:...
     'Content-Disposition: attachment; filename="myproject....tar.gz"\r\n',
     'Content-Transfer-Encoding: binary\r\n',
     'Content-Length: ...\r\n']
    >>> import tarfile
    >>> tar = tarfile.open(fileobj=StringIO(browser.contents))

In the produced tarball, output directories present in the zcml configuration are respected in the tarball::

    >>> files = [a.name for a in tar];files.sort();pprint(files)
    ['.',
     '1',
     '1/myproject',
     '1/myproject/test',
     '2',
     '2/myproject',
     '2/myproject/test1',
     '3',
     '3/myproject',
     '3/myproject/test2']
    >>> templates = dict([(a.name,a) for a in tar if 'test' in a.name])
    >>> t2 = templates['2/myproject/test1']
    >>> t3 = templates['3/myproject/test2']
    >>> t1 = templates['1/myproject/test']

Options filled in the interface are well interpreted in templates
::

    >>> pprint([a for a  in tar.extractfile(t1).read().split('\n') if a.strip()])
    ['namespace                 =>            %(namespace)s',
     'nested_namespace          =>            %(package)s',
     'version                   =>            1.0',
     'author                    =>            tim burton',
     'author_email              =>            tim burton@foo.com',
     'tp1option                 =>            False',
     'tp1option2                =>            Project Monster',
     'tp1option3                =>            True',
     'keywords                  =>            ',
     'license_name              =>            GPL',
     'project_name              =>            My Big Project']

The project_name entered for project1 is shared in project2
::

    >>> pprint([a for a  in tar.extractfile(t2).read().split('\n') if a.strip()])
    ["'namespace'                =>         '%(namespace)s'",
     "'nested_namespace'         =>         '%(package)s'",
     "'version'                  =>         '1.0'",
     "'author'                   =>         'tim burton'",
     "'author_email'             =>         'tim burton@foo.com'",
     "'keywords'                 =>         ''",
     "'license_name'             =>         'GPL'",
     "'project_name'             =>         'My Big Project'",
     "'tp2option'                =>         'tp2option'",
     "'tp2opton2'                =>         'tp2opton2'"]

the aliased project_name (tma) takes efffect in the third template
::

    >>> pprint([a for a  in tar.extractfile(t3).read().split('\n') if a.strip()])
    ["'namespace'                =>         '%(namespace)s'",
     "'nested_namespace'         =>         '%(package)s'",
     "'version'                  =>         '1.0'",
     "'author'                   =>         'tim burton'",
     "'author_email'             =>         'tim burton@foo.com'",
     "'keywords'                 =>         ''",
     "'license_name'             =>         'GPL'",
     "'project_name'             =>         'tma'",
     "'tp3option3'               =>         'Project %s'",
     "'tp3option'                =>         'Project %s'"]




Changelog
=========

1.1
----------

* documentation, because webbuilder needs to be installed in dev mode, anyhow.

1.0 
----------------
* Initial release