You are here: Home / Documentation / How To's / How to protect yourself from buildout problems

How to protect yourself from buildout problems

by nguyen — published Aug 27, 2010 10:35 AM, last modified Aug 16, 2016 10:50 AM
procedures for testing egg/product changes/additions made with buildout or easy_install

Most people who run buildout have at least on one occasion had it fail in the sense that it would grab a slew of new updated eggs that they never actually intended!  In that respect it's always risky to run buildout on a Zope you need to have running because people depend on it.  

Here we have established procedures to prevent our production Zopes from failing because of such issues:

  • we require that egg changes be tested first on a clone of the production Zope; 
  • we do not run bin/buildout and easy_install directly -- we have simple scripts that first check in (we currently use RCS locally) the relevant files (buildout.cfg, easy_install.pth - should add versions.cfg too, now that I think about it) before running bin/buildout or easy_install, so that if the result causes a mess we can roll back the change by going back to the known-good versions of buildout.cfg and easy_install.pth
  • we do not allow developers to test anything on production Zopes or Plone sites... they must develop and test using their local Zope/Plone (on their laptop), and when the time comes to do integration testing, we do so on the cloned Zopes, and only if everything goes well *and* the updated/new eggs/products are thoroughly tested there do we make the same change in our production Zopes
We put all the scripts below in /root/bin and run them out of there.
Here is an example of our script:

# check in buildout.cfg in case we need to roll back later
echo "(cd $DIR; ci -l -w$final_user -m$0 buildout.cfg)"
(cd $DIR; ci -l -w$final_user -m$0 buildout.cfg)

(cd $DIR; bin/buildout)
Here is an example script:
[ "$ARG" = "" -o "$ARG" = "help" -o "$ARG" = "-h" -o "$ARG" = "--help" -o "$ARG" = "-u" -o "$ARG" = "?" ] && (echo "$0 [ -u | -h | help | --help ] ( productname | productURL ) "; echo ""; echo "    help: show this message"; echo "    provide a product name eg Products.PloneFormGen or a full URL to an egg eg."; echo "" ) && exit 0

# check in easy-install.pth in case we need to roll back later                                                                                                              
echo "(cd /UWO/Plone-3.1-01/lib/python2.4/site-packages; ci -l -w$final_user -m$0 easy-install.pth)"
(cd /UWO/Plone-3.1-01/lib/python2.4/site-packages; ci -l -w$final_user -m$0 easy-install.pth)

echo "Running /opt/Plone-3.1-01/Python-2.4/bin/easy_install --prefix=/UWO/Plone-3.1-01/ --always-unzip $ARG"
/opt/Plone-3.1-01/Python-2.4/bin/easy_install --prefix=/UWO/Plone-3.1-01/ --always-unzip $ARG
Here is how we clone a Zope via the rsync command.  I left the actual invocation of the rsync command commented out and just echo the command because I'm paranoid about actually running rsync -- it can really cause I/O thrashing so we run it at off peak hours.  When I am ready to run it for real I copy and paste the command to the shell.  You may also wonder why in this example we are rsync'ing two directory structures: that is because for our early production Zopes we were trying to keep our default Plone eggs and products (maintained with buildout) separate from the third party and custom ones (which are installed with easy_install).  
# Make a working clone on a remote system                                                                                                                                   

#[ -e ${DEST}.old ] && (echo "Removing old dir ${DEST}.old "; rm -rf ${DEST}.old)                                                                                           
#[ -e ${DEST} ] && (echo "Renaming ${DEST} to ${DEST}.old"; mv ${DEST} ${DEST}.old)                                                                                         
#echo "Copying $SOURCE to $DEST"; cp -pr $SOURCE $DEST                                                                                                                      
echo rsync -av --delete $SOURCE $DEST
#rsync -av --delete $SOURCE $DEST                                                                                                                                                    

#[ -e ${DESTUWO}.old ] && (echo "Removing old dir ${DESTUWO}.old "; rm -rf ${DESTUWO}.old)                                                                                  
#[ -e ${DESTUWO} ] && (echo "Renaming ${DESTUWO} to ${DESTUWO}.old"; mv ${DESTUWO} ${DESTUWO}.old)                                                                          
#echo "Copying $SOURCEUWO to $DESTUWO"; cp -pr $SOURCEUWO $DESTUWO                                                                                                          
echo rsync -av --delete $SOURCEUWO $DESTUWO
#rsync -av --delete $SOURCEUWO $DESTUWO                                                                                                                                              

#echo "Backing up the data files with repozo"; $FSBACKUPSCRIPT