Continuous Integration support on system level

  1. Continuous Integration support on system level
    1. Startup plan
    2. Development plan
  2. Requirements
    1. Business requirements
    2. Architecture
    3. Users
    4. Data
    5. Functions
      1. build reference image
      2. create VPS
      3. setup vps
      4. install tested package and test scripts package
      5. run tests, store results in the db, dump overview to stdout
      6. cleanup vps
      7. destroy vps
    6. User interface
    7. Default configuration
    8. Hardware and software compatability
    9. Security and confidentiality
    10. License
    11. Coding language and coding style
  3. Risk profile
  4. Payment profile
  5. Future development
  6. Work progress
    1. 2007.01.09
    2. 2007.01.11
    3. 2007.01.12
    4. 2007.08.31

Stage: development

Startup plan

  1. Business requirements
  2. Software requirements
  3. Detail to tests
  4. Create initial development plan

Done on Friday Trac

Development plan

TBD - left on Friday Trac

Requirements

Business requirements

Continuous Integration on system level, in customers' environment is desperately needed by business. This is first and most important measure for quality assurance.

In short, there should be tool that would:

  1. Run on cvs/svn checkin
  2. Check out top version and run all bundled tests, including, probably, FF and IE-visual tests
  3. Report status (including errors)by email (short report) and put the report on the web (detailed version)
  4. If there were no errors, move the predefined label (e.g. Product-Current) to the version, that was checked out and tested.

It should support running in the following virtualisation environments:

  • qemu
  • xen
  • kvm
  • openvz
  • vserver

Following target OS should be finally supported:

  • ALT Linux
  • Debian GNU/Linux
  • Fedora Core
  • CentOS
  • Windows

First release should support only openvz and ALT Linux and Debian.

See also:

Architecture

Five components:

  • VPS image creation, see TotalHosting and spt
  • VPS management software, see TotalHosting
  • virtualization solution (qemu/xen/kvm/openvz/vserver)
  • test scripts to be run inside VPS
  • glue scripts

Test scripts to be run inside VPS are the responsibility of the tested product. Best are the scripts that are also shipped in the form of the package of target OS (e.g. rpm or deb).

Users

Unix users

Data

  • Reference image of target VPS
  • Script to create the reference image of a target VPS
  • Setup script to be executed in the target VPS
  • Test script
  • Teardown script
  • Script to build a package for a target VPS from svn checkout (including test scripts, preferably)

Functions

build reference image

Invocation: shell ci-buildimage <profile> <size> <image>

Uses spt to build tgz image and then transforms it into qemu image.

Event flow:

  1. System builds altlinux tgz image using supplied spt <profile>
  2. System creates qemu image <image> of the specified <size>
  3. System unwinds tgz image into qemu image and makes it bootable.

Timing: less than 30m for local repository

Tests:

  1. Spt profile is invalid.
  2. Spt gives errors during tgz build.
  3. <image> file already exists.
  4. <size> is invalid, per qemu-img specification
  5. Success test: minimal bootable alt linux configuratoin on <image>

Tickets: #1

create VPS

Invocation: shell ci-startvps <image> <qemu options>

Event flow:

  1. System starts the supplied <image> with the specified <qemu options>
  2. As soon as the <image> is booted up and is net-responsive, it returns the string to be used to connect to the root shell

Timing: less than 10m for minimal system

Tests:

  1. <image> doesn't exist
  2. <image> is not a valid qemu image
  3. <image> is not bootable
  4. <qemu options> are invalid
  5. OS boots up but doesn't respond to the network within 10 minutes
  6. Success test with the minimal bootable configuration, ssh access

Tickets: #2

setup vps

Invocation: shell ci-setupvps <setup.d dir> <connect string ...>

Event flow:

  1. System uses the <connect string...> to connect the live vps
  2. Upon successful connect, system executes all files in the <setup.d dir> in the VPS. If any of files fails, it stops execution.

Timing: less than 1m for minimal system with single ls script

Tests:

  1. VPS is not alive
  2. No scripts in <setup.d dir>
  3. Script in <setup.d dir> fails
  4. Success test.

Tickets: #3

install tested package and test scripts package

Invocation: shell ci-installpkgs <pkg list file> <connect string...>

Event flow:

  1. System uses the <connect string...> to connect the live vps, fails if it can't.
  2. System installs every packages listed in the <pkg list file> in the order they've listed. E.g. <pkg list file> contains 
    package1 package2
package3
    Then apt would be invoked for package1 package2 first and, if it is successful, for package3

Timing: less than 5m for two packages and local repository

Tests:

  1. VPS is not alive
  2. Empty <pkg list file>
  3. Package installation fails
  4. Success test.

Tickets: #4

run tests, store results in the db, dump overview to stdout

Invocation: shell ci-runtests [-k] <tests.d dir> <connect string...>

Event flow:

  1. System users the <connect string...> to connect to the live VPS, fails if it can not.
  2. System executes every script in <tests.d dir> in alpabetical order, collecting the results.
  3. If test script fails and no -k switch (same as in make -k) specified, system stops test run.
  4. System stores the stdout and strerr of test run in the database and dumps the overview (list of scripts run and their result) to stdout for further consumption (e.g. for ci-runtests ... | mail email@report invocation}}}

Timing: less than 1m for simple 'ls' test

Tests:

  1. VPS is not alive.
  2. No tests
  3. Test fails and no -k switch specified
  4. Test fail and -k switch specified
  5. Success test

Tickets: #5

cleanup vps

Invocation: shell ci-cleanup <cleanup.d dir> <connect string...>

Event flow:

  1. System uses the <connect string...> to connect to the live VPS, fails if it can not.
  2. System executes every script in <cleanup.d dir> in alphabetical order. If script fails, System stop the execution with an error message.

Timing: less than 1m for simple 'ls' script

Tests:

  1. VPS is not alive
  2. No cleanup scripts
  3. Cleanup script fails
  4. Success test w/o shutdown

Tickets: #6

destroy vps

Shutdown the vps

Invocation: shell ci-shutdown <connect script...>

Event flow:

  1. System uses the <connect string...> to connect to the live VPS, fails if it can not.
  2. System starts shutdown process and returns only when the VPS is not alive.

Timing: less than 10 minutes

Tests:

  1. VPS is not alive
  2. Shutdown is not complete for 10m
  3. Success test with shutdown

Tickets: #7

User interface

Shell

Default configuration

No data.

Hardware and software compatability

  • stock PC
  • ALT Linux Sisyphus
  • python 2.4
  • nosetests 0.9
  • qemu-0.8.2-alt1

Security and confidentiality

Unix.

License

GPL v2.

Coding language and coding style

  • python 2.4
  • PEP8
  • pylint score 8+
  • test coverage 100%

Risk profile

Below the identified project risks are enumerated.

  • Requirements change or incomplete
  • Tests are incomplete
  • Real development velocity too low comparing to initially estimated
  • Platform/solution/environment, stated in requirements, makes impossible to satisfy project goals
  • Non-development delays (e.g. waiting too long for answers)
  • Software defects in the developed Product
  • Software defects in 3rd party products

Payment profile

Project considered delivered, when included into TdLinux

Completion criteries:

  • all functions implemented
  • all tests are automated and passed
  • project is packaged as tar.bz2 and ALT rpm
  • release tar.bz2 and uploaded to KDS web site and to SourceForge
  • rpm is uploaded to Sisyphus

Future development

Work progress

2007.01.09

Done software requirements.

2007.01.11

Done tests.

2007.01.12

Done initial plan.

2007.08.31

Revived due to talk with GlobalLogic and interest from rPath