PyBeast - python mutation testing tool

  1. PyBeast - python mutation testing tool
    1. Startup plan
    2. Development plan
  2. Requirements
    1. Business requirements
    2. Architecture
    3. Users
    4. Data
    5. Functions
      1. 1. Initialize mutation set from grammar
      2. 2. Test source code mutation
    6. User interface
    7. Default configuration
    8. Hardware and software environment
    9. Security and confidentiality
    10. Style requirements
    11. License and license compatability
  3. Risk profile
  4. Payment profile
  5. Future development
  6. Work progress
    1. 2006.12.21
    2. 2006.12.23
    3. 2007.01.03
    4. 2007.09.20

Stage: development

Startup plan

  1. Business requirements (done)
  2. Spike the possible implementation of "Initialize mutation set from grammar" #353
  3. Spike the PEP8 python code generation from an AST #354
  4. Software requirements #350
  5. Tests #351
  6. Initial development plan #352

Development plan

Current development plan is in {1}.

Requirements

Business requirements

Implement mutation testing by inducing a syntacticaly correct changes (mutations) into the tested source code. After that run unit tests and make sure there's a bug. If there was no bug detected, we've got a bug in the tests.

Architecture

Simple python script.

Users

Shell user.

Data

  1. Source code under test.
  2. Tests under test.
  3. Mutation set, with the relative probability of each mutation.
  4. Language grammar.

Functions

For each:

  • call method
  • event flow
  • timings
  • tests

1. Initialize mutation set from grammar

Call method: python call

Event flow:

  1. The grammar is read in
  2. For each grammar definition statement read, the following is done:
    1. All possible grammar-correct substitutions, are generated and added to the mutation list
  3. Mutation list entries are made unique
  4. Mutations are generated as a python source code and output to the standard output with all weights made equal.

Timings: on referenece hardware the mutation set for the python grammar should be generated in less than 10 minutes. Desired time - less that a minute.

Tests:

  1. Generate a mutation set for the ASDL buildin types, see #353
  2. Parse and generate the proper mutation set for expr-subgrammar, see #353
  3. Parse expr grammar and generate the proper mutation set.
  4. Parse full python grammar and generate the proper mutation set.

Tickets: #372, #1, #2, #3

2. Test source code mutation

Call method: shell utility invocation

testmutation mutations.py src.python.module test.python.module failed-mutation/python/module/dir

Event flow:

  1. System evaluates mutations.py - the mutations list, which includes mutation weights
  2. System reads src.python.module and parses into AST
  3. System executes a single mutation over the AST.
  4. System runs test.python.module over the mutated and compiled AST. If tests didn't caught an change, system dumps mutated code into failed-mutation/python/module/dir

Timings:

  1. Parse and mutation generation over the complete body of PyBeast should take less than 1 sec.
  2. Complete process of testing a single PyBeast mutation should take less than 1 minute.

Tests:

  1. Generate AST mutation.
  2. Test AST mutation, test fails.
  3. Test AST mutation, test passes, have to dump out mutated code.

Tickets: #376, #4, #5, #6, #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, #19

User interface

Shell utilities.

Default configuration

Initial mutation set is generated basing on the complete Python grammar.

Hardware and software environment

Uses:

  • OS: ALT Linux Sisyphus
  • python 2.4.x
  • commodity PC, for testing the devel.kds.priv would be used - quite loaded AthlonX2 64

Security and confidentiality

Not an issue

Style requirements

  • Python 2.4.x
  • PEP 8
  • 100% test coverage with coverage.py v.2.6.
  • pylint score 8 or better
  • tests pass own testing for at least 10 iterations

License and license compatability

GPL v.2. Thus should use only GPL-compatible libraries.

Risk profile

Below the identified project risks are enumerated.

  • Requirements change or incomplete
  • Tests are incomplete
  • Real development velocity too low comparing to initially estimated
  • Platform/solution/environment, stated in requirements, makes impossible to satisfy project goals
  • Non-development delays (e.g. waiting too long for answers)
  • Software defects in the developed Product
  • Software defects in 3rd party products

Payment profile

Internal friday project.

Completion criteries:

  • all functions implemented
  • all tests are automated and passed
  • project is packaged as tar.bz2 and ALT rpm

Future development

Work progress

2006.12.21

Got an idea working on #286

2006.12.23

Started work on software requirements

2007.01.03

Changed mutation set generation from shell call to python call. Will use metaclasses there.

2007.09.20

Moved to new hosting