This is the mail archive of the systemtap@sources.redhat.com mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

RE: got any tomatoes


We are exploring this at Intel; I'm optimistic something
will come together ... Seems there's an OSDL testing site
that might be useful in a number of ways.

What about IBM and RedHat?

Brad


-----Original Message-----
From: Karen Bennet [mailto:bennet@redhat.com] 
Sent: Tuesday, May 10, 2005 4:15 AM
To: Chen, Brad
Cc: systemtap@sources.redhat.com
Subject: Re: got any tomatoes



Comments below .  This is great start of mapping
out the testing plan/objectives and issues.  Thanks for
pulling this together.

Chen, Brad wrote:

>Here is the draft test plan outline. Feedback would be
>much appreciated. Depending on feedback I'll look for 
>an appropriate home for this doc in the CVS repository.
>
>Key issues for everybody to think about:
>- How can we prioritize our testing efforts to get the
>  best possible results from the limited test resources
>  we will have at our disposal?
>- What existing projects have test infrastructure we'd
>  like to leverage? How can we motivate them to help us?
>
Good question; is there any existing test infrastructure within
IBM and/or Intel that could be used? 

>
>Brad
>
>
>
>  
>
>-----------------------------------------------------------------------
-
>
>
>Systemtap
>High Level Test Plan
>Brad Chen
>brad dot chen at intel dot com
>May 2005
>
>1. OVERVIEW
>
>Testing is going to be really important in this project.
>
>1.1 Prioritizing Test Activities
>
>Safety is a critical success factor for Systemtap; many people will
>criticize or reject the system if it is conspicuously less safe than
>DTrace.  It must be improbable/impossible that a system crash occur
>because of Systemtap. Quality is a key concern for safety, and that
>makes testing a high priority for this project. It must be improbable
>that a system crash occur because of a Systemtap bug.
>
>This tends to suggests giving highest priority to tests and bugs
>related to a potential system corruption issue.
>
>1.2 Recruiting Collaborators
>
>It would be incredibly helpful if existing kernel and performance
>tools test teams would consider incorporating Systemtap tests into
>their existing test infrastructure. Who might we work with on this?
>What kind of bribes would it require? Candidates:
>- Distribution Vendors (RedHat, ...)
>- Big OEMs (IBM, SGI, HP, ...)
>- ???
>
We might want to consider universities as a possibility
to help with testing.

Any chance that OSDL and/or LSB groups would
be interested in helping with the testing aspect
of SystemTap and

>
>1.3 Setting Realistic Goals
>
>The sad truth is we probably won't be able to do all the testing
>we'd like to. How can we prioritize tests effectively so our
>limited resources go to doing the most important testing first?
>

>
>2. HIGH LEVEL TEST PLAN
>
>2.1 Unit Tests. Confirm correct functioning of each component in
isolation
>Purpose: Test components in isolation. Find most bugs (as many as
>possible) during software development rather than end-to-end
>testing.
>
>2.2 Kernel Tests
>Purpose: Make sure the kernel builds, boots, runs correctly for
>standard kernel test suite, and is crashproof. 
>Should include an additional set of unit tests and regression tests
>that require installing Systemtap and booting the kernel.
>
Are we going to do this testing on the different kernels?
Do we see that the same tests would be used for
virtualization /systemtap testing?

>
>2.3 Integration Test
>Purpose: Comprehensive test of the correct functioning of Systemtap
>Installation/config testing. That the system can be installed and
>configured on all platforms.
>Functional Test matrix defined by platforms x scripts x workloads
>Scaleability testing defined by platforms x (scripts, workloads) pairs
>Error handling test matrix defined by platforms x error cases
>-- includes portability, scaleability, error case testing
>

>
>2.4 Stress Test
>Stress test matrix defined by platforms x scripts x workloads 
>
>2.5 Platforms
>All tests are assumed executed independently on each platform by the
>platform owner.
>
>- Processor Architectures:
>	x86
>	x86-64
>	Itanium
>	PPC
>- Operating Systems:
>	RedHat EL4 U2
>	other Linux releases to follow...
>- MP configurations:
>	uniprocessors, dual-core, HT, multiprocessors up to 8-way
>
>Responsibilities
>- IA32: RedHat, IBM, Intel to test
>- x86-64: RedHat, IBM, Intel to test
>- PPC: IBM to test
>- Itanium: Intel to test
>
Vara,  Are there going to be any plans to test on s390?

>
>3. UNIT TESTS
>
>* All major components will have unit testing infrastructure,
>  maintained by the component owner.
>* Unit tests will be executable as a part of the build process.
>  Suggested target: "make unittest"
>* Kernel components should provide test jigs to allow user-level
>  testing whenever possible.
>* Unit testing will include a regression suite. The regression
>  suite should cover all fixed bugs for the component.
>* Unit tests must cover both functionality for correct scripts
>  as well as error handling
>
>Each of the following components should create and maintain their
>own unit tests and unit testing plan:
>3.1 parser
>3.2 translator
>3.3 runtime
>3.4 kprobes
>3.5 validator
>3.6 tapset infrastructure
>3.8 tapsets
>
>4. Kernel Test
>
>4.1 Build, boot, and Systemtap unit tests
>4.2 Pass on a subset of ltp
>  Suggestion: ltp-20050405
>  http://ltp.sourceforge.net
>  http://prdownloads.sourceforge.net/ltp/ltp-full-20050405.tgz?download
>4.3 Systemtap small: Pass on ltp subset with each of the following
>  Systemtap scripts:
>  - systemcall profiler
>  - scheduler profiler
>  - statistical call-graph
>
>5. Integration Test
>
>5.1 Installation testing
>
>5.2 Functional test matrix
>
>5.2.1 Test Scripts
>- System call profiler
>  - systemwide
>  - by process
>  - other variants
>- Statistical call graph profiler
>  - time-based
>  - event-based
>- Scheduler activity tracer
>- Lock activity tracer
>- Arbitrary simultaneous combinations of the above
>
>5.2.2 Workloads
>subset of ltp
>
>5.3 Scaleability testing
>Test MP platforms, largest first, on the following script/workload
>pairs:
>- TBD...
>
>5.4 Error case testing.
>A growing list of error cases to check:
>- infinite recursive: direct, indirect
>- infinite loop
>- division by zero
>- underflow, overflow
>- stack overflow
>- excessively large array
>- invalid pointers, array bounds errors (hopefully impossible to test)
>- memory read/write restrictions
>- control flow restrictions
>- bogus runtime
>- illegal instrumentation (in Systemtap runtime, etc.)
>- kernel stack overflow
>- large number of threads
>- frequent thread affinity changes
>- ...
>
>6. Stress Testing
>
>Stress testing uses the same platforms and test scripts as the
>Integration tests, with some additional test scripts and a 
>different set of workloads.
>
>6.1 Scenarios for Stress testing
>
>- Increase interrupt frequency until failure, single timer-based probe
>- Increase interrupt frequency until failure, all kernel functions
(plus
>  timer-based probe if necessary)
>- Instrument all possible taps and then run LTP subset.
>- Find the worst-case case signal-stack kernel code 
>  (probably a driver), and instrument it with kprobes/systemtap to see
>  what happens.
>- max number of probes
>- max probe frequency: one probe
>- max probe frequency: many probes
>
Question:  What is the max number of probes for SystemTap?

>
>======================================================================
>NOTES
>
>Test workloads
>- Oracle
>- PostgreSQL
>- MySQL
>- kernel stress tests
>
Vara,  Is IBM planning to test DB2 ?

>
>References:
>	http://ltp.sourceforge.net
>	http://ltp.sourceforge.net/25_testplan.php
>	http://ltp.sourceforge.net/execmatrix.php
>
http://prdownloads.sourceforge.net/ltp/ltp-full-20050405.tgz?download
>  
>



Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]