Metrics

JET has a system for test cases and test setups to communicate metrics to the test infrastructure, so that they end up in the report files.

A metric is a combination

A name (e.g. "throughput", "free memory", "user cpu")
A value (e.g. 64, 34, 42)
A measurement unit (e.g "TPS", "MB/s", "secs", "%")
A time this metric has been reported on

Examples of metrics can be, but is not limited to:

Throughput
Response time
CPU load
Memory usage
Network utilization
...

This is how metrics end up in the XML file:

   <metrics>
     <metric id="metric_0">
       <name>threads</name>
       <value>2</value>
       <unit>thread</unit>
       <timestamp>2010-03-22T15:56:32</timestamp>
     </metric>
     <metric id="metric_2">
       <name>sys</name>
       <value>0.000000</value>
       <unit>second</unit>
       <timestamp>2010-03-22T15:56:32</timestamp>
    </metric>
  </metrics>

jet~framework/jet-example-memcached has an example of load testing and metrics, check the source there for more information than this page provides.

Manual Metrics

You can add metrics manually from e.g. test cases or setups. This is handy if you want to parse output from some other command or run a thread in the background that collects metrics as your test runs.

 addMetric(new PerformanceMetric("avg_set", "123", "usec"));

You can also add attributes to the metrics

 PerformanceMetric metric = new PerformanceMetric("load", "60", "%")
 metric.addAttribute("host", getHostName())
 metric.addAttribute("phase", "cooldown")

MemcachetestTestCase.java provide an example of manual metrics.

Metrics from the load framework

The load framework in JET generates metrics for its performance.

Examples:

 [INFO] jetbatch-err: 24/01 13:46:49.638 INF Thr{0018} RunJET.gatherTestResult: There were 12 metrics:
 [INFO] jetbatch-err: 1) transactions: 315 transaction
 [INFO] jetbatch-err: 2) commits: 315 transaction
 [INFO] jetbatch-err: 3) unhandled_exceptions: 0 exception
 [INFO] jetbatch-err: 4) statements_prepared: 0 statement
 [INFO] jetbatch-err: 5) statements_executed: 0 statement
 [INFO] jetbatch-err: 6) commits: 315 transaction
 [INFO] jetbatch-err: 7) rollbacks: 0 transaction
 [INFO] jetbatch-err: 8) retries: 0 transaction
 [INFO] jetbatch-err: 9) reconnects: 0 count
 [INFO] jetbatch-err: 10) latency: 0 ms
 [INFO] jetbatch-err: 11) total_time: 501 ms
 [INFO] jetbatch-err: 12) throughput: 628.7425 tps

   <metric id="metric_0">
     <name>transactions</name>
     <value>315</value>
     <unit>transaction</unit>
     <timestamp>2011-01-24T13:46:46</timestamp>
   </metric>
   <metric id="metric_1">
     <name>commits</name>
     <value>315</value>
     <unit>transaction</unit>
     <timestamp>2011-01-24T13:46:46</timestamp>
   </metric>