[Linaro-validation] Parse output

[Albarran, Josue]

Hi,

I ran a simple LTP-DDT test case that produces the following output http://pastebin.ubuntu.com/21280373/.
Is there a way to parse this output so that it shows up in results section of the dashboard as a pass. I know you can call lava-test-case from within the script and do the parsing inside the script, but I want to do this for many different test cases. Is there an easier way to do it without changing the test cases?

I tried the parsing in the test definition as follows http://pastebin.ubuntu.com/21280894/.

I will appreciate any help, thanks.

Josue

[Chase Qi]

Hi Josue,

Post by Albarran, Josue
Hi,
I ran a simple LTP-DDT test case that produces the following output http://pastebin.ubuntu.com/21280373/.
Is there a way to parse this output so that it shows up in results section of the dashboard as a pass. I know you can call lava-test-case from within the script and do the parsing inside the script, but I want to do this for many different test cases. Is there an easier way to do it without changing the test cases?
I tried the parsing in the test definition as follows http://pastebin.ubuntu.com/21280894/.

The parse pattern is good, it matches your ltp output. I checked with python re module manually, the output shown as below. So if you don’t see the result in LAVA results, I think it is not the problem of the pattern, might be sth else.

Maybe your are using LAVA V2, I haven’t tried to run ltp with LAVA V2.

Post by Albarran, Josue

import re
print re.match("^(?!.+ED)(?P<test_case_id>\\w+)\\s+(?P<result>PASS|FAIL)\\s+\\d+", "ETH_XS_FUNC_PING PASS 0").groupdict()

{'test_case_id': 'ETH_XS_FUNC_PING', 'result': 'PASS'}

Post by Albarran, Josue
I will appreciate any help, thanks.
Josue
_______________________________________________
linaro-validation mailing list
https://lists.linaro.org/mailman/listinfo/linaro-validation

--
Thanks,
Chase

[Albarran, Josue]

Yes, I'm currently using LAVA v2. Thanks for the information.

-----Original Message-----
From: Chase Qi [mailto:***@linaro.org]
Sent: Thursday, July 28, 2016 10:40 AM
To: Albarran, Josue
Cc: linaro-***@lists.linaro.org
Subject: Re: [Linaro-validation] Parse output

Hi Josue,

Post by Albarran, Josue
Hi,
I ran a simple LTP-DDT test case that produces the following output http://pastebin.ubuntu.com/21280373/.
Is there a way to parse this output so that it shows up in results section of the dashboard as a pass. I know you can call lava-test-case from within the script and do the parsing inside the script, but I want to do this for many different test cases. Is there an easier way to do it without changing the test cases?
I tried the parsing in the test definition as follows http://pastebin.ubuntu.com/21280894/.

The parse pattern is good, it matches your ltp output. I checked with python re module manually, the output shown as below. So if you don’t see the result in LAVA results, I think it is not the problem of the pattern, might be sth else.

Maybe your are using LAVA V2, I haven’t tried to run ltp with LAVA V2.

Post by Albarran, Josue

import re
print re.match("^(?!.+ED)(?P<test_case_id>\\w+)\\s+(?P<result>PASS|FAIL)\\s+\\d+", "ETH_XS_FUNC_PING PASS 0").groupdict()

{'test_case_id': 'ETH_XS_FUNC_PING', 'result': 'PASS'}

Post by Albarran, Josue
I will appreciate any help, thanks.
Josue
_______________________________________________
linaro-validation mailing list
https://lists.linaro.org/mailman/listinfo/linaro-validation

[Neil Williams]

On Thu, 28 Jul 2016 15:05:05 +0000

Post by Albarran, Josue
Hi,
I ran a simple LTP-DDT test case that produces the following output
http://pastebin.ubuntu.com/21280373/. Is there a way to parse this
output so that it shows up in results section of the dashboard as a
pass. I know you can call lava-test-case from within the script and
do the parsing inside the script, but I want to do this for many
different test cases. Is there an easier way to do it without
changing the test cases?

V2 docs are going to recommend that the implicit pattern at the end of
the test definition is hard to get right and this is exactly how it
presents. It is a python regex but it has to go through a variety of
format tools and debugging the pattern in LAVA takes a lot of time.

The simplest solution here is to take a log of the output, write a
custom script that reliably parses the output and which you can test
locally. Then either wrap the ltp in that or write the ltp output to a
file and process the file. It saves a lot of time in the long run.
Scripts like this will remain compatible with V1 and V2.

This doesn't involve changing the test cases and you've done most of
the work already by creating a pattern that matches locally - the same
process you used to create that pattern is the process to use to create
the wrapper script.

(This is what we're doing with the lava-server unit tests.)

Post by Albarran, Josue
I tried the parsing in the test definition as follows
http://pastebin.ubuntu.com/21280894/.
I will appreciate any help, thanks.
Josue

[Albarran, Josue]

Ok, so you're saying to use something like this script https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/ltp-ddt.sh to wrap the test call and parse the output in that script?

I see this script is being called in the following test definition: https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/ubuntu/ltp-ddt.yaml.

My goal is to have this wrapper script to be able to see a list of pass or fail with the test cases that were called.

Also, I see lava-test-case-attach is used in the script, isn't it unsupported for now in LAVA v2?

Thanks,

Josue

-----Original Message-----
From: linaro-validation [mailto:linaro-validation-***@lists.linaro.org] On Behalf Of Neil Williams
Sent: Thursday, July 28, 2016 11:08 AM
To: linaro-***@lists.linaro.org
Subject: Re: [Linaro-validation] Parse output

On Thu, 28 Jul 2016 15:05:05 +0000

Post by Albarran, Josue
Hi,
I ran a simple LTP-DDT test case that produces the following output
http://pastebin.ubuntu.com/21280373/. Is there a way to parse this
output so that it shows up in results section of the dashboard as a
pass. I know you can call lava-test-case from within the script and do
the parsing inside the script, but I want to do this for many
different test cases. Is there an easier way to do it without changing
the test cases?

V2 docs are going to recommend that the implicit pattern at the end of the test definition is hard to get right and this is exactly how it presents. It is a python regex but it has to go through a variety of format tools and debugging the pattern in LAVA takes a lot of time.

The simplest solution here is to take a log of the output, write a custom script that reliably parses the output and which you can test locally. Then either wrap the ltp in that or write the ltp output to a file and process the file. It saves a lot of time in the long run.
Scripts like this will remain compatible with V1 and V2.

This doesn't involve changing the test cases and you've done most of the work already by creating a pattern that matches locally - the same process you used to create that pattern is the process to use to create the wrapper script.

(This is what we're doing with the lava-server unit tests.)

Post by Albarran, Josue
I tried the parsing in the test definition as follows
http://pastebin.ubuntu.com/21280894/.
I will appreciate any help, thanks.
Josue

[Neil Williams]

On Fri, 29 Jul 2016 16:56:06 +0000

Post by Albarran, Josue
Ok, so you're saying to use something like this script
https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/ltp-ddt.sh
to wrap the test call and parse the output in that script?

A lot of these things are easier in languages other than shell, but
yes. That particular script only seems to do part of the job - I'm not
familiar with LTP, so can't really tell what it is trying to do but it
doesn't call lava-test-case.

From a cursory glance, something like this
https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/ltp-realtime2LAVA.py
may be a better example. Where that is printing test_case_id: it may as
well simple call lava-test-case with the same values.

Simplest thing to do is to get a complete log and run tests locally
with your own script.

Post by Albarran, Josue
https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/ubuntu/ltp-ddt.yaml.
My goal is to have this wrapper script to be able to see a list of
pass or fail with the test cases that were called.

.... which means that it can then call lava-test-case <name> --result
directly.

Post by Albarran, Josue
Also, I see lava-test-case-attach is used in the script, isn't it
unsupported for now in LAVA v2?

In this script, lava-test-case-attach is making a tarball available -
that's not the results but a log file or other artefacts. In LAVA V2,
we are implementing a publishing API which allows test jobs to push
files to configured servers for this kind of purpose.

The problem with lava-test-case-attach and V1 is that it needs special
handling on the device and is not stored in a sane way on the server.

Post by Albarran, Josue
Thanks,
Josue
-----Original Message-----
From: linaro-validation
Parse output
On Thu, 28 Jul 2016 15:05:05 +0000

Post by Albarran, Josue
Hi,
I ran a simple LTP-DDT test case that produces the following output
http://pastebin.ubuntu.com/21280373/. Is there a way to parse this
output so that it shows up in results section of the dashboard as a
pass. I know you can call lava-test-case from within the script and
do the parsing inside the script, but I want to do this for many
different test cases. Is there an easier way to do it without
changing the test cases?

V2 docs are going to recommend that the implicit pattern at the end
of the test definition is hard to get right and this is exactly how
it presents. It is a python regex but it has to go through a variety
of format tools and debugging the pattern in LAVA takes a lot of time.
The simplest solution here is to take a log of the output, write a
custom script that reliably parses the output and which you can test
locally. Then either wrap the ltp in that or write the ltp output to
a file and process the file. It saves a lot of time in the long run.
Scripts like this will remain compatible with V1 and V2.
This doesn't involve changing the test cases and you've done most of
the work already by creating a pattern that matches locally - the
same process you used to create that pattern is the process to use to
create the wrapper script.
(This is what we're doing with the lava-server unit tests.)

Post by Albarran, Josue
I tried the parsing in the test definition as follows
http://pastebin.ubuntu.com/21280894/.
I will appreciate any help, thanks.
Josue

--
Neil Williams
=============
http://www.linux.codehelp.co.uk/