On 2 August 2016 at 11:30, Milosz Wasilewski
Post by Milosz WasilewskiPost by Neil WilliamsOn 2 August 2016 at 10:09, Milosz Wasilewski
Post by Milosz WasilewskiHi,
Chase did an excellent job and put together a piece of code that
allows us local execution of lava-test-shell. This means we can use
our 'lava' test definitions on the boards that are not deployed in any
- prototyping tests for LAVA
- semi-automated execution of tests on targets that are not deployed in any LAB
Major part of this code is taken directly from lava dispatcher. There
are slight modifications but I would like to keep them to minimum or
remove at all (if possible). So the question follows - is there a way
to achieve the same goal with only LAVA code?
For the migration to V2, we kept compatibility with the lava test
shell definitions - as far as runtime operation is concerned. Once the
V1 code is removed, we do have plans for improving lava test shell in
ways which will break compatibility, but changes cannot be made until
then. In the meantime, the advice being written into the documentation
is to prepare custom scripts in the test definition repository which
can do all the parsing and other work in a self-contained way which
can be tested locally against log files from the device. This is a
erm, do you suggest not to use LAVA at all?
To not be held back by Lava Test Shell, which is part of the test
action part of LAVA.
LAVA gets you to the prompt of a login shell and provides some helpers
for LAVA-specific information but I am convinced that the rest of
lava-test-shell needs to be improved to make it more flexible for
other test suites, other deployments and for non-POSIX systems. The
lava-test-shell definition can - and arguably should - become just a
few lines calling existing scripts with parameters.
I'm suggesting not to write tests which rely upon Lava Test Shell
beyond the bare minimum of what's necessary for automation. To write
custom scripts that can do the same thing on a different architecture
and come up with similar results, supporting testing locally in a
clean environment without needing anything from LAVA to be installed
locally.
Post by Milosz WasilewskiPost by Neil Williamsmore robust way of ensuring that operations happen as expected.
lava-test-shell is primarily intended for very simple command
operation. lava-test-case itself will always have problems with much
more than a single command with arguments to be able to pass in
parameters from the test job. When things become sufficiently complex
that the test definition YAML actually needs to be debugged, it is
I totally disagree here. Practice shows that the pretty simple steps
might result in errors that don't happen locally.
https://git.linaro.org/qa/test-definitions.git/commit/74dcae69247c5741921ec8006e2723b39991fa61
? That just looks like the test environment was not clean. If by
"local" you mean a dirty user system instead of a clean chroot / VM or
other way of emulating what any automation actually does (gives you a
clean environment) then this just looks like the remote end changed
behaviour and the local test didn't catch it because it used an
existing copy instead of installing fresh as the test must do.
What I'm getting at is this: What's necessary for reproducing a LAVA
test is a series of log files of the output that you expect to get
from the test operation(s), quite possibly from a completely different
architecture. A custom script which uses a sensible language can then
do lots of reliable, clever parsing of the log to get exactly the
right answers for that test operation. Where possible, use a script
that someone else has already written for this or if none exists,
write it so that it can be contributed upstream, without reference to
LAVA. That stage can be done without any access to the device itself -
the values will change but the format of the output should be the same
(or else it's a bug in the test operation). That script can also do
package installation and other setup tasks, equivalent to what would
be needed to run up the test on a different architecture in a clean
chroot environment. This moves all the complex work into a script
which can be easily run in a local clean environment, regardless of
the architecture/device. Let LAVA do the device-specific & automation
stuff and give you a prompt on the device with your script available
and at least the same list of packages installed as in your clean
local chroot. From that point on, the test operation itself - whether
it's LTP or something else - is handled by the same script as is used
in the local environment. It's trivial to check for lava-test-case in
the $PATH or ../bin/ directory, at which point the script can print
the results or call lava-test-case --result with the results of the
parsing. The Lava Test Shell definition then becomes not much more
than calling that script with parameters - 2 lines at best. The script
does the download of any code needed, the compilation, the execution
and the parsing - exactly as it would in a clean local chroot
environment.
It's about testing one thing at a time - when your own custom script
does all the work, it is trivial to test that script locally - it is
also trivial to use that script in a variety of other test
environments, not just LAVA. Lava Test Shell is a helper, an enabler.
It was never intended to become a limitation on running the tests
elsewhere or to end up needing a version of itself to be available for
local installation. That is something we need to fix in Lava Test
Shell - it needs to stop tempting people into relying on it to such a
degree that it makes the test definition itself non-portable.
Portability is a good thing and custom scripts are the way to achieve
that - especially as all the necessary support for this is already
available and has been for some years.
Tests run in LAVA need to be thought of as code which is intended to
be upstreamed, but not into LAVA. LAVA needs to do only the minimum
operations required to get the test running under automation and then,
critically, get out of your way as test writer. This means that
running the same test locally is part of the design and needs nothing
from LAVA. We're making progress with this principle in the V2 code
and the tests themselves can benefit in the same way.
Post by Milosz WasilewskiPost by Neil Williamsbetter to move that logic into another script, written in a language
with genuine support for conditionals, logic, arrays and dictionaries.
The script/program can then call lava-test-case and other support
scripts or output the results if not running in LAVA. Such scripts can
improve the package installation work as well by only installing
something if the parameters actually require it. The emphasis will
shift to doing less in lava-test-shell in the same way that V2 has
removed the assumptions and misguided helpers in the rest of the code,
to allow the test shell to have easier support for alternative test
platforms and utilities.
Lava test shell actually provides little benefit for targets not
deployed in LAVA, apart from possibly familiarity for current LAVA
test writers, over a common stand-alone script.
I'll play devil's advocate here - it's not lava test shell I'm after
but the ability to run existing tests. I agree that the tests aren't
perfect and we're already trying to remove all dependencies on LAVA.
But this will take time and the results are expected 'in the
meantime'.
Exactly - lava-test-shell will (eventually) stop getting in the way of
running existing tests in LAVA. In the meantime, custom scripts which
work in a clean environment locally and in LAVA are the best approach
and can already be supported. The lava-test-shell definition then just
becomes run: steps: ./scripts/process.py --param value --param2 value2
lava-test-shell is just a helper and when that becomes a limitation,
the correct solution is to do the work in a more competent language
because, as test writer, you control which languages are available.
lava-test-shell has to try and get by with not much more than busybox
ash as the lowest common denominator. Please don't expect it to do
everything - let it provide you with a directory layout containing
your scripts, some basic information about the job and a way of
reporting test case results - that's about all it should be doing
outside of the Multinode API. Then do the rest in ways that allow for
simple local testing in a clean environment.
This allows us to then move lava-test-shell and the test action
support in LAVA V2 into areas where there is no shell and to make it
easier to work with other, established, test harnesses.
There shouldn't be a need for test definitions to rely on LAVA pattern
parsing and fixup dictionaries or much more than a basic set of
dependencies to install - that is under the control of the test writer
and adopting that into the test scripts makes it possible to run that
test outside of LAVA too.
This is what we're doing with the LAVA tests on the LAVA software
itself - to make it easier to run those tests in Debian and other
environments where the software is the same but the architecture /
device does not matter so much.
Post by Milosz WasilewskiPost by Neil WilliamsIn LAVA V2, if you have a local instance, there is no need to
replicate the code from lava_test_shell, the complete overlay is
available as a tarball and that's exactly the same tarball - with any
V2 changes and deployment_data changes - as was used in the test.
We're still working out the best way to expose that overlay, currently
it is only built on the worker. However, the advice remains the same
that tests definitions which are sufficiently complex to need
prototyping or debugging should go into custom scripts running against
log files which can then be switched to read from stdin or similar.
Problem with this approach is one has to install LAVA v2 somewhere.
The problem we're trying to solve is running tests on remote targets
that can't be accessed by other means. There is no easy access to
PDU/IPMI. Logging to console requires custom scripts.
That's just to get hold of the overlay.tar.gz. With a custom script
that does the work, you just need the output of the test itself,
whether it's LTP or something else. You shouldn't need a local
lava-test-shell or the overlay - only a few log files of the expected
/ real output and a local clean environment which can be used to do
the setup.
--
Neil Williams
=============
***@linaro.org
http://www.linux.codehelp.co.uk/