GCC includes a testsuite subdirectory that's kinda dejagnu
but isn't really, with some differences in .exp file naming
in the .sum, and a lack of structure in the .log files ... but
can still get something out of it.
PR32150: support multiple --target_board dejagnu runs
Reworked i-dejagnu-parser to fish out the "variations", i.e., data
from distinct --target_board=FOO{,BAR} in dejagnu logs. These clauses
result in duplicated runs of .exp expfiles, and generally duplicated
subtest names.
Prior to this patch, some random subset of the results would get
associated with the expfile, with no variation awareness. Now, the
variation string is added to the expfile name as a "[suffix]". e.g.:
"gdb.gdb/test_outcomes.exp [native-gdbserver/-m32]". Unfortunately,
this changes the analysis database naming convention, so for
consistency and testrun-to-testrun comparisons with current tooling, a
complete reindex over a testrun git repo should be considered. It also
changes i/o traditions whereever expfiles are named.
run-r-grid-testcase.sh: new test script for r-grid-testcase that runs all invocation levels (toplevel, as well as automake, dejagnu, glic for test and subtest levels) and compares the output to a predetermined outcome
r-grid-testcase: Add check to ensure the database is accessible before running any commands. Add in logging messages to show what sql query is currently being run.
Previous code falsely associated found subtest.trs/.log files with any
parent test directory (identified by presence of a test-suite.log
file), which fails especially when there are multiple nested test
directories. This doesn't seem to happen with normal automake tests,
however, valgrind's proposed tests do include nested subdirs.
r-grid-testcase: Remove parsing of the results data, instead send the query
information out in a list containing dict of every row of the query. This is to
avoid unnecessary double parsing and issues arising from converting the dict
keys to strings.
r-httpd-browse: Moved the parsing from r-grid-testcase here
r-grid-testcase: add trsfile, logfile info to the dict for automake invocations
and trfile, outfile info to the dict for glibc invocations. Add testsuite type
info to the dict.
r-httpd-browse: Add links to allow navigating to r-grid-testcase test and
subtest level views from the automake and glibc pages
Housam Alamour [Wed, 26 Jun 2024 22:15:31 +0000 (18:15 -0400)]
r-grid: change ordering of the sql queries to reflect the hierarchy of the
invocation levels (omnitest top level, test, subtest). Change the query
formatting to be more consistent
When invoked from r-httpd-browse's summary view, all the frontend
needs to know if there exist any dejagnu/etc. testsuite details, not
the details themselves. Add a new output flag family ("has....") to
r-show-testrun, and consume it in r-httpd-browse. This saves some
database traffic ergo page latency.
Housam Alamour [Tue, 25 Jun 2024 00:11:35 +0000 (20:11 -0400)]
r-grid-testcase: For top-level invocation, store the testsuite string as a tuple in
the key of the count dict, rather than in the h/v key of the cell dict as it
makes more sense for the testsuite info to be associated to each
pass/fail/other entry in each cell.
r-httpd-browse: Add support for the new top-level invocation in the grid view.
This view will shows test result information aggregated on the testrun level,
rather than individual test level.
Frank Ch. Eigler [Mon, 24 Jun 2024 19:22:54 +0000 (15:22 -0400)]
gridview: tolerate failed r-grid-testcase queries
- move keyword-enumeration logic over to r-httpd-browse
- r-grid-testcase can still fail bigly if results empty or whatever
- but r-httpd-browse will attempt to produce at least an empty grid
with a functional search form
Housam Alamour [Mon, 24 Jun 2024 18:46:31 +0000 (14:46 -0400)]
r-grid-testcase: add new invocation level --toplevel that returns test result
information aggregated on the testrun level, rather than individual test level.
This pulls information for all testsuites of each testrun from
omnitest_summary_v and presents it in the same cell-dictionary format as the test
level, but with additional testsuite information in the key of each cell entry
Housam Alamour [Fri, 21 Jun 2024 21:21:18 +0000 (17:21 -0400)]
r-grid-testcase: Add new queries for the subtests views of each testsuite.
This maintains support for the dejagnu subtest view via r-httpd-browse.
The queries for the other subtest have been implemented but a way to
select which type of testsuite query to use is still in the works.
Housam Alamour [Fri, 21 Jun 2024 19:04:51 +0000 (15:04 -0400)]
g-gridview-test-summary: new script to create a view combining
data from 'automake', 'dejagnu', and 'glibc' testsuites.
r-grid-testcase: move the gridview_test_summary_v view creation
to g-gridview-test-summary and run it as a subprocess.
Housam Alamour [Wed, 19 Jun 2024 21:28:13 +0000 (17:28 -0400)]
r-grid-testcase: update query for test level invocation to use a new view
that incorporates all testsuite information. This breaks the subtest level views.
Housam Alamour [Thu, 30 May 2024 23:52:47 +0000 (19:52 -0400)]
r-httpd-browse.in: Add ability to access difference/regression view
Add checkboxes to each of the testruns in the grid cells These can be used with the
diff/regress buttons to access the diff/regression view via r_diff_testruns()
Housam Alamour [Tue, 28 May 2024 21:05:28 +0000 (17:05 -0400)]
r-httpd-browse: natsort the table values in the grid-testcase.html template render
- Using the natsort library to sort the table_data dictionary keys in reverse order
- Removing unnecessary sorting of results in the template
- Removing unused code for storing vertical and horizontal keyword values in the dictionary
Housam Alamour [Wed, 22 May 2024 19:02:45 +0000 (15:02 -0400)]
r-grid-testcase: Add third keyword filter argument
This commit adds a new third keyword filter argument to further filter results.
The break condition for the search loop was modified to allow for a single
search expansion if no results where found in the first loop itteration
r-httpd-browse: Include form input for third keyword
This change includes a form input for the third keyword in the table.
Housam Alamour [Tue, 21 May 2024 18:04:14 +0000 (14:04 -0400)]
r-httpd-browse: Don't show subtest information on expfile table view. Move the
building of the "counts" subdictionary to r-grid-testcsae.
r-grid-testcase: Don't load
additional subtest information when in expfile view. Only load result counts
Housam Alamour [Thu, 16 May 2024 20:29:27 +0000 (16:29 -0400)]
r-httpd-browse: remove unused </div>. Change 'check' and 'X' symbols to unicode
hex in the Jinja template. Remove unneeded key from dict reading loop.
r-grid-testcase: Clean up the code by removing code
blocks that do not need to be in the while loop outside. Add
comment block to explain the main while loop. Exit loop if max
number of keywords was reached.
Housam Alamour [Wed, 8 May 2024 21:51:35 +0000 (17:51 -0400)]
r-grid-testcae: update to the SQLITE_MAX_VARIABLE_NUMBER patch to be more dynamic
and equally divide the keyword limit among the h and v keywords. Removed duplicate
variable in other_variables
The jstable.js inner sorting function now uses String.localeCompare(...., {numeric:true})
to approximate strverscmp() style natural sorting. In addition, for sorting purposes,
empty cells are interpreted as "0".
Housam Alamour [Mon, 6 May 2024 22:03:02 +0000 (18:03 -0400)]
r-grid-testcase: fix issue caused by reaching the max number of variables allowed for the version this version of sqlite. SQLITE_MAX_VARIABLE_NUMBER = 999, so the v_keywords and h_keywords have been limited to a combined total of 999 now. https://www.sqlite.org/limits.html
r-grid-testrun: Expand the grid-cell if it is too small. This is done via dynamic expansion of the search limits if number of desired cells from the vertical and horizontal limits is not reached.
New limits are based on the ratio of total cells to desired cells. Added
checks to break the loop if the maximum number of iterations is reached
or if the lengths of v_keywords and h_keywords haven't changed,
indicating all data has been read. Also implemented slicing of the results
dictionary to remove results that go over the limit. Added button to jump to the cell with the base test in it for easier navigation
r-httpd-browse: Fix box styling.
r-grid-testcase: Add h_keywords and v_keywords to the template render in r-grid-testcase
r-httpd-browse: Display any empty cells that do not contain information
r-httpd-browse (r-grid-testcase view): Fix the icons in the cells
showing the favicon of the copyleft symbol instead of the bunsen icon. Set --horizontal default of r-grid-testcase to be uname-m
r-httpd-browse (r-grid-testcase view): move the total results count for each cell to the right to make the view more compact.Fix commas not showing between the results, use bunsen favicon for the testrun icon. Fix link for each result to point to the dgexpfile view instead of dejagnu view. Add horizontal and vertical information to the hover-over/title attribute.
Change hashes in the cell to be unicode icons instead of hashes and make them link to r-grid-view tool using that testrun hash. Add clickable links to each of the result icons as well as the ability to see the gitcommit hash when you hover over via the title attribute. Add nonbreaking spaces to the grid cell text.
Housam Alamour [Tue, 20 Feb 2024 15:16:17 +0000 (10:16 -0500)]
r-grid-testcase: new tool
Introduces the r-grid-testcase tool, a Python script that fetches and
summarizes test/subtest results from the Bunsen database which are
proximate to a base, input test/subtest. Proximity is defined in 2
dimensions, the vertical and horizontal axis keywords selected from
the base test. The proximity of each test is defined as the distance
of the test from the base test in the testrun_cluster table.
The tool was designed to have its output consumed by r-httpd-browse to
create a grid-cell view of the results, although it can be used
independently. It takes in a testrun id, specific test name (expfile),
and optionally a subtest name alongside vertical and horizontal
keywords and limits on how many clusters to search. The output is a
json object where the keys are a tuple-string of the vertical and
horizontal keyword, and the value is a list that contains information
about all the tests that match those keyword filters (expfile name,
subtest name, result, etc.).
It works by first identifying the horizontal/vertical keyword values
of the base subtest, then using these keywords to build a proximate
keyword list for each dimension, then finally doing a search on the
database for test runs that match these keywords. All this is done
with multiple sql queries that may join several tables from the
database together. The relevant information is then put into a Jinja
template and sent to the output.
r-httpd-browse has been updated to include a '+' button for each test
file in the dejagnu view and a '+' button for each subtest in the
dgexpfile view. Clicking this button will open the r-grid-testcase
web-view using that test/subtest. The webview consists of a gridded
table that displays all the proximate tests. The view hides some cell
information behind buttons for the sake of compactness. The base test
run cell is coloured green and the text is bold. The web-view allows
the user to change the filter values when calling r-grid-testcase.
r-httpd-browse grid view works by calling r-grid-testcase, consuming
the output then passing the output to an html template that also makes
use of some javascript.
The current r-grid-testcase and web-view version only works with
dejagnu-style tests but compatibility with other testsuites is planned
for the future
Frank Ch. Eigler [Tue, 27 Feb 2024 19:46:42 +0000 (14:46 -0500)]
r-httpd-browse: use werkzeug multiprocess mode by default
r-httpd-browse spends a fair bit of CPU time in python land, massaging
the json data coming from lower level r-* tools. During this time, it
cannot act concurrently, even in the werkzeug threaded=True mode,
because it's all blocking python code. Add new --threaded /
--processes options, with a default on the latter, to run report jobs
in concurrent subprocesses. This allows genuine concurrency.
pipeline: run dead & new testrun jobs in an interleaved way
This way, even following a big t-untag-testruns batch, where timeouts
might kill the job before completion, the bunsen pipeline can still
make progress on new testruns too.
These scripts use complex sql operations that can cause the generation
of temporary tables (for distinct/order-by evaluation). Direct sqlite
to keep these results in memory rather than bother the OS with disk
access.