<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" /><style type="text/css"><!--
#msg dl { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fc0 solid; padding: 6px; }
#msg ul, pre { overflow: auto; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<title>[24816] trunk/WebKitTools</title>
</head>
<body>
<div id="msg">
<dl>
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/24816">24816</a></dd>
<dt>Author</dt> <dd>aroben</dd>
<dt>Date</dt> <dd>2007-08-01 23:24:50 -0700 (Wed, 01 Aug 2007)</dd>
</dl>
<h3>Log Message</h3>
<pre> When DRT crashes, record stderr and restart DRT
This prevents a DRT crash from causing the next few hundred tests to
"fail" because DRT is no longer running.
I also changed the terminology that run-webkit-tests uses in its
output a bit, so that crashing tests are referred to as "crashes"
instead of "failures".
Reviewed by Mark.
* Scripts/run-webkit-tests: Detect a crash and record it as a tool
failure.
(sub openDumpTool): Use open3 so that we can access stderr.
(sub dumpToolCrashed): Added.
(sub printFailureMessageForTest): Added.
(sub htmlForExpectedAndActualResults): Added.
(sub deleteExpectedAndActualResults): Added.
(sub recordActualResultsAndDiff): Added.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkWebKitToolsChangeLog">trunk/WebKitTools/ChangeLog</a></li>
<li><a href="#trunkWebKitToolsScriptsrunwebkittests">trunk/WebKitTools/Scripts/run-webkit-tests</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkWebKitToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/WebKitTools/ChangeLog (24815 => 24816)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/WebKitTools/ChangeLog 2007-08-02 04:21:28 UTC (rev 24815)
+++ trunk/WebKitTools/ChangeLog 2007-08-02 06:24:50 UTC (rev 24816)
</span><span class="lines">@@ -1,3 +1,25 @@
</span><ins>+2007-08-01 Adam Roben <aroben@apple.com>
+
+ When DRT crashes, record stderr and restart DRT
+
+ This prevents a DRT crash from causing the next few hundred tests to
+ "fail" because DRT is no longer running.
+
+ I also changed the terminology that run-webkit-tests uses in its
+ output a bit, so that crashing tests are referred to as "crashes"
+ instead of "failures".
+
+ Reviewed by Mark.
+
+ * Scripts/run-webkit-tests: Detect a crash and record it as a tool
+ failure.
+ (sub openDumpTool): Use open3 so that we can access stderr.
+ (sub dumpToolCrashed): Added.
+ (sub printFailureMessageForTest): Added.
+ (sub htmlForExpectedAndActualResults): Added.
+ (sub deleteExpectedAndActualResults): Added.
+ (sub recordActualResultsAndDiff): Added.
+
</ins><span class="cx"> 2007-07-30 Darin Adler <darin@apple.com>
</span><span class="cx">
</span><span class="cx"> Reviewed by Tim Hatcher.
</span></span></pre></div>
<a id="trunkWebKitToolsScriptsrunwebkittests"></a>
<div class="modfile"><h4>Modified: trunk/WebKitTools/Scripts/run-webkit-tests (24815 => 24816)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/WebKitTools/Scripts/run-webkit-tests 2007-08-02 04:21:28 UTC (rev 24815)
+++ trunk/WebKitTools/Scripts/run-webkit-tests 2007-08-02 06:24:50 UTC (rev 24816)
</span><span class="lines">@@ -55,6 +55,7 @@
</span><span class="cx"> use FindBin;
</span><span class="cx"> use Getopt::Long;
</span><span class="cx"> use IPC::Open2;
</span><ins>+use IPC::Open3;
</ins><span class="cx"> use Time::HiRes qw(time);
</span><span class="cx">
</span><span class="cx"> use lib $FindBin::Bin;
</span><span class="lines">@@ -63,6 +64,7 @@
</span><span class="cx">
</span><span class="cx"> sub openDumpTool();
</span><span class="cx"> sub closeDumpTool();
</span><ins>+sub dumpToolCrashed();
</ins><span class="cx"> sub closeHTTPD();
</span><span class="cx"> sub countAndPrintLeaks($$$);
</span><span class="cx"> sub fileNameWithNumber($$);
</span><span class="lines">@@ -74,10 +76,14 @@
</span><span class="cx"> sub splitpath($);
</span><span class="cx"> sub isTextOnlyTest($);
</span><span class="cx"> sub expectedDirectoryForTest($);
</span><ins>+sub printFailureMessageForTest($$);
</ins><span class="cx"> sub toURL($);
</span><span class="cx"> sub toWindowsPath($);
</span><span class="cx"> sub closeCygpaths();
</span><span class="cx"> sub validateSkippedArg($$;$);
</span><ins>+sub htmlForExpectedAndActualResults($);
+sub deleteExpectedAndActualResults($);
+sub recordActualResultsAndDiff($$);
</ins><span class="cx">
</span><span class="cx"> # Argument handling
</span><span class="cx"> my $shouldCheckLeaks = '';
</span><span class="lines">@@ -110,6 +116,7 @@
</span><span class="cx"> my $expectedTag = "expected";
</span><span class="cx"> my $actualTag = "actual";
</span><span class="cx"> my $diffsTag = "diffs";
</span><ins>+my $errorTag = "stderr";
</ins><span class="cx">
</span><span class="cx"> my $programName = basename($0);
</span><span class="cx"> my $launchSafariDefault = $launchSafari ? "launch" : "do not launch";
</span><span class="lines">@@ -606,7 +613,24 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>- if (!defined $expected) {
</del><ins>+ if (dumpToolCrashed()) {
+ $result = "crash";
+
+ printFailureMessageForTest($test, "crashed");
+
+ my $dir = "$testResultsDirectory/$base";
+ $dir =~ s|/([^/]+)$|| or die "Failed to find test name from base\n";
+ mkpath $dir;
+
+ open CRASH, ">", "$testResultsDirectory/$base-$errorTag.txt" or die;
+ print CRASH <ERROR>;
+ close CRASH;
+
+ deleteExpectedAndActualResults($base);
+ recordActualResultsAndDiff($base, $actual);
+
+ closeDumpTool();
+ } elsif (!defined $expected) {
</ins><span class="cx"> if ($verbose) {
</span><span class="cx"> print "new " . ($resetResults ? "result" : "test") ."\n";
</span><span class="cx"> $atLineStart = 1;
</span><span class="lines">@@ -621,8 +645,7 @@
</span><span class="cx"> print EXPECTED $actual;
</span><span class="cx"> close EXPECTED;
</span><span class="cx"> }
</span><del>- unlink "$testResultsDirectory/$base-$actualTag.txt";
- unlink "$testResultsDirectory/$base-$diffsTag.txt";
</del><ins>+ deleteExpectedAndActualResults($base);
</ins><span class="cx"> unless ($resetResults) {
</span><span class="cx"> # Always print the file name for new tests, as they will probably need some manual inspection.
</span><span class="cx"> # in verbose mode we already printed the test case, so no need to do it again.
</span><span class="lines">@@ -639,29 +662,20 @@
</span><span class="cx"> $atLineStart = 1;
</span><span class="cx"> }
</span><span class="cx"> $result = "match";
</span><del>- unlink "$testResultsDirectory/$base-$actualTag.txt";
- unlink "$testResultsDirectory/$base-$diffsTag.txt";
</del><ins>+ deleteExpectedAndActualResults($base);
</ins><span class="cx"> } else {
</span><del>- unless ($verbose) {
- print "\n" unless $atLineStart;
- print "$test -> ";
- }
- print "failed\n";
- $atLineStart = 1;
-
</del><span class="cx"> $result = "mismatch";
</span><span class="cx">
</span><ins>+ printFailureMessageForTest($test, "failed");
+
</ins><span class="cx"> my $dir = "$testResultsDirectory/$base";
</span><span class="cx"> $dir =~ s|/([^/]+)$|| or die "Failed to find test name from base\n";
</span><span class="cx"> my $testName = $1;
</span><span class="cx"> mkpath $dir;
</span><span class="cx">
</span><del>- open ACTUAL, ">", "$testResultsDirectory/$base-$actualTag.txt" or die;
- print ACTUAL $actual;
- close ACTUAL;
</del><ins>+ deleteExpectedAndActualResults($base);
+ recordActualResultsAndDiff($base, $actual);
</ins><span class="cx">
</span><del>- system "diff -u \"$expectedDir/$base-$expectedTag.txt\" \"$testResultsDirectory/$base-$actualTag.txt\" > \"$testResultsDirectory/$base-$diffsTag.txt\"";
-
</del><span class="cx"> if ($pixelTests && $diffPNG && $diffPNG ne "") {
</span><span class="cx"> $imagesPresent{$base} = 1;
</span><span class="cx">
</span><span class="lines">@@ -790,10 +804,10 @@
</span><span class="cx"> match => "succeeded",
</span><span class="cx"> mismatch => "had incorrect layout",
</span><span class="cx"> new => "were new",
</span><del>- fail => "failed (tool did not execute successfully)",
</del><ins>+ crash => "crashed",
</ins><span class="cx"> );
</span><span class="cx">
</span><del>-for my $type ("match", "mismatch", "new", "fail") {
</del><ins>+for my $type ("match", "mismatch", "new", "crash") {
</ins><span class="cx"> my $c = $counts{$type};
</span><span class="cx"> if ($c) {
</span><span class="cx"> my $t = $text{$type};
</span><span class="lines">@@ -824,17 +838,9 @@
</span><span class="cx"> for my $test (@{$tests{mismatch}}) {
</span><span class="cx"> my $base = $test;
</span><span class="cx"> $base =~ s/\.[a-zA-Z]+$//;
</span><del>- my $expectedDir = expectedDirectoryForTest($base);
- copy("$expectedDir/$base-$expectedTag.txt", "$testResultsDirectory/$base-$expectedTag.txt");
</del><span class="cx"> print HTML "<tr>\n";
</span><span class="cx"> print HTML "<td><a href=\"" . toURL("$testDirectory/$test") . "\">$test</a></td>\n";
</span><del>- if (-s "$testResultsDirectory/$base-$diffsTag.txt") {
- print HTML "<td><a href=\"$base-$expectedTag.txt\">expected</a></td>\n";
- print HTML "<td><a href=\"$base-$actualTag.txt\">actual</a></td>\n";
- print HTML "<td><a href=\"$base-$diffsTag.txt\">diffs</a></td>\n";
- } else {
- print HTML "<td></td><td></td><td></td>\n";
- }
</del><ins>+ print HTML htmlForExpectedAndActualResults($base);
</ins><span class="cx"> if ($pixelTests) {
</span><span class="cx"> if ($imagesPresent{$base}) {
</span><span class="cx"> print HTML "<td><a href=\"$base-$expectedTag.png\">expected image</a></td>\n";
</span><span class="lines">@@ -848,15 +854,17 @@
</span><span class="cx"> print HTML "</table>\n";
</span><span class="cx"> }
</span><span class="cx">
</span><del>-if ($counts{fail}) {
- print HTML "<p>Tests that caused the DumpRenderTree tool to fail:</p>\n";
</del><ins>+if ($counts{crash}) {
+ print HTML "<p>Tests that caused the DumpRenderTree tool to crash:</p>\n";
</ins><span class="cx"> print HTML "<table>\n";
</span><del>- for my $test (@{$tests{fail}}) {
</del><ins>+ for my $test (@{$tests{crash}}) {
</ins><span class="cx"> my $base = $test;
</span><span class="cx"> $base =~ s/\.[a-zA-Z]+$//;
</span><span class="cx"> my $expectedDir = expectedDirectoryForTest($base);
</span><span class="cx"> print HTML "<tr>\n";
</span><span class="cx"> print HTML "<td><a href=\"" . toURL("$testDirectory/$test") . "\">$base</a></td>\n";
</span><ins>+ print HTML htmlForExpectedAndActualResults($base);
+ print HTML "<td><a href=\"$base-$errorTag.txt\">stderr</a></td>\n";
</ins><span class="cx"> print HTML "</tr>\n";
</span><span class="cx"> }
</span><span class="cx"> print HTML "</table>\n";
</span><span class="lines">@@ -1092,7 +1100,7 @@
</span><span class="cx"> if ($useValgrind) {
</span><span class="cx"> $dumpTool = "valgrind";
</span><span class="cx"> }
</span><del>- $dumpToolPID = open2(\*IN, \*OUT, $dumpTool, @args) or die "Failed to start tool: $dumpTool\n";
</del><ins>+ $dumpToolPID = open3(\*OUT, \*IN, \*ERROR, $dumpTool, @args) or die "Failed to start tool: $dumpTool\n";
</ins><span class="cx"> $isDumpToolOpen = 1;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -1102,10 +1110,19 @@
</span><span class="cx">
</span><span class="cx"> close IN;
</span><span class="cx"> close OUT;
</span><ins>+ close ERROR;
</ins><span class="cx"> waitpid $dumpToolPID, 0;
</span><span class="cx"> $isDumpToolOpen = 0;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+sub dumpToolCrashed()
+{
+ return 0 unless $isDumpToolOpen;
+
+ my $pid = waitpid(-1, WNOHANG);
+ return $pid == $dumpToolPID;
+}
+
</ins><span class="cx"> sub openHTTPDIfNeeded()
</span><span class="cx"> {
</span><span class="cx"> return if $isHttpdOpen;
</span><span class="lines">@@ -1236,6 +1253,18 @@
</span><span class="cx"> return (-f "$platformTestDirectory/$base-$expectedTag.txt") ? $platformTestDirectory : $expectedDirectory;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+sub printFailureMessageForTest($$)
+{
+ my ($test, $description) = @_;
+
+ unless ($verbose) {
+ print "\n" unless $atLineStart;
+ print "$test -> ";
+ }
+ print "$description\n";
+ $atLineStart = 1;
+}
+
</ins><span class="cx"> my %cygpaths = ();
</span><span class="cx">
</span><span class="cx"> sub openCygpathIfNeeded($)
</span><span class="lines">@@ -1308,3 +1337,39 @@
</span><span class="cx"> die "Invalid argument '" . $value . "' for option $option" unless $validSkippedValues{$value};
</span><span class="cx"> $treatSkipped = $value;
</span><span class="cx"> }
</span><ins>+
+sub htmlForExpectedAndActualResults($)
+{
+ my ($base) = @_;
+
+ return "<td></td><td></td><td></td>\n" unless -s "$testResultsDirectory/$base-$diffsTag.txt";
+
+ return "<td><a href=\"$base-$expectedTag.txt\">expected</a></td>\n"
+ . "<td><a href=\"$base-$actualTag.txt\">actual</a></td>\n"
+ . "<td><a href=\"$base-$diffsTag.txt\">diffs</a></td>\n";
+}
+
+sub deleteExpectedAndActualResults($)
+{
+ my ($base) = @_;
+
+ unlink "$testResultsDirectory/$base-$actualTag.txt";
+ unlink "$testResultsDirectory/$base-$diffsTag.txt";
+ unlink "$testResultsDirectory/$base-$errorTag.txt";
+}
+
+sub recordActualResultsAndDiff($$)
+{
+ my ($base, $actual) = @_;
+
+ return unless length($actual);
+
+ open ACTUAL, ">", "$testResultsDirectory/$base-$actualTag.txt" or die "Couldn't open actual results file for $base";
+ print ACTUAL $actual;
+ close ACTUAL;
+
+ my $expectedDir = expectedDirectoryForTest($base);
+ copy("$expectedDir/$base-$expectedTag.txt", "$testResultsDirectory/$base-$expectedTag.txt");
+
+ system "diff -u \"$testResultsDirectory/$base-$expectedTag.txt\" \"$testResultsDirectory/$base-$actualTag.txt\" > \"$testResultsDirectory/$base-$diffsTag.txt\"";
+}
</ins></span></pre>
</div>
</div>
</body>
</html>