Thursday, May 31, 2018

Mobile BDD with Appium and Cucumber: Capturing Testing Data (Part 3)

Of a series: Part 1, Part 2.

The code for this exercise is available on the WITH_GIF branch.

One of the problems with doing automated UI testing in a CI environment is understanding failures. Today we are going to look at extending our Cucumber drivers to help with that. We are going to make a recording of what we are doing on the client side, and capture the log information from the client when there is a failure.

Cucumber for Java, like JUnit or TestNG or whatever else you might for testing has a @Before and @After annotation that you can use to set up state for a test. The thing is, the "test" here is going to be a Scenario in your Cucumber tests. We are going to start here, though, with a before and after Step bit of code, so we need to do that ourselves.  Revisiting our BaseSteps class...


private void beforeStep() {
    
}

private void afterStep() {
    
}


private void doStep(ThrowRunnable runnable) throws Exception {
    beforeStep();
    try {
        runnable.run();
    } finally {
        afterStep()
;    }
}

private interface ThrowRunnable {
    void run() throws Exception;
}

Here we have created a method we can use to wrap a step with a generic beforeStep() and afterStep() method. We will need to get these invoked, but with Java 8+ closures, this is easy. We simply do a no args call in each of our step methods.

@Then("the \"(.*)\" is gone")
public void assertMissing(String text) throws Exception {
    doStep(()->strategy.assertMissing(text));
}

Now, let's start by getting a screenshot and logs before and after each step. We will create a Recorder class with some static fields we will use to capture this information.

public class Recorder {
    private static final Logger LOGGER = Logger.getLogger(
                Recorder.class.getCanonicalName()
    );
    private static List<File> IMAGES;
    private static List<LogEntry> LOGS;
    public static void record(File file) {
        IMAGES.add(file);
    }

    public static void log(List<LogEntry> logs){
        LOGS = logs;
    }
}


Now, let's instrument our platform strategies to give us this information. For Android:

@Override
public List<LogEntry> getLogEntries() {
    return getDriver().manage().logs().get("logcat").filter(Level.ALL);
}
@Override
public File getScreenshotAsFile() {
    return getDriver().getScreenshotAs(OutputType.FILE);
}

... and iOS:

@Override
public List<LogEntry> getLogEntries() {
    List<LogEntry> allEntries = new ArrayList<>();
    getDriver().manage().logs().getAvailableLogTypes()
            .stream()
            .filter(Objects::nonNull)
            .flatMap(s -> {
                try {
                    return getDriver().manage().logs().get(s)
                                      .filter(Level.ALL).stream();
                } catch (Exception e) {
                    return Stream.empty();
                }
            })
            .filter(Objects::nonNull)
            .forEach(allEntries::add);
    allEntries.sort((o1, o2) -> Long.compare(o2.getTimestamp(), 
                                             o1.getTimestamp()));
    return allEntries;
}

public File getScreenshotAsFile() {
    return getDriver().getScreenshotAs(OutputType.FILE);
}

Since iOS has a few different log files, we need to merge them all together into a single sorted list. For Android, hey, "logcat" is probably what we want anyway. Each of the drivers will give us a screenshot to a temp file.

Now, let's revisit the beforeStep() and afterStep() we created earlier, and capture all this information.

private void beforeStep() {
    Recorder.record(strategy.getScreenshotAsFile());
}

@SuppressWarnings("unchecked")
private void afterStep() {
    Recorder.log(strategy.getLogEntries());
    Recorder.record(strategy.getScreenshotAsFile());
}

So we get a screenshot before and after each step, and record the logs after each step.

Now let's bring it all together and persist our information for failing Scenarios. We can do this by adding the @Before and @After hook annotations to our recorder class. This will create a new instance of the class, but we can still refer to the static variables.

@Before
public void initialize() {
    IMAGES = new ArrayList<>();
    LOGS = new ArrayList<>();
} @After public void finalize(Scenario scenario) throws IOException { if (scenario.isFailed()) { File outDir = new File("build/cucumber-images"); outDir.mkdirs(); outDir.mkdir(); if(IMAGES.isEmpty()){ return; } BufferedImage first = ImageIO.read(IMAGES.iterator().next()); File destination = new File(outDir, scenario.getName().replaceAll("[^\\w]", "_") + ".gif"); try ( ImageOutputStream outputStream = new FileImageOutputStream(destination); AnimatedGIFEncoder encoder = new AnimatedGIFEncoder(outputStream, first.getType(), 750, true)) { IMAGES.stream() .map(f -> {
                        try {
                            return ImageIO.read(f);
                        } catch (Exception e) {
                            throw new RuntimeException(e);
                        }
                    })
                    .forEach(i -> {
                        try {
                            encoder.writeToSequence(i);
                        } catch (IOException e) {
                            throw new RuntimeException(e);
                        }
                    });
        }
        LOGGER.info("Wrote scenario animation to " + 
                    destination.getAbsolutePath());
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ByteStreams.copy(new FileInputStream(destination), baos);
        scenario.embed(baos.toByteArray(), "image/gif");
        scenario.embed(logFile(), "text/plain");
    }
}

private byte[] logFile(){
    StringBuilder sb = new StringBuilder();
    LOGS.stream()
            .map(e-> new Date(e.getTimestamp()) + "," + 
                e.getLevel().getName() + ", " + e.getMessage()
            )
            .forEach(line-> sb.append(line).append("\n"));
    return sb.toString().getBytes(Charsets.UTF_8);
}


So in our @Before we initialize the static members. Then in the @After we finalize everything. If there are no images we can bounce. If there are, we will create an AnimatedGIFEncoder class and add all the images to it. I'm not going to get into the image processing thing, but you should pay attention to the last two methods of the finalize() method: by getting the Cucumber Scenario object passed into the method at the end, we can embed other data in the results by MIME type.

Now if we want to see the data we collect, we can add a reporting plugin to our build.gradle file:

buildscript {
    repositories {
        maven {
            url "http://repo.bodar.com"
        }
        maven {
            url "https://plugins.gradle.org/m2/"
        }
    }
    dependencies {
        classpath "com.github.samueltbrown:gradle-cucumber-plugin:0.9"
        classpath "gradle.plugin.com.github.spacialcircumstances:" +
              "gradle-cucumber-reporting:0.0.11"
    }
}

plugins {
    id 'java'
    id "com.github.samueltbrown.cucumber" version "0.9"
    id 'idea'
    id "com.github.spacialcircumstances.gradle-cucumber-reporting" version "0.0.11"
}

cucumberReports {
    outputDir = file("$project.buildDir/reports")
    buildName = '0'
    reports = files("$project.buildDir/cucumber.json")
}
// stuff here


tasks.cucumber.finalizedBy generateCucumberReports

Now when our cucumber gradle cucumber task runs, we will get a report telling us what failed, like so:



(this image not animated)

No comments:

Post a Comment