bUnit version of testing-library.com

[egil]

In the front-end testing world, https://testing-library.com/ API surface has become very popular.

As far as I can see, it's a bunch of helper methods that allow users to easily traverse the DOM, interact with it, and assert against it. All the core APIs should be fairly straightforward to support based on bUnit's API surface, the queries, the firing of events, the async "wait for", e.g. for elements to appear or disappear, accessibility checks, etc.

The cheat sheet for the DOM Testing Library can be found here: https://testing-library.com/docs/dom-testing-library/cheatsheet

Below are the initial design and docs for bUnit's version of Testing Library. The additional APIs would be part of the bunit.web project, as the are all related to interacting with the rendered DOM:

---

Querying and interacting with rendered markup

bUnit includes methods that make it easy to query, interact, and use the rendered markup (DOM), similar to how a user would. They are inspired by the https://testing-library.com library from the JavaScript world. Check out their Guiding Principles for additional tips for writing good tests using the assertion methods in this library.

Mapping between Testing Library and bUnit conventions

Testing Library	bUnit	Description
getBy*	Find*	Returns the matching node for a query, and throw a ElementNotFoundException exception if no elements match or if more than one match is found (use FindAll* instead if more than one element is expected).
queryBy*	Find*OrDefault	Returns the matching node for a query, or null if no elements match. This is useful for asserting an element that is not present. Throws an error if more than one match is found (use FindAll* and assert the returned collection is empty instead, if this is OK).
findBy*	Find*Async	Returns a Task that completes when the matching element is found which matches the given query. Throws a TimeoutException if no element is found or if more than one element is found after a default timeout of 1000ms. If you need to find more than one element, use FindAll*Async.
getAllBy*	FindExactly*	Returns a collection of exactly the number of matching nodes for a query. If a different amount of elements are found an exception is thrown.
queryAllBy*	FindAll*	Returns a collection of all matching nodes for a query. The collection is empty if no elements are found.
findAllBy*	FindAll*Async	Returns a Task that completes when any matching elements are found which matches the given query. Throws a TimeoutException if no elements are found after a default timeout of 1000ms.
-	FindExactly*Async	Returns a Task that completes when exactly the number of matching elements are found for a given query. Throws a TimeoutException if no elements are found after a default timeout of 1000ms.

As the table above shows, the method names are slightly simplified in bUnit compared to Testing Library, and more aligned with what one would expect in .NET / C# code, e.g. using FindOrDefault instead of queryBy*.

Using queries

The query methods are available on the IRenderedFragment type and IInputElement types. When a query method is issued on an IRenderedFragment, it will search the entire component tree for matching elements. When a query method is issued on an IInputElement, it will only search the children of that element.

Query options

Each query method has a default set of options that can be overridden by passing in an options object. The options object is optional, and if not provided, the default options will be used.

TextMatch delegate

Most of the query methods can be passed a TextMatch, in addition to a string or a RegEx type.

The TextMatch delegate is used to specify how a given text should be matched if the default string comparer should not be used. This is the signature for TextMatch:

public delegate bool TextMatch(string? text, IInputElement? element);

Where text is the text that should be matched, and element is the element that contains the text. The element parameter is optional and can be null if the text is not contained in an element.

TODO: is this possible in bUnit?

Matching text (precision)

Instead of the text matching feature of Testing Library, bUnit uses the StringComparison enum to specify how to match text. The default is StringComparison.Ordinal, which is the same as the default in Testing Library, but you can also use StringComparison.OrdinalIgnoreCase if you want to ignore case or any of the other options for string comparison available in .NET.

Normalization

Similar to how MarkupMatches normalize text before comparison, the query methods will normalize the text before matching. By default, normalization consists of trimming whitespace from the start and end of text and collapsing multiple adjacent whitespace characters into a single space.

If you want to prevent that normalization or provide alternative normalization (e.g. to remove Unicode control characters), you can provide a normalizer delegate in the options object. This function will be given a string and is expected to return a normalized version of that string.

The signature for the normalizer delegate is:

public delegate string TextNormalizer(string text);

Wait for options

The WaitForOptions class is used to specify how long to wait for an element to appear in the DOM, and how often to check for the element. The default is to wait for 1000ms and check every 50ms.

public record class WaitForOptions
{
    public static readonly WaitForOptions Default = new();

    public TimeSpan Timeout { get; init; } = TimeSpan.FromSeconds(1);
    public TimeSpan Interval { get; init; } = TimeSpan.FromMilliseconds(50);
}

Previously, the WaitFor* methods in bUnit only retried the wait predicate after every render. I am not sure if it makes sense to introduce a recurring retry interval like Testing Library provides, but it is something to consider.

In addition, the OnTimeout?: (error: Error) => Error call back does not seem to be useful in C#/.NET, and neither does mutationObserverOptions?: MutationObserverInit.

The core Find* method

bUnit ships with Find(string cssSelector) and FindAll(string cssSelector) methods already, and it makes sense to continue having these. They are updated to match the structure described above, and replace the WaitForElement and WaitForElements methods:

public IInputElement Find(string cssSelector);
public IInputElement? FindOrDefault(string cssSelector);
public Task<IInputElement> FindAsync(string cssSelector, WaitForOptions? waitForOptions = null);
public IReadOnlyList<IInputElement> FindAll(string cssSelector);
public Task<IReadOnlyList<IInputElement>> FindAllAsync(string cssSelector, WaitForOptions? waitForOptions = null);
public IReadOnlyList<IInputElement> FindExactly(string cssSelector, int exactCount);
public Task<IReadOnlyList<IInputElement>> FindExactlyAsync(string cssSelector, int exactCount, WaitForOptions? waitForOptions = null);

ByRole

Check out the ByRole query in Testing Library for a description of the ByRole query and its options.

Here is the C# API for the ByRole query:

public IInputElement FindByRole(string role, ByRoleOptions? options = null);
public IInputElement FindByRole(TextMatch roleMatcher, ByRoleOptions? options = null);
public IInputElement FindByRole(RegEx roleRegex, ByRoleOptions? options = null);

public IInputElement? FindByRoleOrDefault(string role, ByRoleOptions? options = null);
public IInputElement? FindByRoleOrDefault(TextMatch roleMatcher, ByRoleOptions? options = null);
public IInputElement? FindByRoleOrDefault(RegEx roleRegex, ByRoleOptions? options = null);

public Task<IInputElement> FindByRoleAsync(string role, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IInputElement> FindByRoleAsync(TextMatch roleMatcher, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IInputElement> FindByRoleAsync(RegEx roleRegex, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);

public IReadOnlyList<IInputElement> FindAllByRole(string role, ByRoleOptions? options = null);
public IReadOnlyList<IInputElement> FindAllByRole(TextMatch roleMatcher, ByRoleOptions? options = null);
public IReadOnlyList<IInputElement> FindAllByRole(RegEx roleRegex, ByRoleOptions? options = null);

public Task<IReadOnlyList<IInputElement>> FindAllByRoleAsync(string role, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IReadOnlyList<IInputElement>> FindAllByRoleAsync(TextMatch roleMatcher, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IReadOnlyList<IInputElement>> FindAllByRoleAsync(RegEx roleRegex, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);

public IReadOnlyList<IInputElement> FindExactlyByRole(string role, int exactCount, ByRoleOptions? options = null);
public IReadOnlyList<IInputElement> FindExactlyByRole(TextMatch roleMatcher, int exactCount, ByRoleOptions? options = null);
public IReadOnlyList<IInputElement> FindExactlyByRole(RegEx roleRegex, int exactCount, ByRoleOptions? options = null);

public Task<IReadOnlyList<IInputElement>> FindExactlyByRoleAsync(string role, int exactCount, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IReadOnlyList<IInputElement>> FindExactlyByRoleAsync(TextMatch roleMatcher, int exactCount, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IReadOnlyList<IInputElement>> FindExactlyByRoleAsync(RegEx roleRegex, int exactCount, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);

// See https://testing-library.com/docs/queries/byrole/#options for 
// description of each of the options.
public record class ByRoleOptions
{
    public static readonly ByRoleOptions Default = new();

    public StringComparison ComparisonType { get; init; } = StringComparison.Ordinal;
    public bool Hidden { get; init; } = false;
    public bool? Selected { get; init; }
    public bool? Checked { get; init; }
    public bool? Current { get; init; }
    public bool? Pressed { get; init; }
    public bool? Expanded { get; init;}
    public bool QueryFallbacks { get; init; } = false;    
    public int? Level { get; init; }
    public TextMatch? Name { get; init; }
    public TextMatch? Description { get; init; }
    public TextNormalizer? Normalizer { get; init; }
}

ByLabelText

Check out the ByLabelText query in Testing Library for a description of the ByLabelText query and its options.

Here is the C# API for the ByLabelText query:

public IInputElement FindByLabelText(string role, ByLabelTextOptions? options = null);
public IInputElement FindByLabelText(TextMatch roleMatcher, ByLabelTextOptions? options = null);
public IInputElement FindByLabelText(RegEx roleRegex, ByLabelTextOptions? options = null);

public IInputElement? FindByLabelTextOrDefault(string role, ByLabelTextOptions? options = null);
public IInputElement? FindByLabelTextOrDefault(TextMatch roleMatcher, ByLabelTextOptions? options = null);
public IInputElement? FindByLabelTextOrDefault(RegEx roleRegex, ByLabelTextOptions? options = null);

public Task<IInputElement> FindByLabelTextAsync(string role, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IInputElement> FindByLabelTextAsync(TextMatch roleMatcher, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IInputElement> FindByLabelTextAsync(RegEx roleRegex, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);

public IReadOnlyList<IInputElement> FindAllByLabelText(string role, ByLabelTextOptions? options = null);
public IReadOnlyList<IInputElement> FindAllByLabelText(TextMatch roleMatcher, ByLabelTextOptions? options = null);
public IReadOnlyList<IInputElement> FindAllByLabelText(RegEx roleRegex, ByLabelTextOptions? options = null);

public Task<IReadOnlyList<IInputElement>> FindAllByLabelTextAsync(string role, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IReadOnlyList<IInputElement>> FindAllByLabelTextAsync(TextMatch roleMatcher, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IReadOnlyList<IInputElement>> FindAllByLabelTextAsync(RegEx roleRegex, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);

public IReadOnlyList<IInputElement> FindExactlyByLabelText(string role, int exactCount, ByLabelTextOptions? options = null);
public IReadOnlyList<IInputElement> FindExactlyByLabelText(TextMatch roleMatcher, int exactCount, ByLabelTextOptions? options = null);
public IReadOnlyList<IInputElement> FindExactlyByLabelText(RegEx roleRegex, int exactCount, ByLabelTextOptions? options = null);

public Task<IReadOnlyList<IInputElement>> FindExactlyByLabelTextAsync(string role, int exactCount, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IReadOnlyList<IInputElement>> FindExactlyByLabelTextAsync(TextMatch roleMatcher, int exactCount, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IReadOnlyList<IInputElement>> FindExactlyByLabelTextAsync(RegEx roleRegex, int exactCount, ByLabelTextOptions? options = null, WaitForOptions? waitForOptions = null);

public record class ByLabelTextOptions
{
    public static readonly ByLabelTextOptions Default = new();
    public StringComparison ComparisonType { get; init; } = StringComparison.Ordinal;
    public TextNormalizer? Normalizer { get; init; }
    // A css selector to match the child element of the selected label element.
    public string Selector { get; init; } = "*";
}

The remaining queries follow the same pattern as the above queries and are not much different.

Firing events

Event firing, or dispatching, as it's called in bUnit until now, will stay pretty much the same, but with a few improvements.

We will drop the overloads that take each property in an EventArgs type and just keep the overload that takes an optional EventArg. If none is provided, we will create a default EventArgs type for the event in question.

[egil]

I did search a bit and found https://github.com/morganlaneap/blazor-testing-library from @morganlaneap that does try to adopt some of the testing library API and combine it with bUnit and Fluent Assertions. It's not complete, and the future plans are not clear, but perhaps Morgan is interested in a collaboration.

[morganlaneap]

@egil thanks for finding my project! I originally created it as I come from a React Testing Library approach of testing, and noticed bUnit didn't have anything like it. It's current form only has the functionality that I needed for a Blazor website I made, but it definitely would be possible to expand it. I haven't worked on it recently as I've had no need for it, but I'd be open to getting started on it again and implementing more functionality.

[egil]

Btw., my inspiration for adding opening this issue was @bpugh and his recent talk with Khalid on JetBrainsTV. I would love to get your take on this as well @bpugh, i.e. inputs and suggestions, what is missing from bUnit, etc.

[egil]

@egil thanks for finding my project! I originally created it as I come from a React Testing Library approach of testing, and noticed bUnit didn't have anything like it. It's current form only has the functionality that I needed for a Blazor website I made, but it definitely would be possible to expand it. I haven't worked on it recently as I've had no need for it, but I'd be open to getting started on it again and implementing more functionality.

@morganlaneap that's great to hear. we're still debating whether or not it makes sense to put something like this under the bUnit umbrella explicitly, e.g. as a bunit.assert package. If we go that route of having a bunit.assert package, you are of course welcome to contribute and be part of the team that builds it, or if you don't like the direction and our take on the API, continue with what you are building (gues without saying).

[morganlaneap]

@egil I think having something like this in bUnit would be beneficial. A bunit.assert package which follows the same way Testing Library does it would work well. I've essentially tried to port some of the stuff from it to be used with Blazor, and given this approach of testing is proven to work well in Javascript land, I think that's a good argument to replicate it in C# land.

[egil]

Hi all,

I started going through the Testing Library docs and matching their concepts to bUnit and .NET. Let me know what you think:

bUnit.assert

This library provides a set of assertion methods for use in unit tests. The assertions are inspired by the https://testing-library.com library from the JavaScript world. Check out their Guiding Principles for additional tips for writing good tests using the assertion methods in this library.

Mapping between Testing Library and bUnit conventions

Testing Library	bUnit	Description
getBy*	Find*	Returns the matching node for a query, and throw a ElementNotFoundException exception if no elements match or if more than one match is found (use FindAll* instead if more than one element is expected).
queryBy*	Find*OrDefault	Returns the matching node for a query, or null if no elements match. This is useful for asserting an element that is not present. Throws an error if more than one match is found (use FindAll* and assert the returned collection is empty instead, if this is OK).
findBy*	Find*Async	Returns a Task that completes when the matching element is found which matches the given query. Throws a TimeoutException if no element is found or if more than one element is found after a default timeout of 1000ms. If you need to find more than one element, use FindAll*Async.
getAllBy* and queryAllBy*	FindAll*	Returns a collection of all matching nodes for a query. The collection is empty if no elements are found.
findAllBy*	FindAll*Async	Returns a Task that completes when any matching elements are found which matches the given query. Throws a TimeoutException if no elements are found after a default timeout of 1000ms.

As the table above shows, the method names are slightly simplified in bUnit compared to Testing Library, and more aligned with what one would expect in .NET / C# code, e.g. using FindOrDefault instead of queryBy*.

Using queries

The query methods are available on the IComponentFragment type and IElement types. When a query method is issued on an IComponentFragment, it will search the entire component tree for matching elements. When a query method is issued on an IElement, it will only search the children of that element.

Query options

Each query method has a default set of options that can be overridden by passing in an options object. The options object is optional, and if not provided, the default options will be used.

TextMatch delegate

Most of the query methods can be passed a TextMatch, in addition to a string or a RegEx type.

The TextMatch delegate is used to specify how a given text should be matched, if the default string comparer should not be used. This is the signature for TextMatch:

public delegate bool TextMatch(string? text, IElement? element);

Where text is the text that should be matched, and element is the element that contains the text. The element parameter is optional, and can be null if the text is not contained in an element.

TODO: is this possible in bUnit?

Matching text (precision)

Instead of the text matching feature of Testing Library, bUnit uses the StringComparison enum to specify how to match text. The default is StringComparison.Ordinal, which is the same as the default in Testing Library, but you can also use StringComparison.OrdinalIgnoreCase if you want to ignore case or any of the other options for string comparison available in .NET.

Normalizion

Similar to how MarkupMatches normalize text before comparison, the query methods will normalize the text before matching. By default, normalization consists of trimming whitespace from the start and end of text, and collapsing multiple adjacent whitespace characters into a single space.

If you want to prevent that normalization, or provide alternative normalization (e.g. to remove Unicode control characters), you can provide a normalizer delegate in the options object. This function will be given a string and is expected to return a normalized version of that string.

The signature for the normalizer delegate is:

public delegate string TextNormalizer(string text);

Wait for options

The WaitForOptions class is used to specify how long to wait for an element to appear in the DOM, and how often to check for the element. The default is to wait for 1000ms, and check every 50ms.

public record class WaitForOptions
{
    public static readonly WaitForOptions Default = new();

    public TimeSpan Timeout { get; init; } = TimeSpan.FromSeconds(1);
    public TimeSpan Interval { get; init; } = TimeSpan.FromMilliseconds(50);
}

public record class WaitForAllOptions : WaitForOptions
{
    public static readonly WaitForAllOptions Default = new();

    /// <summary>
    /// Gets or sets the number of elements that should be found.
    /// If not specified, the wait will terminate when one or more elements are found.
    /// </summary>
    public int? Count { get; init; }
}

Previously, the WaitFor* methods in bUnit only retried the wait predicate after every render. I am not sure if it makes sense to introduce a recurring retry interval like Testing Library provides, but it is something to consider.

In addition, the OnTimeout?: (error: Error) => Error call back does not seem to useful in C#/.NET, and neither does mutationObserverOptions?: MutationObserverInit.

The core Find* method

bUnit ships with Find(string cssSelector) and FindAll(string cssSelector) methods already, and it makes sense to continue having these. They are updated to match the structure described above, and replaces the WaitForElement and WaitForElements methods:

public IElement Find(string cssSelector);
public IElement? FindOrDefault(string cssSelector);
public Task<IElement> FindAsync(string cssSelector, WaitForOptions? waitForOptions = null);
public IReadOnlyList<IElement> FindAll(string cssSelector);
public Task<IReadOnlyList<IElement>> FindAllAsync(string cssSelector, WaitForAllOptions? waitForOptions = null);

ByRole

Check out the ByRole query in Testing Library for a description of the ByRole query and its options.

Here is the C# API for the ByRole query:

public IElement FindByRole(string role, ByRoleOptions? options = null);
public IElement FindByRole(TextMatch roleMatcher, ByRoleOptions? options = null);
public IElement FindByRole(RegEx roleRegex, ByRoleOptions? options = null);

public IElement? FindByRoleOrDefault(string role, ByRoleOptions? options = null);
public IElement? FindByRoleOrDefault(TextMatch roleMatcher, ByRoleOptions? options = null);
public IElement? FindByRoleOrDefault(RegEx roleRegex, ByRoleOptions? options = null);

public Task<IElement> FindByRoleAsync(string role, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IElement> FindByRoleAsync(TextMatch roleMatcher, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);
public Task<IElement> FindByRoleAsync(RegEx roleRegex, ByRoleOptions? options = null, WaitForOptions? waitForOptions = null);

public IReadOnlyList<IElement> FindAllByRole(string role, ByRoleOptions? options = null);
public IReadOnlyList<IElement> FindAllByRole(TextMatch roleMatcher, ByRoleOptions? options = null);
public IReadOnlyList<IElement> FindAllByRole(RegEx roleRegex, ByRoleOptions? options = null);

public Task<IReadOnlyList<IElement>> FindAllByRoleAsync(string role, ByRoleOptions? options = null, WaitForAllOptions? waitForOptions = null);
public Task<IReadOnlyList<IElement>> FindAllByRoleAsync(TextMatch roleMatcher, ByRoleOptions? options = null, WaitForAllOptions? waitForOptions = null);
public Task<IReadOnlyList<IElement>> FindAllByRoleAsync(RegEx roleRegex, ByRoleOptions? options = null, WaitForAllOptions? waitForOptions = null);

// See https://testing-library.com/docs/queries/byrole/#options for 
// description of each of the options.
public record class ByRoleOptions
{
    public static readonly ByRoleOptions Default = new();

    public StringComparison ComparisonType { get; init; } = StringComparison.Ordinal;
    public bool Hidden { get; init; } = false;
    public bool? Selected { get; init; }
    public bool? Checked { get; init; }
    public bool? Current { get; init; }
    public bool? Pressed { get; init; }
    public bool? Expanded { get; init;}
    public bool QueryFallbacks { get; init; } = false;    
    public int? Level { get; init; }
    public TextMatch? Name { get; init; }
    public TextMatch? Description { get; init; }
    public TextNormalizer? Normalizer { get; init; }
}

ByLabelText

Check out the ByLabelText query in Testing Library for a description of the ByLabelText query and its options.

Here is the C# API for the ByLabelText query:

public IInputElement FindByLabelText(string role, ByLabelTextOptions? options = null);
public IInputElement FindByLabelText(TextMatch roleMatcher, ByLabelTextOptions? options = null);
public IInputElement FindByLabelText(RegEx roleRegex, ByLabelTextOptions? options = null);

public IInputElement? FindByLabelTextOrDefault(string role, ByLabelTextOptions? options = null);
public IInputElement? FindByLabelTextOrDefault(TextMatch roleMatcher, ByLabelTextOptions? options = null);
public IInputElement? FindByLabelTextOrDefault(RegEx roleRegex, ByLabelTextOptions? options = null);

public Task<IInputElement> FindByLabelTextAsync(string role, ByLabelTextOptions? options = null);
public Task<IInputElement> FindByLabelTextAsync(TextMatch roleMatcher, ByLabelTextOptions? options = null);
public Task<IInputElement> FindByLabelTextAsync(RegEx roleRegex, ByLabelTextOptions? options = null);

public IReadOnlyList<IInputElement> FindAllByLabelText(string role, ByLabelTextOptions? options = null);
public IReadOnlyList<IInputElement> FindAllByLabelText(TextMatch roleMatcher, ByLabelTextOptions? options = null);
public IReadOnlyList<IInputElement> FindAllByLabelText(RegEx roleRegex, ByLabelTextOptions? options = null);

public Task<IReadOnlyList<IInputElement>> FindAllByLabelTextAsync(string role, ByLabelTextOptions? options = null);
public Task<IReadOnlyList<IInputElement>> FindAllByLabelTextAsync(TextMatch roleMatcher, ByLabelTextOptions? options = null);
public Task<IReadOnlyList<IInputElement>> FindAllByLabelTextAsync(RegEx roleRegex, ByLabelTextOptions? options = null);

public record class ByLabelTextOptions
{
    public static readonly ByLabelTextOptions Default = new();
    public StringComparison ComparisonType { get; init; } = StringComparison.Ordinal;
    public TextNormalizer? Normalizer { get; init; }
    // A css selector to match the child element of the selected label element.
    public string Selector { get; init; } = "*";
}

The remaining queries follows the same pattern as the above queries and are not much different.

Firing events

Event firing, or dispatching, as its called in bUnit until now, will stay pretty much the same, but with a few improvements.

We will drop the overloads that take each property in an EventArgs type and just keep the overload that takes an optional EventArg. If none is provided, we will create a default EventArgs type for the event in question.

[morganlaneap]

It's definitely a good start, this is what I had in mind when I wanted to create something like this. Seems like the right path to go down. The only thing I'd say is that in testing library, I believe if getAll cannot find any elements that match, then an exception is thrown. Do we want to replicated that here, rather than FindAll always returning an empty array if no results? Though thinking about this I'm not sure if there's a valid use case for that, as you could just check Find throws an exception to assert something doesn't exist - just thinking out loud.

[egil]

It's definitely a good start, this is what I had in mind when I wanted to create something like this. Seems like the right path to go down. The only thing I'd say is that in testing library, I believe if getAll cannot find any elements that match, then an exception is thrown. Do we want to replicated that here, rather than FindAll always returning an empty array if no results? Though thinking about this I'm not sure if there's a valid use case for that, as you could just check Find throws an exception to assert something doesn't exist - just thinking out loud.

Appreciate your input.

When I read the description of getAll, I didnt really get the benefit of it throwing. If you want to assert that there are no elements found, I find it more obvious to do a FindAll(...).Should().BeEmpty() instead of just having the FindAll throw. The first makes the code much more readable.

However, there is an option to specify the expected count in FindAllOptions, that will cause it to throw throw if the expected count does not match.

But my ultimate goal was not to introduce too many variants of the query methods, that does not signal explicitly what they do. Find and FindAll does not surprise, at least not me 🙂

That part, the getAll and queryAll methods, is not intuitive, at least not coming from .NET. And here we do have the convention of postfixing methods with "Async", so that means we do not have both a query and getAll methods.

Finally, the Find and FindAll methods already exist in bUnit, so existing users will recognize their meaning.

That said, I have no personal experience with Testing Library and their API, so I am very eager to get all the input I can.

[morganlaneap]

Yeah that makes sense. I think testing library just does it "because that's how we do it". In fact, it actually makes more sense having the FindAll(...).Should().BeEmpty() approach as it's more explicit. Good stuff!

[bpugh]

Btw., my inspiration for adding opening this issue was @bpugh and his recent talk with Khalid on JetBrainsTV. I would love to get your take on this as well @bpugh, i.e. inputs and suggestions, what is missing from bUnit, etc.

That's awesome and thanks for starting this! I had debated opening an issue about this but wasn't sure if it was considered in scope of the bUnit project but I think a companion library makes sense.

I have a few bits of feedback.

This is minor, but I'm not sure I understand the reason for naming the library bunit.assert since to me it implies it would contain additional assertions instead of queries which I believe is what the xunit.assert library contains which is also similar to libraries like FluentAssertions.
Though we could also include additional helpful assertions like what jest-dom provides for testing-library, so maybe a more general name like bunit.helpers?

To clarify a couple points about the testing-library api:

The reason get* and find* throw is to be able to provide more helpful error messages. But then the query* methods are necessary to be able to test for non existence of elements.
So they tend to be used like expect(screen.queryByText('loading')).toBeNull() or expect(screen.queryAllByRole('alert')).toEqual([]);. Likewise, because the getAll* throws, if you have this assertion expect(screen.getAllByRole('link')).toHaveLength(2); and if no links are found then it will give the following error:

TestingLibraryElementError: Unable to find an accessible element with the role "link"

Here are the accessible roles:

  button:

  Name "Hello World":
  <button />

  --------------------------------------------------

<body>
  <div>
    <button>
      <span>
        Hello
      </span>

      <span>
        World
      </span>
    </button>
  </div>
</body>

as opposed to if it returned an empty array then you would only get a message like "expected length 2 but received 0". This is why there is even a linter rule to enforce people use the correct query for the circumstance. And while you can use this behavior to implicitly assert that an element exists, the recommended practice is to have an explicit assertion like: expect(screen.getByRole('alert')).toBeInTheDocument().

That being said, I agree the names aren't very intuitive especially in C# land (I've had to remind myself that find is the async one) and I like the conventions of Find*Async and Find*OrDefault. I do think there's value in throwing when searching for a collection in the context of tests, in which case it would seem like the most consistent thing to do would be to map getAllBy* to FindAll* which would throw, and map queryAllBy* to something like FindAll*OrDefault. Alternatively, maybe we can just add a method FindAtLeastOneBy* that mimics the behavior of getAllBy*? Or just relying on specifying a count with FindAllOptions works too.

The error messages aren't really shown in the docs but you can see them in the test suite: https://github.com/testing-library/dom-testing-library/blob/main/src/__tests__/role.js. Probably not necessary to fully implement all the detailed info at least initially, but I at least found it handy to output the markup automatically when a query failed.

I also wonder if it would be worth defining an subset of the testing-library api to support initially since some of the queries/options will require the library do some heavy lifting that testing-library is able do by leveraging other npm packages. For example, the Description filter of the ByRole query requires calculating the accessible description of an element which testing-library does by using the dom-accessibility-api package. And for getting a mapping of HTML elements to their implicit roles they use aria-query. I didn't see equivalent nuget packages but for the basic implementation of ByRole we could copy the mapping from aria-query easily enough I think.

For me at least, I find ByRole and ByLabelText used by far the most so even just implementing some version of those two would add a lot of value.

I hope this is helpful and thanks for all your work on bUnit!

[egil]

Thank you for your input @bpugh.

This is minor, but I'm not sure I understand the reason for naming the library bunit.assert since to me it implies it would contain additional assertions instead of queries which I believe is what the xunit.assert library contains which is also similar to libraries like FluentAssertions. Though we could also include additional helpful assertions like what jest-dom provides for testing-library, so maybe a more general name like bunit.helpers?

I agree. After reading through the Testing Library's docs, I realize it's mostly a query API, and I am considering just including the additional query APIs into bunit.web, since they do not conflict with other assertion libraries. Assertion libraries like xunit.assert, FluentAssertions or Shouldly have opinionated APIs, so including bUnit specific assertion APIs are just going to feel out of place with whatever assertion lib the user is already using. That would not be the case with the additional query methods.

The same can be said for the event dispatch methods (fireEvent in Testing Library), e.g. Click(), that bUnit already includes.

To clarify a couple points about the testing-library api:

The reason get* and find* throw is to be able to provide more helpful error messages. But then the query* methods are necessary to be able to test for non existence of elements. So they tend to be used like expect(screen.queryByText('loading')).toBeNull() or expect(screen.queryAllByRole('alert')).toEqual([]);. Likewise, because the getAll* throws, if you have this assertion expect(screen.getAllByRole('link')).toHaveLength(2); and if no links are found then it will give the following error:

TestingLibraryElementError: Unable to find an accessible element with the role "link"

Here are the accessible roles:

  button:

  Name "Hello World":
  <button />

  --------------------------------------------------

<body>
  <div>
    <button>
      <span>
        Hello
      </span>

      <span>
        World
      </span>
    </button>
  </div>
</body>

as opposed to if it returned an empty array then you would only get a message like "expected length 2 but received 0". This is why there is even a linter rule to enforce people use the correct query for the circumstance. And while you can use this behavior to implicitly assert that an element exists, the recommended practice is to have an explicit assertion like: expect(screen.getByRole('alert')).toBeInTheDocument().

That being said, I agree the names aren't very intuitive especially in C# land (I've had to remind myself that find is the async one) and I like the conventions of Find*Async and Find*OrDefault. I do think there's value in throwing when searching for a collection in the context of tests, in which case it would seem like the most consistent thing to do would be to map getAllBy* to FindAll* which would throw, and map queryAllBy* to something like FindAll*OrDefault. Alternatively, maybe we can just add a method FindAtLeastOneBy* that mimics the behavior of getAllBy*? Or just relying on specifying a count with FindAllOptions works too.

The error messages aren't really shown in the docs but you can see them in the test suite: https://github.com/testing-library/dom-testing-library/blob/main/src/__tests__/role.js. Probably not necessary to fully implement all the detailed info at least initially, but I at least found it handy to output the markup automatically when a query failed.

I really like having the "actual" markup in the error message for the Find and FindAll methods, as we do with MarkupMatches assertion method. In essence, we would extend the ElementNotFoundException to include a pretty print of the rendered fragments DOM.

My concern is making the API as intuitive to a C#/.NET dev as possible though, and providing a map for users coming from Testing Library where we differ. I'm not a huge fan of FindAtLeastOneBy kind of APIs, which I guess include a FindAll that throws if zero is found, because what if somebody needs at least two or three, etc.. That's also why we settled on having WaitForElements take an optional expected count, to enable the "throw if expected count not found" scenario.

However, I guess the expectedCount overload of the method may not be as easy to discover as just having separate methods. An alternative could be to include a FindExactlyBy* that takes an expectedCount argument. I think that name indicates well that it will only succeed if the expectedCount of elements is found.

An alternative is to mirror the LINQ methods, and create special overload versions of Where, Single, and First, etc., however, they do not offer us the possibility to say "I want to find exactly N elements" easily, but we could make them throw an exception that does include the markup that actually does exist.

I also wonder if it would be worth defining an subset of the testing-library api to support initially since some of the queries/options will require the library do some heavy lifting that testing-library is able do by leveraging other npm packages. For example, the Description filter of the ByRole query requires calculating the accessible description of an element which testing-library does by using the dom-accessibility-api package. And for getting a mapping of HTML elements to their implicit roles they use aria-query. I didn't see equivalent nuget packages but for the basic implementation of ByRole we could copy the mapping from aria-query easily enough I think.

I agree. We would start with the basic options and get a release out that supports them. It was not exactly clear to me what Description vs. Name would work, and if they are mutually exclusive, but I was planning to the port the unit tests from Testing Library and have them drive our implementation. As long as we are getting the same results as they are, we should be solid ground. In theory, we should be able to port all the needed logic from JavaScript, as long as AngleSharp implements all the native features that the various NPM libs depend on in the browser.

For me at least, I find ByRole and ByLabelText used by far the most so even just implementing some version of those two would add a lot of value.

I hope this is helpful and thanks for all your work on bUnit!

Thanks, it's super helpful to get this kind of input and I hope you are able to help out refining our version of Testing Library further!

[bpugh]

Sounds like a good plan!

It was not exactly clear to me what Description vs. Name would work, and if they are mutually exclusive

They're not mutually exclusive since elements can have both so specifying both options would return elements that match on the Name and Description.