Future of assert_cli

[killercup]

@epage asked in #40 (comment):

I'm curious, what do you see needing done before moving this to 1.0? While I should get back to my regularly schedule project, I would like to also help work towards more crates hitting 1.0.

Good question! I'd like to make the arg parsing a bit leaner (cf. #25 and #32) and maybe improve the error output (#17, #18, #30).

My general goal is to make a library that makes writing tests like these easy and fun. It'll probably involve generating some test boilerplate code, and I'm not entirely sure what I want that API to look like. I'm not even sure if that's in scope for assert_cli. What do you think?

EDIT

The building blocks for assert_cli 1.0 are

• https://github.com/assert-rs/assert_fs/
• https://github.com/assert-rs/assert_cmd/
• https://github.com/assert-rs/predicates-rs/
• https://github.com/Freyskeyd/environment ?

[epage]

tl;dr After considering this for the last couple days (and digging into the issues), I feel like PR #43 is sufficient for releasing 1.0.

I feel the main categories of 1.0 blockers

• API changes
  • Use duct internally? #32 duct mostly buys us
    • piping, see my comments on waltz.
    • advanced command lines parsing, see comment on Deprecate macro in favor of clever Assert::command(&str)? #25
  • Deprecate macro in favor of clever Assert::command(&str)? #25, see comment on it
  • Feature: Customizable Panic-Format #18 I feel like this could be done without breaking compatibility
• Usable.
  • Feature: show line numbers and/or context in diffs #17 from my own testing, this seems like there is enough output
  • Feature: Customizable Panic-Format #18 I feel like a default output policy should be sufficient for release
  • Pick one: Error location or error context #30 Closed, I don't quite remember enough about it to be actionable.
  • I audited the output in PR Increase context on failures #43

RE Waltz's tests

While I see a lot of value in what you're doing with waltz, I feel like that has a longer path to stabilization than the core of assert_cli.

assert_cli does a great job of doing one thing very well, testing short lived programs. If we take the approach of trying to have assert_cli cover additional use cases, whether its the side effect testing done in waltz or the orchestrating multiple programs with duct's piping, I feel like it might detract from its primary strengths.

Also, thinking about it some more, side effect testing might have value outside of running programs. Waltz's test harness could be really nice for also wrapping functions.

[killercup]

Thanks for the comprehensive list! Okay, let's keep this crate small! I'll just have to come up with a name for the crate that does the waltz cli spec thing ;) I'd like to ping some more of assert_cli's users about feedback on 1.0. Ed Page <notifications@github.com> schrieb am Mi. 27. Sep. 2017 um 05:01:

…

tl;dr After considering this for the last couple days (and digging into the issues), I feel like PR #43 <#43> is sufficient for releasing 1.0. I feel the main categories of 1.0 blockers - API changes - #32 <#32> duct mostly buys us - piping, see my comments on waltz. - advanced command lines parsing, see comment on #25 <#25> - #25 <#25>, see comment on it - #18 <#18> I feel like this could be done without breaking compatibility - Usable. - #17 <#17> from my own testing, this seems like there is enough output - #18 <#18> I feel like a default output policy should be sufficient for release - #30 <#30> Closed, I don't quite remember enough about it to be actionable. - I audited the output in PR #43 <#43> RE Waltz's tests While I see a lot of value in what you're doing with waltz, I feel like that has a longer path to stabilization than the core of assert_cli. assert_cli does a great job of doing one thing very well, testing short lived programs. If we take the approach of trying to have assert_cli cover additional use cases, whether its the side effect testing done in waltz or the orchestrating multiple programs with duct's piping, I feel like it might detract from its primary strengths. Also, thinking about it some more, side effect testing might have value outside of running programs. Waltz's test harness could be really nice for also wrapping functions. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABOX8GuxZ_uyShM6clH3PKNWmMVluJoks5smbqRgaJpZM4Pi0Dm> .

[killercup]

Pinging some people who used assert_cli in their projects (found using Github search):

We are thinking about doing a 1.0 release and would love to hear your opinion on the crate and ideas for its future!

• @livioribeiro who used it for cargo-readme integration tests
• @bjgill who updated cargo-edit to assert_cli 0.4
• @mattgathu who quite recently used assert_cli in rget
• @jxson who uses it in some tests of to-directory

[mattgathu]

Hello @killercup @epage thank you for building such an amazing crate!! It has been really helpful.

I have been using it to test a rust cli app I'm writing using the assert_cli::Assert::main_binary interface. This is part of my contribution for Increasing Rust's Reach

I was surprised to find assert_cli::Assert::main_binary missing in your docs examples considering how helpful it is!! I had to dig a little deeper into your docs to get it.

I'm planning to:

• Write a Pull Request to enhance the docs and add example for assert_cli::Assert::main_binary and other useful interfaces.
• Write a How to test Rust CLI apps blog post, with introduction to writing tests with assert_cli

Let me know what you think!

[mattgathu]

About future ideas, I have been wondering how feasible it is to test for colored output?

If my CLI binary prints out colored text, can I test this?

What are your thoughts on this.

[epage]

I was surprised to find assert_cli::Assert::main_binary missing in your docs examples considering how helpful it is!! I had to dig a little deeper into your docs to get it.

Yeah, I suspect thats from it being developed later and that it makes it harder to run the examples as tests. I agree that more attention should be drawn to it.

Let me know what you think!

Sounds great! The main thing to watch out for is not reducing coverage.

[epage]

About future ideas, I have been wondering how feasible it is to test for colored output? If my CLI binary prints out colored text, can I test this?

Most programs detect if they are outputting to a console and don't print color if they are. Some have a "force color" option.

So either we'd need to figure out how to fake being a console or clients need to have a force color option and set it.

When forced, I assume color just shows up in stdout as weird characters. Can you just manually enter those characters in to test it? Do you have a proposal for what an API might look like for more advanced color testing?

[epage]

Once #46 goes in, should we also need to wait for Environment to hit 1.0?

[killercup]

Good question. I'm okay with publishing this with a re-export of the 0.1. I also think that Environment could be 1.0 already.

[Freyskeyd]

I would like to have a way to remove variable from Environment before 1.0.
something like:

let env: Environment = vec![("foo", "bar"), ("bar", "baz")].into();
...
env.remove("foo");

I would also like to have a has/check method:

let env = Environment::inherit();

if env.has("foo") && env.check_if("bar", "baz") {
    ...
}

Having a macro style would be a good way to use it:

let e = inherit_env!(
    "foo" -> "bar", // <--- don't know if I can do that
    "bar" -> "baz",
);

let e = empty_env!();

assert_cli::Assert::command(&["printenv"])
     .with_env(inherit_env!( "bar" -> "baz" ))
     .stdout().contain("bar=baz")
     .unwrap();

[epage]

let e = inherit_env!(
    "foo" -> "bar", // <--- don't know if I can do that
    "bar" -> "baz",
);

I think I generally see people use the => syntax when they experiment with these type of things, for example https://github.com/kmcallister/literator

[Freyskeyd]

@epage yep that's looking better.

Another question is: Macro style must expend to compile version? or not?

inherit_env! { "bar" => "baz" }

Becomes

Environment::inherit().insert("bar", "baz").compile()
or just
Environment::inherit().insert("bar", "baz")

But it's a discussion on Environment repository I think^^

[epage]

@killercup just recently came across https://github.com/emk/subtitles-rs. Looks like some of its building blocks are similar to the type of thing you are trying to do with waltz.

I've not explored it enough to know all of the implications of TestDir, if that concept is even compatible with my projects. I could see the idea of splitting the file system tests out into is own crate so the two can be composed together. I could see using the file system stuff in my lower level integration tests as well.