Toys-Core User Guide

Toys-Core is the command line framework underlying Toys. It implements most of the core functionality of Toys, including the tool DSL, argument parsing, loading Toys files, online help, subprocess control, and so forth. Toys-Core can be used to create custom command line executables, or it can be used to provide mixins or templates in your gem to help your users define tools related to your gem's functionality.

If this is your first time using Toys-Core, we recommend starting with the README, which includes a tutorial that introduces how to create simple command line executables using Toys-Core, customize the behavior, and package your executable in a gem. You should also be familiar with Toys itself, including how to define tools by writing Toys files, how to interpret arguments and flags, and how to use the Toys execution environment. For background, please see the Toys README and Toys User's Guide. Together, those resources will likely give you enough information to begin creating your own basic command line executables.

This user's guide covers all the features of Toys-Core in much more depth. Read it when you're ready to unlock all the capabilities of Toys-Core to create sophisticated command line tools.

Conceptual overview

Toys-Core is a command line framework in the traditional sense. It is intended as the core component of the Toys gem, but is designed generically for writing custom command line executables in Ruby. The framework provides common facilities such as argument parsing and online help. Your executable can then choose and configure those facilities, and implement the actual behavior.

The entry point for Toys-Core is the CLI object. Typically your executable script instantiates a CLI, configures it with the desired implementation code, and runs it.

Implementation code is provided by tool sources, which could be blocks, toys files, and other ways to define functionality. Tool sources define functionality using the Toys DSL, the same DSL used by Toys itself, supporting tools, subtools, flags, arguments, help text, and all the other features of Toys.

An executable can customize its own facilities for writing tools by providing built-in mixins and built-in templates, and can implement default behavior across all tools by providing middleware.

Most executables will provide a set of static tools, but it is possible to support user-provided tools, as Toys does, by defining a tool source pointing at user-controlled files.

An executable can customize many aspects of its behavior, such as the logging output, error handling, and even shell tab completion.

Finally, Toys-Core can also be used to publish Toys extensions, collections of mixins, templates, and/or predefined tools that can be distributed as gems to enhance Toys for other users.

Using the CLI object

The Toys::CLI object is the main entry point for Toys-Core. Most command line executables based on Toys-Core use it as follows:

  • Instantiate a CLI object, passing configuration parameters to the constructor.
  • Define the functionality of the CLI, either inline by passing it blocks, or by providing paths to tool files.
  • Call the Toys::CLI#run method, passing it the command line arguments (e.g. from ARGV).
  • Handle the result code, normally by passing it to Kernel#exit.

To get access to the CLI object, or any other Toys-Core classes, you first need to ensure that the toys-core gem is loaded, and require "toys-core".

Following is a simple "hello world" example using the CLI:

#!/usr/bin/env ruby

require "toys-core"

# Instantiate a CLI with the default options
cli = Toys::CLI.new

# Define the functionality
cli.add_source do
  desc "My first executable!"
  flag :whom, default: "world"
  def run
    puts "Hello, #{whom}!"
  end
end

# Run the CLI, passing the command line arguments
result = cli.run(*ARGV)

# Handle the result code.
exit(result)

Try testing out that script, by writing it into a file, e.g greet.rb, setting the execute bit, and running it.

$ chmod a+x greet.rb
$ ./greet.rb
$ ./greet.rb --whom=Ruby

Configuring the CLI

Generally, you control CLI features by passing arguments to its constructor. These features include:

Each of the actual parameters is covered in detail in the documentation for Toys::CLI#initialize. The configuration of a CLI cannot be changed once the CLI is constructed. If you need a CLI with modified configuration, use Toys::CLI#child, which creates a copy of the CLI with any modifications you request.

CLI execution

This section provides some detail on how a CLI executes your code.

A CLI resolves its configuration into a Toys::Runner, the object that actually runs tools. When you call Toys::CLI#run, it passes the command line to that Runner, which carries out three phases:

  • Loading in which the Runner identifies which tool to run, and loads the tool from a tool source, which could be a block passed to the CLI, a file loaded from the file system, git, or other location.
  • Context building, in which the Runner parses the command-line arguments according to the flags and arguments declared by the tool, instantiates the tool, and populates the Toys::Context object (which is self when the tool is executed)
  • Running, which involves running any initializers defined on the tool, applying middleware, running the tool's code, and handling errors.

The Loader

When Toys needs the definition of a tool, it queries the Toys::Loader. The loader object is configured with a set of tool sources representing ways to define a tool. These sources may be blocks passed directly to the CLI, or directories and files loaded from the file system, from gems, or even from remote git repositories. When a tool is requested by name, the loader is responsible for locating the tool definition in those sources, and arbitrating which definition wins when more than one source defines the same tool name. A tool definition is represented by Toys::ToolDefinition.

One important property of the loader is that it is lazy. It queries tool sources only when it has reason to believe that a tool it is looking for may be defined there. For example, if your tools are defined in a directory structure, a tool named foo bar might live in the file foo/bar.rb. The loader will open that file, if it exists, only when the foo bar tool is requested. If instead foo qux is requested, the foo/bar.rb file is never even opened.

Perhaps more subtly, if you call Toys::CLI#add_source to define tools, the block is stored in the source list but not called immediately. Only when a tool is requested does the block actually execute. Furthermore, if you have tool blocks inside the block, the loader will execute only those that are relevant to a tool it wants. Hence:

cli.add_source do
  tool "foo" do
    def run
      puts "foo called"
    end
  end

  tool "bar" do
    def run
      puts "bar called"
    end
  end
end

If only foo is requested, the loader will execute the tool "foo" do block to get that tool definition, but will not execute the tool "bar" do block.

We will discuss more about the features of the loader below in the section on defining functionality.

Building context

Once a tool is defined, the Runner prepares it for execution by building a Toys::Context object. This object is self during tool runtime, and it includes:

  • The tool's methods, including its run entrypoint method.
  • Access to core tool functionality such as exit codes and logging.
  • The results from parsing the command line arguments
  • The runtime environment, including the tool's name, where the tool was defined, detailed results from argumet parsing, and so forth.

Much of this information is stored in a data hash, whose keys are defined as constants under Toys::Context::Key.

Argument parsing is directed by the Toys::ArgParser class. This class, for the most part, replicates the semantics of the standard Ruby OptionParser class, but it implements a few extra features and cleans up a few ambiguities. It is concerned only with the command line: it produces the parsed flag and argument values, along with any usage errors. The rest of the context data, such as the logger, the tool definition and its name and source, and the verbosity, is provided by the Runner.

A Runner can also be given arbitrary additional data by its caller, which is merged into the context underneath the data the Runner provides itself, so it cannot override a runtime-owned key. The CLI context key arrives this way, supplied by the CLI itself. A tool run through a Runner constructed directly will therefore see nil from Toys::Context#cli, but it always sees a Runner from Toys::Context#runner.

Running the tool and error handling

The running phase involves:

  • Running the tool's initializers (if any) in order.
  • Running the tool's middleware. Each middleware "wraps" the execution of subsequent middleware and the final tool execution, and has the opportunity to inject functionality before and after the main execution, or even to forgo or replace the main functionality, similar to Rack middleware.
  • Executing the tool itself by calling its run method (or any alternate entrypoint set by the tool).

Errors and signals are handled in two stages. If an exception reaches the Runner, whether from argument parsing, from the middleware, or from the tool itself, the Runner wraps it in a Toys::ContextualError tagged with the tool's name, its arguments, and the path to the file where it was defined. The Runner then passes that wrapper to its error handler, which decides what to report and what result code to return. Tools themselves can also intercept errors and handle them via the on_usage_error handler.

Signals are treated differently: they are never wrapped. A SignalException propagates as itself, so that each tool it passes through gets the chance to intercept it with an on_interrupt or on_signal handler, and so that any signal no tool handled reaches the error handler, and ultimately the Ruby VM, still recognizable as a signal. This holds within a single run, including across delegation; a tool that starts a separate nested run is a different matter. See the section on error handling for more details.

Multiple runs

A Toys::Runner can be reused to run multiple tools. It holds no state specific to a single run, so every run gets its own Toys::Context, and the Toys::CLI that owns the Runner can be reused in the same way. This may save on loading overhead, as the tools can be loaded just once and their definitions reused for multiple runs.

Reuse is safe as far as the Runner itself is concerned. Whether you can run two tools at the same time, in separate threads, depends entirely on the tools: a tool can modify global state such as the Ruby load path, the process environment, or a logger shared with other tools. See the logger argument to Toys::CLI#initialize for one such caveat.

A CLI's Runner is available as Toys::CLI#runner. Use it when you need run options that Toys::CLI#run does not expose, such as wrap_errors and handle_errors; it carries the CLI's configuration, so a tool run through it still sees the CLI. Note that Toys::CLI#child builds a new CLI with a runner of its own.

Defining functionality

Toys-Core uses (and indeed, provides the underlying implementation of) the familiar Toys DSL that you can read about in the Toys README and Toys User's Guide. This section assumes familiarity with those techniques for defining tools.

Here we will cover how to use the Toys-Core interfaces to define the different types of tool sources that provide tool definitions. We'll also look more closely at how tool definition works, providing insights into lazy loading and the tool prioritization system.

Writing tools in blocks

If you are writing your own command line executable using Toys-Core, often the easiest way to define your tools is to add a "block" tool source. The "hello world" example at the start of this guide uses this technique:

#!/usr/bin/env ruby

require "toys-core"

cli = Toys::CLI.new

# Define the functionality by passing a block to the CLI
cli.add_source do
  desc "My first executable!"
  flag :whom, default: "world"
  def run
    puts "Hello, #{whom}!"
  end
end

result = cli.run(*ARGV)
exit(result)

Here we called Toys::CLI#add_source and passed it a block containing Toys DSL syntax. It configures the "root tool", that is, the functionality of the program if you do not pass a tool name on the command line. You can also include "tool" blocks within the main block to define named tools and subtools, just as you would in a normal Toys file.

Writing tool files

If you want to define tools in separate files, you can do so and pass the file paths to Toys::CLI#add_source.

#!/usr/bin/env ruby

require "toys-core"

cli = Toys::CLI.new

# Load a file defining the functionality
cli.add_source("/usr/local/share/my_tool.rb")

result = cli.run(*ARGV)
exit(result)

The contents of /usr/local/share/my_tool.rb could then be:

desc "My first executable!"
flag :whom, default: "world"
def run
  puts "Hello, #{whom}!"
end

You can point to a specific file to load, or to a Toys directory, whose contents will be loaded similarly to how a .toys directory is loaded.

The CLI also provides high-level lookup methods that search for files named .toys.rb or directories named .toys. (These names can be configured with the toplevel_tool_file_name and toplevel_tool_dir_name options to the CLI constructor.) These methods, Toys::CLI#add_search_path and Toys::CLI#add_search_path_hierarchy, implement the actual behavior of Toys in which it looks for any available files in the current directory or its parents.

One particularly common use case is to package your command line executable along with the tool files it uses, into a gem for distribution. In such a case, you can define your tools in a particular directory in the gem and use Toys::CLI#add_source to point to that directory. See the section on packaging your executable for details on this technique.

Configuring sources

Passing a block or a file path directly to Toys::CLI#add_source is actually just a convenient shorthand. The standard usage is to pass in a source spec object that describes the source.

The block example above could be written to create the source spec explicitly:

#!/usr/bin/env ruby

require "toys-core"

cli = Toys::CLI.new

# Define the functionality in a "block" source spec:
source = Toys::SourceSpec.block do
  desc "My first executable!"
  flag :whom, default: "world"
  def run
    puts "Hello, #{whom}!"
  end
end

# Add it to the CLI
cli.add_source(source)

result = cli.run(*ARGV)
exit(result)

Similarly, to create a source spec for a file system path, pass it to Toys::SourceSpec.path. Full source spec objects are useful because they can include several attributes governing how the source behaves, including setting the context directory for tools loaded from that source, and setting a name for the source that will appear in error messages and documentation. For example:

#!/usr/bin/env ruby

require "toys-core"

cli = Toys::CLI.new

# A source that loads tools from a directory with a particular context directory
source = Toys::SourceSpec.path("/usr/local/share/my_tools/",
                               context_directory: "/var/my_project",
                               source_name: "My local tools directory")
cli.add_source(source)

result = cli.run(*ARGV)
exit(result)

The Toys::CLI#add_search_path and Toys::CLI#add_search_path_hierarchy convenience methods also provide a way to set the context_directory. In fact, by default they set the context directory to the directory containing the toys file/directory to load, which is the behavior of the Toys gem itself.

Gem and Git sources

Toys-Core provides two less common ways to source tools: from a RubyGem and from a remote git repository. This is similar to using the load_gem and load_git DSL directives to load tools from these sources. There is no shorthand for the gem and git source types; you need to create a source spec explicitly using Toys::SourceSpec.gem or Toys::SourceSpec.git and add it to your CLI. For example:

#!/usr/bin/env ruby

require "toys-core"

cli = Toys::CLI.new

# A source that loads tools from a gem with a particular context directory
source = Toys::SourceSpec.gem("toys-release", version: "~> 0.9",
                              context_directory: "/var/my_project")
cli.add_source(source)

result = cli.run(*ARGV)
exit(result)

Tool priority

It is possible to configure a CLI with multiple files, directories, and/or blocks with tool definitions. Indeed, this is how the toys gem itself is configured: loading tools from the current directory and its ancestry, from global directories, and from builtins. When a CLI is configured to load tools from multiple sources, it combines them. However, if multiple sources define a tool of the same name, only one definition will "win", the one from the source with the highest priority.

Each time a tool source is added to a CLI using Toys::CLI#add_source, Toys::CLI#add_search_path, or similar, that new source is added to a prioritized list. By default it is added to the end of the list, at a lower priority level than previously added sources. Thus, any tools defined in the new source would be overridden by tools of the same name defined in previously added sources.

#!/usr/bin/env ruby

require "toys-core"

cli = Toys::CLI.new

# Add a block defining a tool called "hello"
cli.add_source do
  tool "hello" do
    def run
      puts "Hello from the first source block!"
    end
  end
end

# Add a lower-priority block defining a tool with the same name
cli.add_source do
  tool "hello" do
    def run
      puts "Hello from the second source block!"
    end
  end
end

# Runs the tool defined in the first block
result = cli.run("hello")
exit(result)

When defining tool blocks or loading tools from files, you can also add the new source at the front of the priority list by passing an argument:

# Add tools with the highest priority
cli.add_source(high_priority: true) do
  tool "hello" do
    def run
      puts "Hello from the second source block!"
    end
  end
end

Priorities are used by the toys gem when loading tools from different directories. Any .toys.rb file or .toys directory is added to the CLI at the front of the list, with the highest priority. Parent directories are added at subsequently lower priorities, and common directories such as the home directory are loaded at the lowest priority.

Customizing built-in mixins and templates

Mixins and templates are two of the most useful mechanisms for sharing code and generating code for tools. In the main Toys gem, a certain set of mixins are built-in and can be referenced via symbols. For example, the exec mixin that provides facilities for running and controlling external processes, can be included using include :exec. In this section, we see how to define your own "built-in" mixins and templates that can be referenced via symbol.

"Built-in" mixins and templates (and middleware, which we shall cover later) are provided via the Toys::ModuleLookup mechanism. ModuleLookup lets you select a directory for "standard" instances. By default, Toys-Core establishes the toys/standard_mixins directory in the gem as the standard directory for mixins, and whenever you reference a mixin by symbol, it is used to determine the name of a file to open and the name of a module to load. You can, however, change this directory and provide a different ModuleLookup when constructing a CLI object.

Suppose, for example, you are writing a gem my_tools that uses Toys-Core, and you have a directory in your gem's lib called my_tools/mixins where you want your standard mixins to live. You could define mixins there:

# This file is my_tools/mixins/foo_mixin.rb

require "toys-core"

module MyTools
  module Mixins
    module FooMixin
      include Toys::Mixin

      def foo
        puts "Foo was called"
      end
    end
  end
end

Here is how you could configure a CLI to load standard mixins from that directory, and then use the above mixin.

# This file is my_tools.rb

require "toys-core"

my_mixin_lookup = Toys::ModuleLookup.new.add_path("my_tools/mixins")
cli = Toys::CLI.new(mixin_lookup: my_mixin_lookup)

cli.add_source do
  def run
    include :foo_mixin
    foo
  end
end

When you configure a ModuleLookup, you provide one or more paths, which are path prefixes that are used in a require statement. In the above example, we used the path my_tools/mixins for the ModuleLookup. Now when the CLI uses this ModuleLookup to find a mixin called :foo_mixin, it will attempt to require "my_tools/mixins/foo_mixin", which matches the file where we defined our mixin.

Notice also that foo_mixin.rb above defines FooMixin within a specific module hierarchy, corresponding to the file name my_tools/mixins/foo_mixin.rb according to standard Ruby naming conventions. The fully-qualified module name for the mixin must match this expected name, constructed from the path provided to the ModuleLookup and the name of the mixin. You can change the way this name mapping occurs, by providing the :module_base argument to the ModuleLookup constructor.

Template lookup happens similarly. Toys-Core does not provide a set of default templates, but the Toys gem does; the StandardCLI class used by Toys sets the :template_lookup to point to the toys/templates directory in that gem's library. If you want to customize the default template lookup for your Toys-based library, you can similarly provide your own ModuleLookup. This will let you control how templates are resolved when specified by symbol.

Customizing diagnostic output

Toys provides diagnostic logging and error reporting that can be customized by the CLI. This section explains how to control logging output and levels, and how to customize signal handling and exception reporting.

Toys-Core provides a class called Toys::Utils::StandardUI that implements the diagnostic output format used by the toys gem. We'll look at how to use the StandardUI after discussing each type of diagnostic output.

Logging

Toys provides a Logger for each tool execution. Tools can access this Logger by calling the logger method, or by getting the Toys::Context::Key::LOGGER context object.

#!/usr/bin/env ruby

require "toys-core"

cli = Toys::CLI.new

cli.add_source do
  tool "hello" do
    def run
      logger.info "This log entry is displayed in verbose mode."
    end
  end
end

result = cli.run(*ARGV)
exit(result)

Log level and verbosity

The logging level is controlled by the verbosity setting when the tool is invoked. This built-in attribute starts at 0, and by convention can be increased or decreased by the user by passing the --verbose or --quiet flags. (These flags are not provided by the CLI itself, but are implemented by middleware, which we will cover later.) Its final setting is then mapped to a Logger level threshold.

By default, a verbosity of 0 maps to log level Logger::WARN. Entries logged at level Logger::WARN or higher are displayed, whereas entries logged at Logger::INFO or Logger::DEBUG are suppressed. If the user increases the verbosity by passing --verbose or -v, a verbosity of 1 will move the log level threshold down to Logger::INFO.

You can modify the starting verbosity value by passing it to Toys::CLI#run. Passing verbosity: 1 will set the starting verbosity to 1, meaning Logger::INFO entries will display but Logger::DEBUG entries will not. If the invoker then provides an extra --verbose flag, the verbosity will further increase to 2, allowing Logger::DEBUG entries to appear.

# ...
result = cli.run(*ARGV, verbosity: 1)
exit(result)

You can also modify the log level that verbosity 0 maps to by passing the base_level argument to the CLI constructor. The following causes verbosity 0 to map to Logger::INFO rather than Logger::WARN.

cli = Toys::CLI.new(base_level: Logger::INFO)

If you do not pass a base_level, verbosity 0 maps to whatever level the logger has before a run adjusts it. A nested run that shares the same logger with the run that called it uses that same base level rather than the level the caller adjusted it to, so verbosity does not compound: if a tool running at verbosity 1 calls another tool at verbosity 1, both log at Logger::INFO.

Customizing the logger

Toys-Core configures its default logger with the default logging formatter, and configures it to log to STDERR. If you want to change any of these settings, you can provide your own logger by passing a logger to the CLI constructor constructor.

my_logger = Logger.new("my_logfile.log")
cli = Toys::CLI.new(logger: my_logger)

A logger passed directly to the CLI is global. The CLI will attempt to use it for every execution, even if multiple executions are happening concurrently. In the concurrent case, this might cause problems if those executions attempt to use different verbosity settings, as the log level thresholds will conflict. If your CLI might be run multiple times concurrently, we recommend instead passing a logger_factory to the CLI constructor. This is a Proc that will be invoked to create a new logger for each execution.

my_logger_factory = Proc.new do
  Logger.new("my_logfile.log")
end
cli = Toys::CLI.new(logger_factory: my_logger_factory)

StandardUI logging

Toys::Utils::StandardUI implements the logger used by the toys gem, which formats log entries with the severity and timestamp using ANSI coloring.

You can use this logger by passing the proc returned by Toys::Utils::StandardUI#logger_factory_proc to the CLI constructor:

standard_ui = Toys::Utils::StandardUI.new
cli = Toys::CLI.new(logger_factory: standard_ui.logger_factory_proc)

You can also customize the logger by subclassing StandardUI and overriding its methods or adjusting its parameters. In particular, you can alter the Toys::Utils::StandardUI#log_header_severity_styles mapping to adjust styling, or override Toys::Utils::StandardUI#create_logger or Toys::Utils::StandardUI#format_log_entry to adjust content and formatting.

Handling errors

If an unhandled exception (specifically an exception represented by a subclass of StandardError or ScriptError) occurs during tool execution, Toys-Core first wraps the exception in a Toys::ContextualError. This error type provides various context fields such as an estimate of where in the tool source the error may have occurred. It also provides the original exception in the cause field.

When one tool invokes another, whether through delegate_to or by calling Toys::Runner#run or Toys::CLI#run from within a tool, the wrappers nest, so that each level records the tool name, arguments, and source location for its own tool. In that case the cause field holds the next Toys::ContextualError in the chain rather than the original exception. Use Toys::ContextualError#root_cause to reach the original exception regardless of how deeply it is nested.

Signals are not wrapped. If a signal such as an interrupt (represented by a SignalException) is received during tool execution, and no tool intercepts it with an on_interrupt or on_signal handler, it propagates unwrapped. This means a tool that delegates to another tool can still intercept a signal raised while the inner tool was running. That is not true of a tool that invokes another tool by calling Toys::Runner#run itself; see nested runs below.

Then, Toys-Core invokes the error handler, a Proc held by the Toys::Runner. You normally set it as a configuration argument when constructing a CLI, which passes it along to the Runner it creates. An error handler takes the error as its argument and should perform any desired final handling of an unhandled exception, such as displaying the error to the terminal, or reraising the exception. The handler should then return the desired result code for the run.

The argument is one of the following:

  • A Toys::ContextualError wrapper. This is how an ordinary error arrives.
  • A bare StandardError or ScriptError, if the run disabled error wrapping. Toys::CLI#run always wraps, so this happens only if you call Toys::Runner#run yourself with wrap_errors: false, on a Runner you constructed or on the CLI's own Toys::CLI#runner.
  • A bare SignalException, which is never wrapped.
my_error_handler = Proc.new do |error|
  # Propagate signals out and let the Ruby VM handle them. Signals arrive
  # unwrapped, so this is a direct type check on the argument.
  raise error if error.is_a?(SignalException)
  # Handle any other exception types by printing a message. Here `error` is
  # normally a Toys::ContextualError; use root_cause to reach the original
  # exception.
  $stderr.puts "An error occurred. Please contact your administrator."
  # Return the result code
  255
end
cli = Toys::CLI.new(error_handler: my_error_handler)

If you do not set an error handler, the error is raised out of the Toys::CLI#run call as-is. Signals are raised directly so that the Ruby VM can handle them normally. For other exceptions, the outermost Toys::ContextualError wrapper is raised so that a rescue block has access to the context information.

Nested runs

A tool can run another tool in the same process by calling Toys::Runner#run on its Toys::Context#runner. Such a nested run performs its own error handling by default, exactly as the outer run does, so an error raised by the inner tool is routed to the configured error handler before the calling tool sees anything. What happens next depends on the handler:

  • With the default handler, the error is reraised, so it propagates into the calling tool and that tool does not continue past the call.
  • With a reporting handler such as the one from Toys::Utils::StandardUI, the error is printed and a result code is returned from the nested Toys::Runner#run call. The calling tool continues, and must check that result code itself.

This also applies to signals, and is the exception to the rule above that a signal propagates as itself through every tool it passes through. With a reporting handler, an interrupt raised by the inner tool is caught by the nested run's error handling, printed, and converted to result code 130, so the calling tool's on_interrupt handler never fires.

Delegation behaves differently: a delegated tool is part of the same run, so its errors are not handled separately. The error handler fires once, at the outermost run.

If you want a nested run to leave errors to the caller, pass handle_errors: false to Toys::Runner#run.

StandardUI error handling

Toys::Utils::StandardUI provides the error handler used by the toys gem. For normal exceptions, this standard handler displays the exception to STDERR, along with some contextual information such as the tool name and arguments and the location in the tool source where the error occurred, and returns an appropriate result code, typically 1.

A backtrace is also displayed, which can have "uninteresting" frames omitted by passing an array of library directories in the backtrace_omit_prefixes argument. It is often useful, for example, to pass Toys.framework_lib_paths to omit Toys framework files from the backtrace.

For signals, this standard handler displays a brief message noting the signal or interrupt, and returns the conventional result code of 128 + signo (e.g. 130 for interrupts).

You can use this error handler by passing the proc returned by Toys::Utils::StandardUI#error_handler_proc to the CLI constructor:

standard_ui = Toys::Utils::StandardUI.new
cli = Toys::CLI.new(error_handler: standard_ui.error_handler_proc)

You can also customize the error handler by subclassing StandardUI and overriding its methods. In particular, you can alter what is displayed in response to errors or signals by overriding Toys::Utils::StandardUI#display_error_notice or Toys::Utils::StandardUI#display_signal_notice, respectively, and you can alter how exit codes are generated by overriding Toys::Utils::StandardUI#exit_code_for.

Nonstandard exceptions

Toys-Core error handling handles normal exceptions that are subclasses of StandardError, errors coming from Ruby file loading and parsing that are subclasses of ScriptError, and signals that are subclasses of SignalException. The first two are wrapped in a Toys::ContextualError before being passed to the error handler; signals are passed through unwrapped.

Other exceptions such as NoMemoryError or SystemStackError are not handled by Toys, and are raised directly out of the Toys::CLI#run.

Customizing default behavior

Command line tools often have a set of common behaviors, such as online help, flags that control verbosity, and handlers for option parsing errors and corner cases. In Toys-Core, a few of these common behaviors are built into the CLI class as described above, but others are implemented and configured using middleware.

Toys Middleware is analogous to middleware in other frameworks. It is code that "wraps" tools defined in a Toys CLI and makes modifications. Middleware can, for example, modify the tool's properties such as its description, modify the arguments accepted by the tool, and/or modify the execution of the tool, by injecting code before and/or after the tool's execution, or even replacing the execution altogether.

Introducing middleware

A middleware object must duck-type Toys::Middleware, although it does not necessarily need to include the module itself. Toys::Middleware defines two methods, Toys::Middleware#config and Toys::Middleware#run. The first is is called after a tool is defined, and lets the middleware modify the tool's definition, e.g. to modify or provide defaults for properties such as description and common flags. The second is called when a tool is executed, and lets the middleware modify the tool's execution.

Middleware is arranged in a stack, where each middleware object "wraps" the objects below it. Each middleware object's methods can implement its own functionality, and then either pass control to the next middleware in the stack, or stop processing and disable the rest of the stack. In particular, if a middleware stops processing during the Toys::Middleware#run call, the normal tool execution is also canceled; hence, middleware can even be used to replace normal tool execution.

Configuring middleware

Middleware is normally configured as part of the CLI object. Each CLI includes an ordered list, a stack, of middleware specifications, each represented by Toys::Middleware::Spec. A middleware spec can reference a specific middleware object, a class to instantiate, or a name that can be looked up from a directory of middleware class files. You can pass an array of these specs to a CLI object when you instantiate it.

A useful example can be seen in the default Toys CLI behavior. If you do not provide a middleware stack when instantiating Toys::CLI, the class uses a default stack that looks approximately like this:

[
  Toys::Middleware.spec(:set_default_descriptions),
  Toys::Middleware.spec(:show_help, help_flags: true, fallback_execution: true),
  Toys::Middleware.spec(:handle_usage_errors),
  Toys::Middleware.spec(:add_verbosity_flags),
]

Each of the names, e.g. :set_default_descriptions, is the name of a Ruby file in the toys-core gem under toys/standard_middleware. You can configure the middleware system to recognize middleware by name, by providing a middleware lookup object, of type Toys::ModuleLookup. This object is configured with one or more directories, and if you provide a name, it looks for an appropriate module of that name in a ruby file in those directories. By default, the middleware lookup in Toys::CLI looks for middleware in the toys/standard_middleware directory in the toys-core gem, but you can configure it to look elsewhere.

Note also that, in the case of :show_help, the stack above also includes some options that are passed to the Toys::StandardMiddleware::ShowHelp middleware constructor when it is instantiated.

You can also look at the middleware stack in the Toys::StandardCLI class in the toys gem to see the middleware as the toys executable configures it.

Built-in middlewares

The toys-core gem provides several useful middleware classes that you can use when configuring your own CLI. These live in the toys/standard_middlware directory, and are available by name if you keep the default middleware lookup. These built-in middlewares include:

Writing your own middleware

Writing your own middleware is as simple as writing a class that implements the Toys::Middleware#config and/or Toys::Middleware#run methods. The middleware class need not include the Toys::Middleware module; it merely needs to duck-type at least one of its methods. Your class can then be used in the stack of middleware specifications.

Example: TimingMiddleware

An example would probably do best to illustrate how to write middleware. The following is a simple middleware that adds the --show-timing flag to every tool. When the flag is set, the middleware displays how long the tool took to execute.

class TimingMiddleware
  # This is a context key that will be used to store the "--show-timing"
  # flag state. {Toys::UniqueKey} is a convenient way to create a unique key,
  # with a useful name, but you can also simply use `Object.new`.
  KEY = Toys::UniqueKey.new("TimingMiddleware::KEY")

  # This method intercepts tool configuration. We use it to add a flag that
  # enables timing display.
  def config(tool, _loader)
    # Add a flag to control this functionality. Suppress collisions, i.e.
    # just silently do nothing if the tool has already added a flag called
    # "--show-timing".
    tool.add_flag(KEY, "--show-timing", report_collisions: false)

    # Calling yield passes control to the rest of the middleware stack.
    # Normally you should call yield, to ensure that the remaining
    # middleware can run. If you omit this, no additional middleware will
    # be able to run tool configuration. Note you can also perform
    # additional processing after the yield call, i.e. after the rest of
    # the middleware stack has run.
    yield
  end

  # This method intercepts tool execution. We use it to collect timing
  # information, and display it if the flag has been provided in the
  # command line arguments.
  def run(context)
    # Read monotonic time at the start of execution.
    start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)

    # Call yield to run the rest of the middleware stack, including the
    # actual tool execution. If you omit this, you will prevent the rest of
    # the middleware stack, AND the actual tool execution, from running.
    # So you could omit the yield call if your goal is to replace tool
    # execution with your own code.
    yield

    # Read monotonic time again after execution.
    end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)

    # Display the elapsed time, if the tool was passed the "--show-timing"
    # flag.
    puts "Tool took #{end_time - start_time} secs" if context[KEY]
  end
end

We can now insert our middleware into the stack when we create a CLI. Here we'll take that "default" stack we saw earlier and add our timing middleware at the top of the stack. We put it here so that its execution "wraps" all the other middleware, and thus its timing measurement includes the latency incurred by other middleware (including middleware that replaces execution such as :show_help).

my_middleware_stack = [
  Toys::Middleware.spec(TimingMiddleware),
  Toys::Middleware.spec(:set_default_descriptions),
  Toys::Middleware.spec(:show_help, help_flags: true, fallback_execution: true),
  Toys::Middleware.spec(:handle_usage_errors),
  Toys::Middleware.spec(:add_verbosity_flags),
]
cli = Toys::CLI.new(middleware_stack: my_middleware_stack)

Now, every tool run by this CLI will have the --show-timing flag and associated functionality.

Packaging your executable

Simple executables can be written into a single Ruby file and run as a script. We saw an example of this at the beginning of this guide. The only requirement is that the user must have Ruby installed, as well as the toys-core gem.

However, if you want to distribute an executable using Toys-Core, you may find it best to create a Rubygem. Your users can install your gem, which can be configured to bring in the toys-core gem automatically as a dependency, and the executable will be added to their path. For more information on the basic Rubygems mechanism for this, see the Rubygems documentation.

The Toys GitHub repository https://github.com/dazuma/toys includes some examples of executables packaged in gems. Let's take a look at these.

A gem with a simple executable

The simple-gem example illustrates packaging a simple executable that can be implemented in a single file using Toys::CLI#add_source. It demonstrates the gem toys-core-simple-example that provides an executable also called toys-core-simple-example.

simple-gem/
|
+- bin/
|  |
|  +- toys-core-simple-example
|
+- lib/
|  |
|  +- toys-core-simple-example.rb
|
+- toys-core-simple-example.gemspec

The executable file toys-core-simple-example should live in the bin/ directory in the Rubygem. By convention, Rubygems expects executables to live there, and it will ensure they are available in the user's $PATH. The file should have its executable bit set (e.g. rwxr-xr-x.) For this example, we just used the code from the beginning of this guide.

require "toys-core"

cli = ::Toys::CLI.new

cli.add_source do
  desc "Display a simple greeting"
  flag :whom, default: "world"
  def run
    puts "Hello, #{whom}!"
  end
end

exit(cli.run(::ARGV))

Note that there is also a lib/toys-core-simple-example.rb file that is basically empty. Rubygems are meant to contain Ruby libraries, and get confused if there aren't any Ruby library files. So we include this token file. It doesn't actually need to include anything; it just needs to be present.

Finally, let's note a few features of the gemspec:

::Gem::Specification.new do |spec|
  spec.name = "toys-core-simple-example"
  spec.version = "0.0.1"
  spec.authors = ["Daniel Azuma"]
  spec.email = ["dazuma@gmail.com"]

  spec.summary = "An example command line gem created using toys-core"
  spec.description =
    "An example command line gem created using toys-core. For more" \
    " information on toys-core, see https://github.com/dazuma/toys"
  spec.license = "MIT"
  spec.homepage = "https://github.com/dazuma/toys"

  spec.files = ::Dir.glob("*.md") + ::Dir.glob("bin/*")
  spec.required_ruby_version = ">= 2.7.0"
  spec.require_paths = ["lib"]

  spec.bindir = "bin"
  spec.executables = ["toys-core-simple-example"]

  spec.add_dependency "toys-core", "~> 0.21"
end

First, the bin directory and the executable file needs to be included in the gem (i.e. present in spec.files.) Second, we declare the executable by setting spec.bindir and spec.executables, so that Rubygems knows the executable should be present in the user's $PATH. Finally, note that toys-core is declared as a dependency.

That's all there is to it. You can build and install this gem, and the command line executable program toys-core-simple-example will be made available.

A gem using a tools directory

For more complex programs, you may want the actual tool definitions to live in a directory of files, much like you'd use a .toys directory to write and manage more complex sets of Toys tools. To package such a program in a gem, just include the tools directory in the gem, and load it into the CLI using Toys::CLI#add_source. The multi-file-gem example example illustrates how to do this.

multi-file-gem/
|
+- bin/
|  |
|  +- toys-core-multi-file-example
|
+- lib/
|  |
|  +- toys-core-multi-file-example.rb
|
+- tools/
|  |
|  +- greet.rb
|  |
|  +- new-repo.rb
|
+- toys-core-multi-file-example.gemspec

In this example, we chose to implement the application in the lib directory, and just have the executable be a brief script that loads and runs that implementation.

The executable bin/toys-core-multi-file-example:

#!/usr/bin/env ruby
require "toys-core-multi-file-example"
ToysCoreExample.new.run

And the library that gets required lib/toys-core-multi-file-example.rb:

require "toys-core"

class ToysCoreExample
  def initialize
    @cli = ::Toys::CLI.new
    @cli.add_source(::File.join(::File.dirname(__dir__), "tools"))
  end

  def run
    exit(@cli.run(::ARGV))
  end
end

Note the file path we pass to add_source backs out to the gem's root directory, and adds the tools directory from there. You can do this because the lib and tools directories will always be installed as part of the gem. (You might want to do this instead of creating that relative path from the executable itself, so that it's possible to move the executable elsewhere.)

In the tools/ directory, you can define tools just like you would normally. You also have access to all the features of Toys tool definition, such as hierarchical subtools, and even .lib and .data subdirectories if you need shared code or data.

Finally, the gemspec:

::Gem::Specification.new do |spec|
  spec.name = "toys-core-multi-file-example"
  spec.version = "0.0.1"
  spec.authors = ["Daniel Azuma"]
  spec.email = ["dazuma@gmail.com"]

  spec.summary = "An example command line gem created using toys-core"
  spec.description =
    "An example command line gem created using toys-core. For more" \
    " information on toys-core, see https://github.com/dazuma/toys"
  spec.license = "MIT"
  spec.homepage = "https://github.com/dazuma/toys"

  spec.files = ::Dir.glob("*.md") + ::Dir.glob("bin/*") +
               ::Dir.glob("lib/**/*.rb") + ::Dir.glob("tools/**/*.rb")
  spec.required_ruby_version = ">= 2.7.0"
  spec.require_paths = ["lib"]

  spec.bindir = "bin"
  spec.executables = ["toys-core-multi-file-example"]

  spec.add_dependency "toys-core", "~> 0.21"
end

Note that we include the tools directory and its contents in spec.files to ensure they are included in the gem.

Overview of Toys-Core classes

This reference section provides a roadmap to the classes in Toys-Core.

The Toys-Core framework can roughly be divided into sections, as follows. Each of these is implemented in the corresponding Ruby file under lib.

Tool definition classes

These provide the objects that make up the definition of a tool.

  • Toys::ToolDefinition - This is the main class, representing the complete definition of a single tool.
  • Toys::Flag - The definition of a single flag, which may have arguments. Contained in a tool definition.
  • Toys::FlagGroup - The definition of a flag group, a collection of related flags that may have common requirement settings. Contained in a tool definition.
  • Toys::PositionalArg - The definition of a positional argument, which could be required or optional. There is also a special case for an arbitrary-length array of "remaining" args. Contained in a tool definition.
  • Toys::Acceptor - Represents how to validate arguments (either positional or flag arguments) and optionally convert from strings to Ruby objects. This module contains various classes implementing both "well-known" acceptors as defined in the OptionParser interface, and provides various additional options as well as the interface for custom acceptors.
  • Toys::Completion - Represents how to autocomplete command line arguments. Like acceptor, this module contains classes for well-known completion techniques, and the interfaces needed for implementing custom completions.

Tool definition DSL

The DSL is implemented under the lib/dsl subdirectory, and provides the objects defining the DSL directives.

  • Toys::DSL::Tool - This defines the main DSL, including all directives for defining a tool, including the tool directive that defines a subtool block.
  • Toys::DSL::Flag - This defines the directives available within a flag directive block.
  • Toys::DSL::FlagGroup - This defines the directives available within a flag_group directive block and related directives such as exactly_one.
  • Toys::DSL::PositionalArg - This defines the directives available within blocks passed to the positional argument directives such as required_arg.
  • Toys::Tool - This is a base class that you can use to define tools using the class syntax.

Tool loading and resolution

This section includes classes implementing the tool loading logic as described above under the Loader.

  • Toys::Loader - This class implements the lazy tool loader. It is created from a fixed list of the various sources of tool definitions (such as files and blocks) and responds to requests for tools by name, returning a Toys::ToolDefinition.
  • Toys::SourceSpec - A source spec is an unresolved description of a source: its kind (path, git, gem, or block) and the information needed to locate it. Build one with Toys::SourceSpec.path, Toys::SourceSpec.git, Toys::SourceSpec.gem, or Toys::SourceSpec.block, and hand it to Toys::CLI#add_source. Creating a spec touches no file system, git remote, or gem.
  • Toys::SourceList - This class holds the ordered list of source specs that a loader is created from, assigning each a priority. A Toys::CLI owns one of these, and its source-adding methods delegate to it, but you can also use it directly if you are constructing a Toys::Loader yourself.
  • Toys::SourceInfo - This object provides metadata about a source for a tool definition, which could be a directory, a file, or a block. It is what a source spec resolves to: it locates gem and git sources on the local file system, activating gems and populating a local git cache as needed. That resolution happens in the loader, at the first tool lookup, rather than when the source is added; pass git_cache or gems_util to Toys::CLI#initialize if you need control over the Toys::Utils::GitCache or Toys::Utils::Gems objects it uses. This information is used to reference at the source when listing tools, and when reporting errors.
  • Toys::InputFile - This is the module that contains actual tool definition classes. Tool definitions (and any other constants and classes defined with them) are not placed at the top level, but in submodules of InputFile, so that they do not clash with other definitions.

Tool execution

This section includes classes involved in tool execution

  • Toys::CLI - The configuration and entry point for the framework. It provides a general configuration interface for all of the Toys features, owns a Toys::Loader that it uses to load tool definitions, and owns a Toys::Runner that it uses to respond to command line invocations.
  • Toys::Runner - The environment in which tools run. Given a command line, it looks up the tool, parses the arguments, builds the Toys::Context, applies the tool's middleware, runs the tool, and wraps any resulting error in a Toys::ContextualError. Most applications should use Toys::CLI#run rather than creating a Toys::Runner directly.
  • Toys::ArgParser - A service that parses command line argument lists and matches the given arguments against the tool's formal flags and arguments definition, producing the parsed values along with any usage errors.
  • Toys::Context - This class is self during a tool's execution, and the tool's methods, including the entrypoint run method, are defined in a subclass of this class. This class also provides methods for retrieving flag and argument values and other contextual information.

Code sharing

The mixin definition is Toys::Mixin; this module should be included in every mixin. Toys-Core also provides a suite of standard mixins under the Toys::StandardMixins module.

The middleware interface is defined by Toys::Middleware, which also provides a set of module methods and classes for defining middleware specifications as used to configure a Toys::CLI.

The template definition is Toys::Template; this module should be included in every template. Toys-Core does not itself provide any templates, but the toys gem includes a suite of common templates useful for Ruby project tools such as builds and tests.

Exception classes

Exception classes are defined in lib/errors.rb.

  • Toys::ContextualError - This is the error that is generally raised from Toys::CLI#run. It wraps an actual error, and provides source information indicating where in the tool definition the error was raised. When one tool invokes another, these wrappers nest, one per tool; use Toys::ContextualError#root_cause to reach the original error.
  • Toys::ArgParsingError - Raised during argument parsing to indicate that parsing failed. If present, it will contain one or more individual Toys::ArgParser::UsageError exceptions.
  • Toys::NotRunnableError - Raised during tool execution if the tool has no entrypoint. (In the toys gem, this is generally caught by the help system and redirected to display the usage screen.)
  • Toys::ToolDefinitionError - Raised during tool definition in response to semantic issues with the definition.
  • Toys::ToolSourceError - Raised during tool loading to indicate failure to load files or other resources involved in tool definition.

Support and utility classes

These classes provide functional support for different parts of the system

  • Toys::Compat - A set of logic for distinguishing differences in Ruby capability by Ruby version and platform.
  • Toys::ModuleLookup - A utility for looking up modules by symbolic name. This is how "well-known" mixins and templates are looked up by symbol.
  • Toys::WrappableString - An object representing a string that knows how to be text-wrapped. This is used in fields such as descriptions.

Additional classes live under lib/utils/. These are distinguished because, unlike all other classes under toys-core, they are not loaded by default. Any use of these classes must be preceded by an appropriate require. In general, classes are put here if they are both (1) of a utility nature, and (2) not necessarily going to be used by every tool execution, so it may be beneficial to defer require-ing the file until/unless it's actually needed.