pluggy

The pytest plugin system

pluggy is the crystallized core of plugin management and hook calling for pytest.

In fact, pytest is itself composed as a set of pluggy plugins which are invoked in sequence according to a well defined set of protocols. Some 200+ plugins use pluggy to extend and customize pytest‘s default behaviour.

In essence, pluggy enables function hooking so you can build “pluggable” systems.

How’s it work?

A plugin is a namespace which defines hook functions.

pluggy manages plugins by relying on:

  • a hook specification - defines a call signature
  • a set of hook implementations - aka callbacks
  • the hook caller - a call loop which collects results

where for each registered hook specification, a hook call will invoke up to N registered hook implementations.

pluggy accomplishes all this by implementing a request-response pattern using function subscriptions and can be thought of and used as a rudimentary busless publish-subscribe event system.

pluggy‘s approach is meant to let a designer think carefuly about which objects are explicitly needed by an extension writer. This is in contrast to subclass-based extension systems which may expose unecessary state and behaviour or encourage tight coupling in overlying frameworks.

A first example

import pluggy

hookspec = pluggy.HookspecMarker("myproject")
hookimpl = pluggy.HookimplMarker("myproject")


class MySpec(object):
    """A hook specification namespace.
    """
    @hookspec
    def myhook(self, arg1, arg2):
        """My special little hook that you can customize.
        """


class Plugin_1(object):
    """A hook implementation namespace.
    """
    @hookimpl
    def myhook(self, arg1, arg2):
        print("inside Plugin_1.myhook()")
        return arg1 + arg2


class Plugin_2(object):
    """A 2nd hook implementation namespace.
    """
    @hookimpl
    def myhook(self, arg1, arg2):
        print("inside Plugin_2.myhook()")
        return arg1 - arg2


# create a manager and add the spec
pm = pluggy.PluginManager("myproject")
pm.add_hookspecs(MySpec)

# register plugins
pm.register(Plugin_1())
pm.register(Plugin_2())

# call our `myhook` hook
results = pm.hook.myhook(arg1=1, arg2=2)
print(results)

Running this directly gets us:

$ python docs/examples/firstexample.py

inside Plugin_2.myhook()
inside Plugin_1.myhook()
[-1, 3]

For more details and advanced usage please read on.

Defining and Collecting Hooks

A plugin is a namespace type (currently one of a class or module) which defines a set of hook functions.

As mentioned in The Plugin Registry, all plugins which define hooks are managed by an instance of a pluggy.PluginManager which defines the primary pluggy API.

In order for a PluginManager to detect functions in a namespace intended to be hooks, they must be decorated using special pluggy marks.

Marking hooks

The HookspecMarker and HookimplMarker decorators are used to mark functions for detection by a PluginManager:

from pluggy import HookspecMarker, HookimplMarker

hookspec = HookspecMarker('project_name')
hookimpl = HookimplMarker('project_name')

Each decorator type takes a single project_name string as its lone argument the value of which is used to mark hooks for detection by by a similarly configured PluginManager instance.

That is, a mark type called with project_name returns an object which can be used to decorate functions which will then be detected by a PluginManager which was instantiated with the the same project_name value.

Furthermore, each hookimpl or hookspec decorator can configure the underlying call-time behavior of each hook object by providing special options passed as keyword arguments.

Note

The following sections correspond to similar documentation in pytest for Writing hook functions and can be used as a supplementary resource.

Implementations

A hook implementation (hookimpl) is just a (callback) function which has been appropriately marked.

hookimpls are loaded from a plugin using the register() method:

import sys
from pluggy import PluginManager, HookimplMarker

hookimpl = HookimplMarker('myproject')

@hookimpl
def setup_project(config, args):
    """This hook is used to process the initial config
    and possibly input arguments.
    """
    if args:
        config.process_args(args)

    return config

pm = PluginManager('myproject')

# load all hookimpls from the local module's namespace
plugin_name = pm.register(sys.modules[__name__])

Optional validation

Normally each hookimpl should be validated a against a corresponding hook specification. If you want to make an exception then the hookimpl should be marked with the "optionalhook" option:

@hookimpl(optionalhook=True)
def setup_project(config, args):
    """This hook is used to process the initial config
    and possibly input arguments.
    """
    if args:
        config.process_args(args)

    return config

Call time order

By default hooks are called in LIFO registered order, however, a hookimpl can influence its call-time invocation position using special attributes. If marked with a "tryfirst" or "trylast" option it will be executed first or last respectively in the hook call loop:

import sys
from pluggy import PluginManager, HookimplMarker

hookimpl = HookimplMarker('myproject')

@hookimpl(trylast=True)
def setup_project(config, args):
    """Default implementation.
    """
    if args:
        config.process_args(args)

    return config


class SomeOtherPlugin(object):
    """Some other plugin defining the same hook.
    """
    @hookimpl(tryfirst=True)
    def setup_project(config, args):
        """Report what args were passed before calling
        downstream hooks.
        """
        if args:
            print("Got args: {}".format(args))

        return config

pm = PluginManager('myproject')

# load from the local module's namespace
pm.register(sys.modules[__name__])
# load a plugin defined on a class
pm.register(SomePlugin())

For another example see the hook function ordering section of the pytest docs.

Note

tryfirst and trylast hooks are still invoked in LIFO order within each category.

Wrappers

A hookimpl can be marked with a "hookwrapper" option which indicates that the function will be called to wrap (or surround) all other normal hookimpl calls. A hookwrapper can thus execute some code ahead and after the execution of all corresponding non-wrappper hookimpls.

Much in the same way as a @contextlib.contextmanager, hookwrappers must be implemented as generator function with a single yield in its body:

@hookimpl(hookwrapper=True)
def setup_project(config, args):
    """Wrap calls to ``setup_project()`` implementations which
    should return json encoded config options.
    """
    if config.debug:
        print("Pre-hook config is {}".format(
            config.tojson()))

    # get initial default config
    defaults = config.tojson()

    # all corresponding hookimpls are invoked here
    outcome = yield

    for item in outcome.get_result():
        print("JSON config override is {}".format(item))

    if config.debug:
        print("Post-hook config is {}".format(
            config.tojson()))

    if config.use_defaults:
        outcome.force_result(defaults)

The generator is sent a pluggy._Result object which can be assigned in the yield expression and used to override or inspect the final result(s) returned back to the caller using the force_result() or get_result() methods.

Note

Hook wrappers can not return results (as per generator function semantics); they can only modify them using the _Result API.

Also see the hookwrapper section in the pytest docs.

Specifications

A hook specification (hookspec) is a definition used to validate each hookimpl ensuring that an extension writer has correctly defined their callback function implementation .

hookspecs are defined using similarly marked functions however only the function signature (its name and names of all its arguments) is analyzed and stored. As such, often you will see a hookspec defined with only a docstring in its body.

hookspecs are loaded using the add_hookspecs() method and normally should be added before registering corresponding hookimpls:

import sys
from pluggy import PluginManager, HookspecMarker

hookspec = HookspecMarker('myproject')

@hookspec
def setup_project(config, args):
    """This hook is used to process the inital config and input
    arguments.
    """

pm = PluginManager('myproject')

# load from the local module's namespace
pm.add_hookspecs(sys.modules[__name__])

Registering a hookimpl which does not meet the constraints of its corresponding hookspec will result in an error.

A hookspec can also be added after some hookimpls have been registered however this is not normally recommended as it results in delayed hook validation.

Note

The term hookspec can sometimes refer to the plugin-namespace which defines hookspec decorated functions as in the case of pytest‘s hookspec module

Enforcing spec validation

By default there is no strict requirement that each hookimpl has a corresponding hookspec. However, if you’d like you enforce this behavior you can run a check with the check_pending() method. If you’d like to enforce requisite hookspecs but with certain exceptions for some hooks then make sure to mark those hooks as optional.

Opt-in arguments

To allow for hookspecs to evolve over the lifetime of a project, hookimpls can accept less arguments then defined in the spec. This allows for extending hook arguments (and thus semantics) without breaking existing hookimpls.

In other words this is ok:

@hookspec
def myhook(config, args):
    pass

@hookimpl
def myhook(args):
    print(args)

whereas this is not:

@hookspec
def myhook(config, args):
    pass

@hookimpl
def myhook(config, args, extra_arg):
    print(args)

First result only

A hookspec can be marked such that when the hook is called the call loop will only invoke up to the first hookimpl which returns a result other then None.

@hookspec(firstresult=True)
def myhook(config, args):
    pass

This can be useful for optimizing a call loop for which you are only interested in a single core hookimpl. An example is the pytest_cmdline_main central routine of pytest.

Also see the first result section in the pytest docs.

Historic hooks

You can mark a hookspec as being historic meaning that the hook can be called with call_historic() before having been registered:

@hookspec(historic=True)
def myhook(config, args):
    pass

The implication is that late registered hookimpls will be called back immediately at register time and can not return a result to the caller.**

This turns out to be particularly useful when dealing with lazy or dynamically loaded plugins.

For more info see Historic calls.

The Plugin Registry

pluggy manages plugins using instances of the pluggy.PluginManager.

A PluginManager is instantiated with a single str argument, the project_name:

import pluggy
pm = pluggy.PluginManager('my_project_name')

The project_name value is used when a PluginManager scans for hook functions defined on a plugin. This allows for multiple plugin managers from multiple projects to define hooks alongside each other.

Registration

Each PluginManager maintains a plugin registry where each plugin contains a set of hookimpl definitions. Loading hookimpl and hookspec definitions to populate the registry is described in detail in the section on Defining and Collecting Hooks.

In summary, you pass a plugin namespace object to the register() and add_hookspec() methods to collect hook implementations and specfications from plugin namespaces respectively.

You can unregister any plugin‘s hooks using unregister() and check if a plugin is registered by passing its name to the is_registered() method.

Loading setuptools entry points

You can automatically load plugins registered through setuptools entry points with the load_setuptools_entrypoints() method.

An example use of this is the pytest entry point.

Blocking

You can block any plugin from being registered using set_blocked() and check if a given plugin is blocked by name using is_blocked().

Inspection

You can use a variety of methods to inspect the both the registry and particular plugins in it:

Parsing mark options

You can retrieve the options applied to a particular hookspec or hookimpl as per Marking hooks using the parse_hookspec_opts() and parse_hookimpl_opts() respectively.

Calling Hooks

The core functionality of pluggy enables an extension provider to override function calls made at certain points throughout a program.

A particular hook is invoked by calling an instance of a pluggy._HookCaller which in turn loops through the 1:N registered hookimpls and calls them in sequence.

Every pluggy.PluginManager has a hook attribute which is an instance of this pluggy._HookRelay. The _HookRelay itself contains references (by hook name) to each registered hookimpl‘s _HookCaller instance.

More practically you call a hook like so:

import sys
import pluggy
import mypluginspec
import myplugin
from configuration import config

pm = pluggy.PluginManager("myproject")
pm.add_hookspecs(mypluginspec)
pm.register(myplugin)

# we invoke the _HookCaller and thus all underlying hookimpls
result_list = pm.hook.myhook(config=config, args=sys.argv)

Note that you must call hooks using keyword arguments syntax!

Hook implementations are called in LIFO registered order: the last registered plugin’s hooks are called first. As an example, the below assertion should not error:

from pluggy import PluginManager, HookimplMarker

hookimpl = HookimplMarker('myproject')

class Plugin1(object):
    def myhook(self, args):
        """Default implementation.
        """
        return 1

class Plugin2(object):
    def myhook(self, args):
        """Default implementation.
        """
        return 2

class Plugin3(object):
    def myhook(self, args):
        """Default implementation.
        """
        return 3

pm = PluginManager('myproject')
pm.register(Plugin1())
pm.register(Plugin2())
pm.register(Plugin3())

assert pm.hook.myhook(args=()) == [3, 2, 1]

Collecting results

By default calling a hook results in all underlying hookimpls functions to be invoked in sequence via a loop. Any function which returns a value other then a None result will have that result appended to a list which is returned by the call.

The only exception to this behaviour is if the hook has been marked to return its First result only in which case only the first single value (which is not None) will be returned.

Historic calls

A historic call allows for all newly registered functions to receive all hook calls that happened before their registration. The implication is that this is only useful if you expect that some hookimpls may be registered after the hook is initially invoked.

Historic hooks must be specially marked and called using the pluggy._HookCaller.call_historic() method:

# call with history; no results returned
pm.hook.myhook.call_historic(config=config, args=sys.argv)

# ... more of our program ...

# late loading of some plugin
import mylateplugin

# historic call back is done here
pm.register(mylateplugin)

Note that if you call_historic() the _HookCaller (and thus your calling code) can not receive results back from the underlying hookimpl functions.

Calling with extras

You can call a hook with temporarily participating implementation functions (that aren’t in the registry) using the pluggy._HookCaller.call_extra() method.

Calling with a subset of registered plugins

You can make a call using a subset of plugins by asking the PluginManager first for a _HookCaller with those plugins removed using the pluggy.PluginManager.subset_hook_caller() method.

You then can use that _HookCaller to make normal, call_historic(), or call_extra() calls as necessary.

Built-in tracing

pluggy comes with some batteries included hook tracing for your debugging needs.

Call tracing

To enable tracing use the pluggy.PluginManager.enable_tracing() method which returns an undo function to disable the behaviour.

pm = PluginManager('myproject')
# magic line to set a writer function
pm.trace.root.setwriter(print)
undo = pm.enable_tracing()

Call monitoring

Instead of using the built-in tracing mechanism you can also add your own before and after monitoring functions using pluggy.PluginManager.add_hookcall_monitoring().

The expected signature and default implementations for these functions is:

def before(hook_name, methods, kwargs):
    pass

def after(outcome, hook_name, methods, kwargs):
    pass

Public API

Please see the Api Reference.

Development

Great care must taken when hacking on pluggy since multiple mature projects rely on it. Our Github integrated CI process runs the full tox test suite on each commit so be sure your changes can run on all required Python interpreters and pytest versions.

Release Policy

Pluggy uses Semantic Versioning. Breaking changes are only foreseen for Major releases (incremented X in “X.Y.Z”). If you want to use pluggy in your project you should thus use a dependency restriction like "pluggy>=0.1.0,<1.0" to avoid surprises.