Docs / Marionette Collective

Marionette Collective

The Marionette Collective, also known as MCollective, is a framework for building server orchestration or parallel job execution systems. Most people use it to programmatically execute administrative tasks on clusters of servers.

MCollective has some unique strengths for working with large numbers of servers:

  • Instead of relying on a static list of hosts to command, it uses metadata-based discovery and filtering. It can use a rich data source like PuppetDB, or can do real-time discovery across the network.
  • Instead of directly connecting to each host (which can be resource-intensive and slow), it uses publish/subscribe middleware to communicate in parallel with many hosts at once.

To get an immediate feel for what this means, check out the videos on the Screencasts page. Then, keep reading below for further info and links.

We’ve also created a Vagrant-based demo, where you can easily experiment with MCollective.

What is MCollective and what does it allow you to do

  • Interact with small to very large clusters of servers
  • Use a broadcast paradigm for request distribution. All servers get all requests at the same time, requests have filters attached and only servers matching the filter will act on requests. There is no central asset database to go out of sync, the network is the only source of truth.
  • Break free from ever more complex naming conventions for hostnames as a means of identity. Use a very rich set of meta data provided by each machine to address them. Meta data comes from Puppet, Facter, or other sources.
  • Comes with simple to use command line tools to call remote agents.
  • Ability to write custom reports about your infrastructure.
  • A number of agents to manage packages, services and other common components are available from the community.
  • Allows you to write simple RPC style agents, clients and Web UIs in an easy to understand language - Ruby
  • Extremely pluggable and adaptable to local needs
  • Middleware systems already have rich authentication and authorization models, leverage these as a first line of control. Include fine grained Authentication using SSL or RSA, Authorization and Auditing of requests. You can see more details in the Security Overview.
  • Re-use the ability of middleware to do clustering, routing and network isolation to realize secure and scalable setups.

Pluggable Core

We aim to provide a stable core framework that allows you to build it out into a system that meets your own needs, we are pluggable in the following areas:

  • Replace our choice of middleware - STOMP compliant middleware - with your own like AMQP based.
  • Replace our authorization system with one that suits your local needs
  • Replace our serialization - Ruby Marshal and YAML based - with your own like JSONSchema that is cross language.
  • Add sources of data - using Puppet’s data is easy, and you can configure or build plugins for other data sources.
  • Create a central inventory of services leveraging MCollective as transport that can run and distribute inventory data on a regular basis.

MCollective is licensed under the Apache 2 license.

Next Steps and Further Reading

Introductory and Tutorial Pages

Internal References and Developer Docs

  • Finding it hard to do something complex with Simple RPC? See WritingAgents for what lies underneath
  • Role based security, authentication and authorization using ActiveMQ
  • Structure of Request and Reply messages

↑ Back to top