(Belatedly) Announcing pymmcore-plus: an ecosystem of pure-python tools for running experiments with micro-manager core

[talley]

NOTE: cross-posting this here from the image.sc forum as it may be of interest to this community as well

Hi All

We’ve been working on the pymmcore-plus/napari-micromanager ecosystem for a couple years now, but never really made any sort of public announcements. Thanks to the efforts of many people (notably @Federico_Gasparoli and Ian Hunt-Isaak, with great input from many others) I think the toolset is now pretty feature rich and relatively stable with a good amount of testing behind it.

Since many may still be unaware of the libraries, I’d like to give a brief overview and welcome you to give it a try if you want to drive microscopes in a pure python/C++ environment while integrating micro-manager’s device adapters (without requiring Java or any cross-process communication).

pymmcore(-plus)

When we started on napari-micromanager, we wanted to directly communicate with the C++ core of micromanager, so we began with pymmcore: the official python bindings for MMCore. If you’ve used it in the past, you may have felt that it lacked some developer niceties (such as type hinting and IDE autocompletion). We upstreamed type stubs that now give you a nice developer UX (this is the case regardless of whether you use pymmcore directly or pymmcore-plus):

However, there were still many things that were limiting us in an event-loop-based environment like napari/Qt, and we also wanted to be able to execute multi-dimensional experiments similar to the micro-manager application (the logic of which was all implemented in originally in clojure and later in Java, rather than C++), so we started building pymmcore-plus: a library of utilities on top of pymmcore that adds all of the stuff we found we needed. The main object there is pymmcore_plus.CMMCorePlus, a subclass of pymmcore.CMMCore. That library has reached a nice level of maturity, and now includes:

• A full-featured and customizeable multi-dimensional acquisition engine implemented in pure python
• More flexible event handling and core-monitoring capabilities (particularly useful if you want to build GUIs on top of it)
• extended core documentation, so you don’t have to browse the Java docs or C++ doxygen docs when working in a python world.
• generally more pythonic feel and object oriented APIs (e.g. implementing a proper MutableMapping interface for pymmcore’s Configuration objects, etc…).

note: we have chosen not to rename all of the camelCase names to their snake_case equivalents. Similar to other python C-extensions, like Qt, we find that it often gets even more confusing to remember which “regime” you’re working in, so we just stick with the MMCore API.

Acquisition engine and useq-schema

In particular, I’d like to draw your attention to the multidimensional acquisition engine included in pymmcore-plus, which offers both high level and low level control and execution of a sequence of events.

In order to better define exactly what an experiment is (before worrying about exactly how to execute it), we built useq-schema: a set of types to declaratively define a sequence of events to be executed on a microscope. In YAML, an example experiment might look like this (but there are many ways to define it):

channels:
   - config: DAPI
     exposure: 1.0
grid_plan:
  columns: 2
  rows: 2
stage_positions:
   - x: 1.0
     y: 1.0
     z: 1.0
time_plan:
  interval: '0:00:00.100000'
  loops: 2
z_plan:
  range: 3.0
  step: 1.0

pymmcore-plus implements support for that specification, which means that you can define, load, and run a full experiment from python code or a YAML/JSON file:

from pymmcore_plus import CMMCorePlus
import useq

mmc = CMMCorePlus.instance()
mmc.loadSystemConfiguration()

# create a sequence, here directly in python
mda_sequence = useq.MDASequence(
    time_plan={"interval": 2, "loops": 10},
    z_plan={"range": 4, "step": 0.5},
    channels=[
        {"config": "DAPI", "exposure": 50},
        {"config": "FITC", "exposure": 20},
    ]
)

# Run it!
mmc.run_mda(mda_sequence)

At runtime, that MDASequence object behaves like an iterable of event objects that the pymmcore-plus acquisition engine consumes and executes.

side note, any acquisition scheme, (such as ImSwitch, python-microscope, or the java half of micro-manager etc…) could conceivable adopt the useq-schema specification) which would make it easier to adopt/swap different engines.

engine customization

If the high-level interface isn’t flexible enough for your needs, you can fully customize the acquisition engine to your needs to, for example:

• write and integrate python drivers for hardware for which a micro-manager device adapter does not exist (without losing the ability to drive devices that micro-manager does control)
• generally execute any arbitrary python code in the same process as the C++ engine itself, with direct access to images and data as it comes from the engine.
• you could even pick and choose what parts of the hardware you want micromanager to drive, for example to swap in a high-performance camera driver

technically, the MDA Runner doesn’t even need to be controlling micro-manager’s core. It passes an iterable of useq-schema MDAEvents to an MDAEngine that knows how to execute each event. Our reference MDAEngine drives the micro-manager MMCore object, but you can customize as well if desired.

event customization

There are a number of different ways to generate/modify events in realtime, in response to acquisition data, for event-driven acquisition (aka “smart microscopy”). You can use python generators, queue.Queue objects, or any Iterable[MDAEvent] pattern that you like. Many thanks to Willi Stepp and Kyle Douglass for fruitful discussions on that topic.

pymmcore-widgets

Doing this all in a pure-python environment of course comes at the cost of losing MMStudio (micro-manager’s Java/ImageJ GUI). So we’ve been building a set of Qt widgets to control pymmcore-plus, and we’ve made a lot of progress towards parity with the controls available in the Java GUI, including:

• device property browser
• a full hardware configuration wizard
• configuration group editors/selectors
• multi-dimensional acquisition window
• most of the other little buttons like snap/live/stage/roi-control, etc…

These widgets generally stay in sync with the state of the underlying C++ core object thanks to the event/signals system in pymmcore-plus. You can pick and arrange these widgets however you choose to make your own custom pure-python micro-manager GUI.

Notably, the MDA widgets output useq-schema objects: so even if you don’t care about micro-manager, you could conceivably take advantage of the MDA acquisition GUI if your downstream processing implements support for useq-schema.

napari-micromanager

The highest level project (the thing that started this all) is napari-micromanager. It is essentially little more than an “opinionated” arrangement of the widgets in pymmcore-widgets, with the events from the pymmcore-plus acquisition engine connected to the napari viewer:

Admittedly, this is the project that has received the least amount of direct attention, as we’ve been focusing on building the lower level parts in a composable way that should be useful beyond a napari context. But it works in many ways! and we welcome more people to give it a try if they already use micro-manager on their scopes (you can use the same config file).

features not yet implemented

One of the main things you’ll find lacking at the moment is any sort of file IO/writer specification in pymmcore-plus itself. While napari-micromanager does have basic support for writing the experiment out to zarr and/or tiff, this is an area that could use more attention in the future. You can, always connect directly to the frameReady events coming of the engine to write the data to disk or otherwise process the data as best suits your needs. This is something we do plan to add, but given the huge variety of experimental needs and existing solutions, we really don’t want to recreate the wheel or impose a specific set of constraints.

feedback

If you are looking to drive microscopes in a pure-python environment, and want to be able to integrate micro-managers rich set of device adapters, we hope you’ll give it a try. As always feedback is greatly appreciated. If you find bugs, have questions, or would like to request features, feel free to open an issue